Advertisement

Wavelet-Based Dual-Branch Network for Image Demoiréing

Conference paper
  • 568 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12358)

Abstract

When smartphone cameras are used to take photos of digital screens, usually moiré patterns result, severely degrading photo quality. In this paper, we design a wavelet-based dual-branch network (WDNet) with a spatial attention mechanism for image demoiréing. Existing image restoration methods working in the RGB domain have difficulty in distinguishing moiré patterns from true scene texture. Unlike these methods, our network removes moiré patterns in the wavelet domain to separate the frequencies of moiré patterns from the image content. The network combines dense convolution modules and dilated convolution modules supporting large receptive fields. Extensive experiments demonstrate the effectiveness of our method, and we further show that WDNet generalizes to removing moiré artifacts on non-screen images. Although designed for image demoiréing, WDNet has been applied to two other low-level vision tasks, outperforming state-of-the-art image deraining and deraindrop methods on the Rain100h and Raindrop800 data sets, respectively.

Keywords

Deep learning Image demoiréing Wavelet 

Supplementary material

504454_1_En_6_MOESM1_ESM.pdf (36.8 mb)
Supplementary material 1 (pdf 37650 KB)

References

  1. 1.
    Bao, L., Song, Y., Yang, Q., Yuan, H., Wang, G.: Tree filtering: efficient structure-preserving smoothing with a minimum spanning tree. TIP 23, 555–569 (2013)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Bell, S., Lawrence Zitnick, C., Bala, K., Girshick, R.: Inside-outside net: detecting objects in context with skip pooling and recurrent neural networks. In: CVPR (2016)Google Scholar
  3. 3.
    Blau, Y., Michaeli, T.: The perception-distortion tradeoff. In: CVPR (2018)Google Scholar
  4. 4.
    Chang, Y., Yan, L., Zhong, S.: Transformed low-rank model for line pattern noise removal. In: ICCV (2017)Google Scholar
  5. 5.
    Eigen, D., Krishnan, D., Fergus, R.: Restoring an image taken through a window covered with dirt or rain. In: CVPR (2013)Google Scholar
  6. 6.
    Fu, J., et al.: Dual attention network for scene segmentation. In: CVPR (2019)Google Scholar
  7. 7.
    Fujieda, S., Takayama, K., Hachisuka, T.: Wavelet convolutional neural networks for texture classification. arXiv preprint arXiv:1707.07394 (2017)
  8. 8.
    Gao, F., Huang, T., Wang, J., Sun, J., Hussain, A., Yang, E.: Dual-branch deep convolution neural network for polarimetric SAR image classification. Appl. Sci. 7, 447 (2017)CrossRefGoogle Scholar
  9. 9.
    Gharbi, M., Chaurasia, G., Paris, S., Durand, F.: Deep joint demosaicking and denoising. TOG 35, 1–12 (2016)CrossRefGoogle Scholar
  10. 10.
    Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 3rd edn. Prentice-Hall, Upper Saddle River (2007)Google Scholar
  11. 11.
    Gueguen, L., Sergeev, A., Kadlec, B., Liu, R., Yosinski, J.: Faster neural networks straight from JPEG. In: NeurIPS (2018)Google Scholar
  12. 12.
    Guo, T., Cherukuri, V., Monga, V.: Dense123’color enhancement dehazing network. In: CVPRW (2019)Google Scholar
  13. 13.
    Ham, B., Cho, M., Ponce, J.: Robust image filtering using joint static and dynamic guidance. In: CVPR (2015)Google Scholar
  14. 14.
    He, B., Wang, C., Shi, B., Duan, L.Y.: Mop moire patterns using MopNet. In: ICCV (2019)Google Scholar
  15. 15.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)Google Scholar
  16. 16.
    Hu, X., Fu, C.W., Zhu, L., Heng, P.A.: Depth-attentional features for single-image rain removal. In: CVPR (2019)Google Scholar
  17. 17.
    Hu, X., Fu, C.W., Zhu, L., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection and removal. TPAMI (2019)Google Scholar
  18. 18.
    Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR (2017)Google Scholar
  19. 19.
    Huang, H., He, R., Sun, Z., Tan, T.: Wavelet-SRNet: a wavelet-based CNN for multi-scale face super resolution. In: ICCV (2017)Google Scholar
  20. 20.
    Isobe, T., et al.: Video super-resolution with temporal group attention. In: CVPR (2020)Google Scholar
  21. 21.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR (2017)Google Scholar
  22. 22.
    Jang, D.W., Park, R.H.: DenseNet with deep residual channel-attention blocks for single image super resolution. In: CVPRW (2019)Google Scholar
  23. 23.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_43CrossRefGoogle Scholar
  24. 24.
    Karacan, L., Erdem, E., Erdem, A.: Structure-preserving image smoothing via region covariances. TOG 32, 1–11 (2013)CrossRefGoogle Scholar
  25. 25.
    Kim, T.H., Park, S.I.: Deep context-aware descreening and rescreening of halftone images. TOG 37, 1–12 (2018)Google Scholar
  26. 26.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  27. 27.
    Lefkimmiatis, S.: Universal denoising networks: a novel CNN architecture for image denoising. In: CVPR (2018)Google Scholar
  28. 28.
    Levinskis, A.: Convolutional neural network feature reduction using wavelet transform. Elektronika ir Elektrotechnika 19, 61–64 (2013)CrossRefGoogle Scholar
  29. 29.
    Li, Q., Shen, L., Guo, S., Lai, Z.: Wavelet integrated CNNs for noise-robust image classification. In: CVPR (2020)Google Scholar
  30. 30.
    Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: ECCV (2018)Google Scholar
  31. 31.
    Lin, H., Fu, X., Jing, C., Ding, X., Huang, Y.: A\(^2\)Net: Adjacent aggregation networks for image raindrop removal. arXiv preprint arXiv:1811.09780 (2018)
  32. 32.
    Liu, B., Shu, X., Wu, X.: Demoiréing of camera-captured screen images using deep convolutional neural network. arXiv preprint arXiv:1804.03809 (2018)
  33. 33.
    Liu, F., Yang, J., Yue, H.: Moiré pattern removal from texture images via low-rank and sparse matrix decomposition. In: VCIP (2015)Google Scholar
  34. 34.
    Liu, L., Jia, X., Liu, J., Tian, Q.: Joint demosaicing and denoising with self guidance. In: CVPR (2020)Google Scholar
  35. 35.
    Liu, P., Zhang, H., Zhang, K., Lin, L., Zuo, W.: Multi-level wavelet-CNN for image restoration. In: CVPRW (2018)Google Scholar
  36. 36.
    Liu, Y., Li, Q., Sun, Z.: Attribute-aware face aging with wavelet-based generative adversarial networks. In: CVPR (2019)Google Scholar
  37. 37.
    Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: ICCV (2015)Google Scholar
  38. 38.
    Oyallon, E., Belilovsky, E., Zagoruyko, S.: Scaling the scattering transform: deep hybrid networks. In: ICCV (2017)Google Scholar
  39. 39.
    Qian, R., Tan, R.T., Yang, W., Su, J., Liu, J.: Attentive generative adversarial network for raindrop removal from a single image. In: CVPR (2018)Google Scholar
  40. 40.
    Qin, J., Xie, Z., Shi, Y., Wen, W.: Difficulty-aware image super resolution via deep adaptive dual-network. In: ICME (2019)Google Scholar
  41. 41.
    Quan, Y., Deng, S., Chen, Y., Ji, H.: Deep learning for seeing through window with raindrops. In: ICCV (2019)Google Scholar
  42. 42.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  43. 43.
    Sasada, R., Yamada, M., Hara, S., Takeo, H., Shimura, K.: Stationary grid pattern removal using 2D technique for moiré-free radiographic image display. In: Medical Imaging (2003)Google Scholar
  44. 44.
    Shen, L., Yue, Z., Chen, Q., Feng, F., Ma, J.: Deep joint rain and haze removal from a single image. In: ICPR (2018)Google Scholar
  45. 45.
    Si, J., et al.: Dual attention matching network for context-aware feature sequence based person re-identification. In: CVPR (2018)Google Scholar
  46. 46.
    Siddiqui, H., Boutin, M., Bouman, C.A.: Hardware-friendly descreening. TIP 19, 746–757 (2009)MathSciNetzbMATHGoogle Scholar
  47. 47.
    Sidorov, D.N., Kokaram, A.C.: Removing moire from degraded video archives. In: ESPC (2002)Google Scholar
  48. 48.
    Sidorov, D.N., Kokaram, A.C.: Suppression of moiré patterns via spectral analysis. In: VCIP (2002)Google Scholar
  49. 49.
    Su, Z., Luo, X., Deng, Z., Liang, Y., Ji, Z.: Edge-preserving texture suppression filter based on joint filtering schemes. TMM 15, 535–548 (2012)Google Scholar
  50. 50.
    Subr, K., Soler, C., Durand, F.: Edge-preserving multiscale image decomposition based on local extrema. TOG 28, 1–9 (2009)CrossRefGoogle Scholar
  51. 51.
    Sun, B., Li, S., Sun, J.: Scanned image descreening with image redundancy and adaptive filtering. TIP 23, 3698–3710 (2014)MathSciNetzbMATHGoogle Scholar
  52. 52.
    Sun, Y., Schaefer, S., Wang, W.: Image structure retrieval via \( l\_0 \) minimization. TVCG 24, 2129–2139 (2017)Google Scholar
  53. 53.
    Sun, Y., Yu, Y., Wang, W.: Moiré photo restoration using multiresolution convolutional neural networks. TIP 27, 4160–4172 (2018)zbMATHGoogle Scholar
  54. 54.
    Tai, Y., Yang, J., Liu, X., Xu, C.: MemNet: a persistent memory network for image restoration. In: ICCV (2017)Google Scholar
  55. 55.
    Wang, P., et al.: Understanding convolution for semantic segmentation. In: WACV (2018)Google Scholar
  56. 56.
    Wei, Z., Wang, J., Nichol, H., Wiebe, S., Chapman, D.: A median-Gaussian filtering framework for Moiré pattern noise removal from X-ray microscopy image. Micron 43, 170–176 (2012)CrossRefGoogle Scholar
  57. 57.
    Williams, T., Li, R.: Wavelet pooling for convolutional neural networks. In: ICLR (2018)Google Scholar
  58. 58.
    Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via L0 gradient minimization. TOG 30, 11 (2011)Google Scholar
  59. 59.
    Xu, L., Yan, Q., Xia, Y., Jia, J.: Structure extraction from texture via relative total variation. TOG 31, 1–10 (2012)Google Scholar
  60. 60.
    Xu, X., et al.: Rendering portraitures from monocular camera and beyond. In: ECCV (2018)Google Scholar
  61. 61.
    Yang, J., Liu, F., Yue, H., Fu, X., Hou, C., Wu, F.: Textured image demoiréing via signal decomposition and guided filtering. TIP 26, 3528–3541 (2017)zbMATHGoogle Scholar
  62. 62.
    Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: CVPR (2017)Google Scholar
  63. 63.
    Yoo, J., Uh, Y., Chun, S., Kang, B., Ha, J.W.: Photorealistic style transfer via wavelet transforms. In: CVPR (2019)Google Scholar
  64. 64.
    Yuan, S., Timofte, R., Leonardis, A., Slabaugh, G.: NTIRE 2020 challenge on image demoireing: methods and results. In: CVPRW (2020)Google Scholar
  65. 65.
    Yuan, S., Timofte, R., Slabaugh, G., Leonardis, A.: AIM 2019 challenge on image demoireing: dataset and study. In: ICCVW (2019)Google Scholar
  66. 66.
    Yuan, S., et al.: AIM 2019 challenge on image demoireing: methods and results. In: ICCVW (2019)Google Scholar
  67. 67.
    Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. TIP 26, 3142–3155 (2017)MathSciNetzbMATHGoogle Scholar
  68. 68.
    Zhang, Q., Shen, X., Xu, L., Jia, J.: Rolling guidance filter. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 815–830. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10578-9_53CrossRefGoogle Scholar
  69. 69.
    Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: CVPR (2018)Google Scholar
  70. 70.
    Zheng, B., Yuan, S., Slabaugh, G., Leonardis, A.: Image demoireing with learnable bandpass filters. In: CVPR (2020)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of Science and Technology of ChinaHefeiChina
  2. 2.Noah’s Ark Lab, Huawei TechnologiesShenzhenChina
  3. 3.Huawei Cloud BUBeijingChina

Personalised recommendations