Abstract
Automated image de-fencing is an important area of computer vision that deals with the problem of virtually removing fence structures, if any, from images and produce aesthetically pleasing images without the fence structures. Unlike most of the previous de-fencing approaches that employ a two-stage process of fence mask detection followed by image inpainting, here we present a single-stage end-to-end conditional generative adversarial network-based de-fencing model that takes as input a fenced image and produces the corresponding de-fenced image in only 16 ms. The proposed network has been trained using an extensive dataset of fenced and ground-truth de-fenced image pairs by employing a combination of adversarial loss, L1 loss, perceptual loss, and estimated fence mask loss till convergence. The experimental results shows that our approach is capable of successfully handling images with even broken, irregular, and occluded fence structures. Qualitative and quantitative comparative study with previous de-fencing methods also show that our approach outperforms these existing techniques in terms of both response time and quality of de-fencing.
Similar content being viewed by others
References
Liu, Y., Belkina, T., Hays, J., Lublinerman, R.: Image de-fencing. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)
Farid, M.S., Mahmood, A., Grangetto, M.: Image de-fencing framework with hybrid inpainting algorithm. Signal Image Video Process. 10(7), 1193–1201 (2016)
Kumar, V., Mukherjee, J., Mandal, S.K.D.: Image defencing via signal demixing. In: Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, pp. 1–8 (2016)
Khalid, M., Yousaf, M.M., Murtaza, K., Sarwar, S.M.: Image de-fencing using histograms of oriented gradients. Signal Image Video Process. 12(6), 1173–1180 (2018)
Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)
Gupta, D., Jain, S., Tripathi, U., Chattopadhyay, P., Wang, L.: A robust and efficient image de-fencing approach using conditional generative adversarial networks. Signal Image Video Process. 15(2), 297–305 (2021)
Hays, J., Leordeanu, M., Efros, A.A., Liu, Y.: Discovering texture regularity as a higher-order correspondence problem. In: Proceedings of the European Conference on Computer Vision, pp. 522–535 (2006)
Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)
Park, M., Brocklehurst, K., Collins, R.T., Liu, Y.: Image de-fencing revisited. In: Proceedings of the Asian Conference on Computer Vision, pp. 422–434 (2010)
Kumar, V., Mukherjee, J., Mandal, S.K.D.: Combinatorial exemplar-based image inpainting. In: Proceedings of the Workshop on Combinatorial Image Analysis, pp. 284–298 (2015)
Kumar, V., Mukhopadhyay, J., Mandal, S.K.D.: Modified exemplar-based image inpainting via primal-dual optimization. In: Proceedings of the International Conference on Pattern Recognition and Machine Intelligence, pp. 116–125 (2015)
Shang, W., Zhu, P., Ren, D.: Semi-supervised learning to remove fences from a single image. In: Proceedings of the Chinese Conference on Pattern Recognition and Computer Vision, pp. 79–90 (2020)
Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 9446–9454 (2018)
Jonna, S., Nakka, K.K., Sahay, R.R.: My camera can see through fences: a deep learning approach for image de-fencing. In: Proceedings of the Asian Conference on Pattern Recognition, pp. 261–265 (2015)
Jonna, S., Nakka, K.K., Sahay, R.R.: Deep learning based fence segmentation and removal from an image using a video sequence. In: Proceedings of the European Conference on Computer Vision, pp. 836–851. Springer (2016)
Du, C., Kang, B., Xu, Z., Dai, J., Nguyen, T.: Accurate and efficient video de-fencing using convolutional neural networks and temporal information. In: Proceedings of the International Conference on Multimedia and Expo, pp. 1–6 (2018)
Negi, C.S., Mandal, K., Sahay, R.R., Kankanhalli, M.S.: Super-resolution de-fencing: simultaneous fence removal and high-resolution image recovery using videos. In: Proceedings of the Conference on Multimedia and Expo Workshops, pp. 1–6 (2014)
Jonna, S., Satapathy, S., Sahay, R.R.: Stereo image de-fencing using smartphones. In: Proceedings of the Conference on Acoustics, Speech and Signal Processing, pp. 1792–1796 (2017)
Yadong, M., Liu, W., Yan, S.: Video de-fencing. IEEE Trans. Circuits Syst. Video Technol. 24(7), 1111–1121 (2013)
Yi, R., Wang, J., Tan, P.: Automatic fence segmentation in videos of dynamic scenes. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 705–713 (2016)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Springer (2015)
Li, C., Wand, M.: Precomputed real-time texture synthesis with markovian generative adversarial networks. In: European conference on computer vision, pp. 702–716. Springer (2016)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Proceedings of the European Conference on Computer Vision, pp. 694–711 (2016)
Wen, L., Li, X., Li, X., Gao, L.: A new transfer learning based on VGG-19 network for fault diagnosis. In: 23rd International Conference on Computer Supported Cooperative Work in Design, pp. 205–209. IEEE (2019)
Bradley, D., Roth, G.: Adaptive thresholding using the integral image. J. Graph. Tools 12(2), 13–21 (2007)
Chen, Q., Xu, J., Koltun, V.: Fast image processing with fully-convolutional networks (2017)
Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks (2017)
Dosselmann, R., Yang, X.D.: A comprehensive assessment of the structural similarity index. Signal Image Video Process. 5(1), 81–91 (2011)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)
Acknowledgements
The authors would like to thank NVIDIA for supporting their research with a Titan Xp GPU, and all those who participated in the survey for computing the MOS.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Mishra, U., Agrawal, A., Mathew, J.C.R. et al. An efficient approach for image de-fencing based on conditional generative adversarial network. SIViP 17, 147–155 (2023). https://doi.org/10.1007/s11760-022-02215-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-022-02215-1