Skip to main content
SpringerLink
Log in

FF-GAN: Feature Fusion GAN for Monocular Depth Estimation

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12305))

Included in the following conference series:

  • 2624 Accesses

Abstract

Since the results of CNN methods for monocular depth estimation generally suffer the problem of visual dissatisfaction, we propose Feature Fusion GAN (FF-GAN) to address this issue. First, an end-to-end network based on encoder-decoder structure is proposed as the generator of FF-GAN, which can exploit the information of different scales. The encoder of our generator fuse features in different levels with a feature fusion module. The component which can obtain the information of multi-scale receptive field is the main part of the decoder of our generator. Second, in order to match the generator, the discriminator of FF-GAN is designed to efficiently learn the information of different scales by applying pyramid structure. Experiments on public datasets demonstrate the effectiveness of our generator and discriminator. Compared with the CNN methods, the results predicted by FF-GAN are significantly improved in terms of texture loss and edge blur while ensuring accuracy, and the visual effect is better.

This work is supported by Science and Technology Application Innovation Research Project of Zhejiang Provincial Public Security Department (2019TJYYCX007) and North China University of Technology Student Science and Technology Activities Project Funding (218051360020XN114/004).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ha, H., Im, S., Park, J., et al.: High-quality depth from uncalibrated small motion clip. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 5413–5421. IEEE, Piscataway (2016)

    Google Scholar 

  2. Karsch, K., Liu, C., Kang, S.B.: Depth transfer: depth extraction from videos using nonparametric sampling. In: Hassner, T., Liu, C. (eds.) Dense Image Correspondences for Computer Vision, pp. 173–205. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-23048-1_9

    Chapter  Google Scholar 

  3. Godard, C., Aodha, O., Firman, M., et al.: Digging into self-supervised monocular depth estimation. In: International Conference on Computer Vision, pp. 3827–3837 (2019)

    Google Scholar 

  4. Laina, I., Rupprecht, C., Belagiannis, V., et al.: Deeper depth prediction with fully convolutional residual networks. In: Proceedings of IEEE Conference on 3D vision, pp. 239–248. IEEE, Piscataway (2016)

    Google Scholar 

  5. Zhuo, W., Salzmann, M., He, X.M., et al.: Indoor scene structure analysis for single image depth estimation. In: Conference on Computer Vision and Pattern Recognition, pp. 614–622. IEEE Press, New York (2015)

    Google Scholar 

  6. Liu, F.Y., Shen, C.H., Lin, G.S., et al.: Learning depth from single monocular images using deep convolutional neural field. IEEE Trans. Pattern Anal. Mach. Intell. 38(10), 2024–2039 (2016)

    Article  Google Scholar 

  7. Yang, N., Wang, R., Stückler, J., Cremers, D.: Deep virtual stereo odometry: leveraging deep depth prediction for monocular direct sparse odometry. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 835–852. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01237-3_50

    Chapter  Google Scholar 

  8. Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv: 1411.1784v1. https://arxiv.org/abs/1411.1784. Accessed 20 April 2020

  9. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33715-4_54

    Chapter  Google Scholar 

  10. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving the KITTI vision benchmark suite. In: Conference on Computer Vision and Pattern Recognition, pp. 3354–3361. IEEE Press, New York (2012)

    Google Scholar 

  11. Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. In: Proceedings of International Conference on Neural Information Processing Systems, pp. 2366–2374 (2014)

    Google Scholar 

  12. Liao, B., Li, H.W.: Image depth estimation model based on atrous convolutional neural network. J. Comput. Appl. 39(1), 267–274 (2019)

    Google Scholar 

  13. He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778. IEEE, Piscataway (2016)

    Google Scholar 

  14. Ranjan, A., Jampani, V., Balles, L., et al.: Competitive collaboration: joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 12240–12249 (2019)

    Google Scholar 

  15. Goodfellow, I., Pouget-Abadie, J., Mirza, M., et al.: Generative adversarial nets. In: International Conference on Neural Information Processing Systems, pp. 2672–2680 (2014)

    Google Scholar 

  16. Isola, P., Zhu, J.Y., Zhou, T.H., et al.: Image-to-image translation with conditional adversarial networks. In: Conference on Computer Vision and Pattern Recognition, pp. 5967–5976 (2017)

    Google Scholar 

  17. Zhu, J.Y., Park, T., Isola, P., et al.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: International Conference on Computer Vision, pp. 2242–2251 (2017)

    Google Scholar 

  18. Mejjati, Y.A., Richardt, C., Tompkin, J., et al.: Unsupervised attention-guided image-to-image translation. In: The 32nd Annual Conference on Neural Information Processing Systems, pp. 3693–3703 (2018)

    Google Scholar 

  19. Chen, L., Papandreou, G., Schroff, F., et al.: Rethinking atrous convolution for semantic image segmentation. arXiv Preprint: 1706.05587 (2017)

    Google Scholar 

  20. Liu, W., et al.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2

    Chapter  Google Scholar 

  21. Ma, F., Karaman, S.: Sparse-to-dense: depth prediction from sparse depth samples and a single image. In: International Conference on Robotics and Automation, pp. 1–8. IEEE Press, New York (2018)

    Google Scholar 

  22. Blau, Y., Mechrez, R., Timofte, R., Michaeli, T., Zelnik-Manor, L.: The 2018 PIRM challenge on perceptual image super-resolution. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 334–355. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_21

    Chapter  Google Scholar 

  23. Ma, C., Yang, C., Yang, X., et al.: Learning a no-reference quality metric for single-image super-resolution. Comput. Vis. Image Underst. 158(1), 1–16 (2017)

    Article  MathSciNet  Google Scholar 

  24. Mittal, A., Soundararajan, R., Bovik, A.C., et al.: Making a completely blind image quality analyzer. IEEE Sign. Process. Lett. 20(3), 209–212 (2013)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Information Science and Technology, North China University of Technology, Beijing, China

    Ruiming Jia & Tong Li

  2. Digital Content Technology and Media Service Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Fei Yuan

Authors
  1. Ruiming Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fei Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ruiming Jia .

Editor information

Editors and Affiliations

  1. Peking University, Beijing, China

    Yuxin Peng

  2. Nanjing University of Information Science and Technology, Nanjing, China

    Qingshan Liu

  3. Dalian University of Technology, Dalian, China

    Huchuan Lu

  4. Chinese Academy of Sciences, Beijing, China

    Zhenan Sun

  5. Chinese Academy of Sciences, Beijing, China

    Chenglin Liu

  6. Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

    Xilin Chen

  7. Peking University, Beijing, China

    Hongbin Zha

  8. Nanjing University of Science and Technology, Nanjing, China

    Jian Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jia, R., Li, T., Yuan, F. (2020). FF-GAN: Feature Fusion GAN for Monocular Depth Estimation. In: Peng, Y., et al. Pattern Recognition and Computer Vision. PRCV 2020. Lecture Notes in Computer Science(), vol 12305. Springer, Cham. https://doi.org/10.1007/978-3-030-60633-6_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-60633-6_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-60632-9

  • Online ISBN: 978-3-030-60633-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation