Advertisement

Stereo Matching Using Conditional Adversarial Networks

  • Hualong Huang
  • Bo Huang
  • Hongtao Lu
  • Huiyu Weng
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10636)

Abstract

Recently, adversarial networks have attracted increasing attentions for the promising results of generative tasks. In this paper we present the first application of conditional adversarial networks to stereo matching task. Our approach performs a conditional adversarial training process on two networks: a generator that learns the mapping from a pair of RGB images to a dense disparity map, and a discriminator that distinguishes whether the disparity map comes from the ground truth or from the generator. Here, both the generator and the discriminator take the same RGB image pair as an input condition. During this conditional adversarial training process, our discriminator gradually captures high-level contextual features to detect inconsistencies between the ground truth and the generated disparity maps. These high-level contextual features are incorporated into loss function in order to further help the generator to correct predicted disparity maps. We evaluate our model on the Scene Flow dataset and an improvement is achieved compared with the most related work pix2pix.

Keywords

Stereo matching Conditional adversarial networks 

Notes

Acknowledgments

This paper is supported by NSFC(No.61772330, 61272247, 61533012, 61472075), the 863 National High Technology Re-search and Development Program of China (SS2015AA020501), the Basic Research Project of Innovation Action Plan (16JC1402800) and the Major Basic Research Program (15JC1400103) of Shanghai Science and Technology Committee.

References

  1. 1.
    Abadi, M., Agarwal, A., Barham, P., et al.: TensorFlow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016)
  2. 2.
    Bromley, J., Guyon, I., LeCun, Y., et al.: Signature verification using a s̈iameseẗime delay neural network. In: Advances in Neural Information Processing Systems, pp. 737–744 (1994)Google Scholar
  3. 3.
    Fua, P.: A parallel stereo algorithm that produces dense depth maps and preserves image features. Mach. Vis. Appl. 6, 35–49 (1993)CrossRefGoogle Scholar
  4. 4.
    Gidaris, S., Komodakis, N.: Detect, Replace, Refine: Deep Structured Prediction for Pixel Wise Labeling. arXiv preprint arXiv:1612.04770 (2016)
  5. 5.
    Girshick, R., Donahue, J., Darrell, T., et al.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)Google Scholar
  6. 6.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (2014)Google Scholar
  7. 7.
    Hirschmuller, H.: Accurate and efficient stereo processing by semiglobal matching and mutual information. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 807–814. IEEE (2005)Google Scholar
  8. 8.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
  9. 9.
    Isola, P., Zhu, J.Y., Zhou, T., et al.: Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:1611.07004 (2016)
  10. 10.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  11. 11.
    Kingma, D., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  12. 12.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)Google Scholar
  13. 13.
    Luc, P., Couprie, C., Chintala, S., et al.: Semantic Segmentation using Adversarial Networks. arXiv preprint arXiv:1611.08408 (2016)
  14. 14.
    Luo, W., Schwing, A.G., Urtasun, R.: Efficient deep learning for stereo matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5695–5703 (2016)Google Scholar
  15. 15.
    Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  16. 16.
    Mayer, N., Ilg, E., Hausser, P., et al.: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), arXiv:1512.02134 (2016)
  17. 17.
    Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1520–1528 (2015)Google Scholar
  18. 18.
    Park, M.G., Yoon, K.J.: Leveraging stereo matching with learning-based confidence measures. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 101–109 (2015)Google Scholar
  19. 19.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). doi: 10.1007/978-3-319-24574-4_28 CrossRefGoogle Scholar
  20. 20.
    Scharstein, D., Pal, C.: Learning conditional random fields for stereo. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2007)Google Scholar
  21. 21.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1), 7–42 (2002)CrossRefMATHGoogle Scholar
  22. 22.
    Seki, A., Pollefeys, M.: Patch based confidence prediction for dense disparity map. In: British Machine Vision Conference, 10 September 2016Google Scholar
  23. 23.
    Zbontar, J., LeCun, Y.: Computing the stereo matching cost with a convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1592–1599 (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Hualong Huang
    • 1
  • Bo Huang
    • 1
  • Hongtao Lu
    • 1
  • Huiyu Weng
    • 1
  1. 1.Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Department of Computer Science and EngineeringShanghai Jiao Tong UniversityShanghaiPeople’s Republic of China

Personalised recommendations