Disparity Estimation Using Convolutional Neural Networks with Multi-scale Correlation

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10636)


Disparity estimation is a long-standing task in computer vision and multiple approaches have been proposed to solve this problem. A recent work based on convolutional neural networks, which uses a correlation layer to perform the matching process, has achieved state-of-the-art results for the disparity estimation task. This correlation layer employs a single kernel unit which is not suitable for low texture content and repeated patterns. In this paper we tackle this problem by using a multi-scale correlation layer with several correlation kernels and different scales. The major target is to integrate the information of the local matching process by combining the benefits of using both a small correlating scale for fine details and bigger scales for larger areas. Furthermore, we investigate the training approach using horizontally elongated patches that fits the disparity estimation task. The results obtained demonstrate the benefits of the proposed approach on both synthetic and real images.


Disparity estimation Convolutional neural networks Multi-scale correlation Stereo vision Depth estimation 


  1. 1.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002)CrossRefMATHGoogle Scholar
  2. 2.
    De-Maeztu, L., Mattoccia, S., Villanueva, A., Cabeza, R.: Linear stereo matching. In: Proceedings of the 2011 International Conference on Computer Vision, Barcelona, pp. 1708–1715 (2011)Google Scholar
  3. 3.
    Sun, J., Zheng, N.-N., Shum, H.-Y.: Stereo matching using belief propagation. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, TPAMI, vol. 25, no. 7, pp. 787–800 (2003)Google Scholar
  4. 4.
    Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information. IEEE Trans. Patt. Anal. Mach. Intell. 30(2), 328–341 (2008)CrossRefGoogle Scholar
  5. 5.
    Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., Navab, N.: Deeper depth prediction with fully convolutional residual networks. CoRR abs/1606.00373 (2016)Google Scholar
  6. 6.
    Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., Brox, T.: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 4040–4048 (2016)Google Scholar
  7. 7.
    Zagoruyko, S., Komodakis, N.: Learning to compare image patches via convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4353–4361 (2015)Google Scholar
  8. 8.
    Zbontar, J., LeCun, Y.: Computing the stereo matching cost with a convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1592–1599 (2014)Google Scholar
  9. 9.
    Luo, W., Schwing, A.G., Urtasun, R.: Efficient deep learning for stereo matching. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 5695–5703 (2016)Google Scholar
  10. 10.
    Zbontar, J., LeCun, Y.: Stereo matching by training a convolutional neural network to compare image patches. J. Mach. Learn. Res. 17, 4 (2016)MATHGoogle Scholar
  11. 11.
    Seki, A., Pollefeys, M.: Patch based confidence prediction for dense disparity map. In: Proceedings of the British Machine Vision Conference, BMVC, p. 23 (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.X’ian Jiaotong Liverpool UniversitySuzhouChina
  2. 2.University of Bozen-BolzanoBolzanoItaly

Personalised recommendations