Skip to main content
SpringerLink
Log in

Adaptive disparity computation using local and non-local cost aggregations

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

A new method is proposed to adaptively compute the disparity of stereo matching by choosing one of the alternative disparities from local and non-local disparity maps. The initial two disparity maps can be obtained from state-of-the-art local and non-local stereo algorithms. Then, the more reasonable disparity is selected. We propose two strategies to select the disparity. One is based on the magnitude of the gradient in the left image, which is simple and fast. The other utilizes the fusion move to combine the two proposal labelings (disparity maps) in a theoretically sound manner, which is more accurate. Finally, we propose a texture-based sub-pixel refinement to refine the disparity map. Experimental results using Middlebury datasets demonstrate that the two proposed selection strategies both perform better than individual local or non-local algorithms. Moreover, the proposed method is compatible with many local and non-local algorithms that are widely used in stereo matching.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Bleyer M, Rhemann C, Rother C (2011) Patchmatch stereo - stereo matching with slanted support windows: In: British machine vision conference, pp 14.1–14.11

  2. Boykov Y, Veksler O, Zabih R (2001) Fast approximate energy minimization via graph cuts. IEEE Trans Pattern Anal Mach Intell 23(11):1222–1239

    Article  Google Scholar 

  3. Brockers R, Hund M, Mertsching B (2005) Stereo vision using cost-relaxation with 3d support regions. Cortex 9:11

    Google Scholar 

  4. Crouzil A, Massip-Pailhes L, Castan S (1996) A new correlation criterion based on gradient fields similarity. In: International conference on pattern recognition, vol 1. IEEE, pp 632–636

  5. Drouyer S, Beucher S, Bilodeau M, Moreaud M, Sorbier L (2017) Sparse stereo disparity map densification using hierarchical image segmentation. In: International symposium on mathematical morphology and its applications to signal and image processing, pp 172–184

    Google Scholar 

  6. Ghaleb FF, Youness EA, Elmezain M, Dewdar FS (2015) Vision-based hand gesture spotting and recognition using crf and svm. J Softw Eng Appl 8(07):313

    Article  Google Scholar 

  7. Hirschmuller H (2005) Accurate and efficient stereo processing by semi-global matching and mutual information. In: Computer vision and pattern recognition, vol 2. IEEE, pp 807–814

  8. Huang X, Yuan C, Zhang J (2015) Graph cuts stereo matching based on patch-match and ground control points constraint. In: Pacific rim conference on multimedia, Springer, pp 14–23

  9. Jiao J, Wang R, Wang W, Dong S, Wang Z, Gao W (2014) Local stereo matching with improved matching cost and disparity refinement. IEEE MultiMedia 21 (4):16–27

    Article  Google Scholar 

  10. Kim KR, Kim CS (2016) Adaptive smoothness constraints for efficient stereo matching using texture and edge information. In: IEEE International conference on image processing, pp 3429–3433

  11. Kolmogorov V, Rother C (2007) Minimizing nonsubmodular functions with graph cuts-a review. IEEE Trans Pattern Anal Mach Intell 29(7):1274

    Article  Google Scholar 

  12. Kolmogorov V, Zabih R (2001) Computing visual correspondence with occlusions using graph cuts. In: IEEE Conference on computer vision, vol 2. IEEE, pp 508–515

  13. Kong D, Tao H (2004) A method for learning matching errors for stereo computation. In: British machine vision conference, vol 1, p 2

  14. Lempitsky V, Rother C, Roth S, Blake A (2010) Fusion moves for markov random field optimization. IEEE Trans Pattern Anal Mach Intell 32(8):1392

    Article  Google Scholar 

  15. Li L, Zhang S, Yu X, Zhang L (2016) Pmsc: Patchmatch-based superpixel cut for accurate stereo matching. IEEE Transactions on Circuits and Systems for Video Technology

  16. Li L, Yu X, Zhang S, Zhao X, Zhang L (2017) 3d cost aggregation with multiple minimum spanning trees for stereo matching. Applied Optics

  17. Mei X, Sun X, Zhou M, Jiao S, Wang H, Zhang X (2011) On building an accurate stereo matching system on graphics hardware. In: IEEE Conference on computer vision, IEEE, pp 467–474

  18. Mei X, Sun X, Dong W, Wang H, Zhang X (2013) Segment-tree based cost aggregation for stereo matching. In: Computer vision and pattern recognition, pp 313–320

  19. Miyazaki D, Matsushita Y, Ikeuchi K (2009) Interactive shadow removal from a single image using hierarchical graph cut pp 234–245

  20. Mizukami Y, Okada K, Nomura A, Nakanishi S (2012) Sub-pixel disparity search for binocular stereo vision. In: International conference on pattern recognition, pp 364–367

  21. Narducci F, Ricciardi S, Vertucci R (2016) Enabling consistent hand-based interaction in mixed reality by occlusions handling. Multimedia Tools and Applications 75(16):9549–9562

    Article  Google Scholar 

  22. Ogawara K (2010) Approximate belief propagation by hierarchical averaging of outgoing messages. In: International conference on pattern recognition, pp 1368–1372

  23. Olsson C, Ulen J, Boykov Y (2013) In defense of 3d-label stereo. In: Computer vision and pattern recognition, pp 1730–1737

  24. Ošep A, Hermans A, Engelmann F, Klostermann D, Mathias M, Leibe B (2016) Multi-scale object candidates for generic object tracking in street scenes. In: 2016 ieee international conference on Robotics and automation (icra), IEEE, pp 3180–3187

  25. Park H, Lee KM (2016) Look wider to match image patches with convolutional neural networks. IEEE Signal Processing Letters

  26. Park M, Yoon K (2016) As-planar-as-possible depth map estimation. IEEE Transactions Pattern Anal

  27. Peng Y, Li G, Wang R, Wang W (2015) Stereo matching with space-constrained cost aggregation and segmentation-based disparity refinement. In: Three-dimensional image processing, measurement (3DIPM), and applications, p 939309

  28. Psota ET, Kowalczuk J, Mittek M, Prez LC (2016) Map disparity estimation using hidden markov trees. In: IEEE International conference on computer vision

  29. Rameau F, Ha H, Joo K, Choi J, Park K, Kweon IS (2016) A real-time augmented reality system to see-through cars. IEEE Trans Vis Comput Graph 22(11):2395–2404

    Article  Google Scholar 

  30. Rhemann C, Hosni A, Bleyer M, Rother C, Gelautz M (2011) Fast cost-volume filtering for visual correspondence and beyond. In: Computer vision and pattern recognition, pp 3017–3024

  31. Scharstein D (1994) Matching images by comparing their gradient fields. International conference on pattern recognition, vol 1. IEEE, pp 572–575

  32. Scharstein D, Szeliski R (2002) A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int J Comput Vis 47(1-3):7–42

    Article  Google Scholar 

  33. Scharstein D, Hirschmüller H, Kitajima Y, Krathwohl G, Nešić N, Wang X, Westling P (2014) High-resolution stereo datasets with subpixel-accurate ground truth. In: German conference on pattern recognition, Springer, pp 31–42

  34. Shu X, Qi GJ, Tang J, Wang J (2015) Weakly-shared deep transfer networks for heterogeneous-domain knowledge propagation. In: ACM International conference on multimedia, pp 35–44

  35. Suarez J, Murphy RR (2012) Hand gesture recognition with depth images: a review. In: Ro-man, 2012 IEEE, IEEE, pp 411–417

  36. Tan P, Monasse P (2014) Stereo disparity through cost aggregation with guided filter. Image Processing on Line, 4:252–275. https://doi.org/10.5201/ipol.2014.78

    Article  Google Scholar 

  37. Taniai T, Matsushita Y, Naemura T (2014) Graph cut based continuous stereo matching using locally shared labels. In: Computer vision and pattern recognition, pp 1613–1620

  38. Taniai T, Matsushita Y, Sato Y, Naemura T (2016) Continuous stereo matching using local expansion moves. Computer Vision and Pattern Recognition

  39. Tian Y, Long Y, Xia D, Yao H, Zhang J (2015) Handling occlusions in augmented reality based on 3d reconstruction method. Neurocomputing 156:96–104

    Article  Google Scholar 

  40. Ummenhofer B, Zhou H, Uhrig J, Mayer N, Ilg E, Dosovitskiy A, Brox T (2017) Demon: Depth and motion network for learning monocular stereo. In: IEEE Conference on computer vision and pattern recognition (CVPR), vol 5

  41. Vu DT, Chidester B, Yang H, Do MN, Lu J (2014) Efficient hybrid tree-based stereo matching with applications to postcapture image refocusing. IEEE Trans Image Process 23(8):3428–3442

    Article  MathSciNet  Google Scholar 

  42. Wang L, Yang R, Gong M, Liao M (2014) Real-time stereo using approximated joint bilateral filtering and dynamic programming. J Real-Time Image Proc 9(3):447–461

    Article  Google Scholar 

  43. Woodford OJ, Torr PHS, Reid ID, Fitzgibbon AW (2008) Global stereo reconstruction under second order smoothness priors. In: Computer vision and pattern recognition, pp 1–8

  44. Yang Q (2012) A non-local cost aggregation method for stereo matching. In: Computer vision and pattern recognition, IEEE, pp 1402–1409

  45. Ye X, Li J, Wang H, Huang H, Zhang X (2017) Efficient stereo matching leveraging deep local and context information. IEEE Access

  46. Yoon KJ, Kweon IS (2006) Adaptive support-weight approach for correspondence search. IEEE Trans Pattern Anal Mach Intell 28(4):650–656

    Article  Google Scholar 

  47. Yu T, Lin RS, Super B, Tang B (2007) Efficient message representations for belief propagation. In: IEEE Conference on computer vision, IEEE, pp 1–8

  48. Zabih R, Woodfill J (1994) Non-parametric local transforms for computing visual correspondence. In: European conference on computer vision, Springer, pp 151–158

  49. Zbontar J, LeCun Y (2016) Stereo matching by training a convolutional neural network to compare image patches. J Mach Learn Res 17:1–32

    MATH  Google Scholar 

  50. Zhan Y, Gu Y, Huang K, Zhang C, Hu K (2015) Accurate image-guided stereo matching with efficient matching cost and disparity refinement. IEEE Transactions on Circuits and Systems for Video Technology

  51. Zhang C, Li Z, Cheng Y, Cai R (2015) Meshstereo: a global stereo model with mesh alignment regularization for view interpolation. In: IEEE International conference on computer vision, pp 2057–2065

  52. Zhang K, Lu J, Lafruit G (2009) Cross-based local stereo matching using orthogonal integral images. IEEE Trans Circuits Syst Video Technol 19(7):1073–1079

    Article  Google Scholar 

  53. Zhou C, Zhang H, Shen X, Jia J (2017) Unsupervised learning of stereo matching. In: IEEE International conference on computer vision, pp 1576–1584

  54. Zhou T, Brown M, Snavely N, Lowe DG (2017) Unsupervised learning of depth and ego-motion from video pp 6612–6619

Download references

Acknowledgments

The authors would like to thank Qing Ran for her instructive discussion of this paper. This work was supported by the National Natural Science Foundation of China under Grants Nos. 61732015 and 61472349.

Author information

Authors and Affiliations

  1. State Key Lab of CAD, CG, Zhejiang University, Hangzhou, 310058, China

    Qicong Dong & Jieqing Feng

Authors
  1. Qicong Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jieqing Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jieqing Feng.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dong, Q., Feng, J. Adaptive disparity computation using local and non-local cost aggregations. Multimed Tools Appl 77, 31647–31663 (2018). https://doi.org/10.1007/s11042-018-6236-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6236-6

Keywords

Search

Navigation