Skip to main content

Advertisement

SpringerLink
Log in

A new stereo matching energy model based on image local features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper constructs an energy model based on local features used in stereo matching. The local features include the similarity between different image areas, the matching cost function pattern, the connection between neighbor pixels, and the occlusion geometric relationship. Based on these features, we define the weight of each data term and smoothing term in the energy function and then design an algorithm to solve the energy model and get disparity results. The significant improvements of this paper include as following. 1) We modify the structure of the energy function. First, we define the weight of the data term based on the reliability of its corresponding disparity result, which is obtained by cost function features and the occlusion geometric relationship. Then we define the weight of the smoothing term by analyzing the characteristic relation between neighbor super-pixels. We can also reduce the computational complexity by detecting and reducing some low-strength connections. 2) We proposed an algorithm based on pairwise Markov random field (MRF) (Taniai et al., IEEE Trans Pattern Anal Machine Intell 40(11): 2725–2739, 2017) and local greedy iteratively, which can be used to solve the energy model. 3) In post-optimation, we select some areas with severe occlusion and fewer matching clues for post-interpolation fitting to optimize the results. The experiment shows that the proposed method reduced the average percentage of bad pixels (in bad 3) to 6.06 on the Middlebury dataset and 1.42 on the KITTI dataset. Finally, we compare our results with those of MC-Cnn (Zbontar and LeCun 2015), CF-Net (Shen et al., 2021), Guided-Stereo (Poggi et al., 2019), Gwc-Net (Guo et al., 2019) and Patchmatch-Net(PM-Net) (Wang et al., 2021) to verify the improved speed and accuracy of our algorithm, especially at recognizing the depth of changing edges and small objects. This paper’s relevant research can contribute to practical engineering practices such as assisted vision, intelligent driving, and robot grasping control.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Algorithm 1
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Algorithm 2
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available in “MiddleBury” at ”https://vision.middlebury.edu/stereo/data/” and in “KITTI” at “http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=stereo””.

References

  1. Bai C, Ma Q, Hao P, Liu Z, Zhang J (2018) Improving stereo matching algorithm with adaptive cross-scale cost aggregation. Int J Adv Robot Syst 15(1):1729881417751544

    Article  Google Scholar 

  2. Bleyer M, Rhemann C, Rother C (2011) Patchmatch stereo-stereo matching with slanted support windows. In: Bmvc, vol 11, pp 1–11

  3. Chang J-R, Chen Y-S (2018) Pyramid stereo matching network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5410–5418

  4. Chang Y-J, Ho Y-S (2017) Pixel-based adaptive normalized cross correlation for illumination invariant stereo matching. Electronic Imaging 2017(5):124–129

    Article  Google Scholar 

  5. Chang T-A, Lu X, Yang J-F (2017) Robust stereo matching with trinary cross color census and triple image-based refinements. EURASIP Journal on Advances in Signal Processing 2017(1):1–13

    Article  Google Scholar 

  6. Chen Q, Bise R, Gu L, Zheng Y, Sato I, Hwang J-N, Imanishi N, Aiso S (2017) Virtual blood vessels in complex background using stereo x-ray images. In: Proceedings of the IEEE international conference on computer vision workshops, pp 99–106

  7. Chen C, Seff A, Kornhauser A, Xiao J (2015) Deepdriving: Learning affordance for direct perception in autonomous driving. In: Proceedings of the IEEE international conference on computer vision, pp 2722–2730

  8. Cheng F, Zhang H, Yuan D, Sun M (2014) Stereo matching by using the global edge constraint. Neurocomputing 131:217–226

    Article  Google Scholar 

  9. Geiger A, Roser M, Urtasun R (2010) Efficient large-scale stereo matching. In: Asian Conference on computer vision. Springer, pp 25–38

  10. Gu X, Fan Z, Zhu S, Dai Z, Tan F, Tan P (2020) Cascade cost volume for high-resolution multi-view stereo and stereo matching. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2495–2504

  11. Guo X, Yang K, Yang W, Wang X, Li H (2019) Group-wise correlation stereo network. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3273–3282

  12. Hamid MS, Abd Manap N, Hamzah RA, Kadmin AF (2020) Stereo matching algorithm based on deep learning: A survey. Journal of King Saud University-Computer and Information Sciences

  13. Hamzah RA, Kadmin AF, Hamid MS, Ghani SFA, Ibrahim H (2018) Improvement of stereo matching algorithm for 3d surface reconstruction. Signal Process Image Commun 65:165–172

    Article  Google Scholar 

  14. Hirschmüller H, Innocent PR, Garibaldi J (2002) Real-time correlation-based stereo vision with reduced border errors. Int J Comput Vis 47(1):229–246

    Article  MATH  Google Scholar 

  15. Hosni A, Bleyer M, Gelautz M, Rhemann C (2009) Local stereo matching using geodesic support weights. In: 2009 16th IEEE international conference on image processing (ICIP). IEEE, pp 2093–2096

  16. Huang C-S, Huang Y-H, Chan D-Y, Yang J-F (2020) Shape-reserved stereo matching with segment-based cost aggregation and dual-path refinement. EURASIP Journal on Image and Video Processing 2020(1):1–19

    Article  Google Scholar 

  17. Jellal RA, Lange M, Wassermann B, Schilling A, Zell A (2017) Ls-elas: Line segment based efficient large scale stereo matching. In: 2017 IEEE international conference on robotics and automation (ICRA). IEEE, pp 146–152

  18. Jiao J, Yang Q, He S, Gu S, Zhang L, Lau RW (2017) Joint image denoising and disparity estimation via stereo structure pca and noise-tolerant cost. Int J Comput Vis 124(2):204–222

    Article  MathSciNet  Google Scholar 

  19. Jingui Z, Ying W, Liyuan M (2018) A new stereo matching algorithm based on adaptive weight sad algorithm and census algorithm. Bulletin of Surveying and Mapping (11),11

  20. Knyaz VA, Kniaz VV, Remondino F, Zheltov SY, Gruen A (2020) 3d reconstruction of a complex grid structure combining uas images and deep learning. Remote Sens 12(19):3128

    Article  Google Scholar 

  21. Kong L, Sun X, Rahman M, Xu M (2020) A 3d measurement method for specular surfaces based on polarization image sequences and machine learning. CIRP Ann 69(1):497–500

    Article  Google Scholar 

  22. Kong L, Zhu J, Ying S (2021) Local stereo matching using adaptive cross-region-based guided image filtering with orthogonal weights. Math Probl Eng 2021

  23. Lai H-Y, Tsai Y-H, Chiu W-C (2019) Bridging stereo matching and optical flow via spatiotemporal correspondence. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1890–1899

  24. Lee I, Moon B (2017) An improved stereo matching algorithm with robustness to noise based on adaptive support weight. J Inform Process Syst 13 (2):256–267

    Google Scholar 

  25. Li H, Li Z, Huang J, Meng B, Zhang Z (2021) Accurate hierarchical stereo matching based on 3d plane labeling of superpixel for stereo images from rovers. Int J Adv Robot Syst 18(2):17298814211002113

    Article  Google Scholar 

  26. Li L, Zhang S, Yu X, Zhang L (2016) Pmsc: Patchmatch-based superpixel cut for accurate stereo matching. IEEE Trans Circuits Syst Video Technol 28(3):679–692

    Article  Google Scholar 

  27. Lim J, Lee S (2018) Patchmatch-based robust stereo matching under radiometric changes. IEEE Trans Pattern Anal Machine Intell 41(5):1203–1212

    Article  Google Scholar 

  28. Lin C, Li Y, Xu G, Cao Y (2017) Optimizing zncc calculation in binocular stereo matching. Signal Process Image Commun 52:64–73

    Article  Google Scholar 

  29. Luo W, Schwing AG, Urtasun R (2016) Efficient deep learning for stereo matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5695–5703

  30. Mehltretter M, Heipke C (2019) Cnn-based cost volume analysis as confidence measure for dense matching. In: Proceedings of the IEEE/CVF international conference on computer vision workshops, pp 0–0

  31. Mozerov MG, Van De Weijer J (2019) One-view occlusion detection for stereo matching with a fully connected crf model. IEEE Trans Image Process 28 (6):2936–2947

  32. Odena A, Dumoulin V, Olah C (2016) Deconvolution and checkerboard artifacts. Distill 1(10):3

    Article  Google Scholar 

  33. Pang J, Sun W, Ren JS, Yang C, Yan Q (2017) Cascade residual learning: a two-stage convolutional neural network for stereo matching. In: Proceedings of the IEEE international conference on computer vision workshops, pp 887–895

  34. Poggi M, Pallotti D, Tosi F, Mattoccia S (2019) Guided stereo matching. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 979–988

  35. Shen Z, Dai Y, Rao Z (2021) Cfnet: Cascade and fused cost volume for robust stereo matching. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13906–13915

  36. Shi B, Shi S, Wu J, Chen M (2019) A new basic correlation measurement for stereo matching

  37. Sun J, Liu Y, Ding Y, Zhu X, Xi J (2018) Ncc feature matching optimized algorithm based on constraint fusion. In: 2018 IEEE 3rd international conference on image, vision and computing (ICIVC). IEEE, pp 336–341

  38. Taniai T, Matsushita Y, Sato Y, Naemura T (2017) Continuous 3d label stereo matching using local expansion moves. IEEE Trans Pattern Anal Machine Intell 40(11):2725–2739

    Article  Google Scholar 

  39. Wang F, Galliani S, Vogel C, Speciale P, Pollefeys M (2021) Patchmatchnet: Learned multi-view patchmatch stereo. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14194–14203

  40. Wang H, Pathan MS, Dev S (2021) Stereo matching based on visual sensitive information. In: 2021 6th international conference on image, vision and computing (ICIVC). IEEE, pp 312–316

  41. Wang J, Zickler T (2019) Local detection of stereo occlusion boundaries. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3818–3827

  42. Wu W, Zhu H, Yu S, Shi J (2019) Stereo matching with fusing adaptive support weights. IEEE Access 7:61960–61974

    Article  Google Scholar 

  43. Xu H, Zhang J (2020) Aanet: Adaptive aggregation network for efficient stereo matching. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1959–1968

  44. Xue T, Owens A, Scharstein D, Goesele M, Szeliski R (2019) Multi-frame stereo matching with edges, planes, and superpixels. Image Vis Comput 91:103771

    Article  Google Scholar 

  45. Yang Q (2012) A non-local cost aggregation method for stereo matching. In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 1402–1409

  46. Yang Q (2013) Hardware-efficient bilateral filtering for stereo matching. IEEE Trans Pattern Anal Machine Intell 36(5):1026–1032

    Article  Google Scholar 

  47. Ye X, Li J, Wang H, Huang H, Zhang X (2017) Efficient stereo matching leveraging deep local and context information. IEEE Access 5:18745–18755

    Article  Google Scholar 

  48. Zbontar J, LeCun Y (2015) Computing the stereo matching cost with a convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1592–1599

  49. Zbontar J, LeCun Y et al (2016) Stereo matching by training a convolutional neural network to compare image patches. J Mach Learn Res 17(1):2287–2318

    MATH  Google Scholar 

  50. Zhang X, Gao Q, Pan D, Cao PC, Huang DH (2021) Research on spatial positioning system of fruits to be picked in field based on binocular vision and ssd model. In: Journal of physics: conference series, vol 1748. IOP Publishing, p 042011

  51. Zhang C, Li Z, Cheng Y, Cai R, Chao H, Rui Y (2015) Meshstereo: a global stereo model with mesh alignment regularization for view interpolation. In: Proceedings of the IEEE international conference on computer vision, pp 2057–2065

  52. Zhang H, Li H, Wang Z, Yue Y, Chen S (2020) Geometry and context guided refinement for stereo matching. IET Image Process 14 (12):2652–2659

    Article  Google Scholar 

  53. Zhang B, Zhu D (2019) Key technologies of robot navigation based on machine vision: A review. Automatic Control, Mechatronics and Industrial Engineering, 31–36

  54. Zhi T, Pires BR, Hebert M, Narasimhan SG (2018) Deep material-aware cross-spectral stereo matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1916–1925

  55. Zhu H, Yin J, Yuan D (2017) Svcv: segmentation volume combined with cost volume for stereo matching. IET Comput Vis 11(8):733–743

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratory of Algorithms for Cognitive Models, School of Computer Science, Fudan University, Shanghai, 200433, China

    Zhang Hongjin, Wei Hui & Ma Gang

Authors
  1. Zhang Hongjin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Hui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ma Gang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Hui.

Ethics declarations

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

(mp4 5.21 MB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hongjin, Z., Hui, W. & Gang, M. A new stereo matching energy model based on image local features. Multimed Tools Appl 82, 35651–35684 (2023). https://doi.org/10.1007/s11042-023-14706-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14706-5

Keywords

Search

Navigation