Abstract
This paper presents an effective framework for correspondence field estimation. The core idea is to construct pixel-level and superpixel-level patch matching to achieve high accuracy estimation as well as fast speed computation. To this end, a hybrid edge-preserving supported weighting approach is first developed, which contributes to better performance on the pixel level, especially on those in the regions of fine structures. Then, a local Minimum Spanning Tree (MST) is constructed to describe regions and develop the adaptive smooth penalty weights, so that the over-patching in large textureless regions can be effectively avoided. In addition, the MST is further extended to handle occlusions in way of edge preserving strategy. Finally, all the above treatments are collected into an optimization model where the objective function is developed in terms of Markov Random Filed (MRF). In computation, a fast yet efficient iterative optimization strategy is developed. Our approach achieves favorable place on optical flow benchmark, which locates at the top two and top four for endpoint error and angular error evaluations among more than 130 approaches listed in the webpage.
Similar content being viewed by others
References
Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Susstrunk S (2012) Slic superpixels compared to state-of-the-art superpixel methods. TPAMI 34(11):2274–2282
Baker S, Scharstein D, Lewis JP, Roth S, Black MJ, Szeliski R (2011) A database and evaluation methodology for optical flow. Int J Comput Vis 92(1):1–31
Bao L, Song Y, Yang Q, Yuan H, Wang G (2014) Tree filtering: efficient structure-preserving smoothing with a minimum spanning tree. TIP 23(2):555–569
Bao L, Yang Q, Jin H (2014) Fast edge-preserving patchmatch for large displacement optical flow. IEEE TIP
Barnes C, Adviser-Finkelstein A (2011) Patchmatch: a fast randomized matching algorithm with application to image and video. Princeton University
Barnes C, Shechtman E, Finkelstein A, Goldman D (2009) Patchmatch: a randomized correspondence algorithm for structural image editing. TOG 28(3):24
Besse F, Rother C, Fitzgibbon A, Kautz J (2013) Pmbp: patchmatch belief propagation for correspondence field estimation. IJCV, 1–12
Bleyer M, Rhemann C, Rother C (2011) Patchmatch stereo-stereo matching with slanted support windows. In: BMVC, vol 11, pp 1–11
Chen Z, Jin H, Lin Z, Cohen S, Wu Y (2013) Large displacement optical flow from nearest neighbor fields. In: 2013 IEEE conference on computer vision and pattern recognition, pp 2443–2450
Chen Z, Jin H, Lin Z, Cohen S, Wu Y (2013) Large displacement optical flow from nearest neighbor fields. In: CVPR. IEEE, pp 2443–2450
Dai L, Zhang F, Mei X, Zhang X (2015) Fast minimax path-based joint depth interpolation. IEEE Signal Process Lett 22(5):623–627
Drulea M, Nedevschi S (2013) Motion estimation using the correlation transform. TIP 22(8):3260–3270
Gupta RK, Cho SY (2010) Real-time stereo matching using adaptive binary window. 3DPVT 2:1–8
Hornáček M, Besse F, Kautz J, Fitzgibbon A, Rother C (2014) Highly overparameterized optical flow using patchmatch belief propagation. In: ECCV. Springer, pp 220–234
Hosni A, Bleyer M, Gelautz M, Rhemann C (2009) Local stereo matching using geodesic support weights. In: ICIP. IEEE, pp 2093–2096
Kim TH, Lee HS, Lee KM (2013) Optical flow via locally adaptive fusion of complementary data costs. In: ICCV. IEEE, pp 3344–3351
Lempitsky V, Rother C, Roth S, Blake A (2010) Fusion moves for Markov random field optimization. TPAMI 32(8):1392–1405
Lu J, Yang H, Min D, Do MN (2013) Patch match filter: efficient edge-aware filtering meets randomized search for fast correspondence field estimation. In: CVPR. IEEE, pp 1854–1861
Mohamed MA, Mertsching B (2012) Tv-l1 optical flow estimation with image details recovering based on modified census transform. In: Advances in visual computing. Springer, pp 482–491
Olsson C, Ulén J, Boykov Y (2013) In defense of 3d-label stereo. In: CVPR. IEEE, pp 1730–1737
Scharstein D, Szeliski R (2002) A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47(1–3):7–42
Shen S (2013) Accurate multiple view 3d reconstruction using patch-based stereo for large-scale scenes. TIP 22(5):1901–1914
Szeliski R, Zabih R, Scharstein D, Veksler O, Kolmogorov V, Agarwala A, Tappen M, Rother C (2008) A comparative study of energy minimization methods for Markov random fields with smoothness-based priors. TPAMI 30(6):1068–1080
Taniai T, Matsushita Y, Naemura T (2014) Graph cut based continuous stereo matching using locally shared labels. In: CVPR. IEEE, pp 1613–1620
Tombari F, Mattoccia S, Di Stefano L (2007) Segmentation-based adaptive support for accurate stereo correspondence. In: Advances in image and video technology. Springer, pp 427–438
Xu L, Jia J, Matsushita Y (2010) Motion detail preserving optical flow estimation. In: CVPR, pp 1293–1300
Yang Q (2012) A non-local cost aggregation method for stereo matching. In: CVPR. IEEE, pp 1402–1409
Yoon KJ, Kweon IS (2006) Adaptive support-weight approach for correspondence search. TPAMI 28(4):650–656
Acknowledgements
This work was also supported by the National Key R&D Program of China (Grant 2018YFB2100602). This work is supported by the National Natural Science Foundation of China (Nos. 61620106003, 61971418, 61771026, 61671451 and 61571046).
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Feihu Zhang and Shibiao Xu contributed equally to this work and share the first authorship.
Rights and permissions
About this article
Cite this article
Zhang, F., Xu, S. & Zhang, X. High accuracy correspondence field estimation via MST based patch matching. Multimed Tools Appl 79, 13291–13309 (2020). https://doi.org/10.1007/s11042-020-08633-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-08633-y