Abstract
Image interpolation has a wide range of applications such as frame rate-up conversion and free viewpoint TV. Despite significant progresses, it remains an open challenge especially for image pairs with large displacements. In this paper, we first propose a novel optimization algorithm for motion estimation, which combines the advantages of both global optimization and a local parametric transformation model. We perform optimization over dynamic label sets, which are modified after each iteration using the prior of piecewise consistency to avoid local minima. Then we apply it to an image interpolation framework including occlusion handling and intermediate image interpolation. We validate the performance of our algorithm experimentally, and show that our approach achieves state-of-the-art performance.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Jiang, H. Z.; Sun, D. Q.; Jampani, V.; Yang, M. H.; Learned-Miller, E.; Kautz, J. Super SloMo: High quality estimation of multiple intermediate frames for video interpolation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9000–9008, 2018.
Nie, Y.; Zhang, Z.; Sun, H.; Su, T.; Li, G. Homography propagation and optimization for wide-baseline street image interpolation. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 10, 2328–2341, 2017.
Geiger, A.; Lenz, P.; Urtasun, R. Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3354–3361, 2012.
Bao, W. B.; Lai, W. S.; Ma, C.; Zhang, X. Y.; Gao, Z. Y.; Yang, M. H. Depth-aware video frame interpolation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3698–3707, 2019.
Menze, M.; Heipke, C.; Geiger, A. Discrete optimization for optical flow. In: Pattern Recognition. Lecture Notes in Computer Science, Vol. 9358. Gall, J.; Gehler, P.; Leibe, B. Eds. Springer Cham, 16–28, 2015.
Liu, C.; Yuen, J.; Torralba, A. SIFT flow: Dense correspondence across scenes and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 5, 978–994, 2011.
Revaud, J.; Weinzaepfel, P.; Harchaoui, Z.; Schmid, C. EpicFlow: Edge-preserving interpolation of correspondences for optical flow. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1164–1172, 2015.
Horn, B. K. P.; Schunck, B. G. Determining optical flow. Artificial Intelligence Vol. 17, Nos. 1–3, 185–203, 1981.
Zhao, S. Y.; Sheng, Y. L.; Dong, Y.; Chang, E. I. C.; Xu, Y. MaskFlownet: Asymmetric feature matching with learnable occlusion mask. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6277–6286, 2020.
Sun, D. Q.; Yang, X. D.; Liu, M. Y.; Kautz, J. PWC-net: CNNs for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8934–8943, 2018.
Hui, T. W.; Tang, X. O.; Loy, C. C. A lightweight optical flow CNN- revisiting data fidelity and regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence doi: https://doi.org/10.1109/TPAMI.2020.2976928, 2020.
Yang, G.; Ramanan, D. Volumetric correspondence networks for optical flow. In: Proceedings of the 33rd Conference on Neural Information Processing Systems, 794–805, 2019.
Chen, Q. F.; Koltun, V. Full flow: Optical flow estimation by global optimization over regular grids. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4706–4714, 2016.
Chen, J.; Cai, Z.; Lai, J.; Xie, X. A filtering-based framework for optical ow estimation. IEEE Transactions on Circuits and Systems for Video Technology Vol. 29, No. 5, 1350–1364, 2019.
Chen, J.; Cai, Z. M.; Lai, J. H.; Xie, X. H. Fast optical flow estimation based on the split bregman method. IEEE Transactions on Circuits and Systems for Video Technology Vol. 28, No. 3, 664–678, 2018.
Bleyer, M.; Rhemann, C.; Rother, C. PatchMatch stereo-stereo matching with slanted support windows. In: Proceedings of the British Machine Vision Conference, 14.1-14.11, 2011.
Taniai, T.; Matsushita, Y.; Naemura, T. Graph cut based continuous stereo matching using locally shared labels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1613–1620, 2014.
Veksler, O. Reducing search space for stereo correspondence with graph cuts. In: Proceedings of the British Machine Vision Conference, 709–718, 2006.
Kothapa, R.; Pacheco, J.; Sudderth, E. Max-product particle belief propagation. Master project report. Dept. of Computer Science, Brown University, 2011. Available at http://cs.brown.edu/research/pubs/theses/masters/2011/kothapa.pdf.
Besse, F.; Rother, C.; Fitzgibbon, A.; Kautz, J. PMBP: PatchMatch belief propagation for correspondence field estimation. International Journal of Computer Vision Vol. 110, No. 1, 2–13, 2014.
Li, Y.; Min, D. B.; Brown, M. S.; Do, M. N.; Lu, J. B. SPM-BP: Sped-up PatchMatch belief propagation for continuous MRFs. In: Proceedings of the IEEE International Conference on Computer Vision, 4006–4014, 2015.
Hornáček, M.; Besse, F.; Kautz, J.; Fitzgibbon, A.; Rother, C. Highly overparameterized optical flow using PatchMatch belief propagation. In: Computer Vision — ECCV 2014. Lecture Notes in Computer Science, Vol. 8691. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 220–234, 2014.
Baker, S.; Scharstein, D.; Lewis, J. P.; Roth, S.; Black, M. J.; Szeliski, R. A database and evaluation methodology for optical flow. International Journal of Computer Vision Vol. 92, No. 1, 1–31, 2011.
Hur, J.; Roth, S. Optical flow estimation in the deep learning age. In: Modelling Human Motion. Noceti, N.; Sciutti, A.; Rea, F. Eds. Springer Cham, 119–140, 2020.
Jeong, S. G.; Lee, C.; Kim, C. S. Motion-compensated frame interpolation based on multihypothesis motion estimation and texture optimization. IEEE Transactions on Image Processing Vol. 22, No. 11, 4497–4509, 2013.
Mahajan, D.; Huang, F. C.; Matusik, W.; Ramamoorthi, R.; Belhumeur, P. Moving gradients: A path-based method for plausible image interpolation. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 42, 2009.
Stich, T.; Linz, C.; Wallraven, C.; Cunningham, D.; Magnor, M. Perception-motivated interpolation of image sequences. ACM Transactions on Applied Perception Vol. 8, No. 2, Article No. 11, 2011.
Meyer, S.; Wang, O.; Zimmer, H.; Grosse, M.; Sorkine-Hornung, A. Phase-based frame interpolation for video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1410–1418, 2015.
Long, G. C.; Kneip, L.; Alvarez, J. M.; Li, H. D.; Zhang, X. H.; Yu, Q. F. Learning image matching by simply watching video. In: Computer Vision — ECCV 2016. Lecture Notes in Computer Science, Vol. 9910. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 434–450, 2016.
Niklaus, S.; Mai, L.; Liu, F. Video frame interpolation via adaptive convolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2270–2279, 2017.
Niklaus, S.; Mai, L.; Liu, F. Video frame interpolation via adaptive separable convolution. In: Proceedings of the IEEE International Conference on Computer Vision, 261–270, 2017.
Chen, Z. Y.; Jin, H. L.; Lin, Z.; Cohen, S.; Wu, Y. Large displacement optical flow from nearest neighbor fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2443–2450, 2013.
Bao, L. C.; Yang, Q. X.; Jin, H. L. Fast edge-preserving PatchMatch for large displacement optical flow. IEEE Transactions on Image Processing Vol. 23, No. 12, 4996–5006, 2014.
Barnes, C.; Shechtman, E.; Finkelstein, A.; Goldman, D. B. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 24, 2009.
Penner, E.; Zhang, L. Soft 3D reconstruction for view synthesis. ACM Transactions on Graphics Vol. 36, No. 6, Article No. 235, 2017.
Hedman, P.; Kopf, J. Instant 3D photography. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 101, 2018.
Chaurasia, G.; Duchene, S.; Sorkine-Hornung, O.; Drettakis, G. Depth synthesis and local warps for plausible image-based navigation. ACM Transactions on Graphics Vol. 32, No. 3, Article No. 30, 2013.
Wang, S.; Wang, R. G. Robust view synthesis in wide-baseline complex geometric environments. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2297–2301, 2019.
Zhou, T. H.; Tulsiani, S.; Sun, W. L.; Malik, J.; Efros, A. A. View synthesis by appearance flow. In: Computer Vision — ECCV 2016. Lecture Notes in Computer Science, Vol. 9908. Springer Cham, 286–301, 2016.
Xu, Z. X.; Bi, S.; Sunkavalli, K.; Hadap, S.; Su, H.; Ramamoorthi, R. Deep view synthesis from sparse photometric images. ACM Transactions on Graphics Vol. 38, No. 4, Article No. 76, 2019.
Mildenhall, B.; Srinivasan, P. P.; Tancik, M.; Barron, J. T.; Ramamoorthi, R.; Ng, R. NeRF: Representing scenes as neural radiance fields for view synthesis. In: Computer Vision — ECCV 2020. Lecture Notes in Computer Science, Vol. 12346. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer International Publishing, 405–421, 2020.
Yu, A.; Ye, V.; Tancik, M.; Kanazawa, A. pixelNeRF: Neural radiance fields from one or few images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Felzenszwalb, P. F.; Huttenlocher, D. R. Efficient belief propagation for early vision. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004.
Achanta, R.; Shaji, A.; Smith, K.; Lucchi, A.; Fua, P.; Süsstrunk, S. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 34, No. 11, 2274–2282, 2012.
Fischler, M. A.; Bolles, R. C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM Vol. 24, No. 6. 381–395, 1981.
Hu, Y.; Li, Y.; Song, R. Robust interpolation of correspondences for large displacement optical flow. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4791–4799, 2017.
Kaviani, H. R.; Shirani, S. Iterative mask generation method for handling occlusion in optical ow assisted view interpolation. In: Proceedings of the IEEE International Conference on Image Processing, 3387–3391, 2015.
Burt, P. J.; Adelson, E. H. A multiresolution spline with application to image mosaics. ACM Transactions on Graphics Vol. 2, No. 4, 217–236, 1983.
Butler, D. J.; Wulff, J.; Stanley, G. B.; Black, M. J. A naturalistic open source movie for optical flow evaluation. In: Computer Vision — ECCV 2012. Lecture Notes in Computer Science, Vol. 7577. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Berlin Heidelberg, 611–625, 2012.
Menze, M.; Geiger, A. Object scene flow for autonomous vehicles. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3061–3070, 2015.
Dosovitskiy, A.; Fischer, P.; Ilg, E.; Häusser, P.; Hazirbas, C.; Golkov, V.; Smagt, P.; Cremers, D.; Brox, T. FlowNet: Learning optical flow with convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, 2758–2766, 2015.
Mayer, N.; Ilg, E.; Häusser, P.; Fischer, P.; Cremers, D.; Dosovitskiy, A.; Brox, T. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4040–4048, 2016.
Huang, P. H.; Matzen, K.; Kopf, J.; Ahuja, N.; Huang, J. B. DeepMVS: Learning multi-view stereopsis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2821–2830, 2018.
Acknowledgements
This project was supported by the National Key Technology Research and Development Program of China (No. 2017YFB1002601), PKU-Baidu Fund (No. 2019BD007), and National Natural Science Foundation of China (NSFC) (No. 61632003).
Author information
Authors and Affiliations
Corresponding author
Additional information
Yuan Chang is currently a Ph.D. student at Peking University. He received his B.S. degree from the School of Mathematical Sciences, University of Science and Technology of China, in 2016. His research interests include computer vision, computer graphics, and image computing.
Congyi Zhang is a postdoctoral fellow at the University of Hong Kong. He received his B.Sc. degree from the School of Mathematical Science, Fudan University, in 2012, and his Ph.D. degree from the School of Electronics Engineering and Computer Science, Peking University, in 2019. His research interests include computer vision, human-computer interaction, and computer graphics.
Yisong Chen received his B.S. degree in information engineering from Xi’an Jiaotong University, China, in 1996, and his Ph.D. degree in computer science from Nanjing University, China, in 2001. From 2001 to 2003, he was a postdoctoral researcher with the HumanComputer Interaction and Multimedia Laboratory, Peking University. From 2003 to 2005, he was a research fellow with the Image Computing Group, City University of Hong Kong. From 2005 to 2006, he was a senior research fellow with the Heudiasyc Laboratory, Centre National de la Recherche Scientifique, University of Technology of Compiegne, France. In 2007, he joined the Department of Computer Science, Peking University, as an associate professor. His research interests include computer vision, image computing, and pattern recognition.
Guoping Wang is a professor of computer science, Peking University, and director of the Graphics and Interactive Technology Laboratory, Peking University. He obtained his Ph.D degree from the Institute of Mathematics, Fudan University in 1997. He became a full professor in Dept. of Computer Science, Peking University in 2002. He was awarded by the National Science Fund for Distinguished Young Scholars in 2009. His research interests include virtual reality, computer graphics, human-computer interaction and multimedia.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Chang, Y., Zhang, C., Chen, Y. et al. Homography-guided stereo matching for wide-baseline image interpolation. Comp. Visual Media 8, 119–133 (2022). https://doi.org/10.1007/s41095-021-0225-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-021-0225-9