Skip to main content
SpringerLink
Log in

LGP-MVS: combined local and global planar priors guidance for indoor multi-view stereo

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Multi-view stereo (MVS) has long been a subject for researchers in the computer vision field. Due to the unreliable photometric consistency in low-textured areas, the existing PatchMatch MVS methods have low accuracy and completeness when recovering the depth information of low-textured areas in indoor environments. To solve the above problem that PatchMatch methods always fail in the textureless regions, we propose a global-local planar priors jointly optimized PatchMatch MVS method. The algorithm constructs global-local planar priors and uses a dynamic texture-related multi-view aggregation cost to balance photometric consistency and planar priors. The validity of the algorithm is verified by quantitative and qualitative analysis of depth maps and 3D reconstruction on multiple real-world scenes from ScanNet Dataset and ETH3D benchmark, also synthetic indoor scenes from ICL-NUIM Dataset. Our method can effectively recover the depth information in the textureless regions, so as to obtain the 3D model with high precision.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Wang, W., Gao, W.: Efficient multi-plane extraction from massive 3d points for modeling large-scale urban scenes. Vis. Comput. 35(5), 625–638 (2019)

    Article  Google Scholar 

  2. Gan, J., Wilbert, A., Thormählen, T., Drescher, P., Hagens, R.: Multi-view photometric stereo using surface deformation. Vis. Comput. 34(11), 1551–1561 (2018)

    Article  Google Scholar 

  3. Strecha, C., Von Hansen, W., Van Gool, L., Fua, P., Thoennessen, U.: On benchmarking camera calibration and multi-view stereo for high resolution imagery. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)

  4. Schops, T., Schonberger, J.L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., Geiger, A.: A multi-view stereo benchmark with high-resolution images and multi-camera videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3260–3269 (2017)

  5. Knapitsch, A., Park, J., Zhou, Q.-Y., Koltun, V.: Tanks and temples: benchmarking large-scale scene reconstruction. ACM Trans. Graph. (ToG) 36(4), 1–13 (2017)

    Article  Google Scholar 

  6. Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: Scannet: richly-annotated 3d reconstructions of indoor scenes. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR), IEEE (2017)

  7. Woodford, O., Torr, P., Reid, I., Fitzgibbon, A.: Global stereo reconstruction under second-order smoothness priors. IEEE Trans. Pattern Anal. Mach. Intell. 31(12), 2115–2128 (2009)

    Article  Google Scholar 

  8. Gallup, D., Frahm, J.-M., Pollefeys, M.: Piecewise planar and non-planar stereo for urban scene reconstruction. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1418–1425. IEEE (2010)

  9. Zheng, E., Dunn, E., Jojic, V., Frahm, J.-M.: Patchmatch based joint view selection and depthmap estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1510–1517 (2014)

  10. Galliani, S., Lasinger, K., Schindler, K.: Massively parallel multiview stereopsis by surface normal diffusion. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 873–881 (2015)

  11. Schönberger, J.L., Zheng, E., Frahm, J.-M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In: European Conference on Computer Vision, pp. 501–518. Springer, Berlin (2016)

  12. Wei, M., Yan, Q., Luo, F., Song, C., Xiao, C.: Joint bilateral propagation upsampling for unstructured multi-view stereo. Vis. Comput. 35(6), 797–809 (2019)

    Article  Google Scholar 

  13. Xu, Q., Tao, W.: Multi-scale geometric consistency guided multi-view stereo. In: Computer Vision and Pattern Recognition (CVPR) (2019)

  14. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24 (2009)

    Article  Google Scholar 

  15. Romanoni, A., Matteucci, M.: Tapa-MVS: textureless-aware patchmatch multi-view stereo. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10413–10422 (2019)

  16. Liao, J., Fu, Y., Yan, Q., Xiao, C.: Pyramid multi-view stereo with local consistency. Comput. Graph. Forum 38, 335–346 (2019)

    Article  Google Scholar 

  17. Xu, Q., Tao, W.: Planar prior assisted patchmatch multi-view stereo. In: AAAI Conference on Artificial Intelligence (AAAI) (2020)

  18. Handa, A., Whelan, T., McDonald, J.B., Davison, A.J.: A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In: IEEE International Conference on Robotics and Automation, ICRA, Hong Kong, China (2014)

  19. Schöps, T., Schönberger, J.L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., Geiger, A.: “Eth3d benchmark” https://www.eth3d.net

  20. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), vol. 1, pp. 519–528. IEEE (2006)

  21. Hernández, C., Furukawa, Y.: Multi-view stereo: a tutorial. Comput. Graph. Vision 2(9), 1–148 (2013)

    Google Scholar 

  22. Vogiatzis, G., Esteban, C.H., Torr, P.H.S., Cipolla, R.: Multiview stereo via volumetric graph-cuts and occlusion robust photo-consistency. IEEE Trans. Pattern Anal. Mach. Intell. 29(12), 2241–2246 (2007)

    Article  Google Scholar 

  23. Ulusoy, A.O., Geiger, A., Black, M.J.: Towards probabilistic volumetric reconstruction using ray potentials. In: 2015 International Conference on 3D Vision, pp. 10–18. IEEE (2015)

  24. Vu, H.-H., Labatut, P., Pons, J.-P., Keriven, R.: High accuracy and visibility-consistent dense multiview stereo. IEEE Trans. Pattern Anal. Mach. Intell. 34(5), 889–901 (2011)

    Article  Google Scholar 

  25. Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. 32(8), 1362–1376 (2009)

    Article  Google Scholar 

  26. Locher, A., Perdoch, M., Van Gool, L.: Progressive prioritized multi-view stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3244–3252 (2016)

  27. Campbell, N.D.F., Vogiatzis, G., Hernández, C., Cipolla, R.: Using multiple hypotheses to improve depth-maps for multi-view stereo. In: European Conference on Computer Vision, pp. 766–779. Springer, Cham (2008)

  28. Wang, Y., Guan, T., Chen, Z., Luo, Y., Luo, K., Ju, L.: Mesh-guided multi-view stereo with pyramid architecture. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2036–2045 (2020)

  29. Li, Z., Zuo, W., Wang, Z., Zhang, L.: Confidence-based large-scale dense multi-view stereo. IEEE Trans. Image Process. 29, 7176–7191 (2020)

    Article  MATH  Google Scholar 

  30. Xu, Z., Liu, Y., Shi, X., Wang, Y., Zheng, Y.: Marmvs: matching ambiguity reduced multiple view stereo for efficient large scale scene reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June (2020)

  31. Tan, B., Xue, N., Bai, S., Wu, T., Xia, G.-S.: Planetr: structure-guided transformers for 3d plane recovery. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4186–4195 (2021)

  32. Geiger, A., Roser, M., Urtasun, R.: Efficient large-scale stereo matching. In: Asian conference on computer vision, pp. 25–38. Springer, Cham (2010)

  33. Schöps, T., Schönberger, J.L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., Geiger, A.: A multi-view stereo benchmark with high-resolution images and multi-camera videos. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  34. Sun, S., Liu, J., Li, Y., Ying, H., Zhai, Z., Mou, Y.: Adaptive pixelwise inference multi-view stereo. In: Thirteenth International Conference on Graphics and Image Processing, vol. 12083, pp. 165–172 (2022)

  35. Wang, F., Galliani, S., Vogel, C., Speciale, P., Pollefeys, M.: Patchmatchnet: learned multi-view patchmatch stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14194–14203 (2021)

  36. Wang, F., Galliani, S., Vogel, C., Pollefeys, M.: Itermvs: iterative probability estimation for efficient multi-view stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8606–8615 (2022)

  37. Su, W., Xu, Q., Tao, W.: Uncertainty-guided multi-view stereo network for depth estimation. IEEE Trans. Circuits Syst. Video Technol. 32, 7796–7808 (2022)

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China under Grants 62176096 and 61991412.

Author information

Authors and Affiliations

  1. National Key Laboratory of Science and Technology on Multispectral Information Processing, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, China

    Weihang Kong, Qingshan Xu, Wanjuan Su, Siyuan Xu & Wenbing Tao

Authors
  1. Weihang Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qingshan Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wanjuan Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Siyuan Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wenbing Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wenbing Tao.

Ethics declarations

Conflicts of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kong, W., Xu, Q., Su, W. et al. LGP-MVS: combined local and global planar priors guidance for indoor multi-view stereo. Vis Comput 39, 6421–6433 (2023). https://doi.org/10.1007/s00371-022-02737-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02737-2

Keywords

Search

Navigation