Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Energy-based multi-view piecewise planar stereo

基于能量的多视图分段平面立体视觉

Abstract

The piecewise planar model (PPM) is an effective means of approximating a complex scene by using planar patches to give a complete interpretation of the spatial points reconstructed from projected 2D images. The traditional piecewise planar stereo methods suffer from either a very restricted number of directions for plane detection or heavy reliance on the segmentation accuracy of superpixels. To address these issues, we propose a new multi-view piecewise planar stereo method in this paper. Our method formulates the problem of complete scene reconstruction as a multi-level energy minimization problem. To detect planes along principal directions, a novel energy formulation with pair-wise potentials is used to assign an optimal plane for each superpixel in an iterative manner, where reliable scene priors and geometric constraints are incorporated to enhance the modeling efficacy and inference efficiency. To detect non-principal-direction planes, we adopt a multi-direction plane sweeping with a restricted search space method to generate reliable candidate planes. To handle the multi-surface straddling problem of a single superpixel, a superpixel sub-segmenting scheme is proposed and a robust P n Potts model-like higher-order potential is introduced to refine the resulting depth map. Our method is a natural integration of pixel- and superpixel-level multi-view stereos under a unified energy minimization framework. Experimental results for standard data sets and our own data sets show that our proposed method can satisfactorily handle many challenging factors (e.g., slanted surfaces and poorly textured regions) and can obtain accurate piecewise planar depth maps.

创新点

对于基于图像的三维场景重建,由于光照变化、透视畸变、弱纹理等干扰因素的存在,传统像素级与区域级的重建算法通常难以获得可靠的结果。为了解决此问题,本文提出一种新颖的基于能量的场景分段平面重建算法。根据场景分段平面假设,本文算法在MRF(Markov Random Field)能量最小化框架下将场景完整结构推断问题转换为沿场景主方向与非主方向的平面标记以及平面级的场景结构优化等问题进行求解,由于候选平面集与融合灰度一致性度量、空间几何与可见性约束、空间平面先验的能量函数的高可靠性,因而可以快速获取完整、准确的场景模型。实验结果表明,本文算法不但可以有效地解决场景中弱纹理、倾斜表面等区域的重建问题,而且可以克服传统相关算法依赖特定场景模型(如Manhattan 场景模型)、易受图像过分割精度的影响等缺点,整体上具有较高的可靠性与效率。

This is a preview of subscription content, log in to check access.

References

  1. 1

    Hong L, Chen G. Segment-based stereo matching using graph cuts. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, 2004. 74–81

  2. 2

    Zitnick C L, Kang S B. Stereo for image-based rendering using image over-segmentation. Int J Comput Vision, 2007, 75: 49–65

  3. 3

    Klaus A, Sormann M, Karner K. Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: Proceedings of the 18th International Conference on Pattern Recognition (ICPR’06), Hong Kong, 2006. 15–18

  4. 4

    Wang Z F, Zheng Z G. A region based stereo matching algorithm using cooperative optimization. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, 2008. 1–8

  5. 5

    Boykov Y, Veksler O, Zabih R. Fast approximate energy minimization via graph cuts. IEEE Trans Pattern Anal Mach Intell, 2001, 23: 1222–1239

  6. 6

    Bleyer M, Rhemann C, Rother C. PatchMatch stereo - stereo matching with slanted support windows. In: Proceedings of British Machine Vision Conference, Dundee, 2011. 1–11

  7. 7

    Cigla C, Zabulis X, Alatan A A. Segment-based stereo-matching via plane and angle sweeping. In: Proceedings of 3DTV Conference, Kos Island, 2007. 1–4

  8. 8

    Furukawa Y, Curless B, Seitz S, et al. Manhattan-world stereo. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami, 2009. 1422–1429

  9. 9

    Furukawa Y, Ponce J. Accurate, dense, and robust multi-view stereopsis. IEEE Trans Pattern Anal Mach Intell, 2009, 32: 1362–1376

  10. 10

    Mičušík B, Košecká J. Piecewise planar city 3D modeling from street view panoramic sequences. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami, 2009. 2906–2912

  11. 11

    Mičušík B, Košecká J. Multi-view superpixel stereo in urban environments. Int J Comput Vision, 2010, 89: 106–119

  12. 12

    Gallup D, Frahm J M, Mordohai P, et al. Real-time plane-sweeping stereo with multiple sweeping directions. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, 2007. 1–8

  13. 13

    Gallup D, Frahm J M, Pollefeys M. Piecewise planar and non-planar stereo for urban scene reconstruction. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, 2010. 1418–1425

  14. 14

    Sinha S N, Steedly D, Szeliski R. Piecewise planar stereo for image-based rendering. In: Proceedings of IEEE 12th International Conference on Computer Vision, Kyoto, 2009. 1881–1888

  15. 15

    Kim H, Xiao H, Max N. Piecewise planar scene reconstruction and optimization for multi-view stereo. In: Proceedings of the 11th Asian Conference on Computer Vision. Berlin: Springer, 2012. 191–204

  16. 16

    Chauve A L, Labatut P, Pons J P. Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, 2010. 1261–1268

  17. 17

    Kowdle A, Sinha S N, Szeliski R. Multiple view object cosegmentation using appearance and stereo cues. In: Proceedings of the 12th European Conference on Computer Vision. Berlin: Springer, 2012. 789–803

  18. 18

    Bodis-Szomoru A, Riemenschneider H, van Gool L. Fast, approximate piecewise-planar modeling based on sparse structure-from-motion and superpixel. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Columbus, 2014. 469–476

  19. 19

    Gould S, Fulton R, Koller D. Decomposing a scene into geometric and semantically consistent regions. In: Proceedings of IEEE 12th International Conference on Computer Vision, Kyoto, 2009. 1–8

  20. 20

    Comaniciu D, Meer P. Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell, 2002, 24: 603–619

  21. 21

    Fischler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM, 1981, 24: 381–395

  22. 22

    Isack H, Boykov Y. Energy-based geometric multi-model fitting. Int J Comput Vision, 2010, 92: 123–147

  23. 23

    Delong A, Osokin A, Isack H N, et al. Fast approximate energy minimization with label costs. Int J Comput Vision, 2012, 96: 1–27

  24. 24

    Pham T T, Chin T J, Yu J, et al. The random cluster model for robust geometric fitting. IEEE Trans Pattern Anal Mach Intell, 2014, 36: 1658–1671

  25. 25

    Tola E, Strecha C, Fua P. Efficient large-scale multi-view stereo for ultra high-resolution image sets. Mach Vision Appl, 2012, 23: 903–920

  26. 26

    Frey B J, Dueck D. Clustering by passing messages between data points. Science, 2007, 315: 972–976

  27. 27

    Kohli P, Ladicky L, Torr P H S. Robust higher order potentials for enforcing label consistency. Int J Comput Vision, 2009, 82: 302–324

Download references

Author information

Correspondence to Wei Wang.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wang, W., Hu, L. & Hu, Z. Energy-based multi-view piecewise planar stereo. Sci. China Inf. Sci. 60, 32101 (2017). https://doi.org/10.1007/s11432-015-0710-5

Download citation

Keywords

  • piecewise planar model
  • depth map
  • plane fitting
  • energy optimization
  • multi-view stereo
  • 032101

关键词

  • 能量优化
  • 分段平面模型
  • 深度图
  • 平面拟合
  • 多视立体视觉