Robust dense reconstruction by range merging based on confidence estimation

Abstract

Although the stereo matching problem has been extensively studied during the past decades, automatically computing a dense 3D reconstruction from several multiple views is still a difficult task owing to the problems of textureless regions, outliers, detail loss, and various other factors. In this paper, these difficult problems are handled effectively by a robust model that outputs an accurate and dense reconstruction as the final result from an input of multiple images captured by a normal camera. First, the positions of the camera and sparse 3D points are estimated by a structure-from-motion algorithm and we compute the range map with a confidence estimation for each image in our approach. Then all the range maps are integrated into a fine point cloud data set. In the final step we use a Poisson reconstruction algorithm to finish the reconstruction. The major contributions of the work lie in the following points: effective range-computation and confidence-estimation methods are proposed to handle the problems of textureless regions, outliers and detail loss. Then, the range maps are merged into the point cloud data in terms of a confidence-estimation. Finally, Poisson reconstruction algorithm completes the dense mesh. In addition, texture mapping is also implemented as a post-processing work for obtaining good visual effects. Experimental results are presented to demonstrate the effectiveness of the proposed approach.

This is a preview of subscription content, access via your institution.

References

  1. 1

    Chen J W, Bautembach D, Izadi S. Scalable real-time volumetric surface reconstruction. ACM Trans Graphic, 2013, 32: 1–16

    MATH  Google Scholar 

  2. 2

    Chen Y D, Hao C Y, Wu W, et al. Recursive video segmentation (in Chinese). Sci Sin Inform, 2014, 44: 1361–1369

    Google Scholar 

  3. 3

    Hao C Y, Chen Y D, Wu W, et al. Image completion with perspective constraint based on a single image. Sci China Inf Sci, 2015, 58: 092109

    Article  Google Scholar 

  4. 4

    Collins R T. A space-sweep approach to true multi-image matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, 1996. 358–363

    Google Scholar 

  5. 5

    Seitz S M, Dyer C R. Photorealistic scene reconstruction by voxel coloring. Int J Comput Vision, 1999, 35: 151–173

    Article  Google Scholar 

  6. 6

    Newcombe R A, Davison A J. Live dense reconstruction with a single moving camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, 2010. 1498–1505

    Google Scholar 

  7. 7

    Chen Y D, Hao C Y, Wu W, et al. Live accurate and dense reconstruction from a handheld camera. Comput Animat Virtual Worlds, 2013, 24: 387–397

    Article  Google Scholar 

  8. 8

    Furukawa Y, Ponce J. Accurate, dense, and robust multiview stereopsis. IEEE Trans Pattern Anal Mach Intell, 2010, 32: 1362–1376

    Article  Google Scholar 

  9. 9

    Furukawa Y, Curless B, Seitz S M, et al. Towards internet-scale multi-view stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, 2010. 1434–1441

    Google Scholar 

  10. 10

    Wang Y, Ji X Y, Dai Q H. Key technologies of light field capture for 3d reconstruction in microscopic scene. Sci China Inf Sci, 2010, 53: 1917–1930

    Article  Google Scholar 

  11. 11

    Agarwal S, Furukawa Y, Snavely N, et al. Building rome in a day. Commun ACM, 2011, 54: 105–112

    Article  Google Scholar 

  12. 12

    Sun J, Li Y, Kang S B, et al. Symmetric stereo matching for occlusion handling. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, 2005. 399–406

    Google Scholar 

  13. 13

    Strecha C, Fransens R, van Gool L. Combined depth and outlier estimation in multi-view stereo. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, 2006. 2394–2401

    Google Scholar 

  14. 14

    Ryan K, Camillo J T. Optical flow with geometric occlusion estimation and fusion of multiple frames. In: Proceedings of the 10th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition, Hongkong, 2015. 364–377

    Google Scholar 

  15. 15

    Zhang G F, Jia J Y, Hua W, et al. Robust bilayer segmentation and motion depth estimation with a handheld camera. IEEE Trans Pattern Anal Mach Intell, 2011, 33: 603–617

    Article  Google Scholar 

  16. 16

    Felzenszwalb P F, Huttenlocher D P. Efficient belief propagation for early vision. Int J Comput Vision, 2006, 70: 41–54

    Article  Google Scholar 

  17. 17

    Woodford O, Torr P, Reid I, et al. Global stereo reconstruction under second-order smoothness priors. IEEE Trans Pattern Anal Mach Intell, 2009, 31: 2115–2128

    Article  Google Scholar 

  18. 18

    Fuhrmann S, Goesele M. Fusion of depth maps with multiple scales. In: Proceedings of the SIGGRAPH Asia Conference. New York: ACM, 2011. 1–8

    Google Scholar 

  19. 19

    Zach C, Pock T, Bischof H. A globally optimal algorithm for robust TV-L1 range image integration. In: Proceedings of the IEEE International Conference on Computer Vision, Rio de Janeiro, 2007. 1–8

    Google Scholar 

  20. 20

    Stühmer J, Gumhold S, Cremers D. Real-time dense geometry from a handheld camera. In: Proceedings of the Conference on Pattern Recognition, Lecture Notes in Computer Science, Berlin, 2010. 11–20

    Google Scholar 

  21. 21

    Graber G, Pock T, Bischof H. Online 3d reconstruction using convex optimization. In: Proceedings of IEEE International Conference on Computer Vision Workshops, Barcelona, 2011. 708–711

    Google Scholar 

  22. 22

    Bischof H. Dense reconstruction on-the-fly. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Washington, 2012. 1450–1457

    Google Scholar 

  23. 23

    Shan Q, Curless B, Furukawa Y, et al. Occluding contours for multi-view stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Washington, 2014. 4002–4009

    Google Scholar 

  24. 24

    Snavely N, Seitz S M, Szeliski R. Photo tourism: exploring photo collections in 3D. In: Proceedings of the ACM SIGGRAPH, New York, 2006. 835–846

    Google Scholar 

  25. 25

    Wu C C, Agarwal S, Curless B, et al. Multicore bundle adjustment. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Providence, 2011. 3057–3064

    Google Scholar 

  26. 26

    Dong Z L, Zhang G F, Jia J Y, et al. Efficient keyframe-based real-time camera tracking. Comput Vision Image Und, 2014, 118: 97–110

    MathSciNet  Article  Google Scholar 

  27. 27

    Kazhdan M, Hoppe H. Screened Poisson surface reconstruction. ACM Trans Graphic, 2013, 32: 1–13

    Article  MATH  Google Scholar 

  28. 28

    Kang S B, Szeliski R. Extracting view-dependent depth maps from a collection of images. Int J Comput Vision, 2004, 58: 139–163

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Chuanyan Hao.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Hao, C., Wu, W. et al. Robust dense reconstruction by range merging based on confidence estimation. Sci. China Inf. Sci. 59, 092103 (2016). https://doi.org/10.1007/s11432-015-0957-4

Download citation

Keywords

  • stereo matching
  • 3D reconstruction
  • textureless regions
  • outliers
  • details loss
  • range map