Skip to main content
SpringerLink
Log in

Accurate, dense and shading-aware multi-view stereo reconstruction using metaheuritic optimization

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Among the modern means of 3D geometry creation that exist in the literature, there are the Multi-View Stereo (MVS) reconstruction methods that received much attention from the research community and the multimedia industry. In fact, several methods showed that it is possible to recover geometry only from images with reconstruction accuracies paralleling that of excessively expensive laser scanners. The majority of these methods perform on images such as online community photo collection and estimate the surface position with its orientation by minimizing a matching cost function defined over a small local region. However, these datasets not only they are large but also contain more challenging scenes setups with different photometric effects; therefore fine-grained details of an object’s surface cannot be captured. This paper presents a robust multi-view stereo method based on metaheuristic optimization namely the Particle Swarm Optimization (PSO) in order to find the optimal depth, orientation, and surface roughness. To deal with the various shading and stereo mismatch problems caused by rough surfaces, shadows, and interreflections, we propose to use a robust matching/energy function which is a combination of two similarity measurements. Finally, our method computes individual depth maps that can be merged into compelling scene reconstructions. The proposed method is evaluated quantitatively using well-known Middlebury datasets and the obtained results show a high completeness score and comparable accuracy to those of the current top performing algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. http://vision.middlebury.edu/mview/eval/

References

  1. Clerc M, Kennedy J (2002) The particle swarm - explosion, stability, and convergence in a multidimensional complex space. IEEE Trans Evol Comput 6(1):58–73

    Article  Google Scholar 

  2. Esteban CH, Schmitt F (2004) Silhouette and stereo fusion for 3d object modeling. Comput Vis Image Underst 96(3):367–392

    Article  Google Scholar 

  3. Esteban CH, Vogiatzis G, Cipolla R (2008) Multiview photometric stereo. IEEE Trans Pattern Anal Mach Intell 30(3):548–554

    Article  Google Scholar 

  4. Faugeras OD, Keriven R (1998) Variational principles, surface evolution, pdes, level set methods, and the stereo problem. IEEE Trans Image Process 7(3):336–344

    Article  MathSciNet  MATH  Google Scholar 

  5. Fuhrmann S, Langguth F, Goesele M (2014) MVE - A multi-view reconstruction environment. In: 2014 eurographics workshop on graphics and cultural heritage, GCH 2014, Darmstadt, Germany, October 6-8, 2014, pp 11–18

  6. Furukawa Y, Hernȧndez C. (2015) Multi-view stereo: A tutorial. Foundations and Trends in Computer Graphics and Vision 9(1-2):1–148

    Article  Google Scholar 

  7. Furukawa Y, Ponce J (2007) Accurate, dense, and robust multi-view stereopsis. In: 2007 IEEE computer society conference on computer vision and pattern recognition (CVPR 2007), 18-23 June 2007. Minneapolis, Minnesota, USA

  8. Galliani S, Lasinger K, Schindler K (2015) Massively parallel multiview stereopsis by surface normal diffusion. In: 2015 IEEE international conference on computer vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp 873–881

  9. Gallup D, Frahm J, Mordohai P, Yang Q, Pollefeys M (2007) Real-time plane-sweeping stereo with multiple sweeping directions. In: 2007 IEEE computer society conference on computer vision and pattern recognition (CVPR 2007), 18-23 June 2007. Minneapolis, Minnesota, USA

  10. Goesele M, Curless B, Seitz SM (2006) Multi-view stereo revisited. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR 2006), 17-22 June 2006, New York, NY, USA, pp 2402–2409

  11. Goesele M, Snavely N, Curless B, Hoppe H, Seitz SM (2007) Multi-view stereo for community photo collections. In: IEEE 11th international conference on computer vision, ICCV 2007, Rio de Janeiro, Brazil, October 14-20, 2007, pp 1–8

  12. Harltey R, Zisserman A (2006) Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, Cambridge

    Google Scholar 

  13. Hertzmann A, Seitz SM (2005) Example-based photometric stereo: shape reconstruction with general, varying brdfs. IEEE Trans Pattern Anal Mach Intell 27(8):1254–1264

    Article  Google Scholar 

  14. Hirschmu̇ller H., Scharstein D (2009) Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans Pattern Anal Mach Intell 31(9):1582–1599

    Article  Google Scholar 

  15. Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe RA, Kohli P, Shotton J, Hodges S, Freeman D, Davison AJ, Fitzgibbon AW (2011) Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th annual ACM symposium on user interface software and technology, Santa Barbara, CA, USA, October 16-19, 2011, pp 559–568

  16. Kazhdan MM, Hoppe H (2013) Screened poisson surface reconstruction. ACM Trans Graph 32(3):29,1–29,13

    Article  MATH  Google Scholar 

  17. Langguth F, Sunkavalli K, Hadap S, Goesele M (2016) Shading-aware multi-view stereo. In: Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III, pp 469–485

  18. Li Z, Wang K, Zuo W, Meng D, Zhang L (2016) Detail-preserving and content-aware variational multi-view stereo reconstruction. IEEE Trans Image Process 25(2):864–877

    Article  MathSciNet  MATH  Google Scholar 

  19. Mei X, Sun X, Zhou M, Jiao S, Wang H, Zhang X (2011) On building an accurate stereo matching system on graphics hardware. In: IEEE international conference on computer vision workshops, ICCV 2011 Workshops, Barcelona, Spain, November 6-13, 2011, pp 467–474

  20. Newcombe RA, Lovegrove S, Davison AJ (2011) DTAM: dense tracking and mapping in real-time. In: IEEE international conference on computer vision, ICCV 2011, Barcelona, Spain, November 6-13, 2011, pp 2320–2327

  21. Oren M, Nayar SK (1993) Diffuse reflectance from rough surfaces. In: Conference on computer vision and pattern recognition, CVPR 1993, 15-17 June, 1993, New York, NY, USA, pp 763–764

  22. Oren M, Nayar SK (1994) Generalization of lambert’s reflectance model. In: Proceedings of the 21st annual conference on computer graphics and interactive techniques, SIGGRAPH ’94, pp 239–246

  23. Oxholm G, Nishino K (2014) Multiview shape and reflectance from natural illumination. In: 2014 IEEE conference on computer vision and pattern recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pp 2163–2170

  24. Ranftl R, Gehrig S, Pock T, Bischof H (2012) Pushing the limits of stereo using variational stereo estimation. In: 2012 IEEE intelligent vehicles symposium, IV 2012, Alcal de Henares, Madrid, Spain, June 3-7, 2012, pp 401–407

  25. Scharstein D, Szeliski R (2002) A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int J Comput Vis 47(1-3):7–42

    Article  MATH  Google Scholar 

  26. Schlick C (1994) A survey of shading and reflectance models. Comput Graph Forum 13(2):121–131

    Article  Google Scholar 

  27. Seitz SM, Curless B, Diebel J, Scharstein D, Szeliski R (2006) A comparison and evaluation of multi-view stereo reconstruction algorithms. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR 2006), 17-22 June 2006, New York, NY, USA, pp 519–528

  28. Semerjian B (2014) A new variational framework for multiview surface reconstruction. In: 2014 - 13th European conference computer vision - ECCV, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI, pp 719–734

  29. Shen S (2013) Accurate multiple view 3d reconstruction using patch-based stereo for large-scale scenes. IEEE Trans Image Process 22(5):1901–1914

    Article  MathSciNet  MATH  Google Scholar 

  30. Sinha SN, Pollefeys M (2005) Multi-view reconstruction using photo-consistency and exact silhouette constraints: A maximum-flow formulation. In: 10th IEEE international conference on computer vision (ICCV 2005), 17-20 October 2005, Beijing, China, pp 349–356

  31. Strecha C, von Hansen W, Gool LJV, Fua P, Thoennessen U (2008) On benchmarking camera calibration and multi-view stereo for high resolution imagery. In: 2008 IEEE computer society conference on computer vision and pattern recognition CVPR 2008, 24-26 June 2008. Anchorage, Alaska, USA

  32. Wei J, Resch B, Lensch HPA (2014) Multi-view depth map estimation with cross-view consistency. In: British machine vision conference, BMVC 2014, Nottingham, UK, September 1-5, 2014

  33. Wen-Chao Che Nzen CHEN PYS (2015) Stochastic optimization based 3d dense reconstruction from multiple views with high accuracy and completeness. Journal of Information Science and Engineering

  34. Wu C, Wilburn B, Matsushita Y, Theobalt C (2011) High-quality shape from multi-view stereo and shading under general illumination. In: The 24th IEEE conference on computer vision and pattern recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, pp 969–976

  35. Zhan Y, Gu Y, Huang K, Zhang C, Hu K (2016) Accurate image-guided stereo matching with efficient matching cost and disparity refinement. IEEE Trans Circuits Syst Video Technol 26(9):1632–1645

    Article  Google Scholar 

  36. Zhu Z, Stamatopoulos C, Fraser CS (2015) Accurate and occlusion-robust multi-view stereo. ISPRS J Photogramm Remote Sens 109(Supplement C):47–61

    Article  Google Scholar 

  37. Zollhȯfer M, Dai A, Innmann M, Wu C, Stamminger M, Theobalt C, Nießner M (2015) Shading-based refinement on volumetric signed distance functions. ACM Trans Graph 34(4):96,1–96,14

    Article  MATH  Google Scholar 

Download references

Acknowledgements

We thank all photographers who provided their images. We also like to thank Michael Goesele, Simon Fuhrmann and Romuald Perrot for their helpful comments and suggestions.

Funding

This study was not sponsored by any organization.

Author information

Authors and Affiliations

  1. Department of Computer Science, LESIA, University of Mohamed Khider, 07000, Biskra, Algeria

    Abdelhak Saouli & Mohamed Chaouki Babahenini

  2. LIRIS, University of Claude Bernard Lyon 43, Villeurbanne, France

    Sofiane Medjram

Authors
  1. Abdelhak Saouli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Chaouki Babahenini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sofiane Medjram
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abdelhak Saouli.

Ethics declarations

This manuscript has not been published and is not under consideration for publication elsewhere.

This article does not contain any studies with human or animal subjects.

Conflict of interests

Saouli Abdelhak declares that he has no conflict of interest. Mohamed Chaouki Babahenini declares that he has no conflict of interest. Sofiane Medjram declares that he has no conflict of interest

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saouli, A., Babahenini, M.C. & Medjram, S. Accurate, dense and shading-aware multi-view stereo reconstruction using metaheuritic optimization. Multimed Tools Appl 78, 15053–15077 (2019). https://doi.org/10.1007/s11042-018-6904-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6904-6

Keywords

Search

Navigation