Pixelwise View Selection for Unstructured Multi-View Stereo

  • Johannes L. SchönbergerEmail author
  • Enliang Zheng
  • Jan-Michael Frahm
  • Marc Pollefeys
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9907)


This work presents a Multi-View Stereo system for robust and efficient dense modeling from unstructured image collections. Our core contributions are the joint estimation of depth and normal information, pixelwise view selection using photometric and geometric priors, and a multi-view geometric consistency term for the simultaneous refinement and image-based depth and normal fusion. Experiments on benchmarks and large-scale Internet photo collections demonstrate state-of-the-art performance in terms of accuracy, completeness, and efficiency.


Source Image Reprojection Error View Selection Source Patch Scene Representation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Supplementary material

Supplementary material 1 (mp4 24388 KB)

419975_1_En_31_MOESM2_ESM.pdf (5.3 mb)
Supplementary material 2 (pdf 5466 KB)


  1. 1.
    Schaffalitzky, F., Zisserman, A.: Multi-view matching for unordered image sets, or how do i organize my holiday snaps? In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part I. LNCS, vol. 2350, pp. 414–431. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  2. 2.
    Snavely, N., Seitz, S., Szeliski, R.: Photo tourism: exploring photo collections in 3D. In: ACM Transactions on Graphics (2006)Google Scholar
  3. 3.
    Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S., Szeliski, R.: Building Rome in a day. In: ICCV (2009)Google Scholar
  4. 4.
    Frahm, J.-M.: Building Rome on a cloudless day. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 368–381. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15561-1_27 CrossRefGoogle Scholar
  5. 5.
    Heinly, J., Schönberger, J.L., Dunn, E., Frahm, J.M.: Reconstructing the world* in six days *(as captured by the Yahoo 100 million image dataset). In: CVPR (2015)Google Scholar
  6. 6.
    Zheng, E., Wu, C.: Structure from motion using structure-less resection. In: ICCV (2015)Google Scholar
  7. 7.
    Schönberger, J.L., Radenović, F., Chum, O., Frahm, J.M.: From single image query to detailed 3D reconstruction. In: CVPR (2015)Google Scholar
  8. 8.
    Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: CVPR (2016)Google Scholar
  9. 9.
    Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. In: CVPR (2007)Google Scholar
  10. 10.
    Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Towards internet-scale multi-view stereo. In: CVPR (2010)Google Scholar
  11. 11.
    Bailer, C., Finckh, M., Lensch, H.P.A.: Scale robust multi view stereo. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 398–411. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33712-3_29 Google Scholar
  12. 12.
    Shan, Q., Adams, R., Curless, B., Furukawa, Y., Seitz, S.M.: The visual turing test for scene reconstruction. In: 3DV (2013)Google Scholar
  13. 13.
    Shan, Q., Curless, B., Furukawa, Y., Hernandez, C., Seitz, S.M.: Occluding contours for multi-view stereo. In: CVPR (2014)Google Scholar
  14. 14.
    Zheng, E., Dunn, E., Jojic, V., Frahm, J.M.: Patchmatch based joint view selection and depthmap estimation. In: CVPR (2014)Google Scholar
  15. 15.
    Galliani, S., Lasinger, K., Schindler, K.: Massively parallel multiview stereopsis by surface normal diffusion. In: ICCV (2015)Google Scholar
  16. 16.
    Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M., Moore, R.: Real-time human pose recognition in parts from single depth images. Comm. ACM 56(1), 116–124 (2013)CrossRefGoogle Scholar
  17. 17.
    Chen, S.E., Williams, L.: View interpolation for image synthesis. In: Conference on Computer Graphics and Interactive Techniques (1993)Google Scholar
  18. 18.
    Forster, C., Pizzoli, M., Scaramuzza, D.: Air-ground localization and map augmentation using monocular dense reconstruction. In: IROS (2013)Google Scholar
  19. 19.
    Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: CVPR (2006)Google Scholar
  20. 20.
    Strecha, C., von Hansen, W., Gool, L.V., Fua, P., Thoennessen, U.: On benchmarking camera calibration and multi-view stereo for high resolution imagery. In: CVPR (2008)Google Scholar
  21. 21.
    Intille, S.S., Bobick, A.F.: Disparity-space images and large occlusion stereo. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 801, pp. 179–186. Springer, Heidelberg (1994)CrossRefGoogle Scholar
  22. 22.
    Kanade, T., Okutomi, M.: A stereo matching algorithm with an adaptive window: theory and experiment. IEEE Trans. Pattern Anal. Mach. Intell. 16(9), 920–932 (1994)CrossRefGoogle Scholar
  23. 23.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47(1), 7–42 (2002)CrossRefzbMATHGoogle Scholar
  24. 24.
    Rhemann, C., Hosni, A., Bleyer, M., Rother, C., Gelautz, M.: Fast cost-volume filtering for visual correspondence and beyond. In: CVPR (2011)Google Scholar
  25. 25.
    Campbell, N.D.F., Vogiatzis, G., Hernández, C., Cipolla, R.: Using multiple hypotheses to improve depth-maps for multi-view stereo. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5302, pp. 766–779. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-88682-2_58 CrossRefGoogle Scholar
  26. 26.
    Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Manhattan-world stereo. In: CVPR (2009)Google Scholar
  27. 27.
    Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Reconstructing building interiors from images. In: CVPR (2009)Google Scholar
  28. 28.
    Jancosek, M., Pajdla, T.: Multi-view reconstruction preserving weakly-supported surfaces. In: CVPR (2011)Google Scholar
  29. 29.
    Hane, C., Zach, C., Cohen, A., Angst, R., Pollefeys, M.: Joint 3D scene reconstruction and class segmentation. In: CVPR (2013)Google Scholar
  30. 30.
    Tung, T., Nobuhara, S., Matsuyama, T.: Complete multi-view reconstruction of dynamic scenes from probabilistic fusion of narrow and wide baseline stereo. In: ICCV (2009)Google Scholar
  31. 31.
    Ji, D., Dunn, E., Frahm, J.-M.: 3D reconstruction of dynamic textures in crowd sourced data. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 143–158. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10590-1_10 Google Scholar
  32. 32.
    Oswald, M., Cremers, D.: A convex relaxation approach to space time multi-view 3D reconstruction. In: ICCV Workshops (2013)Google Scholar
  33. 33.
    Martin-Brualla, R., Gallup, D., Seitz, S.M.: 3D time-lapse reconstruction from internet photos. In: ICCV (2015)Google Scholar
  34. 34.
    Radenović, F., Schönberger, J.L., Ji, D., Frahm, J.M., Chum, O., Matas, J.: From dusk till dawn: modeling in the dark. In: CVPR (2016)Google Scholar
  35. 35.
    Yang, Q., Wang, L., Yang, R., Stewénius, H., Nistér, D.: Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling. IEEE Trans. Pattern Anal. Mach. Intell. 31(3), 492–504 (2009)CrossRefGoogle Scholar
  36. 36.
    Sun, J., Li, Y., Kang, S.B., Shum, H.Y.: Symmetric stereo matching for occlusion handling. In: CVPR (2005)Google Scholar
  37. 37.
    Zitnick, C.L., Kanade, T.: A cooperative algorithm for stereo matching and occlusion detection. IEEE Trans. Pattern Anal. Mach. Intell. 22(7), 675–684 (2000)CrossRefGoogle Scholar
  38. 38.
    Kang, S.B., Szeliski, R., Chai, J.: Handling occlusions in dense multi-view stereo. In: CVPR (2001)Google Scholar
  39. 39.
    Strecha, C., Fransens, R., Van Gool, L.: Wide-baseline stereo from multiple views: a probabilistic account. In: CVPR (2004)Google Scholar
  40. 40.
    Strecha, C., Fransens, R., Van Gool, L.: Combined depth and outlier estimation in multi-view stereo. In: CVPR (2006)Google Scholar
  41. 41.
    Gallup, D., Frahm, J.M., Mordohai, P., Pollefeys, M.: Variable baseline/resolution stereo. In: CVPR (2008)Google Scholar
  42. 42.
    Gallup, D., Frahm, J.M., Mordohai, P., Yang, Q., Pollefeys, M.: Real-time plane-sweeping stereo with multiple sweeping directions. In: CVPR (2007)Google Scholar
  43. 43.
    Burt, P., Wixson, L., Salgian, G.: Electronically directed focal stereo. In: ICCV (1995)Google Scholar
  44. 44.
    Birchfield, S., Tomasi, C.: Multiway cut for stereo and motion with slanted surfaces. In: ICCV (1999)Google Scholar
  45. 45.
    Zabulis, X., Daniilidis, K.: Multi-camera reconstruction based on surface normal estimation and best viewpoint selection. In: 3DPVT (2004)Google Scholar
  46. 46.
    Bleyer, M., Rhemann, C., Rother, C.: Patchmatch stereo-stereo matching with slanted support windows. In: BMVC (2011)Google Scholar
  47. 47.
    Goesele, M., Snavely, N., Curless, B., Hoppe, H., Seitz, S.M.: Multi-view stereo for community photo collections. In: CVPR (2007)Google Scholar
  48. 48.
    Zach, C.: Fast and high quality fusion of depth maps. In: 3DPVT (2008)Google Scholar
  49. 49.
    Gallup, D., Pollefeys, M., Frahm, J.-M.: 3D reconstruction using an n-layer heightmap. In: Goesele, M., Roth, S., Kuijper, A., Schiele, B., Schindler, K. (eds.) Pattern Recognition. LNCS, vol. 6376, pp. 1–10. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  50. 50.
    Zheng, E., Dunn, E., Raguram, R., Frahm, J.M.: Efficient and scalable depthmap fusion. In: BMVC (2012)Google Scholar
  51. 51.
    Neal, R.M., Hinton, G.E.: A view of the EM algorithm that justifies incremental, sparse, and other variants. In: Jordan, M.I. (ed.) Learning in Graphical Models. Springer, Berlin (1998)Google Scholar
  52. 52.
    Heise, P., Jensen, B., Klose, S., Knoll, A.: Variational patchmatch multiview reconstruction and refinement. In: CVPR (2015)Google Scholar
  53. 53.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Berlin (2006)zbMATHGoogle Scholar
  54. 54.
    Hirschmüller, H., Scharstein, D.: Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Anal. Mach. Intell. 31(9), 1582–1599 (2009)CrossRefGoogle Scholar
  55. 55.
    Yoon, K.J., Kweon, I.S.: Locally adaptive support-weight approach for visual correspondence search. In: CVPR (2005)Google Scholar
  56. 56.
    Zhang, G., Jia, J., Wong, T.T., Bao, H.: Recovering consistent video depth maps via bundle optimization. In: CVPR (2008)Google Scholar
  57. 57.
    Merrell, P., Akbarzadeh, A., Wang, L., Mordohai, P., Frahm, J.M., Yang, R., Nistér, D., Pollefeys, M.: Real-time visibility-based fusion of depth maps. In: CVPR (2007)Google Scholar
  58. 58.
    Waechter, M., Moehrle, N., Goesele, M.: Let there be color! Large-scale texturing of 3D reconstructions. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 836–850. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10602-1_54 Google Scholar
  59. 59.
    Kazhdan, M., Hoppe, H.: Screened poisson surface reconstruction. ACM Trans. Graph. (TOG) 32(3), 29 (2013)CrossRefzbMATHGoogle Scholar
  60. 60.
    Hu, X., Mordohai, P.: Least commitment, viewpoint-based, multi-view stereo. In: 3DIMPVT (2012)Google Scholar
  61. 61.
    Tylecek, R., Sara, R.: Refinement of surface mesh for accurate multi-view reconstruction. IJVR 9(1), 45–54 (2010)Google Scholar
  62. 62.
    Zaharescu, A., Boyer, E., Horaud, R.: Topology-adaptive mesh deformation for surface evolution, morphing, and multiview reconstruction. IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 823–837 (2011)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Johannes L. Schönberger
    • 1
    Email author
  • Enliang Zheng
    • 2
  • Jan-Michael Frahm
    • 2
  • Marc Pollefeys
    • 1
    • 3
  1. 1.ETH ZürichZürichSwitzerland
  2. 2.UNC Chapel HillChapel HillUSA
  3. 3.MicrosoftRedmondUSA

Personalised recommendations