Skip to main content
SpringerLink
Log in

A Super-Resolution Framework for High-Accuracy Multiview Reconstruction

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

We present a variational framework to estimate super-resolved texture maps on a 3D geometry model of a surface from multiple images. Given the calibrated images and the reconstructed geometry, the proposed functional is convex in the super-resolution texture. Using a conformal atlas of the surface, we transform the model from the curved geometry to the flat charts and solve it using state-of-the-art and provably convergent primal–dual algorithms. In order to improve image alignment and quality of the texture, we extend the functional to also optimize for a normal displacement map on the surface as well as the camera calibration parameters. Since the sub-problems for displacement and camera parameters are non-convex, we revert to relaxation schemes in order to robustly estimate a minimizer via sequential convex programming. Experimental results confirm that the proposed super-resolution framework allows to recover textured models with significantly higher level-of-detail than the individual input images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Notes

  1. http://cocolib.net.

  2. The multi-view datasets in Fig. 18 are publicly available on our webpage, http://cvpr.in.tum.de.

References

  • Allne, C., Pons, J. P., & Keriven, R. (2008). Seamless image-based texture atlases using multi-band blending. In 19th International Conference on Pattern Recognition.

  • Attouch, H., Buttazzo, G., & Michaille, G. (2006). Variational analysis in Sobolev and BV spaces. MPS-SIAM series on optimization. Philadelphia, PA: Society for Industrial and Applied Mathematics.

    Book  Google Scholar 

  • Aubry, M., Kolev, K., Goldluecke, B., & Cremers, D. (2011). Decoupling photometry and geometry in dense variational camera calibration. In Proceedings of ICCV.

  • Baker, S., & Kanade, T. (2002). Limits on super-resolution and how to break them. PAMI, 24(9), 1167–1183.

    Article  Google Scholar 

  • Bernardini, F., Martin, I., & Rushmeier, H. (2001). High-quality texture reconstruction from multiple scans. IEEE Transactions on Visualization and Computer Graphics, 7(4), 318–332.

    Article  Google Scholar 

  • Bertalmio, M., Sapiro, G., Cheng, L. T., & Osher, S. (2001). Variational problems and PDEs on implicit surfaces. In Proceedings of the IEEE Workshop on Variational and Level Set Methods (VLSM’01) (pp. 186–193).

  • Bhat, P., Zitnick, C., Snavelny, N., Agarwala, A., Agrawala, M., Cohen, M., et al. (2007). Using photographs to enhance videos of a static scene. In Eurographics Symposium on Rendering.

  • Bresson, X., Esedoglu, S., Vandergheynst, P., Thiran, J. P., & Osher, S. (2007). Fast global minimization of the active contour/snake model. Journal of Mathematical Imaging and Vision, 28, 151–167.

    Article  MathSciNet  Google Scholar 

  • Capel, D., & Zisserman, A. (2001). Super-resolution from multiple views using learnt image models. In Proceedings of CVPR (Vol. 2, pp. 627–634).

  • Chambolle, A., & Pock, T. (2011). A first-order primal–dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision, 40(1), 120–145.

    Article  MATH  MathSciNet  Google Scholar 

  • Donnelly, W. (2005). GPU Gems 2, chapter per-pixel displacement mapping with distance functions. Amsterdam: Addison-Wesley Longman.

    Google Scholar 

  • Eisemann, M., Decker, B. D., Magnor, M., Bekaert, P., de Aguiar, E., Ahmed, N., et al. (2008). Floating textures. Computer Graphics Forum (Proceedings of Eurographics), 27(2), 409–418.

    Article  Google Scholar 

  • Floater, M. S., & Hormann, K. (2006). Surface parameterization: A tutorial and survey. In Advances in multiresolution for geometric modelling, mathematics and visualization (pp. 157–168). Berlin: Springer.

  • Fransens, R., Strecha, C., & van Gool, L. (2007). Optical flow based super-resolution: A probabilistic approach. Computer Vision and Image Understanding, 106(1), 106–115.

    Article  Google Scholar 

  • Furukawa, Y., & Ponce, J. (2009). Accurate camera calibration from multi-view stereo and bundle adjustment. International Journal of Computer Vision, 84, 257–268.

    Article  Google Scholar 

  • Goldluecke, B., & Cremers, D. (2009a). A superresolution framework for high-accuracy multiview reconstruction. In Pattern Recognition (Proceedings of DAGM).

  • Goldluecke, B., & Cremers, D. (2009b). Superresolution texture maps for multiview reconstruction. In Proceedings of ICCV.

  • Goldluecke, B., Strekalovskiy, E., & Cremers, D. (2012). The natural total variation which arises from geometric measure theory. SIAM Journal on Imaging Sciences.

  • Gu, X., & Yau, S.T. (2003). Global conformal surface parameterization. In Proceedings of the 2003 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (Vol. 43, pp. 127–137).

  • Hartley, R., & Zisserman, A. (2004). Multiple view geometry in computer vision (2nd ed.). Cambridge, MA: Cambridge University Press.

    Book  MATH  Google Scholar 

  • Jin, H., Cremers, D., Wang, D., Yezzi, A., Prados, E., & Soatto, S. (2008). 3-D reconstruction of shaded objects from multiple images under unknown illumination. International Journal of Computer Vision, 76(3), 245–256.

    Article  Google Scholar 

  • Klein, G., & Murray, D. (2007). Parallel tracking and mapping for small AR workspaces. In 6th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 225–234). http://ewokrampage.wordpress.com/.

  • Kolev, K., Klodt, M., Brox, T., & Cremers, D. (2009). Continuous global optimization in multiview 3D reconstruction. International Journal of Computer Vision, 84(1), 80–96.

    Article  Google Scholar 

  • Kolev, K., Klodt, M., Brox, T., Esedoglu, S., & Cremers, D. (2007). Continuous global optimization in multiview 3D reconstruction. In Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR) (Vol. 4679, pp. 441–452).

  • Kolev, K., Pock, T., & Cremers, D. (2010). Anisotropic minimal surfaces integrating photoconsistency and normal information for multiview stereo. In European Conference on Computer Vision (ECCV), Heraklion, Greece.

  • Lempitsky, V., & Ivanov, D. (2007). Seamless mosaicing of image-based texture maps. In Proceedings of CVPR (Vol. 1, pp. 1–6).

  • Lensch, H., Heidrich, W., & Seidel, H. P. (2001). A silhouette-based algorithm for texture registration and stitching. Graphical Models, 63(4), 245–262.

    Article  MATH  Google Scholar 

  • Lévy, B., Petitjean, S., Ray, N., & Maillot, J. (2003). Least squares conformal maps for automatic texture atlas generation. ACM Transactions on Graphics (SIGGRAPH), 21(3), 362–371.

    Google Scholar 

  • Lui, L. M., Wang, Y., & Chan, T. F. (2005) Solving PDEs on manifold using global conformal parameterization. In Variational, Geometric, and Level Set Methods in Computer Vision: Third International Workshop (VLSM) (pp. 309–319).

  • Mitzel, D., Pock, T., Schoenemann, T., & Cremers, D. (2009). Video super resolution using duality based TV-L1 optical flow. In Pattern Recognition (Proceedings of DAGM), Jena, Germany.

  • Pock, T., Cremers, D., Bischof, H., & Chambolle, A. (2009). An algorithm for minimizing the piecewise smooth Mumford–Shah functional. In Proceedings of ICCV, Kyoto, Japan.

  • Pock, T., Schoenemann, T., Graber, G., Bischof, H., & Cremers, D. (2008). A convex formulation of continuous multi-label problems. In Proceedings of ECCV.

  • Rav-Acha, A., Kohli, P., Rother, C., & Fitzgibbon, A. (2008). Unwrap mosaics: A new representation for video editing. Proceedings of the ACM SIGGRAPH, 27(3), 17–25.

    Article  Google Scholar 

  • Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D, 60, 259 –268.

    Google Scholar 

  • Schoenemann, T., & Cremers, D. (2008). High resolution motion layer decomposition using dual-space graph cuts. In Proceedings of CVPR (pp. 1–7).

  • Seitz, S., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006). A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proceedings of CVPR (pp. 519–528).

  • Snavely, N., Seitz, S., & Szeliski, R. (2006). Phototourism: Exploring image collections in 3D. In Proceedings of the ACM SIGGRAPH. http://phototour.cs.washington.edu/bundler/.

  • Sroubek, F., Cristobal, G., & Flusser, J. (2007). A unified approach to superresolution and multichannel blind deconvolution. IEEE Transactions on Image Processing, 16(9), 2322–2332.

    Article  MathSciNet  Google Scholar 

  • Stam, J. (2003). Flows on surfaces of arbitrary topology. ACM Transactions on Graphics (SIGGRAPH), 22(3), 724–731.

    Article  Google Scholar 

  • Strecha, C., von Hansen, W., Gool, L. V., Fua, P., & Thoennessen, U. (2008). On benchmarking camera calibration and multi-view stereo for high resolution imagery. In Proceedings of CVPR.

  • Theobalt, C., Ahmed, N., Lensch, H., Magnor, M., & Seidel, H. P. (2007). Seeing people in different light-joint shape, motion, and reflectance capture. IEEE Transactions on Visualization and Computer Graphics, 13(4), 663–674.

    Article  Google Scholar 

  • Triggs, B., McLauchlan, P., Hartley, R., & Fitzgibbon, A. (2000). Bundle adjustment—A modern synthesis. In Vision algorithms: Theory and practice, lecture notes in computer science (Vol. 1883, pp. 298–372). Berlin: Springer.

  • Unal, G., Yezzi, A., Soatto, S., & Slabaugh, G. (2007). A variational approach to problems in calibration of multiple cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 1322–1338.

    Google Scholar 

  • Wang, Y., Gu, X., Hayashi, K., Chan, T. F., Thompson, P., & Yau, S. T. (2005). Surface parameterization using Riemann surface structure. In Proceedings of ICCV (Vol. 2, pp. 1061–1066).

  • Wanner, S., & Goldluecke, B. (2012). Spatial and angular variational super-resolution of 4D light fields. In Proceedings of ECCV.

  • Zach, C., Pock, T., & Bischof, H. (2007). A duality based approach for realtime TV-L1 optical flow. In Pattern Recognition (Proceedings of DAGM) (pp. 214–223).

Download references

Acknowledgments

We thank Martin R. Oswald for providing the visualization in Fig. 1. This work was supported by the ERC Starting Grant “Convex Vision”.

Author information

Authors and Affiliations

  1. Heidelberg Collaboratory for Image Processing, University of Heidelberg, Heidelberg, Germany

    Bastian Goldlücke

  2. Computer Science Department, Technical University of Munich, Munich, Germany

    Mathieu Aubry & Daniel Cremers

  3. Department of Computer Science, ETH Zurich, Zurich, Switzerland

    Kalin Kolev

Authors
  1. Bastian Goldlücke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mathieu Aubry
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kalin Kolev
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniel Cremers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bastian Goldlücke.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Goldlücke, B., Aubry, M., Kolev, K. et al. A Super-Resolution Framework for High-Accuracy Multiview Reconstruction. Int J Comput Vis 106, 172–191 (2014). https://doi.org/10.1007/s11263-013-0654-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-013-0654-8

Keywords

Search

Navigation