International Journal of Computer Vision

, Volume 106, Issue 2, pp 172–191

A Super-Resolution Framework for High-Accuracy Multiview Reconstruction

  • Bastian Goldlücke
  • Mathieu Aubry
  • Kalin Kolev
  • Daniel Cremers
Article

Abstract

We present a variational framework to estimate super-resolved texture maps on a 3D geometry model of a surface from multiple images. Given the calibrated images and the reconstructed geometry, the proposed functional is convex in the super-resolution texture. Using a conformal atlas of the surface, we transform the model from the curved geometry to the flat charts and solve it using state-of-the-art and provably convergent primal–dual algorithms. In order to improve image alignment and quality of the texture, we extend the functional to also optimize for a normal displacement map on the surface as well as the camera calibration parameters. Since the sub-problems for displacement and camera parameters are non-convex, we revert to relaxation schemes in order to robustly estimate a minimizer via sequential convex programming. Experimental results confirm that the proposed super-resolution framework allows to recover textured models with significantly higher level-of-detail than the individual input images.

Keywords

Multi-view 3D reconstruction Texture reconstruction Super-resolution Camera calibration Variational methods 

References

  1. Allne, C., Pons, J. P., & Keriven, R. (2008). Seamless image-based texture atlases using multi-band blending. In 19th International Conference on Pattern Recognition.Google Scholar
  2. Attouch, H., Buttazzo, G., & Michaille, G. (2006). Variational analysis in Sobolev and BV spaces. MPS-SIAM series on optimization. Philadelphia, PA: Society for Industrial and Applied Mathematics.CrossRefGoogle Scholar
  3. Aubry, M., Kolev, K., Goldluecke, B., & Cremers, D. (2011). Decoupling photometry and geometry in dense variational camera calibration. In Proceedings of ICCV.Google Scholar
  4. Baker, S., & Kanade, T. (2002). Limits on super-resolution and how to break them. PAMI, 24(9), 1167–1183.CrossRefGoogle Scholar
  5. Bernardini, F., Martin, I., & Rushmeier, H. (2001). High-quality texture reconstruction from multiple scans. IEEE Transactions on Visualization and Computer Graphics, 7(4), 318–332.CrossRefGoogle Scholar
  6. Bertalmio, M., Sapiro, G., Cheng, L. T., & Osher, S. (2001). Variational problems and PDEs on implicit surfaces. In Proceedings of the IEEE Workshop on Variational and Level Set Methods (VLSM01) (pp. 186–193).Google Scholar
  7. Bhat, P., Zitnick, C., Snavelny, N., Agarwala, A., Agrawala, M., Cohen, M., et al. (2007). Using photographs to enhance videos of a static scene. In Eurographics Symposium on Rendering.Google Scholar
  8. Bresson, X., Esedoglu, S., Vandergheynst, P., Thiran, J. P., & Osher, S. (2007). Fast global minimization of the active contour/snake model. Journal of Mathematical Imaging and Vision, 28, 151–167.CrossRefMathSciNetGoogle Scholar
  9. Capel, D., & Zisserman, A. (2001). Super-resolution from multiple views using learnt image models. In Proceedings of CVPR (Vol. 2, pp. 627–634).Google Scholar
  10. Chambolle, A., & Pock, T. (2011). A first-order primal–dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision, 40(1), 120–145.CrossRefMATHMathSciNetGoogle Scholar
  11. Donnelly, W. (2005). GPU Gems 2, chapter per-pixel displacement mapping with distance functions. Amsterdam: Addison-Wesley Longman.Google Scholar
  12. Eisemann, M., Decker, B. D., Magnor, M., Bekaert, P., de Aguiar, E., Ahmed, N., et al. (2008). Floating textures. Computer Graphics Forum (Proceedings of Eurographics), 27(2), 409–418.CrossRefGoogle Scholar
  13. Floater, M. S., & Hormann, K. (2006). Surface parameterization: A tutorial and survey. In Advances in multiresolution for geometric modelling, mathematics and visualization (pp. 157–168). Berlin: Springer.Google Scholar
  14. Fransens, R., Strecha, C., & van Gool, L. (2007). Optical flow based super-resolution: A probabilistic approach. Computer Vision and Image Understanding, 106(1), 106–115.CrossRefGoogle Scholar
  15. Furukawa, Y., & Ponce, J. (2009). Accurate camera calibration from multi-view stereo and bundle adjustment. International Journal of Computer Vision, 84, 257–268.CrossRefGoogle Scholar
  16. Goldluecke, B., & Cremers, D. (2009a). A superresolution framework for high-accuracy multiview reconstruction. In Pattern Recognition (Proceedings of DAGM).Google Scholar
  17. Goldluecke, B., & Cremers, D. (2009b). Superresolution texture maps for multiview reconstruction. In Proceedings of ICCV.Google Scholar
  18. Goldluecke, B., Strekalovskiy, E., & Cremers, D. (2012). The natural total variation which arises from geometric measure theory. SIAM Journal on Imaging Sciences.Google Scholar
  19. Gu, X., & Yau, S.T. (2003). Global conformal surface parameterization. In Proceedings of the 2003 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (Vol. 43, pp. 127–137).Google Scholar
  20. Hartley, R., & Zisserman, A. (2004). Multiple view geometry in computer vision (2nd ed.). Cambridge, MA: Cambridge University Press.CrossRefMATHGoogle Scholar
  21. Jin, H., Cremers, D., Wang, D., Yezzi, A., Prados, E., & Soatto, S. (2008). 3-D reconstruction of shaded objects from multiple images under unknown illumination. International Journal of Computer Vision, 76(3), 245–256.CrossRefGoogle Scholar
  22. Klein, G., & Murray, D. (2007). Parallel tracking and mapping for small AR workspaces. In 6th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 225–234). http://ewokrampage.wordpress.com/.
  23. Kolev, K., Klodt, M., Brox, T., & Cremers, D. (2009). Continuous global optimization in multiview 3D reconstruction. International Journal of Computer Vision, 84(1), 80–96.CrossRefGoogle Scholar
  24. Kolev, K., Klodt, M., Brox, T., Esedoglu, S., & Cremers, D. (2007). Continuous global optimization in multiview 3D reconstruction. In Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR) (Vol. 4679, pp. 441–452).Google Scholar
  25. Kolev, K., Pock, T., & Cremers, D. (2010). Anisotropic minimal surfaces integrating photoconsistency and normal information for multiview stereo. In European Conference on Computer Vision (ECCV), Heraklion, Greece.Google Scholar
  26. Lempitsky, V., & Ivanov, D. (2007). Seamless mosaicing of image-based texture maps. In Proceedings of CVPR (Vol. 1, pp. 1–6).Google Scholar
  27. Lensch, H., Heidrich, W., & Seidel, H. P. (2001). A silhouette-based algorithm for texture registration and stitching. Graphical Models, 63(4), 245–262.CrossRefMATHGoogle Scholar
  28. Lévy, B., Petitjean, S., Ray, N., & Maillot, J. (2003). Least squares conformal maps for automatic texture atlas generation. ACM Transactions on Graphics (SIGGRAPH), 21(3), 362–371.Google Scholar
  29. Lui, L. M., Wang, Y., & Chan, T. F. (2005) Solving PDEs on manifold using global conformal parameterization. In Variational, Geometric, and Level Set Methods in Computer Vision: Third International Workshop (VLSM) (pp. 309–319).Google Scholar
  30. Mitzel, D., Pock, T., Schoenemann, T., & Cremers, D. (2009). Video super resolution using duality based TV-L1 optical flow. In Pattern Recognition (Proceedings of DAGM), Jena, Germany.Google Scholar
  31. Pock, T., Cremers, D., Bischof, H., & Chambolle, A. (2009). An algorithm for minimizing the piecewise smooth Mumford–Shah functional. In Proceedings of ICCV, Kyoto, Japan.Google Scholar
  32. Pock, T., Schoenemann, T., Graber, G., Bischof, H., & Cremers, D. (2008). A convex formulation of continuous multi-label problems. In Proceedings of ECCV.Google Scholar
  33. Rav-Acha, A., Kohli, P., Rother, C., & Fitzgibbon, A. (2008). Unwrap mosaics: A new representation for video editing. Proceedings of the ACM SIGGRAPH, 27(3), 17–25.CrossRefGoogle Scholar
  34. Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D, 60, 259 –268.Google Scholar
  35. Schoenemann, T., & Cremers, D. (2008). High resolution motion layer decomposition using dual-space graph cuts. In Proceedings of CVPR (pp. 1–7).Google Scholar
  36. Seitz, S., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006). A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proceedings of CVPR (pp. 519–528).Google Scholar
  37. Snavely, N., Seitz, S., & Szeliski, R. (2006). Phototourism: Exploring image collections in 3D. In Proceedings of the ACM SIGGRAPH. http://phototour.cs.washington.edu/bundler/.
  38. Sroubek, F., Cristobal, G., & Flusser, J. (2007). A unified approach to superresolution and multichannel blind deconvolution. IEEE Transactions on Image Processing, 16(9), 2322–2332.CrossRefMathSciNetGoogle Scholar
  39. Stam, J. (2003). Flows on surfaces of arbitrary topology. ACM Transactions on Graphics (SIGGRAPH), 22(3), 724–731.CrossRefGoogle Scholar
  40. Strecha, C., von Hansen, W., Gool, L. V., Fua, P., & Thoennessen, U. (2008). On benchmarking camera calibration and multi-view stereo for high resolution imagery. In Proceedings of CVPR.Google Scholar
  41. Theobalt, C., Ahmed, N., Lensch, H., Magnor, M., & Seidel, H. P. (2007). Seeing people in different light-joint shape, motion, and reflectance capture. IEEE Transactions on Visualization and Computer Graphics, 13(4), 663–674.CrossRefGoogle Scholar
  42. Triggs, B., McLauchlan, P., Hartley, R., & Fitzgibbon, A. (2000). Bundle adjustment—A modern synthesis. In Vision algorithms: Theory and practice, lecture notes in computer science (Vol. 1883, pp. 298–372). Berlin: Springer. Google Scholar
  43. Unal, G., Yezzi, A., Soatto, S., & Slabaugh, G. (2007). A variational approach to problems in calibration of multiple cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 1322–1338.Google Scholar
  44. Wang, Y., Gu, X., Hayashi, K., Chan, T. F., Thompson, P., & Yau, S. T. (2005). Surface parameterization using Riemann surface structure. In Proceedings of ICCV (Vol. 2, pp. 1061–1066).Google Scholar
  45. Wanner, S., & Goldluecke, B. (2012). Spatial and angular variational super-resolution of 4D light fields. In Proceedings of ECCV.Google Scholar
  46. Zach, C., Pock, T., & Bischof, H. (2007). A duality based approach for realtime TV-L1 optical flow. In Pattern Recognition (Proceedings of DAGM) (pp. 214–223).Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Bastian Goldlücke
    • 1
  • Mathieu Aubry
    • 2
  • Kalin Kolev
    • 3
  • Daniel Cremers
    • 2
  1. 1.Heidelberg Collaboratory for Image ProcessingUniversity of HeidelbergHeidelbergGermany
  2. 2.Computer Science DepartmentTechnical University of MunichMunichGermany
  3. 3.Department of Computer ScienceETH ZurichZurichSwitzerland

Personalised recommendations