International Journal of Computer Vision

, Volume 102, Issue 1–3, pp 91–111 | Cite as

Fully Automatic Registration of Image Sets on Approximate Geometry

  • M. Corsini
  • M. Dellepiane
  • F. Ganovelli
  • R. Gherardi
  • A. Fusiello
  • R. Scopigno
Article

Abstract

The photorealistic acquisition of 3D objects often requires color information from digital photography to be mapped on the acquired geometry, in order to obtain a textured 3D model. This paper presents a novel fully automatic 2D/3D global registration pipeline consisting of several stages that simultaneously register the input image set on the corresponding 3D object. The first stage exploits Structure From Motion (SFM) on the image set in order to generate a sparse point cloud. During the second stage, this point cloud is aligned to the 3D object using an extension of the 4 Point Congruent Set (4PCS) algorithm for the alignment of range maps. The extension accounts for models with different scales and unknown regions of overlap. In the last processing stage a global refinement algorithm based on mutual information optimizes the color projection of the aligned photos on the 3D object, in order to obtain high quality textures. The proposed registration pipeline is general, capable of dealing with small and big objects of any shape, and robust. We present results from six real cases, evaluating the quality of the final colors mapped onto the 3D object. A comparison with a ground truth dataset is also presented.

Keywords

2D-3D registration Image-geometry alignment Texture registration 3D scanning 

References

  1. Aiger, D., Mitra, N. J., & Cohen-Or, D. (2008). 4-Points congruent sets for robust pairwise surface registration. ACM Transactions on Graphics, 27, 85:1–85:10. CrossRefGoogle Scholar
  2. Arya, S., Mount, D. M., Netanyahu, N. S., Silverman, R., & Wu, A. Y. (1998). An optimal algorithm for approximate nearest neighbor searching fixed dimensions. Journal of the ACM, 45, 891–923. MathSciNetMATHCrossRefGoogle Scholar
  3. Besl, P. J., & McKay, N. D. (1992). A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2), 239–256. CrossRefGoogle Scholar
  4. Bonarrigo, F., & Signoroni, A. (2011). An enhanced ‘optimization-on-a-manifold’ framework for global registration of 3D range data. In Proc. of 3DIMPVT ’11 (pp. 350–357). Washington: IEEE Computer Society. Google Scholar
  5. Bradley, D., Boubekeur, T., & Heidrich, W. (2008). Accurate multi-view reconstruction using robust binocular stereo and surface meshing. In IEEE conference on computer vision and pattern recognition (CVPR2008) (pp. 1–8). CrossRefGoogle Scholar
  6. Brown, M., & Lowe, D. (2003). Recognising panoramas. In Proc. int. conf. computer vision (Vol. 2, pp. 1218–1225). CrossRefGoogle Scholar
  7. Brown, M., & Lowe, D. G. (2005). Unsupervised 3D object recognition and reconstruction in unordered datasets. In Proc. int. conf. on 3D digital imaging and modeling. Google Scholar
  8. Brunie, L., Lavallée, S., & Szeliski, R. (1992). Using force fields derived from 3D distance maps for inferring the attitude of a 3D rigid object. In Proc. of the second European conference on computer vision (ECCV’92) (pp. 670–675). Berlin: Springer. Google Scholar
  9. Callieri, M., Cignoni, P., Corsini, M., & Scopigno, R. (2008). Masked photo blending: mapping dense photographic dataset on high-resolution 3D models. Computers & Graphics, 32(4), 464–473. CrossRefGoogle Scholar
  10. Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., & Ranzuglia, G. (2008). Meshlab: an open-source mesh processing tool. In Sixth eurographics Italian chapter conference (pp. 129–136). Eurographics. Google Scholar
  11. Cleju, I., & Saupe, D. (2007). Stochastic optimization of multiple texture registration using mutual information. In Proceedings of the 29th DAGM conference on pattern recognition (pp. 517–526). Berlin: Springer. Google Scholar
  12. Cohen-Steiner, D., Alliez, P., & Desbrun, M. (2004). Variational shape approximation. ACM Transactions on Graphics, 23, 905–914. CrossRefGoogle Scholar
  13. Corsini, M., Dellepiane, M., Ponchio, F., & Scopigno, R. (2009). Image-to-geometry registration: a mutual information method exploiting illumination-related geometric properties. Computer Graphics Forum, 28(7), 1755–1764. CrossRefGoogle Scholar
  14. Dellepiane, M., Callieri, M., Ponchio, F., & Scopigno, R. (2008). Mapping highly detailed colour information on extremely dense 3d models: the case of David’s restoration. Computer Graphics Forum, 27(8), 2178–2187. CrossRefGoogle Scholar
  15. Dellepiane, M., Marroquim, R., Callieri, M., Cignoni, P., & Scopigno, R. (2012). Flow-based local optimization for image-to-geometry projection. IEEE Transactions on Visualization and Computer Graphics, 18(3), 463–474. CrossRefGoogle Scholar
  16. Eisemann, M., De Decker, B., Magnor, M., Bekaert, P., de Aguiar, E., Ahmed, N., Theobalt, C., & Sellent, A. (2008). Floating textures. Computer Graphics Forum (Proc. of Eurographics), 27(2), 409–418. CrossRefGoogle Scholar
  17. Farenzena, M., Fusiello, A., & Gherardi, R. (2009). Structure-and-motion pipeline on a hierarchical cluster tree. In IEEE int. workshop on 3-D digital imaging and modeling, Kyoto, Japan. Google Scholar
  18. Fitzgibbon, A. W., & Zisserman, A. (1998). Automatic camera recovery for closed and open image sequences. In Proc. Europ. conf. computer vision (ECCV1998) (pp. 311–326). Google Scholar
  19. Franken, T., Dellepiane, M., Ganovelli, F., Cignoni, P., Montani, C., & Scopigno, R. (2005). Minimizing user intervention in registering 2D images to 3D models. The Visual Computer, 21(8–10), 619–628. CrossRefGoogle Scholar
  20. Früh, C., & Zakhor, A. (2003). Constructing 3D city models by merging aerial and ground views. IEEE Computer Graphics and Applications, 23, 52–61. CrossRefGoogle Scholar
  21. Furukawa, Y., & Ponce, J. (2010). Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(8), 1362–1376. CrossRefGoogle Scholar
  22. Gehua Yang, G. Y., Becker, J., & Stewart, C. V. (2007). Estimating the location of a camera with respect to a 3d model. In Proc. of the sixth international conference on 3-D digital imaging and modeling (3DIM2007) (pp. 159–166). Los Alamitos: IEEE Computer Society. CrossRefGoogle Scholar
  23. Gherardi, R., Farenzena, M., & Fusiello, A. (2010). Improving the efficiency of hierarchical structure-and-motion. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR 2010) (pp. 1594–1600). Google Scholar
  24. Gibson, S., Cook, J., Howard, T., Hubbold, R., & Oram, D. (2002). Accurate camera calibration for off-line, video-based augmented reality. In IEEE/ACM int. symp. on mixed and augmented reality. Google Scholar
  25. Goesele, M., Curless, B., & Seitz, S. M. (2006). Multi-view stereo revisited. In Proc. of CVPR ’06 (Vol. 2, pp. 2402–2409). Los Alamitos: IEEE Computer Society. Google Scholar
  26. Horn, B. K. P. (1987). Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America, A, 4(4), 629–642. MathSciNetCrossRefGoogle Scholar
  27. Ikeuchi, K., Nakazawa, A., Hasegawa, K., & Ohishi, T. (2003). The great Buddha project: modeling cultural heritage for vr systems through observation. In Proc. of the 2nd IEEE/ACM international symposium on mixed and augmented reality (ISMAR’03) (p. 7). Los Alamitos: IEEE Computer Society. CrossRefGoogle Scholar
  28. Irschara, A., Zach, C., & Bischof, H. (2007). Towards wiki-based dense city modeling. In Proc. int. conf. computer vision (ICCV2007) (pp. 1–8). Google Scholar
  29. Johnson, A. (1997). Spin-images: a representation for 3-d surface matching. Ph.D. thesis, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA. Google Scholar
  30. Kalogerakis, E., Nowrouzezahrai, D., Simari, P., & Singh, K. (2009). Extracting lines of curvature from noisy point clouds. Computer Aided Design, 41(4), 282–292. CrossRefGoogle Scholar
  31. Kamberov, G., Kamberova, G., Chum, O., Obdrzalek, S., Martinec, D., Kostkova, J., Pajdla, T., Matas, J., & Sara, R. (2006). 3D geometry from uncalibrated images. In Proc. 2nd int. symp. on visual computing. Google Scholar
  32. Kazhdan, M., Bolitho, M., & Hoppe, H. (2006). Poisson surface reconstruction. In A. Sheffer & K. Polthier (Eds.), Eurographics symposium on geometry processing (SGP2006) (pp. 61–70). Cagliari: Eurographics Association. Google Scholar
  33. Krishnan, S., Lee, P. Y., Moore, J. B., & Venkatasubramanian, S. (2005). Global registration of multiple 3D point sets via optimization-on-a-manifold. In Proc. of the 3rd eurographics symposium on geometry processing (SGP2005). Cagliari: Eurographics Association. Google Scholar
  34. Krishnan, S., Lee, P. Y., Moore, J. B., & Venkatasubramanian, S. (2007). Optimisation-on-a-manifold for global registration of multiple 3D point sets. International Journal of Intelligent Systems Technologies and Applications, 3(3/4), 319–340. CrossRefGoogle Scholar
  35. Lensch, H. P. A., Heidrich, W., & Seidel, H. P. (2000). Automated texture registration and stitching for real world models. In PG ’00: proceedings of the 8th pacific conference on computer graphics and applications (p. 317). Los Alamitos: IEEE Computer Society. CrossRefGoogle Scholar
  36. Li, X., & Guskov, I. (2005). Multi-scale features for approximate alignment of point-based surfaces. In Proc. of the 3rd eurographics symposium on geometry processing (SGP2005). Cagliari: Eurographics Association. Google Scholar
  37. Liu, L., & Stamos, I. (2005). Automatic 3D to 2D registration for the photorealistic rendering of urban scenes. In CVPR (Vol. 2, pp. 137–143). Los Alamitos: IEEE Computer Society. Google Scholar
  38. Liu, L., Stamos, I., Yu, G., Wolberg, G., & Zokai, S. (2006). Multiview geometry for texture mapping 2D images onto 3D range data. In Proc. of the 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06) (Vol. 2, pp. 2293–2300). Los Alamitos: IEEE Computer Society. Google Scholar
  39. Lowe, D. G. (1991). Fitting parameterized three-dimensional models to images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13, 441–450. CrossRefGoogle Scholar
  40. Maes, F., Collignon, A., Vandeermeulen, D., Marchal, G., & Suetens, P. (1997). Multimodality image registration by maximization of mutual information. IEEE Transactions on Medical Imaging, 16, 187–198. CrossRefGoogle Scholar
  41. Makadia, A., Patterson, A., & Daniilidis, K. (2006). Fully automatic registration of 3d point clouds. In IEEE computer society conference on computer vision and pattern recognition (Vol. 1, pp. 1297–1304). Google Scholar
  42. Malzbender, T., Gelb, D., & Wolters, H. (2001). Polynomial texture maps. In SIGGRAPH’01 (pp. 519–528). New York: ACM. Google Scholar
  43. Matsushita, K., & Kaneko, T. (1999). Efficient and handy texture mapping on 3D surfaces. Computer Graphics Forum, 18(3), 349–358. CrossRefGoogle Scholar
  44. Neugebauer, P. J., & Klein, K. (1999). Texturing 3D models of real world objects from multiple unregistered photographic views. Computer Graphics Forum, 18(3), 245–256. CrossRefGoogle Scholar
  45. Ni, K., Steedly, D., & Dellaert, F. (2007). Out-of-core bundle adjustment for large-scale 3D reconstruction. In Proc. int. conf. computer vision (pp. 1–8). Google Scholar
  46. Nistér, D. (2000). Reconstruction from uncalibrated sequences with a hierarchy of trifocal tensors. In Proc. Europ. conf. computer vision (ECCV2000) (pp. 649–663). Google Scholar
  47. Pintus, R., Gobbetti, E., & Combet, R. (2011). Fast and robust semi-automatic registration of photographs to 3D geometry. In The 12th international symposium on virtual reality, archaeology and cultural heritage. To appear. Google Scholar
  48. Pluim, J., Maintz, J., & Viergever, M. (2003). Mutual-information-based registration of medical images: a survey. IEEE Transactions on Medical Imaging, 22(8), 986–1004. CrossRefGoogle Scholar
  49. Pons, J. P., Keriven, R., & Faugeras, O. (2007). Multi-view stereo reconstruction and scene flow estimation with a global image-based matching score. International Journal of Computer Vision, 72(2), 179–193. CrossRefGoogle Scholar
  50. Portelli, D., Ganovelli, F., Tarini, M., Cignoni, P., Dellepiane, M., & Scopigno, R. (2010). A framework for user-assisted sketch-based fitting of geometric primitives. In Proceedings of WSCG, the 18th int. conference on computer graphics, visualization and computer vision. Google Scholar
  51. Pottmann, H., Huang, Q. X., Yang, Y. L., & Hu, S. M. (2006). Geometry and convergence analysis of algorithms for registration of 3d shapes. International Journal of Computer Vision, 67(3), 277–296. CrossRefGoogle Scholar
  52. Powell, M. J. D. (2008). Developments of NEWUOA for minimization without derivatives. IMA Journal of Numerical Analysis, 28(4), 649–664. MathSciNetMATHCrossRefGoogle Scholar
  53. Pulli, K. (1999). Multiview registration for large data sets. In Proc. of the 2nd international conference on 3-D digital imaging and modeling (3DIM’99) (pp. 160–168). Washington, DC: IEEE Computer Society. CrossRefGoogle Scholar
  54. Pulli, K., Abi-Rached, H., Duchamp, T., Shapiro, L. G., & Stuetzle, W. (1998). Acquisition and visualization of colored 3D objects. In Proc. of the 14th international conference on pattern recognition (ICPR’98) (Vol. 1, p. 11). Los Alamitos: IEEE Computer Society. Google Scholar
  55. Rusinkiewicz, S., & Levoy, M. (2001). Efficient variants of the ICP algorithm. In Proc. of the third international conference on 3-D digital imaging and modeling (pp. 145–152). CrossRefGoogle Scholar
  56. Sequeira, V., & Goncalves, J. G. (2002). 3D reality modelling: Photo-realistic 3d models of real world scenes. In International symposium on 3D data processing visualization and transmission (p. 776). CrossRefGoogle Scholar
  57. Shum, H. Y., Ke, Q., & Zhang, Z. (1999). Efficient bundle adjustment with virtual key frames: a hierarchical approach to multi-frame structure from motion. In Proc. int. conf. computer vision and pattern rec. Google Scholar
  58. Skelly, L., & Sclaroff, S. (2007). Improved feature descriptors for 3-D surface matching. In Proc. SPIE conf. on two- and three-dimensional methods for inspection and metrology V (Vol. 6762). Google Scholar
  59. Snavely, N., Seitz, S. M., & Szeliski, R. (2006). Photo tourism: exploring photo collections in 3D. In SIGGRAPH’06 (pp. 835–846). Google Scholar
  60. Sottile, M., Dellepiane, M., Cignoni, P., & Scopigno, R. (2010). Mutual correspondences: an hybrid method for image-to-geometry registration. In Eurographics Italian chapter conference 2010 (pp. 81–88). EG Google Scholar
  61. Stamos, I., Liu, L., Chen, C., Wolberg, G., Yu, G., & Zokai, S. (2008). Integrating automated range registration with multiview geometry for the photorealistic modeling of large-scale scenes. International Journal of Computer Vision, 78, 237–260. CrossRefGoogle Scholar
  62. Steedly, D., Essa, I., & Dellaert, F. (2003). Spectral partitioning for structure from motion. In Proc. int. conf. computer vision (ICCV2003) (pp. 649–663). Google Scholar
  63. Strecha, C., von Hansen, W., Van Gool, L., Fua, P., & Thoennessen, U. (2008). On benchmarking camera calibration and multi-view stereo for high resolution imagery. In CVPR’08 (pp. 1–8). Google Scholar
  64. Vergauwen, M., & Gool, L. V. (2006). Web-based 3D reconstruction service. Machine Vision and Applications, 17(6), 411–426. CrossRefGoogle Scholar
  65. Viola, P., William, M., & Wells, I. (1997). Alignment by maximization of mutual information. International Journal of Computer Vision, 24(2), 137–154. CrossRefGoogle Scholar
  66. Wu, C., Clipp, B., Li, X., Frahm, J. M., & Pollefeys, M. (2008). 3d model matching with viewpoint-invariant patches (vip). In CVPR’08. Los Alamitos: IEEE Computer Society. Google Scholar
  67. Zhao, W., Nister, D., & Hsu, S. (2005). Alignment of continuous video onto 3d point clouds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8), 1305–1318. CrossRefGoogle Scholar
  68. Zheng, H., Cleju, I., & Saupe, D. (2009). Highly-automatic mi based multiple 2D/3D image registration using self-initialized geodesic feature correspondences. In H. Zha, R. I. Taniguchi, & S. J. Maybank (Eds.), Lecture notes in computer science: Vol. 3. ACCV (Vol. 5996, pp. 426–435). Berlin: Springer. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • M. Corsini
    • 1
  • M. Dellepiane
    • 1
  • F. Ganovelli
    • 1
  • R. Gherardi
    • 2
  • A. Fusiello
    • 3
  • R. Scopigno
    • 1
  1. 1.Visual Computing Lab (ISTI-CNR)PisaItaly
  2. 2.Toshiba Cambridge Research LaboratoryCambridgeUK
  3. 3.University of VeronaVeronaItaly

Personalised recommendations