Techniques for Seamless Color Registration and Mapping on Dense 3D Models

  • Ruggero Pintus
  • Enrico Gobbetti
  • Marco Callieri
  • Matteo Dellepiane
Chapter
Part of the Geotechnologies and the Environment book series (GEOTECH, volume 16)

Abstract

Today’s most widely used 3D digitization approach is a combination of active geometric sensing, mainly using laser scanning, with active or passive color sensing, mostly using digital photography. Producing a seamless colored object, starting from a geometric representation and a set of photographs, is a data fusion problem requiring effective solutions for image-to-geometry registration, and color mapping and blending. This chapter provides a brief survey of the state-of-the-art solutions, ranging from manual approaches to fully scalable automated methods.

Keywords

Image-to-geometry registration Color mapping Color blending 3D reconstruction 

References

  1. Agus M, Gobbetti E, Villanueva AJ, Mura C, Pajarola R (2014) SOAR: Stochastic optimization for affine global point set registration. In: Proceedings of the 19th international workshop on vision, modeling and visualization (VMV), Oct 2014, Darmstadt, pp 103–110Google Scholar
  2. Aiger D, Mitra NJ, Cohen-Or D (2008) 4-points congruent sets for robust pairwise surface registration. ACM Trans Graph (TOG) 27:85CrossRefGoogle Scholar
  3. Allène C, Philippe Pons J, Keriven R (2008) Seamless image-based texture atlases using multi-band blending. In: Proceeding of the ICPR. IEEE, pp 1–4Google Scholar
  4. Arikan M, Preiner R, Scheiblauer C, Jeschke S, Wimmer M (2014) Large-scale point-cloud visualization through localized textured surface reconstruction. IEEE Trans Vis Comput Graph 20(9):1280–1292CrossRefGoogle Scholar
  5. Banno A, Ikeuchi K (2010) Omnidirectional texturing based on robust 3D registration through Euclidean reconstruction from two spherical images. Comput Vis Image Underst 114(4):491–499CrossRefGoogle Scholar
  6. Baumberg A (2002) Blending images for texturing 3D models. In: Proceeding of the BMVC, Citeseer, vol 3, p 5Google Scholar
  7. Berkiten S, Fan X, Rusinkiewicz S (2014) Merge2-3D: combining multiple normal maps with 3D surfaces. In: 2nd international conference on 3D vision, Dec 2014, vol 1. IEEE, Los Alamitos/Washington/Tokyo, pp 440–447CrossRefGoogle Scholar
  8. Bettio F, Gobbetti E, Merella E, Pintus R (2013) Improving the digitization of shape and color of 3D artworks in a cluttered environment. In: Proceedings of the digital heritage, Oct 2013. IEEE, Piscataway, pp 23–30. Best Paper AwardGoogle Scholar
  9. Bettio F, Villanueva AJ, Merella E, Marton F, Gobbetti E, Pintus R (2015) Mont’e Scan: effective shape and color digitization of cluttered 3D artworks. J Comput Cult Herit (JOCCH) 8(1):4:1–4:23Google Scholar
  10. Bernardini F, Martin IM, Rushmeier H (2001) High-quality texture reconstruction from multiple scans. IEEE Trans Vis Comput Graph 7(4):318–332CrossRefGoogle Scholar
  11. Bernardini F, Rushmeier H (2002) The 3D model acquisition pipeline. Comput Gr Forum 21:149–172CrossRefGoogle Scholar
  12. Borgeat L, Poirier G, Beraldin A, Godin G, Massicotte P, Picard M (2009) A framework for the registration of color images with 3D models. In: Proceedings of the 16th international conference on image processing (ICIP’09), IEEE Press, Piscataway, NJ, USA, pp 69–72Google Scholar
  13. Brunie L, Lavallée S, Szeliski R (1992) Using force fields derived from 3D distance maps for inferring the attitude of a 3D rigid object. In: Computer Vision ECCV’92. Springer, pp 670–675Google Scholar
  14. Callieri M, Cignoni P, Corsini M, Scopigno R (2008) Masked photo blending: mapping dense photographic data set on high-resolution sampled 3D models. Comput Graph 32(4):464–473CrossRefGoogle Scholar
  15. Callieri M, Cignoni P, Scopigno R (2002) Reconstructing textured meshes from multiple range RGB maps. In: VMV, Erlangen, pp 419–426Google Scholar
  16. Chuang M, Luo L, Brown BJ, Rusinkiewicz S, Kazhdan M (2009) Estimating the laplace-beltrami operator by restricting 3D functions. Comput Gr Forum 28:1475–1484. Wiley Online LibraryGoogle Scholar
  17. Cleju I, Saupe D (2007) Stochastic optimization of multiple texture registration using mutual information. Springer, KonstanzCrossRefGoogle Scholar
  18. Corsini M, Callieri M, Cignoni P (2008) Stereo light probe. Comput Gr Forum 27:291–300. Wiley Online LibraryGoogle Scholar
  19. Corsini M, Dellepiane M, Ganovelli F, Gherardi R, Fusiello A, Scopigno R (2013) Fully automatic registration of image sets on approximate geometry. Int J Comput Vis 102(1–3):91–111CrossRefGoogle Scholar
  20. Corsini M, Dellepiane M, Ponchio F, Scopigno R (2009) Image-to-geometry registration: a mutual information method exploiting illumination-related geometric properties. Comput Gr Forum 28:1755–1764. Wiley Online LibraryGoogle Scholar
  21. Debevec P, Yizhou Yu, Borshukov G (1998) Efficient view-dependent image-based rendering with projective texture-mapping. Springer, BerkeleyCrossRefGoogle Scholar
  22. Dellepiane M, Benedetti L, Scopigno R (2010) Removing shadows for color projection using sun position estimation. In: Proceedings of the 11th international conference on virtual reality, archaeology and cultural heritage. Eurographics Association, pp 55–62Google Scholar
  23. Dellepiane M, Callieri M, Corsini M, Cignoni P, Scopigno R (2010) Improved color acquisition and mapping on 3D models via flash-based photography. J Comput Cult Herit (JOCCH) 2(4):9Google Scholar
  24. Dellepiane M, Callieri M, Ponchio F, Scopigno R (2008) Mapping highly detailed colour information on extremely dense 3D models: the case of David’s restoration. Comput Gr Forum 27:2178–2187. Wiley Online LibraryGoogle Scholar
  25. Dellepiane M, Marroquim R, Callieri M, Cignoni P, Scopigno R (2012) Flow-based local optimization for image-to-geometry projection. IEEE Trans Vis Comput Graph 18(3):463–474CrossRefGoogle Scholar
  26. Dellepiane M, Scopigno R (2013) Global refinement of image-to-geometry registration for color projection on 3D models. In: Digital heritage international congress. The Eurographics Association, pp 39–46Google Scholar
  27. Eisemann M, De Decker B, Magnor M, Bekaert P, De Aguiar E, Ahmed N, Theobalt C, Sellent A (2008) Floating textures. Comput Gr Forum 27:409–418. Wiley Online LibraryGoogle Scholar
  28. Franken T, Dellepiane M, Ganovelli F, Cignoni P, Montani C, Scopigno R (2005) Minimizing user intervention in registering 2D images to 3D models. Vis Commun 21(8–10):619–628CrossRefGoogle Scholar
  29. Frahm JM, Fite-Georgel P, Gallup D, Johnson T, Raguram R, Wu C, Jen Y-H, Dunn E, Clipp B, Lazebnik S et al (2010) Building Rome on a cloudless day. In: Computer vision–ECCV 2010. Springer, pp 368–381Google Scholar
  30. Fruh C, Zakhor A (2001) Fast 3D model generation in urban environments. In: Proceedings of the MFI. IEEE, pp 165–170Google Scholar
  31. Fruh C, Zakhor A (2003) Constructing 3D city models by merging aerial and ground views. Comput Gr Appl IEEE 23(6):52–61CrossRefGoogle Scholar
  32. Gal R, Wexler Y, Ofek E, Hoppe H, Cohen-Or D (2010) Seamless montage for texturing models. Comput Gr Forum 29:479–486. Wiley Online LibraryGoogle Scholar
  33. Goldluecke B, Cremers D (2009) Superresolution texture maps for multiview reconstruction. In: Proceedings of the ICCV. IEEE, pp 1677–1684Google Scholar
  34. Guo R, Dai Q, Hoiem D (2011) Single-image shadow detection and removal using paired regions. In: Proceedings of the CVPR. IEEE, pp 2033–2040Google Scholar
  35. Haber T, Fuchs C, Bekaer P, Seidel H-P, Goesele M, Lensch HP et al (2009) Relighting objects from image collections. In: Proceedings of the CVPR. IEEE, pp 627–634Google Scholar
  36. Hantak C, Lastra A (2006) Metrics and optimization techniques for registration of color to laser range scans. In: International symposium on 3D data processing, visualization, and transmission, third. IEEE, pp 551–558Google Scholar
  37. Ikeuchi K, Oishi T, Takamatsu J, Sagawa R, Nakazawa A, Kurazume R, Nishino K, Kamakura M, Okamoto Y (2007) The Great Buddha project: digitally archiving, restoring, and analyzing cultural heritage objects. Int J Comput Vis 75(1):189–208CrossRefGoogle Scholar
  38. Kaminsky RS, Snavely N, Seitz SM, Szeliski R (2009) Alignment of 3D point clouds to overhead images. In: Proceedings of the CVPR workshops. IEEE, pp 63–70Google Scholar
  39. Koutsoudis A, Vidmar B, Ioannakis G, Arnaoutoglou F, Pavlidis G, Chamzas C (2014) Multi-image 3D reconstruction data evaluation. J Cult Herit 15(1):73–79CrossRefGoogle Scholar
  40. Kim SJ, Lin HT, Lu Z, Suesstrunk S, Lin S, Brown MS (2012) A new in-camera imaging model for color computer vision and its application. IEEE Trans PAMI 34(12):2289–2302CrossRefGoogle Scholar
  41. Lensch H, Heidrich W, Seidel H-P (2000) Automated texture registration and stitching for real world models. In: Proceedings of the pacific graphics. IEEE, pp 317–452Google Scholar
  42. Lensch HPA, Heidrich W, Seidel H-P (2001) A silhouette-based algorithm for texture registration and stitching. Graph Model 63(4):245–262CrossRefGoogle Scholar
  43. Lempitsky V, Ivanov D (2007) Seamless Mosaicing of image-based texture maps. In: Proceedings of the CVPR. IEEE, pp 1–6Google Scholar
  44. Levoy M, Pulli K, Curless B, Rusinkiewicz S, Koller D, Pereira L, Ginzton M, Anderson S, Davis J, Ginsberg J et al (2000) The digital michelangelo project: 3D scanning of large statues. In: Proceedings of the SIGGRAPH. ACM, New York, pp 131–144Google Scholar
  45. Li Y, Low K-L (2009) Automatic registration of color images to 3D geometry. In: Proceedings of the CGI. CGS, pp 21–28Google Scholar
  46. Liu L, Stamos I (2005) Automatic 3D to 2D registration for the photorealistic rendering of urban scenes. In: Proceedings of the CVPR. IEEE, vol 2, pp 137–143Google Scholar
  47. Liu L, Stamos I, Gene Yu, Wolberg G, Zokai S (2006) Multiview geometry for texture mapping 2D images onto 3D range data. In: Proceedings of the CVPR. IEEE, vol 2, pp 2293–2300Google Scholar
  48. Lowe DG (1991) Fitting parameterized three-dimensional models to images. IEEE Trans Pattern Anal Mach Intell 13(5):441–450CrossRefGoogle Scholar
  49. Malzbender T, Gelb D, Wolters H (2001) Polynomial texture maps. In: Proceedings of the 28th annual conference on computer graphics and interactive techniques. ACM, pp 519–528Google Scholar
  50. Mastin A, Kepner J, Fisher J (2009) Automatic registration of LIDAR and optical images of urban scenes. In: Proceedings of the CVPR. IEEE, pp 2639–2646Google Scholar
  51. Matsushita K, Kaneko T (1999) Efficient and handy texture mapping on 3D surfaces. Comput Gr Forum, Blackwell Publishers, Oxford, UK, vol 18, pp 349–358Google Scholar
  52. Nehab D, Rusinkiewicz S, Davis J, Ramamoorthi R (2005) Efficiently combining positions and normals for precise 3D geometry. ACM Trans Graph (TOG) 24(3):536–543CrossRefGoogle Scholar
  53. Neugebauer PJ, Klein K (1999) Texturing 3D models of real world objects from multiple unregistered photographic views. Comput Gr Forum 18:245–256. Wiley Online LibraryGoogle Scholar
  54. Palma G, Callieri M, Dellepiane M, Scopigno R (2012) A statistical method for SVBRDF approximation from video sequences in general lighting conditions. Comput Gr Forum 31:1491–1500. Wiley Online LibraryGoogle Scholar
  55. Pintus R, Gobbetti E (2015) A fast and robust framework for semi-automatic and automatic registration of photographs to 3D geometry. J Comput Cult Herit (JOCCH) 7(4):23:1–23:23Google Scholar
  56. Pintus R, Gobbetti E, Callieri M (2011a) Fast low-memory seamless photo blending on massive point clouds using a streaming framework. J Comput Cult Herit (JOCCH) 4(2):6Google Scholar
  57. Pintus R, Gobbetti E, Callieri M (2011b) A streaming framework for seamless detailed photo blending on massive point clouds. In: Proceedings of the eurographics area papers. pp 25–32Google Scholar
  58. Pintus R, Gobbetti E, Combet R (2011c) Fast and robust semi-automatic registration of photographs to 3D geometry. In: The 12th international symposium on virtual reality, archaeology and cultural heritage Oct 2011. Eurographics Association, pp 9–16Google Scholar
  59. Pulli K, Abi-Rached H, Duchamp T, Shapiro LG, Stuetzle W (1998) Acquisition and visualization of colored 3D objects. In: Proceedings of the ICPR. IEEE, pp 11–15Google Scholar
  60. Remondino F (2011) Heritage recording and 3D modeling with photogrammetry and 3D scanning. Remote Sens 3(6):1104–1138CrossRefGoogle Scholar
  61. Rocchini C, Cignoni P, Montani C, Scopigno R (2002) Acquiring, stitching and blending diffuse appearance attributes on 3D models. Vis Comput 18(3):186–204CrossRefGoogle Scholar
  62. Sequeira V, Gonçalves GMJ (2002) 3D reality modelling: photo-realistic 3D models of real world scenes. In: Proceedings of the 3DPVT. IEEE, pp 776–783Google Scholar
  63. Sinha SN, Steedly D, Szeliski R, Agrawala M, Pollefeys M (2008) Interactive 3D architectural modeling from unordered photo collections. ACM Trans Gr (TOG) 27:159Google Scholar
  64. Stamos I, Alien PK (2001) Automatic registration of 2-D with 3-D imagery in urban environments. In: Proceedings of the ICCV. IEEE, vol 2, pp 731–736Google Scholar
  65. Stamos I, Liu L, Chen C, Wolberg G, Yu G, Zokai S (2008) Integrating automated range registration with multiview geometry for the photorealistic modeling of large-scale scenes. Int J Comput Vis 78(2–3):237–260CrossRefGoogle Scholar
  66. Viola P, Wells WM III (1997) Alignment by maximization of mutual information. Int J Comput Vis 24(2):137–154CrossRefGoogle Scholar
  67. Waechter M, Moehrle N, Goesele M (2014) Let there be color! large-scale texturing of 3D reconstructions. In: Computer vision–ECCV 2014. Springer, pp 836–850Google Scholar
  68. Walkowski F, Johnston RA, Price NB (2008) Texture mapping for the Fastscan hand-held laser scanner. In: Proceedings of the IVCNZ. IEEE, pp 1–6Google Scholar
  69. Williams N, Low K-L, Hantak C, Pollefeys M, Lastra A (2004) Automatic image alignment for 3D environment modeling. In: Proceedings of the computer graphics and image processing. IEEE, pp 388–395Google Scholar
  70. Wu C, Clipp B, Xiaowei Li, Frahm J-M, Pollefeys M (2008) 3D model matching with viewpoint-invariant patches (VIP). In: Proceedings of the CVPR. IEEE, pp 1–8Google Scholar
  71. Xu S-C, Ye X, Wu Y, Zhang S (2005) Highlight detection and removal based on chromaticity. In: Image analysis and recognition. Springer, pp 199–206Google Scholar
  72. Xu L, Li E, Li J, Chen Y, Zhang Y (2010) A general texture mapping framework for image-based 3D modeling. In: Proceedings of the international conference on image processing, Hong kong, pp 2713–2716Google Scholar
  73. Yang G, Becker J, Stewart CV (2007) Estimating the location of a camera with respect to a 3D model. In: Proceedings of the 3DIM. IEEE, pp 159–166Google Scholar
  74. Zhao W, Nister D, Hsu S (2005) Alignment of continuous video onto 3D point clouds. IEEE Trans Pattern Anal Mach Intell 27(8):1305–1318CrossRefGoogle Scholar
  75. Zheng H, Cleju I, Saupe D (2010) Highly-automatic mi based multiple 2D/3D image registration using self-initialized geodesic feature correspondences. In: Computer vision–ACCV 2009. Springer, pp 426–435Google Scholar
  76. Zhou Q-Y, Koltun V (2014) Color map optimization for 3D reconstruction with consumer depth cameras. ACM Trans Gr (TOG) 33(4):155Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Ruggero Pintus
    • 1
  • Enrico Gobbetti
    • 1
  • Marco Callieri
    • 2
  • Matteo Dellepiane
    • 2
  1. 1.CRS4, Visual Computing GroupCagliariItaly
  2. 2.ISTI-CNR, Visual Computing GroupPisaItaly

Personalised recommendations