Skip to main content

Advertisement

SpringerLink
Log in

Integrating Automated Range Registration with Multiview Geometry for the Photorealistic Modeling of Large-Scale Scenes

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

The photorealistic modeling of large-scale scenes, such as urban structures, requires a fusion of range sensing technology and traditional digital photography. This paper presents a system that integrates automated 3D-to-3D and 2D-to-3D registration techniques, with multiview geometry for the photorealistic modeling of urban scenes. The 3D range scans are registered using our automated 3D-to-3D registration method that matches 3D features (linear or circular) in the range images. A subset of the 2D photographs are then aligned with the 3D model using our automated 2D-to-3D registration algorithm that matches linear features between the range scans and the photographs. Finally, the 2D photographs are used to generate a second 3D model of the scene that consists of a sparse 3D point cloud, produced by applying a multiview geometry (structure-from-motion) algorithm directly on a sequence of 2D photographs. The last part of this paper introduces a novel algorithm for automatically recovering the rotation, scale, and translation that best aligns the dense and sparse models. This alignment is necessary to enable the photographs to be optimally texture mapped onto the dense model. The contribution of this work is that it merges the benefits of multiview geometry with automated registration of 3D range scans to produce photorealistic models with minimal human interaction. We present results from experiments in large-scale urban scenes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ambler, A. P., Barrow, H. G., Brown, C. M., Burstall, R. M., & Popplestone, R. J. (1973). A versatile computer-controlled assembly system. In The 3rd international joint conference on artificial intelligence.

  • Beardsley, P. A., Zisserman, A. P., & Murray, D. W. (1997). Sequential updating of projective and affine structure from motion. International Journal of Computer Vision 23(3), 235–259.

    Article  Google Scholar 

  • Bellon, O. R. P., & Silva, L. (2002). New improvements to range image segmentation by edge detection. IEEE Signal Processing Letters 9(2), 23–45.

    Article  Google Scholar 

  • Bernardini, F., & Rushmeier, H. (2002). The 3D model acquisition pipeline. Computer Graphics Forum 21(2), 149–172.

    Article  Google Scholar 

  • Besl, P. J., & Jain, R. C. (1988). Segmentation through variable-order surface fitting. IEEE Transactions on Pattern Analysis and Machine Intelligence 10(2), 167–192.

    Article  Google Scholar 

  • Besl, P. J., & McKay, N. D. (1992). A method for registration of 3D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2) (1992).

  • Cass, T. (1997). Polynomial–time geometric matching for object recognition. International Journal of Computer Vision 21(1–2), 37–61.

    Article  Google Scholar 

  • Chen, C. (2007). Range segmentation and registration for 3D modeling of large-scale urban scenes. PhD thesis, City University of New York.

  • Chen, C., & Stamos, I. (2005). Semi-automatic range to range registration: a feature-based method. In The 5th international conference on 3-D digital imaging and modeling (pp. 254–261), Ottawa, June 2005.

  • Chen, C., & Stamos, I. (2006). Range image registration based on circular features. In International symposium on 3D data processing, visualization and transmission, Chapel Hill, June 2006.

  • Christy, S., & Horaud, R. (1999). Iterative pose computation from line correspondences. Journal of Computer Vision and Image Understanding 73(1), 137–144.

    Article  MATH  Google Scholar 

  • Faugeras, O. (1996). Three-dimensional computer vision. Cambridge: MIT Press.

    Google Scholar 

  • Faugeras, O., Luong, Q. T., & Papadopoulos, T. (2001). The geometry of multiple images. Cambridge: MIT Press.

    MATH  Google Scholar 

  • Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Graphics and Image Processing 24(6), 381–395.

    MathSciNet  Google Scholar 

  • Früh, C., & Zakhor, A. (2003). Constructing 3D city models by merging aerial and ground views. Computer Graphics and Applications 23(6), 52–11.

    Article  Google Scholar 

  • Hartley, R., & Zisserman, A. (2003). Multiple view geometry in computer vision. (2nd ed.). Cambridge: Cambridge University Press.

    Google Scholar 

  • Hausler, G., & Ritter, D. (1999). Feature-based object recognition and localization in 3D-space, using a single video image. Journal of Computer Vision and Image Understanding 73(1), 64–81.

    Article  Google Scholar 

  • Heyden, A., & Astrom, K. (1996) Euclidean reconstruction from constant intrinsic parameters. In International conference on pattern recognition (Vol. 1, pp. 339–343) (1992).

  • Horaud, R., Dornaika, F., Lamiroy, B., & Christy, S. (1997). Object pose: the link between weak perspective, paraperspective, and full perspective. International Journal of Computer Vision, 22(2).

  • Huber, D. F., & Hebert, M. (2003). Fully automatic registration of multiple 3D data sets. Image and Vision Computing 21(7), 637–650.

    Article  Google Scholar 

  • Huttenlocher, D., & Ullman, S. (1990). Recognizing solid objects by alignment with an image. International Journal of Computer Vision 5(7), 195–212.

    Article  Google Scholar 

  • Ikeuchi, K. (2003). The great Buddha project, In IEEE ISMAR03, Tokyo, Japan, November 2003.

  • Jacobs, D. W. (1997). Matching 3-D models to 2-D images. International Journal of Computer Vision 21(1–2), 123–153.

    Article  Google Scholar 

  • Johnson, A. E., & Hebert, M. (1999). Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(5), 433–449.

    Article  Google Scholar 

  • Jurie, F. (1999). Solution of the simultaneous pose and correspondence problem using Gaussian error model. Journal of Computer Vision and Image Understanding 73(3), 357–373.

    Article  MATH  Google Scholar 

  • Leica Geosystems. http://hds.leica-geosystems.com/.

  • Liu, L. (2007). Automated registration of 2D images with 3D range data in a photorealistic modeling system of urban scenes. PhD thesis, City University of New York.

  • Liu, L., & Stamos, I. (2005). Automatic 3D to 2D registration for the photorealistic rendering of urban scenes. In IEEE conference on computer vision and pattern recognition (Vol. II, pp. 137–143), San Diego, CA, June 2005.

  • Liu, L., & Stamos, I. (2007). A systematic approach for 2D-image to 3D-range registration in urban environments. In Virtual Representations and modeling of large-scale environments (VRML), in conjuction with ICCV. Brazil: Rio De Janeiro.

    Google Scholar 

  • Liu, L., Stamos, I., Yu, G., Wolberg, G., & Zokai, S. (2006). Multiview geometry for texture mapping 2D images onto 3D range data. In IEEE conference on computer vision and pattern recognition (Vol. II, pp. 2293–2300), New York City, June 2006.

  • Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoint. International Journal of Computer Vision 60(2), 91–110.

    Article  Google Scholar 

  • Ma, Y., Soatto, S., Kosecka, J., & Sastry, S. (2003). An invitation to 3-D vision: from images to geometric models. Berlin: Springer.

    Google Scholar 

  • Marshall, G. L. D., & Martin, R. (2001). Robust segmentation of primitives from range data in the presence of geometric degeneracy. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(3), 304–314.

    Article  Google Scholar 

  • Oberkampf, D., DeMenthon, D., & Davis, L. (1996) Iterative pose estimation using coplanar feature points. Computer Vision Graphics and Image Processing, 63(3) (1996).

  • Pollefeys, M., & Gool, L. V. (1997). A stratified approach to metric self-calibration. In IEEE conference on computer vision and pattern recognition (pp. 407–412) (1997).

  • Pollefeys, M., Gool, L. V., Vergauwen, M., Verbiest, F., Cornelis, K., Tops, J., & Koch, R. (2004). Visual modeling with a hand-held camera. International Journal of Computer Vision 59(3), 207–232.

    Article  Google Scholar 

  • Pulli, K., & Pietikãinen, M. (1993). Range image segmentation based on decomposition of surface normals. In Proceedings of the Scandinavian conference on image analysis.

  • Pulli, K., Abi-Rached, H., Duchamp, T., Shapiro, L. G., & Stuetzle, W. (1998). Acquisition and visualization of colored 3D objects. In International conference on pattern recognition (Vol. 1, p. 11), Australia, 1998.

  • Quan, L., & Lan, Z. (1999). Linear N–point camera pose determination. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(7).

  • Rusinkiewicz, S., & Levoy, M. (2001). Efficient variants of the ICP algorithm. In The 3rd international conference on 3-D digital imaging and modeling, June 2001.

  • Schaffalitzky, F., & Zisserman, A. (2001). Viewpoint invariant texture matching and wide baseline stereo. In IEEE international conference on computer vision (pp. 636–643), July 2001.

  • Sequeira, V., & Concalves, J. (2002). 3D reality modeling: photo-realistic 3D models of real world scenes. In International symposium on 3D data processing, visualization and transmission (pp. 776–783).

  • Stamos, I., & Allen, P. K. (2001). Automatic registration of 3-D with 2-D imagery in urban environments. In IEEE International conference on computer vision (pp. 731–736).

  • Stamos, I., & Allen, P. K. (2002). Geometry and texture recovery of scenes of large scale. Journal of Computer Vision and Image Understanding 88(2), 94–118.

    Article  MATH  Google Scholar 

  • Stamos, I., & Leordeanu, M. (2003). Automated feature-based range registration of urban scenes of large scale. In IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 555–561).

  • Triggs, B. (1996). Factorization methods for projective structure and motion. In IEEE conference on computer vision and pattern recognition (pp. 845–851).

  • Troccoli, A., & Allen, P. K. (2004). A shadow based method for image to model registration. In 2nd IEEE workshop on video and image registration, July 2004.

  • Tuytelaars, T., & Gool, L. J. V. (2004). Matching widely separated views based on affine invariant regions. International Journal of Computer Vision 59(1), 61–85.

    Article  Google Scholar 

  • Visual Information Technology Group, Canada. (2005). http://iit-iti.nrc-cnrc.gc.ca/about-sujet/vit-tiv_e.html.

  • Wami, M. A., & Batchelor, B. G. (1994). Edge-region-based segmentation of range images. IEEE Transactions on Pattern Analysis and Machine Intelligence 16(3), 314–319.

    Article  Google Scholar 

  • Wells, W. (1997). Statistical approaches to feature–based object recognition. International Journal of Computer Vision 21(1–2), 63–98.

    Article  Google Scholar 

  • Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(11), 1330–1334.

    Article  Google Scholar 

  • Zhao, H., & Shibasaki, R. (2003). Reconstructing a textured CAD model of an urban environment using vehicle-borne laser range scanners and line cameras. Machine Vision and Applications 14(1), 35–41.

    Article  Google Scholar 

  • Zhao, W., Nister, D., & Hsu, S. (2005). Alignment of continuous video onto 3D point clouds. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(8), 1305–1318.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Hunter College/CUNY, New York, NY, USA

    Ioannis Stamos, Lingyun Liu & Chao Chen

  2. Department of Computer Science, City College of New York/CUNY, New York, NY, USA

    George Wolberg & Gene Yu

  3. Brainstorm Technology LLC, New York, NY, USA

    Siavash Zokai

Authors
  1. Ioannis Stamos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lingyun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. George Wolberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gene Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Siavash Zokai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ioannis Stamos.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Stamos, I., Liu, L., Chen, C. et al. Integrating Automated Range Registration with Multiview Geometry for the Photorealistic Modeling of Large-Scale Scenes. Int J Comput Vis 78, 237–260 (2008). https://doi.org/10.1007/s11263-007-0089-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-007-0089-1

Keywords

Search

Navigation