Skip to main content

Systems and Experiment Paper: Construction of Panoramic Image Mosaics with Global and Local Alignment

International Journal of Computer Vision volume 36, pages 101–130 (2000)Cite this article

An Erratum to this article was published on 01 July 2002

Abstract

This paper presents a complete system for constructing panoramic image mosaics from sequences of images. Our mosaic representation associates a transformation matrix with each input image, rather than explicitly projecting all of the images onto a common surface (e.g., a cylinder). In particular, to construct a full view panorama, we introduce a rotational mosaic representation that associates a rotation matrix (and optionally a focal length) with each input image. A patch-based alignment algorithm is developed to quickly align two images given motion models. Techniques for estimating and refining camera focal lengths are also presented.

In order to reduce accumulated registration errors, we apply global alignment (block adjustment) to the whole sequence of images, which results in an optimally registered image mosaic. To compensate for small amounts of motion parallax introduced by translations of the camera and other unmodeled distortions, we use a local alignment (deghosting) technique which warps each image based on the results of pairwise local image registrations. By combining both global and local alignment, we significantly improve the quality of our image mosaics, thereby enabling the creation of full view panoramic mosaics with hand-held cameras.

We also present an inverse texture mapping algorithm for efficiently extracting environment maps from our panoramic image mosaics. By mapping the mosaic onto an arbitrary texture-mapped polyhedron surrounding the origin, we can explore the virtual environment using standard 3D graphics viewers and hardware without requiring special-purpose players.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

References

  • Anandan, P. 1989. A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 2(3):283–310.

    Google Scholar 

  • Anandan, P., Sawhrey, H., Shashua, A., and Simoncelli, E. (Eds.) 1995. In IEEE Workshop on Representations of Visual Scenes, Cambridge, Massachusetts, IEEE Computer Society Press.

    Google Scholar 

  • Ayache, N. 1989. Vision Stér#x00E9;oscopique et Perception Multisensorielle. InterEditions: Paris.

    Google Scholar 

  • Bergen, J.R., Anandan, P., Hanna, K.J., and Hingorani, R. 1992. Hierarchical model-based motion estimation. In Second European Conference on Computer Vision (ECCV'92), Santa Margherita Liguere, Italy, Springer-Verlag, pp. 237–252.

    Google Scholar 

  • Black, M.J. and Rangarajan, A. 1996. On the unification of line processes, outlier rejection, and robust statistics with applications in early vision. International Journal of Computer Vision, 19(1):57–91.

    Google Scholar 

  • Burt, P.J. and Adelson, E.H. 1983. A multiresolution spline with applications to image mosaics. ACM Transactions on Graphics, 2(4):217–236.

    Google Scholar 

  • Chen, S. and Williams, L. 1993. View interpolation for image synthesis. In Computer Graphics (SIGGRAPH'93), pp. 279–288.

  • Chen, S.E. 1995. QuickTime VR—an image-based approach to virtual environment navigation. In Computer Graphics (SIGGRAPH' 95), pp. 29–38.

  • Chiang, M.-C. and Boult, T.E. 1996. Efficient image warping and super-resolution. In IEEE Workshop on Applications of Computer Vision (WACV'96), Sarasota, Florida, IEEE Computer Society, pp. 56–61.

    Google Scholar 

  • Danielsson, P.E. 1980. Euclidean distance mapping. Computer Graphics and Image Processing, 14:227–248.

    Google Scholar 

  • Davis, J. 1998. Mosaics of scenes with moving objects. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'98), Santa Barbara, pp. 354–360.

  • Fua, P. and Leclerc, Y.G. 1994. Using 3-dimensional meshes to combine image-based and geometry-based constraints. In Third European Conference on Computer Vision (ECCV'94), Stockholm, Sweden, Springer-Verlag, pp. 281–291.

    Google Scholar 

  • Golub, G. and Van Loan, C.F. 1996. Matrix Computation, 3rd edn. The John Hopkins University Press: Baltimore.

    Google Scholar 

  • Gortler, S.J., Grzeszczuk, R., Szeliski, R., and Cohen, M.F. 1996. The lumigraph. In Computer Graphics Proceedings, Annual Conference Series, Proc. SIGGRAPH'96, New Orleans, ACM SIGGRAPH, pp. 43–54.

  • Greene, N. 1986. Environment mapping and other applications of world projections. IEEE Computer Graphics and Applications, 6(11):21–29.

    Google Scholar 

  • Hansen, M., Anandan, P., Dana, K., van der Wal, G., and Burt, P. 1994. Real-time scene stabilization and mosaic construction. In IEEE Workshop on Applications of Computer Vision (WACV'94), Sarasota, Florida, pp. 54–62.

  • Hartley, R.I. 1994. Self-calibration from multiple views of a rotating camera. In Third European Conference on Computer Vision (ECCV'94), Stockholm, Sweden, Springer-Verlag, pp. 471–478.

    Google Scholar 

  • Heckbert, P. 1989. Fundamentals of Texture Mapping and Image Warping. Master' Thesis, The University of California at Berkeley.

    Google Scholar 

  • Horn, B.K.P. 1987. Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America A, 4(4):629–642.

    Google Scholar 

  • Horn, B.K.P., Hilden, H.M., and Negahdaripour, S. 1988. Closedform solution of absolute orientation using orthonormal matrices. Journal of the Optical Society of America A, 5(7):1127–1135.

    Google Scholar 

  • Irani, M. and Peleg, S. 1991. Improving resolution by image registration. Graphical Models and Image Processing, 53(3):231–239.

    Google Scholar 

  • Irani, M., Anandan, P., and Hsu, S. 1995a. Mosaic based representations of video sequences and their applications. In Fifth International Conference on Computer Vision (ICCV'95), Cambridge, Massachusetts, pp. 605–611.

  • Irani, M., Hsu, S., and Anandan, P. 1995b. Video compression using mosaic representations. Signal Processing: Image Communication, 7:529–552.

    Google Scholar 

  • Kanatani, K. 1994. Analyis of 3-D rotation fitting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(5):543–549.

    Google Scholar 

  • Kang, S.B. 1997. A survey of image-based rendering techniques. Technical Report 97/4, Digital Equipment Corporation, Cambridge Research Lab.

  • Kang, S.B. and Weiss, R. 1997. Characterization of errors in compositing panoramic images. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'97), San Juan, Puerto Rico, pp. 103–109.

  • Kumar, R., Anandan, P., and Hanna, K. 1994. Shape recovery from multiple views: Aparallax based approach. In Image Understanding Workshop, Morgan Kaufmann Publishers: Monterey, CA, pp. 947–955.

    Google Scholar 

  • Kumar, R., Anandan, P., Irani, M., Bergen, J., and Hanna, K. 1995. Representation of scenes from collections of images. In IEEE Workshop on Representations of Visual Scenes, Cambridge, Massachusetts, pp. 10–17.

  • Le Gall, D. 1991. MPEG: A video compression standard for multimedia applications. Communications of the ACM, 34(4):44–58.

    Google Scholar 

  • Lee, M.-C. et al. 1997. A layered video object coding system using sprite and affine motion model. IEEE Transactions on Circuits and Systems for Video Technology, 7(1):130–145.

    Google Scholar 

  • Levoy, M. and Hanrahan, P. 1996. Light field rendering. In Computer Graphics Proceedings, Annual Conference Series, Proc. SIGGRAPH'96, New Orleans, ACM SIGGRAPH, pp. 31–42.

  • Lucas, B.D. and Kanade, T. 1981. An iterative image registration technique with an application in stereo vision. In Seventh International Joint Conference on Artificial Intelligence (IJCAI-81), Vancouver, pp. 674–679.

  • Malde, H.E. 1983. Panoramic photographs. American Scientist, 71(2):132–140.

    Google Scholar 

  • Mann, S. and Picard, R.W. 1994. Virtual bellows: Constructing highquality images from video. In First IEEE International Conference on Image Processing (ICIP-94), Austin, Texas, pp. 363–367.

  • McMillan, L. and Bishop, G. 1995. Plenoptic modeling: An imagebased rendering system. In Computer Graphics (SIGGRAPH'95), pp. 39–46.

  • Meehan, J. 1990. Panoramic Photography. Watson-Guptill.

  • Milgram, D.L. 1977. Adaptive techniques for photomosaicking. IEEE Transactions on Computers, C-26(11):1175–1180.

    Google Scholar 

  • Moffitt, F.H. and Mikhail, E.M. 1980. Photogrammetry, 3rd edn. Harper & Row: New York.

    Google Scholar 

  • Murray, R.M., Li, Z.X., and Sastry, S.S. 1994. A Mathematical Introduction to Robotic Manipulation. CRC Press.

  • Nayar, S. 1997. Catadioptric omnidirectional camera. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'97), San Juan, Puerto Rico, pp. 482–488.

  • Nielson, G.M. 1993. Scattered data modeling. IEEE Computer Graphics and Applications, 13(1):60–70.

    Google Scholar 

  • Peleg, S. and Herman, J. 1997. Panoramic mosaics by manifold projection. In IEEE Computer Society Conference on Computer Vision andPattern Recognition (CVPR'97), San Juan, Puerto Rico, pp. 338–343.

  • Press, W.H., Flannery, B.P., Teukolsky, S.A., and Vetterling, W.T. 1992. Numerical Recipes in C: The Art of Scientific Computing, 2nd edn. Cambridge University Press: Cambridge, England.

    Google Scholar 

  • Quam, L.H. 1984. Hierarchicalwarp stereo. In Image Understanding Workshop, Science Applications International Corporation, New Orleans, Louisiana, pp. 149–155.

    Google Scholar 

  • Rosenfeld, A. and Kak, A.C. 1976. Digital Picture Processing. Academic Press: New York.

    Google Scholar 

  • Sato, Y., Wheeler, M., and Ikeuchi, K. 1997. Object shape and reflectance modeling from observation. In Computer Graphics Proceedings, Annual Conference Series, Proc. SIGGRAPH'97, Los Angeles, ACM SIGGRAPH, pp. 379–387.

  • Sawhney, H.S. 1994. Simplifying motion and structure analysis using planar parallax and image warping. In Twelfth International Conference on Pattern Recognition (ICPR'94), Jerusalem, Israel, IEEE Computer Society Press, pp. 403–408.

    Google Scholar 

  • Sawhney, H.S. and Ayer, S. 1996. Compact representation of videos through dominant multiple motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(8):814–830.

    Google Scholar 

  • Sawhney, H.S., Hsu, S., and Kumar, R. 1998. Robust video mosaicing through topology inference and local to global alignment. In Fifth European Conference on Computer Vision (ECCV'98), Freiburg, Germany, Springer-Verlag, pp. 103–119.

    Google Scholar 

  • Shi, J. and Tomasi, C. 1994. Good features to track. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'94), Seattle, Washington, IEEE Computer Society, pp. 593–600.

    Google Scholar 

  • Shum, H.-Y. and Szeliski, R. 1997. Panoramic image mosaicing. Technical Report MSR-TR-97-23, Microsoft Research.

  • Shum, H.-Y. and Szeliski, R. 1998. Construction and refinement of panoramic mosaics with global and local alignment. In Sixth International Conference on Computer Vision (ICCV'98), Bombay, pp. 953–958.

  • Stein, G. 1995. Accurate internal camera calibration using rotation, with analysis of sources of error. In Fifth International Conference on Computer Vision (ICCV'95), Cambridge, Massachusetts, pp. 230–236.

  • Szeliski, R. 1994. Image mosaicing for tele-reality applications. In IEEE Workshop on Applications of Computer Vision (WACV'94), Sarasota, Florida, IEEE Computer Society, pp. 44–53.

    Google Scholar 

  • Szeliski, R. and Kang, S.B. 1995. Direct methods for visual scene reconstruction. In IEEE Workshop on Representations of Visual Scenes, Cambridge, Massachusetts, pp. 26–33.

  • Szeliski, R. 1996. Video mosaics for virtual environments. IEEE Computer Graphics and Applications, 16(2):22–30.

    Google Scholar 

  • Szeliski, R. and Shum, H.-Y. 1997. Creating full view panoramic image mosaics and texture-mapped models. In Computer Graphics (SIGGRAPH'97), pp. 251–258.

  • Tian, Q. and Huhns, M.N. 1986. Algorithms for subpixel registration. Computer Vision, Graphics, and Image Processing, 35:220–233.

    Google Scholar 

  • Tsai, R.Y. 1987. A versatile camera calibration technique for highaccuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE Journal of Robotics and Automation, RA-3(4):323–344.

    Google Scholar 

  • Weghorst, H., Hooper, G., and Greenberg, D.P. 1984. Improved computational methods for ray tracing. ACM Transactions on Graphics, 3(1):52069.

    Google Scholar 

  • Westover, L. 1990. Footprint evaluation for volume rendering. In Computer Graphics (SIGGRAPH' 90 Proceedings), Baskett, F. (Eds.), pp. 367–376.

  • Williams, L. 1983. Pyramidal parametrics. Computer Graphics, 17(3):1–11.

    Google Scholar 

  • Wolberg, G. 1990. Digital Image Warping. IEEE Computer Society Press: Los Alamitos, California.

    Google Scholar 

  • Wolf, P.R. 1974. Elements of Photogrammetry. McGraw-Hill: New York.

    Google Scholar 

  • Wood, D.N., Finkelstein, A., Hughes, J.F., Thayer, C.E., and Salesin, D.H. 1997. Multiperspective panoramas for cel animation. In Computer Graphics Proceedings, Annual Conference Series, Proc. SIGGRAPH'97, Los Angeles, ACM SIGGRAPH, pp. 243–250.

  • Xiong, Y. and Turkowski, K. 1997. Creating image-based VR using a self-calibrating fisheye lens. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'97), San Juan, Puerto Rico, pp. 237–243.

Download references

Author information

Authors and Affiliations

  1. Microsoft Research, One Microsoft Way, Redmond, WA, 98052, USA

    Heung-Yeung Shum & Richard Szeliski

Authors
  1. Heung-Yeung Shum
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Richard Szeliski
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

An erratum to this article can be found at http://dx.doi.org/10.1023/A:1016051024520

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Shum, HY., Szeliski, R. Systems and Experiment Paper: Construction of Panoramic Image Mosaics with Global and Local Alignment. International Journal of Computer Vision 36, 101–130 (2000). https://doi.org/10.1023/A:1008195814169

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1008195814169

  • image mosaics
  • virtual environment modeling
  • panoramas
  • parametric motion estimation
  • global alignment
  • local alignment