Skip to main content
SpringerLink
Log in

Panoramic video stitching from commodity HDTV cameras

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Digital camera and smartphone technologies have made high-quality images and video pervasive and abundant. Combining or stitching collections of images from a variety of viewpoints into an extended panoramic image is a common and popular function for such devices. Extending this functionality to video however, poses many new challenges due to the demand for both spatial and temporal continuity. Multi-view video stitching (also called panoramic video stitching) is an emerging, common research area in computer vision, image/video processing and computer graphics and has wide applications in virtual reality, virtual tourism, surveillance, and human computer interaction. In this paper, we study and solve the major technical and practical problems in the complete process of stitching a high-resolution multi-view video into a high-resolution panoramic video. The challenges addressed include video stabilization, efficient high-definition multi-view video alignment and stitching, color correction, and blurred frame detection and repair. The proposed approaches have been successfully applied in a high-quality virtual reality system—the Virtual Exercise Environment (VEE) system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. An alternative approach is to compute a 2D homography from each of the selected frames using its own SIFT feature correspondences, and then average the estimated homography matrices for all of the frames. This method is not effective in practice.

References

  1. Alhwarin, F., Ristić-Durrant, D., Gräer, A.: Pca-sift: a more distinctive representation for local image descriptors. In: Proceedings of DAGM 2010, LNCS 6376, pp. 222–231 (2010)

  2. Anguelov, D., Dulong, C., Filip, D., Frueh, C., Lafon, S., Lyon, R., Ogale, A., Vincent, L., Weaver, J.: Google street view: capturing the world at street level. Computer 43, 32–38 (2010)

    Article  Google Scholar 

  3. Brown, M.: Autostitch. http://www.cs.bath.ac.uk/brown/autostitch/autostitch.html

  4. Brown, M., Lowe, D.: Recognising panoramas. In: Proceedings of International Conference on Computer Vision (ICCV’03), vol. 2, pp. 1218–1225 (2003)

  5. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)

    Article  Google Scholar 

  6. Caspi, Y., Irani, M.: Spatio-temporal alignment of sequences. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1409–1424 (2002)

    Google Scholar 

  7. Chern, N.K., Neow, N.P.A., Ang, M.H. Jr.: Practical issues in pixel-based autofocusing for machine vision. In: Proceedings of 2001 IEEE International Conference on Robotics and Automation, pp. 2791–2796 (2001)

  8. Cho, T.S., Zitnick, L., Joshi, N., Kang, S.B., Szeliski, R., Freeman, W.T.: Image restoration by matching gradient distributions. IEEE Trans. Pattern Anal. Mach. Intell. 34(4), 683–694 (2012)

    Article  Google Scholar 

  9. Choi, S., Kim, T., Yu, W.: Robust video stabilization to outlier motion using adaptive ransac. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1897–1902 (2009)

  10. de Villiers, J., le Roux, F.: Omnidirectional maritime surveillance. In: Proceedings of CSIR 3rd Biennial Conference (2010)

  11. de Villiers, J.. Real-time photogrammetric stitching of high resolution video on COTS hardware. In: Proceedings of 2009 International Symposium on Optomechatronic Technologies (2009)

  12. Fisher, Y. (eds): Fractal Image Compression: Theory and Application. Springer, New York (1995)

    Google Scholar 

  13. El-Saban, M., Izz, M., Kaheel, A.: Fast stitching of videos captured from freely moving devices by exploiting temporal redundancy. In: Proceedings of 17th IEEE International Conference on Image Processing (2010)

  14. El-Saban, M., Izz, M., Kaheel, A., Refaat, M.: Improved optimal seam selection blending for fast video stitching of videos captured from freely moving devices. In: Proceedings of 18th IEEE International Conference on Image Processing (2011)

  15. El-Saban, M., Refaat, M., Kaheel, A., Abdul-Hamid, A.: Stitching videos streamed by mobile phones in real-time. In: Proceedings of 2009 ACM Multimedia Conference, pp. 1009–1010 (2009)

  16. Erasmus, S., Smith, K.: An automatic focusing and astigmatism correction system for the sem and ctem. J. Microsc. 127, 185–199 (1982)

    Article  Google Scholar 

  17. Firestone, L., Cook, K., Talsania, N., Preston, K.: Comparison of autofocus methods for automated microscopy. Cytometry 12, 195–206 (1991)

    Article  Google Scholar 

  18. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–C395 (1981)

    Article  MathSciNet  Google Scholar 

  19. Google. Google street view. http://www.google.com/streetview.

  20. Goshtasby, A.A.: 2-D and 3-D Image Registration. Wiley, Hoboken (2005)

    Google Scholar 

  21. Grey Point: Ladybug2 camera. http://www.ptgrey.com/products/ladybug2/ladybug2_360_video_camera.asp

  22. Gupta, A., Joshi, N., Zitnick, L., Cohen M., Curless, B.: Single image deblurring using motion density functions. In: Proceedings of 10th European Conference on Computer Vision (ECCV’10) (2010)

  23. Jansson, P.: Deconvolution of Image and Spectra, 2nd edn. Academic Press, London(1997)

    Google Scholar 

  24. Kaheel, A., El-saban, M., Refaat, M., Ezz, M.: Mobicast: a system for collaborative event casting using mobile phones. In: Proceedings of 8th International Conference on Mobile and Ubiquitous Multimedia (2009)

  25. Ke, Y., Sukthankar, R.: Pca-sift: a more distinctive representation for local image descriptors. In: Proceedings of 2004 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’04), vol. 2, pp. 506–513 (2004)

  26. Levin, A.: Blind motion deblurring using image statistics. In: 2006 Advances in Neural Information Processing Systems, pp. 841–848 (2006)

  27. Levin, A., Zomet, A., Peleg, S., Weiss, Y.: Seamless image stitching in the gradient domain. In: ECCV’04, pp. 377–389 (2004)

  28. Li, K., Zhou, S.: A fast sift feature matching algorithm for image registration. In: Proceedings of 2011 International Conference on Multimedia and Signal Processing (CMSP’11), pp. 89–93 (2011)

  29. Litvin, A., Konrad, J., Karl, W.: Probabilistic video stabilization using kalman filtering and mosaicking. In: Proceedings of IS&T/SPIE Symposium Electronic Imaging, Image, and Video Communications, pp. 663–674 (2003)

  30. Liu, L., Peng, F., Tian, Y., Xu, Y., Zhao, K.: Fast image matching for localization in deep-sea based on the simplified sift (scale invariant feature transform) algorithm. In: Proceedings of 2nd International Conference on Space Information Technology, p. 67953A (2007)

  31. Liu, L., Wang, Y., Wang, Y.: Sift based automatic tie-point extraction for multitemporal sar images. In: Proceedings of 2008 International Workshop on Education Technology and Training & 2008 International Workshop on Geoscience and Remote Sensing (2008)

  32. Liu, R., Li, Z., Jia, J.: Image partial blur detection and classification. In: Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)

  33. Lowe, D.: Object recognition from local scale-invariant features. In: Proceedings of International Conference on Computer Vision (ICCV’99), vol. 2, pp. 1150–1157 (1999)

  34. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  35. Matsushita, Y., Ofek, E., Ge, W., Tang, X., Shum, H.-Y.: Full-frame video stabilization with motion inpainting. IEEE Trans. Pattern Anal. Mach. Intell. 7, 1150–1163 (2006)

    Article  Google Scholar 

  36. Microsoft. Photosynth. photosynth.net

  37. Nokia. Nokia panorama. http://store.ovi.com/content/112209

  38. Ong, E.P., Lin, W.S., Lu, Z.K., Yao, S.S., Yang, X.K., Jiang, L.F.: No-reference quality metric for measuring image blur. In: Proceedings of 2003 International Conference on Image Processing, pp. 469–472 (2003)

  39. Pilu, M.: Video stabilization as a variational problem and numerical solution with the viterbi method. In: Procedings of 2004 IEEE Conference on Computer Vision and Pattern Recognition, pp. 625–630 (2004)

  40. Raskar, R., Agrawal, A., Tumblin, J.: Coded exposure photography: motion deblurring using fluttered shutter. ACM Trans. Graph. 25(3), 795–804 (2006)

    Article  Google Scholar 

  41. Reinhard, E., Adhikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. IEEE Comput. Graph. Appl. 21(5), 34–41 (2001)

    Article  Google Scholar 

  42. Shan, Q., Jia, J., Agarwala, A.: High-quality motion deblurring from a single image. ACM Trans. Graph. 27(3), art no. 73 (2008)

    Google Scholar 

  43. Simon, D.: Optimal State Estimation: Kalman, H Infinity, and Nonlinear Approaches. Wiley, London (2006)

    Book  Google Scholar 

  44. Singhai, J., Rawat, P.: Review of motion estimation and video stabilization techniques for hand held mobile video. Signal Image Process. Int. J. 2(2), 159–168 (2011)

  45. Sahil S., Peter S., Peter R., Johannes, U.: Combining mutual information and scale invariant feature transform for fast and robust multisensor sar image registration. In: Proceedings of 75th ASRPS Conference (2009)

  46. Szeliski, R.: Image mosaicing for tele-reality applications. In: Proceedings of IEEE Workshop on Applications of Computer Vision, pp. 44–53 (1994)

  47. Szeliski, R.: Image Alignment and Stitching: A tutorial. Technical Report MSR-TR-2004-92, Microsoft Research, One Microsoft Way, Redmond, WA (2004)

  48. Szeliski, R.: Video mosaics for virtual environments. Comput. Graph. Appl. 16(2), 22–30 (1996)

    Article  Google Scholar 

  49. Tuo, H., Jing, Z., Zhang, T.: Aerial sequence image mosaic using reduced sift descriptors. In: Proceedings of MIPPR 2007, SPIE 6786 (2007)

  50. Wang, Z., Bovik A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  51. Wikipedia. Mean square weighted deviation. http://en.wikipedia.org/wiki/Mean_square_weighted_deviation

  52. Xu, L., Jia, J.: Two-phase kernel estimation for robust motion deblurring. In: Proceedings of 11th European Conference on Computer Vision (ECCV’10), vol. 1, pp. 157–170 (2010)

  53. Xu, W., Jeong, J., Mulligan, J.: Augmenting exercise systems with virtual exercise environment. In: Advances in Visual Computing, LNCS 5875, pp. 490–499 (2009)

  54. Xu, W., Mulligan, J.: Performance evaluation of color correction approaches for automatic multi-view image and video stitching. In: Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’10), pp. 263–270 (2010)

  55. Xu, W., Mulligan, J., Xu, D.: Detecting and classifying blurred image regions. In: Proceedings of 2013 IEEE International Conference on Multimedia and Expo (ICME’13) (2013) (to appear)

  56. Xu, Y., Qin, S.: A new approach to video stabilization with iterative smoothing. In: Proceedings of 10th IEEE International Conference on Signal Processing pp. 1224–1227 (2010)

  57. Zheng, Y.J., Yu, J.Y., Kang, S.B., Lin, S., Kambhamettu, C.: Sigle-image vignetting correction using radial gradient symmetry. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’08), vol. 2, pp. 1–8 (2008)

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, University of Science and Technology of China, Hefei, China

    Wei Xu

  2. Flashback Technologies Inc., Boulder, CO, USA

    Jane Mulligan

Authors
  1. Wei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jane Mulligan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Xu.

Additional information

Communicated by P. Pala.

Electronic supplementary material

Below is the link to the electronic supplementary material.

MPG (1457 KB)

MPG (1453 KB)

MPG (1371 KB)

MPG (2846 KB)

PDF (5918 KB)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Xu, W., Mulligan, J. Panoramic video stitching from commodity HDTV cameras. Multimedia Systems 19, 407–426 (2013). https://doi.org/10.1007/s00530-013-0316-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-013-0316-2

Keywords

Search

Navigation