Skip to main content
SpringerLink
Log in

Efficient Video Rectification and Stabilisation for Cell-Phones

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

This article presents a method for rectifying and stabilising video from cell-phones with rolling shutter (RS) cameras. Due to size constraints, cell-phone cameras have constant, or near constant focal length, making them an ideal application for calibrated projective geometry. In contrast to previous RS rectification attempts that model distortions in the image plane, we model the 3D rotation of the camera. We parameterise the camera rotation as a continuous curve, with knots distributed across a short frame interval. Curve parameters are found using non-linear least squares over inter-frame correspondences from a KLT tracker. By smoothing a sequence of reference rotations from the estimated curve, we can at a small extra cost, obtain a high-quality image stabilisation. Using synthetic RS sequences with associated ground-truth, we demonstrate that our rectification improves over two other methods. We also compare our video stabilisation with the methods in iMovie and Deshaker.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ait-Aider, O., & Berry, F. (2009). Structure and kinematics triangulation with a rolling shutter stereo rig. In IEEE international conference on computer vision.

    Google Scholar 

  • Ait-Aider, O., Bartoli, A., & Andreff, N. (2007). Kinematics from lines in a single rolling shutter image. In CVPR’07, Minneapolis, USA.

    Google Scholar 

  • Apple Inc. (2010). iMovie’09 video stabilizer. http://www.apple.com/ilife/imovie/.

  • Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2007). A database and evaluation methodology for optical flow. In IEEE international conference on computer vision (ICCV07), Rio de Janeiro, Brazil.

    Google Scholar 

  • Baker, S., Bennett, E., Kang, S. B., & Szeliski, R. (2010). Removing rolling shutter wobble. In IEEE conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society.

    Google Scholar 

  • Bernstein, J. (2003). An overview of MEMS inertial sensing technology. Sensors Magazine, 2003(1).

  • Buehler, C., Bosse, M., & McMillan, L. (2001). Non-metric image-based rendering for video stabilization. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR’01) (pp. 609–614).

    Google Scholar 

  • Chang, L. W., Liang, C. K., & Chen, H. (2005). Analysis and compensation of rolling shutter distortion for CMOS image sensor arrays. In International symposium on communications (ISCOM05).

    Google Scholar 

  • Cho, W. H., & Kong, K. S. (2007). Affine motion based CMOS distortion analysis and CMOS digital image stabilization. IEEE Transactions on Consumer Electronics, 53(3), 833–841.

    Article  Google Scholar 

  • Cho, W. H., Kim, D. W., & Hong, K. S. (2007). CMOS digital image stabilization. IEEE Transactions on Consumer Electronics, 53(3), 979–986.

    Article  Google Scholar 

  • Chun, J. B., Jung, H., & Kyung, C. M. (2008). Suppressing rolling-shutter distortion of CMOS image sensors by motion vector detection. IEEE Transactions on Consumer Electronics, 54(4), 1479–1487.

    Article  Google Scholar 

  • Forssén, P. E., & Ringaby, E. (2010). Rectifying rolling shutter video from hand-held devices. In IEEE conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society.

    Google Scholar 

  • Gamal, A. E., & Eltoukhy, H. (2005). CMOS image sensors. IEEE Circuits and Devices Magazine.

  • Geyer, C., Meingast, M., & Sastry, S. (2005). Geometric models of rolling-shutter cameras. In 6th OmniVis WS.

    Google Scholar 

  • Gramkow, C. (2001). On averaging rotations. International Journal of Computer Vision, 42(1/2), 7–16.

    Article  MathSciNet  MATH  Google Scholar 

  • Green, R., Mahler, H., & Siau, J. (1983). Television picture stabilizing system. US Patent 4,403,256.

  • Harris, C. G., & Stephens, M. (1988). A combined corner and edge detector. In 4th Alvey vision conference (pp. 147–151).

    Google Scholar 

  • Hartley, R., & Zisserman, A. (2000). Multiple view geometry in computer vision. Cambridge: Cambridge University Press.

    MATH  Google Scholar 

  • Liang, C. K., Chang, L. W., & Chen, H. (2008). Analysis and compensation of rolling shutter effect. IEEE Transactions on Image Processing, 17(8), 1323–1330.

    Article  MathSciNet  Google Scholar 

  • Liu, F., Gleicher, M., Jin, H., & Agarwala, A. (2009). Content-preserving warps for 3D video stabilization. ACM Transactions on Graphics.

  • Liu, F., Gleicher, M., Wang, J., Jin, H., & Agarwala, A. (2011a). Subspace video stabilization. ACM Transactions on Graphics, 30(1).

  • Liu, F., Gleicher, M., Wang, J., Jin, H., & Agarwala, A. (2011b). Subspace video stabilization, supplemental video set 1. http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/.

  • Lucas, B., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In IJCAI81 (pp. 674–679).

    Google Scholar 

  • Morimoto, C., & Chellappa, R. (1997). Fast 3D stabilization and mosaic construction. In Proc. IEEE conference on computer vision and pattern recognition (CVPR97) (pp. 660–665). Puerto Rico, San Juan.

    Chapter  Google Scholar 

  • Nicklin, S. P., Fisher, R. D., & Middleton, R. H. (2007). Rolling shutter image compensation. In LNAI: Vol. 4434. Robocup 2006 (pp. 402–409).

    Chapter  Google Scholar 

  • Park, F., & Ravani, B. (1997). Smooth invariant interpolation of rotations. ACM Transactions on Graphics, 16(3), 277–295.

    Article  Google Scholar 

  • Ringaby, E. (2010). Rolling shutter dataset with ground truth. http://www.cvl.isy.liu.se/research/rs-dataset.

  • Shi, J., & Tomasi, C. (1994). Good features to track. In IEEE conference on computer vision and pattern recognition, CVPR’94, Seattle.

    Google Scholar 

  • Shoemake, K. (1985). Animating rotation with quaternion curves. In Int. conf. on CGIT (pp. 245–254).

    Google Scholar 

  • Thalin, G. (2010). Deshaker. video stabilizer plugin, v2, 5 for VirtualDub. http://www.guthspot.se/video/deshaker.htm.

  • Whyte, O., Sivic, J., Zisserman, A., & Ponce, J. (2010). Non-uniform deblurring for shaken images. In IEEE conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society.

    Google Scholar 

  • Yao, Y. S., Burlina, P., Chellappa, R., & Wu, T. H. (1995). Electronic image stabilization using multiple visual cues. In International conference on image processing (ICIP’95) (pp. 191–194). Washington DC, USA.

    Google Scholar 

  • Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330–1334.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Linköping University, 581 83, Linköping, Sweden

    Erik Ringaby & Per-Erik Forssén

Authors
  1. Erik Ringaby
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Per-Erik Forssén
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Erik Ringaby.

Electronic Supplementary Material

Below are the links to the electronic supplementary material.

(MPG 2.76 MB)

(MPG 2.76 MB)

(MPG 2.72 MB)

(MPG 2.77 MB)

(MPG 2.71 MB)

(MPG 2.79 MB)

(MPG 2.76 MB)

(MPG 3.27 MB)

(MPG 3.25 MB)

(MPG 3.24 MB)

(MPG 3.26 MB)

(MPG 3.92 MB)

(MPG 3.50 MB)

(MPG 3.49 MB)

(MPG 3.49 MB)

(MPG 4.18 MB)

(MPG 3.27 MB)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ringaby, E., Forssén, PE. Efficient Video Rectification and Stabilisation for Cell-Phones. Int J Comput Vis 96, 335–352 (2012). https://doi.org/10.1007/s11263-011-0465-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-011-0465-8

Keywords

Search

Navigation