Advertisement

Efficient Video Rectification and Stabilisation for Cell-Phones

Abstract

This article presents a method for rectifying and stabilising video from cell-phones with rolling shutter (RS) cameras. Due to size constraints, cell-phone cameras have constant, or near constant focal length, making them an ideal application for calibrated projective geometry. In contrast to previous RS rectification attempts that model distortions in the image plane, we model the 3D rotation of the camera. We parameterise the camera rotation as a continuous curve, with knots distributed across a short frame interval. Curve parameters are found using non-linear least squares over inter-frame correspondences from a KLT tracker. By smoothing a sequence of reference rotations from the estimated curve, we can at a small extra cost, obtain a high-quality image stabilisation. Using synthetic RS sequences with associated ground-truth, we demonstrate that our rectification improves over two other methods. We also compare our video stabilisation with the methods in iMovie and Deshaker.

This is a preview of subscription content, log in to check access.

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

References

  1. Ait-Aider, O., & Berry, F. (2009). Structure and kinematics triangulation with a rolling shutter stereo rig. In IEEE international conference on computer vision.

  2. Ait-Aider, O., Bartoli, A., & Andreff, N. (2007). Kinematics from lines in a single rolling shutter image. In CVPR’07, Minneapolis, USA.

  3. Apple Inc. (2010). iMovie’09 video stabilizer. http://www.apple.com/ilife/imovie/.

  4. Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2007). A database and evaluation methodology for optical flow. In IEEE international conference on computer vision (ICCV07), Rio de Janeiro, Brazil.

  5. Baker, S., Bennett, E., Kang, S. B., & Szeliski, R. (2010). Removing rolling shutter wobble. In IEEE conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society.

  6. Bernstein, J. (2003). An overview of MEMS inertial sensing technology. Sensors Magazine, 2003(1).

  7. Buehler, C., Bosse, M., & McMillan, L. (2001). Non-metric image-based rendering for video stabilization. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR’01) (pp. 609–614).

  8. Chang, L. W., Liang, C. K., & Chen, H. (2005). Analysis and compensation of rolling shutter distortion for CMOS image sensor arrays. In International symposium on communications (ISCOM05).

  9. Cho, W. H., & Kong, K. S. (2007). Affine motion based CMOS distortion analysis and CMOS digital image stabilization. IEEE Transactions on Consumer Electronics, 53(3), 833–841.

  10. Cho, W. H., Kim, D. W., & Hong, K. S. (2007). CMOS digital image stabilization. IEEE Transactions on Consumer Electronics, 53(3), 979–986.

  11. Chun, J. B., Jung, H., & Kyung, C. M. (2008). Suppressing rolling-shutter distortion of CMOS image sensors by motion vector detection. IEEE Transactions on Consumer Electronics, 54(4), 1479–1487.

  12. Forssén, P. E., & Ringaby, E. (2010). Rectifying rolling shutter video from hand-held devices. In IEEE conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society.

  13. Gamal, A. E., & Eltoukhy, H. (2005). CMOS image sensors. IEEE Circuits and Devices Magazine.

  14. Geyer, C., Meingast, M., & Sastry, S. (2005). Geometric models of rolling-shutter cameras. In 6th OmniVis WS.

  15. Gramkow, C. (2001). On averaging rotations. International Journal of Computer Vision, 42(1/2), 7–16.

  16. Green, R., Mahler, H., & Siau, J. (1983). Television picture stabilizing system. US Patent 4,403,256.

  17. Harris, C. G., & Stephens, M. (1988). A combined corner and edge detector. In 4th Alvey vision conference (pp. 147–151).

  18. Hartley, R., & Zisserman, A. (2000). Multiple view geometry in computer vision. Cambridge: Cambridge University Press.

  19. Liang, C. K., Chang, L. W., & Chen, H. (2008). Analysis and compensation of rolling shutter effect. IEEE Transactions on Image Processing, 17(8), 1323–1330.

  20. Liu, F., Gleicher, M., Jin, H., & Agarwala, A. (2009). Content-preserving warps for 3D video stabilization. ACM Transactions on Graphics.

  21. Liu, F., Gleicher, M., Wang, J., Jin, H., & Agarwala, A. (2011a). Subspace video stabilization. ACM Transactions on Graphics, 30(1).

  22. Liu, F., Gleicher, M., Wang, J., Jin, H., & Agarwala, A. (2011b). Subspace video stabilization, supplemental video set 1. http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/.

  23. Lucas, B., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In IJCAI81 (pp. 674–679).

  24. Morimoto, C., & Chellappa, R. (1997). Fast 3D stabilization and mosaic construction. In Proc. IEEE conference on computer vision and pattern recognition (CVPR97) (pp. 660–665). Puerto Rico, San Juan.

  25. Nicklin, S. P., Fisher, R. D., & Middleton, R. H. (2007). Rolling shutter image compensation. In LNAI: Vol. 4434. Robocup 2006 (pp. 402–409).

  26. Park, F., & Ravani, B. (1997). Smooth invariant interpolation of rotations. ACM Transactions on Graphics, 16(3), 277–295.

  27. Ringaby, E. (2010). Rolling shutter dataset with ground truth. http://www.cvl.isy.liu.se/research/rs-dataset.

  28. Shi, J., & Tomasi, C. (1994). Good features to track. In IEEE conference on computer vision and pattern recognition, CVPR’94, Seattle.

  29. Shoemake, K. (1985). Animating rotation with quaternion curves. In Int. conf. on CGIT (pp. 245–254).

  30. Thalin, G. (2010). Deshaker. video stabilizer plugin, v2, 5 for VirtualDub. http://www.guthspot.se/video/deshaker.htm.

  31. Whyte, O., Sivic, J., Zisserman, A., & Ponce, J. (2010). Non-uniform deblurring for shaken images. In IEEE conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society.

  32. Yao, Y. S., Burlina, P., Chellappa, R., & Wu, T. H. (1995). Electronic image stabilization using multiple visual cues. In International conference on image processing (ICIP’95) (pp. 191–194). Washington DC, USA.

  33. Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330–1334.

Download references

Author information

Correspondence to Erik Ringaby.

Electronic Supplementary Material

Below are the links to the electronic supplementary material.

(MPG 2.76 MB)

(MPG 2.76 MB)

(MPG 2.72 MB)

(MPG 2.77 MB)

(MPG 2.71 MB)

(MPG 2.79 MB)

(MPG 2.76 MB)

(MPG 3.27 MB)

(MPG 3.25 MB)

(MPG 3.24 MB)

(MPG 3.26 MB)

(MPG 3.92 MB)

(MPG 3.50 MB)

(MPG 3.49 MB)

(MPG 3.49 MB)

(MPG 4.18 MB)

(MPG 3.27 MB)

(MPG 2.76 MB)

(MPG 2.76 MB)

(MPG 2.72 MB)

(MPG 2.77 MB)

(MPG 2.71 MB)

(MPG 2.79 MB)

(MPG 2.76 MB)

(MPG 3.27 MB)

(MPG 3.25 MB)

(MPG 3.24 MB)

(MPG 3.26 MB)

(MPG 3.92 MB)

(MPG 3.50 MB)

(MPG 3.49 MB)

(MPG 3.49 MB)

(MPG 4.18 MB)

(MPG 3.27 MB)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Ringaby, E., Forssén, P. Efficient Video Rectification and Stabilisation for Cell-Phones. Int J Comput Vis 96, 335–352 (2012). https://doi.org/10.1007/s11263-011-0465-8

Download citation

Keywords

  • Cell-phone
  • Rolling shutter
  • CMOS
  • Video stabilisation