International Journal of Computer Vision

, Volume 96, Issue 3, pp 335–352

Efficient Video Rectification and Stabilisation for Cell-Phones

Article

Abstract

This article presents a method for rectifying and stabilising video from cell-phones with rolling shutter (RS) cameras. Due to size constraints, cell-phone cameras have constant, or near constant focal length, making them an ideal application for calibrated projective geometry. In contrast to previous RS rectification attempts that model distortions in the image plane, we model the 3D rotation of the camera. We parameterise the camera rotation as a continuous curve, with knots distributed across a short frame interval. Curve parameters are found using non-linear least squares over inter-frame correspondences from a KLT tracker. By smoothing a sequence of reference rotations from the estimated curve, we can at a small extra cost, obtain a high-quality image stabilisation. Using synthetic RS sequences with associated ground-truth, we demonstrate that our rectification improves over two other methods. We also compare our video stabilisation with the methods in iMovie and Deshaker.

Keywords

Cell-phone Rolling shutter CMOS Video stabilisation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Supplementary material

(MPG 2.76 MB)

(MPG 2.76 MB)

(MPG 2.72 MB)

(MPG 2.77 MB)

(MPG 2.71 MB)

(MPG 2.79 MB)

(MPG 2.76 MB)

(MPG 3.27 MB)

(MPG 3.25 MB)

(MPG 3.24 MB)

(MPG 3.26 MB)

(MPG 3.92 MB)

(MPG 3.50 MB)

(MPG 3.49 MB)

(MPG 3.49 MB)

(MPG 4.18 MB)

(MPG 3.27 MB)

References

  1. Ait-Aider, O., & Berry, F. (2009). Structure and kinematics triangulation with a rolling shutter stereo rig. In IEEE international conference on computer vision. Google Scholar
  2. Ait-Aider, O., Bartoli, A., & Andreff, N. (2007). Kinematics from lines in a single rolling shutter image. In CVPR’07, Minneapolis, USA. Google Scholar
  3. Apple Inc. (2010). iMovie’09 video stabilizer. http://www.apple.com/ilife/imovie/.
  4. Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2007). A database and evaluation methodology for optical flow. In IEEE international conference on computer vision (ICCV07), Rio de Janeiro, Brazil. Google Scholar
  5. Baker, S., Bennett, E., Kang, S. B., & Szeliski, R. (2010). Removing rolling shutter wobble. In IEEE conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society. Google Scholar
  6. Bernstein, J. (2003). An overview of MEMS inertial sensing technology. Sensors Magazine, 2003(1). Google Scholar
  7. Buehler, C., Bosse, M., & McMillan, L. (2001). Non-metric image-based rendering for video stabilization. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR’01) (pp. 609–614). Google Scholar
  8. Chang, L. W., Liang, C. K., & Chen, H. (2005). Analysis and compensation of rolling shutter distortion for CMOS image sensor arrays. In International symposium on communications (ISCOM05). Google Scholar
  9. Cho, W. H., & Kong, K. S. (2007). Affine motion based CMOS distortion analysis and CMOS digital image stabilization. IEEE Transactions on Consumer Electronics, 53(3), 833–841. CrossRefGoogle Scholar
  10. Cho, W. H., Kim, D. W., & Hong, K. S. (2007). CMOS digital image stabilization. IEEE Transactions on Consumer Electronics, 53(3), 979–986. CrossRefGoogle Scholar
  11. Chun, J. B., Jung, H., & Kyung, C. M. (2008). Suppressing rolling-shutter distortion of CMOS image sensors by motion vector detection. IEEE Transactions on Consumer Electronics, 54(4), 1479–1487. CrossRefGoogle Scholar
  12. Forssén, P. E., & Ringaby, E. (2010). Rectifying rolling shutter video from hand-held devices. In IEEE conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society. Google Scholar
  13. Gamal, A. E., & Eltoukhy, H. (2005). CMOS image sensors. IEEE Circuits and Devices Magazine. Google Scholar
  14. Geyer, C., Meingast, M., & Sastry, S. (2005). Geometric models of rolling-shutter cameras. In 6th OmniVis WS. Google Scholar
  15. Gramkow, C. (2001). On averaging rotations. International Journal of Computer Vision, 42(1/2), 7–16. MathSciNetMATHCrossRefGoogle Scholar
  16. Green, R., Mahler, H., & Siau, J. (1983). Television picture stabilizing system. US Patent 4,403,256. Google Scholar
  17. Harris, C. G., & Stephens, M. (1988). A combined corner and edge detector. In 4th Alvey vision conference (pp. 147–151). Google Scholar
  18. Hartley, R., & Zisserman, A. (2000). Multiple view geometry in computer vision. Cambridge: Cambridge University Press. MATHGoogle Scholar
  19. Liang, C. K., Chang, L. W., & Chen, H. (2008). Analysis and compensation of rolling shutter effect. IEEE Transactions on Image Processing, 17(8), 1323–1330. MathSciNetCrossRefGoogle Scholar
  20. Liu, F., Gleicher, M., Jin, H., & Agarwala, A. (2009). Content-preserving warps for 3D video stabilization. ACM Transactions on Graphics. Google Scholar
  21. Liu, F., Gleicher, M., Wang, J., Jin, H., & Agarwala, A. (2011a). Subspace video stabilization. ACM Transactions on Graphics, 30(1). Google Scholar
  22. Liu, F., Gleicher, M., Wang, J., Jin, H., & Agarwala, A. (2011b). Subspace video stabilization, supplemental video set 1. http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/.
  23. Lucas, B., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In IJCAI81 (pp. 674–679). Google Scholar
  24. Morimoto, C., & Chellappa, R. (1997). Fast 3D stabilization and mosaic construction. In Proc. IEEE conference on computer vision and pattern recognition (CVPR97) (pp. 660–665). Puerto Rico, San Juan. CrossRefGoogle Scholar
  25. Nicklin, S. P., Fisher, R. D., & Middleton, R. H. (2007). Rolling shutter image compensation. In LNAI: Vol. 4434. Robocup 2006 (pp. 402–409). CrossRefGoogle Scholar
  26. Park, F., & Ravani, B. (1997). Smooth invariant interpolation of rotations. ACM Transactions on Graphics, 16(3), 277–295. CrossRefGoogle Scholar
  27. Ringaby, E. (2010). Rolling shutter dataset with ground truth. http://www.cvl.isy.liu.se/research/rs-dataset.
  28. Shi, J., & Tomasi, C. (1994). Good features to track. In IEEE conference on computer vision and pattern recognition, CVPR’94, Seattle. Google Scholar
  29. Shoemake, K. (1985). Animating rotation with quaternion curves. In Int. conf. on CGIT (pp. 245–254). Google Scholar
  30. Thalin, G. (2010). Deshaker. video stabilizer plugin, v2, 5 for VirtualDub. http://www.guthspot.se/video/deshaker.htm.
  31. Whyte, O., Sivic, J., Zisserman, A., & Ponce, J. (2010). Non-uniform deblurring for shaken images. In IEEE conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society. Google Scholar
  32. Yao, Y. S., Burlina, P., Chellappa, R., & Wu, T. H. (1995). Electronic image stabilization using multiple visual cues. In International conference on image processing (ICIP’95) (pp. 191–194). Washington DC, USA. Google Scholar
  33. Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330–1334. CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.Department of Electrical EngineeringLinköping UniversityLinköpingSweden

Personalised recommendations