Advertisement

Image Stitching and Rectification for Hand-Held Cameras

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12352)

Abstract

In this paper, we derive a new differential homography that can account for the scanline-varying camera poses in Rolling Shutter (RS) cameras, and demonstrate its application to carry out RS-aware image stitching and rectification at one stroke. Despite the high complexity of RS geometry, we focus in this paper on a special yet common input—two consecutive frames from a video stream, wherein the inter-frame motion is restricted from being arbitrarily large. This allows us to adopt simpler differential motion model, leading to a straightforward and practical minimal solver. To deal with non-planar scene and camera parallax in stitching, we further propose an RS-aware spatially-varying homogarphy field in the principle of As-Projective-As-Possible (APAP). We show superior performance over state-of-the-art methods both in RS image stitching and rectification, especially for images captured by hand-held shaking cameras.

Keywords

Rolling Shutter Image rectification Image stitching Differential homography Homography field Hand-held cameras 

Notes

Acknowledgement

We would like to thank Buyu Liu, Gaurav Sharma, Samuel Schulter, and Manmohan Chandraker for proofreading and support of this work. We are also grateful to all the reviewers for their constructive suggestions.

Supplementary material

Supplementary material 1 (mp4 25969 KB)

Supplementary material 2 (mp4 33927 KB)

504444_1_En_15_MOESM3_ESM.pdf (16.2 mb)
Supplementary material 3 (pdf 16598 KB)

References

  1. 1.
    Albl, C., Kukelova, Z., Larsson, V., Polic, M., Pajdla, T., Schindler, K.: From two rolling shutters to one global shutter. In: CVPR (2020)Google Scholar
  2. 2.
    Albl, C., Kukelova, Z., Pajdla, T.: R6p-rolling shutter absolute camera pose. In: CVPR (2015)Google Scholar
  3. 3.
    Albl, C., Kukelova, Z., Pajdla, T.: Rolling shutter absolute pose problem with known vertical direction. In: CVPR (2016)Google Scholar
  4. 4.
    Albl, C., Sugimoto, A., Pajdla, T.: Degeneracies in rolling shutter SfM. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 36–51. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46454-1_3CrossRefGoogle Scholar
  5. 5.
    Bapat, A., Price, T., Frahm, J.M.: Rolling shutter and radial distortion are features for high frame rate multi-camera tracking. In: CVPR (2018)Google Scholar
  6. 6.
    Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)CrossRefGoogle Scholar
  7. 7.
    Chang, C.H., Sato, Y., Chuang, Y.Y.: Shape-preserving half-projective warps for image stitching. In: CVPR (2014)Google Scholar
  8. 8.
    Chen, Y.S., Chuang, Y.Y.: Natural image stitching with the global similarity prior. In: ECCV (2016)Google Scholar
  9. 9.
    Cox, D.A., Little, J., O’shea, D.: Using Algebraic Geometry, vol. 185. Springer, Heidelberg (2006).  https://doi.org/10.1007/b138611CrossRefzbMATHGoogle Scholar
  10. 10.
    Dai, Y., Li, H., Kneip, L.: Rolling shutter camera relative pose: generalized epipolar geometry. In: CVPR (2016)Google Scholar
  11. 11.
    Grundmann, M., Kwatra, V., Castro, D., Essa, I.: Calibration-free rolling shutter removal. In: ICCP (2012)Google Scholar
  12. 12.
    Haresh, S., Kumar, S., Zia, M.Z., Tran, Q.H.: Towards anomaly detection in dashcam videos. In: IV (2020)Google Scholar
  13. 13.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)zbMATHGoogle Scholar
  14. 14.
    Hartley, R.I.: In defense of the eight-point algorithm. IEEE Trans. Pattern Anal. Mach. Intell. 19(6), 580–593 (1997)CrossRefGoogle Scholar
  15. 15.
    Hedborg, J., Forssén, P.E., Felsberg, M., Ringaby, E.: Rolling shutter bundle adjustment. In: CVPR (2012)Google Scholar
  16. 16.
    Heeger, D.J., Jepson, A.D.: Subspace methods for recovering rigid motion I: algorithm and implementation. Int. J. Comput. Vis. 7(2), 95–117 (1992)CrossRefGoogle Scholar
  17. 17.
    Herrmann, C., et al.: Robust image stitching with multiple registrations. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 53–69. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01216-8_4CrossRefGoogle Scholar
  18. 18.
    Herrmann, C., Wang, C., Bowen, R.S., Keyder, E., Zabih, R.: Object-centered image stitching. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 846–861. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01219-9_50CrossRefGoogle Scholar
  19. 19.
    Horn, B.K.: Motion fields are hardly ever ambiguous. Int. J. Comput. Vis. 1(3), 259–274 (1988)CrossRefGoogle Scholar
  20. 20.
    Im, S., Ha, H., Choe, G., Jeon, H.G., Joo, K., So Kweon, I.: High quality structure from small motion for rolling shutter cameras. In: ICCV (2015)Google Scholar
  21. 21.
    Ito, E., Okatani, T.: Self-calibration-based approach to critical motion sequences of rolling-shutter structure from motion. In: CVPR (2017)Google Scholar
  22. 22.
    Klingner, B., Martin, D., Roseborough, J.: Street view motion-from-structure-from-motion. In: ICCV (2013)Google Scholar
  23. 23.
    Kukelova, Z., Albl, C., Sugimoto, A., Pajdla, T.: Linear solution to the minimal absolute pose rolling shutter problem. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11363, pp. 265–280. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-20893-6_17CrossRefGoogle Scholar
  24. 24.
    Lao, Y., Aider, O.A.: Rolling shutter homography and its applications. In: IEEE Trans. Pattern Anal. Mach. Intell. (2020)Google Scholar
  25. 25.
    Lao, Y., Ait-Aider, O.: A robust method for strong rolling shutter effects correction using lines with automatic feature selection. In: CVPR (2018)Google Scholar
  26. 26.
    Lao, Y., Ait-Aider, O., Bartoli, A.: Rolling shutter pose and ego-motion estimation using shape-from-template. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 477–492. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01216-8_29CrossRefGoogle Scholar
  27. 27.
    Lee, K.Y., Sim, J.Y.: Warping residual based image stitching for large parallax. In: CVPR (2020)Google Scholar
  28. 28.
    Li, S., Yuan, L., Sun, J., Quan, L.: Dual-feature warping-based motion model estimation. In: ICCV (2015)Google Scholar
  29. 29.
    Liao, T., Li, N.: Single-perspective warps in natural image stitching. IEEE Trans. Image Process. 29, 724–735 (2019)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Lin, C.C., Pankanti, S.U., Natesan Ramamurthy, K., Aravkin, A.Y.: Adaptive as-natural-as-possible image stitching. In: CVPR (2015)Google Scholar
  31. 31.
    Lin, K., Jiang, N., Cheong, L.-F., Do, M., Lu, J.: SEAGULL: seam-guided local alignment for parallax-tolerant image stitching. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 370–385. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46487-9_23CrossRefGoogle Scholar
  32. 32.
    Lin, K., Jiang, N., Liu, S., Cheong, L.F., Do, M., Lu, J.: Direct photometric alignment by mesh deformation. In: CVPR (2017)Google Scholar
  33. 33.
    Lin, W.Y., Liu, S., Matsushita, Y., Ng, T.T., Cheong, L.F.: Smoothly varying affine stitching. In: CVPR (2011)Google Scholar
  34. 34.
    Liu, F., Gleicher, M., Jin, H., Agarwala, A.: Content-preserving warps for 3D video stabilization. ACM Trans. Graph. (TOG) 28(3), 1–9 (2009)Google Scholar
  35. 35.
    Liu, P., Cui, Z., Larsson, V., Pollefeys, M.: Deep shutter unrolling network. In: CVPR (2020)Google Scholar
  36. 36.
    Liu, S., Yuan, L., Tan, P., Sun, J.: Bundled camera paths for video stabilization. ACM Trans. Graph. (TOG) 32(4), 1–10 (2013)Google Scholar
  37. 37.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  38. 38.
    Ma, Y., Košecká, J., Sastry, S.: Linear differential algorithm for motion recovery: a geometric approach. Int. J. Comput. Vis. 36(1), 71–89 (2000)CrossRefGoogle Scholar
  39. 39.
    Ma, Y., Soatto, S., Kosecka, J., Sastry, S.S.: An Invitation to 3-D Vision: From Images to Geometric Models, vol. 26. Springer, Heidelberg (2012)zbMATHGoogle Scholar
  40. 40.
    Magerand, L., Bartoli, A., Ait-Aider, O., Pizarro, D.: Global optimization of object pose and motion from a single rolling shutter image with automatic 2D-3D matching. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7572, pp. 456–469. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33718-5_33CrossRefGoogle Scholar
  41. 41.
    Meingast, M., Geyer, C., Sastry, S.: Geometric models of rolling-shutter cameras. In: Workshop on Omnidirectional Vision, Camera Networks and Non-Classical Cameras (2005)Google Scholar
  42. 42.
    Mohan, M.M., Rajagopalan, A., Seetharaman, G.: Going unconstrained with rolling shutter deblurring. In: ICCV (2017)Google Scholar
  43. 43.
    Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: ORB-SLAM: a versatile and accurate monocular slam system. IEEE Trans. Rob. 31(5), 1147–1163 (2015)CrossRefGoogle Scholar
  44. 44.
    Muratov, O., Slynko, Y., Chernov, V., Lyubimtseva, M., Shamsuarov, A., Bucha, V.: 3DCapture: 3D reconstruction for a smartphone. In: CVPRW (2016)Google Scholar
  45. 45.
    Oth, L., Furgale, P., Kneip, L., Siegwart, R.: Rolling shutter camera calibration. In: CVPR (2013)Google Scholar
  46. 46.
    Punnappurath, A., Rengarajan, V., Rajagopalan, A.: Rolling shutter super-resolution. In: ICCV (2015)Google Scholar
  47. 47.
    Purkait, P., Zach, C.: Minimal solvers for monocular rolling shutter compensation under ackermann motion. In: WACV (2018)Google Scholar
  48. 48.
    Purkait, P., Zach, C., Leonardis, A.: Rolling shutter correction in Manhattan world. In: ICCV (2017)Google Scholar
  49. 49.
    Rengarajan, V., Balaji, Y., Rajagopalan, A.: Unrolling the shutter: CNN to correct motion distortions. In: CVPR (2017)Google Scholar
  50. 50.
    Rengarajan, V., Rajagopalan, A.N., Aravind, R.: From bows to arrows: rolling shutter rectification of urban scenes. In: CVPR (2016)Google Scholar
  51. 51.
    Rengarajan, V., Rajagopalan, A.N., Aravind, R., Seetharaman, G.: Image registration and change detection under rolling shutter motion blur. IEEE Trans. Pattern Anal. Mach. Intell. 39(10), 1959–1972 (2016)CrossRefGoogle Scholar
  52. 52.
    Ringaby, E., Forssén, P.E.: Efficient video rectification and stabilisation for cell-phones. Int. J. Comput. Vis. 96(3), 335–352 (2012)CrossRefGoogle Scholar
  53. 53.
    Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: ICCV (2011)Google Scholar
  54. 54.
    Saurer, O., Koser, K., Bouguet, J.Y., Pollefeys, M.: Rolling shutter stereo. In: ICCV (2013)Google Scholar
  55. 55.
    Saurer, O., Pollefeys, M., Hee Lee, G.: Sparse to dense 3D reconstruction from rolling shutter images. In: CVPR (2016)Google Scholar
  56. 56.
    Saurer, O., Pollefeys, M., Lee, G.H.: A minimal solution to the rolling shutter pose estimation problem. In: IROS (2015)Google Scholar
  57. 57.
    Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: CVPR (2016)Google Scholar
  58. 58.
    Schubert, D., Demmel, N., Usenko, V., Stuckler, J., Cremers, D.: Direct sparse odometry with rolling shutter. In: ECCV (2018)Google Scholar
  59. 59.
    Szeliski, R., et al.: Image alignment and stitching: a tutorial. Found. Trends® Comput. Graph. Vis. 2(1), 1–104 (2007)CrossRefGoogle Scholar
  60. 60.
    Tran, Q.-H., Chin, T.-J., Carneiro, G., Brown, M.S., Suter, D.: In defence of RANSAC for outlier rejection in deformable registration. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7575, pp. 274–287. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33765-9_20CrossRefGoogle Scholar
  61. 61.
    Vasu, S., Mohan, M.M., Rajagopalan, A.: Occlusion-aware rolling shutter rectification of 3D scenes. In: CVPR (2018)Google Scholar
  62. 62.
    Vasu, S., Rajagopalan, A.N., Seetharaman, G.: Camera shutter-independent registration and rectification. IEEE Trans. Image Process. 27(4), 1901–1913 (2017)MathSciNetCrossRefGoogle Scholar
  63. 63.
    Zaragoza, J., Chin, T.J., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. In: CVPR (2013)Google Scholar
  64. 64.
    Zaragoza, J., Chin, T.J., Tran, Q.H., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1285–1298 (2014)CrossRefGoogle Scholar
  65. 65.
    Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: CVPR (2014)Google Scholar
  66. 66.
    Zhuang, B., Cheong, L.F., Hee Lee, G.: Rolling-shutter-aware differential SFM and image rectification. In: ICCV (2017)Google Scholar
  67. 67.
    Zhuang, B., Cheong, L.F., Hee Lee, G.: Baseline desensitizing in translation averaging. In: CVPR (2018)Google Scholar
  68. 68.
    Zhuang, B., Tran, Q.H., Ji, P., Cheong, L.F., Chandraker, M.: Learning structure-and-motion-aware rolling shutter correction. In: CVPR (2019)Google Scholar
  69. 69.
    Zhuang, B., Tran, Q.H., Lee, G.H., Cheong, L.F., Chandraker, M.: Degeneracy in self-calibration revisited and a deep learning solution for uncalibrated SLAM. In: IROS (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.NEC Labs AmericaPrincetonUSA

Personalised recommendations