Advertisement

Rolling Shutter Pose and Ego-Motion Estimation Using Shape-from-Template

  • Yizhen LaoEmail author
  • Omar Ait-Aider
  • Adrien Bartoli
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11206)

Abstract

We propose a new method for the absolute camera pose problem (PnP) which handles Rolling Shutter (RS) effects. Unlike all existing methods which perform 3D-2D registration after augmenting the Global Shutter (GS) projection model with the velocity parameters under various kinematic models, we propose to use local differential constraints. These are established by drawing an analogy with Shape-from-Template (SfT). The main idea consists in considering that RS distortions due to camera ego-motion during image acquisition can be interpreted as virtual deformations of a template captured by a GS camera. Once the virtual deformations have been recovered using SfT, the camera pose and ego-motion are computed by registering the deformed scene on the original template. This 3D-3D registration involves a 3D cost function based on the Euclidean point distance, more physically meaningful than the re-projection error or the algebraic distance based cost functions used in previous work. Results on both synthetic and real data show that the proposed method outperforms existing RS pose estimation techniques in terms of accuracy and stability of performance in various configurations.

Keywords

Rolling Shutter Pose estimation Shape-from-Template 

Notes

Acknowledgement

This work has been sponsored by the French government research program “Investissements d’Avenir” through the IDEX-ISITE initiative 16-IDEX-0001 (CAP 20-25), the IMobS3 Laboratory of Excellence (ANR-10-LABX-16-01) and the RobotEx Equipment of Excellence (ANR-10-EQPX-44). This research was also financed by the European Union through the Regional Competitiveness and Employment program -2014-2020- (ERDF - AURA region) and by the AURA region. This research has received funding from the EUs FP7 through the ERC research grant 307483 FLEXABLE.

References

  1. 1.
    Frank, E., Hall, M.: A simple approach to ordinal classification. In: De Raedt, L., Flach, P. (eds.) ECML 2001. LNCS (LNAI), vol. 2167, pp. 145–156. Springer, Heidelberg (2001).  https://doi.org/10.1007/3-540-44795-4_13CrossRefGoogle Scholar
  2. 2.
    Ait-Aider, O., Bartoli, A., Andreff, N.: Kinematics from lines in a single rolling shutter image. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–6. IEEE (2007)Google Scholar
  3. 3.
    Ait-Aider, O., Berry, F.: Structure and kinematics triangulation with a rolling shutter stereo rig. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1835–1840 (2009)Google Scholar
  4. 4.
    Saurer, O., Pollefeys, M., Hee Lee, G.: Sparse to dense 3D reconstruction from rolling shutter images. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016Google Scholar
  5. 5.
    Saurer, O., Koser, K., Bouguet, J.Y., Pollefeys, M.: Rolling shutter stereo. In: 2013 IEEE International Conference on Computer Vision (ICCV) (2013)Google Scholar
  6. 6.
    Hedborg, J., Ringaby, E., Forssén, P.E., Felsberg, M.: Structure and motion estimation from rolling shutter video. In: Computer Vision Workshops (ICCV Workshops) (2011)Google Scholar
  7. 7.
    Hedborg, J., Forssen, P.E., Felsberg, M., Ringaby, E.: Rolling shutter bundle adjustment. In: Computer Vision and Pattern Recognition (CVPR) (2012)Google Scholar
  8. 8.
    Dai, Y., Li, H., Kneip, L.: Rolling shutter camera relative pose: generalized epipolar geometry. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4132–4140 (2016)Google Scholar
  9. 9.
    Kim, J.H., Cadena, C., Reid, I.: Direct semi-dense slam for rolling shutter cameras. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 1308–1315, May 2016Google Scholar
  10. 10.
    Albl, C., Sugimoto, A., Pajdla, T.: Degeneracies in rolling shutter SfM. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 36–51. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46454-1_3CrossRefGoogle Scholar
  11. 11.
    Ito, E., Okatani, T.: Self-calibration-based approach to critical motion sequences of rolling-shutter structure from motion. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  12. 12.
    Haralick, R.M., Lee, D., Ottenburg, K., Nolle, M.: Analysis and solutions of the three point perspective pose estimation problem. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1991. Proceedings CVPR 1991, pp. 592–598. IEEE (1991)Google Scholar
  13. 13.
    Gao, X.S., Hou, X.R., Tang, J., Cheng, H.F.: Complete solution classification for the perspective-three-point problem. IEEE Trans. Pattern Anal. Mach. Intell. 25(8), 930–943 (2003)CrossRefGoogle Scholar
  14. 14.
    Wu, Y., Hu, Z.: PnP problem revisited. J. Math. Imaging Vis. 24(1), 131–141 (2006)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Quan, L., Lan, Z.: Linear n-point camera pose determination. IEEE Trans. Pattern Anal. Mach. Intell. 21(8), 774–780 (1999)CrossRefGoogle Scholar
  16. 16.
    Leng, D., Sun, W.: 2009 Finding all the solutions of PnP problem. In: IEEE International Workshop on Imaging Systems and Techniques, IST 2009, pp. 348–352. IEEE (2009)Google Scholar
  17. 17.
    Albl, C., Kukelova, Z., Pajdla, T.: R6P-rolling shutter absolute camera pose. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2292–2300 (2015)Google Scholar
  18. 18.
    Saurer, O., Pollefeys, M., Lee, G.H.: A minimal solution to the rolling shutter pose estimation problem. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1328–1334. IEEE (2015)Google Scholar
  19. 19.
    Albl, C., Kukelova, Z., Pajdla, T.: Rolling shutter absolute pose problem with known vertical direction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3355–3363 (2016)Google Scholar
  20. 20.
    Bartoli, A., Gérard, Y., Chadebecq, F., Collins, T., Pizarro, D.: Shape-from-template. IEEE Trans. Pattern Anal. Mach. Intell. 37(10), 2099–2118 (2015)CrossRefGoogle Scholar
  21. 21.
    Duchamp, G., Ait-Aider, O., Royer, E., Lavest, J.M.: A rolling shutter compliant method for localisation and reconstruction. In: VISAPP (3), pp. 277–284 (2015)Google Scholar
  22. 22.
    Magerand, L., Bartoli, A., Ait-Aider, O., Pizarro, D.: Global optimization of object pose and motion from a single rolling shutter image with automatic 2D-3D matching. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7572, pp. 456–469. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33718-5_33CrossRefGoogle Scholar
  23. 23.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)zbMATHGoogle Scholar
  24. 24.
    Magerand, L., Bartoli, A.: A generic rolling shutter camera model and its application to dynamic pose estimation. In: International Symposium on 3D Data Processing, Visualization and Transmission (2010)Google Scholar
  25. 25.
    Salzmann, M., Fua, P.: Linear local models for monocular reconstruction of deformable surfaces. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 931–944 (2011)CrossRefGoogle Scholar
  26. 26.
    Brunet, F., Bartoli, A., Hartley, R.I.: Monocular template-based 3D surface reconstruction: convex inextensible and nonconvex isometric methods. Comput. Vis. Image Underst. 125, 138–154 (2014)CrossRefGoogle Scholar
  27. 27.
    Collins, T., Bartoli, A.: [POSTER] Realtime shape-from-template: system and applications. In: 2015 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 116–119. IEEE (2015)Google Scholar
  28. 28.
    Chhatkuli, A., Pizarro, D., Bartoli, A., Collins, T.: A stable analytical framework for isometric shape-from-template by surface integration. IEEE Trans. Pattern Anal. Mach. Intell. 39(5), 833–850 (2017)CrossRefGoogle Scholar
  29. 29.
    Malti, A., Bartoli, A., Hartley, R.: A linear least-squares solution to elastic shape-from-template. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1629–1637 (2015)Google Scholar
  30. 30.
    Malti, A., Herzet, C.: Elastic shape-from-template with spatially sparse deforming forces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3337–3345 (2017)Google Scholar
  31. 31.
    Haouchine, N., Dequidt, J., Berger, M.O., Cotin, S.: Single view augmentation of 3D elastic objects. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 229–236. IEEE (2014)Google Scholar
  32. 32.
    Rengarajan, V., Balaji, Y., Rajagopalan, A.: Unrolling the shutter: CNN to correct motion distortions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2291–2299 (2017)Google Scholar
  33. 33.
    Horn, B.K., Hilden, H.M., Negahdaripour, S.: Closed-form solution of absolute orientation using orthonormal matrices. JOSA A 5(7), 1127–1135 (1988)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Institut PascalUniversité Clermont Auvergne/CNRSClermont-FerrandFrance

Personalised recommendations