Skip to main content
SpringerLink
Log in

Asynchronous Structure from Motion at Scale

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Vision systems that provide a 360-degree view are becoming increasingly common in today’s vehicles. These systems are generally composed of several cameras pointing in different directions and rigidly connected to each other. The purpose of these systems is to provide driver assistance in the form of a display, for example by building a Bird’s eye view around the vehicle for parking assistance. In this context, and for reasons of cost and ease of integration, such cameras are generally not synchronized. If non-synchronization is not a problem when it comes to display only, it poses significant issues for more complex computer vision applications (3D reconstruction, motion estimation, etc.). In this article, we propose to use a network of asynchronous cameras to estimate the motion of the vehicle and to find the 3D structure of the scene around it (for example for obstacle detection). Our method relies on the use of at least three images from two adjacent cameras. The poses of the cameras are independently estimated by conventional visual odometry algorithms. Then we show that it is possible to find the absolute scale factor by hypothesizing that the motion of the vehicle is smooth. The results are then refined through a local bundle adjustment on the scale factor and 3D points only. We evaluated our method under real conditions on the KITTI database, and we showed that our method can be generalized to a larger network of cameras thanks to a system developed in our lab.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Appia, V., Hariyani, H., Sivasankaran, S., Liu, S., Chitnis, K., Mueller, M., Batur, U., Agarwa, G.: Surround View Camera System for Adas on Ti’s Tdax Socs. Tech. rep., Texas Instrument (2015)

  2. Baker, S., Nayar, S.: A theory of single-viewpoint catadioptric image formation. Int. J. Comput. Vis. 35(2), 175–196 (1999)

    Article  Google Scholar 

  3. Calonder, M., Lepetit, V., Strecha, C., Fua, P.: Brief: binary robust independent elementary features. In: European Conference on Computer Vision (ECCV), vol. 6314, pp. 778–792. Heraklion, Crete, Greece (2010)

  4. Caspi, Y., Irani, M.: A step towards sequence-to-sequence alignment. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662), vol. 2, pp. 682–689 (2000)

  5. Clipp, B., Kim, J.H., Frahm, J.M., Pollefeys, M., Hartley, R.: Robust 6Dof motion estimation for non-overlapping, multi-camera systems. In: 2008 IEEE Workshop on Applications of Computer Vision, pp. 1–8 (2008)

  6. Comport, A., Malis, E., Rives, P.: Real-time quadrifocal visual odometry. Int. J. Robot. Res. 29(2-3), 245–266 (2010)

    Article  Google Scholar 

  7. Dhome, M., Mennillo, L., Royer, E., Mondot, F., Mousain, J.: Multibody reconstruction of the dynamic scene surrounding a vehicle using a wide baseline and multifocal stereo system. In: Workshop on Planning, Perception and Navigation for Intelligent Vehicles (Satellite Event of IROS’17). Vancouver, Canada. https://hal-clermont-univ.archives-ouvertes.fr/hal-01657751 (2017)

  8. Dooley, D., McGinley, B., Hughes, C., Kilmartin, L., Jones, E., Glavin, M.: A blind-zone detection method using a rear-mounted fisheye camera with combination of vehicle detection methods. IEEE Trans. Intell. Transp. Syst. 17(1), 264–278 (2016)

    Article  Google Scholar 

  9. Engels, C., Stewénius, H., Nistér, D.: Bundle adjustment rules. In: Photogrammetric Computer Vision (2006)

  10. Fraundorfer, F., Scaramuzza, D., Pollefeys, M.: A constricted bundle adjustment parameterization for relative scale estimation in visual odometry. In: IEEE International Conference on Robotics and Automation (2010)

  11. Furgale, P.T., Schwesinger, U., Rufli, M., Derendarz, W., Grimmett, H., Mühlfellner, P., Wonneberger, S., Timpner, J., Rottmann, S., Li, B., Schmidt, B., Nguyen, T., Cardarelli, E., Cattani, S., Bruning, S., Horstmann, S., Stellmacher, M., Mielenz, H., Köser, K., Beermann, M., Hane, C., Heng, L., Lee, G.H., Fraundorfer, F., Iser, R., Triebel, R., Posner, I., Newman, P., Wolf, L.C., Pollefeys, M., Brosig, S., Effertz, J., Pradalier, C., Siegwart, R.: Toward automated driving in cities using close-to-market sensors: an overview of the V-charge project. In: 2013 IEEE Intelligent Vehicles Symposium (IV), Gold Coast City, Australia, June 23-26, 2013, pp. 809–816 (2013)

  12. Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the KITTI dataset. Int. J. Robot. Res. 32(11), 1231–1237 (2013)

    Article  Google Scholar 

  13. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision Benchmark suite. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)

  14. Geiger, A., Ziegler, J., Stiller, C.: Stereoscan: dense 3D reconstruction in real-time. In: Intelligent Vehicles Symposium (IV) (2011)

  15. Häne, C., Heng, L., Lee, G.H., Fraundorfer, F., Furgale, P., Sattler, T., Pollefeys, M.: 3d visual perception for self-driving cars using a multi-camera system: calibration, mapping, localization, and obstacle detection. Image Vision Comput. 68, 14–27 (2017)

    Article  Google Scholar 

  16. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004)

  17. Heng, L., Li, B., Pollefeys, M.: Camodocal: automatic intrinsic and extrinsic calibration of a rig with multiple generic cameras and odometry. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1793–1800 (2013)

  18. Kim, J.H., Hartley, R., Frahm, J.M., Pollefeys, M.: Visual odometry for non-overlapping views using second-order cone programming. In: Yagi, Y., Kang, S.B., Kweon, I.S., Zha, H. (eds.) Computer Vision – ACCV 2007, pp 353–362. Springer, Berlin (2007)

  19. Lébraly, P., Royer, E., Ait-aider, O., Deymier, C., Dhome, M.: Fast calibration of embedded non-overlapping cameras. In: IEEE International Conference on Robotics and Automation, ICRA 2011, Shanghai, China, 9-13 May 2011, pp. 221–227 (2011)

  20. Lepetit, V., Moreno-Noguer, F., Fua, P.: Epnp: an accurate o(n) solution to the pnp problem. Int. J. Comput. Vis. 81(2), 155–166 (2009)

    Article  Google Scholar 

  21. Levenberg, K.: A method for the solution of certain problems in least squares. Q. Appl. Math. 2, 164–168 (1944)

    Article  MathSciNet  Google Scholar 

  22. Lüke, S., Fochler, O., Schaller, T., Regensburger, U.: Traffic jam assistance and automation. In Handbook of Driver Assistance Systems, pp. 1287–1302 (2016)

  23. Mathias, M., Timofte, R., Benenson, R., Van Gool, L.: Traffic sign recognition—How far are we from the solution?. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–8. Dallas, TX, USA (2013)

  24. McCall, J., Trivedi, M.: Video-based lane estimation and tracking for driver assistance: survey, system, and evaluation. IEEE Trans. Intell. Transp. Syst. 7(1), 20–37 (2006)

    Article  Google Scholar 

  25. Mouragnon, E., Lhuillier, M., Dhome, M., Dekeyser, F., Sayd, P.: Real time localization and 3D reconstruction. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 363–370. IEEE (2006)

  26. Mouragnon, E., Lhuillier, M., Dhome, M., Dekeyser, F., Sayd, P.: Generic and real-time structure from motion using local bundle adjustment. Image Vision Comput. 27(8), 1178–1193 (2009)

    Article  Google Scholar 

  27. Nguyen, T., Lhuillier, M.: Self-calibration of omnidirectional multi-cameras including synchronization and rolling shutter. Comput. Vis. Image Underst. 162, 166–184 (2017)

    Article  Google Scholar 

  28. Nister, D.: An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Anal. Mach. Intell. 26, 756–770 (2004)

    Article  Google Scholar 

  29. Pless, R.: Using many cameras as one. In: 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2003), 16-22 June 2003, Madison, WI, USA, pp. 587–593 (2003)

  30. Pollefeys, M., Nikolic, J., Kneip, L., Kazik, T., Siegwart, R.: Real-time 6D stereo visual odometry with non-overlapping fields of view. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 00, pp. 1529–1536 (2012)

  31. Pooley, D.W., Brooks, M.J., van den Hengel, A.J., Chojnacki, W.: A voting scheme for estimating the synchrony of moving-camera videos. In: Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429), vol. 1, pp. I–413–16 (2003)

  32. Ramirez, A., Ohn-bar, E., Trivedi, M.M.: Panoramic stitching for driver assistance and applications to motion saliency-based risk analysis. In: 16th International IEEE Conference on Intelligent Transportation Systems, ITSC 2013, the Hague, the Netherlands, October 6-9, 2013, pp. 597–601 (2013)

  33. Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: European Conference on Computer Vision (ECCV), pp. 430-443. Graz, Austria (2006)

  34. Scaramuzza, D., Fraundorfer, F., Siegwart, R.: Real-time monocular visual odometry for on-road vehicles with 1-Point RANSAC. In: 2009 IEEE International Conference on Robotics and Automation, ICRA 2009, Kobe, Japan, May 12-17, 2009, pp. 4293–4299 (2009)

  35. Scaramuzza, D., Fraundorfer, F., Siegwart, R.: Real-time monocular visual odometry for on-road vehicles with 1-point ransac. In: IEEE International Conference on Robotics and Automation, 2009. ICRA’09. pp. 4293–4299. IEEE (2009)

  36. Schönbein, M., Kitt, B., Lauer, M.: Environmental perception for intelligent vehicles using catadioptric stereo vision systems. In: ECMR, pp. 189–194 (2011)

  37. Sons, M., Lauer, M., Keller, C., Stiller, C.: Mapping and localization using surround view. In: IEEE Intelligent Vehicles Symposium (IV), pp. 1158–1163. Los Angeles, USA (2017)

  38. Svedman, M.: 3-D structure from stereo vision using unsynchronized cameras. In: Masters Thesis, Royal Institute of Technology (KTH (2005)

  39. Timofte, R., Zimmermann, K., Van Gool, L.: Multi-view traffic sign detection, recognition, and 3d localisation. Mach. Vis. Appl. 25(3), 633–647 (2014)

    Article  Google Scholar 

  40. Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment a modern synthesis. In: Vision Algorithms: Theory and Practice, pp. 298–372. Springer (2000)

  41. Wang, Y., Kneip, L.: On scale initialization in non-overlapping multi-perspective visual odometry. In: Computer Vision Systems - 11th International Conference, ICVS 2017, Shenzhen, China, July 10-13, 2017, Revised Selected Papers, pp. 144–157 (2017)

  42. Wolf, L., Zomet, A.: Sequence-to-sequence self calibration. In: Proceedings 7th European Conference on Computer Vision Computer Vision - ECCV 2002, Copenhagen, Denmark, May 28-31, 2002, Part II, pp. 370–382 (2002)

  43. Zhang, B., Appia, V., Pekkucuksen, I., Liu, Y., Umit Batur, A., Shastry, P., Liu, S., Sivasankaran, S., Chitnis, K.: A surround view camera solution for embedded systems. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 662-667. Columbus, USA (2014)

Download references

Author information

Authors and Affiliations

  1. Normandie Univ, UNIROUEN, UNIHAVRE, INSA Rouen, LITIS, Rouen, 76000, France

    Rawia Mhiri, Pascal Vasseur, Stéphane Mousset & Abdelaziz Bensrhair

  2. Normandie Univ, UNIROUEN, ESIGELEC, IRSEEM, Rouen, 76000, France

    Safa Ouerghi & Rémi Boutteau

Authors
  1. Rawia Mhiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Safa Ouerghi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rémi Boutteau
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pascal Vasseur
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stéphane Mousset
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Abdelaziz Bensrhair
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rémi Boutteau.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mhiri, R., Ouerghi, S., Boutteau, R. et al. Asynchronous Structure from Motion at Scale. J Intell Robot Syst 96, 159–177 (2019). https://doi.org/10.1007/s10846-018-0974-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-018-0974-6

Keywords

Search

Navigation