Asynchronous Structure from Motion at Scale

  • Rawia Mhiri
  • Safa Ouerghi
  • Rémi BoutteauEmail author
  • Pascal Vasseur
  • Stéphane Mousset
  • Abdelaziz Bensrhair


Vision systems that provide a 360-degree view are becoming increasingly common in today’s vehicles. These systems are generally composed of several cameras pointing in different directions and rigidly connected to each other. The purpose of these systems is to provide driver assistance in the form of a display, for example by building a Bird’s eye view around the vehicle for parking assistance. In this context, and for reasons of cost and ease of integration, such cameras are generally not synchronized. If non-synchronization is not a problem when it comes to display only, it poses significant issues for more complex computer vision applications (3D reconstruction, motion estimation, etc.). In this article, we propose to use a network of asynchronous cameras to estimate the motion of the vehicle and to find the 3D structure of the scene around it (for example for obstacle detection). Our method relies on the use of at least three images from two adjacent cameras. The poses of the cameras are independently estimated by conventional visual odometry algorithms. Then we show that it is possible to find the absolute scale factor by hypothesizing that the motion of the vehicle is smooth. The results are then refined through a local bundle adjustment on the scale factor and 3D points only. We evaluated our method under real conditions on the KITTI database, and we showed that our method can be generalized to a larger network of cameras thanks to a system developed in our lab.


Structure from motion Asynchronous cameras Scale estimation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



  1. 1.
    Appia, V., Hariyani, H., Sivasankaran, S., Liu, S., Chitnis, K., Mueller, M., Batur, U., Agarwa, G.: Surround View Camera System for Adas on Ti’s Tdax Socs. Tech. rep., Texas Instrument (2015)Google Scholar
  2. 2.
    Baker, S., Nayar, S.: A theory of single-viewpoint catadioptric image formation. Int. J. Comput. Vis. 35(2), 175–196 (1999)CrossRefGoogle Scholar
  3. 3.
    Calonder, M., Lepetit, V., Strecha, C., Fua, P.: Brief: binary robust independent elementary features. In: European Conference on Computer Vision (ECCV), vol. 6314, pp. 778–792. Heraklion, Crete, Greece (2010)Google Scholar
  4. 4.
    Caspi, Y., Irani, M.: A step towards sequence-to-sequence alignment. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662), vol. 2, pp. 682–689 (2000)Google Scholar
  5. 5.
    Clipp, B., Kim, J.H., Frahm, J.M., Pollefeys, M., Hartley, R.: Robust 6Dof motion estimation for non-overlapping, multi-camera systems. In: 2008 IEEE Workshop on Applications of Computer Vision, pp. 1–8 (2008)Google Scholar
  6. 6.
    Comport, A., Malis, E., Rives, P.: Real-time quadrifocal visual odometry. Int. J. Robot. Res. 29(2-3), 245–266 (2010)CrossRefGoogle Scholar
  7. 7.
    Dhome, M., Mennillo, L., Royer, E., Mondot, F., Mousain, J.: Multibody reconstruction of the dynamic scene surrounding a vehicle using a wide baseline and multifocal stereo system. In: Workshop on Planning, Perception and Navigation for Intelligent Vehicles (Satellite Event of IROS’17). Vancouver, Canada. (2017)
  8. 8.
    Dooley, D., McGinley, B., Hughes, C., Kilmartin, L., Jones, E., Glavin, M.: A blind-zone detection method using a rear-mounted fisheye camera with combination of vehicle detection methods. IEEE Trans. Intell. Transp. Syst. 17(1), 264–278 (2016)CrossRefGoogle Scholar
  9. 9.
    Engels, C., Stewénius, H., Nistér, D.: Bundle adjustment rules. In: Photogrammetric Computer Vision (2006)Google Scholar
  10. 10.
    Fraundorfer, F., Scaramuzza, D., Pollefeys, M.: A constricted bundle adjustment parameterization for relative scale estimation in visual odometry. In: IEEE International Conference on Robotics and Automation (2010)Google Scholar
  11. 11.
    Furgale, P.T., Schwesinger, U., Rufli, M., Derendarz, W., Grimmett, H., Mühlfellner, P., Wonneberger, S., Timpner, J., Rottmann, S., Li, B., Schmidt, B., Nguyen, T., Cardarelli, E., Cattani, S., Bruning, S., Horstmann, S., Stellmacher, M., Mielenz, H., Köser, K., Beermann, M., Hane, C., Heng, L., Lee, G.H., Fraundorfer, F., Iser, R., Triebel, R., Posner, I., Newman, P., Wolf, L.C., Pollefeys, M., Brosig, S., Effertz, J., Pradalier, C., Siegwart, R.: Toward automated driving in cities using close-to-market sensors: an overview of the V-charge project. In: 2013 IEEE Intelligent Vehicles Symposium (IV), Gold Coast City, Australia, June 23-26, 2013, pp. 809–816 (2013)Google Scholar
  12. 12.
    Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the KITTI dataset. Int. J. Robot. Res. 32(11), 1231–1237 (2013)CrossRefGoogle Scholar
  13. 13.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision Benchmark suite. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)Google Scholar
  14. 14.
    Geiger, A., Ziegler, J., Stiller, C.: Stereoscan: dense 3D reconstruction in real-time. In: Intelligent Vehicles Symposium (IV) (2011)Google Scholar
  15. 15.
    Häne, C., Heng, L., Lee, G.H., Fraundorfer, F., Furgale, P., Sattler, T., Pollefeys, M.: 3d visual perception for self-driving cars using a multi-camera system: calibration, mapping, localization, and obstacle detection. Image Vision Comput. 68, 14–27 (2017)CrossRefGoogle Scholar
  16. 16.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004)Google Scholar
  17. 17.
    Heng, L., Li, B., Pollefeys, M.: Camodocal: automatic intrinsic and extrinsic calibration of a rig with multiple generic cameras and odometry. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1793–1800 (2013)Google Scholar
  18. 18.
    Kim, J.H., Hartley, R., Frahm, J.M., Pollefeys, M.: Visual odometry for non-overlapping views using second-order cone programming. In: Yagi, Y., Kang, S.B., Kweon, I.S., Zha, H. (eds.) Computer Vision – ACCV 2007, pp 353–362. Springer, Berlin (2007)Google Scholar
  19. 19.
    Lébraly, P., Royer, E., Ait-aider, O., Deymier, C., Dhome, M.: Fast calibration of embedded non-overlapping cameras. In: IEEE International Conference on Robotics and Automation, ICRA 2011, Shanghai, China, 9-13 May 2011, pp. 221–227 (2011)Google Scholar
  20. 20.
    Lepetit, V., Moreno-Noguer, F., Fua, P.: Epnp: an accurate o(n) solution to the pnp problem. Int. J. Comput. Vis. 81(2), 155–166 (2009)CrossRefGoogle Scholar
  21. 21.
    Levenberg, K.: A method for the solution of certain problems in least squares. Q. Appl. Math. 2, 164–168 (1944)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Lüke, S., Fochler, O., Schaller, T., Regensburger, U.: Traffic jam assistance and automation. In Handbook of Driver Assistance Systems, pp. 1287–1302 (2016)Google Scholar
  23. 23.
    Mathias, M., Timofte, R., Benenson, R., Van Gool, L.: Traffic sign recognition—How far are we from the solution?. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–8. Dallas, TX, USA (2013)Google Scholar
  24. 24.
    McCall, J., Trivedi, M.: Video-based lane estimation and tracking for driver assistance: survey, system, and evaluation. IEEE Trans. Intell. Transp. Syst. 7(1), 20–37 (2006)CrossRefGoogle Scholar
  25. 25.
    Mouragnon, E., Lhuillier, M., Dhome, M., Dekeyser, F., Sayd, P.: Real time localization and 3D reconstruction. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 363–370. IEEE (2006)Google Scholar
  26. 26.
    Mouragnon, E., Lhuillier, M., Dhome, M., Dekeyser, F., Sayd, P.: Generic and real-time structure from motion using local bundle adjustment. Image Vision Comput. 27(8), 1178–1193 (2009)CrossRefGoogle Scholar
  27. 27.
    Nguyen, T., Lhuillier, M.: Self-calibration of omnidirectional multi-cameras including synchronization and rolling shutter. Comput. Vis. Image Underst. 162, 166–184 (2017)CrossRefGoogle Scholar
  28. 28.
    Nister, D.: An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Anal. Mach. Intell. 26, 756–770 (2004)CrossRefGoogle Scholar
  29. 29.
    Pless, R.: Using many cameras as one. In: 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2003), 16-22 June 2003, Madison, WI, USA, pp. 587–593 (2003)Google Scholar
  30. 30.
    Pollefeys, M., Nikolic, J., Kneip, L., Kazik, T., Siegwart, R.: Real-time 6D stereo visual odometry with non-overlapping fields of view. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 00, pp. 1529–1536 (2012)Google Scholar
  31. 31.
    Pooley, D.W., Brooks, M.J., van den Hengel, A.J., Chojnacki, W.: A voting scheme for estimating the synchrony of moving-camera videos. In: Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429), vol. 1, pp. I–413–16 (2003)Google Scholar
  32. 32.
    Ramirez, A., Ohn-bar, E., Trivedi, M.M.: Panoramic stitching for driver assistance and applications to motion saliency-based risk analysis. In: 16th International IEEE Conference on Intelligent Transportation Systems, ITSC 2013, the Hague, the Netherlands, October 6-9, 2013, pp. 597–601 (2013)Google Scholar
  33. 33.
    Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: European Conference on Computer Vision (ECCV), pp. 430-443. Graz, Austria (2006)Google Scholar
  34. 34.
    Scaramuzza, D., Fraundorfer, F., Siegwart, R.: Real-time monocular visual odometry for on-road vehicles with 1-Point RANSAC. In: 2009 IEEE International Conference on Robotics and Automation, ICRA 2009, Kobe, Japan, May 12-17, 2009, pp. 4293–4299 (2009)Google Scholar
  35. 35.
    Scaramuzza, D., Fraundorfer, F., Siegwart, R.: Real-time monocular visual odometry for on-road vehicles with 1-point ransac. In: IEEE International Conference on Robotics and Automation, 2009. ICRA’09. pp. 4293–4299. IEEE (2009)Google Scholar
  36. 36.
    Schönbein, M., Kitt, B., Lauer, M.: Environmental perception for intelligent vehicles using catadioptric stereo vision systems. In: ECMR, pp. 189–194 (2011)Google Scholar
  37. 37.
    Sons, M., Lauer, M., Keller, C., Stiller, C.: Mapping and localization using surround view. In: IEEE Intelligent Vehicles Symposium (IV), pp. 1158–1163. Los Angeles, USA (2017)Google Scholar
  38. 38.
    Svedman, M.: 3-D structure from stereo vision using unsynchronized cameras. In: Masters Thesis, Royal Institute of Technology (KTH (2005)Google Scholar
  39. 39.
    Timofte, R., Zimmermann, K., Van Gool, L.: Multi-view traffic sign detection, recognition, and 3d localisation. Mach. Vis. Appl. 25(3), 633–647 (2014)CrossRefGoogle Scholar
  40. 40.
    Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment a modern synthesis. In: Vision Algorithms: Theory and Practice, pp. 298–372. Springer (2000)Google Scholar
  41. 41.
    Wang, Y., Kneip, L.: On scale initialization in non-overlapping multi-perspective visual odometry. In: Computer Vision Systems - 11th International Conference, ICVS 2017, Shenzhen, China, July 10-13, 2017, Revised Selected Papers, pp. 144–157 (2017)Google Scholar
  42. 42.
    Wolf, L., Zomet, A.: Sequence-to-sequence self calibration. In: Proceedings 7th European Conference on Computer Vision Computer Vision - ECCV 2002, Copenhagen, Denmark, May 28-31, 2002, Part II, pp. 370–382 (2002)Google Scholar
  43. 43.
    Zhang, B., Appia, V., Pekkucuksen, I., Liu, Y., Umit Batur, A., Shastry, P., Liu, S., Sivasankaran, S., Chitnis, K.: A surround view camera solution for embedded systems. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 662-667. Columbus, USA (2014)Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Normandie Univ, UNIROUEN, UNIHAVREINSA Rouen, LITISRouenFrance
  2. 2.Normandie Univ, UNIROUENESIGELEC, IRSEEMRouenFrance

Personalised recommendations