Skip to main content
SpringerLink
Log in

Self-calibration and visual SLAM with a multi-camera system on a micro aerial vehicle

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

The use of a multi-camera system enables a robot to obtain a surround view, and thus, maximize its perceptual awareness of its environment. If vision-based simultaneous localization and mapping (vSLAM) is expected to provide reliable pose estimates for a micro aerial vehicle (MAV) with a multi-camera system, an accurate calibration of the multi-camera system is a necessary prerequisite. We propose a novel vSLAM-based self-calibration method for a multi-camera system that includes at least one calibrated stereo camera, and an arbitrary number of monocular cameras. We assume overlapping fields of view to only exist within stereo cameras. Our self-calibration estimates the inter-camera transforms with metric scale; metric scale is inferred from calibrated stereo. On our MAV, we set up each camera pair in a stereo configuration which facilitates the estimation of the MAV’s pose with metric scale. Once the MAV is calibrated, the MAV is able to estimate its global pose via a multi-camera vSLAM implementation based on the generalized camera model. We propose a novel minimal and linear 3-point algorithm that uses relative rotation angle measurements from a 3-axis gyroscope to recover the relative motion of the MAV with metric scale and from 2D-2D feature correspondences. This relative motion estimation does not involve scene point triangulation. Our constant-time vSLAM implementation with loop closures runs on-board the MAV in real-time. To the best of our knowledge, no published work has demonstrated real-time on-board vSLAM with loop closures. We show experimental results from simulation experiments, and real-world experiments in both indoor and outdoor environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://www.ros.org.

  2. https://www.rti.com/products/dds/index.html.

References

  • Agarwal, S., Mierle, K., et al. (2013). Ceres solver. https://code.google.com/p/ceres-solver/.

  • Agrawal, M., Konolige, K., & Blas, M. (2008). Censure: Center surround extremas for realtime feature detection and matching. Computer Vision ECCV 2008 (Vol. 5305, pp. 102–115)., Lecture Notes in Computer Science Berlin, Heidelberg: Springer.

    Chapter  Google Scholar 

  • Brookshire, J., & Teller, S. (2012). Extrinsic calibration from per-sensor egomotion. In Robotics: science and systems (RSS)

  • Carrera, G., Angeli, A., and Davidson, A. (2011a). Lightweight slam and navigation with a multi-camera rig. In European conference on mobile robots (ECMR).

  • Carrera, G., Angeli, A., and Davison, A. (2011b). Slam-based automatic extrinsic calibration of a multi-camera rig. In IEEE international conference on robotics and automation (ICRA).

  • Daniilidis, K. (1999). Hand-eye calibration using dual quaternions. International Journal of Robotics Research (IJRR), 18(3), 286–298.

    Article  MathSciNet  Google Scholar 

  • Furgale, P., Rehder, J., and Siegwart, R. (2013a). Unified temporal and spatial calibration for multi-sensor systems. In IEEE/RSJ international conference on intelligent robots and systems (IROS).

  • Furgale, P., Schwesinger, U., Rufli, M., Derendarz, W., Grimmett, H., Muhlfellner, P., Wonneberger, S., Timpner, J., Rottmann, S., Li, B., Schmidt, B., Nguyen, T., Cardarelli, E., Cattani, S., Bruning, S., Horstmann, S., Stellmacher, M., Mielenz, H., Koser, K., Beermann, M., Hane, C., Heng, L., Lee, G. H., Fraundorfer, F., Iser, R., Triebel, R., Posner, I., Newman, P., Wolf, L., Pollefeys, M., Brosig, S., Effertz, J., Pradalier, C., & Siegwart, R. (2013b). Toward automated driving in cities using close-to-market sensors: An overview of the v-charge project. In IEEE intelligent vehicles symposium (IV).

  • Grossberg, M. and Nayar, S. (2001). A general imaging model and a method for finding its parameters. In IEEE international conference on computer vision (ICCV), volume 2, 2,108–115.

  • Harmat, A., Sharf, I., and Trentini, M. (2012). Parallel tracking and mapping with multiple cameras on an unmanned aerial vehicle. In Intelligent robotics and applications, volume 7506 of Lecture Notes in Computer Science, pp. 421–432.

  • Heng, L., Bürki, M., Lee, G. H., Furgale, P., Siegwart, R., and Pollefeys, M. (2014a). Infrastructure-based calibration of a multi-camera rig. In IEEE international conference on robotics and automation (ICRA).

  • Heng, L., Lee, G. H., & Pollefeys, M. (2014b). Self-calibration and visual slam with a multi-camera system on a micro aerial vehicle. In Robotics science and systems (RSS) .

  • Heng, L., Li, B., and Pollefeys, M. (2013). Camodocal: Automatic intrinsic and extrinsic calibration of a rig with multiple generic cameras and odometry. In IEEE/RSJ international conference on intelligent robots and systems (IROS).

  • Heng, L., Meier, L., Tanskanen, P., Fraundorfer, F., and Pollefeys, M. (2011). Autonomous obstacle avoidance and maneuvering on a vision-guided mav using on-board processing. In IEEE international conference on robotics and automation (ICRA).

  • Kaess, M. and Dellaert, F. (2006). Visual slam with a multi-camera rig. Technical report.

  • Kelly, J., & Sukhatme, G. (2011). Visual-inertial sensor fusion: Localization, mapping and sensor-to-sensor self-calibration. International Journal of Robotics Research (IJRR), 30(1), 56–79.

    Article  Google Scholar 

  • Kim, J., Hwangbo, M., and Kanade, T. (2008). Motion estimation using multiple non-overlapping cameras for small unmanned aerial vehicles. In IEEE international conference on robotics and automation (ICRA).

  • Kneip, L., Scaramuzza, D., and Siegwart, R. (2011). A novel parametrization of the perspective-three-point problem for a direct computation of absolute camera position and orientation. In IEEE conference on computer vision and pattern recognition (CVPR).

  • Lee, G. H., Fraundorfer, F., and Pollefeys, M. (2013a). Minimal solutions for pose estimation of a multi-camera system. In International symposium on robotics research (ISRR).

  • Lee, G. H., Fraundorfer, F., and Pollefeys, M. (2013b). Motion estimation for self-driving cars with a generalized camera. In IEEE conference on computer vision and pattern recognition (CVPR).

  • Lee, G. H., Fraundorfer, F., and Pollefeys, M. (2013c). Robust pose-graph loop-closures with expectation-maximization. In IEEE/RSJ international conference on intelligent robots and systems (IROS).

  • Lee, G. H., Pollefeys, M., and Fraundorfer, F. (2014). Relative pose estimation for a multi-camera system with known vertical direction. In IEEE international conference on computer vision and pattern recognition (CVPR).

  • Li, H., Hartley, R., and Kim, J. (2008). A linear approach to motion estimation using generalized camera models. In IEEE conference on computer vision and pattern recognition (CVPR).

  • Lim, H., Sinha, S., Cohen, M., and Uyttendaele, M. (2012). Real-time image-based 6-dof localization in large-scale environments. In IEEE conference on computer vision and pattern recognition (CVPR), pp. 1043–1050.

  • Lynen, S., Achtelik, M., Weiss, S., Chli, M., and Siegwart, R. (2013). A robust and modular multi-sensor fusion approach applied to mav navigation. In IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 3923–3929.

  • Mei, C. and Rives, P. (2007). Single view point omnidirectional camera calibration from planar grids. In IEEE international conference on robotics and automation (ICRA).

  • Nister, D. (2004). An efficient solution to the five-point relative pose problem. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(6), 756–770.

    Article  Google Scholar 

  • Pless, R. (2003). Using many cameras as one. In IEEE conference on computer vision and pattern recognition (CVPR).

  • Prentice, S., & Roy, N. (2009). The belief roadmap: Efficient planning in belief space by factoring the covariance. International Journal of Robotics Research (IJRR), 28(11–12), 1448–1465.

    Article  Google Scholar 

  • Quan, L., & Lan, Z. D. (1999). Linear n-point camera pose determination. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 21, 774–780.

    Article  Google Scholar 

  • Rublee, E., Rabaud, V., Konolige, K., and Bradski, G. (2011). Orb: An efficient alternative to sift or surf. In IEEE international conference on computer vision (ICCV).

  • Sattler, T., Liebe, B., and Kobbelt, L. (2011). Fast image-based localization using direct 2d-to-3d matching. In IEEE international conference on computer vision (ICCV), pp. 667–674.

  • Schauwecker, K., & Zell, A. (2014). On-board dual-stereo-vision for the navigation of an autonomous mav. Journal of Intelligent and Robotic Systems, 74(1–2), 1–16.

    Article  Google Scholar 

  • Schmid, K., Tomic, T., Ruess, F., Hirschmuller, H., and Suppa, M. (2013). Stereo vision based indoor/outdoor navigation for flying robots. In IEEE/RSJ international conference on intelligent robots and systems (IROS).

  • Schmidt, J., Vogt, F., and Niemann, H. (2005). Calibration-free hand-eye calibration: A structure-from-motion approach. In Pattern Recognition, volume 3663 of Lecture Notes in Computer Science, pp. 67–74.

  • Schweighofer, G., Segvic, S., and Pinz, A. (2008). Online/realtime structure and motion for general camera models. In IEEE workshop on applications of computer vision (WACV), pp. 1–6.

  • Shen, S., Mulgaonkar, Y., Michael, N., & Kumar, V. (2013). Vision-based state estimation and trajectory control towards high-speed flight with a quadrotor. In Robotics: Science and Systems (RSS).

  • Stewénius, H., Nistér, D., Oskarsson, M., and Åström, K. (2005). Solutions to minimal generalized relative pose problems. In OMNIVIS.

  • Strasdat, H., Davison, A., Montiel, J., and Konolige, K. (2011). Double window optimisation for constant time visual slam. In IEEE international conference on computer vision (ICCV).

  • Tribou, M. (2014). Relative Pose Estimation Using Non-overlapping Multicamera Clusters. PhD thesis, University of Waterloo.

  • Weiss, S., Achtelik, M. W., Lynen, S., Achtelik, M. C., Kneip, L., Chli, M., et al. (2013). Monocular vision for long-term micro aerial vehicle state estimation: A compendium. Journal of Field Robotics (JFR), 30(5), 803–831.

    Article  Google Scholar 

Download references

Acknowledgments

The first author is funded by the DSO National Laboratories Postgraduate Scholarship. This work is partially supported by the SNSF V-MAV Grant (DACH framework).

Author information

Authors and Affiliations

  1. Information Division, DSO National Laboratories, 20 Science Park Drive, Singapore, 118230, Singapore

    Lionel Heng

  2. Department of Mechanical Engineering, National University of Singapore, Block EA, #07-08, 9 Engineering Drive 1, Singapore, 117575, Singapore

    Gim Hee Lee

  3. Computer Vision and Geometry Group, ETH Zürich, Universitätstrasse 6, 8092, Zurich, Switzerland

    Marc Pollefeys

Authors
  1. Lionel Heng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gim Hee Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marc Pollefeys
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lionel Heng.

Additional information

This is one of several papers published in Autonomous Robots comprising the “Special Issue on Robotics Science and Systems”

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Heng, L., Lee, G.H. & Pollefeys, M. Self-calibration and visual SLAM with a multi-camera system on a micro aerial vehicle. Auton Robot 39, 259–277 (2015). https://doi.org/10.1007/s10514-015-9466-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-015-9466-8

Keywords

Search

Navigation