Skip to main content
SpringerLink
Log in

Multi-Camera Tracking and Mapping for Unmanned Aerial Vehicles in Unstructured Environments

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Pose estimation for small unmanned aerial vehicles has made large improvements in recent years, leading to vehicles that use a suite of sensors to navigate and explore various environments. In particular, cameras have become popular due to their low weight and power consumption, as well as the large amount of data they capture. However, processing this data to extract useful information has proved challenging, as the pose estimation problem is inherently nonlinear and, depending on the cameras’ field of view, potentially ill-posed. Results from the field of multi-camera egomotion estimation show that these issues can be reduced or eliminated by using multiple cameras positioned appropriately. In this work, we make use of these insights to develop a multi-camera visual pose estimator using ultra wide angle fisheye cameras, leading to a system that has many advantages over traditional visual pose estimators. The system is tested in a variety of configurations and flight scenarios on an unprepared urban rooftop, including landings and takeoffs. To our knowledge, this is the first time a visual pose estimator has been shown to be able to continuously track the pose of a small aerial vehicle throughout the landing and subsequent takeoff maneuvers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Achtelik, M.: Vision-Based Pose Estimation for Autonomous Micro Aerial Vehicles in GPS-Denied Areas. Master’s Thesis, Technische Universität München (2009)

    Google Scholar 

  2. Ahrens, S., Levine, D., Andrews, G., How, J.: Vision-based guidance and control of a hovering vehicle in unknown, GPS-denied environments. In: IEEE International Conference on Robotics and Automation, pp. 2643–2648. Kobe, Japan (2009). doi:10.1109/ROBOT.2009.5152680

  3. Angeli, A., Doncieux, S., Meyer, J.A., Filliat, D.: Real-time visual loop-closure detection. In: IEEE International Conference on Robotics and Automation, pp. 1842–1847 (2008)

  4. Baker, P., Fermüller, C., Aloimonos, Y., Pless, R.: A spherical eye from multiple cameras (makes better models of the world). In: IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. 576–583. Kauai, HI, USA (2001). doi:10.1109/CVPR.2001.990525

  5. Blosch, M., Weiss, S., Scaramuzza, D., Siegwart, R.: Vision Based MAV Navigation in Unknown and Unstructured Environments. In: IEEE International Conference on Robotics and Automation, pp. 21–28. Anchorage, AK, USA (2010)

  6. Carrera, G., Angeli, A., Davison, A.J.: SLAM-based automatic extrinsic calibration of a multi-camera rig. In: IEEE International Conference on Robotics and Automation, pp. 2652–2659. Shanghai, China (2011). doi:10.1109/ICRA.2011.5980294

  7. Clipp, B., Kim, J.H., Frahm, J.M., Pollefeys, M., Hartley, R.I.: Robust 6DOF Motion Estimation for Non-Overlapping, Multi-Camera Systems. In: 9th IEEE Workshop on Applications of Computer Vision, pp. 1–8. Copper Mountain, CO, USA (2008)

  8. Davison, A.J., Reid, I.D., Molton, N., Stasse, O.: MonoSLAM: Real-time single camera SLAM. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 1052–1067 (2007)

    Article  Google Scholar 

  9. Dellaert, F., Kaess, M.: Square root SAM: simultaneous localization and mapping via square root information smoothing. Int. J. Robot. Res. 25, 1181–1203 (2006)

    Article  MATH  Google Scholar 

  10. Demonceaux, C., Vasseur, P., Regard, C.: Omnidirectional vision on UAV for attitude computation. In: IEEE International Conference on Robotics and Automation, pp. 2842–2847. Orlando, FL (2006)

  11. Devernay, F., Faugeras, O.D.: Straight lines have to be straight. Mach. Vis. Appl. 13(1), 14–24 (2001)

    Article  Google Scholar 

  12. Esquivel, S., Woelk, F., Koch, R.: Calibration of a multicamera rig from non-overlapping views. In: 29th DAGM Conference on Pattern Recognition, Vol. 4713, pp. 82–91. Heidelberg, Germany (2007)

  13. Fermüller, C., Aloimonos, Y.: Observability of 3D Motion. Int. J. Comput. Vis. 37, 43–63 (2000)

    Article  MATH  Google Scholar 

  14. Frahm, J.M., Kser, K., Koch, R.: Pose estimation for multi-camera systems. In: 26th DAGM Symposium, pp. 286–293. Tübingen, Germany (2004). doi:10.1007/978-3-540-28649-3_35

  15. Fraundorfer, F., Heng, L., Honegger, D., Lee, G., Meier, L., Tanskanen, P., Pollefeys, M.: Vision-Based Autonomous Mapping and Exploration Using a Quadrotor MAV. In: IEEE International Conference on Intelligent Robots and Systems, pp. 4557–4564. Vilamoura, Portugal (2012)

  16. Gove, D.: The cost of mutexes. http://blogs.oracle.com/d/entry/the_cost_of_mutexes. Accessed 2 June 2013 (2008)

  17. Grisetti, G., Kuemmerle, R., Stachniss, C., Frese, U., Hertzberg, C.: Hierarchical optimization on manifolds for online 2d and 3d mapping. In: IEEE International Conference on Robotics and Automation (2010)

  18. Handa, A., Newcombe, R.A., Angeli, A., Davison, A.J.: Real-Time Camera Tracking: When is High Frame-Rate Best? In: 12th European Conference on Computer Vision, pp. 222–235. Florence, Italy (2012). doi:10.1007/978-3-642-33786-4_17

  19. Harmat, A., Sharf, I., Trentini, M.: Parallel Tracking and Mapping with Multiple Cameras on an Unmanned Aerial Vehicle. In: International Conference on Intelligent Robotics and Applications, pp. 421–432. Montreal, Canada (2012)

  20. Horaud, R., Dornaika, F.: Hand-eye calibration. Int. J. Robot. Res. 14(3), 195–210 (1995)

    Article  Google Scholar 

  21. Kaess, M., Dellaert, F.: Probabilistic structure matching for visual SLAM with a Multi-Camera Rig. Comp. Vision Image Underst. 114, 286–296 (2010)

    Article  Google Scholar 

  22. Kazik, T., Kneip, L., Nikolic, J., Pollefeys, M., Siegwart, R.: Real-time 6D stereo Visual Odometry with non-overlapping fields of view. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1529–1536. Providence, RI, USA (2012). doi:10.1109/CVPR.2012.6247843

  23. Kim, J.H., Hartley, R., Frahm, J.M., Pollefeys, M.: Visual Odometry for Non-overlapping Views Using Second-Order Cone Programming. In: 8th Asian Conference on Computer vision, pp. 353–362. Tokyo, Japan (2007) doi:10.1007/978-3-540-76390-1_35

  24. Kim, J.S., Hwangbo, M., Kanade, T.: Motion Estimation using Multiple Non-Overlapping Cameras for Small Unmanned Aerial Vehicles. In: IEEE International Conference on Robotics and Automation, 3076–3081. Pasadena, CA, USA (2008)

  25. Klein, G., Murray, D.: Parallel Tracking and Mapping for Small AR Workspaces. In: International Symposium on Mixed and Augmented Reality, pp. 225–234. Nara, Japan (2007). doi:10.1109/ISMAR.2007.4538852

  26. Klein, G., Murray, D.: Improving the agility of keyframe-based SLAM. In: 10th European Conference on Computer Vision, 802–815. Marseille, France (2008)

  27. Kümmerle, R., Grisetti, G., Strasdat, H., Konolige, K., Burgard, W.: g2o: A General Framework for Graph Optimization. In: IEEE International Conference on Robotics and Automation, pp. 3607–3613. Shanghai, China (2011). doi:10.1109/ICRA.2011.5979949

  28. Mei, C., Sibley, G., Cummins, M., Newman, P., Reid, I.: RSLAM: a system for large-scale mapping in constant-time using stereo. Int. J. Comput. Vis. 94(2), 198–214 (2011)

    Article  Google Scholar 

  29. Mellinger, D., Michael, N., Kumar, V.: Trajectory generation and control for precise aggressive maneuvers with quadrotors. Int. J. Robot. Res. 31 (5), 664–674 (2012). doi:10.1177/0278364911434236

    Article  Google Scholar 

  30. Neumann, J., Fermuller, C., Aloimonos, Y.: Eyes from eyes: new cameras for structure from motion. In: Third Workshop on Omnidirectional Vision, pp. 19–26. Copenhagen, Denmark (2002). doi:10.1109/OMNVIS.2002.1044486

  31. Pless, R.: Using many cameras as one. In: IEEE Conference on Computer Vision and Pattern Recognition, Vol. 2, pp. 587–93. Madison, WI, USA (2003). doi:10.1109/CVPR.2003.1211520

  32. Quigley, M., Conley, K., Gerkey, B.P., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: ROS: an open-source Robot Operating System. In: ICRA Workshop on Open Source Software. Kobe, Japan (2009)

  33. Ragab, M.E., Wong, K.: Multiple nonoverlapping camera pose estimation. In: IEEE International Conference on Image Processing, pp. 3253–3256. Hong Kong (2010)

  34. Rituerto, A., Puig, L., Guerrero, J.J.: Comparison of omnidirectional and convential monocular systems for visual SLAM. In: Workshop on Omnidirectional Vision, Camera Networks, and Non-classical Cameras. Zaragoza, Spain (2010)

  35. Scaramuzza, D.: Omnidirectional Vision: From Calibration to Robot Motion Estimation. Ph.D. thesis, ETH Zurich (2008)

  36. Schauwecker, K., Zell, A.: On-Board Dual-Stereo-Vision for Autonomous Quadrotor Navigation. In: International Conference on Unmanned Aircraft Systems, pp. 333–342. Atlanta, GA, USA (2013). doi:10.1109/ICUAS.2013.6564706

  37. Schneider, J., Schindler, F., Labe, T., Forstner, W.: Bundle Adjustment for Multi-Camera Systems with Points at Infinity. In: 22nd Congress of the International Society for Photogrammetry and Remote Sensing. Melbourne, Australia (2012)

  38. Sharf, I., Wolf, A., Rubin, M.: Arithmetic and geometric solutions for average rigid-body rotation. Mech. Mach. Theory 45(9), 1239–1251 (2010)

    Article  MATH  Google Scholar 

  39. Sibley, G., Mei, C., Reid, I., Newman, P.: Adaptive relative bundle adjustment. In: Robotics: Science and Systems. Seattle, USA (2009)

  40. Solà, J., Monin, A., Devy, M., Vidal-Calleja, T.: Fusing monocular information in multicamera SLAM. IEEE Trans. Robot. 24 (5), 958–968 (2008)

    Article  Google Scholar 

  41. Strasdat, H., Davison, A., Montiel, J., Konolige, K.: Double window optimisation for constant time visual SLAM. In: IEEE International Conference on Computer Vision, pp. 2352–2359 (2011)

  42. Strasdat, H., Montiel, J.M.M., Davison, A.: Real-Time Monocular SLAM: Why Filter?. In: IEEE International Conference on Robotics and Automation, pp. 2657–2664. Anchorage, AK, USA (2010)

  43. Tarhan, M., Altug, E.: Control of a quadrotor air vehicle by vanishing points in catadioptric images. In: International Symposium on Optomechatronic Technologies, pp. 92–97. Istanbul, Turkey (2009). doi:10.1109/ISOT.2009.5326105

  44. Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment - a modern synthesis, In: International Workshop on Vision Algorithms, pp. 298–372. London, UK (2000)

  45. Weiss, S., Achtelik, M., Kneip, L., Scaramuzza, D., Siegwart, R.: Intuitive 3D Maps for MAV Terrain Exploration and Obstacle Avoidance. J. Intell. Robot. Syst. 61 (1-4), 473–493 (2010)

    Article  Google Scholar 

  46. Weiss, S., Scaramuzza, D., Siegwart, R.: Monocular-SLAM–based navigation for autonomous micro helicopters in GPS-denied environments. J. Field Robot. 28(6), 854–874 (2011)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. McGill University, Montreal, Quebec, CANADA

    Adam Harmat & Inna Sharf

  2. Defence Research and Development Canada - Suffield, Medicine Hat, Alberta, Canada

    Michael Trentini

Authors
  1. Adam Harmat
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Trentini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Inna Sharf
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adam Harmat.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Harmat, A., Trentini, M. & Sharf, I. Multi-Camera Tracking and Mapping for Unmanned Aerial Vehicles in Unstructured Environments. J Intell Robot Syst 78, 291–317 (2015). https://doi.org/10.1007/s10846-014-0085-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-014-0085-y

Keywords

Search

Navigation