Combining Deep Learning and RGBD SLAM for Monocular Indoor Autonomous Flight

  • J. Martinez-CarranzaEmail author
  • L. O. Rojas-Perez
  • A. A. Cabrera-Ponce
  • R. Munguia-Silva
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11289)


We present a system that uses deep learning and visual SLAM for autonomous flight in indoor environments. In this spirit, we use a state-of-the-art CNN architecture to obtain depth estimates, on a frame-to-frame basis, of images obtained from the drone’s onboard camera, and use them in a visual SLAM system to obtain both camera pose estimates with a metric that is further passed to a PID controller, responsible for the autonomous flight. However, because depth estimation and visual SLAM system are computationally intensive tasks, the processing is carried out off-board on a ground control station that receives online imagery and inertial data transmitted by the drone via a WiFi channel during the flight mission. Further, the metric pose estimates are used by the PID controller that communicates back to the vehicle with the caveat that synchronisation issues may arise in between the frame reception and the pose estimation output, typically with the frame reception running at 30 Hz, and the pose estimation at 15 Hz. As a consequence, the controller may also exhibit a delay in the control loop, provoking a flight off-track the trajectory set by the way-points. To mitigate this, we implemented a stochastic filter that estimates velocity and acceleration of the vehicle to predict pose estimates in those frames where no pose estimate is available yet, and when available, to compensate for the communication delay. We have evaluated the use of this methodology for indoor autonomous flight with promising results.



This work has also been partially funded by a CONACYT-INEGI fund with project no. 268528 and the Royal Society through the Newton Advanced Fellowship with reference NA140454.


  1. 1.
    Davison, A.J., Reid, I.D., Molton, N.D., Stasse, O.: MonoSLAM: real-time single camera SLAM. IEEE Trans. Pattern Anal. Mach. Intell. 29, 1052–1067 (2007)CrossRefGoogle Scholar
  2. 2.
    Weiss, S., Scaramuzza, D., Siegwart, R.: Monocular-SLAM-based navigation for autonomous micro helicopters in GPS-denied environments. J. Field Robot. 28, 854–874 (2011)CrossRefGoogle Scholar
  3. 3.
    Magree, D., Mooney, J.G., Johnson, E.N.: Monocular visual mapping for obstacle avoidance on UAVs. J. Intell. Robot. Syst. 74, 17–26 (2014)CrossRefGoogle Scholar
  4. 4.
    Rojas-Perez, L.O., Martinez-Carranza, J.: Metric monocular SLAM and colour segmentation for multiple obstacle avoidance in autonomous flight. In: IEEE 4th RED-UAS (2017)Google Scholar
  5. 5.
    Saxena, A., Chung, S.H., Ng, A.Y.: Learning depth from single monocular images. In: NIPS, vol. 18, MIT Press (2005)Google Scholar
  6. 6.
    Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., Navab, N.: Deeper depth prediction with fully convolutional residual networks. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 239–248, IEEE (2016)Google Scholar
  7. 7.
    Tateno, K., Tombari, F., Laina, I., Navab, N.: CNN-SLAM: real-time dense monocular SLAM with learned depth prediction. arXiv preprint arXiv:1704.03489 (2017)
  8. 8.
    Konam, S.: Vision-based navigation and deep-learning explanation for autonomy, in Masters thesis, Robotics Institute, Carnegie Mellon University (2017)Google Scholar
  9. 9.
    Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans. Robot. 31, 1147–1163 (2015)CrossRefGoogle Scholar
  10. 10.
    Bi, Y., et al.: An MAV localization and mapping system based on dual realsense cameras. In: International Micro Air Vehicles, Conferences Competitions, National University of Singapore, Singapore (2016). Technical ReportGoogle Scholar
  11. 11.
    Li, J., et al.: Real-time simultaneous localization and mapping for UAV: a survey. In: International Micro Air Vehicle Conference and Competition (IMAV) (2010)Google Scholar
  12. 12.
    Bloesch, M., Omari, S., Hutter, M., Siegwart, R.: Robust visual inertial odometry using a direct EKF-based approach. In: IROS (2015)Google Scholar
  13. 13.
    Teixeira, L., Alzugaray, I., Chli, M.: Autonomous aerial inspection using visual-inertial robust localization and mapping. In: Hutter, M., Siegwart, R. (eds.) Field and Service Robotics. SPAR, vol. 5, pp. 191–204. Springer, Cham (2018). Scholar
  14. 14.
    Lin, Y., et al.: Autonomous aerial navigation using monocular visual-inertial fusion. J. Field Robot. 35(1), 23–51 (2018)CrossRefGoogle Scholar
  15. 15.
    Xu, W., Choi, D., Wang, G.: Direct visual-inertial odometry with semi-dense mapping. Comput. Electr. Eng. (2018)Google Scholar
  16. 16.
    Mu, X., Chen, J., Zhou, Z., Leng, Z., Fan, L.: Accurate initial state estimation in a monocular visual-inertial SLAM system. Sensors 18, 506 (2018)CrossRefGoogle Scholar
  17. 17.
    Usenko, V., Engel, J., Stckler, J., Cremers, D.: Direct visual-inertial odometry with stereo cameras. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 1885–1892 (2016)Google Scholar
  18. 18.
    Rebecq, H., Horstschaefer, T., Scaramuzza, D.: Real-time visualinertial odometry for event cameras using keyframe-based nonlinear optimization. In: British Machine Vision Conference (BMVC), vol. 3 (2017)Google Scholar
  19. 19.
    Vidal, A.R., Rebecq, H., Horstschaefer, T., Scaramuzza, D.: Hybrid, frame and event based visual inertial odometry for robust, autonomous navigation of quadrotors. arXiv preprint arXiv:1709.06310 (2017)
  20. 20.
    Mancini, M., Costante, G., Valigi, P., Ciarfuglia, T.A., Delmerico, J., Scaramuzza, D.: Toward domain independence for learning-based monocular depth estimation. IEEE Robot. Autom. Lett. 2, 1778–1785 (2017)CrossRefGoogle Scholar
  21. 21.
    Wang, P., Shen, X., Lin, Z., Cohen, S., Price, B., Yuille, A.L.: Towards unified depth and semantic prediction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2800–2809 (2015)Google Scholar
  22. 22.
    Chakrabarti, A., Shao, J., Shakhnarovich, G.: Depth from a single image by harmonizing overcomplete local network predictions. In: Advances in Neural Information Processing Systems, pp. 2658–2666 (2016)Google Scholar
  23. 23.
    Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: CVPR, vol. 2, p. 7 (2017)Google Scholar
  24. 24.
    Chen, Z., Lam, O., Jacobson, A., Milford, M.: Convolutional neural network-based place recognition. arXiv preprint arXiv:1411.1509 (2014)
  25. 25.
    Kendall, A., Grimes, M., Cipolla, R.: PoseNeT: a convolutional network for real-time 6-DOF camera relocalization. In: IEEE International Conference on Computer Vision, pp. 2938–2946, IEEE (2015)Google Scholar
  26. 26.
    Li, R., Liu, Q., Gui, J., Gu, D., Hu, H.: Indoor relocalization in challenging environments with dual-stream convolutional neural networks. IEEE Trans. Autom. Sci. Eng. 15(2), 651–662 (2017)CrossRefGoogle Scholar
  27. 27.
    Weerasekera, C.S., Garg, R., Reid, I.: Learning deeply supervised visual descriptors for dense monocular reconstruction. arXiv preprint arXiv:1711.05919 (2017)
  28. 28.
    Mukasa, T., Xu, J., Stenger, B.: 3D scene mesh from CNN depth predictions and sparse monocular SLAM. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 921–928 (2017)Google Scholar
  29. 29.
    Yang, S., Song, Y., Kaess, M., Scherer, S.: Pop-up SLAM: Semantic monocular plane SLAM for low-texture environments. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1222–1229. IEEE (2016)Google Scholar
  30. 30.
    Zhou, T., Brown, M., Snavely, N., Lowe, D.G.: Unsupervised learning of depth and ego-motion from video. In: CVPR, vol. 2, p. 7 (2017)Google Scholar
  31. 31.
    Gemeiner, P., Davison, A., Vincze, M.: Improving localization robustness in monocular SLAM using a high-speed camera. In: Robotics: Science and Systems (2008)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • J. Martinez-Carranza
    • 1
    Email author
  • L. O. Rojas-Perez
    • 1
  • A. A. Cabrera-Ponce
    • 1
  • R. Munguia-Silva
    • 1
  1. 1.The Computer Science Department of the Instituto Nacional de Astrofisica Optica y ElectronicaPueblaMexico

Personalised recommendations