Guidance for Autonomous Aerial Manipulator Using Stereo Vision

  • Christoforos KanellakisEmail author
  • George Nikolakopoulos
Open Access


Combining the agility of Micro Aerial Vehicles (MAV) with the dexterity of robotic arms leads to a new era of Aerial Robotic Workers (ARW) targeting infrastructure inspection and maintenance tasks. Towards this vision, this work focuses on the autonomous guidance of the aerial end-effector to either reach or keep desired distance from areas/objects of interest. The proposed system: 1) is structured around a real-time object tracker, 2) employs stereo depth perception to extract the target location within the surrounding scene, and finally 3) generates feasible poses for both the arm and the MAV relative to the target. The performance of the proposed scheme is experimentally demonstrated in multiple scenarios of increasing complexity.


Vision based guidance Aerial manipulator MAV 



This work has received funding from the European Unions Horizon 2020 Research and Innovation Programs under the Grant Agreements No.644128, AEROWORKS and No.730302, SIMS

Funding Information

Open access funding provided by Lulea University of Technology.


  1. 1.
    Mansouri, S.S., Kanellakis, C., Fresk, E., Kominiak, D., Nikolakopoulos, G.: Cooperative uavs as a tool for aerial inspection of the aging infrastructure. In: Field and Service Robotics, pp 177–189. Springer (2018)Google Scholar
  2. 2.
    Lee, A.C., Dahan, M., Weinert, A.J., Amin, S.: Leveraging suas for infrastructure network exploration and failure isolation. J. Intell. Robot. Syst., 1–29 (2018)Google Scholar
  3. 3.
    Yuan, C., Liu, Z., Zhang, Y.: Learning-based smoke detection for unmanned aerial vehicles applied to forest fire surveillance. Journal of Intelligent & Robotic Systems. (2018)
  4. 4.
    Sampedro, C., Rodriguez-Ramos, A., Bavle, H., Carrio, A., de la Puente, P., Campoy, P.: A fully-autonomous aerial robot for search and rescue applications in indoor environments using learning-based techniques. Journal of Intelligent & Robotic Systems. (2018)
  5. 5.
    Kondak, K., Ollero, A., Maza, I., Krieger, K., Albu-Schaeffer, A., Schwarzbach, M., Laiacker, M.: Unmanned aerial systems physically interacting with the environment: Load transportation, deployment, and aerial manipulation. In: Handbook of Unmanned Aerial Vehicles, pp 2755–2785. Springer (2015)Google Scholar
  6. 6.
    Lindsey, Q., Mellinger, D., Kumar, V.: Construction of cubic structures with quadrotor teams. In: Proc. Robotics, Science & Systems VII (2011)Google Scholar
  7. 7.
    Kim, S., Seo, H., Choi, S., Kim, H.J.: Vision-guided aerial manipulation using a multirotor with a robotic arm. IEEE/ASME Trans. Mechatron. 21(4), 1912 (2016)CrossRefGoogle Scholar
  8. 8.
    Seo, H., Kim, S., Kim, H.J.: Aerial grasping of cylindrical object using visual servoing based on stochastic model predictive control. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp 6362–6368. IEEE (2017)Google Scholar
  9. 9.
    Santamaria-Navarro, A., Grosch, P., Lippiello, V., Sola, J., Andrade-Cetto, J.: Uncalibrated visual servo for unmanned aerial manipulation. IEEE/ASME Transactions on Mechatronics (2017)Google Scholar
  10. 10.
    Steich, K., Kamel, M., Beardsley, P., Obrist, M.K., Siegwart, R., Lachat, T.: Tree cavity inspection using aerial robots. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 4856–4862. IEEE (2016)Google Scholar
  11. 11.
    Ramon Soria, P., Arrue, B.C., Ollero, A.: Detection, location and grasping objects using a stereo sensor on uav in outdoor environments. Sensors 17(1), 103 (2017)CrossRefGoogle Scholar
  12. 12.
    Lippiello, V., Cacace, J., Santamaria-Navarro, A., Andrade-Cetto, J., Trujillo, M.A., Esteves, Y.R., Viguria, A.: Hybrid visual servoing with hierarchical task composition for aerial manipulation. IEEE Robot. Autom. Lett. 1(1), 259 (2016)CrossRefGoogle Scholar
  13. 13.
    Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. In: IEEE Transactions on Pattern Analysis and Machine Intelligence.
  14. 14.
    Compact AeRial MAnipulator (CARMA) Assembly overwiew.
  15. 15.
    Wuthier, D., Kominiak, D., Kanellakis, C., Andrikopoulos, G., Fumagalli, M., Schipper, G., Nikolakopoulos, G.: On the design, modeling and control of a novel compact aerial manipulator. In: 2016 24th Mediterranean Conference on Control and Automation (MED), pp 665–670. IEEE (2016)Google Scholar
  16. 16.
    Lynen, S., Achtelik, M., Weiss, S., Chli, M., Siegwart, R.: A robust and modular multi-sensor fusion approach applied to mav navigation. In: Proc. of the IEEE/RSJ Conference on Intelligent Robots and Systems (IROS) (2013)Google Scholar
  17. 17.
    Alexis, K., Nikolakopoulos, G., Tzes, A.: Switching model predictive attitude control for a quadrotor helicopter subject to atmospheric disturbances. Control. Eng. Pract. 19(10), 1195 (2011)CrossRefGoogle Scholar
  18. 18.
    Alexis, K., Nikolakopoulos, G., Tzes, A.: Model predictive quadrotor control: Attitude, altitude and position experimental studies. IET Control Theory Appl. 6(12), 1812 (2012)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Kamel, M., Stastny, T., Alexis, K., Siegwart, R.: Model Predictive Control for Trajectory Tracking of Unmanned Aerial Vehicles Using Robot Operating System, pp 3–39. Springer International Publishing, Cham (2017),
  20. 20.
    Piccinini, P., Prati, A., Cucchiara, R.: Real-time object detection and localization with sift-based clustering. Image Vis. Comput. 30(8), 573 (2012)CrossRefGoogle Scholar
  21. 21.
    Smeulders, A.W., Chu, D.M., Cucchiara, R., Calderara, S., Dehghan, A., Shah, M.: Visual tracking: an experimental survey. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1442 (2014)CrossRefGoogle Scholar
  22. 22.
    Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328 (2008)CrossRefGoogle Scholar
  23. 23.
    Zhan, Q., Liang, Y., Xiao, Y.: Color-based segmentation of point clouds. Laser Scan. 38(3), 155 (2009)Google Scholar
  24. 24.
    Rusu, R.B., Cousins, S.: 3d is here: Point cloud library (pcl). In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp 1–4. IEEE (2011)Google Scholar
  25. 25.
    Oehler, B., Stueckler, J., Welle, J., Schulz, D., Behnke, S.: Efficient multi-resolution plane segmentation of 3d point clouds. Intell Robot Appl, 145–156 (2011)Google Scholar

Copyright information

© The Author(s) 2019

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Robotics Group, Department of Computer, Electrical and Space EngineeringLuleå University of TechnologyLuleåSweden

Personalised recommendations