Abstract
Aerial transportation and manipulation have attracted increasing attention in the unmanned aerial vehicle field, and visual servoing methodology is widely used to achieve the autonomous aerial grasping of a target object. However, the existing marker-based solutions pose a challenge to the practical application of target grasping owing to the difficulty in attaching markers on targets. To address this problem, this study proposes a novel image-based visual servoing controller based on natural features instead of artificial markers. The natural features are extracted from the target images and further processed to provide servoing feature points. A six degree-of-freedom (6-DoF) aerial manipulator system is proposed with differential kinematics deduced to achieve aerial grasping. Furthermore, a controller is designed when the target object is outside a manipulator’s workspace by utilizing both the degrees-of-freedom of unmanned aerial vehicle and manipulator joints. Thereafter, a weight matrix is used as basis to develop a multi-tasking visual servoing framework to integrate the controllers inside and outside the manipulator’s workspace. Lastly, experimental results are provided to verify the effectiveness of the proposed approach.
Article PDF
Similar content being viewed by others
References
Lee H, Kim H J. Estimation, control, and planning for autonomous aerial transportation. IEEE Transactions on Industrial Electronics, 2017, 64, 3369–3379.
Lee H, Kim S, Kim H J. Control of an aerial manipulator using on-line parameter estimator for an unknown payload. Proceedings of the 2015 IEEE International Conference on Automation Science and Engineering, Gothenburg, Sweden, 2015, 316–321.
Kondak K, Huber F, Schwarzbach M, Laiacker M, Sommer D, Bejar M, Ollero A. Aerial manipulation robot composed of an autonomous helicopter and a 7 degrees of freedom industrial manipulator. Proceedings of the 2014 IEEE International Conference on Robotics and Automation, Hong Kong, China, 2014, 2107–2112.
Laiacker M, Huber F, Kondak K. High accuracy visual servoing for aerial manipulation using a 7 degrees of freedom industrial manipulator. Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, Daejeon, Korea, 2016, 1631–1636.
Korpela C, Orsag M, Oh P. Towards valve turning using a dual-arm aerial manipulator. Proceedings of the 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Illinois, USA, 2014, 3411–3416.
Ruggiero F, Lippiello V, Ollero A. Aerial manipulation: A literature review. IEEE Robotics and Automation Letters, 2018, 3, 1957–1964.
Munoz-Morera J, Maza I, Fernandez-Aguera C J, Caballero F, Ollero A. Assembly planning for the construction of structures with multiple UAS equipped with robotic arms. Proceedings of the 2015 International Conference on Unmanned Aircraft Systems, Colorado, USA, 2015, 1049–1058.
Kim S, Choi S, Kim H J. Aerial manipulation using a quadrotor with a two dof robotic arm. Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan, 2013, 4990–4995.
Sanchez M I, Acosta J Á, Ollero A. Integral action in first-order closed-loop inverse kinematics. Application to aerial manipulators. Proceedings of the 2015 IEEE International Conference on Robotics and Automation, Washington, USA, 2015, 5297–5302.
Kim S, Seo H, Shin J, Kim H J. Cooperative aerial manipulation using multirotors with multi-DOF robotic arms. IEEE/ASME Transactions on Mechatronics, 2018, 23, 702–713.
Kim H, Lee H, Choi S, Noh Y K, Kim H J. Motion planning with movement primitives for cooperative aerial transportation in obstacle environment. Proceedings of the 2017 IEEE International Conference on Robotics and Automation, Singapore, Singapore, 2017, 2328–2334.
Lee H, Kim H, Kim W, Kim H J. An integrated framework for cooperative aerial manipulators in unknown environments. IEEE Robotics and Automation Letters, 2018, 3, 2307–2314.
Garimella G, Kobilarov M. Towards model-predictive control for aerial pick-and-place. Proceedings of the 2015 IEEE International Conference on Robotics and Automation, Washington, USA, 2015, 4692–4697.
Cataldi E, Muscio G, Trujillo M A, Rodríguez Y, Pierri F, Antonelli G, Caccavale F, Viguria A, Chiaverini S, Ollero A. Impedance control of an aerial-manipulator: Preliminary results. Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, Daejeon, South Korea, 2016, 3848–3853.
Seo H, Kim S, Kim H J. Locally optimal trajectory planning for aerial manipulation in constrained environments. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, British Columbia, Canada, 2017, 1719–1724.
Rossi R, Santamaria-Navarro A, Andrade-Cetto J, Rocco P. Trajectory generation for unmanned aerial manipulators through quadratic programming. IEEE Robotics and Automation Letters, 2016, 2, 389–396.
Fang L X, Chen H Y, Lou Y J, Li Y J, Liu Y H. Visual grasping for a lightweight aerial manipulator based on NSGA-II and kinematic compensation. Proceedings of the 2018 IEEE International Conference on Robotics and Automation, Queensland, Australia, 2018, 3488–3493.
Thomas J, Loianno G, Sreenath K, Kumar V. Toward image based visual servoing for aerial grasping and perching. Proceedings of the 2014 IEEE International Conference on Robotics and Automation, Hong Kong, China, 2014, 2113–2118.
Thomas J, Loianno G, Polin J, Sreenath K, Kumar V. Toward autonomous avian-inspired grasping for micro aerial vehicles. Bioinspiration & Biomimetics, 2014, 9, 025010.
Kim S, Seo H, Choi S, Kim H J. Vision-guided aerial manipulation using a multirotor with a robotic arm. IEEE/ASME Transactions on Mechatronics, 2016, 21, 1912–1923.
Seo H, Kim S, Kim H J. Aerial grasping of cylindrical object using visual servoing based on stochastic model predictive control. Proceedings of the 2017 IEEE International Conference on Robotics and Automation, Singapore, Singapore, 2017, 6362–6368.
Lippiello V, Cacace J, Santamaria-Navarro A, Andrade-Cetto J, Trujillo M A, Esteves Y R, Viguria A. Hybrid visual servoing with hierarchical task composition for aerial manipulation. IEEE Robotics and Automation Letters, 2015, 1, 259–266.
Santamaria-Navarro A, Lippiello V, Andrade-Cetto J. Task priority control for aerial manipulation. Proceedings of the 2014 IEEE International Symposium on Safety, Security, and Rescue Robotics, Hokkaido, Japan, 2014, 1–6.
Muscio G, Pierri F, Trujillo M A, Cataldi E, Antonelli G, Caccavale F, Viguria A, Chiaverini S, Ollero A. Coordinated control of aerial robotic manipulators: Theory and experiments. IEEE Transactions on Control Systems Technology, 2017, 26, 1406–1413.
Santamaria-Navarro A, Grosch P, Lippiello V, Solà J, Andrade-Cetto J. Uncalibrated visual servo for unmanned aerial manipulation. IEEE/ASME Transactions on Mechatronics, 2017, 22, 1610–1621.
Lippiello V, Fontanelli G A, Ruggiero F. Image-based visual-impedance control of a dual-arm aerial manipulator. IEEE Robotics and Automation Letters, 2018, 3, 1856–1863.
Collewet C, Marchand E. Photometric visual servoing. IEEE Transactions on Robotics, 2011, 27, 828–834.
Bateux Q, Marchand E. Histograms-based visual servoing. IEEE Robotics and Automation Letters, 2016, 2, 80–87.
Bateux Q, Marchand E, Leitner J, Chaumette F, Corke P. Training deep neural networks for visual servoing. Proceedings of the 2018 IEEE International Conference on Robotics and Automation, Queensland, Australia, 2018, 1–8.
Bruno S, Lorenzo S, Luigi V, Giuseppe O. Robotics: Modelling, planning and control. Advanced Textbooks in Control and Signal Processing Series, Springer, London, UK, 2009.
Chaumette F, Hutchinson S. Visual servo control. II. Advanced approaches [Tutorial]. IEEE Robotics & Automation Magazine, 2007, 14, 109–118.
Li Y L, Wang S J, Tian Q, Ding X Q. A survey of recent advances in visual feature detection. Neurocomputing, 2015, 149, 736–751.
Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60, 91–110.
Bay H, Tuytelaars T, Van Gool L. Surf: Speeded up robust features. Proceedings of the European Conference on Computer Vision, Graz, Austria, 2006, 404–417.
Rublee E, Rabaud V, Konolige K, Bradski G. ORB: An efficient alternative to SIFT or SURF. Proceedings of 2011 International Conference on Computer Vision, Barcelona, Spain, 2011, 2564–2571.
Meagher D. Geometric modeling using octree encoding. Computer Graphics and Image Processing, 1982, 19, 129–147.
Hartley R, Zisserman A. Multiple View Geometry in Computer Vision, Cambridge University Press, UK, 2003.
Acknowledgment
This study was partially supported by grants from the National Natural Science Foundation of China (Nos. U1713206, 61673131 and 51975550), the Bureau of Industry and Information Technology of Shenzhen (No. 20170505160946600), and Hong Kong Research Grant Council (No. 14204814).
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Luo, B., Chen, H., Quan, F. et al. Natural Feature-based Visual Servoing for Grasping Target with an Aerial Manipulator. J Bionic Eng 17, 215–228 (2020). https://doi.org/10.1007/s42235-020-0017-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42235-020-0017-4