Usage of multiple drones is necessary for aerial filming applications to ensure redundancy. However, this could inevitably contribute to higher risks of collisions, especially when the number of drones increases. Hence, this motivates us to explore various autonomous flight formation control methods that have the potential to enable multiple drones to effectively track a specific target at the same time. In this paper, we designed a model-free deep reinforcement learning algorithm, which is mainly based on the Deep Recurrent Q-Network concept, for the aforementioned purposes. The proposed algorithm was expanded into single and multi-agent types that enable multiple drones tracking while maintaining formation and preventing collision. The involved rewards in these approaches are two-dimensional in nature and are dependent on the communication system. Using Microsoft AirSim simulator, a virtual environment that includes four virtual drones was developed for experimental purposes. A comparison was made among various methods during the simulations, and the results concluded that the recurrent, single-agent model is the most effective method, being 33% more effective than its recurrent, multi-agent counterparts. The poor performance of the non-recurrent, single-agent baseline model also suggests that the recurrent elements in the network are essential to enable desirable multiple-drones flight.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Second Iteration Video: https://www.youtube.com/watch?v=ZT0SEAQG_U0
Third Iteration Video: https://www.youtube.com/watch?v=OdLcRP5R0MQ
Fourth Iteration Video: https://www.youtube.com/watch?v=aweLkL8Xr18
Codes can be found at the following link: https://github.com/raymondng76/IRS-Practice-Module-Dev
Abadi M, Agarwal A, Barham P et al (2016) TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. Cornell University. arXiv:1603.04467. Accessed 1 Jun 2020
Abughalieh KM, Sababha BH, Rawashdeh NA (2019) A video-based object detection and tracking system for weight sensitive UAVs. Multimed Tools Appl 78:9149–9167. https://doi.org/10.1007/s11042-018-6508-1
Alam MS, Natesha BV, Ashwin TS, et al. (2019) UAV Based cost-effective real-time abnormal event detection using edge computing. Multimed Tools Appl 78:35119–35134. https://doi.org/10.1007/s11042-019-08067-1
Becker-Ehmck P, Karl M, Peters J et al (2020) Learning to Fly via Deep Model-Based Reinforcement Learning. Cornell University. arXiv:2003.08876. Accessed 1 Jun 2020
Bochkovskiy A, Wang CY, Liao HYM (2020) YOLOv4: Optimal Speed and Accuracy of Object Detection. Cornell University. arXiv:2004.10934. Accessed 1 Jun 2020
Bonatti R, Ho C, Wang W et al (2019) Towards a Robust Aerial Cinematography Platform: Localizing and Tracking Moving Targets in Unstructured Environments. Cornell University. arXiv:1904.02319. Accessed 1 Jun 2020
Bonatti R, Zhang Y, Choudhury S et al (2018) Autonomous drone cinematographer: Using artistic principles to create smooth, safe, occlusion-free trajectories for aerial filming. Cornell University. arXiv:1808.09563. Accessed 1 Jun 2020
Chollet F et al (2019) Keras. https://keras.io. Accessed 1 Jun 2020
Cunha R (2017) ICCV2017 Tutorial: Drone vision for cinematography - Drone Formation and Flight Control. MultiDrone. https://multidrone.eu/wp-content/uploads/2017/01/04_Drone_Formation_Flight_Control.pdf. Accessed 1 Jun 2020
Epic Games (2020) Unreal Engine. https://www.unrealengine.com. Accessed 1 Jun 2020
Esmukov K, Tygart A, López A et al (2018) Geocoding library for Python. GitHub repository. https://github.com/geopy/geopy. Accessed 1 Jun 2020
French S (2018) Want to make six figures? Try being a drone pilot. MarketWatch. https://www.marketwatch.com/story/want-to-make-six-figures-try-being-a-drone-pilot-2018-08-10. Accessed 1 Jun 2020
Galvane Q, Fleureau J, Tariolle FL, Guillotel P (2017) Automated Cinematography with Unmanned Aerial Vehicles. Cornell University. arXiv:1712.04353. Accessed 1 Jun 2020
Galvane Q, Lino C, Christie M et al (2018) Directing cinematographic drones. ACM Trans Graph 37(3):1–18. https://doi.org/10.1145/3181975
Hausknecht M, Stone P (2017) Deep recurrent Q-Learning for partially observable MDPs. Cornell University. arXiv:1507.06527. Accessed 1 Jun 2020
Hong S (2019) Autonomous UAV Navigation without Collision using Visual Information in Airsim. GitHub repository. https://github.com/sunghoonhong/AirsimDRL. Accessed 1 Jun 2020
Huang C, Gao F, Pan J, et al. (2018) ACT: An autonomous drone cinematography system for action scenes. 2018 IEEE international conference on robotics and automation (ICRA). Brisbane 7039–7046 https://doi.org/10.1109/ICRA.2018.8460703
Huang C, Yang Z, Kong Y et al (2018) Through-the-lens Drone Filming. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid 4692–4699 https://doi.org/10.1109/IROS.2018.8594333
Huynh NA (2017) Training and Detecting Objects with YOLO3. GitHub repository. https://github.com/experiencor/keras-yolo3. Accessed 1 Jun 2020
James S, Freese M, Davison AJ (2019) PyRep: Bringing V-REP to Deep Robot Learning. Cornell University. arXiv:1906.11176. Accessed 1 Jun 2020
James S, Ma Z, Arrojo DR, Davison AJ (2019) RLBench: The Robot Learning Benchmark & Learning Environment. Cornell University. arXiv:1909.12271. Accessed 1 Jun 2020
Joubert N, Jane LE, Goldman DB, Berthouzoz F et al (2016) Towards a Drone Cinematographer: Guiding Quadrotor Cameras using Visual Composition Principles. Cornell University. arXiv:1610.01691. Accessed 1 Jun 2020
Karney CFF (2013) Algorithms for geodesics. J Geodesy 87:43–55. https://doi.org/10.1007/s00190-012-0578-z
Kostrikov I, Yarats D, Fergus R (2020) Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels. Cornell University. arXiv:2004.13649. Accessed 1 Jun 2020
Krishnan S, Boroujerdian B, Fu W et al (2019) Air Learning: An AI Research Platform for Algorithm-Hardware Benchmarking of Autonomous Aerial Robots. Cornell University. arXiv:1906.00421. Accessed 1 Jun 2020
Kwak J, Park JH, Sung Y (2019) Affective social big data generation algorithm for autonomous controls by CRNN-based end-to-end controls. Multimed Tools Appl 78:27175–27192. https://doi.org/10.1007/s11042-019-7703-4
Liu F, Yang A (2019) Application of gcForest to visual tracking using UAV image sequences. Multimed Tools Appl 78:27933–27956. https://doi.org/10.1007/s11042-019-07864-y
Liu H, Zhao W, Lewis L et al (2019) Attitude Synchronization for Multiple Quadrotors using Reinforcement Learning*. 2019 Chinese Control Conference (CCC). Guangzhou 2480–2483 https://doi.org/10.23919/ChiCC.2019.8865177
Mademlis I, Nikolaidis N, Tefas A, et al. (2018) Autonomous unmanned aerial vehicles filming in dynamic unstructured outdoor environments. IEEE Signal Proc Mag 36(1):147–153. https://doi.org/10.1109/MSP.2018.2875190
Microsoft (2019) Microsoft Drone Rescue. GitHub repository. https://github.com/microsoft/DroneRescue. Accessed 1 Jun 2020
Microsoft (2020) Open source simulator for autonomous vehicles built on Unreal Engine / Unity, from Microsoft AI & Research. GitHub repository. https://github.com/microsoft/AirSim. Accessed 1 Jun 2020
Nägeli T, Alonso-Mora J, Domahidi A et al (2017) Real-Time Motion planning for aerial videography with dynamic obstacle avoidance and viewpoint optimization. IEEE Robot Autom Lett 2(3):1696–1703. https://doi.org/10.1109/LRA.2017.2665693
Nägeli T, Meier L, Domahidi A, et al. (2017) Real-time planning for automated multi-view drone cinematography. ACM Trans Graph 36 (4):1–10. https://doi.org/10.1145/3072959.3073712
Nikolaidis N, Mademlis I, Raptopoulou C, Bull D (2017) ICCV2017 Tutorial: Drone vision for cinematography - Drone Cinematography. MultiDrone. https://multidrone.eu/wp-content/uploads/2017/01/06_Drone-Cinematography.pdf. Accessed 1 Jun 2020
Passalis N, Tefas A (2019) Deep reinforcement learning for controlling frontal person close-up shooting. Neurocomputing 335:37–47. https://doi.org/10.1016/j.neucom.2019.01.046
Redmon J, Farhadi A (2018) YOLOv3: An Incremental Improvement. Cornell University. arXiv:1804.02767. Accessed 1 Jun 2020
Rising J (2015) Drones vs Helicopter - Part 1: How drones are changing the aerial video industry. Flight Evolved. https://flight-evolved.com/drone-vs-helicopter. Accessed 1 Jun 2020
Sabetghadam B, Alcántara A, Capitán J et al (2019) Optimal trajectory planning for autonomous drone cinematography. 2019 european conference on mobile robots (ECMR). Prague 1–7 https://doi.org/10.1109/ECMR.2019.8870950
Tefas A, Nousi P, Passalis N et al (2017) ICCV2017 Tutorial: Drone vision for cinematography - Deep Learning for Drone Vision in Cinematography. MultiDrone. https://multidrone.eu/wp-content/uploads/2017/01/05_Deep-Learning-for-Drone-Vision-in-Cinematography.pdf. Accessed 1 Jun 2020
Torres-González A, Capitán J, Cunha R et al (2017) A multidrone approach for autonomous cinematography planning. In: In: Ollero A, Sanfeliu A, Montano L et al (eds) ROBOT 2017: Third iberian robotics conference. ROBOT 2017. Advances in intelligent systems and computing, vol 693. Springer, Cham, pp 337–349, DOI https://doi.org/10.1007/978-3-319-70833-1_28
Tzutalin (2020) LabelImg is a graphical image annotation tool and label object bounding boxes in images. GitHub repository. https://github.com/tzutalin/labelImg. Accessed 1 Jun 2020
Wang T, Qin R, Chen Y et al (2019) A reinforcement learning approach for UAV target searching and tracking. Multimed Tools Appl 78:4347–4364. https://doi.org/10.1007/s11042-018-5739-5
Yang H, Xie K, Huang S, Huang H (2018) Uncut aerial video via a single sketch. Comput Graph Forum 37:191–199. https://doi.org/10.1111/cgf.13559
Zanol R, Chiariotti F, Zanella A (2019) Drone mapping through multi-agent reinforcement learning. 2019 IEEE Wireless Communications and Networking Conference (WCNC). Marrakesh 1–7 https://doi.org/10.1109/WCNC.2019.8885873
Zhang K, Yang Z, Basar T (2019) Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms. Cornell University. arXiv:1911.10635. Accessed 1 Jun 2020
Conflict of interests
The authors declare that they have no conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Kenneth C. W Goh, Raymond B. C Ng and Yoke-Keong Wong contributed equally.
About this article
Cite this article
Goh, K.C.W., Ng, R.B.C., Wong, YK. et al. Aerial filming with synchronized drones using reinforcement learning. Multimed Tools Appl 80, 18125–18150 (2021). https://doi.org/10.1007/s11042-020-10388-5
- Aerial filming
- Autonomous drones
- Swarm formation control
- Deep reinforcement learning