Skip to main content
SpringerLink
Log in

Aerial filming with synchronized drones using reinforcement learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Usage of multiple drones is necessary for aerial filming applications to ensure redundancy. However, this could inevitably contribute to higher risks of collisions, especially when the number of drones increases. Hence, this motivates us to explore various autonomous flight formation control methods that have the potential to enable multiple drones to effectively track a specific target at the same time. In this paper, we designed a model-free deep reinforcement learning algorithm, which is mainly based on the Deep Recurrent Q-Network concept, for the aforementioned purposes. The proposed algorithm was expanded into single and multi-agent types that enable multiple drones tracking while maintaining formation and preventing collision. The involved rewards in these approaches are two-dimensional in nature and are dependent on the communication system. Using Microsoft AirSim simulator, a virtual environment that includes four virtual drones was developed for experimental purposes. A comparison was made among various methods during the simulations, and the results concluded that the recurrent, single-agent model is the most effective method, being 33% more effective than its recurrent, multi-agent counterparts. The poor performance of the non-recurrent, single-agent baseline model also suggests that the recurrent elements in the network are essential to enable desirable multiple-drones flight.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. Second Iteration Video: https://www.youtube.com/watch?v=ZT0SEAQG_U0

  2. Third Iteration Video: https://www.youtube.com/watch?v=OdLcRP5R0MQ

  3. Fourth Iteration Video: https://www.youtube.com/watch?v=aweLkL8Xr18

  4. Codes can be found at the following link: https://github.com/raymondng76/IRS-Practice-Module-Dev

References

  1. Abadi M, Agarwal A, Barham P et al (2016) TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. Cornell University. arXiv:1603.04467. Accessed 1 Jun 2020

  2. Abughalieh KM, Sababha BH, Rawashdeh NA (2019) A video-based object detection and tracking system for weight sensitive UAVs. Multimed Tools Appl 78:9149–9167. https://doi.org/10.1007/s11042-018-6508-1

    Article  Google Scholar 

  3. Alam MS, Natesha BV, Ashwin TS, et al. (2019) UAV Based cost-effective real-time abnormal event detection using edge computing. Multimed Tools Appl 78:35119–35134. https://doi.org/10.1007/s11042-019-08067-1

    Article  Google Scholar 

  4. Becker-Ehmck P, Karl M, Peters J et al (2020) Learning to Fly via Deep Model-Based Reinforcement Learning. Cornell University. arXiv:2003.08876. Accessed 1 Jun 2020

  5. Bochkovskiy A, Wang CY, Liao HYM (2020) YOLOv4: Optimal Speed and Accuracy of Object Detection. Cornell University. arXiv:2004.10934. Accessed 1 Jun 2020

  6. Bonatti R, Ho C, Wang W et al (2019) Towards a Robust Aerial Cinematography Platform: Localizing and Tracking Moving Targets in Unstructured Environments. Cornell University. arXiv:1904.02319. Accessed 1 Jun 2020

  7. Bonatti R, Zhang Y, Choudhury S et al (2018) Autonomous drone cinematographer: Using artistic principles to create smooth, safe, occlusion-free trajectories for aerial filming. Cornell University. arXiv:1808.09563. Accessed 1 Jun 2020

  8. Chollet F et al (2019) Keras. https://keras.io. Accessed 1 Jun 2020

  9. Cunha R (2017) ICCV2017 Tutorial: Drone vision for cinematography - Drone Formation and Flight Control. MultiDrone. https://multidrone.eu/wp-content/uploads/2017/01/04_Drone_Formation_Flight_Control.pdf. Accessed 1 Jun 2020

  10. Epic Games (2020) Unreal Engine. https://www.unrealengine.com. Accessed 1 Jun 2020

  11. Esmukov K, Tygart A, López A et al (2018) Geocoding library for Python. GitHub repository. https://github.com/geopy/geopy. Accessed 1 Jun 2020

  12. French S (2018) Want to make six figures? Try being a drone pilot. MarketWatch. https://www.marketwatch.com/story/want-to-make-six-figures-try-being-a-drone-pilot-2018-08-10. Accessed 1 Jun 2020

  13. Galvane Q, Fleureau J, Tariolle FL, Guillotel P (2017) Automated Cinematography with Unmanned Aerial Vehicles. Cornell University. arXiv:1712.04353. Accessed 1 Jun 2020

  14. Galvane Q, Lino C, Christie M et al (2018) Directing cinematographic drones. ACM Trans Graph 37(3):1–18. https://doi.org/10.1145/3181975

    Article  Google Scholar 

  15. Hausknecht M, Stone P (2017) Deep recurrent Q-Learning for partially observable MDPs. Cornell University. arXiv:1507.06527. Accessed 1 Jun 2020

  16. Hong S (2019) Autonomous UAV Navigation without Collision using Visual Information in Airsim. GitHub repository. https://github.com/sunghoonhong/AirsimDRL. Accessed 1 Jun 2020

  17. Huang C, Gao F, Pan J, et al. (2018) ACT: An autonomous drone cinematography system for action scenes. 2018 IEEE international conference on robotics and automation (ICRA). Brisbane 7039–7046 https://doi.org/10.1109/ICRA.2018.8460703

  18. Huang C, Yang Z, Kong Y et al (2018) Through-the-lens Drone Filming. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid 4692–4699 https://doi.org/10.1109/IROS.2018.8594333

  19. Huynh NA (2017) Training and Detecting Objects with YOLO3. GitHub repository. https://github.com/experiencor/keras-yolo3. Accessed 1 Jun 2020

  20. James S, Freese M, Davison AJ (2019) PyRep: Bringing V-REP to Deep Robot Learning. Cornell University. arXiv:1906.11176. Accessed 1 Jun 2020

  21. James S, Ma Z, Arrojo DR, Davison AJ (2019) RLBench: The Robot Learning Benchmark & Learning Environment. Cornell University. arXiv:1909.12271. Accessed 1 Jun 2020

  22. Joubert N, Jane LE, Goldman DB, Berthouzoz F et al (2016) Towards a Drone Cinematographer: Guiding Quadrotor Cameras using Visual Composition Principles. Cornell University. arXiv:1610.01691. Accessed 1 Jun 2020

  23. Karney CFF (2013) Algorithms for geodesics. J Geodesy 87:43–55. https://doi.org/10.1007/s00190-012-0578-z

    Article  Google Scholar 

  24. Kostrikov I, Yarats D, Fergus R (2020) Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels. Cornell University. arXiv:2004.13649. Accessed 1 Jun 2020

  25. Krishnan S, Boroujerdian B, Fu W et al (2019) Air Learning: An AI Research Platform for Algorithm-Hardware Benchmarking of Autonomous Aerial Robots. Cornell University. arXiv:1906.00421. Accessed 1 Jun 2020

  26. Kwak J, Park JH, Sung Y (2019) Affective social big data generation algorithm for autonomous controls by CRNN-based end-to-end controls. Multimed Tools Appl 78:27175–27192. https://doi.org/10.1007/s11042-019-7703-4

    Article  Google Scholar 

  27. Liu F, Yang A (2019) Application of gcForest to visual tracking using UAV image sequences. Multimed Tools Appl 78:27933–27956. https://doi.org/10.1007/s11042-019-07864-y

    Article  Google Scholar 

  28. Liu H, Zhao W, Lewis L et al (2019) Attitude Synchronization for Multiple Quadrotors using Reinforcement Learning*. 2019 Chinese Control Conference (CCC). Guangzhou 2480–2483 https://doi.org/10.23919/ChiCC.2019.8865177

  29. Mademlis I, Nikolaidis N, Tefas A, et al. (2018) Autonomous unmanned aerial vehicles filming in dynamic unstructured outdoor environments. IEEE Signal Proc Mag 36(1):147–153. https://doi.org/10.1109/MSP.2018.2875190

    Article  Google Scholar 

  30. Microsoft (2019) Microsoft Drone Rescue. GitHub repository. https://github.com/microsoft/DroneRescue. Accessed 1 Jun 2020

  31. Microsoft (2020) Open source simulator for autonomous vehicles built on Unreal Engine / Unity, from Microsoft AI & Research. GitHub repository. https://github.com/microsoft/AirSim. Accessed 1 Jun 2020

  32. Nägeli T, Alonso-Mora J, Domahidi A et al (2017) Real-Time Motion planning for aerial videography with dynamic obstacle avoidance and viewpoint optimization. IEEE Robot Autom Lett 2(3):1696–1703. https://doi.org/10.1109/LRA.2017.2665693

    Article  Google Scholar 

  33. Nägeli T, Meier L, Domahidi A, et al. (2017) Real-time planning for automated multi-view drone cinematography. ACM Trans Graph 36 (4):1–10. https://doi.org/10.1145/3072959.3073712

    Article  Google Scholar 

  34. Nikolaidis N, Mademlis I, Raptopoulou C, Bull D (2017) ICCV2017 Tutorial: Drone vision for cinematography - Drone Cinematography. MultiDrone. https://multidrone.eu/wp-content/uploads/2017/01/06_Drone-Cinematography.pdf. Accessed 1 Jun 2020

  35. Passalis N, Tefas A (2019) Deep reinforcement learning for controlling frontal person close-up shooting. Neurocomputing 335:37–47. https://doi.org/10.1016/j.neucom.2019.01.046

    Article  Google Scholar 

  36. Redmon J, Farhadi A (2018) YOLOv3: An Incremental Improvement. Cornell University. arXiv:1804.02767. Accessed 1 Jun 2020

  37. Rising J (2015) Drones vs Helicopter - Part 1: How drones are changing the aerial video industry. Flight Evolved. https://flight-evolved.com/drone-vs-helicopter. Accessed 1 Jun 2020

  38. Sabetghadam B, Alcántara A, Capitán J et al (2019) Optimal trajectory planning for autonomous drone cinematography. 2019 european conference on mobile robots (ECMR). Prague 1–7 https://doi.org/10.1109/ECMR.2019.8870950

  39. Tefas A, Nousi P, Passalis N et al (2017) ICCV2017 Tutorial: Drone vision for cinematography - Deep Learning for Drone Vision in Cinematography. MultiDrone. https://multidrone.eu/wp-content/uploads/2017/01/05_Deep-Learning-for-Drone-Vision-in-Cinematography.pdf. Accessed 1 Jun 2020

  40. Torres-González A, Capitán J, Cunha R et al (2017) A multidrone approach for autonomous cinematography planning. In: In: Ollero A, Sanfeliu A, Montano L et al (eds) ROBOT 2017: Third iberian robotics conference. ROBOT 2017. Advances in intelligent systems and computing, vol 693. Springer, Cham, pp 337–349, DOI https://doi.org/10.1007/978-3-319-70833-1_28

  41. Tzutalin (2020) LabelImg is a graphical image annotation tool and label object bounding boxes in images. GitHub repository. https://github.com/tzutalin/labelImg. Accessed 1 Jun 2020

  42. Wang T, Qin R, Chen Y et al (2019) A reinforcement learning approach for UAV target searching and tracking. Multimed Tools Appl 78:4347–4364. https://doi.org/10.1007/s11042-018-5739-5

    Article  Google Scholar 

  43. Yang H, Xie K, Huang S, Huang H (2018) Uncut aerial video via a single sketch. Comput Graph Forum 37:191–199. https://doi.org/10.1111/cgf.13559

    Article  Google Scholar 

  44. Zanol R, Chiariotti F, Zanella A (2019) Drone mapping through multi-agent reinforcement learning. 2019 IEEE Wireless Communications and Networking Conference (WCNC). Marrakesh 1–7 https://doi.org/10.1109/WCNC.2019.8885873

  45. Zhang K, Yang Z, Basar T (2019) Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms. Cornell University. arXiv:1911.10635. Accessed 1 Jun 2020

Download references

Author information

Author notes

    Authors and Affiliations

    1. Institute of Systems Science, National University of Singapore, Singapore, Singapore

      Kenneth C. W Goh, Raymond B. C Ng, Yoke-Keong Wong, Nicholas J. H Ho & Matthew C. H Chua

    Authors
    1. Kenneth C. W Goh
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Raymond B. C Ng
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Yoke-Keong Wong
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Nicholas J. H Ho
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Matthew C. H Chua
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Nicholas J. H Ho.

    Ethics declarations

    Conflict of interests

    The authors declare that they have no conflict of interest.

    Additional information

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Kenneth C. W Goh, Raymond B. C Ng and Yoke-Keong Wong contributed equally.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Goh, K.C.W., Ng, R.B.C., Wong, YK. et al. Aerial filming with synchronized drones using reinforcement learning. Multimed Tools Appl 80, 18125–18150 (2021). https://doi.org/10.1007/s11042-020-10388-5

    Download citation

    • Received:

    • Revised:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s11042-020-10388-5

    Keywords

    Search

    Navigation