Advertisement

Autonomous Drone Cinematographer: Using Artistic Principles to Create Smooth, Safe, Occlusion-Free Trajectories for Aerial Filming

Conference paper
Part of the Springer Proceedings in Advanced Robotics book series (SPAR, volume 11)

Abstract

Autonomous aerial cinematography has the potential to enable automatic capture of aesthetically pleasing videos without requiring human intervention, empowering individuals with the capability of high-end film studios. Current approaches either only handle off-line trajectory generation, or offer strategies that reason over short time horizons and simplistic representations for obstacles, which result in jerky movement and low real-life applicability. In this work we develop a method for aerial filming that is able to trade off shot smoothness, occlusion, and cinematography guidelines in a principled manner, even under noisy actor predictions. We present a novel algorithm for real-time covariant gradient descent that we use to efficiently find the desired trajectories by optimizing a set of cost functions. Experimental results show that our approach creates attractive shots, avoiding obstacles and occlusion 65 times over 1.25 h of flight time, re-planning at 5 Hz with a 10 s time horizon. We robustly film human actors, cars and bicycles performing different motion among obstacles, using various shot types.

Notes

Acknowledgements

We thank Lentin Joseph, Aayush Ahuja, Delong Zhu, and Greg Armstrong for the assistance in field experiments and robot construction. Research presented in this paper was funded by Yamaha Motor Co., Ltd.

Supplementary material

489953_1_En_11_MOESM1_ESM.mp4 (33.1 mb)
Supplementary material 1 (mp4 33922 KB)

References

  1. 1.
    Arijon, D.: Grammar of the Film Language (1976)Google Scholar
  2. 2.
    Barry, A.J., Florence, P.R., Tedrake, R.: High-speed autonomous obstacle avoidance with pushbroom stereo. J. Field Robot. 35(1), 52–68 (2018)CrossRefGoogle Scholar
  3. 3.
    Bowen, C.J., Thompson, R.: Grammar of the Shot. Taylor & Francis (2013)Google Scholar
  4. 4.
    Christie, M., Olivier, P., Normand, J.-M.: Camera control in computer graphics. In: Computer Graphics Forum, vol. 27, pp. 2197–2218. Wiley (2008)Google Scholar
  5. 5.
    Drucker, S.M., Zeltzer, D.: Intelligent camera control in a virtual environment. In: Graphics Interface, pp. 190–190. Citeseer (1994)Google Scholar
  6. 6.
    Galvane, Q., Fleureau, J., Tariolle, F.-L., Guillotel, P.: Automated cinematography with unmanned aerial vehicles. arXiv preprint arXiv:1712.04353 (2017)
  7. 7.
    Galvane, Q., Lino, C., Christie, M., Fleureau, J., Servant, F., Guillotel, P.: Directing cinematographic drones. arXiv preprint arXiv:1712.04216 (2017)
  8. 8.
    Gebhardt, C., Hepp, B., Nägeli, T., Stevšić, S., Hilliges, O.: Airways: optimization-based planning of quadrotor trajectories according to high-level user goals. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 2508–2519. ACM (2016)Google Scholar
  9. 9.
    Gebhardt, C., Stevsic, S., Hilliges, O.: Optimizing for aesthetically pleasing quadrotor camera motion (2018)Google Scholar
  10. 10.
    Gleicher, M., Witkin, A.: Through-the-lens camera control. In: ACM SIGGRAPH Computer Graphics, vol. 26, pp. 331–340. ACM (1992)Google Scholar
  11. 11.
    Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 583–596 (2015)CrossRefGoogle Scholar
  12. 12.
    Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
  13. 13.
    Joubert, N., Roberts, M., Truong, A., Berthouzoz, F., Hanrahan, P.: An interactive tool for designing quadrotor camera shots. ACM Trans. Graph. (TOG) 34(6), 238 (2015)CrossRefGoogle Scholar
  14. 14.
    Joubert, N., Goldman, D.B., Berthouzoz, F., Roberts, M., Landay, J.A., Hanrahan, P., et al.: Towards a drone cinematographer: guiding quadrotor cameras using visual composition principles. arXiv preprint arXiv:1610.01691 (2016)
  15. 15.
    Lan, Z., Shridhar, M., Hsu, D., Zhao, S.: Xpose: reinventing user interaction with flying cameras. In: Robotics Science and Systems (2017)Google Scholar
  16. 16.
    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: European Conference on Computer Vision, pp. 740–755. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  17. 17.
    Lino, C., Christie, M.: Intuitive and efficient camera control with the toric space. ACM Trans. Graph. (TOG) 34(4), 82 (2015)CrossRefGoogle Scholar
  18. 18.
    Lino, C., Christie, M., Ranon, R., Bares, W.: The director’s lens: an intelligent assistant for virtual cinematography. In: Proceedings of the 19th ACM International Conference on Multimedia, pp. 323–332. ACM (2011)Google Scholar
  19. 19.
    Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y. and Berg, A.C.: SSD: single shot multibox detector. In: European Conference on Computer Vision, pp. 21–37. Springer, Heidelberg (2016)CrossRefGoogle Scholar
  20. 20.
    Mellinger, D., Kumar, V.: Minimum snap trajectory generation and control for quadrotors. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 2520–2525. IEEE (2011)Google Scholar
  21. 21.
    Mohta, K., Watterson, M., Mulgaonkar, Y., Liu, S., Chao, Q., Makineni, A., Saulnier, K., Sun, K., Zhu, A., Delmerico, J., et al.: Fast, autonomous flight in GPS-denied and cluttered environments. J. Field Robot. 35(1), 101–120 (2018)CrossRefGoogle Scholar
  22. 22.
    Nägeli, T., Meier, L., Domahidi, A., Alonso-Mora, J., Hilliges, O.: Real-time planning for automated multi-view drone cinematography. ACM Trans. Graph. (TOG) 36(4), 132 (2017)CrossRefGoogle Scholar
  23. 23.
    Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohli, P., Shotton, J., Hodges, S., Fitzgibbon, A.W.: Kinectfusion: real-time dense surface mapping and tracking. In: 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 127–136. IEEE (2011)Google Scholar
  24. 24.
    Ratliff, N., Zucker, M., Bagnell, J.A., Srinivasa, S.: CHOMP: gradient optimization techniques for efficient motion planning. In: IEEE International Conference on Robotics and Automation, ICRA 2009, pp. 489–494. IEEE (2009)Google Scholar
  25. 25.
    Ratliff, N.D., Silver, D., Bagnell, J.A.: Learning to search: functional gradient techniques for imitation learning. Auton. Robot. 27(1), 25–53 (2009)CrossRefGoogle Scholar
  26. 26.
    Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. arXiv preprint (2017)Google Scholar
  27. 27.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)Google Scholar
  28. 28.
    Roberts, M., Hanrahan, P.: Generating dynamically feasible trajectories for quadrotor cameras. ACM Trans. Graph. (TOG) 35(4), 61 (2016)CrossRefGoogle Scholar
  29. 29.
    Schulman, J., Ho, J., Lee, A.X., Awwal, I., Bradlow, H., Abbeel, P.: Finding locally optimal, collision-free trajectories with sequential convex optimization. In: Robotics: Science and Systems, vol. 9, pp. 1–10. Citeseer (2013)Google Scholar
  30. 30.
    Shah, S., Dey, D., Lovett, C., Kapoor, A.: AirSim: high-fidelity visual and physical simulation for autonomous vehicles (2017). https://arxiv.org/abs/1705.05065Google Scholar
  31. 31.
    Shim, D.H., Kim, H.J., Sastry, S.: Decentralized nonlinear model predictive control of multiple flying robots. In: Proceedings of 42nd IEEE Conference on Decision and control, vol. 4, pp. 3621–3626. IEEE (2003)Google Scholar
  32. 32.
    Turpin, M., Michael, N., Kumar, V.: Trajectory design and control for aggressive formation flight with quadrotors. Auton. Robot. 33(1–2), 143–156 (2012)CrossRefGoogle Scholar
  33. 33.
    Xie, K., Yang, H., Huang, S., Lischinski, D., Christie, M., Kai, X., Gong, M., Cohen-Or, D., Huang, H.: Creating and chaining camera moves for quadrotor videography. ACM Trans. Graph. 37, 14 (2018)Google Scholar
  34. 34.
    Zucker, M., Ratliff, N., Dragan, A.D., Pivtoraiko, M., Klingensmith, M., Dellin, C.M., Bagnell, J.A., Srinivasa, S.S.: CHOMP: covariant Hamiltonian optimization for motion planning. Int. J. Robot. Res. 32(9–10), 1164–1193 (2013)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Robotics InstituteCarnegie Mellon UniversityPittsburghUSA
  2. 2.Advanced Technology Research DivisionYamaha Motor Co., Ltd.ShizuokaJapan

Personalised recommendations