Reinforcement Learning and Trustworthy Autonomy

  • Jieliang LuoEmail author
  • Sam Green
  • Peter Feghali
  • George Legrady
  • Çetin Kaya Koç


Cyber-Physical Systems (CPS) possess physical and software interdependence and are typically designed by teams of mechanical, electrical, and software engineers. The interdisciplinary nature of CPS makes them difficult to design with safety guarantees. When autonomy is incorporated, design complexity and, especially, the difficulty of providing safety assurances are increased. Vision-based reinforcement learning is an increasingly popular family of machine learning algorithms that may be used to provide autonomy for CPS. Understanding how visual stimuli trigger various actions is critical for trustworthy autonomy. In this chapter we introduce reinforcement learning in the context of Microsoft’s AirSim drone simulator. Specifically, we guide the reader through the necessary steps for creating a drone simulation environment suitable for experimenting with vision-based reinforcement learning. We also explore how existing vision-oriented deep learning analysis methods may be applied toward safety verification in vision-based reinforcement learning applications.


  1. 1.
    R.N Charette, This car runs on code. IEEE Spectr. 46(3), 3 (2009)CrossRefGoogle Scholar
  2. 2.
    A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems (2012), pp. 1097–1105Google Scholar
  3. 3.
    R.J. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning, in Reinforcement Learning (Springer, Berlin, 1992), pp. 5–32zbMATHGoogle Scholar
  4. 4.
    S. Shah, D. Dey, C. Lovett, A. Kapoor, Airsim: high-fidelity visual and physical simulation for autonomous vehicles, in Field and Service Robotics (2017)Google Scholar
  5. 5.
    L. van der Maaten, G. Hinton, Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)zbMATHGoogle Scholar
  6. 6.
    K. Simonyan, A. Vedaldi, A. Zisserman, Deep inside convolutional networks: visualising image classification models and saliency maps. Preprint, arXiv:1312.6034 (2013)Google Scholar
  7. 7.
    C. Olah, A. Mordvintsev, L. Schubert, Feature visualization, in Distill (2017)Google Scholar
  8. 8.
    R.R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-CAM: visual explanations from deep networks via gradient-based localization, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 618–626Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Jieliang Luo
    • 1
    Email author
  • Sam Green
    • 1
  • Peter Feghali
    • 1
  • George Legrady
    • 1
  • Çetin Kaya Koç
    • 2
    • 3
    • 4
  1. 1.University of California Santa BarbaraSanta BarbaraUSA
  2. 2.İstinye UniversityIstanbulTurkey
  3. 3.Nanjing University of Aeronautics and AstronauticsNanjingChina
  4. 4.University of California Santa BarbaraSanta BarbaraUSA

Personalised recommendations