Safe At Any Speed: A Simulation-Based Test Harness for Autonomous Vehicles

  • Houssam Abbas
  • Matthew O’Kelly
  • Alena RodionovaEmail author
  • Rahul Mangharam
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11267)


The testing of Autonomous Vehicles (AVs) requires driving the AV billions of miles under varied scenarios in order to find bugs, accidents and otherwise inappropriate behavior. Because driving a real AV that many miles is too slow and costly, this motivates the use of sophisticated ‘world simulators’, which present the AV’s perception pipeline with realistic input scenes, and present the AV’s control stack with realistic traffic and physics to which to react. Thus the simulator is a crucial piece of any CAD toolchain for AV testing. In this work, we build a test harness for driving an arbitrary AV’s code in a simulated world. We demonstrate this harness by using the game Grand Theft Auto V (GTA) as world simulator for AV testing. Namely, our AV code, for both perception and control, interacts in real-time with the game engine to drive our AV in the GTA world, and we search for weather conditions and AV operating conditions that lead to dangerous situations. This goes beyond the current state-of-the-art where AVs are tested under ideal weather conditions, and lays the ground work for a more comprehensive testing effort. We also propose and demonstrate necessary analyses to validate the simulation results relative to the real world. The results of such analyses allow the designers and verification engineers to weigh the results of simulation-based testing.


  1. 1.
    Godil, A.A., et al.: Performance metrics for evaluating object and human detection and tracking systems, nIST Interagency/Internal Report (NISTIR) - 7972, July 2014Google Scholar
  2. 2.
    Alexander Blade, A.S.D.: Script hook, September 2017.
  3. 3.
    Baidu: Apollo platform, September 2017.
  4. 4.
    Chen, C., Seff, A., Kornhauser, A., Xiao, J.: Deepdriving: learning affordance for direct perception in autonomous driving. In: International Conference on Computer Vision (2015)Google Scholar
  5. 5.
    Chib, S., Greenberg, E.: Understanding the Metropolis-Hastings algorithm. Am. Stat. 49(4), 327–335 (1995)Google Scholar
  6. 6.
    Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the kitti dataset. Int. J. Robot. Res. 32(11), 1231–1237 (2013)CrossRefGoogle Scholar
  7. 7.
    Johnson-Roberson, M., Barto, C., Mehta, R., Sridhar, S.N., Rosaen, K., Vasudevan, R.: Driving in the matrix: can virtual worlds replace human-generated annotations for real world tasks? In: ICRA, May 2017Google Scholar
  8. 8.
    Kato, S.: Autoware, September 2017.
  9. 9.
    Kundu, D.: Subjective and objective quality evaluation of synthetic and high dynamic range images, Ph.D. Dissertation, May 2016Google Scholar
  10. 10.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  11. 11.
    Redmon, J., Farhadi, A.: Yolo9000: better, faster, stronger.
  12. 12.
    Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: ground truth from computer games. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 102–118. Springer, Cham (2016). Scholar
  13. 13.
    Udacity, A.R.: Udacity self-driving car dataset, p. 2 (2017).
  14. 14.
    Winkler, S.: Analysis of public image and video databases for quality assessment. IEEE J. Sel. Top. Signal Process. 6(6), 616–625 (2012)CrossRefGoogle Scholar
  15. 15.
    Zhao, D., Peng, H.: From the lab to the street, m-City White Paper, May 2017Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Houssam Abbas
    • 1
  • Matthew O’Kelly
    • 1
  • Alena Rodionova
    • 1
    Email author
  • Rahul Mangharam
    • 1
  1. 1.University of PennsylvaniaPhiladelphiaUSA

Personalised recommendations