Efficient Statistical Validation of Autonomous Driving Systems

  • Handi Yu
  • Weijing Shi
  • Mohamed Baker Alawieh
  • Changhao Yan
  • Xuan Zeng
  • Xin LiEmail author
  • Huafeng Yu
Part of the Unmanned System Technologies book series (UST)


Today’s automotive vehicles are often equipped with powerful data processing systems for driver assistance and/or autonomous driving. To meet the rigorous safety standard, ensuring extremely small failure probability over all possible operation conditions is one of the critical tasks for an autonomous driving system. In this chapter, we describe a novel methodology to validate vision-based autonomous driving systems over different circuit corners with consideration of temperature variation and circuit aging. Our approach seamlessly integrates the image data recorded under nominal conditions with comprehensive statistical circuit models to synthetically generate the critical corner cases for which an autonomous driving system is likely to fail. As such, a given automotive system can be robustly validated for these worst-case scenarios that cannot be easily captured by physical experiments. To efficiently estimate the rare failure rate of an autonomous system, we further propose a novel Subset Sampling (SUS) algorithm. In particular, a Markov Chain Monte Carlo algorithm based on graph mapping is developed to accurately estimate the rare failure rate with a minimal amount of test data, thereby minimizing the validation cost. Our numerical experiments show that SUS achieves 15.2× runtime speed-up over the conventional brute-force Monte Carlo method.



This research work has been supported in part by Toyota InfoTechnology Center, Cadence Design Systems, National Key Research and Development Program of China 2016YFB0201304, and National Natural Science Foundation of China (NSFC) research project 61376040, 61574046 and 61674042.


  1. 1.
    A. Lindgren, F. Chen, in State of the art analysis: an overview of advanced driver assistance systems (ADAS) and possible human factors issues. Human Factors and Economics Aspects on Safety (2006), pp. 38–50Google Scholar
  2. 2.
    M. Aeberhard et al., Experience, results and lessons learned from automated driving on Germany’s highways. IEEE Intell. Transp. Syst. Mag. 7(1), 42–57 (2015)CrossRefGoogle Scholar
  3. 3.
    J. Ziegler et al., Making bertha drive—an autonomous journey on a historic route. IEEE Intell. Transp. Syst. Mag. 6(2), 8–20 (2014)CrossRefGoogle Scholar
  4. 4.
    M. Fu, Y. Huang, in A survey of traffic sign recognition. ICWAPR (2010), pp. 119–124Google Scholar
  5. 5.
    N. Kalra, S.M. Paddock, Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability. Transp. Res. A Pol. Pract. 94, 182–193 (2016)CrossRefGoogle Scholar
  6. 6.
    D. Hospach et al., in Simulation of falling rain for robustness testing of video-based surround sensing systems. DATE (2016), pp. 233–236Google Scholar
  7. 7.
    R. Gallen et al., Nighttime visibility analysis and estimation method in the presence of dense fog. IEEE Trans. Intell. Transp. Syst. 16(1), 310–320 (2015)CrossRefGoogle Scholar
  8. 8.
    M. Nentwig, M. Stamminger, in Hardware-in-the-loop testing of computer vision based driver assistance systems. IEEE Intelligent Vehicles Symposium (2011), pp. 339–344Google Scholar
  9. 9.
    M. Nentwig et al., in Concerning the applicability of computer graphics for the evaluation of image processing algorithms. ICVES (2012), pp. 205–210Google Scholar
  10. 10.
    A. Broggi et al., Extensive tests of autonomous driving technologies. IEEE Trans. Intell. Transp. Syst. 14(3), 1403–1415 (2013)CrossRefGoogle Scholar
  11. 11.
    E. Roth, T. Calapoglu, in Advanced driver assistance system testing using OptiX. NVIDIA GTC (2012)Google Scholar
  12. 12.
    L. Raffaëlli et al., in Facing ADAS validation complexity with usage oriented testing. ERTS (2016)Google Scholar
  13. 13.
    M. Lankamp et al., MGSim-simulation tools for multi-core processor architectures, arXiv (2013)Google Scholar
  14. 14.
    S. Au, J. Beck, Estimation of small failure probabilities in high dimensions by subset simulation. Probabilist. Eng. Mech. 16(4), 263–277 (2001)CrossRefGoogle Scholar
  15. 15.
    S. Sun, X. Li, in Fast statistical analysis of rare circuit failure events via subset simulation in high-dimensional variation space. ICCAD (2014), pp. 324–331Google Scholar
  16. 16.
    T. Kuroda, Essential Principles of Image Sensors (CRC Press, Boca Raton, 2014)CrossRefGoogle Scholar
  17. 17.
    G.C. Holst, CCD Arrays, Cameras, and Displays (JCD Publication, Winter Park, 1998)Google Scholar
  18. 18.
    R. Ramanath et al., Color image processing pipeline. IEEE Signal Process. Mag. 22(1), 34–43 (2005)CrossRefGoogle Scholar
  19. 19.
    C.M. Kyung, Theory and Applications of Smart Cameras (Springer, Dordrecht, 2016)CrossRefGoogle Scholar
  20. 20.
    C. Bishop, Pattern Recognition and Machine Learning (Prentice Hall, Upper Saddle River, 2007)zbMATHGoogle Scholar
  21. 21.
    P. Viola, M. Jones, in Rapid object detection using a boosted cascade of simple features. CVPR (2001), pp. I511–I518Google Scholar
  22. 22.
    W. Shi et al., in Efficient statistical validation of machine learning systems for autonomous driving. ICCAD (2016)Google Scholar
  23. 23.
    X. Xi et al., in BSIM4.3.0 MOSFET Model—User’s Manual, vol. 2, no. 1 (University of California, Berkley, 2003), pp. 351–354Google Scholar
  24. 24.
    P. Seitz, A.J. Theuwissen, Single-Photon Imaging (Springer, Berlin, 2011)CrossRefGoogle Scholar
  25. 25.
    G.G. Nampoothiri et al., in Ageing effects on image sensors due to terrestrial cosmic radiation. IS&T/SPIE Electronic Imaging, vol. 7875 (2011)Google Scholar
  26. 26.
    G.H. Chapman et al., in Empirical formula for rates of hot pixel defects based on pixel size, sensor area and ISO. IS&T/SPIE Electronic Imaging, vol. 8659, (2013)Google Scholar
  27. 27.
    H.T. Sencar, N. Memon, Digital Image Forensics (Springer, New York, 2013)CrossRefGoogle Scholar
  28. 28.
    W. Shi et al., An FPGA-based hardware accelerator for traffic sign detection. IEEE Trans. Very Large Scale Integr. Syst. 25(4), 1362–1372 (2017)CrossRefGoogle Scholar
  29. 29.
    P. Dollar, Piotr’s computer vision MATLAB toolbox (PMT),
  30. 30.
    R. Timof et al., in Traffic sign recognition—how far are we from the solution? IJCNN (2013)Google Scholar
  31. 31.
    J. Stallkamp et al., Man vs. computer: benchmarking machine learning algorithms for traffic sign recognition. Neural Netw. 32, 323–332 (2012)CrossRefGoogle Scholar
  32. 32.
    S. Houben et al., in Detection of traffic signs in real-world images: The German traffic sign detection benchmark. IJCNN (2014)Google Scholar
  33. 33.
    S. Sahni, T. Gonzalez, P-complete approximation problems. J. ACM 23(3), 555–565 (1976)MathSciNetCrossRefGoogle Scholar
  34. 34.
    D. Harel, Y. Koren, in Graph Drawing by High Dimensional Embedding. International Symposium on Graph Drawing (Springer, Berlin, 2002). pp. 207–219CrossRefGoogle Scholar
  35. 35.
    S. Banerjee, A. Roy, Linear Algebra and Matrix Analysis for Statistics (Chapman and Hall/CRC, Hoboken, 2014)CrossRefGoogle Scholar
  36. 36.
    I. Borg, Modern Multidimensional Scaling: Theory and Applications (Springer, New York, 2005)zbMATHGoogle Scholar
  37. 37.
    P. Gopalakrishnan, X. Li, L. Pileggi, in Architecture-aware FPGA placement using metric embedding. DAC (2006), pp. 460–465Google Scholar
  38. 38.
    J. Bentley, Multidimensional binary search trees used for associative searching. Commun. ACM 18(9), 509–517 (1975)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Handi Yu
    • 1
  • Weijing Shi
    • 2
  • Mohamed Baker Alawieh
    • 2
  • Changhao Yan
    • 3
  • Xuan Zeng
    • 3
  • Xin Li
    • 1
    • 4
    Email author
  • Huafeng Yu
    • 5
  1. 1.Duke UniversityDurhamUSA
  2. 2.Carnegie Mellon UniversityPittsburghUSA
  3. 3.Fudan UniversityShanghaiP. R. China
  4. 4.Duke Kunshan UniversityKunshanP.R. China
  5. 5.Boeing Research and TechnologyHuntsvilleUSA

Personalised recommendations