Abstract
Driven by the improvements in a variety of domains, autonomous driving is becoming a reality and today, industry aims at moving toward fully autonomous vehicles. High-tech chip manufacturers are designing high-performance and energy-efficient platforms in accordance with safety standard requirements. However, the software used to implement advanced functionalities in autonomous vehicles challenges real-time constraints on those platforms. Hence, there is a clear need for industry-level autonomous driving benchmarks to evaluate platforms and systems. In this paper, we propose ADBench, a benchmarking approach and benchmark suite for state-of-the-art autonomous driving platforms, in accordance with the key modules, structural design and functions of AD systems, building on several industry-level autonomous driving systems. The use of standard benchmarks facilitates the design, verification and validation process of autonomous systems.
Similar content being viewed by others
Notes
A bag is a file format in ROS for storing ROS message data. They are typically created by a tool like rosbag, which subscribes to one or more ROS topics, and stores the serialized message data in a file as it is received. These bag files can also be played back in ROS to the same topics they were recorded from, or even remapped to new topics.
References
Alcon M, Tabani H, Kosmidis L, Mezzetti E, Abella J, Cazorla FJ (2020) Timing of autonomous driving software: problem analysis and prospects for future solutions. IEEE Real-Time Embed Technol Appl Symp (RTAS) IEEE 2020:267–280
Alcon M, Tabani H, Abella J, Kosmidis L, Cazorla FJ (2020) En-route: on enabling resource usage testing for autonomous driving frameworks. In: Proceedings of the 35th annual ACM symposium on applied computing, pp 1953–1962
Alcaide S, Kosmidis L, Tabani H, Hernandez C, Abella J, Cazorla FJ (2018) Safety-related challenges and opportunities for GPUS in the automotive domain. IEEE Micro 38(6):46–55
ApolloAuto, Apollo 3.0 software architecture (2018). https://github.com/ApolloAuto/apollo/blob/master/docs/specs/Apollo_3.0_Software_Architecture.md
ApolloAuto, Perception (2018). https://github.com/ApolloAuto/apollo/blob/r3.0.0/docs/specs/perception_apollo_3.0.md
ApolloAuto (2018) 3D obstacle perception. https://github.com/ApolloAuto/apollo/blob/r3.0.0/docs/specs/3d_obstacle_perception.md
ApolloAuto (2018) Traffic light perception. https://github.com/ApolloAuto/apollo/blob/master/docs/specs/traffic_light.md
ApolloAuto (2018) Multi-sensor fusion localization. https://github.com/ApolloAuto/apollo/blob/r3.0.0/modules/localization/msf/README.md
ApolloAuto (2018) Planning. https://github.com/ApolloAuto/apollo/tree/r3.5.0/modules/planning/README.md
ApolloAuto (2018) Prediction. https://github.com/ApolloAuto/apollo/blob/r3.0.0/modules/prediction/README.md
Clint Whaley R, Aberdeen D, Brett M, Coult N, Castaldo T, Dittrich M, Gaudet D, Goto K, Horner J, Maguire C, Mattox T, Deitz H, Nguyen V, Strazdins P, Ruhe J, Soendergaard P, Staelin C (2018) Automatically Tuned Linear Algebra Software (ATLAS). http://math-atlas.sourceforge.net/
Xianyi Z, Qian W, Saar W, Chothia Z, Shaohu C, Wen L et al (2020) An optimized BLAS library (OpenBLAS). http://www.openblas.net/
Baidu Apollo, an open autonomous driving platform (2018). http://apollo.auto/
Blackford L et al (2002) An updated set of basic linear algebra subprograms (BLAS). ACM Trans Math Softw 28(2):135–151
Chetlur S et al., CUDNN: Efficient primitives for deep learning. ArXiv preprint arXiv:1410.0759
Caesar H, Bankiti V, Lang AH, Vora S, Liong VE, Xu Q, Krishnan A, Pan Y, Baldan G, Beijbom O. Nuscenes: a multimodal dataset for autonomous driving. ArXiv preprint arXiv:1903.11027
Corp T (2018) Tesla autopilot. https://www.tesla.com/autopilot
NVIDIA (2021) cuBLAS. http://docs.nvidia.com/cuda/cublas/
Demler M (2017) Xavier simplifies self-driving cars. In: Microprocessors Report, The Linly Group, June
Dolgov D, Thrun S, Montemerlo M, Diebel J (2010) Path planning for autonomous vehicles in unknown semi-structured environments. Int J Robot Res 29(5):485–501
EEMBC (2019) Introducing the EEMBC MLMark Benchmark. https://www.eembc.org/mlmark/
EEMBC (2019) The \({\rm ADASMark}^{\rm TM}\) Benchmark: A Performance Measurement and Optimization Tool for Automotive Companies Building Next-Generation Advanced Driver-Assistance Systems (ADAS). https://www.eembc.org/adasmark/
Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? The kitti vision benchmark suite. In: Conference on computer vision and pattern recognition (CVPR)
Intel (2020) Intel oneAPI Math Kernel Library: The fastest and most-used math library for Intel-based systems. https://software.intel.com/en-us/intel-mkl
Koopman P, Wagner M (2016) Challenges in autonomous vehicle testing and validation. SAE Int J Trans Saf 4:15–24. https://doi.org/10.4271/2016-01-0128
Merkel D (2014) Docker: lightweight Linux containers for consistent development and deployment. Linux J 2014(239):2
Paz D, Lai P-J, Chan N, Jiang Y, Christensen HI, Autonomous vehicle benchmarking using unbiased metrics. ArXiv preprint arXiv:2006.02518
Pujol R, Tabani H, Kosmidis L, Mezzetti E, Abella J, Cazorla FJ (2019) Generating and exploiting deep learning variants to increase heterogeneous resource utilization in the Nvidia Xavier. In: 31st euromicro conference on real-time systems (ECRTS 2019), Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik
Quigley M et al (2009) ROS: an open-source robot operating system. ICRA workshop on open source software, vol 3. Japan, Kobe, p 5
Redmon J, Farhadi A (2018) YOLOv3: An Incremental Improvement. CoRR. arXiv:1804.02767
Reddi VJ, Cheng C, Kanter D, Mattson P, Schmuelling G, Wu C-J, Anderson B, Breughe M, Charlebois M, Chou W, et al (2020) Mlperf inference benchmark. In: 2020 ACM/IEEE 47th annual international symposium on computer architecture (ISCA). IEEE, pp 446–459
Rong G, Shin BH, Tabatabaee H, Lu Q, Lemke S, Možeiko M, Boise E, Uhm G, Gerow M, Mehta S, et al. Lgsvl simulator: a high fidelity simulator for autonomous driving. ArXiv preprint arXiv:2005.03778
Tabani H, Kosmidis L, Abella J, Cazorla FJ, Bernat G (2019) Assessing the adherence of an industrial autonomous driving framework to iso 26262 software guidelines. In: Proceedings of the 56th annual design automation conference 2019. ACM, p 9
Tabani H, Mazzocchetti F, Benedicte P, Abella J, Cazorla FJ (2021) Performance analysis and optimization opportunities for Nvidia automotive GPUS. J Parallel Distrib Comput 152:21–32
Tabani H, Pujol R, Abella J, Cazorla FJ (2020) A cross-layer review of deep learning frameworks to ease their optimization and reuse. In: IEEE 23rd international symposium on real-time distributed computing (ISORC). IEEE, pp 144–145
NVIDIA (2021) TensorRT: A platform for high-performance deep learning inference. https://developer.nvidia.com/tensorrt
The autoware foundation, autoware: an open autonomous driving platform https://github.com/CPFL/Autoware/ (2016)
NVIDIA (2018) Self-driving Safety Report. https://www.nvidia.com/en-us/self-driving-cars/safety-report/
Wan G, Yang X, Cai R, Li H, Zhou Y, Wang H, Song S (2018) Robust and precise vehicle localization based on multi-sensor fusion in diverse city scenes. In: IEEE international conference on robotics and automation (ICRA), pp 4670–4677
Acknowledgements
This work has been partially supported by the Spanish Ministry of Economy and Competitiveness (MINECO) under Grant TIN2015-65316-P, the SuPerCom European Research Council (ERC) project under the European Union’s Horizon 2020 research and innovation programme (Grant Agreement No. 772773), and the HiPEAC Network of Excellence.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Tabani, H., Pujol, R., Alcon, M. et al. ADBench: benchmarking autonomous driving systems. Computing 104, 481–502 (2022). https://doi.org/10.1007/s00607-021-00975-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00607-021-00975-1