Abstract
Autonomous vehicles must pass effective standard tests to verify their reliability and safety. Accordingly, it is very important to establish a complete scientific test and evaluation system for autonomous vehicles. A comprehensive framework incorporating the design of test scenarios, selection of evaluation indexes, and establishment of an evaluation system is proposed in this paper. The aims of the system are to obtain an objective and quantitative score regarding the intelligence of autonomous vehicles, and to form an automated process in the future development. The proposed framework is built on a simulation platform to ensure the feasibility of the design and implementation of the test scenarios. The design principle for the test scenarios is also presented. To reduce subjective influences, the proposed framework selects objective indexes from four aspects: safety, comfort, driving performance, and standard regulations. The order relation analysis method is adopted to formulate the index weights, and fuzzy comprehensive evaluation is used to quantify the scores. Finally, a numerical example is provided to visually demonstrate the evaluation results for the autonomous vehicles scored by the proposed framework.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
HUANG H, ZHENG X, YANG Y, et al. An integrated architecture for intelligence evaluation of automated vehicles [J]. Accident Analysis & Prevention, 2020, 145: 105681.
HUANG H M, PAVEK K, NOVAK B, et al. A framework for autonomy levels for unmanned systems (ALFUS) [C]//AUVSI’s Unmanned Systems North America. Baltimore: NIST, 2005: 1–9.
DARPA. Urban challenge rules [DB/OL]. (2007-10-27) [2020-11-20]. https://www.grandchallenge.org/grand-challenge/docs/Urban_Challenge_Rules_102707.pdf.
SUN Y, CHEN H. Research on test and evaluation of unmanned ground vehicles [J]. Acta Armamentarii, 2015, 36(6): 978–986 (in Chinese).
SUN Y, XIONG G, CHEN H. Evaluation of the intelligent behaviors of unmanned ground vehicles based on fuzzy-EAHP scheme [J]. Automotive Engineering, 2014, 36(1): 22–27 (in Chinese).
SUN Y. Quantitative evaluation of intelligence levels for unmanned ground vehicles [D]. Beijing: Beijing Institute of Technology, 2014 (in Chinese).
XIONG G M, GAO L, WU S B, et al. Intelligent behaviors and test and evaluation for unmanned ground vehicles [M]. Beijing: Beijing Institute of Technology Press, 2015 (in Chinese).
SON T D, BHAVE A, VAN DER AUWERAER H. Simulation-based testing framework for autonomous driving development [C]//2019 IEEE International Conference on Mechatronics (ICM). Ilmenau: IEEE, 2019: 576–583.
WANG G, DENG W, ZHANG S, et al. A comprehensive testing and evaluation approach for autonomous vehicles [J]. SAE Technical Paper, 2018: 2018-01-0124.
WENG B, RAO S J, DEOSTHALE E, et al. Model predictive instantaneous safety metric for evaluation of automated driving systems [C]//IEEE Intelligent Vehicles Symposium (IV). Las Vegas: IEEE, 2020: 1899–1906.
LI L, HUANG W, LIU Y, et al. Intelligence testing for autonomous vehicles: A new approach [J]. IEEE Transactions on Intelligent Vehicles, 2016, 1 (2): 158–166.
FENG S, FENG Y, YU C, et al. Testing scenario library generation for connected and automated vehicles, part I: Methodology [J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(3): 1573–1582.
FENG S, FENG Y, SUN H, et al. Testing scenario library generation for connected and automated vehicles, part II: case studies [J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(9): 5635–5647.
SAE International. Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles: J3016 [R]. Warrendale: SAE, 2018.
YU Z, XING X, CHEN J. Review on automated vehicle testing technology and its application [J]. Journal of Tongji University (Natural Science), 2019, 47(4): 540–547 (in Chinese).
PEGASUS. PEGASUS joint project [DB/OL]. (2019-05-14) [2020-11-20]. http://www.pegasuspro-jekt.de/en/.
MENZEL T, BAGSCHIK G, MAURER M. Scenarios for development, test and validation of automated vehicles [C]//2018 IEEE Intelligent Vehicles Symposium (IV). Changshu: IEEE, 2018: 1821–1827.
WATANABE H, TOBISCH L, ROST J, et al. Scenario mining for development of predictive safety functions [C]//2019 IEEE International Conference on Vehicular Electronics and Safety (ICVES). Cairo: IEEE, 2019: 1–7.
HighD Dataset. HighD dataset [DB/OL]. [2020-11-20]. https://www.highd-dataset.com/.
DATA.GOV. Safety pilot model deployment data [DB/OL]. (2020-08-21) [2020-11-20]. https://catalog.data.gov/dataset/safety-pilot-model-deployment-data.
UMTRI. Safety pilot model deployment. [DB/OL]. [2020-11-20]. http://safetypilot.umtri.umich.edu/.
RAHMAN M S, ABDEL-ATY M, LEE J, et al. Safety benefits of arterials’ crash risk under connected and automated vehicles [J]. Transportation Research Part C: Emerging Technologies, 2019, 100: 354–371.
CHEN J, LI R, XING X, et al. Survey on intelligence evaluation of autonomous vehicles [J]. Journal of Tongji University (Natural Science), 2019, 47(12): 1785–1790 (in Chinese).
ZHAO Y N, MENG K W, GAO L. The entropy-cost function evaluation method for unmanned ground vehicles [J]. Mathematical Problems in Engineering, 2015, 2015:1-6.
HAYWARD J C. Near-miss determination through use of a scale of danger [C]//51st Annual Meeting of the Highway Research Board. Washington, DC: Highway Research Board, 1972: 24–34.
Standardization Administration. Intelligent transportation systems: Forward vehicle collision warning systems: Performance requirements and test procedures: GB/T 33577-2017 [S]. Beijing: Standards Press of China, 2017 (in Chinese).
MINDERHOUD M M, BOVY P H L. Extended time-to-collision measures for road traffic safety assessment [J]. Accident Analysis & Prevention, 2001, 33(1): 89–97.
BAGDADI O, VÁRHELYI A. Jerky driving: An indicator of accident proneness? [J]. Accident Analysis & Prevention, 2011, 43(4): 1359–1363.
BELLEM H, SCHOENENBERG T, KREMS J F, et al. Objective metrics of comfort: developing a driving style for highly automated vehicles [J]. Transportation Research Part F: Traffic Psychology and Behaviour, 2016, 41: 45–54.
GUO Y. Comprehensive evaluation theory and method [M]. Beijing: Science Press, 2002 (in Chinese).
XIA Q, LIU K, HUANG L, et al. Study of comprehensive evaluation for L2 automated vehicles on field test [C]//2019 3rd Conference on Vehicle Control and Intelligence (CVCI). Hefei: IEEE, 2019: 1–6.
WANG X H, LI T J, DING L L. Evaluation theory and technology of complex large-scale systems [M]. Jinan: Shandong University Press, 2010 (in Chinese).
Author information
Authors and Affiliations
Corresponding author
Additional information
Foundation item
the National Natural Science Foundation of China (No. 61873167), and the Automotive Industry Science and Technology Development Foundation of Shanghai (No. 1904)
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Su, Y., Wang, L. Integrated Framework for Test and Evaluation of Autonomous Vehicles. J. Shanghai Jiaotong Univ. (Sci.) 26, 699–712 (2021). https://doi.org/10.1007/s12204-021-2360-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12204-021-2360-y