Skip to main content
SpringerLink
Log in

Novel Test Scenario Generation Technology for Performance Evaluation of Automated Vehicle

  • Published:
International Journal of Automotive Technology Aims and scope Submit manuscript

An Erratum to this article was published on 04 November 2023

This article has been updated

Abstract

As one of the critical technologies for performance evaluation of automated vehicles, the test scenario generation has been widespread concerned. In this paper, we propose a novel test scenario generation technology based on optimized Latin Hypercube Sampling (OLHS) and Test Matrix method (TM), named HIS-MPSO, which is efficient to generate the test scenario that consider the complexity, coverage, and potential relationships of factors. Based on naturalistic driving data, numerous car-following scenarios are generated by HIS-MPSO. Then, an adaptive cruise control system (ACC) are evaluated in terms of the tracking errors, comfort, and safety using the generated scenarios. Results show that compared with other existing OLHS algorithms, the HIS-MPSO can better restore the relationships among test factors existed in realistic traffic scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Change history

Abbreviations

ACC:

adaptive cruise control system

AV:

automated vehicle

CDF:

cumulative distribution function

Cholesky:

COLHS based on cholesky decomposition

COA:

COLHS based on combinatorial optimization

COLHS:

optimized latin hypercube sampling algorithm for correlation control

CPDF:

conditional probability density function

Corr-errors:

correlation relationship and correlation degree errors between LSTM and OSTM

EPI:

equiprobable interval

GA:

COA based on genetic algorithm

GEPI:

EPI for Gibbs sampling

HIS:

heuristic initialization strategy

HIS-MPSO:

test scenario generation technology proposed in this paper, which is a kind of optimized latin hypercube sampling algorithm

HLSTM:

LSTM generated by HIS-MPSO

LEPI:

EPI for latin hypercube sampling

LHS:

traditional latin hypercube sampling algorithm

LSTM:

scenario test matrix generated by OLHS

LV:

lead vehicle in car-following scenario

MPSO:

modified particle swarm optimization algorithm

OSTM:

original scenario test matrix extracted directly from the naturalistic driving

OLHS:

optimized latin hypercube sampling algorithm

RV:

rear vehicle in car-following scenario

SOLHS:

optimized latin hypercube sampling algorithm for space-filling

References

  • Auckland, R. A., Manning, W. J., Carsten, O. M. J. and Jamson, A. H. (2008). Advanced driver assistance systems: Objective and subjective performance evaluation. Vehicle System Dynamics 46, S1, 883–897.

    Article  Google Scholar 

  • Chen, R. B., Hsieh, D. N., Hung, Y. and Wang, W. (2013). Optimizing Latin hypercube designs by particle swarm. Statistics and Computing 23, 5, 663–676.

    Article  MathSciNet  MATH  Google Scholar 

  • Chen, Y., Wen, J. and Cheng, S. (2012). Probabilistic load flow method based on Nataf transformation and Latin hypercube sampling. IEEE Trans. Sustainable Energy 4, 2, 294–301.

    Article  Google Scholar 

  • Damblin, G., Couplet, M. and Iooss, B. (2013). Numerical studies of space-filling designs: Optimization of Latin Hypercube Samples and subprojection properties. Journal of Simulation 7, 4, 276–289.

    Article  Google Scholar 

  • Emirler, M. T., Güvenç, L. and Güvenç, B. A. (2018). Design and evaluation of robust cooperative adaptive cruise control systems in parameter space. Int. J. Automotive Technology 19, 2, 359–367.

    Article  Google Scholar 

  • Fremont, D. J., Kim, E., Pant, Y. V., Seshia, S. A., Acharya, A., Bruso, X., Wells, P., Lemke, S., Lu, Q. and Mehta, S. (2020). Formal scenario-based testing of autonomous vehicles: From simulation to the real world. IEEE 23rd Int. Conf. Intelligent Transportation Systems (ITSC), Rhodes, Greece.

  • Duan, J., Gao, F. and He, Y. (2022). Test scenario generation and optimization technology for intelligent driving systems. IEEE Intelligent Transportation Systems Magazine 14, 1, 115–127.

    Article  Google Scholar 

  • Feng, S., Feng, Y., Yu, C., Zhang, Y. and Liu, H. X. (2020). Testing scenario library generation for connected and automated vehicles, part I: Methodology. IEEE Trans. Intelligent Transportation Systems 22, 3, 1573–1582.

    Article  Google Scholar 

  • Gauthier, T. D. (2001). Detecting trends using Spearman’s rank correlation coefficient. Environmental Forensics 2, 4, 359–362.

    Article  Google Scholar 

  • Grosso, A., Jamali, A. and Locatelli, M. (2009). Finding maximin latin hypercube designs by iterated local search heuristics. European J. Operational Research 197, 2, 541–547.

    Article  MATH  Google Scholar 

  • Ghodsi, Z., Hari, S. K. S., Frosio, I., Tsai, T., Troccoli, A., Keckler, S. W., Garg, S. and Anandkumar, A. (2021). Generating and characterizing scenarios for safety testing of autonomous vehicles. IEEE Intelligent Vehicles Symp. (IV), Nagoya, Japan.

  • Junietz, P., Bonakdar, F., Klamann, B. and Winner, H. (2018). Criticality metric for the safety validation of automated driving using model predictive trajectory optimization. 21st Int. Conf. Intelligent Transportation Systems (TTSC), Maui, Hawaii, USA.

  • Kennedy, J. and Eberhart, R. (1995). Particle swarm optimization. Int. Conf. Neural Networks (ICNN), Perth, WA, Australia.

  • Kapoor, P. (2017). A Spatio-Temporal Approach to Mitigate Automotive Radar Spoofing Attacks. Ph. D. Dissertation, Binghamton University. Binghamton, NY, USA.

    Google Scholar 

  • Li, W., Lu, L., Xie, X. and Yang, M. (2017). A novel extension algorithm for optimized Latin hypercube sampling. J. Statistical Computation and Simulation 87, 13, 2549–2559.

    Article  MathSciNet  MATH  Google Scholar 

  • Liefvendahl, M. and Stocki, R. (2006). A study on algorithms for optimization of Latin hypercubes. J. Statistical Planning and Inference 136, 9, 3231–3247.

    Article  MathSciNet  MATH  Google Scholar 

  • Ma, W. H. and Peng, H. (1999). A worst-case evaluation method for dynamic systems. J. Dynamic SYstems, Measurement, and Control 121, 2, 191–199.

    Article  Google Scholar 

  • Naus, G., Van Den Bleek, R., Ploeg, J., Scheepers, B., van de Molengraft, R. and Steinbuch, M. (2008). Explicit MPC design and performance evaluation of an ACC Stop-&-Go. American Control Conf. (ACC), Seattle, WA, USA.

  • Rebonato, R. and Jäckel, P. (2011). The most general methodology to create a valid correlation matrix for risk management and option pricing purposes. Available at SSRN 1969689.

  • Roberts, G. O. and Sahu, S. K. (1997). Updating schemes, correlation structure, blocking and parameterization for the Gibbs sampler. J. Royal Statistical Society: Series B (Statistical Methodology) 59, 2, 291–317.

    Article  MathSciNet  MATH  Google Scholar 

  • Sun, H., Feng, S., Yan, X. and Liu, H. X. (2021). Corner case generation and analysis for safety assessment of autonomous vehicles. Transportation Research Record 2615, 11, 587–600.

    Article  Google Scholar 

  • Saifuzzaman, M. and Zheng, Z. (2014). Incorporating human-factors in car-following models: A review of recent developments and research needs. Transportation Research Part C: Emerging Technologies, 48, 379–403.

    Article  Google Scholar 

  • Schmidt, R., Voigt, M. and Mailach, R. (2019). Latin hypercube sampling-based Monte Carlo simulation: Extension of the sample size and correlation control. Uncertainty Management for Robust Industrial Design in Aeronautics (pp. 279–289). Springer, Cham, Switzerland.

    Chapter  Google Scholar 

  • Sippl, C, Schwab, B., Kielar, P. and Djanatliev, A. (2018). Distributed real-time traffic simulation for autonomous vehicle testing in urban environments. 21st Int. Conf. Intelligent Transportation Systems (ITSC), Maui, Hawaii, USA.

  • Tatar, M. (2015). Enhancing ADAS test and validation with automated search for critical situations. Driving Simulation Conf. (DSC), Tübingen, Germany.

  • Ying, Y. and Solomon, O. O. (2017). Research on adaptive cruise control systems and performance analysis using Matlab and Carsim. 5th Int. Conf. Mechanical, Automotive and Materials Engineering (CMAME), Guangzhou, China.

  • Yu, H., C. Y. Chung, K. P. Wong, H. W. Lee, and J. H. Zhang. (2009). Probabilistic load flow evaluation with hybrid latin hypercube sampling and cholesky decomposition. IEEE Transactions on Power Systems 24, 2, 661–667.

    Article  Google Scholar 

  • Zhao, D., Huang, X., Peng, H., Lam, H. and LeBlanc, D. J. (2017). Accelerated evaluation of automated vehicles in car-following maneuvers. IEEE Trans. Intelligent Transportation Systems 19, 3, 733–744.

    Article  Google Scholar 

Download references

Acknowledgement

This work was supported by the National Key R&D Program of China (No.2018YFB 1701600), and the Technology Innovation and Application Development Program of Chongqing, China (No. cstc2019jscx-fxydX0041).

Author information

Authors and Affiliations

  1. Control and Simulation Center, Harbin Institute of Technology, Harbin, 150001, China

    Shuang Li, Wei Li, Ping Ma & Ming Yang

  2. Key Laboratory of Transport Industry of Big Data Application Technologies for Comprehensive Transport, the School of Traffic and Transportation, Beijing Jiaotong University, Beijing, 100044, China

    Penghui Li

  3. State Key Laboratory of Vehicle NVH and Safety Technology, China Automotive Engineering Research Institute Co., Ltd., Chongqing, 401122, China

    Penghui Li

Authors
  1. Shuang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Penghui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ping Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ming Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ming Yang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, S., Li, W., Li, P. et al. Novel Test Scenario Generation Technology for Performance Evaluation of Automated Vehicle. Int.J Automot. Technol. 23, 1295–1312 (2022). https://doi.org/10.1007/s12239-022-0113-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12239-022-0113-z

Key Words

Search

Navigation