Skip to main content
SpringerLink
Log in

Artificial intelligence test: a case study of intelligent vehicles

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

To meet the urgent requirement of reliable artificial intelligence applications, we discuss the tight link between artificial intelligence and intelligence test in this paper. We highlight the role of tasks in intelligence test for all kinds of artificial intelligence. We explain the necessity and difficulty of describing tasks for intelligence test, checking all the tasks that may encounter in intelligence test, designing simulation-based test, and setting appropriate test performance evaluation indices. As an example, we present how to design reliable intelligence test for intelligent vehicles. Finally, we discuss the future research directions of intelligence test.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • A Tragic Loss (2016) https://www.tesla.com/blog/tragic-loss. Accessed April 2018

  • Ackerman E (2014) A better test than Turing. IEEE Spectr 51(10):20–21

    Article  Google Scholar 

  • Ammann P, Jeff O (2017) Introduction to software testing, 2nd edn. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  • Argall BD, Chernova S, Veloso M, Browning B (2009) A survey of robot learning from demonstration. Robot Auton Syst 57(5):469–483

    Article  Google Scholar 

  • Bagnell JA (2015) An invitation to imitation. Technical Report, CMU-RI-TR-15-08, Robotics Institute, Carnegie Mellon University

  • Black R (2009) Managing the testing process: practical tools and techniques for managing hardware and software testing. Wiley, Hoboken

    Google Scholar 

  • Boehm BW (1988) A spiral model of software development and enhancement. IEEE Comput 21(5):61–72

    Article  Google Scholar 

  • Bradley AR, Manna Z (2007) The calculus of computation: decision procedures with applications to verification. Springer, Berlin

    MATH  Google Scholar 

  • Broggi A, Buzzoni M, Debattisti S, Grisleri P, Laghi MC, Medici P, Versari P (2013) Extensive tests of autonomous driving technologies. IEEE Trans Intell Transp Syst 14(3):1403–1415

    Article  Google Scholar 

  • Broggi A, Cerri P, Debattisti S, Laghi MC, Medici P, Molinari D, Panciroli M, Prioletti A (2015) PROUD—public road urban driverless-car test. IEEE Trans Intell Transp Syst 16(6):3508–3519

    Article  Google Scholar 

  • Brown N, Sandholm T (2017) Safe and nested subgame solving for imperfect-information games. https://arxiv.org/abs/1705.02955. Accessed April 2018

  • Browne CB, Powley E, Whitehouse D, Lucas SM, Cowling PI, Rohlfshagen P, Tavener S, Perez D, Samothrakis S, Colton S (2012) A survey of monte carlo tree search methods. IEEE Trans Comput Intell AI Games 4(1):1–43

    Article  Google Scholar 

  • Buehler M, Iagnemma K, Singh S (eds) (2009) The DARPA urban challenge. Springer, Berlin

    Google Scholar 

  • Butakov VA, Ioannou P (2015) Personalized driver/vehicle lane change models for ADAS. IEEE Trans Veh Technol 64(10):4422–4431

    Article  Google Scholar 

  • Campbell M, Egerstedt M, How JP, Murray RM (2010) Autonomous driving in urban environments: approaches, lessons and challenges. Philos Trans R Soc A 368(1928):4649–4672

    Article  Google Scholar 

  • Chen Z, Liu B (2016) Lifelong machine learning. Morgan & Claypool Publishers, San Rafael

    Google Scholar 

  • Cheng PCH (2016) What constitutes an effective representation? In: Jamnik M, Uesaka Y, Elzer Schwartz S (eds) Diagrammatic representation and inference: proceedings from the 9th international conference, diagrams 2016, vol 9781. Lecture notes in computer science. Springer, Berlin

    Chapter  Google Scholar 

  • Classen S, Nichols AL, McPeek R, Breinerd JF (2011) Personality as a predictor of driving performance: an exploratory study. Transp Res F Traffic Psychol Behav 14(5):381–389

    Article  Google Scholar 

  • Coulom R (2008) Whole-history rating: a Bayesian rating system for players of time-varying strength. In: Proceedings of international conference on computers and games, pp 113–124

  • DARPA Grand Challenge, DARPA Urban Challenge (2004–2007) http://archive.darpa.mil/grandchallenge/. Accessed April 2018

  • Ding Z, Jiang C, Zhou MC (2013) Design, analysis and verification of real-time systems based on time Petri net refinement. ACM Transactions in Embedded Computing Systems 12:4:1–4:18. https://doi.org/10.1145/2406336.2406340

    Article  Google Scholar 

  • Elo AE (1978) The rating of chessplayers, past and present. Arco Publishing, New York

    Google Scholar 

  • Eskandarian A (ed) (2012) Handbook of intelligent vehicles. Springer, Berlin

    Google Scholar 

  • Evtimov I, Eykholt K, Fernandes E, Kohno T, Li B, Prakash A, Rahmati A, Song D (2017) Robust physical-world attacks on machine learning models. https://arxiv.org/abs/1707.08945. Accessed April 2018

  • Fagnant DJ, Kockelman K (2015) Preparing a nation for autonomous vehicles: opportunities, barriers and policy recommendations. Transp Res A Policy Practice 77:167–181

    Article  Google Scholar 

  • Fisher DL, Lohrenz M, Moore D, Nadler ED, Pollard JK (2016) Humans and intelligent vehicles: the hope, the help, and the harm. IEEE Trans Intell Veh 1(1):56–67

    Article  Google Scholar 

  • Gaidon A, Wang Q, Cabon Y, Vig E (2016) Virtual worlds as proxy for multi-object tracking analysis. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4340–4349

  • Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 2414–2423

  • George D, Lehrach W, Kansky K, Lázaro-Gredilla M, Laan C, Marthi B, Lou X, Meng Z, Liu Y, Wang H, Lavin A, Phoenix DS (2017) A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs. Science. https://doi.org/10.1126/science.aag2612

    Google Scholar 

  • Goodall NJ (2014) Ethical decision making during automated vehicle crashes. Transp Res Rec 2424:58–65

    Article  Google Scholar 

  • Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Proc Adv Neural Inf Process Syst 27:2672–2680

    Google Scholar 

  • Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge

    MATH  Google Scholar 

  • Greengard S (2017) Gaming machine learning. Commun ACM 60(12):14–16

    Article  Google Scholar 

  • GTSDB, The German Traffic Sign Recognition Benchmark and the German Traffic Sign Detection Benchmark (2014) http://benchmark.ini.rub.de/?section=home&subsection=news. Accessed April 2018

  • Harari YN (2017) Reboot for the AI revolution. Nature 550:324–327

    Article  Google Scholar 

  • Hernández-Orallo J (2017) Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement 48(3):397–447

    Article  Google Scholar 

  • Ho J, Ermon S (2017) Generative adversarial imitation learning. https://arxiv.org/abs/1606.03476. Accessed April 2018

  • Huang WL, Wen D, Geng J, Zheng NN (2014) Task-specific performance evaluation of ugvs: case studies at the IFVC. IEEE Trans Intell Transp Syst 15(5):1969–1979

    Article  Google Scholar 

  • Huizinga D, Adam K (2007) Automated defect prevention: best practices in software management. Wiley, Hoboken

    Book  Google Scholar 

  • IBM, Deep Blue - Overview (1997) IBM Research. http://www-03.ibm.com/ibm/history/ibm100/us/en/icons/deepblue/. Accessed April 2018

  • ImageNet (2016) http://image-net.org. Accessed April 2018

  • Karp RM (1972) Reducibility among combinatorial problems. In: Miller RE, Thacher JW (eds) Complexity of computer computation. Plenum Press, New York, pp 85–103

    Chapter  Google Scholar 

  • Karpathy A (2017) Software 2.0. https://medium.com/@karpathy/software-2-0-a64152b37c35. Accessed April 2018

  • Koopman P, Wagner M (2017) Autonomous vehicle safety: an interdisciplinary challenge. IEEE Intell Transp Syst Mag 9(1):90–96

    Article  Google Scholar 

  • Kroening D, Strichman O (2016) Decision procedures: an algorithmic point of view, 2nd edn. Springer, Berlin

    Book  MATH  Google Scholar 

  • Kuefler A, Morton J, Wheeler T, Kochenderfer M (2017) Imitating driver behavior with generative adversarial networks. In: Proceedings of IEEE intelligent vehicles symposium, pp 204–211

  • Heule MJH, Kullmann O (2017) The science of brute force. Commun ACM 60(8):70–79

    Article  Google Scholar 

  • Kumfer W, Burgess R (2015) Investigation into the role of rational ethics in crashes of automated vehicles. Transp Res Rec 2489:130–136

    Article  Google Scholar 

  • Kurzweil R (2005) The singularity is near. Viking Press, New York

    Google Scholar 

  • Lamb E (2016) Maths proof smashes size record: supercomputer produces a 200-terabyte proof—but is it really mathematics? Nature 534(7605):17–19

    Article  Google Scholar 

  • LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  • Lefèvre S, Carvalho A, Gao Y, Tseng HE, Borrellia F (2015) Driver models for personalised driving assistance. Veh Syst Dyn 53(12):1705–1720

    Article  Google Scholar 

  • Levesque HJ (2014) On our best behavior. Artif Intell 212:27–35

    Article  MATH  Google Scholar 

  • Levesque HJ (2017) Common sense, the Turing test, and the quest for real AI. MIT Press, Cambridge

    MATH  Google Scholar 

  • Li L, Wang FY (2007) Advanced motion control and sensing for intelligent vehicles. Springer, New York

    MATH  Google Scholar 

  • Li L, Wen D, Zheng NN, Shen LC (2012) Cognitive cars: a new frontier for ADAS research. IEEE Trans Intell Transp Syst 13(1):395–407

    Article  Google Scholar 

  • Li L, Huang WL, Liu Y, Zheng NN, Wang FY (2016a) Intelligence testing for autonomous vehicles: a new approach. IEEE Trans Intell Veh 1(2):158–166

    Article  Google Scholar 

  • Li L, Lv Y, Wang FY (2016b) Traffic signal timing via deep reinforcement learning. IEEE/CAA J Autom Sin 3(3):247–254

    Article  MathSciNet  Google Scholar 

  • Li L, Lin Y, Zheng NN, Wang FY (2017) Parallel learning: a perspective and a framework. IEEE/CAA J Autom Sin 4(3):389–395

    Article  MathSciNet  Google Scholar 

  • Liao R (2017) Tencent discovers major loopholes in Google’s AI platform TensorFlow. https://technode.com/2017/12/18/tencent-tensorflow/. Accessed April 2018

  • Licato J, Zhang Z (2017) Evaluating representational systems in artificial intelligence. Artif Intell Rev. https://doi.org/10.1007/s10462-017-9598-7

    Google Scholar 

  • Liu MY, Breuel T, Kautz J (2017) Unsupervised image-to-image translation networks. https://arxiv.org/abs/1703.00848. Accessed April 2018

  • Mackintosh NJ (2011) IQ and human intelligence, 2nd edn. Oxford University Press, Oxford

    Google Scholar 

  • Maurer M, Gerdes JC, Lenz B, Winner H (eds) (2015) Autonomous driving: technical, legal and social aspects. Springer, Berlin

    Google Scholar 

  • Mcguire G, Tugemann B, Civario G (2014) There is no 16-clue sudoku: solving the sudoku minimum number of clues problem via hitting set enumeration. Exp Math 23(2):190–217

    Article  MathSciNet  MATH  Google Scholar 

  • Merel J, Tassa Y, TB D, Srinivasan S, Lemmon J, Wang Z, Wayne G, Heess N (2017) Learning human behaviors from motion capture by adversarial imitation. https://arxiv.org/abs/1707.02201. Accessed April 2018

  • Minsky ML (ed) (1968) Semantic information processing. MIT Press, Cambridge

    MATH  Google Scholar 

  • Moravčík M, Schmid M, Burch N, Lisý V, Morrill D, Bard N, Davis T, Waugh K, Johanson M, Bowling M (2017) DeepStack: expert-level artificial intelligence in heads-up no-limit poker. Science 356:508–513

    Article  MathSciNet  Google Scholar 

  • Newell A, Simon HA (1976) Computer science as empirical inquiry: symbols and search. Commun ACM CACM Homepage 19(3):113–126

    Article  MathSciNet  Google Scholar 

  • Ohlsson S, Sloan RH, Turán G, Urasky A (2017) Measuring an artificial intelligence system’s performance on a verbal IQ test for young children. J Exp Theor Artif Intell 29(4):679–693

    Article  Google Scholar 

  • Raccoon L (1997) Fifty years of progress in software engineering. ACM SIGSOFT Softw Eng Notes 22(1):88–104

    Article  Google Scholar 

  • Redmon J, Farhadi A (2016) YOLO9000: better, faster, stronger. https://arxiv.org/abs/1612.08242. Accessed April 2018

  • Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. https://arxiv.org/abs/1506.02640. Accessed April 2018

  • Richter SR, Vineet V, Roth S, Koltun V (2016) Playing for data: ground truth from computer games. In: European conference on computer vision, pp 102–118

  • Rindermann H, Becker D, Coyle TR (2016) Survey of expert opinion on intelligence: causes of international differences in cognitive ability tests. Front Psychol. https://doi.org/10.3389/fpsyg.2016.00399

    Google Scholar 

  • Ros G, Sellart L, Materzynska J, Vazquez D, Lopez AM (2016) The SYNTHIA dataset: a large collection of synthetic images for semantic segmentation of urban scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3234–3243

  • Russell S, Norvig P (2010) Artificial intelligence: a modern approach, 3rd edn. Pearson Education Limited, London

    MATH  Google Scholar 

  • SAE J3016 (2016) Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems. SAE, Warrendale

    Google Scholar 

  • Santana E, Hotz G (2016) Learning a driving simulator. https://arxiv.org/abs/1608.01230. Accessed April 2018

  • Schoenick C, Clark P, Tafjord O, Turney P, Etzioni O (2017) Moving beyond the Turing test with the Allen AI science challenge. Commun ACM 60(9):60–64

    Article  Google Scholar 

  • Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T, Leach M, Kavukcuoglu K, Graepel T, Hassabis D (2016) Mastering the game of Go with deep neural networks and tree search. Nature 529(7587):484–489

    Article  Google Scholar 

  • Silver D, Hubert T, Schrittwieser J, Antonoglou I, Lai M, Guez A, Lanctot M, Sifre L, Kumaran D, Graepel T, Lillicrap T, Simonyan K, Hassabis D (2017a) Mastering Chess and Shogi by self-play with a general reinforcement learning algorithm. https://arxiv.org/abs/1712.01815. Accessed April 2018

  • Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A, Chen Y, Lillicrap T, Hui F, Sifre L, van den Driessche G, Graepel T, Hassabis D (2017b) Mastering the game of Go without human knowledge. Nature 550:354–359

    Article  Google Scholar 

  • Srinivasan B, Parthasarathi R (2017) A survey of imperatives and action representation formalisms. Artif Intell Rev 48(2):263–297

    Article  Google Scholar 

  • Sternberg RJ (1985) Beyond IQ: a triarchic theory of human intelligence. Cambridge University Press, Cambridge

    Google Scholar 

  • Sternberg RJ, Davidson JE (1983) Insight in the gifted. Educ Psychol 18(1):51–57

    Article  Google Scholar 

  • Thornton SM, Pan S, Erlien SM, Gerdes JC (2017) Incorporating ethical considerations into automated vehicle control. IEEE Trans Intell Transp Syst 18(6):1429–1439

    Google Scholar 

  • Tong Y, Zhao L, Li L, Zhang Y (2015) Stochastic programming model for oversaturated intersection signal timing. Transp Res Part C 58:474–486

    Article  Google Scholar 

  • Turing AM (1950) Computing machinery and intelligence. Mind 59(236):433–460

    Article  MathSciNet  Google Scholar 

  • Veeravasarapu VSR, Hota RN, Rothkopf C, Visvanathan R (2015) Simulations for validation of vision systems. Comput Sci. https://arxiv.org/abs/1512.01030

  • Vinge V (1993) The coming technological singularity: how to survive in the post-human era. In: Landis GA (ed) Vision-21: interdisciplinary science and engineering in the ear of cyberspace. NASA Publication, CP-10129, Washington, pp 11–22

    Google Scholar 

  • von Ahn L, Blum M, Hopper NJ, Langford J (2003) CAPTCHA: using hard AI problems for security. In: Proceedings of international conference on the theory and applications of cryptographic techniques, pp 294–311

  • Wagner M, Koopman P (2015) A philosophy for developing trust in self-driving cars. In: Meyer G, Beiker S (eds) Road vehicle automation 2. Lecture notes in mobility. Springer, Cham. https://doi.org/10.1007/978-3-319-19078-5_14

    Google Scholar 

  • Wang FY, Zhang JJ, Zheng X et al (2016) Where does AlphaGo go: from church-turing thesis to AlphaGo thesis and beyond. IEEE/CAA J Automatica Sin 3:113–120. https://doi.org/10.1109/JAS.2016.7471613

    Article  Google Scholar 

  • Wang L (2016) Directions 2017: BeiDou’s road to global service. GPS World

  • Wang FY, Wang X, Li L, Li L (2016a) Steps toward parallel intelligence. IEEE/CAA J Autom Sin 3(4):345–348

    Article  MathSciNet  Google Scholar 

  • Wang X, Zheng X, Zhang Q, Wang T, Shen D (2016b) Crowdsourcing in ITS: the state of the work and the networking. IEEE Trans Intell Transp Syst 17(6):1596–1605

    Article  Google Scholar 

  • Wang K, Gou C, Zheng N, Rehg JM, Wang FY (2017a) Parallel vision for perception and understanding of complex scenes: methods, framework, and perspectives. Artif Intell Rev 1:1–31

    Google Scholar 

  • Wang X, Jiang R, Li L, Lin Y, Zheng X, Wang FY (2017b) Capturing car-following behaviors by deep learning. IEEE Trans Intell Transp Syst. http://ieeexplore.ieee.org/document/7970189/

  • Watzenig D, Horn M (2017a) Automated driving: safer and more efficient future driving. Springer, Cham

    Book  Google Scholar 

  • Watzenig D, Horn M (2017b) Automated driving: safer and more efficient future driving. Springer, Cham

    Book  Google Scholar 

  • You J. (2017) Deep learning based lane departure detection for automated vehicles. Bachelor Thesis, Tsinghua University

  • Zhao D, Huang X, Peng H, Lam H, Leblanc DJ (2017) Accelerated evaluation of automated vehicles in car-following maneuvers. IEEE Trans Intell Transp Syst. http://ieeexplore.ieee.org/document/7933977/

  • Zheng NN, Tang S, Cheng H, Li Q, Lai G, Wang FY (2004) Toward intelligent driver-assistance and safety warning systems. IEEE Intell Syst 19(2):8–11

    Article  Google Scholar 

  • Zheng NN, Liu ZY, Ren PJ, Ma YQ, Chen ST, Yu SY, Xue JR, Chen BD, Wang FY (2017) Hybrid-augmented intelligence: collaboration and cognition. Front Inf Technol Electron Eng 18(2):153–179

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants 91520301 and 61533019, and the Beijing Municipal Science and Technology Project (No. D171100000317002).

Author information

Authors and Affiliations

  1. Department of Automation, BNRist, Tsinghua University, Beijing, 100084, China

    Li Li

  2. The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Yi-Lun Lin, Fei-Yue Wang, Kunfeng Wang & Wu-Ling Huang

  3. Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, 710049, China

    Nan-Ning Zheng & Yuehu Liu

  4. Department of Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, N2L3G1, Canada

    Dongpu Cao

  5. Qingdao Academy of Intelligent Industries, Qingdao, 266109, Shandong, China

    Yi-Lun Lin & Fei-Yue Wang

  6. VIPioneers (HuiTuo) Inc., Qingdao, 266109, Shandong, China

    Dongpu Cao

Authors
  1. Li Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yi-Lun Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nan-Ning Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fei-Yue Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuehu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dongpu Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Kunfeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Wu-Ling Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fei-Yue Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, L., Lin, YL., Zheng, NN. et al. Artificial intelligence test: a case study of intelligent vehicles. Artif Intell Rev 50, 441–465 (2018). https://doi.org/10.1007/s10462-018-9631-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-018-9631-5

Keywords

Search

Navigation