Artificial Intelligence Review

, Volume 50, Issue 3, pp 441–465 | Cite as

Artificial intelligence test: a case study of intelligent vehicles

  • Li Li
  • Yi-Lun Lin
  • Nan-Ning Zheng
  • Fei-Yue WangEmail author
  • Yuehu Liu
  • Dongpu Cao
  • Kunfeng Wang
  • Wu-Ling Huang


To meet the urgent requirement of reliable artificial intelligence applications, we discuss the tight link between artificial intelligence and intelligence test in this paper. We highlight the role of tasks in intelligence test for all kinds of artificial intelligence. We explain the necessity and difficulty of describing tasks for intelligence test, checking all the tasks that may encounter in intelligence test, designing simulation-based test, and setting appropriate test performance evaluation indices. As an example, we present how to design reliable intelligence test for intelligent vehicles. Finally, we discuss the future research directions of intelligence test.


Artificial intelligence Intelligence test Turing test Simulation test 



This work was supported in part by the National Natural Science Foundation of China under Grants 91520301 and 61533019, and the Beijing Municipal Science and Technology Project (No. D171100000317002).


  1. A Tragic Loss (2016) Accessed April 2018
  2. Ackerman E (2014) A better test than Turing. IEEE Spectr 51(10):20–21CrossRefGoogle Scholar
  3. Ammann P, Jeff O (2017) Introduction to software testing, 2nd edn. Cambridge University Press, CambridgezbMATHGoogle Scholar
  4. Argall BD, Chernova S, Veloso M, Browning B (2009) A survey of robot learning from demonstration. Robot Auton Syst 57(5):469–483CrossRefGoogle Scholar
  5. Bagnell JA (2015) An invitation to imitation. Technical Report, CMU-RI-TR-15-08, Robotics Institute, Carnegie Mellon UniversityGoogle Scholar
  6. Black R (2009) Managing the testing process: practical tools and techniques for managing hardware and software testing. Wiley, HobokenGoogle Scholar
  7. Boehm BW (1988) A spiral model of software development and enhancement. IEEE Comput 21(5):61–72CrossRefGoogle Scholar
  8. Bradley AR, Manna Z (2007) The calculus of computation: decision procedures with applications to verification. Springer, BerlinzbMATHGoogle Scholar
  9. Broggi A, Buzzoni M, Debattisti S, Grisleri P, Laghi MC, Medici P, Versari P (2013) Extensive tests of autonomous driving technologies. IEEE Trans Intell Transp Syst 14(3):1403–1415CrossRefGoogle Scholar
  10. Broggi A, Cerri P, Debattisti S, Laghi MC, Medici P, Molinari D, Panciroli M, Prioletti A (2015) PROUD—public road urban driverless-car test. IEEE Trans Intell Transp Syst 16(6):3508–3519CrossRefGoogle Scholar
  11. Brown N, Sandholm T (2017) Safe and nested subgame solving for imperfect-information games. Accessed April 2018
  12. Browne CB, Powley E, Whitehouse D, Lucas SM, Cowling PI, Rohlfshagen P, Tavener S, Perez D, Samothrakis S, Colton S (2012) A survey of monte carlo tree search methods. IEEE Trans Comput Intell AI Games 4(1):1–43CrossRefGoogle Scholar
  13. Buehler M, Iagnemma K, Singh S (eds) (2009) The DARPA urban challenge. Springer, BerlinGoogle Scholar
  14. Butakov VA, Ioannou P (2015) Personalized driver/vehicle lane change models for ADAS. IEEE Trans Veh Technol 64(10):4422–4431CrossRefGoogle Scholar
  15. Campbell M, Egerstedt M, How JP, Murray RM (2010) Autonomous driving in urban environments: approaches, lessons and challenges. Philos Trans R Soc A 368(1928):4649–4672CrossRefGoogle Scholar
  16. Chen Z, Liu B (2016) Lifelong machine learning. Morgan & Claypool Publishers, San RafaelGoogle Scholar
  17. Cheng PCH (2016) What constitutes an effective representation? In: Jamnik M, Uesaka Y, Elzer Schwartz S (eds) Diagrammatic representation and inference: proceedings from the 9th international conference, diagrams 2016, vol 9781. Lecture notes in computer science. Springer, BerlinCrossRefGoogle Scholar
  18. Classen S, Nichols AL, McPeek R, Breinerd JF (2011) Personality as a predictor of driving performance: an exploratory study. Transp Res F Traffic Psychol Behav 14(5):381–389CrossRefGoogle Scholar
  19. Coulom R (2008) Whole-history rating: a Bayesian rating system for players of time-varying strength. In: Proceedings of international conference on computers and games, pp 113–124Google Scholar
  20. DARPA Grand Challenge, DARPA Urban Challenge (2004–2007) Accessed April 2018
  21. Ding Z, Jiang C, Zhou MC (2013) Design, analysis and verification of real-time systems based on time Petri net refinement. ACM Transactions in Embedded Computing Systems 12:4:1–4:18. CrossRefGoogle Scholar
  22. Elo AE (1978) The rating of chessplayers, past and present. Arco Publishing, New YorkGoogle Scholar
  23. Eskandarian A (ed) (2012) Handbook of intelligent vehicles. Springer, BerlinGoogle Scholar
  24. Evtimov I, Eykholt K, Fernandes E, Kohno T, Li B, Prakash A, Rahmati A, Song D (2017) Robust physical-world attacks on machine learning models. Accessed April 2018
  25. Fagnant DJ, Kockelman K (2015) Preparing a nation for autonomous vehicles: opportunities, barriers and policy recommendations. Transp Res A Policy Practice 77:167–181CrossRefGoogle Scholar
  26. Fisher DL, Lohrenz M, Moore D, Nadler ED, Pollard JK (2016) Humans and intelligent vehicles: the hope, the help, and the harm. IEEE Trans Intell Veh 1(1):56–67CrossRefGoogle Scholar
  27. Gaidon A, Wang Q, Cabon Y, Vig E (2016) Virtual worlds as proxy for multi-object tracking analysis. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4340–4349Google Scholar
  28. Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 2414–2423Google Scholar
  29. George D, Lehrach W, Kansky K, Lázaro-Gredilla M, Laan C, Marthi B, Lou X, Meng Z, Liu Y, Wang H, Lavin A, Phoenix DS (2017) A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs. Science. Google Scholar
  30. Goodall NJ (2014) Ethical decision making during automated vehicle crashes. Transp Res Rec 2424:58–65CrossRefGoogle Scholar
  31. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Proc Adv Neural Inf Process Syst 27:2672–2680Google Scholar
  32. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, CambridgezbMATHGoogle Scholar
  33. Greengard S (2017) Gaming machine learning. Commun ACM 60(12):14–16CrossRefGoogle Scholar
  34. GTSDB, The German Traffic Sign Recognition Benchmark and the German Traffic Sign Detection Benchmark (2014) Accessed April 2018
  35. Harari YN (2017) Reboot for the AI revolution. Nature 550:324–327CrossRefGoogle Scholar
  36. Hernández-Orallo J (2017) Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement 48(3):397–447CrossRefGoogle Scholar
  37. Ho J, Ermon S (2017) Generative adversarial imitation learning. Accessed April 2018
  38. Huang WL, Wen D, Geng J, Zheng NN (2014) Task-specific performance evaluation of ugvs: case studies at the IFVC. IEEE Trans Intell Transp Syst 15(5):1969–1979CrossRefGoogle Scholar
  39. Huizinga D, Adam K (2007) Automated defect prevention: best practices in software management. Wiley, HobokenCrossRefGoogle Scholar
  40. IBM, Deep Blue - Overview (1997) IBM Research. Accessed April 2018
  41. ImageNet (2016) Accessed April 2018
  42. Karp RM (1972) Reducibility among combinatorial problems. In: Miller RE, Thacher JW (eds) Complexity of computer computation. Plenum Press, New York, pp 85–103CrossRefGoogle Scholar
  43. Karpathy A (2017) Software 2.0. Accessed April 2018
  44. Koopman P, Wagner M (2017) Autonomous vehicle safety: an interdisciplinary challenge. IEEE Intell Transp Syst Mag 9(1):90–96CrossRefGoogle Scholar
  45. Kroening D, Strichman O (2016) Decision procedures: an algorithmic point of view, 2nd edn. Springer, BerlinCrossRefzbMATHGoogle Scholar
  46. Kuefler A, Morton J, Wheeler T, Kochenderfer M (2017) Imitating driver behavior with generative adversarial networks. In: Proceedings of IEEE intelligent vehicles symposium, pp 204–211Google Scholar
  47. Heule MJH, Kullmann O (2017) The science of brute force. Commun ACM 60(8):70–79CrossRefGoogle Scholar
  48. Kumfer W, Burgess R (2015) Investigation into the role of rational ethics in crashes of automated vehicles. Transp Res Rec 2489:130–136CrossRefGoogle Scholar
  49. Kurzweil R (2005) The singularity is near. Viking Press, New YorkGoogle Scholar
  50. Lamb E (2016) Maths proof smashes size record: supercomputer produces a 200-terabyte proof—but is it really mathematics? Nature 534(7605):17–19CrossRefGoogle Scholar
  51. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444CrossRefGoogle Scholar
  52. Lefèvre S, Carvalho A, Gao Y, Tseng HE, Borrellia F (2015) Driver models for personalised driving assistance. Veh Syst Dyn 53(12):1705–1720CrossRefGoogle Scholar
  53. Levesque HJ (2014) On our best behavior. Artif Intell 212:27–35CrossRefzbMATHGoogle Scholar
  54. Levesque HJ (2017) Common sense, the Turing test, and the quest for real AI. MIT Press, CambridgezbMATHGoogle Scholar
  55. Li L, Wang FY (2007) Advanced motion control and sensing for intelligent vehicles. Springer, New YorkzbMATHGoogle Scholar
  56. Li L, Wen D, Zheng NN, Shen LC (2012) Cognitive cars: a new frontier for ADAS research. IEEE Trans Intell Transp Syst 13(1):395–407CrossRefGoogle Scholar
  57. Li L, Huang WL, Liu Y, Zheng NN, Wang FY (2016a) Intelligence testing for autonomous vehicles: a new approach. IEEE Trans Intell Veh 1(2):158–166CrossRefGoogle Scholar
  58. Li L, Lv Y, Wang FY (2016b) Traffic signal timing via deep reinforcement learning. IEEE/CAA J Autom Sin 3(3):247–254MathSciNetCrossRefGoogle Scholar
  59. Li L, Lin Y, Zheng NN, Wang FY (2017) Parallel learning: a perspective and a framework. IEEE/CAA J Autom Sin 4(3):389–395MathSciNetCrossRefGoogle Scholar
  60. Liao R (2017) Tencent discovers major loopholes in Google’s AI platform TensorFlow. Accessed April 2018
  61. Licato J, Zhang Z (2017) Evaluating representational systems in artificial intelligence. Artif Intell Rev. Google Scholar
  62. Liu MY, Breuel T, Kautz J (2017) Unsupervised image-to-image translation networks. Accessed April 2018
  63. Mackintosh NJ (2011) IQ and human intelligence, 2nd edn. Oxford University Press, OxfordGoogle Scholar
  64. Maurer M, Gerdes JC, Lenz B, Winner H (eds) (2015) Autonomous driving: technical, legal and social aspects. Springer, BerlinGoogle Scholar
  65. Mcguire G, Tugemann B, Civario G (2014) There is no 16-clue sudoku: solving the sudoku minimum number of clues problem via hitting set enumeration. Exp Math 23(2):190–217MathSciNetCrossRefzbMATHGoogle Scholar
  66. Merel J, Tassa Y, TB D, Srinivasan S, Lemmon J, Wang Z, Wayne G, Heess N (2017) Learning human behaviors from motion capture by adversarial imitation. Accessed April 2018
  67. Minsky ML (ed) (1968) Semantic information processing. MIT Press, CambridgezbMATHGoogle Scholar
  68. Moravčík M, Schmid M, Burch N, Lisý V, Morrill D, Bard N, Davis T, Waugh K, Johanson M, Bowling M (2017) DeepStack: expert-level artificial intelligence in heads-up no-limit poker. Science 356:508–513MathSciNetCrossRefGoogle Scholar
  69. Newell A, Simon HA (1976) Computer science as empirical inquiry: symbols and search. Commun ACM CACM Homepage 19(3):113–126MathSciNetCrossRefGoogle Scholar
  70. Ohlsson S, Sloan RH, Turán G, Urasky A (2017) Measuring an artificial intelligence system’s performance on a verbal IQ test for young children. J Exp Theor Artif Intell 29(4):679–693CrossRefGoogle Scholar
  71. Raccoon L (1997) Fifty years of progress in software engineering. ACM SIGSOFT Softw Eng Notes 22(1):88–104CrossRefGoogle Scholar
  72. Redmon J, Farhadi A (2016) YOLO9000: better, faster, stronger. Accessed April 2018
  73. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. Accessed April 2018
  74. Richter SR, Vineet V, Roth S, Koltun V (2016) Playing for data: ground truth from computer games. In: European conference on computer vision, pp 102–118Google Scholar
  75. Rindermann H, Becker D, Coyle TR (2016) Survey of expert opinion on intelligence: causes of international differences in cognitive ability tests. Front Psychol. Google Scholar
  76. Ros G, Sellart L, Materzynska J, Vazquez D, Lopez AM (2016) The SYNTHIA dataset: a large collection of synthetic images for semantic segmentation of urban scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3234–3243Google Scholar
  77. Russell S, Norvig P (2010) Artificial intelligence: a modern approach, 3rd edn. Pearson Education Limited, LondonzbMATHGoogle Scholar
  78. SAE J3016 (2016) Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems. SAE, WarrendaleGoogle Scholar
  79. Santana E, Hotz G (2016) Learning a driving simulator. Accessed April 2018
  80. Schoenick C, Clark P, Tafjord O, Turney P, Etzioni O (2017) Moving beyond the Turing test with the Allen AI science challenge. Commun ACM 60(9):60–64CrossRefGoogle Scholar
  81. Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T, Leach M, Kavukcuoglu K, Graepel T, Hassabis D (2016) Mastering the game of Go with deep neural networks and tree search. Nature 529(7587):484–489CrossRefGoogle Scholar
  82. Silver D, Hubert T, Schrittwieser J, Antonoglou I, Lai M, Guez A, Lanctot M, Sifre L, Kumaran D, Graepel T, Lillicrap T, Simonyan K, Hassabis D (2017a) Mastering Chess and Shogi by self-play with a general reinforcement learning algorithm. Accessed April 2018
  83. Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A, Chen Y, Lillicrap T, Hui F, Sifre L, van den Driessche G, Graepel T, Hassabis D (2017b) Mastering the game of Go without human knowledge. Nature 550:354–359CrossRefGoogle Scholar
  84. Srinivasan B, Parthasarathi R (2017) A survey of imperatives and action representation formalisms. Artif Intell Rev 48(2):263–297CrossRefGoogle Scholar
  85. Sternberg RJ (1985) Beyond IQ: a triarchic theory of human intelligence. Cambridge University Press, CambridgeGoogle Scholar
  86. Sternberg RJ, Davidson JE (1983) Insight in the gifted. Educ Psychol 18(1):51–57CrossRefGoogle Scholar
  87. Thornton SM, Pan S, Erlien SM, Gerdes JC (2017) Incorporating ethical considerations into automated vehicle control. IEEE Trans Intell Transp Syst 18(6):1429–1439Google Scholar
  88. Tong Y, Zhao L, Li L, Zhang Y (2015) Stochastic programming model for oversaturated intersection signal timing. Transp Res Part C 58:474–486CrossRefGoogle Scholar
  89. Turing AM (1950) Computing machinery and intelligence. Mind 59(236):433–460MathSciNetCrossRefGoogle Scholar
  90. Veeravasarapu VSR, Hota RN, Rothkopf C, Visvanathan R (2015) Simulations for validation of vision systems. Comput Sci.
  91. Vinge V (1993) The coming technological singularity: how to survive in the post-human era. In: Landis GA (ed) Vision-21: interdisciplinary science and engineering in the ear of cyberspace. NASA Publication, CP-10129, Washington, pp 11–22Google Scholar
  92. von Ahn L, Blum M, Hopper NJ, Langford J (2003) CAPTCHA: using hard AI problems for security. In: Proceedings of international conference on the theory and applications of cryptographic techniques, pp 294–311Google Scholar
  93. Wagner M, Koopman P (2015) A philosophy for developing trust in self-driving cars. In: Meyer G, Beiker S (eds) Road vehicle automation 2. Lecture notes in mobility. Springer, Cham. Google Scholar
  94. Wang FY, Zhang JJ, Zheng X et al (2016) Where does AlphaGo go: from church-turing thesis to AlphaGo thesis and beyond. IEEE/CAA J Automatica Sin 3:113–120. CrossRefGoogle Scholar
  95. Wang L (2016) Directions 2017: BeiDou’s road to global service. GPS WorldGoogle Scholar
  96. Wang FY, Wang X, Li L, Li L (2016a) Steps toward parallel intelligence. IEEE/CAA J Autom Sin 3(4):345–348MathSciNetCrossRefGoogle Scholar
  97. Wang X, Zheng X, Zhang Q, Wang T, Shen D (2016b) Crowdsourcing in ITS: the state of the work and the networking. IEEE Trans Intell Transp Syst 17(6):1596–1605CrossRefGoogle Scholar
  98. Wang K, Gou C, Zheng N, Rehg JM, Wang FY (2017a) Parallel vision for perception and understanding of complex scenes: methods, framework, and perspectives. Artif Intell Rev 1:1–31Google Scholar
  99. Wang X, Jiang R, Li L, Lin Y, Zheng X, Wang FY (2017b) Capturing car-following behaviors by deep learning. IEEE Trans Intell Transp Syst.
  100. Watzenig D, Horn M (2017a) Automated driving: safer and more efficient future driving. Springer, ChamCrossRefGoogle Scholar
  101. Watzenig D, Horn M (2017b) Automated driving: safer and more efficient future driving. Springer, ChamCrossRefGoogle Scholar
  102. You J. (2017) Deep learning based lane departure detection for automated vehicles. Bachelor Thesis, Tsinghua UniversityGoogle Scholar
  103. Zhao D, Huang X, Peng H, Lam H, Leblanc DJ (2017) Accelerated evaluation of automated vehicles in car-following maneuvers. IEEE Trans Intell Transp Syst.
  104. Zheng NN, Tang S, Cheng H, Li Q, Lai G, Wang FY (2004) Toward intelligent driver-assistance and safety warning systems. IEEE Intell Syst 19(2):8–11CrossRefGoogle Scholar
  105. Zheng NN, Liu ZY, Ren PJ, Ma YQ, Chen ST, Yu SY, Xue JR, Chen BD, Wang FY (2017) Hybrid-augmented intelligence: collaboration and cognition. Front Inf Technol Electron Eng 18(2):153–179CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V., part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Automation, BNRistTsinghua UniversityBeijingChina
  2. 2.The State Key Laboratory for Management and Control of Complex Systems, Institute of AutomationChinese Academy of SciencesBeijingChina
  3. 3.Institute of Artificial Intelligence and RoboticsXi’an Jiaotong UniversityXi’anChina
  4. 4.Department of Mechanical and Mechatronics EngineeringUniversity of WaterlooWaterlooCanada
  5. 5.Qingdao Academy of Intelligent IndustriesQingdaoChina
  6. 6.VIPioneers (HuiTuo) Inc.QingdaoChina

Personalised recommendations