Artificial intelligence test: a case study of intelligent vehicles

Abstract

To meet the urgent requirement of reliable artificial intelligence applications, we discuss the tight link between artificial intelligence and intelligence test in this paper. We highlight the role of tasks in intelligence test for all kinds of artificial intelligence. We explain the necessity and difficulty of describing tasks for intelligence test, checking all the tasks that may encounter in intelligence test, designing simulation-based test, and setting appropriate test performance evaluation indices. As an example, we present how to design reliable intelligence test for intelligent vehicles. Finally, we discuss the future research directions of intelligence test.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  1. A Tragic Loss (2016) https://www.tesla.com/blog/tragic-loss. Accessed April 2018

  2. Ackerman E (2014) A better test than Turing. IEEE Spectr 51(10):20–21

    Article  Google Scholar 

  3. Ammann P, Jeff O (2017) Introduction to software testing, 2nd edn. Cambridge University Press, Cambridge

    Google Scholar 

  4. Argall BD, Chernova S, Veloso M, Browning B (2009) A survey of robot learning from demonstration. Robot Auton Syst 57(5):469–483

    Article  Google Scholar 

  5. Bagnell JA (2015) An invitation to imitation. Technical Report, CMU-RI-TR-15-08, Robotics Institute, Carnegie Mellon University

  6. Black R (2009) Managing the testing process: practical tools and techniques for managing hardware and software testing. Wiley, Hoboken

    Google Scholar 

  7. Boehm BW (1988) A spiral model of software development and enhancement. IEEE Comput 21(5):61–72

    Article  Google Scholar 

  8. Bradley AR, Manna Z (2007) The calculus of computation: decision procedures with applications to verification. Springer, Berlin

    Google Scholar 

  9. Broggi A, Buzzoni M, Debattisti S, Grisleri P, Laghi MC, Medici P, Versari P (2013) Extensive tests of autonomous driving technologies. IEEE Trans Intell Transp Syst 14(3):1403–1415

    Article  Google Scholar 

  10. Broggi A, Cerri P, Debattisti S, Laghi MC, Medici P, Molinari D, Panciroli M, Prioletti A (2015) PROUD—public road urban driverless-car test. IEEE Trans Intell Transp Syst 16(6):3508–3519

    Article  Google Scholar 

  11. Brown N, Sandholm T (2017) Safe and nested subgame solving for imperfect-information games. https://arxiv.org/abs/1705.02955. Accessed April 2018

  12. Browne CB, Powley E, Whitehouse D, Lucas SM, Cowling PI, Rohlfshagen P, Tavener S, Perez D, Samothrakis S, Colton S (2012) A survey of monte carlo tree search methods. IEEE Trans Comput Intell AI Games 4(1):1–43

    Article  Google Scholar 

  13. Buehler M, Iagnemma K, Singh S (eds) (2009) The DARPA urban challenge. Springer, Berlin

    Google Scholar 

  14. Butakov VA, Ioannou P (2015) Personalized driver/vehicle lane change models for ADAS. IEEE Trans Veh Technol 64(10):4422–4431

    Article  Google Scholar 

  15. Campbell M, Egerstedt M, How JP, Murray RM (2010) Autonomous driving in urban environments: approaches, lessons and challenges. Philos Trans R Soc A 368(1928):4649–4672

    Article  Google Scholar 

  16. Chen Z, Liu B (2016) Lifelong machine learning. Morgan & Claypool Publishers, San Rafael

    Google Scholar 

  17. Cheng PCH (2016) What constitutes an effective representation? In: Jamnik M, Uesaka Y, Elzer Schwartz S (eds) Diagrammatic representation and inference: proceedings from the 9th international conference, diagrams 2016, vol 9781. Lecture notes in computer science. Springer, Berlin

    Google Scholar 

  18. Classen S, Nichols AL, McPeek R, Breinerd JF (2011) Personality as a predictor of driving performance: an exploratory study. Transp Res F Traffic Psychol Behav 14(5):381–389

    Article  Google Scholar 

  19. Coulom R (2008) Whole-history rating: a Bayesian rating system for players of time-varying strength. In: Proceedings of international conference on computers and games, pp 113–124

  20. DARPA Grand Challenge, DARPA Urban Challenge (2004–2007) http://archive.darpa.mil/grandchallenge/. Accessed April 2018

  21. Ding Z, Jiang C, Zhou MC (2013) Design, analysis and verification of real-time systems based on time Petri net refinement. ACM Transactions in Embedded Computing Systems 12:4:1–4:18. https://doi.org/10.1145/2406336.2406340

    Article  Google Scholar 

  22. Elo AE (1978) The rating of chessplayers, past and present. Arco Publishing, New York

    Google Scholar 

  23. Eskandarian A (ed) (2012) Handbook of intelligent vehicles. Springer, Berlin

    Google Scholar 

  24. Evtimov I, Eykholt K, Fernandes E, Kohno T, Li B, Prakash A, Rahmati A, Song D (2017) Robust physical-world attacks on machine learning models. https://arxiv.org/abs/1707.08945. Accessed April 2018

  25. Fagnant DJ, Kockelman K (2015) Preparing a nation for autonomous vehicles: opportunities, barriers and policy recommendations. Transp Res A Policy Practice 77:167–181

    Article  Google Scholar 

  26. Fisher DL, Lohrenz M, Moore D, Nadler ED, Pollard JK (2016) Humans and intelligent vehicles: the hope, the help, and the harm. IEEE Trans Intell Veh 1(1):56–67

    Article  Google Scholar 

  27. Gaidon A, Wang Q, Cabon Y, Vig E (2016) Virtual worlds as proxy for multi-object tracking analysis. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4340–4349

  28. Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 2414–2423

  29. George D, Lehrach W, Kansky K, Lázaro-Gredilla M, Laan C, Marthi B, Lou X, Meng Z, Liu Y, Wang H, Lavin A, Phoenix DS (2017) A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs. Science. https://doi.org/10.1126/science.aag2612

    Google Scholar 

  30. Goodall NJ (2014) Ethical decision making during automated vehicle crashes. Transp Res Rec 2424:58–65

    Article  Google Scholar 

  31. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Proc Adv Neural Inf Process Syst 27:2672–2680

    Google Scholar 

  32. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge

    Google Scholar 

  33. Greengard S (2017) Gaming machine learning. Commun ACM 60(12):14–16

    Article  Google Scholar 

  34. GTSDB, The German Traffic Sign Recognition Benchmark and the German Traffic Sign Detection Benchmark (2014) http://benchmark.ini.rub.de/?section=home&subsection=news. Accessed April 2018

  35. Harari YN (2017) Reboot for the AI revolution. Nature 550:324–327

    Article  Google Scholar 

  36. Hernández-Orallo J (2017) Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement 48(3):397–447

    Article  Google Scholar 

  37. Ho J, Ermon S (2017) Generative adversarial imitation learning. https://arxiv.org/abs/1606.03476. Accessed April 2018

  38. Huang WL, Wen D, Geng J, Zheng NN (2014) Task-specific performance evaluation of ugvs: case studies at the IFVC. IEEE Trans Intell Transp Syst 15(5):1969–1979

    Article  Google Scholar 

  39. Huizinga D, Adam K (2007) Automated defect prevention: best practices in software management. Wiley, Hoboken

    Google Scholar 

  40. IBM, Deep Blue - Overview (1997) IBM Research. http://www-03.ibm.com/ibm/history/ibm100/us/en/icons/deepblue/. Accessed April 2018

  41. ImageNet (2016) http://image-net.org. Accessed April 2018

  42. Karp RM (1972) Reducibility among combinatorial problems. In: Miller RE, Thacher JW (eds) Complexity of computer computation. Plenum Press, New York, pp 85–103

    Google Scholar 

  43. Karpathy A (2017) Software 2.0. https://medium.com/@karpathy/software-2-0-a64152b37c35. Accessed April 2018

  44. Koopman P, Wagner M (2017) Autonomous vehicle safety: an interdisciplinary challenge. IEEE Intell Transp Syst Mag 9(1):90–96

    Article  Google Scholar 

  45. Kroening D, Strichman O (2016) Decision procedures: an algorithmic point of view, 2nd edn. Springer, Berlin

    Google Scholar 

  46. Kuefler A, Morton J, Wheeler T, Kochenderfer M (2017) Imitating driver behavior with generative adversarial networks. In: Proceedings of IEEE intelligent vehicles symposium, pp 204–211

  47. Heule MJH, Kullmann O (2017) The science of brute force. Commun ACM 60(8):70–79

    Article  Google Scholar 

  48. Kumfer W, Burgess R (2015) Investigation into the role of rational ethics in crashes of automated vehicles. Transp Res Rec 2489:130–136

    Article  Google Scholar 

  49. Kurzweil R (2005) The singularity is near. Viking Press, New York

    Google Scholar 

  50. Lamb E (2016) Maths proof smashes size record: supercomputer produces a 200-terabyte proof—but is it really mathematics? Nature 534(7605):17–19

    Article  Google Scholar 

  51. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  52. Lefèvre S, Carvalho A, Gao Y, Tseng HE, Borrellia F (2015) Driver models for personalised driving assistance. Veh Syst Dyn 53(12):1705–1720

    Article  Google Scholar 

  53. Levesque HJ (2014) On our best behavior. Artif Intell 212:27–35

    Article  MATH  Google Scholar 

  54. Levesque HJ (2017) Common sense, the Turing test, and the quest for real AI. MIT Press, Cambridge

    Google Scholar 

  55. Li L, Wang FY (2007) Advanced motion control and sensing for intelligent vehicles. Springer, New York

    Google Scholar 

  56. Li L, Wen D, Zheng NN, Shen LC (2012) Cognitive cars: a new frontier for ADAS research. IEEE Trans Intell Transp Syst 13(1):395–407

    Article  Google Scholar 

  57. Li L, Huang WL, Liu Y, Zheng NN, Wang FY (2016a) Intelligence testing for autonomous vehicles: a new approach. IEEE Trans Intell Veh 1(2):158–166

    Article  Google Scholar 

  58. Li L, Lv Y, Wang FY (2016b) Traffic signal timing via deep reinforcement learning. IEEE/CAA J Autom Sin 3(3):247–254

    MathSciNet  Article  Google Scholar 

  59. Li L, Lin Y, Zheng NN, Wang FY (2017) Parallel learning: a perspective and a framework. IEEE/CAA J Autom Sin 4(3):389–395

    MathSciNet  Article  Google Scholar 

  60. Liao R (2017) Tencent discovers major loopholes in Google’s AI platform TensorFlow. https://technode.com/2017/12/18/tencent-tensorflow/. Accessed April 2018

  61. Licato J, Zhang Z (2017) Evaluating representational systems in artificial intelligence. Artif Intell Rev. https://doi.org/10.1007/s10462-017-9598-7

    Google Scholar 

  62. Liu MY, Breuel T, Kautz J (2017) Unsupervised image-to-image translation networks. https://arxiv.org/abs/1703.00848. Accessed April 2018

  63. Mackintosh NJ (2011) IQ and human intelligence, 2nd edn. Oxford University Press, Oxford

    Google Scholar 

  64. Maurer M, Gerdes JC, Lenz B, Winner H (eds) (2015) Autonomous driving: technical, legal and social aspects. Springer, Berlin

    Google Scholar 

  65. Mcguire G, Tugemann B, Civario G (2014) There is no 16-clue sudoku: solving the sudoku minimum number of clues problem via hitting set enumeration. Exp Math 23(2):190–217

    MathSciNet  Article  MATH  Google Scholar 

  66. Merel J, Tassa Y, TB D, Srinivasan S, Lemmon J, Wang Z, Wayne G, Heess N (2017) Learning human behaviors from motion capture by adversarial imitation. https://arxiv.org/abs/1707.02201. Accessed April 2018

  67. Minsky ML (ed) (1968) Semantic information processing. MIT Press, Cambridge

    Google Scholar 

  68. Moravčík M, Schmid M, Burch N, Lisý V, Morrill D, Bard N, Davis T, Waugh K, Johanson M, Bowling M (2017) DeepStack: expert-level artificial intelligence in heads-up no-limit poker. Science 356:508–513

    MathSciNet  Article  Google Scholar 

  69. Newell A, Simon HA (1976) Computer science as empirical inquiry: symbols and search. Commun ACM CACM Homepage 19(3):113–126

    MathSciNet  Article  Google Scholar 

  70. Ohlsson S, Sloan RH, Turán G, Urasky A (2017) Measuring an artificial intelligence system’s performance on a verbal IQ test for young children. J Exp Theor Artif Intell 29(4):679–693

    Article  Google Scholar 

  71. Raccoon L (1997) Fifty years of progress in software engineering. ACM SIGSOFT Softw Eng Notes 22(1):88–104

    Article  Google Scholar 

  72. Redmon J, Farhadi A (2016) YOLO9000: better, faster, stronger. https://arxiv.org/abs/1612.08242. Accessed April 2018

  73. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. https://arxiv.org/abs/1506.02640. Accessed April 2018

  74. Richter SR, Vineet V, Roth S, Koltun V (2016) Playing for data: ground truth from computer games. In: European conference on computer vision, pp 102–118

  75. Rindermann H, Becker D, Coyle TR (2016) Survey of expert opinion on intelligence: causes of international differences in cognitive ability tests. Front Psychol. https://doi.org/10.3389/fpsyg.2016.00399

    Google Scholar 

  76. Ros G, Sellart L, Materzynska J, Vazquez D, Lopez AM (2016) The SYNTHIA dataset: a large collection of synthetic images for semantic segmentation of urban scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3234–3243

  77. Russell S, Norvig P (2010) Artificial intelligence: a modern approach, 3rd edn. Pearson Education Limited, London

    Google Scholar 

  78. SAE J3016 (2016) Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems. SAE, Warrendale

    Google Scholar 

  79. Santana E, Hotz G (2016) Learning a driving simulator. https://arxiv.org/abs/1608.01230. Accessed April 2018

  80. Schoenick C, Clark P, Tafjord O, Turney P, Etzioni O (2017) Moving beyond the Turing test with the Allen AI science challenge. Commun ACM 60(9):60–64

    Article  Google Scholar 

  81. Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T, Leach M, Kavukcuoglu K, Graepel T, Hassabis D (2016) Mastering the game of Go with deep neural networks and tree search. Nature 529(7587):484–489

    Article  Google Scholar 

  82. Silver D, Hubert T, Schrittwieser J, Antonoglou I, Lai M, Guez A, Lanctot M, Sifre L, Kumaran D, Graepel T, Lillicrap T, Simonyan K, Hassabis D (2017a) Mastering Chess and Shogi by self-play with a general reinforcement learning algorithm. https://arxiv.org/abs/1712.01815. Accessed April 2018

  83. Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A, Chen Y, Lillicrap T, Hui F, Sifre L, van den Driessche G, Graepel T, Hassabis D (2017b) Mastering the game of Go without human knowledge. Nature 550:354–359

    Article  Google Scholar 

  84. Srinivasan B, Parthasarathi R (2017) A survey of imperatives and action representation formalisms. Artif Intell Rev 48(2):263–297

    Article  Google Scholar 

  85. Sternberg RJ (1985) Beyond IQ: a triarchic theory of human intelligence. Cambridge University Press, Cambridge

    Google Scholar 

  86. Sternberg RJ, Davidson JE (1983) Insight in the gifted. Educ Psychol 18(1):51–57

    Article  Google Scholar 

  87. Thornton SM, Pan S, Erlien SM, Gerdes JC (2017) Incorporating ethical considerations into automated vehicle control. IEEE Trans Intell Transp Syst 18(6):1429–1439

    Google Scholar 

  88. Tong Y, Zhao L, Li L, Zhang Y (2015) Stochastic programming model for oversaturated intersection signal timing. Transp Res Part C 58:474–486

    Article  Google Scholar 

  89. Turing AM (1950) Computing machinery and intelligence. Mind 59(236):433–460

    MathSciNet  Article  Google Scholar 

  90. Veeravasarapu VSR, Hota RN, Rothkopf C, Visvanathan R (2015) Simulations for validation of vision systems. Comput Sci. https://arxiv.org/abs/1512.01030

  91. Vinge V (1993) The coming technological singularity: how to survive in the post-human era. In: Landis GA (ed) Vision-21: interdisciplinary science and engineering in the ear of cyberspace. NASA Publication, CP-10129, Washington, pp 11–22

    Google Scholar 

  92. von Ahn L, Blum M, Hopper NJ, Langford J (2003) CAPTCHA: using hard AI problems for security. In: Proceedings of international conference on the theory and applications of cryptographic techniques, pp 294–311

  93. Wagner M, Koopman P (2015) A philosophy for developing trust in self-driving cars. In: Meyer G, Beiker S (eds) Road vehicle automation 2. Lecture notes in mobility. Springer, Cham. https://doi.org/10.1007/978-3-319-19078-5_14

    Google Scholar 

  94. Wang FY, Zhang JJ, Zheng X et al (2016) Where does AlphaGo go: from church-turing thesis to AlphaGo thesis and beyond. IEEE/CAA J Automatica Sin 3:113–120. https://doi.org/10.1109/JAS.2016.7471613

    Article  Google Scholar 

  95. Wang L (2016) Directions 2017: BeiDou’s road to global service. GPS World

  96. Wang FY, Wang X, Li L, Li L (2016a) Steps toward parallel intelligence. IEEE/CAA J Autom Sin 3(4):345–348

    MathSciNet  Article  Google Scholar 

  97. Wang X, Zheng X, Zhang Q, Wang T, Shen D (2016b) Crowdsourcing in ITS: the state of the work and the networking. IEEE Trans Intell Transp Syst 17(6):1596–1605

    Article  Google Scholar 

  98. Wang K, Gou C, Zheng N, Rehg JM, Wang FY (2017a) Parallel vision for perception and understanding of complex scenes: methods, framework, and perspectives. Artif Intell Rev 1:1–31

    Google Scholar 

  99. Wang X, Jiang R, Li L, Lin Y, Zheng X, Wang FY (2017b) Capturing car-following behaviors by deep learning. IEEE Trans Intell Transp Syst. http://ieeexplore.ieee.org/document/7970189/

  100. Watzenig D, Horn M (2017a) Automated driving: safer and more efficient future driving. Springer, Cham

    Google Scholar 

  101. Watzenig D, Horn M (2017b) Automated driving: safer and more efficient future driving. Springer, Cham

    Google Scholar 

  102. You J. (2017) Deep learning based lane departure detection for automated vehicles. Bachelor Thesis, Tsinghua University

  103. Zhao D, Huang X, Peng H, Lam H, Leblanc DJ (2017) Accelerated evaluation of automated vehicles in car-following maneuvers. IEEE Trans Intell Transp Syst. http://ieeexplore.ieee.org/document/7933977/

  104. Zheng NN, Tang S, Cheng H, Li Q, Lai G, Wang FY (2004) Toward intelligent driver-assistance and safety warning systems. IEEE Intell Syst 19(2):8–11

    Article  Google Scholar 

  105. Zheng NN, Liu ZY, Ren PJ, Ma YQ, Chen ST, Yu SY, Xue JR, Chen BD, Wang FY (2017) Hybrid-augmented intelligence: collaboration and cognition. Front Inf Technol Electron Eng 18(2):153–179

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants 91520301 and 61533019, and the Beijing Municipal Science and Technology Project (No. D171100000317002).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Fei-Yue Wang.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Li, L., Lin, Y., Zheng, N. et al. Artificial intelligence test: a case study of intelligent vehicles. Artif Intell Rev 50, 441–465 (2018). https://doi.org/10.1007/s10462-018-9631-5

Download citation

Keywords

  • Artificial intelligence
  • Intelligence test
  • Turing test
  • Simulation test