Artificial Intelligence Review

, Volume 48, Issue 3, pp 397–447 | Cite as

Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement

  • José Hernández-OralloEmail author


The evaluation of artificial intelligence systems and components is crucial for the progress of the discipline. In this paper we describe and critically assess the different ways AI systems are evaluated, and the role of components and techniques in these systems. We first focus on the traditional task-oriented evaluation approach. We identify three kinds of evaluation: human discrimination, problem benchmarks and peer confrontation. We describe some of the limitations of the many evaluation schemes and competitions in these three categories, and follow the progression of some of these tests. We then focus on a less customary (and challenging) ability-oriented evaluation approach, where a system is characterised by its (cognitive) abilities, rather than by the tasks it is designed to solve. We discuss several possibilities: the adaptation of cognitive tests used for humans and animals, the development of tests derived from algorithmic information theory or more integrated approaches under the perspective of universal psychometrics. We analyse some evaluation tests from AI that are better positioned for an ability-oriented evaluation and discuss how their problems and limitations can possibly be addressed with some of the tools and ideas that appear within the paper. Finally, we enumerate a series of lessons learnt and generic guidelines to be used when an AI evaluation scheme is under consideration.


AI evaluation AI competitions Machine intelligence Cognitive abilities Universal psychometrics Turing test 



I thank the organisers of the AEPIA Summer School On Artificial Intelligence, held in September 2014, for giving me the opportunity to give a lecture on ‘AI Evaluation’. This paper was born out of and evolved through that lecture. The information about many benchmarks and competitions discussed in this paper have been contrasted with information from and discussions with many people: M. Bedia, A. Cangelosi, C. Dimitrakakis, I. GarcÍa-Varea, Katja Hofmann, W. Langdon, E. Messina, S. Mueller, M. Siebers and C. Soares. Figure 4 is courtesy of F. Martínez-Plumed. Finally, I thank the anonymous reviewers, whose comments have helped to significantly improve the balance and coverage of the paper. This work has been partially supported by the EU (FEDER) and the Spanish MINECO under Grants TIN 2013-45732-C4-1-P, TIN 2015-69175-C4-1-R and by Generalitat Valenciana PROMETEOII2015/013.


  1. Abel D, Agarwal A, Diaz F, Krishnamurthy A, Schapire RE (2016) Exploratory gradient boosting for reinforcement learning in complex domains. arXiv preprint arXiv:1603.04119
  2. Adams S, Arel I, Bach J, Coop R, Furlan R, Goertzel B, Hall JS, Samsonovich A, Scheutz M, Schlesinger M, Shapiro SC, Sowa J (2012) Mapping the landscape of human-level artificial general intelligence. AI Mag 33(1):25–42CrossRefGoogle Scholar
  3. Adams SS, Banavar G, Campbell M (2016) I-athlon: towards a multi-dimensional Turing test. AI Mag 37(1):78–84CrossRefGoogle Scholar
  4. Alcalá J, Fernández A, Luengo J, Derrac J, García S, Sánchez L, Herrera F (2010) Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework. J Mult Valued Logic Soft Comput 17:255–287Google Scholar
  5. Alexander JRM, Smales S (1997) Intelligence, learning and long-term memory. Personal Individ Differ 23(5):815–825CrossRefGoogle Scholar
  6. Alpcan T, Everitt T, Hutter M (2014) Can we measure the difficulty of an optimization problem? In: IEEE information theory workshop (ITW)Google Scholar
  7. Alur R, Bodik R, Juniwal G, Martin MMK, Raghothaman M, Seshia SA, Singh R, Solar-Lezama A, Torlak E, Udupa A (2013) Syntax-guided synthesis. In: Formal methods in computer-aided design (FMCAD), 2013, IEEE, pp 1–17Google Scholar
  8. Alvarado N, Adams SS, Burbeck S, Latta C (2002) Beyond the Turing test: performance metrics for evaluating a computer simulation of the human mind. In: Proceedings of the 2nd international conference on development and learning, IEEE, pp 147–152Google Scholar
  9. Amigoni F, Bastianelli E, Berghofer J, Bonarini A, Fontana G, Hochgeschwender N, Iocchi L, Kraetzschmar G, Lima P, Matteucci M, Miraldo P, Nardi D, Schiaffonati V (2015) Competitions for benchmarking: task and functionality scoring complete performance assessment. IEEE Robot Autom Mag 22(3):53–61CrossRefGoogle Scholar
  10. Anderson J, Lebiere C (2003) The Newell test for a theory of cognition. Behav Brain Sci 26(5):587–601Google Scholar
  11. Anderson J, Baltes J, Cheng CT (2011) Robotics competitions as benchmarks for AI research. Knowl Eng Rev 26(01):11–17CrossRefGoogle Scholar
  12. Arel I, Rose DC, Karnowski TP (2010) Deep machine learning—a new frontier in artificial intelligence research. IEEE Comput Intell Mag 5(4):13–18CrossRefGoogle Scholar
  13. Asada M, Hosoda K, Kuniyoshi Y, Ishiguro H, Inui T, Yoshikawa Y, Ogino M, Yoshida C (2009) Cognitive developmental robotics: a survey. IEEE Trans Auton Ment Dev 1(1):12–34CrossRefGoogle Scholar
  14. Aziz H, Brill M, Fischer F, Harrenstein P, Lang J, Seedig HG (2015) Possible and necessary winners of partial tournaments. J Artif Intell Res 54:493–534MathSciNetzbMATHGoogle Scholar
  15. Bache K, Lichman M (2013) UCI machine learning repository.
  16. Bagnall AJ, Zatuchna ZV (2005) On the classification of maze problems. In: Bull L, Kovacs T (eds) Foundations of learning classifier system. Studies in fuzziness and soft computing, vol. 183, Springer, pp 305–316.
  17. Baldwin D, Yadav SB (1995) The process of research investigations in artificial intelligence - a unified view. IEEE Trans Syst Man Cybern 25(5):852–861CrossRefGoogle Scholar
  18. Bellemare MG, Naddaf Y, Veness J, Bowling M (2013) The arcade learning environment: an evaluation platform for general agents. J Artif Intell Res 47:253–279Google Scholar
  19. Besold TR (2014) A note on chances and limitations of psychometric ai. In: KI 2014: advances in artificial intelligence. Springer, pp 49–54Google Scholar
  20. Biever C (2011) Ultimate IQ: one test to rule them all. New Sci 211(2829, 10 September 2011):42–45CrossRefGoogle Scholar
  21. Borg M, Johansen SS, Thomsen DL, Kraus M (2012) Practical implementation of a graphics Turing test. In: Advances in visual computing. Springer, pp 305–313Google Scholar
  22. Boring EG (1923) Intelligence as the tests test it. New Repub 35–37Google Scholar
  23. Bostrom N (2014) Superintelligence: paths, dangers, strategies. Oxford University Press, OxfordGoogle Scholar
  24. Brazdil P, Carrier CG, Soares C, Vilalta R (2008) Metalearning: applications to data mining. Springer, New YorkzbMATHGoogle Scholar
  25. Bringsjord S (2011) Psychometric artificial intelligence. J Exp Theor Artif Intell 23(3):271–277CrossRefGoogle Scholar
  26. Bringsjord S, Schimanski B (2003) What is artificial intelligence? Psychometric AI as an answer. In: International joint conference on artificial intelligence, pp 887–893Google Scholar
  27. Brundage M (2016) Modeling progress in ai. AAAI 2016 Workshop on AI, Ethics, and SocietyGoogle Scholar
  28. Buchanan BG (1988) Artificial intelligence as an experimental science. Springer, New YorkCrossRefGoogle Scholar
  29. Buhrmester M, Kwang T, Gosling SD (2011) Amazon’s mechanical turk a new source of inexpensive, yet high-quality, data? Perspect Psychol Sci 6(1):3–5CrossRefGoogle Scholar
  30. Bursztein E, Aigrain J, Moscicki A, Mitchell JC (2014) The end is nigh: generic solving of text-based captchas. In: Proceedings of the 8th USENIX conference on Offensive Technologies, USENIX Association, p 3Google Scholar
  31. Campbell M, Hoane AJ, Hsu F (2002) Deep Blue. Artif Intell 134(1–2):57–83zbMATHCrossRefGoogle Scholar
  32. Cangelosi A, Schlesinger M, Smith LB (2015) Developmental robotics: from babies to robots. MIT Press, CambridgeGoogle Scholar
  33. Caputo B, Müller H, Martinez-Gomez J, Villegas M, Acar B, Patricia N, Marvasti N, Üsküdarlı S, Paredes R, Cazorla M et al (2014) Imageclef 2014: overview and analysis of the results. In: Information access evaluation. Multilinguality, multimodality, and interaction, Springer, pp 192–211Google Scholar
  34. Carlson A, Betteridge J, Kisiel B, Settles B, Hruschka ER Jr, Mitchell TM (2010) Toward an architecture for never-ending language learning. In: AAAI, vol 5, p 3Google Scholar
  35. Carroll JB (1993) Human cognitive abilities: a survey of factor-analytic studies. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  36. Caruana R (1997) Multitask learning. Mach Learn 28(1):41–75MathSciNetCrossRefGoogle Scholar
  37. Chaitin GJ (1982) Gödel’s theorem and information. Int J Theor Phys 21(12):941–954zbMATHCrossRefGoogle Scholar
  38. Chandrasekaran B (1990) What kind of information processing is intelligence? In: The foundation of artificial intelligence—a sourcebook. Cambridge University Press, pp 14–46Google Scholar
  39. Chater N (1999) The search for simplicity: a fundamental cognitive principle? Q J Exp Psychol Sect A 52(2):273–302CrossRefGoogle Scholar
  40. Chater N, Vitányi P (2003) Simplicity: a unifying principle in cognitive science? Trends Cogn Sci 7(1):19–22CrossRefGoogle Scholar
  41. Chu Z, Gianvecchio S, Wang H, Jajodia S (2010) Who is tweeting on twitter: human, bot, or cyborg? In: Proceedings of the 26th annual computer security applications conference, ACM, pp 21–30Google Scholar
  42. Cochran WG (2007) Sampling techniques. Wiley, New YorkzbMATHGoogle Scholar
  43. Cohen PR, Howe AE (1988) How evaluation guides AI research: the message still counts more than the medium. AI Mag 9(4):35Google Scholar
  44. Cohen Y (2013) Testing and cognitive enhancement. Technical repor, National Institute for Testing and Evaluation, Jerusalem, IsraelGoogle Scholar
  45. Conrad JG, Zeleznikow J (2013) The significance of evaluation in AI and law: a case study re-examining ICAIL proceedings. In: Proceedings of the 14th international conference on artificial intelligence and law, ACM, pp 186–191Google Scholar
  46. Conrad JG, Zeleznikow J (2015) The role of evaluation in ai and law. In: Proceedings of the 15th international conference on artificial intelligence and law, pp 181–186Google Scholar
  47. Deary IJ, Der G, Ford G (2001) Reaction times and intelligence differences: a population-based cohort study. Intelligence 29(5):389–399CrossRefGoogle Scholar
  48. Decker KS, Durfee EH, Lesser VR (1989) Evaluating research in cooperative distributed problem solving. Distrib Artif Intell 2:487–519Google Scholar
  49. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30MathSciNetzbMATHGoogle Scholar
  50. Detterman DK (2011) A challenge to Watson. Intelligence 39(2–3):77–78CrossRefGoogle Scholar
  51. Dimitrakakis C (2016) Personal communicationGoogle Scholar
  52. Dimitrakakis C, Li G, Tziortziotis N (2014) The reinforcement learning competition 2014. AI Mag 35(3):61–65CrossRefGoogle Scholar
  53. Dowe DL (2013) Introduction to Ray Solomonoff 85th memorial conference. In: Dowe DL (ed) Algorithmic probability and friends. Bayesian prediction and artificial intelligence, lecture notes in computer science, vol 7070. Springer, Berlin, pp 1–36CrossRefGoogle Scholar
  54. Dowe DL, Hajek AR (1997) A computational extension to the Turing Test. In: Proceedings of the 4th conference of the Australasian cognitive science society, University of Newcastle, NSW, AustraliaGoogle Scholar
  55. Dowe DL, Hajek AR (1998) A non-behavioural, computational extension to the Turing test. In: International conference on computational intelligence and multimedia applications (ICCIMA’98), Gippsland, Australia, pp 101–106Google Scholar
  56. Dowe DL, Hernández-Orallo J (2012) IQ tests are not for machines, yet. Intelligence 40(2):77–81CrossRefGoogle Scholar
  57. Dowe DL, Hernández-Orallo J (2014) How universal can an intelligence test be? Adapt Behav 22(1):51–69CrossRefGoogle Scholar
  58. Drummond C (2009) Replicability is not reproducibility: nor is it good science. In: Proceedings of the evaluation methods for machine learning workshop at the 26th ICML, Montreal, CanadaGoogle Scholar
  59. Drummond C, Japkowicz N (2010) Warning: statistical benchmarking is addictive. Kicking the habit in machine learning. J Exp Theor Artif Intell 22(1):67–80zbMATHCrossRefGoogle Scholar
  60. Duan Y, Chen X, Houthooft R, Schulman J, Abbeel P (2016) Benchmarking deep reinforcement learning for continuous control. arXiv preprint arXiv:1604.06778
  61. Eden AH, Moor JH, Soraker JH, Steinhart E (2013) Singularity hypotheses: a scientific and philosophical assessment. Springer, New YorkzbMATHGoogle Scholar
  62. Edmondson W (2012) The intelligence in ETI—what can we know? Acta Astronaut 78:37–42CrossRefGoogle Scholar
  63. Elo AE (1978) The rating of chessplayers, past and present, vol 3. Batsford, LondonGoogle Scholar
  64. Embretson SE, Reise SP (2000) Item response theory for psychologists. L. Erlbaum, HillsdaleGoogle Scholar
  65. Evans JM, Messina ER (2001) Performance metrics for intelligent systems. NIST Special Publication SP, pp 101–104Google Scholar
  66. Everitt T, Lattimore T, Hutter M (2014) Free lunch for optimisation under the universal distribution. In: 2014 IEEE Congress on evolutionary computation (CEC), IEEE, pp 167–174Google Scholar
  67. Falkenauer E (1998) On method overfitting. J Heuristics 4(3):281–287zbMATHCrossRefGoogle Scholar
  68. Feldman J (2003) Simplicity and complexity in human concept learning. Gen Psychol 38(1):9–15Google Scholar
  69. Ferrando PJ (2009) Difficulty, discrimination, and information indices in the linear factor analysis model for continuous item responses. Appl Psychol Meas 33(1):9–24MathSciNetCrossRefGoogle Scholar
  70. Ferrando PJ (2012) Assessing the discriminating power of item and test scores in the linear factor-analysis model. Psicológica 33:111–139Google Scholar
  71. Ferri C, Hernández-Orallo J, Modroiu R (2009) An experimental comparison of performance measures for classification. Pattern Recogn Lett 30(1):27–38CrossRefGoogle Scholar
  72. Ferrucci D, Brown E, Chu-Carroll J, Fan J, Gondek D, Kalyanpur AA, Lally A, Murdock J, Nyberg E, Prager J et al (2010) Building Watson: an overview of the DeepQA project. AI Mag 31(3):59–79CrossRefGoogle Scholar
  73. Fogel DB (1991) The evolution of intelligent decision making in gaming. Cybern Syst 22(2):223–236CrossRefGoogle Scholar
  74. Gaschnig J, Klahr P, Pople H, Shortliffe E, Terry A (1983) Evaluation of expert systems: issues and case studies. Build Exp Syst 1:241–278Google Scholar
  75. Geissman JR, Schultz RD (1988) Verification & validation. AI Exp 3(2):26–33Google Scholar
  76. Genesereth M, Love N, Pell B (2005) General game playing: overview of the AAAI competition. AI Mag 26(2):62Google Scholar
  77. Gerónimo D, López AM (2014) Datasets and benchmarking. In: Vision-based pedestrian protection systems for intelligent vehicles. Springer, pp 87–93Google Scholar
  78. Goertzel B, Pennachin C (eds) (2007) Artificial general intelligence. Springer, New YorkzbMATHGoogle Scholar
  79. Goertzel B, Arel I, Scheutz M (2009) Toward a roadmap for human-level artificial general intelligence: embedding HLAI systems in broad, approachable, physical or virtual contexts. Artif Gen Intell Roadmap InitiatGoogle Scholar
  80. Goldreich O, Vadhan S (2007) Special issue on worst-case versus average-case complexity editors’ foreword. Comput complex 16(4):325–330MathSciNetCrossRefGoogle Scholar
  81. Gordon BB (2007) Report on panel discussion on (re-)establishing or increasing collaborative links between artificial intelligence and intelligent systems. In: Messina ER, Madhavan R (eds) Proceedings of the 2007 workshop on performance metrics for intelligent systems, pp 302–303Google Scholar
  82. Gulwani S, Hernández-Orallo J, Kitzelmann E, Muggleton SH, Schmid U, Zorn B (2015) Inductive programming meets the real world. Commun ACM 58(11):90–99CrossRefGoogle Scholar
  83. Hand DJ (2004) Measurement theory and practice. A Hodder Arnold Publication, LondonzbMATHGoogle Scholar
  84. Hernández-Orallo J (2000a) Beyond the Turing test. J Logic Lang Inf 9(4):447–466MathSciNetzbMATHCrossRefGoogle Scholar
  85. Hernández-Orallo J (2000b) On the computational measurement of intelligence factors. In: Meystel A (ed) Performance metrics for intelligent systems workshop. National Institute of Standards and Technology, Gaithersburg, pp 1–8Google Scholar
  86. Hernández-Orallo J (2000c) Thesis: computational measures of information gain and reinforcement in inference processes. AI Commun 13(1):49–50Google Scholar
  87. Hernández-Orallo J (2010) A (hopefully) non-biased universal environment class for measuring intelligence of biological and artificial systems. In: Artificial general intelligence, 3rd International Conference. Atlantis Press, Extended report at, pp 182–183
  88. Hernández-Orallo J (2014) On environment difficulty and discriminating power. Auton Agents Multi-Agent Syst. 29(3):402–454. doi: 10.1007/s10458-014-9257-1
  89. Hernández-Orallo J, Dowe DL (2010) Measuring universal intelligence: towards an anytime intelligence test. Artif Intell 174(18):1508–1539MathSciNetCrossRefGoogle Scholar
  90. Hernández-Orallo J, Dowe DL (2013) On potential cognitive abilities in the machine kingdom. Minds Mach 23:179–210CrossRefGoogle Scholar
  91. Hernández-Orallo J, Minaya-Collado N (1998) A formal definition of intelligence based on an intensional variant of Kolmogorov complexity. In: Proceedings of international symposium of engineering of intelligent systems (EIS’98), ICSC Press, pp 146–163Google Scholar
  92. Hernández-Orallo J, Dowe DL, España-Cubillo S, Hernández-Lloreda MV, Insa-Cabrera J (2011) On more realistic environment distributions for defining, evaluating and developing intelligence. In: Schmidhuber J, Thórisson K, Looks M (eds) Artificial general intelligence, LNAI, vol 6830. Springer, New York, pp 82–91CrossRefGoogle Scholar
  93. Hernández-Orallo J, Flach P, Ferri C (2012a) A unified view of performance metrics: translating threshold choice into expected classification loss. J Mach Learn Res 13(1):2813–2869MathSciNetzbMATHGoogle Scholar
  94. Hernández-Orallo J, Insa-Cabrera J, Dowe DL, Hibbard B (2012b) Turing Tests with Turing machines. In: Voronkov A (ed) Turing-100, EPiC Series, vol 10, pp 140–156Google Scholar
  95. Hernández-Orallo J, Dowe DL, Hernández-Lloreda MV (2014) Universal psychometrics: measuring cognitive abilities in the machine kingdom. Cogn Syst Res 27:50–74CrossRefGoogle Scholar
  96. Hernández-Orallo J, Martínez-Plumed F, Schmid U, Siebers M, Dowe DL (2016) Computer models solving intelligence test problems: progress and implications. Artif Intell 230:74–107MathSciNetCrossRefGoogle Scholar
  97. Herrmann E, Call J, Hernández-Lloreda MV, Hare B, Tomasello M (2007) Humans have evolved specialized skills of social cognition: the cultural intelligence hypothesis. Science 317(5843):1360–1366CrossRefGoogle Scholar
  98. Hibbard B (2009) Bias and no free lunch in formal measures of intelligence. J Artif Gen Intell 1(1):54–61Google Scholar
  99. Hingston P (2010) A new design for a Turing Test for bots. In: 2010 IEEE symposium on computational intelligence and games (CIG), IEEE, pp 345–350Google Scholar
  100. Hingston P (2012) Believable bots: can computers play like people?. Springer, New YorkCrossRefGoogle Scholar
  101. Ho TK, Basu M (2002) Complexity measures of supervised classification problems. IEEE Trans Pattern Anal Mach Intell 24(3):289–300CrossRefGoogle Scholar
  102. Hutter M (2007) Universal algorithmic intelligence: a mathematical top\(\rightarrow \)down approach. In: Goertzel B, Pennachin C (eds) Artificial general intelligence, cognitive technologies. Springer, Berlin, pp 227–290CrossRefGoogle Scholar
  103. Igel C, Toussaint M (2005) A no-free-lunch theorem for non-uniform distributions of target functions. J Math Model Algorithms 3(4):313–322MathSciNetzbMATHCrossRefGoogle Scholar
  104. Insa-Cabrera J (2016) Towards a universal test of social intelligence. Ph.D. thesis, Departament de Sistemes Informátics i Computació, UPVGoogle Scholar
  105. Insa-Cabrera J, Dowe DL, España-Cubillo S, Hernández-Lloreda MV, Hernández-Orallo J (2011a) Comparing humans and ai agents. In: Schmidhuber J, Thórisson K, Looks M (eds) Artificial general intelligence, LNAI, vol 6830. Springer, New York, pp 122–132CrossRefGoogle Scholar
  106. Insa-Cabrera J, Dowe DL, Hernández-Orallo J (2011) Evaluating a reinforcement learning algorithm with a general intelligence test. In: Lozano JA, Gamez JM (eds) Current topics in artificial intelligence. CAEPIA 2011, LNAI series 7023. Springer, New YorkGoogle Scholar
  107. Insa-Cabrera J, Benacloch-Ayuso JL, Hernández-Orallo J (2012) On measuring social intelligence: experiments on competition and cooperation. In: Bach J, Goertzel B, Iklé M (eds) AGI, lecture notes in computer science, vol 7716. Springer, New York, pp 126–135Google Scholar
  108. Jacoff A, Messina E, Weiss BA, Tadokoro S, Nakagawa Y (2003) Test arenas and performance metrics for urban search and rescue robots. In: Proceedings of 2003 IEEE/RSJ international conference on intelligent robots and systems, 2003 (IROS 2003), IEEE, vol 4, pp 3396–3403Google Scholar
  109. Japkowicz N, Shah M (2011) Evaluating learning algorithms. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  110. Jiang J (2008) A literature survey on domain adaptation of statistical classifiers.
  111. Johnson M, Hofmann K, Hutton T, Bignell D (2016) The Malmo platform for artificial intelligence experimentation. In: International joint conference on artificial intelligence (IJCAI)Google Scholar
  112. Keith TZ, Reynolds MR (2010) Cattell–Horn–Carroll abilities and cognitive tests: what we’ve learned from 20 years of research. Psychol Schools 47(7):635–650Google Scholar
  113. Ketter W, Symeonidis A (2012) Competitive benchmarking: lessons learned from the trading agent competition. AI Mag 33(2):103CrossRefGoogle Scholar
  114. Khreich W, Granger E, Miri A, Sabourin R (2012) A survey of techniques for incremental learning of HMM parameters. Inf Sci 197:105–130CrossRefGoogle Scholar
  115. Kim JH (2004) Soccer robotics, vol 11. Springer, New YorkGoogle Scholar
  116. Kitano H, Asada M, Kuniyoshi Y, Noda I, Osawa E (1997) Robocup: the robot world cup initiative. In: Proceedings of the first international conference on autonomous agents, ACM, pp 340–347Google Scholar
  117. Kleiner K (2011) Who are you calling bird-brained? An attempt is being made to devise a universal intelligence test. Economist 398(8723, 5 March 2011):82Google Scholar
  118. Knuth DE (1973) Sorting and searching, volume 3 of the art of computer programming. Addison-Wesley, ReadingzbMATHGoogle Scholar
  119. Koza JR (2010) Human-competitive results produced by genetic programming. Genet Program Evolvable Mach 11(3–4):251–284CrossRefGoogle Scholar
  120. Krueger J, Osherson D (1980) On the psychology of structural simplicity. In: Jusczyk PW, Klein RM (eds) The nature of thought: essays in honor of D. O. Hebb. Psychology Press, London, pp 187–205Google Scholar
  121. Langford J (2005) Clever methods of overfitting. Machine Learning (Theory).
  122. Langley P (1987) Research papers in machine learning. Mach Learn 2(3):195–198Google Scholar
  123. Langley P (2011) The changing science of machine learning. Mach Learn 82(3):275–279zbMATHCrossRefGoogle Scholar
  124. Langley P (2012) The cognitive systems paradigm. Adv Cogn Syst 1:3–13Google Scholar
  125. Lattimore T, Hutter M (2013) No free lunch versus Occam’s razor in supervised learning. Algorithmic Probability and Friends. Springer, Bayesian Prediction and Artificial Intelligence, pp 223–235Google Scholar
  126. Leeuwenberg ELJ, Van Der Helm PA (2012) Structural information theory: the simplicity of visual form. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  127. Legg S, Hutter M (2007a) Tests of machine intelligence. In: Lungarella M, Iida F, Bongard J, Pfeifer R (eds) 50 Years of Artificial Intelligence, Lecture Notes in Computer Science, vol 4850, Springer Berlin Heidelberg, pp 232–242. doi: 10.1007/978-3-540-77296-5_22
  128. Legg S, Hutter M (2007b) Universal intelligence: a definition of machine intelligence. Minds Mach 17(4):391–444CrossRefGoogle Scholar
  129. Legg S, Veness J (2013) An approximation of the universal intelligence measure. Algorithmic Probability and Friends. Springer, Bayesian Prediction and Artificial Intelligence, pp 236–249Google Scholar
  130. Levesque HJ (2014) On our best behaviour. Artif Intell 212:27–35MathSciNetzbMATHCrossRefGoogle Scholar
  131. Levesque HJ, Davis E, Morgenstern L (2012) The winograd schema challenge. In: Proceedings of the thirteenth international conference on the principles of knowledge representation and reasoning, pp 552–561Google Scholar
  132. Levin LA (1973) Universal sequential search problems. Prob Inf Transm 9(3):265–266Google Scholar
  133. Levin LA (1986) Average case complete problems. SIAM J Comput 15:285–286MathSciNetzbMATHCrossRefGoogle Scholar
  134. Levin LA (2013) Universal heuristics: how do humans solve unsolvable problems? In: Dowe DL (ed) Algorithmic probability and friends. Bayesian prediction and artificial intelligence, lecture notes in computer science, vol 7070. Springer, New York, pp 53–54CrossRefGoogle Scholar
  135. Li M, Vitányi P (2008) An introduction to Kolmogorov complexity and its applications, 3rd edn. Springer, New YorkzbMATHCrossRefGoogle Scholar
  136. Livingstone D (2006) Turing’s test and believable AI in games. Comput Entertain CIE 4(1):6CrossRefGoogle Scholar
  137. Llargues-Asensio JM, Peralta J, Arrabales R, González-Bedía M, Cortez P, López-Peña AL (2014) Artificial intelligence approaches for the generation and assessment of believable human-like behaviour in virtual characters. Expert Systems with ApplicationsGoogle Scholar
  138. Long D, Fox M (2003) The 3rd international planning competition: results and analysis. J Artif Intell Res JAIR 20:1–59zbMATHCrossRefGoogle Scholar
  139. Lord FM (1980) Applications of item response theory to practical testing problems. Erlbaum, MahwahGoogle Scholar
  140. Macià N, Bernadó-Mansilla E (2014) Towards UCI+: a mindful repository design. Inf Sci 261:237–262CrossRefGoogle Scholar
  141. Madhavan R, Tunstel E, Messina E (2009) Performance evaluation and benchmarking of intelligent systems. Springer, New YorkCrossRefGoogle Scholar
  142. Mahoney MV (1999) Text compression as a test for artificial intelligence. In: Proceedings of the national conference on artificial intelligence, AAAI, p 970Google Scholar
  143. Marché C, Zantema H (2007) The termination competition. In: Term rewriting and applications, Springer, pp 303–313Google Scholar
  144. Marcus G, Rossi F, Veloso M (2016) Beyond the Turing test (special issue). AI Mag 37(1):3–101CrossRefGoogle Scholar
  145. Masum H, Christensen S (2003) The turing ratio: a framework for open-ended task metrics. J Evol TechnolGoogle Scholar
  146. Masum H, Christensen S, Oppacher F (2002) The turing ratio: metrics for open-ended tasks. In: GECCO, Citeseer, pp 973–980Google Scholar
  147. McCarthy J (2007) What is artificial intelligence. Technical report, Stanford University.
  148. McCorduck P (2004) Machines who think. A K Peters/CRC Press, Boca RatonGoogle Scholar
  149. McDermott J, White DR, Luke S, Manzoni L, Castelli M, Vanneschi L, Jaśkowski W, Krawiec K, Harper R, Jong KD, O’Reilly UM (2012) Genetic programming needs better benchmarks. In: Proceedings of the 14th international conference on Genetic and evolutionary computation conference. ACM, Philadelphia, pp 791–798Google Scholar
  150. McGuigan M (2006) Graphics Turing Test. arXiv preprint arXiv:cs/0603132
  151. Melkikh AV (2014) The no free lunch theorem and hypothesis of instinctive animal behavior. Artif Intell Res 3(4):p43CrossRefGoogle Scholar
  152. Mellenbergh GJ (1994) Generalized linear item response theory. Psychol Bull 115(2):300CrossRefGoogle Scholar
  153. Mesnil G, Dauphin Y, Glorot X, Rifai S, Bengio Y, Goodfellow IJ, Lavoie E, Muller X, Desjardins G, Warde-Farley D, et al (2012) Unsupervised and transfer learning challenge: a deep learning approach. JMLR: Workshop and Conference Proceedings, 2012 ICML Workshop on Unsupervised and Transfer Learning vol 27, pp 97–110Google Scholar
  154. Messina E, Meystel A, Reeker L (2001) PerMIS 2001, white paper. In: Meystel AM, Messina ER (eds) Measuring the performance and intelligence of systems: proceedings of the 2001 PerMIS Workshop, September 4, 2001, National Institute of Standards and Technology (NIST) Special Publication 982. Gaithersburg, pp 3–15Google Scholar
  155. Meystel A (2000) Permis 2000 white paper: measuring performance and intelligence of systems with autonomy. In: Meystel AM, Messina ER (eds) Measuring the performance and intelligence of systems: proceedings of the 2000 PerMIS Workshop, August 14–16, 2000, National Institute of Standards and Technology (NIST) Special Publication 970. Gaithersburg, pp 1–34Google Scholar
  156. Meystel A, Albus J, Messina E, Leedom D (2003a) Performance measures for intelligent systems: measures of technology readiness. Technical report, DTIC DocumentGoogle Scholar
  157. Meystel A, Albus J, Messina E, Leedom D (2003) Permis 2003 white paper: performance measures for intelligent systems—measures of technology readiness. In: Meystel AM, Messina ER (eds) Measuring the performance and intelligence of systems: proceedings of the 2003 PerMIS Workshop, National Institute of Standards and Technology (NIST) Special Publication 1014. GaithersburgGoogle Scholar
  158. Minsky ML (ed) (1968) Semantic information processing. MIT Press, CambridgezbMATHGoogle Scholar
  159. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533CrossRefGoogle Scholar
  160. Morgenstern L, Davis E, Ortiz-Jr CL (2016) Planning, executing, and evaluating the Winograd schema challenge. AI Mag 37(1):50–54CrossRefGoogle Scholar
  161. Mueller S, Jones M, Minnery B, Hiland JM (2007) The bica cognitive decathlon: a test suite for biologically-inspired cognitive agents. In: Proceedings of behavior representation in modeling and simulation conference, NorfolkGoogle Scholar
  162. Mueller ST (2010) A partial implementation of the BICA cognitive decathlon using the psychology experiment building language (PEBL). Int J Mach Conscious 2(02):273–288CrossRefGoogle Scholar
  163. Mueller ST, Minnery BS (2008) Adapting the Turing Test for embodied neurocognitive evaluation of biologically-inspired cognitive agents. In: Proceedings of 2008 AAAI fall symposium on biologically inspired cognitive architecturesGoogle Scholar
  164. Newell A (1973) You can’t play 20 questions with nature and win: projective comments on the papers of this symposium. In: Chase W (ed) Vis Inf Process. Academic Press, New York, pp 283–308Google Scholar
  165. Newell A (1980) Physical symbol systems. Cogn Sci 4(2):135–183CrossRefGoogle Scholar
  166. Newell A (1990) Unified theories of cognition. Harvard University, CambridgeGoogle Scholar
  167. Newell A, Simon HA (1976) Computer science as empirical inquiry: symbols and search. Commun ACM 19(3):113–126MathSciNetCrossRefGoogle Scholar
  168. Nizamani AR (2015) Reasoning with bounded cognitive resources. Ph.D. thesis, Department of Applied Information Technology, Chalmers University of Technology & University of Gothenburg, SwedenGoogle Scholar
  169. Oppy G, Dowe DL (2011) The Turing Test. In: Zalta EN (ed) Stanford Encyclopedia of Philosophy, Stanford University.
  170. Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359CrossRefGoogle Scholar
  171. Perez D, Samothrakis S, Togelius J, Schaul T, Lucas S, Couëtoux A, Lee J, Lim CU, Thompson T (2015) The 2014 general video game playing competition. IEEE Transactions on Computational Intelligence and AI in GamesGoogle Scholar
  172. Potthast M, Hagen M, Gollub T, Tippmann M, Kiesel J, Rosso P, Stamatatos E, Stein B (2013) Overview of the 5th international competition on plagiarism detection. CLEF (2013) Evaluation labs and workshop working notes papers, pp 23–26 September. Valencia, SpainGoogle Scholar
  173. Proudfoot D (2011) Anthropomorphism and AI: Turing’s much misunderstood imitation game. Artif Intell 175(5):950–957MathSciNetCrossRefGoogle Scholar
  174. Quinn AJ, Bederson BB (2011) Human computation: a survey and taxonomy of a growing field. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, pp 1403–1412Google Scholar
  175. Rajani S (2011) Artificial intelligence—man or machine. Int J Inf Technol 4(1):173–176Google Scholar
  176. Rao RB, Fung G, Rosales R (2008) On the dangers of cross-validation. an experimental evaluation. In: SDM, SIAM, pp 588–596Google Scholar
  177. Rohrer B (2010) Accelerating progress in artificial general intelligence: choosing a benchmark for natural world interaction. J Artif Gen Intell 2(1):1–28CrossRefGoogle Scholar
  178. Rothenberg J, Paul J, Kameny I, Kipps JR, Swenson M (1987) Evaluating expert system tools: a framework and methodology-workshops. Technical report, DTIC DocumentGoogle Scholar
  179. Russell S, Norvig P (2009) Artificial intelligence: a modern approach. Prentice Hall, Upper Saddle RiverzbMATHGoogle Scholar
  180. Sanghi P, Dowe DL (2003) A computer program capable of passing IQ tests. In: 4th international conference on cognitive science (ICCS’03), Sydney, pp 570–575Google Scholar
  181. Schaeffer J, Burch N, Bjornsson Y, Kishimoto A, Muller M, Lake R, Lu P, Sutphen S (2007) Checkers is solved. Science 317(5844):1518MathSciNetzbMATHCrossRefGoogle Scholar
  182. Schaie KW (2010) Primary mental abilities. Corsini Encyclopedia of PsychologyGoogle Scholar
  183. Schaul T (2014) An extensible description language for video games. IEEE Trans Comput Intell AI Games PP(99):1–1. doi: 10.1109/TCIAIG.2014.2352795
  184. Schenck C (2013) Intelligence tests for robots: Solving perceptual reasoning tasks with a humanoid robot. Master’s thesis, Iowa State UniversityGoogle Scholar
  185. Schlenoff C, Scott H, Balakirsky S (2011) Performance evaluation of intelligent systems at the National Institute of Standards and Technology (NIST). Technical report, DTIC DocumentGoogle Scholar
  186. Schmid U, Ragni M (2015) Comparing computer models solving number series problems. In: Artificial general intelligence. Springer, pp 352–361Google Scholar
  187. Schweizer P (1998) The truly total Turing test. Minds Mach 8(2):263–272MathSciNetCrossRefGoogle Scholar
  188. Searle JR (1980) Minds, brains, and programs. Behav Brain Sci 3:417–457CrossRefGoogle Scholar
  189. Seber GAF, Salehi MM (2013) Adaptive cluster sampling. In: Adaptive sampling designs. Springer, pp 11–26Google Scholar
  190. Settles B (2012) Active learning. Synth Lect Artif Intell Mach Learn 6(1):1–114MathSciNetzbMATHCrossRefGoogle Scholar
  191. Shettleworth SJ (2010) Cognition, evolution, and behavior. Oxford University Press, OxfordGoogle Scholar
  192. Shettleworth SJ, Bloom P, Nadel L (2013) Fundamentals of comparative cognition. Oxford University Press, OxfordGoogle Scholar
  193. Shieber SM (2016) Principles for designing an AI competition, or why the Turing test fails as an inducement prize. AI Mag 37(1):91–96CrossRefGoogle Scholar
  194. Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M et al (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484–489CrossRefGoogle Scholar
  195. Simmons R (2000) Survivability and competence as measures of intelligent systems. In: Meystel AM, Messina ER (eds) Measuring the performance and intelligence of systems: proceedings of the 2000 PerMIS Workshop, August 14–16, 2000, National Institute of Standards and Technology (NIST) Special Publication 970. Gaithersburg, pp 162–163Google Scholar
  196. Simon HA (1995) Artificial intelligence: an empirical science. Artif Intell 77(1):95–127CrossRefGoogle Scholar
  197. Sloman A, Scheutz M (2002) A framework for comparing agent architectures. Proceedings of UKCI 2Google Scholar
  198. Smith WD (2002) Rating systems for gameplayers, and learning. NEC, Princeton, NJ, Technical report, pp 93–104Google Scholar
  199. Smith WD (2006) Mathematical definition of “intelligence” (and consequences). Unpublished reportGoogle Scholar
  200. Soares C (2009) UCI++: improved support for algorithm selection using datasetoids. In: Advances in knowledge discovery and data mining. Springer, pp 499–506Google Scholar
  201. Solomonoff R (1996) Does algorithmic probability solve the problem of induction. Inf Stat Induction Sci 7–8Google Scholar
  202. Solomonoff RJ (1964) A formal theory of inductive inference. Part I. Inf Control 7(1):1–22MathSciNetzbMATHCrossRefGoogle Scholar
  203. Solomonoff RJ (1984) Optimum sequential search. Oxbridge Research, Cambridge.
  204. Srinivasan R (2002) Importance sampling: applications in communications and detection. Springer, New YorkzbMATHCrossRefGoogle Scholar
  205. Starkie B, van Zaanen M, Estival D (2006) The Tenjinno machine translation competition. In: Grammatical inference: algorithms and applications. Springer, pp 214–226Google Scholar
  206. Sternberg RJ (ed) (2000) Handbook of intelligence. Cambridge University Press, CambridgeGoogle Scholar
  207. Strannegård C, Amirghasemi M, Ulfsbücker S (2013a) An anthropomorphic method for number sequence problems. Cogn Syst Res 22–23:27–34CrossRefGoogle Scholar
  208. Strannegård C, Nizamani A, Sjöberg A, Engström F (2013b) Bounded Kolmogorov complexity based on cognitive models. In: Kühnberger KU, Rudolph S, Wang P (eds) Artificial general intelligence. Lecture notes in computer science, vol 7999. Springer, Berlin Heidelberg, pp 130–139CrossRefGoogle Scholar
  209. Strickler RE (1973) Change in selected characteristics of students between ninth and twelfth grade as related to high school curriculumGoogle Scholar
  210. Sturtevant N (2012) Benchmarks for grid-based pathfinding. Trans Comput Intell AI Games 4(2):144–148.
  211. Sutcliffe G (2009) The TPTP problem library and associated infrastructure: the FOF and CNF Parts, v3.5.0. J Autom Reason 43(4):337–362zbMATHCrossRefGoogle Scholar
  212. Sutcliffe G, Suttner C (2006) The state of CASC. AI Commun 19(1):35–48MathSciNetzbMATHGoogle Scholar
  213. Thrun S (1996) Is learning the n-th thing any easier than learning the first? In: Advances in neural information processing systems, pp 640–646Google Scholar
  214. Thrun S, Pratt L (2012) Learning to learn. Springer, New YorkzbMATHGoogle Scholar
  215. Thurstone LL (1938a) Primary mental abilities. Psychometric monographsGoogle Scholar
  216. Thurstone LL (1938b) Primary mental abilities. Psychometric monographsGoogle Scholar
  217. Togelius J, Yannakakis GN, Karakovskiy S, Shaker N (2012) Assessing believability. In: Believable bots, Springer, pp 215–230Google Scholar
  218. Torrey L, Shavlik J (2009) Transfer learning. Handb Res Mach Learn Appl 3:17–35Google Scholar
  219. Turing AM (1950) Computing machinery and intelligence. Mind 59:433–460MathSciNetCrossRefGoogle Scholar
  220. Valiant LG (1984) A theory of the learnable. Commun ACM 27(11):1134–1142zbMATHCrossRefGoogle Scholar
  221. Vallati M, Chrpa L, Grzes M, McCluskey TL, Roberts M, Sanner S (2015) The 2014 international planning competition: progress and trends. AI Mag 36(3):90–98CrossRefGoogle Scholar
  222. van Rijn JN, Bischl B, Torgo L, Gao B, Umaashankar V, Fischer S, Winter P, Wiswedel B, Berthold MR, Vanschoren J (2013) Openml: a collaborative science platform. In: Machine learning and knowledge discovery in databases. Springer, pp 645–649Google Scholar
  223. Vanschoren J, Blockeel H, Pfahringer B, Holmes G (2012) Experiment databases. Mach Learn 87(2):127–158MathSciNetzbMATHCrossRefGoogle Scholar
  224. Vanschoren J, van Rijn JN, Bischl B, Torgo L (2014) Openml: networked science in machine learning. ACM SIGKDD Explor Newsl 15(2):49–60CrossRefGoogle Scholar
  225. Vázquez D, López AM, Marín J, Ponsa D, Gerónimo D (2014) Virtual and real world adaptation for pedestrian detection. IEEE Trans Pattern Anal Mach Intell 36(4):797–809. doi: 10.1109/TPAMI.2013.163 CrossRefGoogle Scholar
  226. Vere SA (1992) A cognitive process shell. Behav Brain Sci 15(03):460–461CrossRefGoogle Scholar
  227. von Ahn L (2009) Human computation. In: Design automation conference, 2009. DAC’09. 46th ACM/IEEE, IEEE, pp 418–419Google Scholar
  228. von Ahn L, Blum M, Langford J (2004) Telling humans and computers apart automatically. Commun ACM 47(2):56–60CrossRefGoogle Scholar
  229. von Ahn L, Maurer B, McMillen C, Abraham D, Blum M (2008) RECAPTCHA: human-based character recognition via web security measures. Science 321(5895):1465MathSciNetzbMATHCrossRefGoogle Scholar
  230. Wallace CS, Boulton DM (1968) An information measure for classification. Comput J 11(2):185–194zbMATHCrossRefGoogle Scholar
  231. Wallace CS, Dowe DL (1999) Minimum message length and Kolmogorov complexity. Comput J 42(4):270–283 (special issue on Kolmogorov complexity)zbMATHCrossRefGoogle Scholar
  232. Wang G, Mohanlal M, Wilson C, Wang X, Metzger M, Zheng H, Zhao BY (2012) Social Turing tests: crowdsourcing sybil detection. arXiv preprint arXiv:1205.3856
  233. Wang P (2010) The evaluation of agi systems. In: Proceedings of the third conference on artificial general intelligence, Citeseer, pp 164–169Google Scholar
  234. Warwick K (2014) Turing Test success marks milestone in computing history. University or Reading Press Release,Google Scholar
  235. Wasserman EA, Zentall TR (2006) Comparative cognition: Experimental explorations of animal intelligence. Oxford University Press, OxfordGoogle Scholar
  236. Watkins CJCH, Dayan P (1992) Q-learning. Mach Learn 8(3):279–292zbMATHGoogle Scholar
  237. Weiss DJ (2011) Better data from better measurements using computerized adaptive testing. J Methods Meas Soc Sci 2(1):1–27MathSciNetCrossRefGoogle Scholar
  238. Weizenbaum J (1966) ELIZA—a computer program for the study of natural language communication between man and machine. Commun ACM 9(1):36–45CrossRefGoogle Scholar
  239. Wellman M, Reeves D, Lochner K, Vorobeychik Y (2004) Price prediction in a trading agent competition. J Artif Intell Res JAIR 21:19–36Google Scholar
  240. White DR, McDermott J, Castelli M, Manzoni L, Goldman BW, Kronberger G, Jaśkowski W, O’Reilly UM, Luke S (2013) Better GP benchmarks: community survey results and proposals. Genet Program Evolvable Mach 14:3–29. doi: 10.1007/s10710-012-9177-2 CrossRefGoogle Scholar
  241. Whiteson S, Tanner B, White A (2010) The reinforcement learning competitions. AI Mag 31(2):81–94CrossRefGoogle Scholar
  242. Whiteson S, Tanner B, Taylor ME, Stone P (2011) Protecting against evaluation overfitting in empirical reinforcement learning. In: 2011 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), IEEE, pp 120–127Google Scholar
  243. Williams PL, Beer RD (2010) Information dynamics of evolved agents. In: From animals to animats 11, Springer, pp 38–49Google Scholar
  244. Winikoff M, Cranefield S (2014) On the testability of bdi agent systems. J Artif Intell Res JAIR 51:71–131MathSciNetzbMATHGoogle Scholar
  245. Wolpert DH (1996) The lack of a priori distinctions between learning algorithms. Neural Comput 8(7):1341–1390CrossRefGoogle Scholar
  246. Wolpert DH (2012) What the no free lunch theorems really mean; how to improve search algorithms. Technical report, Santa fe Institute Working PaperGoogle Scholar
  247. Wolpert DH, Macready WG (1995) No free lunch theorems for search. Technical report SFI-TR-95-02-010 (Santa Fe Institute)Google Scholar
  248. Wolpert DH, Macready WG (2005) Coevolutionary free lunches. IEEE Trans Evol Comput 9(6):721–735CrossRefGoogle Scholar
  249. Yampolskiy RV (2015) Artificial superintelligence: a futuristic approach. CRC Press, Boca RatonGoogle Scholar
  250. Yonck R (2012) Toward a standard metric of machine intelligence. World Future Rev 4(2):61–70CrossRefGoogle Scholar
  251. You J (2015) Beyond the turing test. Science 347(6218):116–116CrossRefGoogle Scholar
  252. Zatuchna Z, Bagnall A (2009) Learning mazes with aliasing states: an LCS algorithm with associative perception. Adapt Behav 17(1):28–57CrossRefGoogle Scholar
  253. Zhou ZH (2012) Ensemble methods: foundations and algorithms. CRC Press, Boca RatonGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  1. 1.DSICUniversitat Politècnica de ValènciaValenciaSpain

Personalised recommendations