Tests of Machine Intelligence

  • Shane Legg
  • Marcus Hutter
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4850)


Although the definition and measurement of intelligence is clearly of fundamental importance to the field of artificial intelligence, no general survey of definitions and tests of machine intelligence exists. Indeed few researchers are even aware of alternatives to the Turing test and its many derivatives. In this paper we fill this gap by providing a short survey of the many tests of machine intelligence that have been proposed.


Machine Intelligence Intelligence Test Kolmogorov Complexity World Knowledge Turing Test 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alvarado, N., Adams, S., Burbeck, S., Latta, C.: Beyond the Turing test: Performance metrics for evaluating a computer simulation of the human mind. In: Performance Metrics for Intelligent Systems Workshop, Gaithersburg, MD, USA, North-Holland, Amsterdam (2002)Google Scholar
  2. 2.
    Bell, T.C., Cleary, J.G., Witten, I.H.: Text compression. Prentice-Hall, Englewood Cliffs (1990)Google Scholar
  3. 3.
    Block, N.: Psychologism and behaviorism. Philosophical Review 90, 5–43 (1981)CrossRefGoogle Scholar
  4. 4.
    Bringsjord, S., Schimanski, B.: What is artificial intelligence? Psychometric AI as an answer. In: Eighteenth International Joint Conference on Artificial Intelligence, vol. 18, pp. 887–893 (2003)Google Scholar
  5. 5.
    Chaitin, G.J.: Gödel’s theorem and information. International Journal of Theoretical Physics 22, 941–954 (1982)CrossRefMathSciNetGoogle Scholar
  6. 6.
    Dowe, D.L., Hajek, A.R.: A non-behavioural, computational extension to the Turing test. In: ICCIMA 1998. International Conference on Computational Intelligence & Multimedia Applications, pp. 101–106. Gippsland, Australia (1998)Google Scholar
  7. 7.
    Eisner, J.: Cognitive science and the search for intelligence. Invited paper presented to the Socratic Society, University of Cape Town (1991)Google Scholar
  8. 8.
    Ford, K.M., Hayes, P.J.: On computational wings: Rethinking the goals of artificial intelligence. Scientific American, Special edn. (4) (1998)Google Scholar
  9. 9.
    French, R.M.: Subcognition and the limits of the Turing test. Mind 99, 53–65 (1990)CrossRefMathSciNetGoogle Scholar
  10. 10.
    Gottfredson, L.S.: Why g matters: The complexity of everyday life. Intelligence 24(1), 79–132 (1997)CrossRefGoogle Scholar
  11. 11.
    Harnad, S.: Minds, machines and Searle. Journal of Theoretical and Experimental Artificial Intelligence 1, 5–25 (1989)CrossRefGoogle Scholar
  12. 12.
    Hernández-Orallo, J.: Beyond the Turing test. Journal of Logic, Language and Information 9(4), 447–466 (2000)zbMATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Hernández-Orallo, J.: On the computational measurement of intelligence factors. In: Performance Metrics for Intelligent Systems Workshop, Gaithersburg, MD, USA, pp. 1–8 (2000)Google Scholar
  14. 14.
    Hernández-Orallo, J., Minaya-Collado, N.: A formal definition of intelligence based on an intensional variant of Kolmogorov complexity. In: EIS 1998. Proceedings of the International Symposium of Engineering of Intelligent Systems, pp. 146–163. ICSC Press (1998)Google Scholar
  15. 15.
    Hutchens, J.L.: How to pass the Turing test by cheating (1996),
  16. 16.
    Hutter, M.: Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability, p. 300 Springer, Berlin (2005),
  17. 17.
    Hutter, M.: The Human knowledge compression prize (2006),
  18. 18.
    Legg, S., Hutter, M.: A collection of definitions of intelligence. In: Goertzel, B. (ed.) Proc. 1st Annual artificial general intelligence workshop (to appear), Online version shane/intelligence.htmlGoogle Scholar
  19. 19.
    Legg, S., Hutter, M.: A formal measure of machine intelligence. In: Benelearn 2006. Annual Machine Learning Conference of Belgium and The Netherlands, Ghent (2006)Google Scholar
  20. 20.
    Levin, L.A.: Universal sequential search problems. Problems of Information Transmission 9, 265–266 (1973)Google Scholar
  21. 21.
    Li, M., Vitányi, P.M.B.: An introduction to Kolmogorov complexity and its applications, 2nd edn. Springer, Heidelberg (1997)zbMATHGoogle Scholar
  22. 22.
    Loebner, H.G.: The Loebner prize — The first Turing test (1990),
  23. 23.
    Mahoney, M.V.: Text compression as a test for artificial intelligence. In: AAAI/IAAI (1999)Google Scholar
  24. 24.
    Masum, H., Christensen, S., Oppacher, F.: The Turing ratio: Metrics for open-ended tasks. In: GECCO 2002. Proceedings of the Genetic and Evolutionary Computation Conference, pp. 973–980. Morgan Kaufmann Publishers, New York (2002)Google Scholar
  25. 25.
    Sanghi, P., Dowe, D.L.: A computer program capable of passing I.Q. tests. In: Sloot, P.M.A., Abramson, D., Bogdanov, A.V., Gorbachev, Y.E., Dongarra, J.J., Zomaya, A.Y. (eds.) ICCS 2003. LNCS, vol. 2657, pp. 570–575. Springer, Heidelberg (2003)Google Scholar
  26. 26.
    Saygin, A., Cicekli, I., Akman, V.: Turing test: 50 years later. Minds and Machines 10 (2000)Google Scholar
  27. 27.
    Schweizer, P.: The truly total Turing test. Minds and Machines 8, 263–272 (1998)CrossRefGoogle Scholar
  28. 28.
    Searle, J.: Minds, brains, and programs. Behavioral & Brain Sciences 3, 417–458 (1980)CrossRefGoogle Scholar
  29. 29.
    Shieber, S.: Lessons from a restricted Turing test. CACM: Communications of the ACM 37 (1994)Google Scholar
  30. 30.
    Smith, W.D.: Mathematical definition of “intelligence” (and consequences) (2006),
  31. 31.
    Treister-Goren, A., Dunietz, J., Hutchens, J.L.: The developmental approach to evaluating artificial intelligence – a proposal. In: Performance Metrics for Intelligence Systems (2000)Google Scholar
  32. 32.
    Treister-Goren, A., Hutchens, J.L.: Creating AI: A unique interplay between the development of learning algorithms and their education. In: Proceeding of the First International Workshop on Epigenetic Robotics (2001)Google Scholar
  33. 33.
    Turing, A.M.: Computing machinery and intelligence. Mind (October 1950)Google Scholar
  34. 34.
    Voss, P.: Essentials of general intelligence: The direct path to AGI. In: Goertzel, B., Pennachin, C. (eds.) Artificial General Intelligence, Springer, Heidelberg (2005)Google Scholar
  35. 35.
    Watt, S.: Naive psychology and the inverted Turing test. Psycoloquy 7(14) (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Shane Legg
    • 1
  • Marcus Hutter
    • 2
  1. 1.IDSIA, Galleria 2, Manno-Lugano CH-6928Switzerland
  2. 2.RSISE/ANU/NICTA, Canberra, ACT, 0200Australia

Personalised recommendations