Tests of Machine Intelligence

  • Shane Legg
  • Marcus Hutter
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4850)


Although the definition and measurement of intelligence is clearly of fundamental importance to the field of artificial intelligence, no general survey of definitions and tests of machine intelligence exists. Indeed few researchers are even aware of alternatives to the Turing test and its many derivatives. In this paper we fill this gap by providing a short survey of the many tests of machine intelligence that have been proposed.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alvarado, N., Adams, S., Burbeck, S., Latta, C.: Beyond the Turing test: Performance metrics for evaluating a computer simulation of the human mind. In: Performance Metrics for Intelligent Systems Workshop, Gaithersburg, MD, USA, North-Holland, Amsterdam (2002)Google Scholar
  2. 2.
    Bell, T.C., Cleary, J.G., Witten, I.H.: Text compression. Prentice-Hall, Englewood Cliffs (1990)Google Scholar
  3. 3.
    Block, N.: Psychologism and behaviorism. Philosophical Review 90, 5–43 (1981)CrossRefGoogle Scholar
  4. 4.
    Bringsjord, S., Schimanski, B.: What is artificial intelligence? Psychometric AI as an answer. In: Eighteenth International Joint Conference on Artificial Intelligence, vol. 18, pp. 887–893 (2003)Google Scholar
  5. 5.
    Chaitin, G.J.: Gödel’s theorem and information. International Journal of Theoretical Physics 22, 941–954 (1982)CrossRefMathSciNetGoogle Scholar
  6. 6.
    Dowe, D.L., Hajek, A.R.: A non-behavioural, computational extension to the Turing test. In: ICCIMA 1998. International Conference on Computational Intelligence & Multimedia Applications, pp. 101–106. Gippsland, Australia (1998)Google Scholar
  7. 7.
    Eisner, J.: Cognitive science and the search for intelligence. Invited paper presented to the Socratic Society, University of Cape Town (1991)Google Scholar
  8. 8.
    Ford, K.M., Hayes, P.J.: On computational wings: Rethinking the goals of artificial intelligence. Scientific American, Special edn. (4) (1998)Google Scholar
  9. 9.
    French, R.M.: Subcognition and the limits of the Turing test. Mind 99, 53–65 (1990)CrossRefMathSciNetGoogle Scholar
  10. 10.
    Gottfredson, L.S.: Why g matters: The complexity of everyday life. Intelligence 24(1), 79–132 (1997)CrossRefGoogle Scholar
  11. 11.
    Harnad, S.: Minds, machines and Searle. Journal of Theoretical and Experimental Artificial Intelligence 1, 5–25 (1989)CrossRefGoogle Scholar
  12. 12.
    Hernández-Orallo, J.: Beyond the Turing test. Journal of Logic, Language and Information 9(4), 447–466 (2000)MATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Hernández-Orallo, J.: On the computational measurement of intelligence factors. In: Performance Metrics for Intelligent Systems Workshop, Gaithersburg, MD, USA, pp. 1–8 (2000)Google Scholar
  14. 14.
    Hernández-Orallo, J., Minaya-Collado, N.: A formal definition of intelligence based on an intensional variant of Kolmogorov complexity. In: EIS 1998. Proceedings of the International Symposium of Engineering of Intelligent Systems, pp. 146–163. ICSC Press (1998)Google Scholar
  15. 15.
    Hutchens, J.L.: How to pass the Turing test by cheating (1996), www.cs.umbc.edu/471/current/papers/hutchens.pdf
  16. 16.
    Hutter, M.: Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability, p. 300 Springer, Berlin (2005), http://www.hutter1.net/ai/uaibook.htm
  17. 17.
    Hutter, M.: The Human knowledge compression prize (2006), http://prize.hutter1.net
  18. 18.
    Legg, S., Hutter, M.: A collection of definitions of intelligence. In: Goertzel, B. (ed.) Proc. 1st Annual artificial general intelligence workshop (to appear), Online version www.idsia.ch/ shane/intelligence.htmlGoogle Scholar
  19. 19.
    Legg, S., Hutter, M.: A formal measure of machine intelligence. In: Benelearn 2006. Annual Machine Learning Conference of Belgium and The Netherlands, Ghent (2006)Google Scholar
  20. 20.
    Levin, L.A.: Universal sequential search problems. Problems of Information Transmission 9, 265–266 (1973)Google Scholar
  21. 21.
    Li, M., Vitányi, P.M.B.: An introduction to Kolmogorov complexity and its applications, 2nd edn. Springer, Heidelberg (1997)MATHGoogle Scholar
  22. 22.
    Loebner, H.G.: The Loebner prize — The first Turing test (1990), http://www.loebner.net/Prizef/loebner-prize.html
  23. 23.
    Mahoney, M.V.: Text compression as a test for artificial intelligence. In: AAAI/IAAI (1999)Google Scholar
  24. 24.
    Masum, H., Christensen, S., Oppacher, F.: The Turing ratio: Metrics for open-ended tasks. In: GECCO 2002. Proceedings of the Genetic and Evolutionary Computation Conference, pp. 973–980. Morgan Kaufmann Publishers, New York (2002)Google Scholar
  25. 25.
    Sanghi, P., Dowe, D.L.: A computer program capable of passing I.Q. tests. In: Sloot, P.M.A., Abramson, D., Bogdanov, A.V., Gorbachev, Y.E., Dongarra, J.J., Zomaya, A.Y. (eds.) ICCS 2003. LNCS, vol. 2657, pp. 570–575. Springer, Heidelberg (2003)Google Scholar
  26. 26.
    Saygin, A., Cicekli, I., Akman, V.: Turing test: 50 years later. Minds and Machines 10 (2000)Google Scholar
  27. 27.
    Schweizer, P.: The truly total Turing test. Minds and Machines 8, 263–272 (1998)CrossRefGoogle Scholar
  28. 28.
    Searle, J.: Minds, brains, and programs. Behavioral & Brain Sciences 3, 417–458 (1980)CrossRefGoogle Scholar
  29. 29.
    Shieber, S.: Lessons from a restricted Turing test. CACM: Communications of the ACM 37 (1994)Google Scholar
  30. 30.
    Smith, W.D.: Mathematical definition of “intelligence” (and consequences) (2006), http://math.temple.edu/~wds/homepage/works.html
  31. 31.
    Treister-Goren, A., Dunietz, J., Hutchens, J.L.: The developmental approach to evaluating artificial intelligence – a proposal. In: Performance Metrics for Intelligence Systems (2000)Google Scholar
  32. 32.
    Treister-Goren, A., Hutchens, J.L.: Creating AI: A unique interplay between the development of learning algorithms and their education. In: Proceeding of the First International Workshop on Epigenetic Robotics (2001)Google Scholar
  33. 33.
    Turing, A.M.: Computing machinery and intelligence. Mind (October 1950)Google Scholar
  34. 34.
    Voss, P.: Essentials of general intelligence: The direct path to AGI. In: Goertzel, B., Pennachin, C. (eds.) Artificial General Intelligence, Springer, Heidelberg (2005)Google Scholar
  35. 35.
    Watt, S.: Naive psychology and the inverted Turing test. Psycoloquy 7(14) (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Shane Legg
    • 1
  • Marcus Hutter
    • 2
  1. 1.IDSIA, Galleria 2, Manno-Lugano CH-6928Switzerland
  2. 2.RSISE/ANU/NICTA, Canberra, ACT, 0200Australia

Personalised recommendations