An Information-Theoretic Predictive Model for the Accuracy of AI Agents Adapted from Psychometrics

  • Nader ChmaitEmail author
  • David L. Dowe
  • Yuan-Fang Li
  • David G. Green
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10414)


We propose a new model to quantitatively estimate the accuracy of artificial agents over cognitive tasks of approximable complexities. The model is derived by introducing notions from algorithmic information theory into a well-known (psychometric) measurement paradigm called Item Response Theory (IRT). A lower bound on accuracy can be guaranteed with respect to task complexity and the breadth of its solution space using our model. This in turn permits formulating the relationship between agent selection cost, task difficulty and accuracy as optimisation problems. Further results indicate some of the settings over which a group of cooperative agents can be more or less accurate than individual agents or other groups.


  1. 1.
    Bien, Z., Bang, W.C., Kim, D.Y., Han, J.S.: Machine intelligence quotient: its measurements and applications. Fuzzy Sets Syst. 127(1), 3–16 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Birnbaum, A.: Some latent trait models and their use in inferring an examinee’s ability. In: Statistical Theories of Mental Test Scores, pp. 395–479 (1968)Google Scholar
  3. 3.
    Chmait, N.: Understanding and measuring collective intelligence across different cognitive systems: an information-theoretic approach (extended abstract). In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI-17 Doctoral Consortium, Melbourne, Australia (2017, to appear)Google Scholar
  4. 4.
    Chmait, N., Dowe, D.L., Li, Y.F., Green, D.G., Insa-Cabrera, J.: Factors of collective intelligence: how smart are agent collectives? In: Proceedings of 22nd European Conference on Artificial Intelligence ECAI, Frontiers in Artificial Intelligence and Applications, vol. 285, pp. 542–550. IOS Press (2016)Google Scholar
  5. 5.
    De Ayala, R.J.: The Theory and Practice of Item Response Theory. Guilford Publications, New York (2013)Google Scholar
  6. 6.
    Dowe, D.L., Hajek, A.R.: A computational extension to the turing test. In: Proceedings 4th Conference of the Australasian Cognitive Science Society, University of Newcastle, NSW, Australia (1997)Google Scholar
  7. 7.
    Dowe, D.L., Hernández-Orallo, J.: IQ tests are not for machines, yet. Intelligence 40(2), 77–81 (2012)CrossRefGoogle Scholar
  8. 8.
    Dowe, D.L., Hernández-Orallo, J., Das, P.K.: Compression and intelligence: social environments and communication. In: Schmidhuber, J., Thórisson, K.R., Looks, M. (eds.) AGI 2011. LNCS (LNAI), vol. 6830, pp. 204–211. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-22887-2_21 CrossRefGoogle Scholar
  9. 9.
    Gottfredson, L.S.: Mainstream science on intelligence: an editorial with 52 signatories, history, and bibliography. Intelligence 24(1), 13–23 (1997)CrossRefGoogle Scholar
  10. 10.
    Hernández-Orallo, J.: Beyond the Turing test. J. Log. Lang. Inf. 9(4), 447–466 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Hernández-Orallo, J.: The Measure of All Minds: Evaluating Natural and Artificial Intelligence. Cambridge University Press, New York (2016)Google Scholar
  12. 12.
    Hernández-Orallo, J., Dowe, D.L.: Measuring universal intelligence: towards an anytime intelligence test. Artif. Intell. 174(18), 1508–1539 (2010)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Hernández-Orallo, J., Insa-Cabrera, J., Dowe, D.L., Hibbard, B.: Turing machines and recursive turing tests. In: AISB/IACAP 2012 Symposium Revisiting Turing and his Test, pp. 28–33 (2012)Google Scholar
  14. 14.
    Insa-Cabrera, J., Dowe, D.L., España-Cubillo, S., Hernández-Lloreda, M.V., Hernández-Orallo, J.: Comparing humans and AI agents. In: Schmidhuber, J., Thórisson, K.R., Looks, M. (eds.) AGI 2011. LNCS (LNAI), vol. 6830, pp. 122–132. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-22887-2_13 CrossRefGoogle Scholar
  15. 15.
    Klein, G.A., King, J.A.: A test for the performance of knowledge-based systems: AIQ. In: Proceedings of AAAI Workshop on Validation and Verification of Expert System, Menlo Park, CA (1988)Google Scholar
  16. 16.
    Kolmogorov, A.N.: Three approaches to the quantitative definition of information. Probl. Inf. Transm. 1(1), 1–7 (1965)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, Hoboken (2004)CrossRefzbMATHGoogle Scholar
  18. 18.
    Legg, S., Hutter, M.: Universal intelligence: a definition of machine intelligence. Mind. Mach. 17(4), 391–444 (2007)CrossRefGoogle Scholar
  19. 19.
    Lempel, A., Ziv, J.: On the complexity of finite sequences. IEEE Trans. Inf. Theory 22(1), 75–81 (1976)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Levin, L.A.: Universal sequential search problems. Probl. Inf. Transm. 9(3), 265–266 (1973)Google Scholar
  21. 21.
    Li, M., Vitányi, P.: An Introduction to Kolmogorov Complexity and Its Applications, 3rd edn. Springer, New York (2008)CrossRefzbMATHGoogle Scholar
  22. 22.
    Lord, F.M., Novick, M.R.: Statistical Theories of Mental Test Scores. Addison-Wesley, Menlo Park (1968)zbMATHGoogle Scholar
  23. 23.
    Martínez-Plumed, F., Prudêncio, R.B., Martínez-Usó, A., Hernández-Orallo, J.: Making sense of item response theory in machine learning. In: Proceedings of 22nd European Conference on Artificial Intelligence (ECAI), Frontiers in Artificial Intelligence and Applications, vol. 285, pp. 1140–1148 (2016)Google Scholar
  24. 24.
    Raven, J.C., Court, J.H.: Raven’s Progressive Matrices and Vocabulary Scales. Oxford Psychologists Press, Oxford (1998)Google Scholar
  25. 25.
    Roid, G.H.: Stanford-Binet Intelligence Scales. Riverside Publishing, Itasca (2003)Google Scholar
  26. 26.
    Sanghi, P., Dowe, D.L.: A computer program capable of passing I.Q. tests. In: Slezak, P. (ed.) Proceedings of 4th International Conference on Cognitive Science (ICCS/ASCS-2003), pp. 570–575, Australia, July 2003Google Scholar
  27. 27.
    Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27(3), 379–423 (1948)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Shapley, L., Grofman, B.: Optimizing group judgmental accuracy in the presence of interdependencies. Public Choice 43(3), 329–343 (1984)CrossRefGoogle Scholar
  29. 29.
    Solomonoff, R.J.: A preliminary report on a general theory of inductive inference. Report ZTB-138. Zator Co 131, Cambridge, MA (1960)Google Scholar
  30. 30.
    Spearman, C.: General intelligence, objectively determined and measured. Am. J. Psychol. 15(2), 201–292 (1904)CrossRefGoogle Scholar
  31. 31.
    Thurstone, L.L.: Primary Mental Abilities. Chicago Press, Chicago (1938)Google Scholar
  32. 32.
    Turing, A.M.: Computing machinery and intelligence. Mind 59, 433–460 (1950)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Wechsler, D.: Wechsler Adult Intelligence Scale-Fourth. Pearson, San Antonio (2008)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Nader Chmait
    • 1
    Email author
  • David L. Dowe
    • 1
  • Yuan-Fang Li
    • 1
  • David G. Green
    • 1
  1. 1.Faculty of Information TechnologyMonash UniversityClaytonAustralia

Personalised recommendations