Advertisement

KANDINSKY Patterns as IQ-Test for Machine Learning

  • Andreas HolzingerEmail author
  • Michael Kickmeier-Rust
  • Heimo Müller
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11713)

Abstract

AI follows the notion of human intelligence which is unfortunately not a clearly defined term. The most common definition given by cognitive science as mental capability, includes, among others, the ability to think abstract, to reason, and to solve problems from the real world. A hot topic in current AI/machine learning research is to find out whether and to what extent algorithms are able to learn abstract thinking and reasoning similarly as humans can do – or whether the learning outcome remains on purely statistical correlation. In this paper we provide some background on testing intelligence, report some preliminary results from 271 participants of our online study on explainability, and propose to use our Kandinsky Patterns as an IQ-Test for machines. Kandinsky Patterns are mathematically describable, simple, self-contained hence controllable test data sets for the development, validation and training of explainability in AI. Kandinsky Patterns are at the same time easily distinguishable from human observers. Consequently, controlled patterns can be described by both humans and computers. The results of our study show that the majority of human explanations was made based on the properties of individual elements in an image (i.e., shape, color, size) and the appearance of individual objects (number). Comparisons of elements (e.g., more, less, bigger, smaller, etc.) were significantly less likely and the location of objects, interestingly, played almost no role in the explanation of the images. The next step is to compare these explanations with machine explanations.

Keywords

Artificial intelligence Human intelligence Intelligence testing IQ-Test Explainable-AI Interpretable machine learning 

Notes

Acknowledgements

We are grateful for interesting discussions with our local and international colleagues and their encouragement. Parts of this project have been funded by the EU projects FeatureCloud, EOSC-Life, EJP-RD and the Austrian FWF Project “explainable AI”, Grant No. P-32554.

References

  1. 1.
    von Ahn, L., Blum, M., Hopper, N.J., Langford, J.: CAPTCHA: using hard AI problems for security. In: Biham, E. (ed.) EUROCRYPT 2003. LNCS, vol. 2656, pp. 294–311. Springer, Heidelberg (2003).  https://doi.org/10.1007/3-540-39200-9_18CrossRefGoogle Scholar
  2. 2.
    Amthauer, R.: Intelligenz-Struktur-Test 2000 R: I-S-T 2000 R Manual, 2nd edn. Hogrefe, Göttingen (2001)Google Scholar
  3. 3.
    Belk, M., Germanakos, P., Fidas, C., Holzinger, A., Samaras, G.: Towards the personalization of CAPTCHA mechanisms based on individual differences in cognitive processing. In: Holzinger, A., Ziefle, M., Hitz, M., Debevc, M. (eds.) SouthCHI 2013. LNCS, vol. 7946, pp. 409–426. Springer, Heidelberg (2013).  https://doi.org/10.1007/978-3-642-39062-3_26CrossRefGoogle Scholar
  4. 4.
    Binet, A.: L’étude expérimentale de l’intelligence. Schleicher fréres & cie, Paris (1903)Google Scholar
  5. 5.
    Bruner, J.S.: Chapter 2: on attributes and concepts. In: Bruner, J.S., Goodnow, J.J., Austin, G.A. (eds.) A Study of Thinking, pp. 25–49. Wiley, Hoboken (1956)Google Scholar
  6. 6.
    Campbell, M., Hoane Jr., A.J., Hsu, F.: Deep blue. Artif. Intell. 134(1–2), 57–83 (2002).  https://doi.org/10.1016/S0004-3702(01)00129-1CrossRefzbMATHGoogle Scholar
  7. 7.
    Conway, A., Kovacs, K.: New and emerging models of human intelligence. Cogn. Sci. (2015).  https://doi.org/10.1002/wcs.1356CrossRefGoogle Scholar
  8. 8.
    Dollar, P., Tu, Z., Belongie, S.: Supervised learning of edges and object boundaries. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), pp. 1964–1971. IEEE (2006).  https://doi.org/10.1109/CVPR.2006.298
  9. 9.
    Dowe, D.L., Hernández-Orallo, J.: IQ tests are not for machines, yet. Intelligence 40(2), 77–81 (2012).  https://doi.org/10.1016/j.intell.2011.12.001CrossRefGoogle Scholar
  10. 10.
    Ferrucci, D., Levas, A., Bagchi, S., Gondek, D., Mueller, E.T.: Watson: beyond jeopardy!. Artif. Intell. 199, 93–105 (2013).  https://doi.org/10.1016/j.artint.2012.06.009CrossRefGoogle Scholar
  11. 11.
    Funke, J.: Handbook of Anthropometry. Springer, Heidelberg (2006)Google Scholar
  12. 12.
    Gardner, H.: Changing Minds: The Art and Science of Changing Our Own and Other People’s Minds. Harvard Business School Press, Boston (2004)Google Scholar
  13. 13.
    Holzinger, A., Biemann, C., Pattichis, C.S., Kell, D.B.: What do we need to build explainable AI systems for the medical domain? arXiv:1712.09923 (2017)
  14. 14.
    Holzinger, A., Kickmeier-Rust, M.D., Müller, H.: Human explanation profiles for random visual patterns as a benchmark for explainable AI. In: Preparation (2019)Google Scholar
  15. 15.
    Holzinger, A., Langs, G., Denk, H., Zatloukal, K., Müller, H.: Causability and explainability of AI in medicine. Wiley Interdisc. Rev.: Data Min. Knowl. Discov. (2019).  https://doi.org/10.1002/widm.1312
  16. 16.
    Holzinger, A., Plass, M., Holzinger, K., Crisan, G.C., Pintea, C.M., Palade, V.: A glass-box interactive machine learning approach for solving NP-hard problems with the human-in-the-loop. arXiv:1708.01104 (2017)
  17. 17.
    Horn, J.L., Cattell, R.B.: Refinement and test of the theory of fluid and crystallized general intelligences. J. Educ. Psychol. 57, 253–270 (1966)CrossRefGoogle Scholar
  18. 18.
    Hubel, D.H., Wiesel, T.N.: Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol. 160(1), 106–154 (1962).  https://doi.org/10.1113/jphysiol.1962.sp006837CrossRefGoogle Scholar
  19. 19.
    Hunt, E.B.: Concept Learning: An Information Processing Problem. Wiley, Hoboken (1962).  https://doi.org/10.1037/13135-001CrossRefGoogle Scholar
  20. 20.
    Jäger, A.O.: Validität von intelligenztests. Diagnostica 32, 272–289 (1986)Google Scholar
  21. 21.
    Koffka, K.: Principles of Gestalt Psychology. Harcourt, New York (1935)Google Scholar
  22. 22.
    Kovacs, K., Conway, A.: Process overlap theory: a unified account of human intelligence. Psychol. Inquiry 27, 151–177 (2016)CrossRefGoogle Scholar
  23. 23.
    Kundel, H.I., Nodine, C.F.: A visual concept shapes image perception. Radiology 146(2), 363–368 (1983)CrossRefGoogle Scholar
  24. 24.
    Lake, B.M., Salakhutdinov, R., Tenenbaum, J.B.: Human-level concept learning through probabilistic program induction. Science 350(6266), 1332–1338 (2015).  https://doi.org/10.1126/science.aab3050MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Laplace, P.S.: Mémoire sur les probabilités. Mémoires de l’Académie Royale des sciences de Paris 1778, 227–332 (1781)Google Scholar
  26. 26.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  27. 27.
    Leeuwenberg, E.L., Boselie, F.: Against the likelihood principle in visual form perception. Psychol. Rev. 95(4), 485–491 (1988)CrossRefGoogle Scholar
  28. 28.
    Legg, S., Hutter, M.: Universal intelligence: a definition of machine intelligence. Minds Mach. 17(4), 391–444 (2007).  https://doi.org/10.1007/s11023-007-9079-xCrossRefGoogle Scholar
  29. 29.
    Li, L., et al.: Parallel testing of vehicle intelligence via virtual-real interaction. Sci. Robot. 4(eaaw4106), 1–3 (2019)Google Scholar
  30. 30.
    Locke, E.A.: Why emotional intelligence is an invalid concept. J. Organ. Behav. 26, 425–431 (2005)CrossRefGoogle Scholar
  31. 31.
    Mitchell, T.M.: Machine Learning. McGraw Hill, New York (1997)zbMATHGoogle Scholar
  32. 32.
    Müller, H., Holzinger, A.: Kandinsky patterns. arXiv:1906.00657 (2019). https://arxiv.org/abs/1906.00657
  33. 33.
    Nambiar, R.: Towards an industry standard for benchmarking artificial intelligence systems. In: 34th International Conference on Data Engineering (ICDE 2018). IEEE (2018).  https://doi.org/10.1109/ICDE.2018.00212
  34. 34.
    Nambiar, R., Ghandeharizadeh, S., Little, G., Boden, C., Dholakia, A.: Industry panel on defining industry standards for benchmarking artificial intelligence. In: Nambiar, R., Poess, M. (eds.) TPCTC 2018. LNCS, vol. 11135, pp. 1–6. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-11404-6_1CrossRefGoogle Scholar
  35. 35.
    Pirolli, P., Russell, D.M.: Introduction to this special issue on sensemaking. Hum.-Comput. Interact. 26, 1–8 (2011)CrossRefGoogle Scholar
  36. 36.
    Preed, V.R.: Handbook of Anthropometry. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-1-4419-1788-1CrossRefGoogle Scholar
  37. 37.
    Raven, J.: The raven’s progressive matrices: change and stability over culture and time. Cogn. Psychol. 41(1), 1–48 (2000).  https://doi.org/10.1006/cogp.1999.0735MathSciNetCrossRefGoogle Scholar
  38. 38.
    Raven, J.C., Court, J.H.: Raven’s Progressive Matrices and Vocabulary Scales. Oxford Psychologists Press, Oxford (1998)Google Scholar
  39. 39.
    Ren, X., Malik, J.: Learning a classification model for segmentation. In: Ninth IEEE International Conference on Computer Vision (ICCV), pp. 10–17. IEEE (2003).  https://doi.org/10.1109/ICCV.2003.1238308
  40. 40.
    Samek, W., Wiegand, T., Müller, K.R.: Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models. arXiv:1708.08296 (2017)
  41. 41.
    Santoro, A., Hill, F., Barrett, D., Morcos, A., Lillicrap, T.: Measuring abstract reasoning in neural networks. In: 35th International Conference on Machine Learning, pp. 4477–4486. PMLR (2018)Google Scholar
  42. 42.
    Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016).  https://doi.org/10.1038/nature16961CrossRefGoogle Scholar
  43. 43.
    Spearman, C.: “General intelligence”, objectively determined and measured. Am. J. Psychol. 15(2), 201–292 (1904)CrossRefGoogle Scholar
  44. 44.
    Tenenbaum, J.B.: Bayesian modeling of human concept learning. In: Solla, S.A., Leen, T.K., Müller, K.R. (eds.) Advances in Neural Information Processing Systems (NIPS 1999), pp. 59–68. NIPS Foundation (1999)Google Scholar
  45. 45.
    Turing, A.M.: Computing machinery and intelligence. Mind 59(236), 433–460 (1950).  https://doi.org/10.1093/mind/LIX.236.433MathSciNetCrossRefGoogle Scholar
  46. 46.
    Urbina, S.: Chapter 2: tests of intelligence. In: Kaufman, S.B. (ed.) The Cambridge Handbook of Intelligence. Cambridge University Press, Cambridge (2011)Google Scholar
  47. 47.
    Uusitalo, L., Simola, J., Kuisma, J.: Perception of abstract and representative visual art. In: TBA, January 2009Google Scholar
  48. 48.
    Wechsler, D.: The Measurement and Appraisal of Adult Intelligence, 4th edn. Williams and Witkins, Baltimore (1958)CrossRefGoogle Scholar
  49. 49.
    Wertheimer, M.: Laws of organization in perceptual forms. In: Ellis, W.D. (ed.) A Source Book of Gestalt Psychology, pp. 71–88. Paul Kegan, London (1938).  https://doi.org/10.1037/11496-005CrossRefGoogle Scholar
  50. 50.
    Yanagisawa, H.: How does expectation affect sensory experience? A theory of relativity in perception. In: 5th International Symposium on Affective Science and Engineering, ISASE 2019, pp. 1–4. Japan Society of Kansei Engineering (2019).  https://doi.org/10.5057/isase.2019-C000014Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  1. 1.Medical University GrazGrazAustria
  2. 2.University of Teacher EducationSt. GallenSwitzerland

Personalised recommendations