Skip to main content
SpringerLink
Log in

Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13699))

Included in the following conference series:

Abstract

Is intelligence realized by connectionist or classicist? While connectionist approaches have achieved superhuman performance, there has been growing evidence that such task-specific superiority is particularly fragile in systematic generalization. This observation lies in the central debate between connectionist and classicist, wherein the latter continually advocates an algebraic treatment in cognitive architectures. In this work, we follow the classicist’s call and propose a hybrid approach to improve systematic generalization in reasoning. Specifically, we showcase a prototype with algebraic representation for the abstract spatial-temporal reasoning task of Raven’s Progressive Matrices (RPM) and present the ALgebra-Aware Neuro-Semi-Symbolic (ALANS) learner. The ALANS learner is motivated by abstract algebra and the representation theory. It consists of a neural visual perception frontend and an algebraic abstract reasoning backend: the frontend summarizes the visual information from object-based representation, while the backend transforms it into an algebraic structure and induces the hidden operator on the fly. The induced operator is later executed to predict the answer’s representation, and the choice most similar to the prediction is selected as the solution. Extensive experiments show that by incorporating an algebraic treatment, the ALANS learner outperforms various pure connectionist models in domains requiring systematic generalization. We further show the generative nature of the learned algebraic representation; it can be decoded by isomorphism to generate an answer.

C. Zhang, S. Xie, B. Jia—Indicates equal contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Augustine, S.: The confessions. Clark (1876)

    Google Scholar 

  2. Bahdanau, D., et al.: Systematic generalization: what is required and can it be learned? In: International Conference on Learning Representations (ICLR) (2019)

    Google Scholar 

  3. Bard, J.F.: Practical Bilevel Optimization: Algorithms and Applications, vol. 30. Springer, Dordrecht (2013)

    MATH  Google Scholar 

  4. Carpenter, P.A., Just, M.A., Shell, P.: What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test. Psychol. Rev. 97(3), 404 (1990)

    Article  Google Scholar 

  5. Chen, X., Liang, C., Yu, A.W., Song, D., Zhou, D.: Compositional generalization via neural-symbolic stack machines. In: Advances in Neural Information Processing Systems (2020)

    Google Scholar 

  6. Chen, Z., Mao, J., Wu, J., Wong, K.Y.K., Tenenbaum, J.B., Gan, C.: Grounding physical concepts of objects and events through dynamic visual reasoning. In: International Conference on Learning Representations (ICLR) (2020)

    Google Scholar 

  7. Chollet, F.: The measure of intelligence. arXiv preprint arXiv:1911.01547 (2019)

  8. Colson, B., Marcotte, P., Savard, G.: An overview of bilevel optimization. Ann. Oper. Res. 153(1), 235–256 (2007)

    Article  MathSciNet  Google Scholar 

  9. Dong, H., Mao, J., Lin, T., Wang, C., Li, L., Zhou, D.: Neural logic machines. In: International Conference on Learning Representations (ICLR) (2018)

    Google Scholar 

  10. Evans, R., Grefenstette, E.: Learning explanatory rules from noisy data. J. Artif. Intell. Res. (JAIR) 61, 1–64 (2018)

    Article  MathSciNet  Google Scholar 

  11. Fodor, J.A.: The Language of Thought, vol. 5. Harvard University Press, Cambridge (1975)

    Google Scholar 

  12. Fodor, J.A., Pylyshyn, Z.W., et al.: Connectionism and cognitive architecture: a critical analysis. Cognition 28(1–2), 3–71 (1988)

    Article  Google Scholar 

  13. d’Avila Garcez, A.S. Broda, K.B., Gabbay, D.M.: Neural-Symbolic Learning Systems: Foundations and Applications. Springer, London (2012)

    Google Scholar 

  14. Han, C., Mao, J., Gan, C., Tenenbaum, J., Wu, J.: Visual concept-metaconcept learning. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)

    Google Scholar 

  15. Hausmann, B.A., Ore, O.: Theory of quasi-groups. Am. J. Math. 59(4), 983–1004 (1937)

    Article  MathSciNet  Google Scholar 

  16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  17. Heath, T.L., et al.: The Thirteen Books of Euclid’s Elements. Courier Corporation (1956)

    Google Scholar 

  18. Hill, F., Santoro, A., Barrett, D.G., Morcos, A.S., Lillicrap, T.: Learning to make analogies by contrasting abstract relational structure. In: International Conference on Learning Representations (ICLR) (2019)

    Google Scholar 

  19. Hofstadter, D.R.: Fluid concepts and Creative Analogies: Computer Models of the Fundamental Mechanisms of Thought. Basic Books, New York (1995)

    Google Scholar 

  20. Holyoak, K.J., Ichien, N., Lu, H.: From semantic vectors to analogical mapping. Curr. Dir. Psychol. Sci. 31, 09637214221098054 (2022)

    Google Scholar 

  21. Hu, S., Ma, Y., Liu, X., Wei, Y., Bai, S.: Hierarchical rule induction network for abstract visual reasoning. arXiv preprint arXiv:2002.06838 (2020)

  22. Hudson, D., Manning, C.D.: Learning by abstraction: The neural state machine. In: Advances in Neural Information Processing Systems (2019)

    Google Scholar 

  23. Humphreys, J.E.: Introduction to Lie Algebras and Representation Theory, vol. 9. Springer, New York (2012)

    Google Scholar 

  24. Jaeggi, S.M., Buschkuehl, M., Jonides, J., Perrig, W.J.: Improving fluid intelligence with training on working memory. Proc. Natl. Acad. Sci. (PNAS) 105(19), 6829–6833 (2008)

    Article  Google Scholar 

  25. James, W.: The Principles of Psychology. Henry Holt and Company, New York (1891)

    Google Scholar 

  26. Kansky, K., et al.: Schema networks: zero-shot transfer with a generative causal model of intuitive physics. In: International Conference on Machine Learning (ICML) (2017)

    Google Scholar 

  27. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2014)

    Google Scholar 

  28. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 (2013)

  29. Lake, B., Baroni, M.: Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In: International Conference on Machine Learning (ICML) (2018)

    Google Scholar 

  30. Little, D.R., Lewandowsky, S., Griffiths, T.L.: A Bayesian model of rule induction in raven’s progressive matrices. In: Annual Meeting of the Cognitive Science Society (CogSci) (2012)

    Google Scholar 

  31. Lovett, A., Forbus, K.: Modeling visual problem solving as analogical reasoning. Psychol. Rev. 124(1), 60 (2017)

    Article  Google Scholar 

  32. Lovett, A., Forbus, K., Usher, J.: A structure-mapping model of Raven’s progressive matrices. In: Annual Meeting of the Cognitive Science Society (CogSci) (2010)

    Google Scholar 

  33. Lovett, A., Tomai, E., Forbus, K., Usher, J.: Solving geometric analogy problems through two-stage analogical mapping. Cogn. Sci. 33(7), 1192–1231 (2009)

    Article  Google Scholar 

  34. Maddy, P.: Believing the axioms. I. J. Symb. Logic 53(2), 481–511 (1988)

    Google Scholar 

  35. Manhaeve, R., Dumancic, S., Kimmig, A., Demeester, T., De Raedt, L.: DeepProbLog: neural probabilistic logic programming. In: Advances in Neural Information Processing Systems (NeurIPS) (2018)

    Google Scholar 

  36. Mao, J., Gan, C., Kohli, P., Tenenbaum, J.B., Wu, J.: The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In: International Conference on Learning Representations (ICLR) (2019)

    Google Scholar 

  37. Marcus, G.: The Algebraic Mind. MIT Press, Cambridge (2001)

    Book  Google Scholar 

  38. Marcus, G.: The next decade in AI: four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177 (2020)

  39. Marcus, G.F., Vijayan, S., Rao, S.B., Vishton, P.M.: Rule learning by seven-month-old infants. Science 283(5398), 77–80 (1999)

    Article  Google Scholar 

  40. McCarthy, J.: Programs with common sense. RLE and MIT Computation Center (1960)

    Google Scholar 

  41. McGreggor, K., Goel, A.: Confident reasoning on raven’s progressive matrices tests. In: AAAI Conference on Artificial Intelligence (AAAI) (2014)

    Google Scholar 

  42. McGreggor, K., Kunda, M., Goel, A.: Fractals and Ravens. Artif. Intell. 215, 1–23 (2014)

    Article  MathSciNet  Google Scholar 

  43. Mekik, C.S., Sun, R., Dai, D.Y.: Similarity-based reasoning, Raven’s matrices, and general intelligence. In: International Joint Conference on Artificial Intelligence (IJCAI) (2018)

    Google Scholar 

  44. Newell, A.: Physical symbol systems. Cogn. Sci. 4(2), 135–183 (1980)

    Article  Google Scholar 

  45. Paszke, A., et al.: Automatic differentiation in PyTorch. In: NIPS Autodiff Workshop (2017)

    Google Scholar 

  46. Peano, G.: Arithmetices principia: Nova methodo exposita. Fratres Bocca (1889)

    Google Scholar 

  47. Rahaman, N., et al.: Dynamic inference with neural interpreters. In: Advances in Neural Information Processing Systems (NeurIPS) (2021)

    Google Scholar 

  48. Raven, J.C.: Mental tests used in genetic studies: the performance of related individuals on tests mainly educative and mainly reproductive. Master’s thesis, University of London (1936)

    Google Scholar 

  49. Raven, J.C., Court, J.H.: Raven’s Progressive Matrices and Vocabulary Scales. Oxford Pyschologists Press, Oxford (1998)

    Google Scholar 

  50. Rocktäschel, T., Riedel, S.: End-to-end differentiable proving. In: Advances in Neural Information Processing Systems (NeurIPS) (2017)

    Google Scholar 

  51. Santoro, A., Hill, F., Barrett, D., Morcos, A., Lillicrap, T.: Measuring abstract reasoning in neural networks. In: International Conference on Machine Learning (ICML) (2018)

    Google Scholar 

  52. Santoro, A., et al.: A simple neural network module for relational reasoning. In: Advances in Neural Information Processing Systems (NeurIPS) (2017)

    Google Scholar 

  53. Serafini, L., d’Garcez, A.: Logic tensor networks: deep learning and logical reasoning from data and knowledge. arXiv preprint arXiv:1606.04422 (2016)

  54. Shegheva, S., Goel, A.: The structural affinity method for solving the Raven’s progressive matrices test for intelligence. In: AAAI Conference on Artificial Intelligence (AAAI) (2018)

    Google Scholar 

  55. Spearman, C.: The Nature of “Intelligence’’ and the Principles of Cognition. Macmillan, New York (1923)

    Google Scholar 

  56. Spearman, C.: The Abilities of Man, vol. 6. Macmillan, New York (1927)

    MATH  Google Scholar 

  57. Steenbrugge, X., Leroux, S., Verbelen, T., Dhoedt, B.: Improving generalization for abstract reasoning tasks using disentangled feature representations. arXiv preprint arXiv:1811.04784 (2018)

  58. Wang, D., Jamnik, M., Lio, P.: Abstract diagrammatic reasoning with multiplex graph networks. In: International Conference on Learning Representations (ICLR) (2020)

    Google Scholar 

  59. Wang, K., Su, Z.: Automatic generation of Raven’s progressive matrices. In: International Joint Conference on Artificial Intelligence (IJCAI) (2015)

    Google Scholar 

  60. Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8(3–4), 229–256 (1992)

    Article  Google Scholar 

  61. Winograd, T.: Procedures as a representation for data in a computer program for understanding natural language. Technical report, MIT Center for Space Research (1971)

    Google Scholar 

  62. Wittgenstein, L.: Philosophical Investigations. Philosophische Untersuchungen. Macmillan, New York (1953)

    MATH  Google Scholar 

  63. Wu, J., Tenenbaum, J.B., Kohli, P.: Neural scene de-rendering. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  64. Wu, Y., Dong, H., Grosse, R., Ba, J.: The scattering compositional learner: discovering objects, attributes, relationships in analogical reasoning. arXiv preprint arXiv:2007.04212 (2020)

  65. Xu, M., Jiang, G., Zhang, C., Zhu, S.C., Zhu, Y.: EST: evaluating scientific thinking in artificial agents. arXiv preprint arXiv:2206.09203 (2022)

  66. Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A., Tenenbaum, J.: CLEVRER: collision events for video representation and reasoning. In: International Conference on Learning Representations (ICLR) (2020)

    Google Scholar 

  67. Yi, K., Wu, J., Gan, C., Torralba, A., Kohli, P., Tenenbaum, J.: Neural-symbolic VQA: disentangling reasoning from vision and language understanding. In: Advances in Neural Information Processing Systems (NeurIPS) (2018)

    Google Scholar 

  68. Zhang, C., Gao, F., Jia, B., Zhu, Y., Zhu, S.C.: Raven: A dataset for relational and analogical visual reasoning. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  69. Zhang, C., Jia, B., Edmonds, M., Zhu, S.C., Zhu, Y.: ACRE: abstract causal reasoning beyond covariation. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    Google Scholar 

  70. Zhang, C., Jia, B., Gao, F., Zhu, Y., Lu, H., Zhu, S.C.: Learning perceptual inference by contrasting. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)

    Google Scholar 

  71. Zhang, C., Jia, B., Zhu, S.C., Zhu, Y.: Abstract spatial-temporal reasoning via probabilistic abduction and execution. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    Google Scholar 

  72. Zhang, C., Zhu, Y., Zhu, S.C.: MetaStyle: three-way trade-off among speed, flexibility, and quality in neural style transfer. In: AAAI Conference on Artificial Intelligence (AAAI) (2019)

    Google Scholar 

  73. Zhang, W., Zhang, C., Zhu, Y., Zhu, S.C.: Machine number sense: a dataset of visual arithmetic problems for abstract and relational reasoning. In: AAAI Conference on Artificial Intelligence (AAAI) (2020)

    Google Scholar 

  74. Zheng, K., Zha, Z.J., Wei, W.: Abstract reasoning with distracting features. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)

    Google Scholar 

  75. Zhu, S.C., Mumford, D., et al.: A stochastic grammar of images. Found. Trends® Comput. Graph. Vis. 2(4), 259–362 (2007)

    Google Scholar 

  76. Zhu, Y., et al.: Dark, beyond deep: a paradigm shift to cognitive AI with humanlike common sense. Engineering 6(3), 310–345 (2020)

    Article  Google Scholar 

Download references

Acknowledgement

We thank Prof. Hongjing Lu and colleagues from UCLA for fruitful discussions. We would also like to thank anonymous reviewers for constructive feedback.

Author information

Authors and Affiliations

  1. University of California, Los Angeles, Los Angeles, CA, 90095, USA

    Chi Zhang, Sirui Xie, Baoxiong Jia, Ying Nian Wu & Song-Chun Zhu

  2. Institute for Artificial Intelligence, Peking University, Beijing, 10080, China

    Song-Chun Zhu & Yixin Zhu

  3. Tsinghua University, Beijing, 10080, China

    Song-Chun Zhu

  4. Beijing Institute for General Artificial Intelligence, Beijing, 10080, China

    Song-Chun Zhu

Authors
  1. Chi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sirui Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Baoxiong Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ying Nian Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Song-Chun Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yixin Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chi Zhang .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 579 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, C., Xie, S., Jia, B., Wu, Y.N., Zhu, SC., Zhu, Y. (2022). Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13699. Springer, Cham. https://doi.org/10.1007/978-3-031-19842-7_40

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19842-7_40

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19841-0

  • Online ISBN: 978-3-031-19842-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation