Skip to main content

More Diverse Training, Better Compositionality! Evidence from Multimodal Language Learning

  • 1544 Accesses

Part of the Lecture Notes in Computer Science book series (LNCS,volume 13531)


Artificial neural networks still fall short of human-level generalization and require a very large number of training examples to succeed. Model architectures that further improve generalization capabilities are therefore still an open research question. We created a multimodal dataset from simulation for measuring the compositional generalization of neural networks in multimodal language learning. The dataset consists of sequences showing a robot arm interacting with objects on a table in a simple 3D environment, with the goal of describing the interaction. Compositional object features, multiple actions, and distracting objects pose challenges to the model. We show that an LSTM-encoder-decoder architecture jointly trained together with a vision-encoder surpasses previous performance and handles multiple visible objects. Visualization of important input dimensions shows that a model that is trained with multiple objects, but not a model trained on just one object, has learnt to ignore irrelevant objects. Furthermore we show that additional modalities in the input improve the overall performance. We conclude that the underlying training data has a significant influence on the model’s capability to generalize compositionally.


  • Compositional generalization
  • Computer vision
  • Multimodality
  • Sequence-to-sequence
  • Robotics

The authors acknowledge support from the German Research Foundation DFG under project CML (TRR 169) and from the BMWK under project SiDiMo.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


  1. 1.

    The source code for the model and the data generation can be found at this link:


  1. Eisermann, A., Lee, J.H., Weber, C., Wermter, S.: Generalization in multimodal language learning from simulation. In: Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021)(2021)

    Google Scholar 

  2. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Adaptive Computation and Machine Learning (2016)

    MATH  Google Scholar 

  3. Greff, K., van Steenkiste, S., Schmidhuber, J.: On the binding problem in artificial neural networks. arXiv:2012.05208 (2020)

  4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv:1512.03385 (2015)

  5. Heinrich, S., Kerzel, M., Strahl, E., Wermter, S.: Embodied multi-modal interaction in language learning: the EMIL data collection. In: Proceedings of the ICDL-EpiRob Workshop on Active Vision, Attention, and Learning (ICDL-Epirob 2018 AVAL). Tokyo, Japan (2018)

    Google Scholar 

  6. Heinrich, S., et al.: Crossmodal language grounding in an embodied neurocognitive model. Front. Neurorobotics 14 (2020)

    Google Scholar 

  7. Keysers, D., et al.: Measuring compositional generalization: a comprehensive method on realistic data. arXiv:1912.09713 (2019)

  8. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv:1412.6980 (2017)

  9. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems. vol. 25, Curran Associates, Inc. (2012)

    Google Scholar 

  10. Lake, B.M., Baroni, M.: Generalization without systematicity: on the compositional skills of sequence-to-sequence recurrent networks. arXiv:1711.00350 (2017)

  11. Lake, B.M., Ullman, T.D., Tenenbaum, J.B., Gershman, S.J.: Building machines that learn and think like people. arXiv:1604.00289 (2016)

  12. LeCun, Y.: Generalization and network design strategies. Technical Report CRG-TR-89-4, University of Toronto (1989)

    Google Scholar 

  13. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–44 (2015)

    Google Scholar 

  14. Loula, J., Baroni, M., Lake, B.M.: Rearranging the familiar: testing compositional generalization in recurrent networks. arXiv:1807.07545 (2018)

  15. Montague, R.: Universal Grammar, vol. 36. Blackwell Publishing Ltd. (1970)

    Google Scholar 

  16. Ruis, L., Andreas, J., Baroni, M., Bouchacourt, D., Lake, B.M.: A benchmark for systematic generalization in grounded language understanding. arXiv:2003.05161 (2020)

  17. Russakovsky, O.: ImageNet large scale visual recognition challenge. arXiv:1409.0575 (2014)

  18. Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: International Conference on Machine Learning, pp. 3319–3328, PMLR (2017)

    Google Scholar 

  19. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. arXiv:1409.3215 (2014)

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Caspar Volquardsen .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Volquardsen, C., Lee, J.H., Weber, C., Wermter, S. (2022). More Diverse Training, Better Compositionality! Evidence from Multimodal Language Learning. In: Pimenidis, E., Angelov, P., Jayne, C., Papaleonidas, A., Aydin, M. (eds) Artificial Neural Networks and Machine Learning – ICANN 2022. ICANN 2022. Lecture Notes in Computer Science, vol 13531. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-15933-6

  • Online ISBN: 978-3-031-15934-3

  • eBook Packages: Computer ScienceComputer Science (R0)