Skip to main content
SpringerLink
Log in

Online Task-free Continual Learning with Dynamic Sparse Distributed Memory

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13685))

Included in the following conference series:

Abstract

This paper addresses the very challenging problem of online task-free continual learning in which a sequence of new tasks is learned from non-stationary data using each sample only once for training and without knowledge of task boundaries. We propose in this paper an efficient semi-distributed associative memory algorithm called Dynamic Sparse Distributed Memory (DSDM) where learning and evaluating can be carried out at any point of time. DSDM evolves dynamically and continually modeling the distribution of any non-stationary data stream. DSDM relies on locally distributed, but only partially overlapping clusters of representations to effectively eliminate catastrophic forgetting, while at the same time, maintaining the generalization capacities of distributed networks. In addition, a local density-based pruning technique is used to control the network’s memory footprint. DSDM significantly outperforms state-of-the-art continual learning methods on different image classification baselines, even in a low data regime. Code is publicly available: https://github.com/Julien-pour/Dynamic-Sparse-Distributed-Memory.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aggarwal, C.C., Hinneburg, A., Keim, D.A.: On the surprising behavior of distance metrics in high dimensional space. In: Van den Bussche, J., Vianu, V. (eds.) 8th International Conference on Database Theory, pp. 420–434. Springer (2001)

    Google Scholar 

  2. Aljundi, R., et al.: Online continual learning with maximal interfered retrieval. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  3. Aljundi, R., Lin, M., Goujaud, B., Bengio, Y.: Gradient based sample selection for online continual learning. Adv. Neural Inf. Process. Syst. 32 (2019)

    Google Scholar 

  4. Ans, B., Rousset, S., French, R.M., Musca, S.: Self-refreshing memory in artificial neural networks: learning temporal sequences without catastrophic forgetting. Connection Sci. 16(2), 71–99 (2004)

    Article  Google Scholar 

  5. Breunig, M.M., Kriegel, H.P., Ng, R.T., Sander, J.: LOF: identifying density-based local outliers, p. 12 (2000)

    Google Scholar 

  6. Bricken, T., Pehlevan, C.: Attention approximates sparse distributed memory. Adv. Neural Inf. Process. Syst. 34, 15301–15315 (2021)

    Google Scholar 

  7. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650–9660 (2021)

    Google Scholar 

  8. Carpenter, G., Grossberg, S.: ART 2: self-organization of stable category recognition codes for analog input patterns. Appl. Opt.26, 4919–4930 (1987)

    Google Scholar 

  9. Castro, F.M., Marín-Jiménez, M.J., Guil, N., Schmid, C., Alahari, K.: End-to-end incremental learning. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Proceedings of the European conference on computer vision (ECCV), pp. 233–248 (2018)

    Google Scholar 

  10. Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with A-GEM. In: International Conference on Learning Representations (2018)

    Google Scholar 

  11. De Lange, M., Tuytelaars, T.: Continual prototype evolution: learning online from non-stationary data streams. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8250–8259 (2021)

    Google Scholar 

  12. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)

    Google Scholar 

  13. Deng, L.: The MNIST database of handwritten digit images for machine learning research. IEEE Sign. Process. Mag. 29(6), 141–142 (2012)

    Article  Google Scholar 

  14. Fahlman, S.E., Lebiere, C.: The cascade-correlation learning architecture. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems 2, [NIPS Conference, Denver, Colorado, USA, November 27–30, 1989], pp. 524–532. Morgan Kaufmann (1989). http://papers.nips.cc/paper/207-the-cascade-correlation-learning-architecture

  15. French, R.M.: Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference. In: Proceedings of the 16th Annual Cognitive Science Society Conference, pp. 335–340 (1994)

    Google Scholar 

  16. French, R.M.: Pseudo-recurrent connectionist networks: an approach to the sensitivity-stability dilemma. Connection Sci. 9(4), 353–380 (1997)

    Google Scholar 

  17. French, R.M.: Catastrophic forgetting in connectionist networks. Trends Cogn. Sci. 3(4), 128–135 (1999)

    Google Scholar 

  18. French, R.M., Chater, N.: Using noise to compute error surfaces in connectionist networks: a novel means of reducing catastrophic forgetting. Neural Comput. 14(7), 1755–1769 (2002)

    Google Scholar 

  19. Goodfellow, I.J., et al.: Generative Adversarial Networks. arXiv:1406.2661 (2014)

  20. Goyal, P., et al.: Self-supervised Pretraining of Visual Features in the Wild. arXiv:2103.01988 (2021)

  21. He, J., Zhu, F.: Online continual learning via candidates voting. In: 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3154–3163 (2022)

    Google Scholar 

  22. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  23. Hely, T.A., Willshaw, D.J., Hayes, G.M.: A new approach to Kanerva’s sparse distributed memory. IEEE Trans. Neural Netw. 8(3), 791–794 (1997). https://doi.org/10.1109/72.572115

    Article  Google Scholar 

  24. Hinton, G., Vinyals, O., Dean, J.: Distilling the Knowledge in a Neural Network. arXiv:1503.02531 (2015)

  25. Jezequel, L., Vu, N., Beaudet, J., Histace, A.: Efficient anomaly detection using self-supervised multi-cue tasks. CoRR arXiv:abs/2111.12379 (2021)

  26. Kanerva, P.: A cerebellar-model associative memory as a generalized random-access memory. In: Thirty-Fourth IEEE Computer Society International Conference: Intellectual Leverage, pp. 770–778 (1989)

    Google Scholar 

  27. Kanerva, P.: Sparse distributed memory and related models. Tech. Rep. NASA-CR-190553, keeler (1992)

    Google Scholar 

  28. Keeler, J.: Capacity for patterns and sequences in Kanerva’ s SDM as compared to other associative memory models. In: Neural Information Processing Systems. American Institute of Physics (1988)

    Google Scholar 

  29. Kemker, R., Kanan, C.: FearNet: Brain-inspired model for incremental learning. In: International Conference on Learning Representations (2018)

    Google Scholar 

  30. Kingma, D.P., Welling, M.: Auto-Encoding Variational Bayes. arXiv:1312.6114 (May 2014)

  31. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., Hadsell, R.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114(13), 3521–3526 (2017)

    Article  MathSciNet  Google Scholar 

  32. Krizhevsky, A.: Learning Multiple Layers of Features from Tiny Images (2009)

    Google Scholar 

  33. Kruschke, J.K.: Human category learning: implications for backpropagation models. Connection Sci. 5(1), 3–36 (1993)

    Google Scholar 

  34. Lee, S., Ha, J., Zhang, D., Kim, G.: A neural dirichlet process mixture model for task-free continual learning. In: 8th ICLR 2020. OpenReview.net (2020). https://openreview.net/forum?id=SJxSOJStPr

  35. Lewandowsky, S.: Gradual unlearning and catastrophic interference: a comparison of distributed architectures. In: Relating theory and data: Essays on human memory in honor of Bennet B. Murdock, pp. 445–476. Lawrence Erlbaum Associates Inc, Hillsdale, NJ, US (1991)

    Google Scholar 

  36. Li, Z., Hoiem, D.: Learning without forgetting. In: 14th European Conference on Computer Vision, ECCV 2016. Computer Vision - 14th European Conference, ECCV 2016, Proceedings, pp. 614–629 (2016)

    Google Scholar 

  37. Lomonaco, V., Maltoni, D.: CORe50: a new dataset and benchmark for continuous object recognition. In: CoRL, pp. 17–26 (2017)

    Google Scholar 

  38. Lopez-Paz, D., Ranzato, M.A.: Gradient episodic memory for continual learning. Adv. Neural Inf. Process. Syst. 30(2017)

    Google Scholar 

  39. Mallya, A., Lazebnik, S.: PackNet: adding multiple tasks to a single network by iterative pruning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7765–7773 (2018)

    Google Scholar 

  40. Marsland, S., Shapiro, J., Nehmzow, U.: A self-organising network that grows when required. Neural Netw. 15(8-9), 1041–1058 (2002)

    Google Scholar 

  41. McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Bower, G.H. (ed.) Psychology of Learning and Motivation, vol. 24, pp. 109–165 (1989)

    Google Scholar 

  42. Orhan, A.E., Gupta, V.V., Lake, B.M.: Self-supervised learning through the eyes of a child. Adv. Neural Inf. Process. Syst. (2020)

    Google Scholar 

  43. Ostapenko, O., Puscas, M., Klein, T., Jähnichen, P., Nabi, M.: Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning. arXiv:1904.03137 [cs] (2019)

  44. Ostapenko, O., Puscas, M.M., Klein, T., Jähnichen, P., Nabi, M.: Learning to remember: a synaptic plasticity driven framework for continual learning. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16–20, 2019, pp. 11321–11329. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPR.2019.01158

  45. Prabhu, A., Torr, P.H.S., Dokania, P.K.: GDumb: a Simple approach that questions our progress in continual learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) Computer Vision - ECCV 2020, vol. 12347, pp. 524–540 (2020)

    Google Scholar 

  46. Ramsauer, H., et al.: Hopfield Networks is All You Need. arXiv:2008.02217 (2021)

  47. Rao, D., Visin, F., Rusu, A.A., Teh, Y.W., Pascanu, R., Hadsell, R.: Continual unsupervised representation learning. CoRR arXiv:abs/1910.14481 (2019)

  48. Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: incremental classifier and representation learning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2001–2010 (2017)

    Google Scholar 

  49. R.M, F.: Semi-distributed representations and catastrophic forgetting in connectionist networks. Connection Sci. 4(3-4), 365–377 (1992)

    Google Scholar 

  50. Robins, A.: Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Sci. 7(2), 123–146 (1995)

    Google Scholar 

  51. Roy, D., Panda, P., Roy, K.: Tree-CNN: a hierarchical deep convolutional neural network for incremental learning. Neural Netw. (2019)

    Google Scholar 

  52. Schwarz, Jet al.: Progress & compress: a scalable framework for continual learning. In: Proceedings of the 35th International Conference on Machine Learning, pp. 4528–4537 (2018)

    Google Scholar 

  53. Serrà, J., Surís, D., Miron, M., Karatzoglou, A.: Overcoming catastrophic forgetting with hard attention to the task. Int. Conf. Mach. Learn. 4548–4557 (2018)

    Google Scholar 

  54. Shanahan, M., Kaplanis, C., Mitrović, J.: Encoders and Ensembles for Task-Free Continual Learning. arXiv:2105.13327 (2021)

  55. Shim, D., Mai, Z., Jeong, J., Sanner, S., Kim, H., Jang, J.: Online class-incremental continual learning with adversarial shapley value. In: Proceedings of the AAAI Conference on Artificial Intelligence 35(11), 9630–9638 (2021)

    Google Scholar 

  56. van de Ven, G.M., Siegelmann, H.T., Tolias, A.S.: Brain-inspired replay for continual learning with artificial neural networks. Nat. Commun. 11(1), 1–14 (2020)

    Google Scholar 

  57. van de Ven, G.M., Tolias, A.S.: Three scenarios for continual learning. arXiv:1904.07734 (2019)

  58. Wang, Y.X., Ramanan, D., Hebert, M.: Growing a brain: fine-tuning by increasing model capacity. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2471–2480 (2019)

    Google Scholar 

  59. Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-UCSD birds 200 (2010)

    Google Scholar 

  60. Wu, C., Herranz, L., Liu, X., wang, y., van de Weijer, J., Raducanu, B.: Memory replay GANs: learning to generate new categories without forgetting. Adv. Neural Inf. Process. Syst. 31 (2018)

    Google Scholar 

  61. Xiang, Y., Fu, Y., Ji, P., Huang, H.: Incremental learning using conditional adversarial networks. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6619–6628 (2019)

    Google Scholar 

  62. Yoon, J., Yang, E., Lee, J., Hwang, S.J.: Lifelong learning with dynamically expandable networks. In: International Conference on Learning Representations (2018)

    Google Scholar 

  63. Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: Proceedings of the 34th International Conference on Machine Learning, pp. 3987–3995 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. ETIS - CYU, ENSEA, CNRS, Paris, France

    Julien Pourcel & Ngoc-Son Vu

  2. LEAD - UBFC, CNRS, Paris, France

    Robert M. French

Authors
  1. Julien Pourcel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ngoc-Son Vu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Robert M. French
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Julien Pourcel .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pourcel, J., Vu, NS., French, R.M. (2022). Online Task-free Continual Learning with Dynamic Sparse Distributed Memory. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13685. Springer, Cham. https://doi.org/10.1007/978-3-031-19806-9_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19806-9_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19805-2

  • Online ISBN: 978-3-031-19806-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation