Skip to main content
SpringerLink
Log in

SPeCiaL: Self-supervised Pretraining for Continual Learning

  • Conference paper
  • First Online:
Continual Semi-Supervised Learning (CSSL 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13418))

Included in the following conference series:

Abstract

This paper presents SPeCiaL: a method for unsupervised pretraining of representations tailored for continual learning. Our approach devises a meta-learning objective that differentiates through a sequential learning process. Specifically, we train a linear model over the representations to match different augmented views of the same image together, each view presented sequentially. The linear model is then evaluated on both its ability to classify images it just saw, and also on images from previous iterations. This gives rise to representations that favor quick knowledge retention with minimal forgetting. We evaluate SPeCiaL in the Continual Few-Shot Learning setting, and show that it can match or outperform other supervised pretraining approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Achille, A., et al.: Life-long disentangled representation learning with cross-domain latent homologies. arXiv preprint arXiv:1808.06508 (2018)

  2. Aljundi, R., et al.: Online continual learning with maximally interfered retrieval. arXiv preprint arXiv:1908.04742 (2019)

  3. Aljundi, R., Lin, M., Goujaud, B., Bengio, Y.: Gradient based sample selection for online continual learning. arXiv preprint arXiv:1903.08671 (2019)

  4. Aljundi, R., Rohrbach, M., Tuytelaars, T.: Selfless sequential learning. arXiv preprint arXiv:1806.05421 (2018)

  5. Antoniou, A., Storkey, A.: Assume, augment and learn: Unsupervised few-shot meta-learning via random labels and data augmentation. arXiv preprint arXiv:1902.09884 (2019)

  6. Assran, M., Ballas, N., Castrejon, L., Rabbat, M.: Supervision accelerates pre-training in contrastive semi-supervised learning of visual representations. arXiv preprint arXiv:2006.10803 (2020)

  7. Beaulieu, S., et al.: Learning to continually learn. arXiv preprint arXiv:2002.09571 (2020)

  8. Bertugli, A., Vincenzi, S., Calderara, S., Passerini, A.: Generalising via meta-examples for continual learning in the wild. arXiv preprint arXiv:2101.12081 (2021)

  9. Caccia, L., Belilovsky, E., Caccia, M., Pineau, J.: Online learned continual compression with adaptive quantization modules. In: International Conference on Machine Learning, pp. 1240–1250. PMLR (2020)

    Google Scholar 

  10. Caccia, M., et al.: Online fast adaptation and knowledge accumulation: a new approach to continual learning. arXiv preprint arXiv:2003.05856 (2020)

  11. Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882 (2020)

  12. Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420 (2018)

  13. Chaudhry, A., et al.: Continual learning with tiny episodic memories (2019)

    Google Scholar 

  14. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)

    Google Scholar 

  15. Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020)

  16. Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: International Conference on Machine Learning, pp. 1126–1135. PMLR (2017)

    Google Scholar 

  17. French, R.M.: Using semi-distributed representations to overcome catastrophic forgetting in connectionist networks. In: Proceedings of the 13th Annual Cognitive Science Society Conference, vol. 1, pp. 173–178 (1991)

    Google Scholar 

  18. Gupta, G., Yadav, K., Paull, L.: Look-ahead meta learning for continual learning. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 11588–11598. Curran Associates, Inc. (2020). https://proceedings.neurips.cc/paper/2020/file/85b9a5ac91cd629bd3afe396ec07270a-Paper.pdf

  19. He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738 (2020)

    Google Scholar 

  20. Hsu, K., Levine, S., Finn, C.: Unsupervised learning via meta-learning. arXiv preprint arXiv:1810.02334 (2018)

  21. Javed, K., White, M.: Meta-learning representations for continual learning. arXiv preprint arXiv:1905.12588 (2019)

  22. Khodadadeh, S., Bölöni, L., Shah, M.: Unsupervised meta-learning for few-shot image classification. arXiv preprint arXiv:1811.11819 (2018)

  23. Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114(13), 3521–3526 (2017)

    Article  MathSciNet  Google Scholar 

  24. Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

    Google Scholar 

  25. Liu, V., Kumaraswamy, R., Le, L., White, M.: The utility of sparse representations for control in reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 4384–4391 (2019)

    Google Scholar 

  26. Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. arXiv preprint arXiv:1706.08840 (2017)

  27. Nguyen, C.V., Li, Y., Bui, T.D., Turner, R.E.: Variational continual learning. arXiv preprint arXiv:1710.10628 (2017)

  28. Qiao, S., Wang, H., Liu, C., Shen, W., Yuille, A.: Weight standardization. arXiv preprint arXiv:1903.10520 (2019)

  29. Rao, D., Visin, F., Rusu, A.A., Teh, Y.W., Pascanu, R., Hadsell, R.: Continual unsupervised representation learning. arXiv preprint arXiv:1910.14481 (2019)

  30. Ravi, S., Larochelle, H.: Optimization as a model for few-shot learning (2016)

    Google Scholar 

  31. Riemer, M., et al.: Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910 (2018)

  32. Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. arXiv preprint arXiv:1705.08690 (2017)

  33. Smith, J., Baer, S., Taylor, C., Dovrolis, C.: Unsupervised progressive learning and the STAM architecture. arXiv preprint arXiv:1904.02021 (2019)

  34. Snell, J., Swersky, K., Zemel, R.S.: Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175 (2017)

  35. Van de Ven, G.M., Tolias, A.S.: Generative replay with feedback connections as a general strategy for continual learning. arXiv preprint arXiv:1809.10635 (2018)

  36. Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., Wierstra, D.: Matching networks for one shot learning. arXiv preprint arXiv:1606.04080 (2016)

  37. Wu, Y., He, K.: Group normalization. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

    Google Scholar 

  38. Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: International Conference on Machine Learning, pp. 3987–3995. PMLR (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. MILA, McGill University, Montreal, Canada

    Lucas Caccia & Joelle Pineau

  2. Facebook AI Research, Montreal, Canada

    Lucas Caccia & Joelle Pineau

Authors
  1. Lucas Caccia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joelle Pineau
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lucas Caccia .

Editor information

Editors and Affiliations

  1. Oxford Brookes University, Oxford, UK

    Fabio Cuzzolin

  2. Huawei Technologies Canada Co., Ltd., Burnaby, BC, Canada

    Kevin Cannons

  3. University of Pisa, Pisa, Italy

    Vincenzo Lomonaco

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Caccia, L., Pineau, J. (2022). SPeCiaL: Self-supervised Pretraining for Continual Learning. In: Cuzzolin, F., Cannons, K., Lomonaco, V. (eds) Continual Semi-Supervised Learning. CSSL 2021. Lecture Notes in Computer Science(), vol 13418. Springer, Cham. https://doi.org/10.1007/978-3-031-17587-9_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-17587-9_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-17586-2

  • Online ISBN: 978-3-031-17587-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation