Skip to main content
SpringerLink
Log in

Continual Variational Autoencoder Learning via Online Cooperative Memorization

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13683))

Included in the following conference series:

Abstract

Due to their inference, data representation and reconstruction properties, Variational Autoencoders (VAE) have been successfully used in continual learning classification tasks. However, their ability to generate images with specifications corresponding to the classes and databases learned during Continual Learning (CL) is not well understood and catastrophic forgetting remains a significant challenge. In this paper, we firstly analyze the forgetting behaviour of VAEs by developing a new theoretical framework that formulates CL as a dynamic optimal transport problem. This framework proves approximate bounds to the data likelihood without requiring the task information and explains how the prior knowledge is lost during the training process. We then propose a novel memory buffering approach, namely the Online Cooperative Memorization (OCM) framework, which consists of a Short-Term Memory (STM) that continually stores recent samples to provide future information for the model, and a Long-Term Memory (LTM) aiming to preserve a wide diversity of samples. The proposed OCM transfers certain samples from STM to LTM according to the information diversity selection criterion without requiring any supervised signals. The OCM framework is then combined with a dynamic VAE expansion mixture network for further enhancing its performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abbasnejad, E., Dick, M., van der Hengel, A.: Infinite variational autoencoder for semi-supervised learning. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5888–5897 (2017)

    Google Scholar 

  2. Achille, A., et al.: Life-long disentangled representation learning with cross-domain latent homologies. In: Proceedings Advances in Neural Information Processing Systems (NeurIPS), pp. 9873–9883 (2018)

    Google Scholar 

  3. Aljundi, R., Lin, M., Goujaud, B., Bengio, Y.: Gradient based sample selection for online continual learning. In: Advances Neural Information Processing Systems (NeurIPS), vol. 33, pp. 11817–11826 (2019)

    Google Scholar 

  4. Aljundi, R., et al.: Online continual learning with maximal interfered retrieval. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 11872–11883 (2019)

    Google Scholar 

  5. Aljundi, R., Kelchtermans, K., Tuytelaars, T.: Task-free continual learning. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11254–11263 (2019)

    Google Scholar 

  6. Bang, J., Kim, H., Yoo, Y., Ha, J.W., Choi, J.: Rainbow memory: continual learning with a memory of diverse samples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8218–8227 (2021)

    Google Scholar 

  7. Belghazi, M.I., et al.: Mutual information neural estimation. In: Proceedings International Conference on Machine Learning (ICML), vol. PMLR 80, pp. 531–540 (2018)

    Google Scholar 

  8. Bousquet, O., Gelly, S., Tolstikhin, I., Simon-Gabriel, C.J., Schoelkopf, B.: From optimal transport to generative modeling: the VEGAN cookbook. arXiv preprint arXiv:1705.07642 (2017)

  9. Burda, Y., Grosse, R., Salakhutdinov, R.: Importance weighted autoencoders. arXiv preprint arXiv:1509.00519 (2015)

  10. Chaudhry, A., et al.: On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486 (2019)

  11. Chen, L., Dai, S., Pu, Y., Li, C., Su, Q., Carin, L.: Symmetric variational autoencoder and connections to adversarial learning. In: Proceedings International Conference on Artificial Intelligence and Statistics (AISTATS) 2018, vol. PMLR 84, pp. 661–669 (2018)

    Google Scholar 

  12. Courty, N., Flamary, R., Tuia, D., Rakotomamonjy, A.: Optimal transport for domain adaptation. IEEE Trans. Pattern Anal. Mach. Intell. 39(9), 1853–1865 (2016)

    Article  Google Scholar 

  13. De Lange, M., Tuytelaars, T.: Continual prototype evolution: learning online from non-stationary data streams. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8250–8259 (2021)

    Google Scholar 

  14. Egorov, E., Kuzina, A., Burnaev, E.: BooVAE: boosting approach for continual learning of VAE. Adv. Neural Inf. Process. Syst. (NeurIPS) 35, 17889–17901 (2021)

    Google Scholar 

  15. Fang, P., Harandi, M., Petersson, L.: Kernel methods in hyperbolic spaces. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10665–10674 (2021)

    Google Scholar 

  16. Fatras, K., Séjourné, T., Flamary, R., Courty, N.: Unbalanced minibatch optimal transport; applications to domain adaptation. In: International Conference on Machine Learning (ICML), vol. PMLR 139. pp. 3186–3197 (2021)

    Google Scholar 

  17. Goldberger, J., Gordon, S., Greenspan, H., et al.: An efficient image similarity measure based on approximations of kl-divergence between two gaussian mixtures. In: Proceedings IEEE International Conference on Computer Vision (ICCV), vol. 3, pp. 487–493 (2003)

    Google Scholar 

  18. Goodfellow, I., et al.: Generative adversarial nets. In: Proceedings Advances in Neural Information Proceedings Systems (NIPS), pp. 2672–2680 (2014)

    Google Scholar 

  19. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Proceedings Advances in Neural Information Processing Systems (NIPS), pp. 6626–6637 (2017)

    Google Scholar 

  20. Higgins, I., et al.: \(\beta \)-VAE: learning basic visual concepts with a constrained variational framework. In: Proceedings International Conference on Learning Representations (ICLR) (2017)

    Google Scholar 

  21. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: Proceedings NIPS Deep Learning Workshop. arXiv preprint arXiv:1503.02531 (2014)

  22. Hua, Y., Zhao, Z., Li, R., Chen, X., Liu, Z., Zhang, H.: Deep learning with long short-term memory for time series prediction. IEEE Commun. Mag. 57(6), 114–119 (2019)

    Article  Google Scholar 

  23. Jung, H., Ju, J., Jung, M., Kim, J.: Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122 (2016)

  24. Kantorovitch, L.: On the translocation of masses. Manag. Sci. 5(1), 1–4 (1958)

    Article  MathSciNet  MATH  Google Scholar 

  25. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 (2013)

  26. Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Nat. Acad. Sci. (PNAS) 114(13), 3521–3526 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  27. Knoblauch, J., Husain, H., Diethe, T.: Optimal continual learning has perfect memory and is NP-hard. In: Proceedings International Conference on Machine Learning (ICML), vol PMLR 119. pp. 5327–5337 (2020)

    Google Scholar 

  28. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report (2009)

    Google Scholar 

  29. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NIPS), pp. 1097–1105 (2012)

    Google Scholar 

  30. Kurle, R., Günnemann, S., van der Smagt, P.: Multi-source neural variational inference. In: Proceedings of AAAI Conference on Artificial Intelligence, vol. 33, pp. 4114–4121 (2019)

    Google Scholar 

  31. Lake, B.M., Salakhutdinov, R., Tenenbaum, J.B.: Human-level concept learning through probabilistic program induction. Science 350(6266), 1332–1338 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  32. Le, Y., Yang, X.: Tiny imagenet visual recognition challenge. CS 231N 7(7), 3 (2015)

    Google Scholar 

  33. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  34. Lee, S., Goldt, S., Saxe, A.: Continual learning in the teacher-student setup: impact of task similarity. In: International Conference on Machine Learning (ICML), vol. PMLR 139. pp. 6109–6119 (2021)

    Google Scholar 

  35. Lee, S., Ha, J., Zhang, D., Kim, G.: A neural Dirichlet process mixture model for task-free continual learning. In: Proceedings International Conference on Learning Representations (ICLR), arXiv preprint arXiv:2001.00689 (2020)

  36. Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2935–2947 (2017)

    Article  Google Scholar 

  37. Liu, H., Gu, X., Samaras, D.: Wasserstein GAN with quadratic transport cost. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4832–4841 (2019)

    Google Scholar 

  38. Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Advances in Neural Information Processing Systems, pp. 700–708 (2017)

    Google Scholar 

  39. Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. In: Advances in Neural Information Processing Systems, pp. 6467–6476 (2017)

    Google Scholar 

  40. Ma, X., Zhou, C., Hovy, E.: MAE: mutual posterior-divergence regularization for variational autoencoders. In: Proceedings International Conference on Learning Representations (ICLR), arXiv preprint arXiv:1901.01498 (2019)

  41. Nguyen, C.V., Li, Y., Bui, T.D., Turner, R.E.: Variational continual learning. In: Proceedings of International Conference on Learning Representations (ICLR), arXiv preprint arXiv:1710.10628 (2018)

  42. Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: a review. Neural Netw. 113, 54–71 (2019)

    Article  Google Scholar 

  43. Raghavan, K., Balaprakash, P.: Formalizing the generalization-forgetting trade-off in continual learning. In: Advances in Neural Information Processing Systems, vol. 34 (2021)

    Google Scholar 

  44. Ramapuram, J., Gregorova, M., Kalousis, A.: Lifelong generative modeling. In: Proceedings International Conference on Learning Representations (ICLR), arXiv preprint arXiv:1705.09847 (2017)

  45. Rao, D., Visin, F., Rusu, A.A., Teh, Y.W., Pascanu, R., Hadsell, R.: Continual unsupervised representation learning. In: Advances Neural Information Processing Systems (NeurIPS), pp. 7645–7655 (2019)

    Google Scholar 

  46. Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: incremental classifier and representation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2001–2010 (2017)

    Google Scholar 

  47. Ren, B., Wang, H., Li, J., Gao, H.: Life-long learning based on dynamic combination model. Appl. Soft Comput. 56, 398–404 (2017)

    Article  Google Scholar 

  48. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Proceedings Advances in Neural Information Processing Systems (NIPS), pp. 2234–2242 (2016)

    Google Scholar 

  49. Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: Advances in Neural Information Processing Systems (NIPS), pp. 2990–2999 (2017)

    Google Scholar 

  50. Sobolev, A., Vetrov, D.: Importance weighted hierarchical variational inference. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 33 (2019)

    Google Scholar 

  51. Takahashi, H., Iwata, T., Yamanaka, Y., Yamada, M., Yagi, S.: Variational autoencoder with implicit optimal priors. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5066–5073 (2019)

    Google Scholar 

  52. Tang, S., Chen, D., Zhu, J., Yu, S., Ouyang, W.: Layerwise optimization by gradient decomposition for continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9634–9643 (2021)

    Google Scholar 

  53. Titsias, M.K., Schwarz, J., Matthews, A.G.D.G., Pascanu, R., Teh, Y.W.: Functional regularisation for continual learning with Gaussian processes. In: Proceedings International Conference on Learning Represenations (ICLR), arXiv preprint arXiv:1901.11356 (2019)

  54. Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders. In: International Conference on Learning Representations (ICLR), arXiv preprint arXiv:1711.01558 (2018)

  55. Vitter, J.S.: Random sampling with a reservoir. ACM Trans. Math. Softw. (TOMS) 11(1), 37–57 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  56. Wang, S., Li, X., Sun, J., Xu, Z.: Training networks in null space of feature covariance for continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 184–193 (2021)

    Google Scholar 

  57. Ye, F., Bors, A.: Lifelong teacher-student network learning. IEEE Trans. Pattern Anal. Mach. Intell. (2021). https://doi.org/10.1109/TPAMI.2021.3092677

  58. Ye, F., Bors, A.G.: Learning latent representations across multiple data domains using lifelong VAEGAN. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12365, pp. 777–795. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58565-5_46

    Chapter  Google Scholar 

  59. Ye, F., Bors, A.G.: Lifelong learning of interpretable image representations. In: Proceedings International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 1–6 (2020)

    Google Scholar 

  60. Ye, F., Bors, A.G.: Mixtures of variational autoencoders. In: 2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 1–6 (2020)

    Google Scholar 

  61. Ye, F., Bors, A.G.: Deep mixture generative autoencoders. IEEE Trans. Neural Netw. Learn. Syst. 33, 1–15 (2021). https://doi.org/10.1109/TNNLS.2021.3071401

    Article  Google Scholar 

  62. Ye, F., Bors, A.G.: Infovaegan: learning joint interpretable representations by information maximization and maximum likelihood. In: Proceedings IEEE International Conference on Image Processing (ICIP), pp. 749–753 (2021). https://doi.org/10.1109/ICIP42928.2021.9506169

  63. Ye, F., Bors, A.G.: Learning joint latent representations based on information maximization. Inform. Sci. 567, 216–236 (2021)

    Article  MathSciNet  Google Scholar 

  64. Ye, F., Bors, A.G.: Lifelong infinite mixture model based on knowledge-driven Dirichlet process. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2021)

    Google Scholar 

  65. Ye, F., Bors, A.G.: Lifelong mixture of variational autoencoders. IEEE Trans. Neural Netw. Learn. Syst. 1–14 (2021). https://doi.org/10.1109/TNNLS.2021.3096457

  66. Ye, F., Bors, A.G.: Lifelong twin generative adversarial networks. In: Proceedings IEEE International Conference on Image Processing (ICIP), pp. 1289–1293 (2021)

    Google Scholar 

  67. Ye, F., Bors, A.G.: Learning an evolved mixture model for task-free continual learning (2022)

    Google Scholar 

  68. Ye, F., Bors, A.G.: Lifelong generative modelling using dynamic expansion graph model. In: AAAI on Artificial Intelligence. AAAI Press (2022)

    Google Scholar 

  69. Zhai, M., Chen, L., Tung, F., He, J., Nawhal, M., Mori, G.: Lifelong GAN: continual learning for conditional image generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2759–2768 (2019)

    Google Scholar 

  70. Zhao, S., Song, J., Ermon, S.: InfoVAE: balancing learning and inference in variational autoencoders. In: Proceedings AAAI Conference on Artificial Intelligence, vol. 33, pp. 5885–5892 (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of York, York YO10 5GH, UK

    Fei Ye & Adrian G. Bors

Authors
  1. Fei Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adrian G. Bors
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adrian G. Bors .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 4650 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ye, F., Bors, A.G. (2022). Continual Variational Autoencoder Learning via Online Cooperative Memorization. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13683. Springer, Cham. https://doi.org/10.1007/978-3-031-20050-2_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20050-2_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20049-6

  • Online ISBN: 978-3-031-20050-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation