Advertisement

GDumb: A Simple Approach that Questions Our Progress in Continual Learning

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12347)

Abstract

We discuss a general formulation for the Continual Learning (CL) problem for classification—a learning task where a stream provides samples to a learner and the goal of the learner, depending on the samples it receives, is to continually upgrade its knowledge about the old classes and learn new ones. Our formulation takes inspiration from the open-set recognition problem where test scenarios do not necessarily belong to the training distribution. We also discuss various quirks and assumptions encoded in recently proposed approaches for CL. We argue that some oversimplify the problem to an extent that leaves it with very little practical importance, and makes it extremely easy to perform well on. To validate this, we propose GDumb that (1) greedily stores samples in memory as they come and; (2) at test time, trains a model from scratch using samples only in the memory. We show that even though GDumb is not specifically designed for CL problems, it obtains state-of-the-art accuracies (often with large margins) in almost all the experiments when compared to a multitude of recently proposed algorithms. Surprisingly, it outperforms approaches in CL formulations for which they were specifically designed. This, we believe, raises concerns regarding our progress in CL for classification. Overall, we hope our formulation, characterizations and discussions will help in designing realistically useful CL algorithms, and GDumb will serve as a strong contender for the same.

Notes

Acknowledgements

AP would like to thank Aditya Bharti, Shyamgopal Karthik, Saujas Vaduguru, and Aurobindo Munagala for helpful discussions. PHS and PD thank EPSRC/MURI grant EP/N019474/1, and Facebook (DeepFakes grant) for their support. This project was supported by the Royal Academy of Engineering under the Research Chair and Senior Research Fellowships scheme. PHS and PD also acknowledge FiveAI UK.

Supplementary material

504434_1_En_31_MOESM1_ESM.pdf (107 kb)
Supplementary material 1 (pdf 106 KB)

References

  1. 1.
    McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of Learning and Motivation (1989)Google Scholar
  2. 2.
    Goodfellow, I.J., Mirza, M., Xiao, D., Courville, A., Bengio, Y.: An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 (2013)
  3. 3.
    Li, Z., Hoiem, D.: Learning without forgetting. TPAMI 40(12), 2935–2947 (2017)CrossRefGoogle Scholar
  4. 4.
    Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: incremental classifier and representation learning. In: CVPR (2017)Google Scholar
  5. 5.
    Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. ICML 70, 3987 (2017)Google Scholar
  6. 6.
    Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. PNAS 114(13), 3521–3526 (2017)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. In: NeurIP (2017)Google Scholar
  8. 8.
    Chaudhry, A., Dokania, P.K., Ajanthan, T., Torr, P.H.: Riemannian walk for incremental learning: understanding forgetting and intransigence. In: ECCV (2018)Google Scholar
  9. 9.
    De Lange, M., et al.: Continual learning: a comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv:1909.08383 (2019)
  10. 10.
    Scheirer, W., Rocha, A., Sapkota, A., Boult, T.: Towards open set recognition. TPAMI 35(7), 1757–1772 (2012)CrossRefGoogle Scholar
  11. 11.
    Aljundi, R., Caccia, L., Belilovsky, E., Caccia, M., Charlin, L., Tuytelaars, T.: Online continual learning with maximally interfered retrieval. In: NeurIPS (2019)Google Scholar
  12. 12.
    Jin, X., Du, J., Ren, X.: Gradient based memory editing for task-free continual learning (2020)Google Scholar
  13. 13.
    Dhar, P., Vikram Singh, R., Peng, K.C., Wu, Z., Chellappa, R.: Learning without memorizing. In: CVPR (2019)Google Scholar
  14. 14.
    Zhang, J., et al.: Class-incremental learning via deep model consolidation. In: WACV (2020)Google Scholar
  15. 15.
    Yu, L., et al.: Semantic drift compensation for class-incremental learning. In: CVPR (2020)Google Scholar
  16. 16.
    Wu, Y., et al.: Large scale incremental learning. In: CVPR (2019)Google Scholar
  17. 17.
    Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: CVPR (2019)Google Scholar
  18. 18.
    Castro, F.M., Marín-Jiménez, M.J., Guil, N., Schmid, C., Alahari, K.: End-to-end incremental learning. In: ECCV (2018)Google Scholar
  19. 19.
    Belouadah, E., Popescu, A.: Il2m: class incremental learning with dual memory. In: ICCV (2019)Google Scholar
  20. 20.
    Zhao, B., Xiao, X., Gan, G., Zhang, B., Xia, S.T.: Maintaining discrimination and fairness in class incremental learning. In: CVPR (2020)Google Scholar
  21. 21.
    Douillard, A., Cord, M., Ollion, C., Robert, T., Valle, E.: Small-task incremental learning. ECCV (2020)Google Scholar
  22. 22.
    Liu, Y., Su, Y., Liu, A.A., Schiele, B., Sun, Q.: Mnemonics training: multi-class incremental learning without forgetting. In: CVPR (2020)Google Scholar
  23. 23.
    Rajasegaran, J., Hayat, M., Khan, S., Khan, F.S., Shao, L.: Random path selection for incremental learning. In: NeurIPS (2019)Google Scholar
  24. 24.
    Rajasegaran, J., Khan, S., Hayat, M., Khan, F.S., Shah, M.: itaml: an incremental task-agnostic meta-learning approach. In: CVPR (2020)Google Scholar
  25. 25.
    Abati, D., Tomczak, J., Blankevoort, T., Calderara, S., Cucchiara, R., Bejnordi, B.E.: Conditional channel gated networks for task-aware continual learning. In: CVPR (2020)Google Scholar
  26. 26.
    Rusu, A.A., et al.: Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016)
  27. 27.
    Yoon, J., Lee, J., Yang, E., Hwang, S.J.: Lifelong learning with dynamically expandable network. In: ICLR (2018)Google Scholar
  28. 28.
    Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: NeurIPS (2017)Google Scholar
  29. 29.
    Schwarz, J., et al.: Progress & compress: a scalable framework for continual learning. ICML (2018)Google Scholar
  30. 30.
    Yoon, J., Kim, S., Yang, E., Hwang, S.J.: Scalable and order-robust continual learning with additive parameter decomposition. In: ICLR (2020)Google Scholar
  31. 31.
    Nguyen, C.V., Li, Y., Bui, T.D., Turner, R.E.: Variational continual learning. In: ICLR (2018)Google Scholar
  32. 32.
    Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: learning what (not) to forget. In: ECCV (2018)Google Scholar
  33. 33.
    Lee, S.W., Kim, J.H., Jun, J., Ha, J.W., Zhang, B.T.: Overcoming catastrophic forgetting by incremental moment matching. In: NeurIPS (2017)Google Scholar
  34. 34.
    Chaudhry, A., et al.: Continual learning with tiny episodic memories. ICML-W (2019)Google Scholar
  35. 35.
    Chaudhry, A., Gordo, A., Lopez-Paz, D., Dokania, P.K., Torr, P.: Using hindsight to anchor past knowledge in continual learning (2020)Google Scholar
  36. 36.
    Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with a-gem. In: ICLR (2019)Google Scholar
  37. 37.
    Aljundi, R., Lin, M., Goujaud, B., Bengio, Y.: Gradient based sample selection for online continual learning. In: NeurIPS (2019)Google Scholar
  38. 38.
    Tulving, E.: Episodic memory: from mind to brain. Ann. Rev. Psychol. 53(1), 1–25 (2002)CrossRefGoogle Scholar
  39. 39.
    Norman, K.A., O’Reilly, R.C.: Modeling hippocampal and neocortical contributions to recognition memory: a complementary-learning-systems approach. Psychol. Rev. 110(4), 611 (2003)CrossRefGoogle Scholar
  40. 40.
    Ren, M., Iuzzolino, M.L., Mozer, M.C., Zemel, R.S.: Wandering within a world: online contextualized few-shot learning. arXiv preprint arXiv:2007.04546 (2020)
  41. 41.
    Ji, X., Henriques, J., Tuytelaars, T., Vedaldi, A.: Automatic recall machines: internal replay, continual learning and the brain. arXiv preprint arXiv:2006.12323 (2020)
  42. 42.
    Hsu, Y.C., Liu, Y.C., Kira, Z.: Re-evaluating continual learning scenarios: a categorization and case for strong baselines. In: NeurIPS-W (2018)Google Scholar
  43. 43.
    Riemer, M., et al.: Learning to learn without forgetting by maximizing transfer and minimizing interference. In: ICLR (2019)Google Scholar
  44. 44.
    Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T.P., Wayne, G.: Experience replay for continual learning. In: NeurIPS (2019)Google Scholar
  45. 45.
    Loshchilov, I., Hutter, F.: Sgdr: stochastic gradient descent with warm restarts. In: ICLR (2017)Google Scholar
  46. 46.
    Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: regularization strategy to train strong classifiers with localizable features. In: ICCV (2019)Google Scholar
  47. 47.
    Yin, H., et al.: Dreaming to distill: data-free knowledge transfer via deepinversion. In: CVPR (2020)Google Scholar
  48. 48.
    Zeno, C., Golan, I., Hoffer, E., Soudry, D.: Task agnostic continual learning using online variational bayes. arXiv preprint arXiv:1803.10123 (2018)
  49. 49.
    Hocquet, G., Bichler, O., Querlioz, D.: Ova-inn: continual learning with invertible neural networks. IJCNN (2020)Google Scholar
  50. 50.
    van de Ven, G.M., Tolias, A.S.: Generative replay with feedback connections as a general strategy for continual learning. arXiv preprint arXiv:1809.10635 (2018)
  51. 51.
    Serra, J., Suris, D., Miron, M., Karatzoglou, A.: Overcoming catastrophic forgetting with hard attention to the task. ICML (2018)Google Scholar
  52. 52.
    Rannen, A., Aljundi, R., Blaschko, M.B., Tuytelaars, T.: Encoder based lifelong learning. In: CVPR (2017)Google Scholar
  53. 53.
    Mallya, A., Lazebnik, S.: Packnet: adding multiple tasks to a single network by iterative pruning. In: CVPR (2018)Google Scholar
  54. 54.
    Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of OxfordOxfordUK
  2. 2.Five AI Ltd.OxfordUK

Personalised recommendations