Topology-Preserving Class-Incremental Learning

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12364)


A well-known issue for class-incremental learning is the catastrophic forgetting phenomenon, where the network’s recognition performance on old classes degrades severely when incrementally learning new classes. To alleviate forgetting, we put forward to preserve the old class knowledge by maintaining the topology of the network’s feature space. On this basis, we propose a novel topology-preserving class-incremental learning (TPCIL) framework. TPCIL uses an elastic Hebbian graph (EHG) to model the feature space topology, which is constructed with the competitive Hebbian learning rule. To maintain the topology, we develop the topology-preserving loss (TPL) that penalizes the changes of EHG’s neighboring relationships during incremental learning phases. Comprehensive experiments on CIFAR100, ImageNet, and subImageNet datasets demonstrate the power of the TPCIL for continuously learning new classes with less forgetting. The code will be released.


Topology-Preserving Class-Incremental Learning (TPCIL) Class-Incremental Learning (CIL) Elastic Hebbian Graph (EHG) Topology-Preserving Loss (TPL) 



This work is sponsored by National Key R&D Program of China under Grand No.2019YFB1312000, National Major Project under Grant No.2017YFC0803905 and SHAANXI Province Joint Key Laboratory of Machine Learning.


  1. 1.
    Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: learning what (not) to forget. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 144–161. Springer, Cham (2018). Scholar
  2. 2.
    Burges, C.J., Ragno, R., Le, Q.V.: Learning to rank with nonsmooth cost functions. In: NeurIPS (2007)Google Scholar
  3. 3.
    Castro, F.M., Marín-Jiménez, M.J., Guil, N., Schmid, C., Alahari, K.: End-to-end incremental learning. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11216, pp. 241–257. Springer, Cham (2018). Scholar
  4. 4.
    Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with A-GEM. arXiv preprint arXiv:1812.00420 (2018)
  5. 5.
    Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)
  6. 6.
    Chen, L.: The topological approach to perceptual organization. Vis. Cogn. 12(4), 553–637 (2005)CrossRefGoogle Scholar
  7. 7.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Li, F.F.: ImageNet: a large-scale hierarchical image database. In: CVPR (2009)Google Scholar
  8. 8.
    Deng, J., Guo, J., Xue, N., Zafeiriou, S.: ArcFace: additive angular margin loss for deep face recognition. arXiv preprint arXiv:1801.07698 (2018)
  9. 9.
    Eden, B., Adrian, P.: IL2M: class incremental learning with dual memory. In: ICCV (2019)Google Scholar
  10. 10.
    French, R.M.: Catastrophic forgetting in connectionist networks. Trends Cogn. Sci. 3(4), 128–135 (1999)CrossRefGoogle Scholar
  11. 11.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015)
  12. 12.
    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. Comput. Sci. 14(7), 38–39 (2015)Google Scholar
  13. 13.
    Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: CVPR (2019)Google Scholar
  14. 14.
    Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Nat. Acad. Sci. 114(13), 3521–3526 (2017)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Tech. rep. Citeseer (2009)Google Scholar
  16. 16.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: NeurIPS (2012)Google Scholar
  17. 17.
    Lee, S.W., Kim, J.H., Jun, J., Ha, J.W., Zhang, B.T.: Overcoming catastrophic forgetting by incremental moment matching. In: NeurIPS (2017)Google Scholar
  18. 18.
    Lee, S., Song, B.C.: Graph-based knowledge distillation by multi-head attention network. In: BMVC (2019)Google Scholar
  19. 19.
    Li, Z., Hoiem, D.: Learning without forgetting. T-PAMI 40(12), 2935–2947 (2018)CrossRefGoogle Scholar
  20. 20.
    Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., Song, L.: SphereFace: deep hypersphere embedding for face recognition. In: CVPR (2017)Google Scholar
  21. 21.
    Liu, Z., Sun, M., Zhou, T., Huang, G., Darrell, T.: Rethinking the value of network pruning. In: ICLR (2019)Google Scholar
  22. 22.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)Google Scholar
  23. 23.
    Lopez-Paz, D., et al.: Gradient episodic memory for continual learning. In: NeurIPS (2017)Google Scholar
  24. 24.
    Ma, Z., Wei, X., Hong, X., Gong, Y.: Bayesian loss for crowd count estimation with point supervision. In: ICCV (2019)Google Scholar
  25. 25.
    Mallya, A., Davis, D., Lazebnik, S.: Piggyback: adapting a single network to multiple tasks by learning to mask weights. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 72–88. Springer, Cham (2018). Scholar
  26. 26.
    Mallya, A., Lazebnik, S.: PackNet: adding multiple tasks to a single network by iterative pruning. In: CVPR (2018)Google Scholar
  27. 27.
    Maltoni, D., Lomonaco, V.: Continuous learning in single-incremental-task scenarios. arXiv preprint arXiv:1806.08568 (2018)
  28. 28.
    Martinetz, T.M.: Competitive Hebbian learning rule forms perfectly topology preserving maps. In: International Conference on Artificial Neural Networks, pp. 427–434 (1993)Google Scholar
  29. 29.
    Martinetz, T., Schulten, K.: Topology representing networks. Neural Netw. 7(3), 507–522 (1994)CrossRefGoogle Scholar
  30. 30.
    Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: a review. Neural Netw. 113, 54–71 (2019)CrossRefGoogle Scholar
  31. 31.
    Park, W., Kim, D., Lu, Y., Cho, M.: Relational knowledge distillation. In: CVPR (2019)Google Scholar
  32. 32.
    Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: incremental classifier and representation learning. In: CVPR (2017)Google Scholar
  33. 33.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. arXiv preprint arXiv:1506.02640 (2015)
  34. 34.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NeurIPS (2015)Google Scholar
  35. 35.
    Serrà, J., Suris, D., Miron, M., Karatzoglou, A.: Overcoming catastrophic forgetting with hard attention to the task. arXiv preprint arXiv:1801.01423 (2018)
  36. 36.
    Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: NeurIPS (2017)Google Scholar
  37. 37.
    Tao, X., Hong, X., Chang, X., Dong, S., Xing, W., Yihong, G.: Few-shot class-incremental learning. In: CVPR (2020)Google Scholar
  38. 38.
    Tao, X., Hong, X., Chang, X., Gong, Y.: Bi-objective continual learning: learning ‘new’ while consolidating ‘known’. In: AAAI, February 2020Google Scholar
  39. 39.
    Wei, N., Zhou, T., Zhang, Z., Zhuo, Y., Chen, L.: Visual working memory representation as a topological defined perceptual object. J. Vis. 19(7), 1–12 (2019)CrossRefGoogle Scholar
  40. 40.
    Wei, X., Zhang, Y., Gong, Y., Zhang, J., Zheng, N.: Grassmann pooling as compact homogeneous bilinear pooling for fine-grained visual classification. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 365–380. Springer, Cham (2018). Scholar
  41. 41.
    Wei, X., Zhang, Y., Gong, Y., Zheng, N.: Kernelized subspace pooling for deep local descriptors. In: CVPR (2018)Google Scholar
  42. 42.
    Wu, Y., et al.: Large scale incremental learning. In: CVPR (2019)Google Scholar
  43. 43.
    Yim, J., Joo, D., Bae, J., Kim, J.: A gift from knowledge distillation: fast optimization, network minimization and transfer learning. In: CVPR (2017)Google Scholar
  44. 44.
    Yoon, J., Yang, E., Lee, J., Hwang, S.J.: Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547 (2017)
  45. 45.
    Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: ICML (2017)Google Scholar
  46. 46.
    Zhai, M., Chen, L., Tung, F., He, J., Nawhal, M., Mori, G.: Lifelong GAN: continual learning for conditional image generation. In: ICCV (2019)Google Scholar
  47. 47.
    Zhuo, L., et al.: Cogradient descent for bilinear optimization. In: CVPR (2020)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Faculty of Electronic and Information EngineeringXi’an Jiaotong UniversityXi’anChina
  2. 2.School of Software EngineeringXi’an Jiaotong UniversityXi’anChina
  3. 3.Research Center for Artificial Intelligence, Peng Cheng LaboratoryShenzhenChina

Personalised recommendations