Comparing Incremental Learning Strategies for Convolutional Neural Networks

  • Vincenzo LomonacoEmail author
  • Davide Maltoni
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9896)


In the last decade, Convolutional Neural Networks (CNNs) have shown to perform incredibly well in many computer vision tasks such as object recognition and object detection, being able to extract meaningful high-level invariant features. However, partly because of their complex training and tricky hyper-parameters tuning, CNNs have been scarcely studied in the context of incremental learning where data are available in consecutive batches and retraining the model from scratch is unfeasible. In this work we compare different incremental learning strategies for CNN based architectures, targeting real-word applications.


Deep learning Incremental learning Convolutional neural networks 


  1. 1.
    Hinton, G., Deng, L., Yu, D., Dahl, G.E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T.N., Kingsbury, B.: Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Process. Mag. 82–97 (2012)Google Scholar
  2. 2.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. Adv. Neural Inf. Process. Syst. (NIPS) 1–9 (2013)Google Scholar
  3. 3.
    Krizhevsky, A., Sulskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. (NIPS) 1–9 (2012)Google Scholar
  4. 4.
    Bengio, Y., Courville, A., Vincent, P.: Unsupervised feature learning and deep learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828Google Scholar
  5. 5.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL visual object classes (voc) challenge. Int J Comput Vis. 88, 303–338 (2010)CrossRefGoogle Scholar
  7. 7.
    Mermillod, M., Bugaiska, A., Bonin, P.: The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects. Front. Psychol. 4, 504 (2013)CrossRefGoogle Scholar
  8. 8.
    Mccloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. Psychol. Learn. Motiv. 109–165 (1989)Google Scholar
  9. 9.
    French, R.M.: Catastrophic forgetting in connectionist networks. Encycl. Cogn. Sci. Nadel/Cogn. (2006)Google Scholar
  10. 10.
    Goodfellow, I.J., Mirza, M., Courville, A., Bengio, Y.: An empirical investigation of catastrophic forgetting in gradient-based neural networks (2012). arXiv preprint arXiv:1312.6211v3 (2015)
  11. 11.
    LeCun, Y., Huang, F.J., Bottou, L.: Learning methods for generic object recognition with invariance to pose and lighting. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision Pattern Recognition, 2004, CVPR 2004, vol. 2, pp. 97–104 (2004)Google Scholar
  12. 12.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the ACM International Conference Multimedia, pp. 675–678 (2014)Google Scholar
  13. 13.
    Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.: DeCAF: a deep convolutional activation feature for generic visual recognition. In: International Conference on Machine Learning (ICML), vol. 32, pp. 647–655 (2014)Google Scholar
  14. 14.
    Razavian, A.S., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition. In: IEEE Computer Society Conference Computer Vision Pattern Recognition Workshops, pp. 512–519 (2014)Google Scholar
  15. 15.
    Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: delving deep into convolutional nets. In: Proceedings of the British Machine Vision Conference pp. 1–11 (2014)Google Scholar
  16. 16.
    Tsai, C.-H., Lin, C.-Y., Lin, C.-J.: Incremental and decremental training for linear classification. In: 20th ACM SIGKDD International Conference Knowledge Discovery Data Mining, KDD 2014, pp. 343–352 (2014)Google Scholar
  17. 17.
    Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: Proceedings of the British Machine Vision Conference 2015, pp. 41.1–41.12 (2015)Google Scholar
  18. 18.
    Pasquale, G., Ciliberto, C., Odone, F., Rosasco, L., Natale, L.: Real-world object recognition with off-the-shelf deep conv nets: how many objects can iCub learn? arXiv:1504.03154 [cs] (2015)
  19. 19.
    Franco, A., Maio, D., Maltoni, D.: The big brother database: evaluating face recognition in smart home environments. In: Tistarelli, M., Nixon, M.S. (eds.) ICB 2009. LNCS, vol. 5558, pp. 142–150. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  20. 20.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Computer Vision Pattern Recognition, vol. 1, pp. I–511–I–518 (2001)Google Scholar
  21. 21.
    Franco, A., Maio, D., Maltoni, D.: Incremental template updating for face recognition in home environments. Pattern Recognit. 43, 2891–2903 (2010)CrossRefzbMATHGoogle Scholar
  22. 22.
    Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: Advances Neural Information Processing Systems (NIPS), vol. 27, pp. 1–9 (2014)Google Scholar
  23. 23.
    Maltoni, D., Lomonaco, V.: Semi-supervised tuning from temporal coherence. Technical report, DISI – University of Bologna, pp. 1–14 (2015).

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.DISI - University of BolognaBolognaItaly

Personalised recommendations