Skip to main content

Online Continual Learning on Sequences

  • Chapter
  • First Online:
Recent Trends in Learning From Data

Part of the book series: Studies in Computational Intelligence ((SCI,volume 896))

Abstract

Online continual learning (OCL) refers to the ability of a system to learn over time from a continuous stream of data without having to revisit previously encountered training samples. Learning continually in a single data pass is crucial for agents and robots operating in changing environments and required to acquire, fine-tune, and transfer increasingly complex representations from non-i.i.d. input distributions. Machine learning models that address OCL must alleviate catastrophic forgetting in which hidden representations are disrupted or completely overwritten when learning from streams of novel input. In this chapter, we summarize and discuss recent deep learning models that address OCL on sequential input through the use (and combination) of synaptic regularization, structural plasticity, and experience replay. Different implementations of replay have been proposed that alleviate catastrophic forgetting in connectionists architectures via the re-occurrence of (latent representations of) input sequences and that functionally resemble mechanisms of hippocampal replay in the mammalian brain. Empirical evidence shows that architectures endowed with experience replay typically outperform architectures without in (online) incremental learning tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    www.vlomonaco.github.io/core50.

References

  1. Aimone, J.B., Wiles, J., Gage, F.H.: Computational influence of adult neurogenesis on memory encoding. Neuron 61, 187–202 (2009)

    Article  Google Scholar 

  2. Anguita, D., Ghio, A., Oneto, L., Ridella, S.: Selecting the hypothesis space for improving the generalization ability of support vector machines. In: IEEE International Joint Conference on Neural Networks (2011)

    Google Scholar 

  3. Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., Baskurt, A.: Sequential deep learning for human action recognition. In: Salah, A.A., Lepri, B. (eds.) Human Behavior Understanding, pp. 29–39. Springer, Berlin (2011)

    Chapter  Google Scholar 

  4. Bellemare, M.G., Naddaf, Y., Veness, J., Bowling, M.: The arcade learning environment: an evaluation platform for general agents. J. Artif. Intell. Res. 47, 253–279 (2013)

    Article  Google Scholar 

  5. Borji, A., Izadi, S., Itti, L.: iLab-20M: a large-scale controlled object dataset to investigate deep learning. In: International Conference of Computer Vision and Pattern Recognition (CVPR), pp. 2221–2230 (2016). https://doi.org/10.1109/CVPR.2016.244

  6. Chen, Z., Liu, B.: Lifelong machine learning. Synth. Lect. Artif. Intell. Mach. Learn. 12(3), 1–207 (2018)

    Article  Google Scholar 

  7. Coraddu, A., Oneto, L., Baldi, F., Anguita, D.: Vessels fuel consumption forecast and trim optimisation: a data analytics perspective. Ocean Eng. 130, 351–370 (2017)

    Google Scholar 

  8. Deng, W., Aimone, J.B., Gage, F.H.: New neurons and new memories: how does adult hippocampal neurogenesis affect learning and memory? Nat. Rev. Neurosci. 11(5), 339–350 (2010)

    Article  Google Scholar 

  9. Díaz-Rodríguez, N., Lomonaco, V., Filliat, D., Maltoni, D.: Don’t forget, there is more than forgetting: new metrics for Continual Learning. In: Workshop on Continual Learning, NeurIPS 2018 (Neural Information Processing Systems), Montreal, Canada (2018). https://hal.archives-ouvertes.fr/hal-01951488

  10. Elfaramawy, N., Barros, P., Parisi, G.I., Wermter, S.: Emotion recognition from body expressions with a neural network architecture. In: Proceedings of the International Conference on Human Agent Interaction (HAI’17), Bielefeld, Germany, pp. 143–149 (2017)

    Google Scholar 

  11. Elman, J.L.: Learning and development in neural networks: the importance of starting small. Cognition 48(1), 71–99 (1993)

    Article  Google Scholar 

  12. Everingham, M., Eslami, S.M.A., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The Pascal visual object classes challenge: a retrospective. Int. J. Comput. Vis. 111(1), 98–136 (2015)

    Article  Google Scholar 

  13. Fan, L., Zhu, Y., Zhu, J., Liu, Z., Zeng, O., Gupta, A., Creus-Costa, J., Savarese, S., Fei-Fei, L.: Surreal: open-source reinforcement learning framework and robot manipulation benchmark. In: Conference on Robot Learning (2018)

    Google Scholar 

  14. Franco, A., Maio, D., Maltoni, D.: The big brother database: evaluating face recognition in smart home environments. In: Advances in Biometrics: 3rd International Conference (ICB), pp. 142–150 (2009)

    Google Scholar 

  15. French, R.M.: Catastrophic forgetting in connectionist networks. Trends Cogn. Sci. 3(4), 128–135 (1999)

    Article  Google Scholar 

  16. Fusi, S., Drew, P.J., Abbott, L.F.: Cascade models of synaptically stored memories. Neuron 45(4), 599–611 (2005)

    Article  Google Scholar 

  17. Gepperth, A., Hammer, B.: Incremental learning algorithms and applications. In: European Symposium on Artificial Neural Networks (ESANN), Bruges, Belgium (2016). https://hal.archives-ouvertes.fr/hal-01418129

  18. Geusebroek, J.M., Burghouts, G.J., Smeulders, A.W.: The Amsterdam Library of object images. Int. J. Comput. Vis. 61(1), 103–112 (2005). https://doi.org/10.1023/B:VISI.0000042993.50813.60

  19. Gorelick, L., Blank, M., Shechtman, E., Irani, M., Basri, R.: Actions as space-time shapes. In: ICCV’05, Beijing, China, pp. 1395–1402 (2005)

    Google Scholar 

  20. Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwinska, A., Colmenarejo, S.G., Grefenstette, E., Ramalho, T., Agapiou, J.E.A.: Hybrid computing using a neural network with dynamic external memory. Nature 538, 471–476 (2016)

    Google Scholar 

  21. Grossberg, S.: How does a brain build a cognitive code? Psychol. Rev. 87, 1–51 (1980)

    Article  Google Scholar 

  22. Hayes, T.L., Cahill, N.D., Kanan, C.: Memory efficient experience replay for streaming learning. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 9769–9776 (2018)

    Google Scholar 

  23. Hayes, T.L., Kemker, R., Cahill, N.D., Kanan, C.: New metrics and experimental paradigms for continual learning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2112–21123 (2018). https://doi.org/10.1109/CVPRW.2018.00273

  24. Holyoak, K., Thagard, P.: The analogical mind. Am. Psychol. 52, 35–44 (1997)

    Article  Google Scholar 

  25. Ioffe, S.: Batch renormalization: towards reducing minibatch dependence in batch-normalized models. In: Advances in Neural Information Processing Systems (NIPS), pp. 1945–1953 (2017)

    Google Scholar 

  26. Ji, S., Xu, W., Yang, M., Yu, K.: 3d convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 221–231 (2013)

    Article  Google Scholar 

  27. Jung, M., Hwang, J., Tani, J.: Self-organization of spatio-temporal hierarchy via learning of dynamic visual image patterns on action sequences. PloS One 10(7), 1–16 (2015). https://doi.org/10.1371/journal.pone.0131214

    Article  Google Scholar 

  28. Karlsson, M., Frank, L.: Awake replay of remote experiences in the hippocampus. Nat. Neurosci. 19(10), 913–918 (2009)

    Article  Google Scholar 

  29. Kemker, R., Kanan, C.: Fearnet: brain-inspired model for incremental learning. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=SJ1Xmf-Rb

  30. Kemker, R., McClure, M., Abitino, A., Hayes, T.L., Kanan, C.: Measuring catastrophic forgetting in neural networks. In: AAAI (2017)

    Google Scholar 

  31. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. (2017)

    Google Scholar 

  32. Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Technical Report, Citeseer (2009)

    Google Scholar 

  33. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  34. Krueger, K.A., Dayan, P.: Flexible shaping: how learning in small steps helps. Cognition 110, 380–394 (2009)

    Article  Google Scholar 

  35. Kudrimoti, H.S., Barnes, C.A., McNaughton, B.L.: Reactivation of hippocampal cell assemblies: effects of behavioral state, experience, and EEG dynamics. J. Neurosci. 19(10), 4090–4101 (1999). https://doi.org/10.1523/JNEUROSCI.19-10-04090.1999

    Article  Google Scholar 

  36. Lake, B.M., Salakhutdinov, R., Tenenbaum, J.B.: Human-level concept learning through probabilistic program induction. Science 350(6266), 1332–1338 (2015)

    Article  MathSciNet  Google Scholar 

  37. LeCun, Y., Cortes, C.: MNIST handwritten digit database. Public (2010). http://yann.lecun.com/exdb/mnist/

  38. LeCun, Y., Huang, F.J., Bottou, L.: Learning methods for generic object recognition with invariance to pose and lighting. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, pp. 97–104 (2004). https://doi.org/10.1109/CVPR.2004.1315150, http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1315150%5Cn, http://www.cs.nyu.edu/~ylclab/data/norb-v1.0-small/

  39. Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., Díaz-Rodríguez, N.: Continual learning for robotics: definition, framework, learning strategies, opportunities and challenges. Inf. Fusion (2019). https://doi.org/10.1016/j.inffus.2019.12.004, https://hal.archives-ouvertes.fr/hal-02381343

  40. Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. (2017)

    Google Scholar 

  41. Lomonaco, V., Desai, K., Culurciello, E., Maltoni, D.: Continual reinforcement learning in 3d non-stationary environments (2019). arXiv:1905.10112

  42. Lomonaco, V., Maltoni, D.: Comparing incremental learning strategies for convolutional neural networks. In: Artificial Neural Networks in Pattern Recognition: 7th IAPR TC3 Workshop (ANNPR 2016), pp. 175–184 (2016). https://doi.org/10.1007/978-3-319-46182-3_15

  43. Lomonaco, V., Maltoni, D.: CORe50: a new dataset and benchmark for continuous object recognition. In: Levine, S., Vanhoucke, V., Goldberg, K. (eds.) Proceedings of the 1st Annual Conference on Robot Learning. Proceedings of Machine Learning Research, vol. 78, pp. 17–26. PMLR (2017). http://proceedings.mlr.press/v78/lomonaco17a.html

  44. Lomonaco, V., Maltoni, D.: CORe50: a new dataset and benchmark for continuous object recognition (2017). arXiv:1705.03550, https://arxiv.org/pdf/1705.03550v1.pdf

  45. Lomonaco, V., Maltoni, D., Pellegrini, L.: fine-grained continual learning. 1–14 (2019). arXiv:1907.03799

  46. Lopez-Paz, D., Ranzato, M.A.: Gradient episodic memory for continual learning. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30, pp. 6467–6476. Curran Associates, Inc. (2017). http://papers.nips.cc/paper/7225-gradient-episodic-memory-for-continual-learning.pdf

  47. Maltoni, D., Lomonaco, V.: Semi-supervised tuning from temporal coherence. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2509–2514 (2016). https://doi.org/10.1109/ICPR.2016.7900013, http://ieeexplore.ieee.org/document/7900013/

  48. Maltoni, D., Lomonaco, V.: Semi-supervised tuning from temporal coherence (2016). arXiv:1511.03163

  49. Maltoni, D., Lomonaco, V.: Continuous learning in single-incremental-task scenarios. Neural Netw. 116, 56–73 (2019). https://doi.org/10.1016/j.neunet.2019.03.010, http://arxiv.org/abs/1806.08568, https://linkinghub.elsevier.com/retrieve/pii/S0893608019300838

  50. Mandlekar, A., Zhu, Y., Garg, A., Booher, J., Spero, M., Tung, A., Gao, J., Emmons, J., Gupta, A., Orbay, E., Savarese, S., Fei-Fei, L.: Roboturk: a crowdsourcing platform for robotic skill learning through imitation. In: Conference on Robot Learning (2018)

    Google Scholar 

  51. Mankowitz, D.J., Žídek, A., Barreto, A., Horgan, D., Hessel, M., Quan, J., Oh, J., van Hasselt, H., Silver, D., Schaul, T.: Unicorn: continual learning with a universal, off-policy agent (2018). arXiv:1802.08294

  52. Marsland, S., Shapiro, J., Nehmzow, U.: A self-organising network that grows when required. Neural Netw. 15(8–9), 1041–1058 (2002)

    Article  Google Scholar 

  53. McClelland, J.L., McNaughton, B.L., O’Reilly, R.C.: Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol. Rev. 102(3), 419 (1995)

    Google Scholar 

  54. Mermillod, M., Bugaiska, A., Bonin, P.: The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects. Front. Psychol. 4, 504 (2013). https://doi.org/10.3389/fpsyg.2013.00504, http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3732997&tool=pmcentrez&endertype=abstract

  55. Mici, L., Parisi, G.I., Wermter, S.: An incremental self-organizing architecture for sensorimotor learning and prediction (2017). arXiv:1712.08521

  56. Mici, L., Parisi, G.I., Wermter, S.: A self-organizing neural network architecture for learning human-object interactions. Neurocomputing 307, 14–24 (2018)

    Article  Google Scholar 

  57. Ming, G.L., Song, H.: Adult neurogenesis in the mammalian brain: significant answers and significant questions. Neuron 70, 687–702 (2011)

    Article  Google Scholar 

  58. Nene, S.A., Nayar, S.K., Murase, H.: Columbia Object Image Library (COIL-100). Technical Report (1996). http://www1.cs.columbia.edu/CAVE/publications/pdfs/Nene_TR96_2.pdf

  59. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 (2011). http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf

  60. Oneto, L., Ridella, S., Anguita, D.: Tikhonov, Ivanov and Morozov regularization for support vector machine learning. Mach. Learn. 103(1), 103–136 (2015)

    Article  MathSciNet  Google Scholar 

  61. Parisi, G., Ji, X., Wermter, S.: On the role of neurogenesis in overcoming catastrophic forgetting. In: NIPS’18, Workshop on Continual Learning, Montreal, Canada (2018)

    Google Scholar 

  62. Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: a review. Neural Netw. 113, 54–71 (2019). https://doi.org/10.1016/j.neunet.2019.01.012, http://www.sciencedirect.com/science/article/pii/S0893608019300231

  63. Parisi, G.I., Magg, S., Wermter, S.: Human motion assessment in real time using recurrent self-organization. In: Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication, New York, NY, pp. 71–79 (2016)

    Google Scholar 

  64. Parisi, G.I., Tani, J., Weber, C., Wermter, S.: Lifelong learning of humans actions with deep neural network self-organization. Neural Netw. 96, 137–149 (2017)

    Article  Google Scholar 

  65. Parisi, G.I., Tani, J., Weber, C., Wermter, S.: Lifelong learning of spatiotemporal representations with dual-memory recurrent self-organization. Front. Neurorobotics 12, 78 (2018). https://doi.org/10.3389/fnbot.2018.00078, https://www.frontiersin.org/article/10.3389/fnbot.2018.00078

  66. Parisi, S., Ramstedt, S., Peters, J.: Goal-driven dimensionality reduction for reinforcement learning. In: Proceedings of the IEEE/RSJ Conference on Intelligent Robots and Systems (IROS) (2017). http://www.ausy.tu-darmstadt.de/uploads/Site/EditPublication/parisi2017iros.pdf

  67. Pasquale, G., Ciliberto, C., Odone, F., Rosasco, L., Natale, L.: Teaching iCub to recognize objects using deep convolutional neural networks. In: Proceedings of Workshop on Machine Learning for Interactive Systems, pp. 21–25 (2015)

    Google Scholar 

  68. Pasquale, G., Ciliberto, C., Rosasco, L., Natale, L.: Object identification from few examples by improving the invariance of a deep convolutional neural network. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4904–4911 (2016). https://doi.org/10.1109/IROS.2016.7759720

  69. Pellegrini, L., Graffieti, G., Lomonaco, V., Maltoni, D.: Latent replay for real-time continual learning (2019). arXiv:1912.01100

  70. Rebuffi, S., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: incremental classifier and representation learning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533–5542 (2017). https://doi.org/10.1109/CVPR.2017.587

  71. Reed, S., de Freitas, N.: Neural programmer interpreters (2015). arXiv:1511.06279

  72. Richardson, F.M., Thomas, M.S.: Critical periods and catastrophic interference effects in the development of self-organizing feature maps. Dev. Sci. 11(3), 371–389 (2008)

    Article  Google Scholar 

  73. Robins, A.: Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci. 7(2), 123–146 (1995). https://doi.org/10.1080/09540099550039318

  74. Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks (2016). ArXiv e-prints

    Google Scholar 

  75. Rusu, A.A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., Hadsell, R.: Sim-to-real robot learning from pixels with progressive nets. In: CoRL’17, Mountain View, CA (2017)

    Google Scholar 

  76. Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: a local SVM approach. In: ICPR’04, Cambridge, UK, pp. 32–36 (2004)

    Google Scholar 

  77. Schwarz, M., Schulz, H., Behnke, S.: RGB-D object recognition and pose estimation based on pre-trained convolutional neural network features. In: IEEE International Conference on Robotics and Automation (ICRA’15), May, 1329–1335 (2015). https://doi.org/10.1109/ICRA.2015.7139363, http://www.ais.uni-bonn.de/papers/ICRA_2015_Schwarz_RGB-D-Objects_Transfer-Learning.pdf

  78. She, Q., Feng, F., Hao, X., Yang, Q., Lan, C., Lomonaco, V., Shi, X., Wang, Z., Guo, Y., Zhang, Y., Qiao, F., Chan, R.H.M.: Openloris-object: a dataset and benchmark towards lifelong object recognition (2019). CoRR arXiv:abs/1911.06487

  79. Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: Advances in Neural Information Processing Systems, pp. 2990–2999 (2017)

    Google Scholar 

  80. Singh, A., Sha, J., Narayan, K.S., Achim, T., Abbeel, P.: BigBIRD: a large-scale 3d database of object instances. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 509–516 (2014). https://doi.org/10.1109/ICRA.2014.6906903

  81. Vahdat, M., Oneto, L., Anguita, D., Funk, M., Rauterberg, M.: A learning analytics approach to correlate the academic achievements of students with interaction data from an educational simulator. In: European Conference on Technology Enhanced Learning (2015)

    Google Scholar 

  82. Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-UCSD birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology (2010)

    Google Scholar 

  83. Wu, C., Herranz, L., Liu, X., Wang, Y., van de Weijer, J., Raducanu, B.: Memory replay GANs: learning to generate new categories without forgetting. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31, pp. 5962–5972. Curran Associates, Inc. (2018)

    Google Scholar 

  84. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017). arXiv:1708.07747

  85. Yu, F., Zhang, Y., Song, S., Seff, A., Xiao, J.: LSUN: construction of a large-scale image dataset using deep learning with humans in the loop (2015). CoRR arXiv:abs/1506.03365, http://dblp.uni-trier.de/db/journals/corr/corr1506.html#YuZSSX15

  86. Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 70, pp. 3987–3995. PMLR, International Convention Centre, Sydney, Australia (2017). http://proceedings.mlr.press/v70/zenke17a.html

Download references

Acknowledgements

The authors would like to thank the ContinualAI organization and the other ContinualAI Research fellows for their support.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to German I. Parisi .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Parisi, G.I., Lomonaco, V. (2020). Online Continual Learning on Sequences. In: Oneto, L., Navarin, N., Sperduti, A., Anguita, D. (eds) Recent Trends in Learning From Data. Studies in Computational Intelligence, vol 896. Springer, Cham. https://doi.org/10.1007/978-3-030-43883-8_8

Download citation

Publish with us

Policies and ethics