Abstract
Online continual learning (OCL) refers to the ability of a system to learn over time from a continuous stream of data without having to revisit previously encountered training samples. Learning continually in a single data pass is crucial for agents and robots operating in changing environments and required to acquire, fine-tune, and transfer increasingly complex representations from non-i.i.d. input distributions. Machine learning models that address OCL must alleviate catastrophic forgetting in which hidden representations are disrupted or completely overwritten when learning from streams of novel input. In this chapter, we summarize and discuss recent deep learning models that address OCL on sequential input through the use (and combination) of synaptic regularization, structural plasticity, and experience replay. Different implementations of replay have been proposed that alleviate catastrophic forgetting in connectionists architectures via the re-occurrence of (latent representations of) input sequences and that functionally resemble mechanisms of hippocampal replay in the mammalian brain. Empirical evidence shows that architectures endowed with experience replay typically outperform architectures without in (online) incremental learning tasks.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Aimone, J.B., Wiles, J., Gage, F.H.: Computational influence of adult neurogenesis on memory encoding. Neuron 61, 187–202 (2009)
Anguita, D., Ghio, A., Oneto, L., Ridella, S.: Selecting the hypothesis space for improving the generalization ability of support vector machines. In: IEEE International Joint Conference on Neural Networks (2011)
Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., Baskurt, A.: Sequential deep learning for human action recognition. In: Salah, A.A., Lepri, B. (eds.) Human Behavior Understanding, pp. 29–39. Springer, Berlin (2011)
Bellemare, M.G., Naddaf, Y., Veness, J., Bowling, M.: The arcade learning environment: an evaluation platform for general agents. J. Artif. Intell. Res. 47, 253–279 (2013)
Borji, A., Izadi, S., Itti, L.: iLab-20M: a large-scale controlled object dataset to investigate deep learning. In: International Conference of Computer Vision and Pattern Recognition (CVPR), pp. 2221–2230 (2016). https://doi.org/10.1109/CVPR.2016.244
Chen, Z., Liu, B.: Lifelong machine learning. Synth. Lect. Artif. Intell. Mach. Learn. 12(3), 1–207 (2018)
Coraddu, A., Oneto, L., Baldi, F., Anguita, D.: Vessels fuel consumption forecast and trim optimisation: a data analytics perspective. Ocean Eng. 130, 351–370 (2017)
Deng, W., Aimone, J.B., Gage, F.H.: New neurons and new memories: how does adult hippocampal neurogenesis affect learning and memory? Nat. Rev. Neurosci. 11(5), 339–350 (2010)
Díaz-Rodríguez, N., Lomonaco, V., Filliat, D., Maltoni, D.: Don’t forget, there is more than forgetting: new metrics for Continual Learning. In: Workshop on Continual Learning, NeurIPS 2018 (Neural Information Processing Systems), Montreal, Canada (2018). https://hal.archives-ouvertes.fr/hal-01951488
Elfaramawy, N., Barros, P., Parisi, G.I., Wermter, S.: Emotion recognition from body expressions with a neural network architecture. In: Proceedings of the International Conference on Human Agent Interaction (HAI’17), Bielefeld, Germany, pp. 143–149 (2017)
Elman, J.L.: Learning and development in neural networks: the importance of starting small. Cognition 48(1), 71–99 (1993)
Everingham, M., Eslami, S.M.A., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The Pascal visual object classes challenge: a retrospective. Int. J. Comput. Vis. 111(1), 98–136 (2015)
Fan, L., Zhu, Y., Zhu, J., Liu, Z., Zeng, O., Gupta, A., Creus-Costa, J., Savarese, S., Fei-Fei, L.: Surreal: open-source reinforcement learning framework and robot manipulation benchmark. In: Conference on Robot Learning (2018)
Franco, A., Maio, D., Maltoni, D.: The big brother database: evaluating face recognition in smart home environments. In: Advances in Biometrics: 3rd International Conference (ICB), pp. 142–150 (2009)
French, R.M.: Catastrophic forgetting in connectionist networks. Trends Cogn. Sci. 3(4), 128–135 (1999)
Fusi, S., Drew, P.J., Abbott, L.F.: Cascade models of synaptically stored memories. Neuron 45(4), 599–611 (2005)
Gepperth, A., Hammer, B.: Incremental learning algorithms and applications. In: European Symposium on Artificial Neural Networks (ESANN), Bruges, Belgium (2016). https://hal.archives-ouvertes.fr/hal-01418129
Geusebroek, J.M., Burghouts, G.J., Smeulders, A.W.: The Amsterdam Library of object images. Int. J. Comput. Vis. 61(1), 103–112 (2005). https://doi.org/10.1023/B:VISI.0000042993.50813.60
Gorelick, L., Blank, M., Shechtman, E., Irani, M., Basri, R.: Actions as space-time shapes. In: ICCV’05, Beijing, China, pp. 1395–1402 (2005)
Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwinska, A., Colmenarejo, S.G., Grefenstette, E., Ramalho, T., Agapiou, J.E.A.: Hybrid computing using a neural network with dynamic external memory. Nature 538, 471–476 (2016)
Grossberg, S.: How does a brain build a cognitive code? Psychol. Rev. 87, 1–51 (1980)
Hayes, T.L., Cahill, N.D., Kanan, C.: Memory efficient experience replay for streaming learning. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 9769–9776 (2018)
Hayes, T.L., Kemker, R., Cahill, N.D., Kanan, C.: New metrics and experimental paradigms for continual learning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2112–21123 (2018). https://doi.org/10.1109/CVPRW.2018.00273
Holyoak, K., Thagard, P.: The analogical mind. Am. Psychol. 52, 35–44 (1997)
Ioffe, S.: Batch renormalization: towards reducing minibatch dependence in batch-normalized models. In: Advances in Neural Information Processing Systems (NIPS), pp. 1945–1953 (2017)
Ji, S., Xu, W., Yang, M., Yu, K.: 3d convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 221–231 (2013)
Jung, M., Hwang, J., Tani, J.: Self-organization of spatio-temporal hierarchy via learning of dynamic visual image patterns on action sequences. PloS One 10(7), 1–16 (2015). https://doi.org/10.1371/journal.pone.0131214
Karlsson, M., Frank, L.: Awake replay of remote experiences in the hippocampus. Nat. Neurosci. 19(10), 913–918 (2009)
Kemker, R., Kanan, C.: Fearnet: brain-inspired model for incremental learning. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=SJ1Xmf-Rb
Kemker, R., McClure, M., Abitino, A., Hayes, T.L., Kanan, C.: Measuring catastrophic forgetting in neural networks. In: AAAI (2017)
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. (2017)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Technical Report, Citeseer (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Krueger, K.A., Dayan, P.: Flexible shaping: how learning in small steps helps. Cognition 110, 380–394 (2009)
Kudrimoti, H.S., Barnes, C.A., McNaughton, B.L.: Reactivation of hippocampal cell assemblies: effects of behavioral state, experience, and EEG dynamics. J. Neurosci. 19(10), 4090–4101 (1999). https://doi.org/10.1523/JNEUROSCI.19-10-04090.1999
Lake, B.M., Salakhutdinov, R., Tenenbaum, J.B.: Human-level concept learning through probabilistic program induction. Science 350(6266), 1332–1338 (2015)
LeCun, Y., Cortes, C.: MNIST handwritten digit database. Public (2010). http://yann.lecun.com/exdb/mnist/
LeCun, Y., Huang, F.J., Bottou, L.: Learning methods for generic object recognition with invariance to pose and lighting. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, pp. 97–104 (2004). https://doi.org/10.1109/CVPR.2004.1315150, http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1315150%5Cn, http://www.cs.nyu.edu/~ylclab/data/norb-v1.0-small/
Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., Díaz-Rodríguez, N.: Continual learning for robotics: definition, framework, learning strategies, opportunities and challenges. Inf. Fusion (2019). https://doi.org/10.1016/j.inffus.2019.12.004, https://hal.archives-ouvertes.fr/hal-02381343
Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. (2017)
Lomonaco, V., Desai, K., Culurciello, E., Maltoni, D.: Continual reinforcement learning in 3d non-stationary environments (2019). arXiv:1905.10112
Lomonaco, V., Maltoni, D.: Comparing incremental learning strategies for convolutional neural networks. In: Artificial Neural Networks in Pattern Recognition: 7th IAPR TC3 Workshop (ANNPR 2016), pp. 175–184 (2016). https://doi.org/10.1007/978-3-319-46182-3_15
Lomonaco, V., Maltoni, D.: CORe50: a new dataset and benchmark for continuous object recognition. In: Levine, S., Vanhoucke, V., Goldberg, K. (eds.) Proceedings of the 1st Annual Conference on Robot Learning. Proceedings of Machine Learning Research, vol. 78, pp. 17–26. PMLR (2017). http://proceedings.mlr.press/v78/lomonaco17a.html
Lomonaco, V., Maltoni, D.: CORe50: a new dataset and benchmark for continuous object recognition (2017). arXiv:1705.03550, https://arxiv.org/pdf/1705.03550v1.pdf
Lomonaco, V., Maltoni, D., Pellegrini, L.: fine-grained continual learning. 1–14 (2019). arXiv:1907.03799
Lopez-Paz, D., Ranzato, M.A.: Gradient episodic memory for continual learning. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30, pp. 6467–6476. Curran Associates, Inc. (2017). http://papers.nips.cc/paper/7225-gradient-episodic-memory-for-continual-learning.pdf
Maltoni, D., Lomonaco, V.: Semi-supervised tuning from temporal coherence. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2509–2514 (2016). https://doi.org/10.1109/ICPR.2016.7900013, http://ieeexplore.ieee.org/document/7900013/
Maltoni, D., Lomonaco, V.: Semi-supervised tuning from temporal coherence (2016). arXiv:1511.03163
Maltoni, D., Lomonaco, V.: Continuous learning in single-incremental-task scenarios. Neural Netw. 116, 56–73 (2019). https://doi.org/10.1016/j.neunet.2019.03.010, http://arxiv.org/abs/1806.08568, https://linkinghub.elsevier.com/retrieve/pii/S0893608019300838
Mandlekar, A., Zhu, Y., Garg, A., Booher, J., Spero, M., Tung, A., Gao, J., Emmons, J., Gupta, A., Orbay, E., Savarese, S., Fei-Fei, L.: Roboturk: a crowdsourcing platform for robotic skill learning through imitation. In: Conference on Robot Learning (2018)
Mankowitz, D.J., Žídek, A., Barreto, A., Horgan, D., Hessel, M., Quan, J., Oh, J., van Hasselt, H., Silver, D., Schaul, T.: Unicorn: continual learning with a universal, off-policy agent (2018). arXiv:1802.08294
Marsland, S., Shapiro, J., Nehmzow, U.: A self-organising network that grows when required. Neural Netw. 15(8–9), 1041–1058 (2002)
McClelland, J.L., McNaughton, B.L., O’Reilly, R.C.: Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol. Rev. 102(3), 419 (1995)
Mermillod, M., Bugaiska, A., Bonin, P.: The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects. Front. Psychol. 4, 504 (2013). https://doi.org/10.3389/fpsyg.2013.00504, http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3732997&tool=pmcentrez&endertype=abstract
Mici, L., Parisi, G.I., Wermter, S.: An incremental self-organizing architecture for sensorimotor learning and prediction (2017). arXiv:1712.08521
Mici, L., Parisi, G.I., Wermter, S.: A self-organizing neural network architecture for learning human-object interactions. Neurocomputing 307, 14–24 (2018)
Ming, G.L., Song, H.: Adult neurogenesis in the mammalian brain: significant answers and significant questions. Neuron 70, 687–702 (2011)
Nene, S.A., Nayar, S.K., Murase, H.: Columbia Object Image Library (COIL-100). Technical Report (1996). http://www1.cs.columbia.edu/CAVE/publications/pdfs/Nene_TR96_2.pdf
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 (2011). http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf
Oneto, L., Ridella, S., Anguita, D.: Tikhonov, Ivanov and Morozov regularization for support vector machine learning. Mach. Learn. 103(1), 103–136 (2015)
Parisi, G., Ji, X., Wermter, S.: On the role of neurogenesis in overcoming catastrophic forgetting. In: NIPS’18, Workshop on Continual Learning, Montreal, Canada (2018)
Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: a review. Neural Netw. 113, 54–71 (2019). https://doi.org/10.1016/j.neunet.2019.01.012, http://www.sciencedirect.com/science/article/pii/S0893608019300231
Parisi, G.I., Magg, S., Wermter, S.: Human motion assessment in real time using recurrent self-organization. In: Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication, New York, NY, pp. 71–79 (2016)
Parisi, G.I., Tani, J., Weber, C., Wermter, S.: Lifelong learning of humans actions with deep neural network self-organization. Neural Netw. 96, 137–149 (2017)
Parisi, G.I., Tani, J., Weber, C., Wermter, S.: Lifelong learning of spatiotemporal representations with dual-memory recurrent self-organization. Front. Neurorobotics 12, 78 (2018). https://doi.org/10.3389/fnbot.2018.00078, https://www.frontiersin.org/article/10.3389/fnbot.2018.00078
Parisi, S., Ramstedt, S., Peters, J.: Goal-driven dimensionality reduction for reinforcement learning. In: Proceedings of the IEEE/RSJ Conference on Intelligent Robots and Systems (IROS) (2017). http://www.ausy.tu-darmstadt.de/uploads/Site/EditPublication/parisi2017iros.pdf
Pasquale, G., Ciliberto, C., Odone, F., Rosasco, L., Natale, L.: Teaching iCub to recognize objects using deep convolutional neural networks. In: Proceedings of Workshop on Machine Learning for Interactive Systems, pp. 21–25 (2015)
Pasquale, G., Ciliberto, C., Rosasco, L., Natale, L.: Object identification from few examples by improving the invariance of a deep convolutional neural network. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4904–4911 (2016). https://doi.org/10.1109/IROS.2016.7759720
Pellegrini, L., Graffieti, G., Lomonaco, V., Maltoni, D.: Latent replay for real-time continual learning (2019). arXiv:1912.01100
Rebuffi, S., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: incremental classifier and representation learning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533–5542 (2017). https://doi.org/10.1109/CVPR.2017.587
Reed, S., de Freitas, N.: Neural programmer interpreters (2015). arXiv:1511.06279
Richardson, F.M., Thomas, M.S.: Critical periods and catastrophic interference effects in the development of self-organizing feature maps. Dev. Sci. 11(3), 371–389 (2008)
Robins, A.: Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci. 7(2), 123–146 (1995). https://doi.org/10.1080/09540099550039318
Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks (2016). ArXiv e-prints
Rusu, A.A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., Hadsell, R.: Sim-to-real robot learning from pixels with progressive nets. In: CoRL’17, Mountain View, CA (2017)
Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: a local SVM approach. In: ICPR’04, Cambridge, UK, pp. 32–36 (2004)
Schwarz, M., Schulz, H., Behnke, S.: RGB-D object recognition and pose estimation based on pre-trained convolutional neural network features. In: IEEE International Conference on Robotics and Automation (ICRA’15), May, 1329–1335 (2015). https://doi.org/10.1109/ICRA.2015.7139363, http://www.ais.uni-bonn.de/papers/ICRA_2015_Schwarz_RGB-D-Objects_Transfer-Learning.pdf
She, Q., Feng, F., Hao, X., Yang, Q., Lan, C., Lomonaco, V., Shi, X., Wang, Z., Guo, Y., Zhang, Y., Qiao, F., Chan, R.H.M.: Openloris-object: a dataset and benchmark towards lifelong object recognition (2019). CoRR arXiv:abs/1911.06487
Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: Advances in Neural Information Processing Systems, pp. 2990–2999 (2017)
Singh, A., Sha, J., Narayan, K.S., Achim, T., Abbeel, P.: BigBIRD: a large-scale 3d database of object instances. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 509–516 (2014). https://doi.org/10.1109/ICRA.2014.6906903
Vahdat, M., Oneto, L., Anguita, D., Funk, M., Rauterberg, M.: A learning analytics approach to correlate the academic achievements of students with interaction data from an educational simulator. In: European Conference on Technology Enhanced Learning (2015)
Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-UCSD birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology (2010)
Wu, C., Herranz, L., Liu, X., Wang, Y., van de Weijer, J., Raducanu, B.: Memory replay GANs: learning to generate new categories without forgetting. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31, pp. 5962–5972. Curran Associates, Inc. (2018)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017). arXiv:1708.07747
Yu, F., Zhang, Y., Song, S., Seff, A., Xiao, J.: LSUN: construction of a large-scale image dataset using deep learning with humans in the loop (2015). CoRR arXiv:abs/1506.03365, http://dblp.uni-trier.de/db/journals/corr/corr1506.html#YuZSSX15
Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 70, pp. 3987–3995. PMLR, International Convention Centre, Sydney, Australia (2017). http://proceedings.mlr.press/v70/zenke17a.html
Acknowledgements
The authors would like to thank the ContinualAI organization and the other ContinualAI Research fellows for their support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Parisi, G.I., Lomonaco, V. (2020). Online Continual Learning on Sequences. In: Oneto, L., Navarin, N., Sperduti, A., Anguita, D. (eds) Recent Trends in Learning From Data. Studies in Computational Intelligence, vol 896. Springer, Cham. https://doi.org/10.1007/978-3-030-43883-8_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-43883-8_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-43882-1
Online ISBN: 978-3-030-43883-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)