Skip to main content
SpringerLink
Log in

Experimental study of rehearsal-based incremental classification of document streams

  • Original Paper
  • Published:
International Journal on Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

This research work proposes a novel protocol for rehearsal-based incremental learning models for the classification of business document streams using deep learning and, in particular, transformer-based natural language processing techniques. When implementing a rehearsal-based incremental classification model, the questions raised most often for parameterizing the model relate to the number of instances from “old” classes (learned in previous training iterations) which need to be kept in memory and the optimal number of new classes to be learned at each iteration. In this paper, we propose an incremental learning protocol that involves training incremental models using a weight-sharing strategy between transformer model layers across incremental training iterations. We provide a thorough experimental study that enables us to determine optimal ranges for various parameters in the context of incremental classification of business document streams. We also study the effect of the order in which the classes are presented to the model for learning and the effects of class imbalance on the model’s performances. Our results reveal no significant difference in the performances of our incrementally trained model and its statically trained counterpart after all training iterations (especially when, in the presence of class imbalance, the most represented classes are learned first). In addition, our proposed approach shows an improvement of 1.55% and 3.66% over a baseline model on two business documents dataset. Based on this experimental study, we provide a list of recommendations for researchers and developers for training rehearsal-based incremental classification models for business document streams. Our protocol can be further re-used for other final applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. ABBYY fine reader (https://www.abbyy.com/ocr-sdk/features/ocr/).

  2. Link to the code and appendix: http://bit.ly/3hC6ved.

  3. Link to the appendix: http://bit.ly/3hC6ved.

  4. Link to the code and appendix: http://bit.ly/3hC6ved.

  5. Link to the appendix: http://bit.ly/3hC6ved.

  6. Link to the appendix: http://bit.ly/3hC6ved.

References

  1. Asim, M.N., Khan, M.U.G., Malik, M.I., Dengel, A., Ahmed, S.: A robust hybrid approach for textual document classification. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1390–1396, IEEE (2019)

  2. Kölsch, A., Afzal, M.Z., Ebbecke, M., Liwicki, M.: Real-time document image classification using deep CNN and extreme learning machines. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 1318–1323, IEEE (2017)

  3. Gallo, I., Calefati, A., Nawaz, S., Janjua, M.K.: Image and encoded text fusion for multi-modal classification. In: 2018 Digital Image Computing: Techniques and Applications (DICTA), pp. 1–7, IEEE (2018)

  4. Rebuffi, S.-A., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: incremental classifier and representation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001–2010 (2017)

  5. Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: learning what (not) to forget. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 139–154 (2018)

  6. Cote, M., Albu, A.B.: Texture sparseness for pixel classification of business document images. Int. J. Doc. Anal. Recognit. (IJDAR) 17(3), 257–273 (2014)

    Article  Google Scholar 

  7. Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., Fu, Y.: Large scale incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 374–382 (2019)

  8. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114(13), 3521–3526 (2017)

    Article  MathSciNet  Google Scholar 

  9. D’Andecy, V.P., Joseph, A., Ogier, J.-M.: Indus: incremental document understanding system focus on document classification. In: 2018 13th IAPR International Workshop on Document Analysis Systems (DAS), pp. 239–244, IEEE (2018)

  10. Alhaj, Y.A., Xiang, J., Zhao, D., Al-Qaness, M.A., Abd Elaziz, M., Dahou, A.: A study of the effects of stemming strategies on Arabic document classification. IEEE Access 7, 32664–32671 (2019)

    Article  Google Scholar 

  11. Shahkolaei, A., Beghdadi, A., Cheriet, M.: Blind quality assessment metric and degradation classification for degraded document images. Signal Process. Image Commun. 76, 11–21 (2019)

    Article  Google Scholar 

  12. Sanchez-Pi, N., Martí, L., Garcia, A.C.B.: Improving ontology-based text classification: an occupational health and security application. J. Appl. Log. 17, 48–58 (2016)

    Article  MathSciNet  Google Scholar 

  13. Gayathri, M., Kannan, R.J.: Ontology based concept extraction and classification of ayurvedic documents. Procedia Comput. Sci. 172, 511–516 (2020)

    Article  Google Scholar 

  14. Walczak, S., Kellogg, D.L.: A heuristic text analytic approach for classifying research articles. Intell. Inf. Manag. 7(1), 7 (2015)

    Google Scholar 

  15. Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A.D., van de Weijer, J.: Class-incremental learning: survey and performance evaluation. arXiv preprint arXiv:2010.15277 (2020)

  16. Luo, Y., Yin, L., Bai, W., Mao, K.: An appraisal of incremental learning methods. Entropy 22(11), 1190 (2020)

    Article  MathSciNet  Google Scholar 

  17. Van de Ven, G.M., Tolias, A.S.: Three scenarios for continual learning. arXiv preprint arXiv:1904.07734 (2019)

  18. Chaudhry, A., Dokania, P.K., Ajanthan, T., Torr, P.H.: Riemannian walk for incremental learning: Understanding forgetting and intransigence. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 532–547 (2018)

  19. Mermillod, M., Bugaiska, A., Bonin, P.: The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects. Front. Psychol. 4, 504 (2013)

    Article  Google Scholar 

  20. Zeng, G., Chen, Y., Cui, B., Yu, S.: Continual learning of context-dependent processing in neural networks. Nat. Mach. Intell. 1(8), 364–372 (2019)

    Article  Google Scholar 

  21. Farajtabar, M., Azizan, N., Mott, A., Li, A.: Orthogonal gradient descent for continual learning. In: International Conference on Artificial Intelligence and Statistics, pp. 3762–3773, PMLR (2020)

  22. Castro, F.M., Marín-Jiménez, M.J., Guil, N., Schmid, C., Alahari, K.: End-to-end incremental learning. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 233–248 (2018)

  23. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

  24. Zhang, J., Zhang, J., Ghosh, S., Li, D., Tasci, S., Heck, L., Zhang, H., Kuo, C.-C.J.: Class-incremental learning via deep model consolidation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1131–1140 (2020)

  25. Lee, K., Lee, K., Shin, J., Lee, H.: Overcoming catastrophic forgetting with unlabeled data in the wild. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 312–321 (2019)

  26. Roy, D., Panda, P., Roy, K.: Tree-CNN: a hierarchical deep convolutional neural network for incremental learning. Neural Netw. 121, 148–160 (2020)

    Article  Google Scholar 

  27. Mandivarapu, J.K., Camp, B., Estrada, R.: Self-net: lifelong learning via continual self-modeling. Front. Artif. Intell. 3, 19 (2020)

    Article  Google Scholar 

  28. Polikar, R., Upda, L., Upda, S.S., Honavar, V.: Learn++: an incremental learning algorithm for supervised neural networks. IEEE Trans. Syst. Man, Cybern. Part C (Appl. Rev.) 31(4), 497–508 (2001)

    Article  Google Scholar 

  29. Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016)

  30. Yoon, J., Yang, E., Lee, J., Hwang, S.J.: Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547 (2017)

  31. Nguyen, C.V., Li, Y., Bui, T.D., Turner, R.E.: Variational continual learning. arXiv preprint arXiv:1710.10628 (2017)

  32. Farquhar, S., Gal, Y.: A unifying Bayesian view of continual learning. arXiv preprint arXiv:1902.06494 (2019)

  33. Chen, Y., Diethe, T., Lawrence, N.: Facilitating bayesian continual learning by natural gradients and stein gradients. In: Continual Learning Workshop of 32nd Conference on Neural Information Processing Systems (NeurIPS 2018) (2019)

  34. Adel, T., Zhao, H., Turner, R.E.: Continual learning with adaptive weights (claw). In: International Conference on Learning Representations (2019)

  35. Ebrahimi, S., Elhoseiny, M., Darrell, T., Rohrbach, M.: Uncertainty-guided continual learning with Bayesian neural networks. arXiv preprint arXiv:1906.02425 (2019)

  36. Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., Zhang, Z., Fu, Y.: Incremental classifier learning with generative adversarial networks. arXiv preprint arXiv:1802.00853 (2018)

  37. Masarczyk, W., Tautkute, I.: Reducing catastrophic forgetting with learning on synthetic data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 252–253 (2020)

  38. Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. arXiv preprint arXiv:1705.08690 (2017)

  39. Kemker, R., Kanan, C.: Fearnet: brain-inspired model for incremental learning. arXiv preprint arXiv:1711.10563 (2017)

  40. Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831–839 (2019)

  41. Arif-Uz-Zaman, K., Cholette, M.E., Ma, L., Karim, A.: Extracting failure time data from industrial maintenance records using text mining. Adv. Eng. Inform. 33, 388–396 (2017)

    Article  Google Scholar 

  42. Mahamoud, I.S., Voerman, J., Coustaty, M., Joseph, A., d’Andecy, V.P., Ogier, J.-M.: Multimodal attention-based learning for imbalanced corporate documents classification. In: International Conference on Document Analysis and Recognition, pp. 223–237, Springer (2021)

  43. Choe, J.-W., Lee, D.-G.: Trends 21 corpus: public web resources and search tools. Stud. Korean Cult. 64, 1–20 (2014)

    Google Scholar 

  44. Hitt, M.A., Ireland, R.D., Hoskisson, R.E.: Strategic management: concepts and cases: competitiveness and globalization. Cengage Learn. (2016)

  45. Ahn, H., Cha, S., Lee, D., Moon, T.: Uncertainty-based continual learning with adaptive regularization. arXiv preprint arXiv:1905.11614 (2019)

  46. Shan, G., Xu, S., Yang, L., Jia, S., Xiang, Y.: Learn#: a novel incremental learning method for text classification. Expert Syst. Appl. 147, 113198 (2020)

    Article  Google Scholar 

  47. Madhusudhanan, S., Jaganathan, S., LS, J.: Incremental learning for classification of unstructured data using extreme learning machine. Algorithms 11(10), 158 (2018)

  48. Jang, J., Kim, Y., Choi, K., Suh, S.: Sequential targeting: an incremental learning approach for data imbalance in text classification. arXiv preprint arXiv:2011.10216, (2020)

  49. Xia, C., Yin, W., Feng, Y., Yu, P.: Incremental few-shot text classification with multi-round new classes: formulation, dataset and system. arXiv preprint arXiv:2104.11882 (2021)

  50. Voerman, J., Mahamoud, I.S., Joseph, A., Coustaty, M., D’Andecy, V.P., Ogier, J.-M.: Toward an incremental classification process of document stream using a cascade of systems. In: Document Analysis and Recognition–ICDAR 2021 Workshops: Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part II 16, pp. 240–254, Springer (2021)

  51. Ravuri, S., Vinyals, O.: Classification accuracy score for conditional generative models. arXiv preprint arXiv:1905.10887 (2019)

  52. Harley, A.W., Ufkes, A., Derpanis, K.G.: Evaluation of deep convolutional nets for document image classification and retrieval. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 991–995, IEEE (2015)

  53. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)

  54. Kenton, J.D.M.W.C., Toutanova, L.K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp. 4171–4186 (2019)

  55. Le, H., Vial, L., Frej, J., Segonne, V., Coavoux, M., Lecouteux, B., Allauzen, A., Crabbé, B., Besacier, L., Schwab, D.: Flaubert: unsupervised language model pre-training for french. arXiv preprint arXiv:1912.05372 (2019)

  56. Emerson, R.W.: Anova and t-tests. J. Vis. Impair. Blind. 111(2), 193–196 (2017)

    Article  Google Scholar 

Download references

Funding

This work is supported by the Region Nouvelle Aquitaine under the grant number 2019-1R50120 (CRASD project) and AAPR2020-2019-8496610 (CRASD2 project) and by the LabCom IDEAS under the Grant Number ANR-18-LCV3-0008.

Author information

Authors and Affiliations

  1. La Rochelle Université, Laboratoire Informatique, Image et Interaction (L3i), 17042, La Rochelle, France

    Usman Malik, Muriel Visani, Nicolas Sidere & Mickael Coustaty

  2. French Military Center for Epidemiology and Public Health (CESPA), 13014, Marseille, France

    Muriel Visani

  3. Yooz, 1 Rue Fleming, 17000, La Rochelle, France

    Aurelie Joseph

Authors
  1. Usman Malik
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Muriel Visani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicolas Sidere
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mickael Coustaty
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aurelie Joseph
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Usman Malik or Muriel Visani.

Ethics declarations

Conflict of interest

The authors have no financial or Conflict of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Malik, U., Visani, M., Sidere, N. et al. Experimental study of rehearsal-based incremental classification of document streams. IJDAR (2024). https://doi.org/10.1007/s10032-024-00467-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10032-024-00467-w

Keywords

Search

Navigation