Evolving Systems

, Volume 1, Issue 1, pp 17–27 | Cite as

Incremental linear discriminant analysis for evolving feature spaces in multitask pattern recognition problems

  • Masayuki Hisada
  • Seiichi OzawaEmail author
  • Kau Zhang
  • Nikola Kasabov
Original Paper


In this paper, we propose a new incremental linear discriminant analysis (ILDA) for multitask pattern recognition (MTPR) problems in which a chunk of training data for a particular task are given sequentially and the task is switched to another related task one after another. The Pang et al.’s ILDA is extended such that a discriminant space of the current task is augmented with effective discriminant vectors that are selected from other tasks based on the class separability. We call this selective augmentation of discriminant vectors knowledge transfer of feature space. In the experiments, the proposed ILDA is evaluated for seven MTPR problems, each of which consists of three recognition tasks. The results demonstrate that the proposed ILDA with knowledge transfer outperforms the conventional ILDA and its naive extension to MTPR problems with regard to both class separability and recognition accuracy. We confirm that the proposed knowledge transfer works well to evolve effective feature spaces online in MTPR problems.


Linear discriminant analysis Multitask learning Incremental learning Feature extraction 



The authors would like to thank Professor Shigeo Abe for his helpful comments and discussions. This research was partially supported by the Ministry of Education, Science, Sports and Culture, Grant-in-Aid for Scientific Research (C) 205002205.


  1. Abu-Mostafa YS (1989) Learning from hints in neural networks. J Complex 6:192–198CrossRefMathSciNetGoogle Scholar
  2. Argyriou A, Evgeniou T, Pontil M (2007) Multi-task feature learning. In: Schölkopf B et al (eds) Advances in neural information processing systems 19, MIT Press, pp 41–48Google Scholar
  3. Blake C, Keogh E, Merz C (1998) UCI repository of machine learning databases.
  4. Caruana R (1995) Learning many related tasks at the same time with backpropagation. In: Tesauro G et al (eds) Advances in neural information processing systems 7, MIT Press, pp 657–664Google Scholar
  5. Caruana R (1997) Multitask learning. Mach Learn 28:41–75CrossRefGoogle Scholar
  6. Carpenter GA, Grossberg S (1998) The ART of adaptive pattern recognition by a self-organizing neural network. IEEE Comput 21:77–88Google Scholar
  7. Cauwenberghs G, Poggio T (2000) Incremental and decremental support vector machine learning. In: Leen T et al (eds) Advances in neural information processing systems 13, MIT PressGoogle Scholar
  8. Kasabov N (2001) Evolving fuzzy neural networks for supervised/unsupervised on-line knowledge-based learning. IEEE Trans Syst Man Cybern B 31:902–918CrossRefGoogle Scholar
  9. Kasabov N (2002) Evolving connectionist systems: methods and applications in bioinformatics, brain study and intelligent machines. Springer, LondonGoogle Scholar
  10. Okada T and Tomita S (1985) An optimal orthonormal system for discriminant analysis. Pattern Recognit 18:139–144CrossRefGoogle Scholar
  11. Ozawa S, Toh SL, Abe S, Pang S, Kasabov N (2005) Incremental learning of feature space and classifier for face recognition. Neural Netw 18:575–584CrossRefGoogle Scholar
  12. Ozawa S, Pang S, Kasabov N (2008) Incremental learning of chunk data for on-line pattern classification systems. IEEE Trans on Neural Netw 19:430–445CrossRefGoogle Scholar
  13. Ozawa S, Roy A, Roussinov D (2009) A multitask learning model for online pattern recognition. IEEE Trans Neural Netw 20:1061–1074CrossRefGoogle Scholar
  14. Pang S, Ozawa S, Kasabov N (2005) Incremental linear discriminant analysis for classification of data streams. IEEE Trans Syst Man Cybern B 35:905–914CrossRefGoogle Scholar
  15. Platt J (1991) A resource allocating network for function interpolation. Neural Comput 3:213–225CrossRefMathSciNetGoogle Scholar
  16. Roy A (2003) Neural networks: how do we make a widely used technology out of it? IEEE Neural Netw Soc Newsl 1:8–12Google Scholar
  17. Silver D and Mercer R (2001) Selective functional transfer: inductive bias from related tasks. In: Proc IASTED Int Conf on Artif Intell and Soft Comput, pp 182–189Google Scholar
  18. Takeuchi Y, Ozawa S, Abe S (2007) An efficient incremental kernel principal component analysis for online feature selection. In: Proc Int Joint Conf on Neural Netwo 2007, pp 1603–1608Google Scholar
  19. Thrun S, Pratt L (1998) Learning to learn, Kluwer, BostonGoogle Scholar
  20. Xiang C, Fan X, Lee T (2006) Face recognition using recursive fisher linear discriminant. IEEE Trans on Image Process 15:2097–2105CrossRefGoogle Scholar
  21. Zheng W (2006) Class-incremental generalized discriminant analysis. Neural Comput 18:979–1006zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer 2010

Authors and Affiliations

  • Masayuki Hisada
    • 1
  • Seiichi Ozawa
    • 1
    Email author
  • Kau Zhang
    • 1
  • Nikola Kasabov
    • 2
  1. 1.Graduate School of EngineeringKobe UniversityKobeJapan
  2. 2.Knowledge Engineering and Discovery Research InstituteAuckland University of TechnologyAucklandNew Zealand

Personalised recommendations