Abstract
This paper presents an incremental learning solution for Linear Discriminant Analysis (LDA) and its applications to object recognition problems. We apply the sufficient spanning set approximation in three steps i.e. update for the total scatter matrix, between-class scatter matrix and the projected data matrix, which leads an online solution which closely agrees with the batch solution in accuracy while significantly reducing the computational complexity. The algorithm yields an efficient solution to incremental LDA even when the number of classes as well as the set size is large. The incremental LDA method has been also shown useful for semi-supervised online learning. Label propagation is done by integrating the incremental LDA into an EM framework. The method has been demonstrated in the task of merging large datasets which were collected during MPEG standardization for face image retrieval, face authentication using the BANCA dataset, and object categorisation using the Caltech101 dataset.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bar-Hillel, A., Hertz, T., Shental, N., & Weinshall, D. (2005). Learning a Mahalanobis metric from equivalence constraints. Journal of Machine Learning Research, 6, 937–965.
Belkin, M., & Niyogi, P. (2004). Learning with local and global consistency. In Proc. of NIPS.
Berg, A., Berg, T., & Malik, J. (2005). Shape matching and object recognition using low distortion correspondence. In Proc. of CVPR.
Bouveyron, C., Girard, S., & Schmid, C. (2004). Dimension reduction and classification methods for object recognition in vision. In Proc. of 5th French-Danish workshop on spatial statistics and image analysis in biology, Saint-Pierre de Chartreuse, France, May 2004.
Chan, C.-H., Kittler, J., & Messer, K. (2007). Multi-scale local binary pattern histograms for face recognition. In Proc. of ICB (pp. 809–818).
Cheng, H., Liu, Z., & Yang, J. (2009). Sparsity induced similarity measure for label propagation. In Proc. of ICCV.
Chin, T.-J., & Suter, D. (2006). Incremental kernel PCA for efficient non-linear feature extraction. In Proc. of BMVC.
De la Torre Frade, F., Gross, R., Baker, S., & Kumar, V. (2005). Representational oriented component analysis (ROCA) for face recognition with one sample image per training class. In Proc. of CVPR.
Fei-Fei, L., Fergus, R., & Perona, P. (2004). Learning generative visual models from few training examples. In Proc. of CVPR workshop on GMBV.
Fukunaga, K. (1990). Introduction to statistical pattern recognition. Boston: Academic Press.
Grauman, K., & Darrell, T. (2005). The pyramid match kernel: discriminative classification with sets of image features. In Proc. of ICCV.
Hall, P., Marshall, D., & Martin, R. (2000). Merging and splitting eigenspace models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(9), 1042–1049.
Hiraoka, K., Hidai, K., Hamahira, M., Mizoguchi, H., Mishima, T., & Yoshizawa, S. (2000). Successive learning of linear discriminant analysis: Sanger-type algorithm. In Proc. of ICPR.
Holub, A. D., Welling, M., & Perona, P. (2005). Exploiting unlabelled data for hybrid object classification. In Proc. of NIPS workshop on inter-class transfer.
Kamei, T., Yamada, A., Kim, T., Kim, H., Hwang, W., & Kee, S. (2002). Advanced face descriptor using Fourier and intensity LDA features. ISO/IEC JTC1/SC29/WG11 M8998, October 2002.
Kim, H.-C., Kim, D., & Bang, S. Y. (2003). Face recognition using LDA mixture model. Pattern Recognition Letters, 24(15), 2815–2821.
Kim, T.-K., Kim, H., Hwang, W., & Kittler, J. (2005). Component-based LDA face description for image retrieval and MPEG-7 standardisation. Image and Vision Computing, 23, 631–642.
Kim, T.-K., Kittler, J., & Cipolla, R. (2007). Discriminative learning and recognition of image set classes using canonical correlations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6), June 2007.
Kim, T.-K., Wong, S.-F., Stenger, B., Kittler, J., & Cipolla, R. (2007). Incremental linear discriminant analysis using sufficient spanning set approximations. In Proc. of CVPR, Minneapolis, MN.
Leibe, B., & Schiele, B. (2003). Analyzing appearance and contour based methods for object categorization. In Proc. of CVPR (pp. 409–415).
Lin, R.-S., Ross, D., Lim, J., & Yang, M.-H. (2005). Adaptive discriminative generative model and its applications. In Proc. of NIPS.
Manjunath, B. S., Salembier, P. P., & Sikora, T. (2002). Introduction to MPEG-7: multimedia content description interface. New York: Wiley.
Messer, K. et al. (2004). Face authentication competition on the BANCA database. In Proc. of ICPR.
Moosmann, F., Triggs, B., & Jurie, F. (2007). Fast discriminative visual codebooks using randomized clustering forests. In Proc. of NIPS.
Mutch, J., & Lowe, D. (2006). Multiclass object recognition with sparse, localized features. In Proc. of CVPR.
Niebles, J. C., Wang, H., & Fei-Fei, L. (2008). Unsupervised learning of human action categories using spatial-temporal words. International Journal of Computer Vision.
Oja, E. (1983). Subspace methods of pattern recognition. Research Studies Press.
Pang, S., Ozawa, S., & Kasabov, N. (2005). Incremental linear discriminant analysis for classification of data streams. IEEE Transactions on System, Man and Cybernetics, 905–914.
Poh, N. et al. (2009). Face video competition. In Proc. of ICPR.
Skocaj, D., & Leonardis, A. (2003). Weighted and robust incremental method for subspace learning. In Proc. of ICCV.
Stenger, B., Woodley, T., Kim, T.-K., Hernandez, C., & Cipolla, R. (2008). AIDIA—adaptive interface for display interaction. In Proc. of BMVC, Leeds, UK.
Tao, X., Ye, J., Li, Q., Janardan, R., & Cherkassky, V. (2004). Efficient kernel discriminant analysis via QR decomposition. In Proc. of NIPS.
Uray, M., Skocaj, D., Roth, P., Bischof, H., & Leonardis, A. (2007). Incremental LDA learning by combining reconstructive and discriminative approaches. In Proc. of BMVC.
Wang, F., & Zhang, C. (2007). Label propagation through linear neighborhoods. In Proc. of ICML.
Wang, X., Han, T. X., & Yan, S. (2009). An HOG-LBP human detector with partial occlusion handling. In Proc. of ICCV, Kyoto.
Winn, J., Criminisi, A., & Minka, T. (2005). Object categorisation by learned universal visual dictionary. In Proc. of ICCV.
Wu, Y., & Huang, S. (2000). View-independent recognition of hand postures. In Proc. of CVPR (pp. 2088–2094).
Yan, J., Zhang, B., Yan, S., Yang, Q., & Li, H. (2004). IMMC: incremental maximum margin criterion. In Proc. of int’l conf. knowledge discovery and data mining.
Ye, J., Li, Q., Xiong, H., Park, H., Janardan, V., & Kumar, V. (2005). IDR/QR: an incremental dimension reduction algorithm via QR decomposition. IEEE Transactions on Knowledge and Data Engineering, 17(9), 1208–1222.
Yu, H., & Yang, H. (2001). A direct LDA algorithm for high-dimensional data—with application to face recognition. Pattern Recognition, 34(10), 2067–2070.
Zhu, X. (2006). Semi-supervised learning literature survey. Computer Sciences TR 1530, University of Wisconsin-Madison.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kim, TK., Stenger, B., Kittler, J. et al. Incremental Linear Discriminant Analysis Using Sufficient Spanning Sets and Its Applications. Int J Comput Vis 91, 216–232 (2011). https://doi.org/10.1007/s11263-010-0381-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-010-0381-3