Abstract
Fisher linear discriminant analysis (LDA) and its kernel extension—kernel discriminant analysis (KDA)—are well known methods that consider dimensionality reduction and classification jointly. While widely deployed in practical problems, there are still unresolved issues surrounding their efficient implementation and their relationship with least mean squared error procedures. In this paper we address these issues within the framework of regularized estimation. Our approach leads to a flexible and efficient implementation of LDA as well as KDA. We also uncover a general relationship between regularized discriminant analysis and ridge regression. This relationship yields variations on conventional LDA based on the pseudoinverse and a direct equivalence to an ordinary least squares estimator. Experimental results on a collection of benchmark data sets demonstrate the effectiveness of our approach.
Chapter PDF
Similar content being viewed by others
Keywords
- Discriminant Analysis
- Linear Discriminant Anal
- Ridge Regression
- Scatter Matrix
- Generalize Singular Value Decomposition
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Baudat, G., Anouar, F.: Generalized discriminant analysis using a kernel approach. Neural Computation 12, 2385–2404 (2000)
Cheng, Y.-Q., Zhuang, Y.-M., Yang, J.-Y.: Optimal Fisher discriminant analysis using the rank decomposition. Pattern Recognition 25(1), 101–111 (1992)
Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. John Wiley and Sons, New York (2001)
Friedman, J.H.: Regularized discriminant analysis. Journal of the American Statistical Association 84(405), 165–175 (1989)
Golub, G.H., Heath, M., Wahba, G.: Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21, 215–223 (1979)
Golub, G.H., Van Loan, C.F.: Matrix Computations. Johns Hopkins University Press, Baltimore (1996)
Hastie, T., Tishiran, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, Heidelberg (2001)
Hoerl, A.E., Kennard, R.W.: Ridge regression. Technometrics 12, 56–67, 69–82 (1970)
Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, Cambridge (1985)
Howland, P., Jeon, M., Park, H.: Structure preserving dimension reduction for clustered text data based on the generalized singular value decomposition. SIAM Journal on Matrix Analysis and Applications 25(1), 165–179 (2003)
Howland, P., Park, H.: Generalizing discriminant analysis using the generalized singular value decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(8), 995–1006 (2004)
Kittler, J., Young, P.C.: A new approach to feature selection based on the Karhunen-Loève expansion. Pattern Recognition 5, 335–352 (1973)
Mika, S., Rätsch, G., Weston, J., Schölkopf, B., Smola, A., Müller, K.R.: Invariant feature extraction and classification in kernel space. In: Advances in Neural Information Processing Systems 12, vol. 12, pp. 526–532 (2000)
Paige, C.C., Saunders, M.A.: Towards a generalized singular value decomposition. SIAM Journal on Numerical Analysis 18(3), 398–405 (1981)
Park, C.H., Park, H.: Nonlinear discriminant analysis using kernel functions and the generalized singular value decomposition. SIAM Journal on Matrix Analysis and Applications 27(1), 87–102 (2005)
Park, C.H., Park, H.: A relationship between linear discriminant analysis and the generalized minimum squared error solution. SIAM Journal on Matrix Analysis and Applications 27(2), 474–492 (2005)
Roth, V., Steinhage, V.: Nonlinear discriminant analysis using kernel functions. In: Advances in Neural Information Processing Systems 12, vol. 12, pp. 568–574 (2000)
Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge (2004)
Webb, A.R.: Statistical Pattern Recognition. John Wiley & Sons, Hoboken (2002)
Ye, J.: Least squares linear discriminant analysis. In: The Twenty-Fourth International Coference on Machine Learning (ICML) (2007)
Ye, J., Li, Q., Xiong, H., Park, H., Janardan, R., Kumar, V.: An incremental dimension reduction algorithm via QR decomposition. In: ACM SIGKDD, pp. 364–373 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhang, Z., Dai, G., Jordan, M.I. (2009). A Flexible and Efficient Algorithm for Regularized Fisher Discriminant Analysis. In: Buntine, W., Grobelnik, M., Mladenić, D., Shawe-Taylor, J. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2009. Lecture Notes in Computer Science(), vol 5782. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04174-7_41
Download citation
DOI: https://doi.org/10.1007/978-3-642-04174-7_41
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-04173-0
Online ISBN: 978-3-642-04174-7
eBook Packages: Computer ScienceComputer Science (R0)