Abstract
k-Nearest Neighbor (kNN) is a commonly used supervised learning method with a simple mechanism: given a testing sample, find the k nearest training samples based on some distance metric, and then use these k ‘‘neighbors” to make predictions. Typically, for classification problems, voting can be used to predict the testing sample as the most frequent class label in the k neighbors; for regression problems, averaging can be used to predict the testing sample as the average of the k real-valued outputs. Besides, the samples can be weighted by the distances in the way that a closer sample is assigned a higher weight.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aha D (ed) (1997) Lazy learning. Kluwer, Norwell
Baudat G, Anouar F (2000) Generalized discriminant analysis using a kernel approach. Neural Comput 12(10):2385–2404
Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput 15(6):1373–1396
Belkin M, Niyogi P, Sindhwani V (2006) Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. J Mach Learn Res 7:2399–2434
Bellman RE (1957) Dynamic programming. Princeton University Press, Princeton
Cover TM, Hart PE (1967) Nearest neighbor pattern classification. IEEE Trans Inf Theory 13(1):21–27
Cox TF, Cox MA (2001) Multidimensional scaling. Chapman & Hall/CRC, London
Davis JV, Kulis B, Jain P, Sra S, Dhillon IS (2007) Information-theoretic metric learning. In: Proceedings of the 24th international conference on machine learning (ICML), Corvalis, OR, pp 209–216
Fisher RA (1936) The use of multiple measurements in taxonomic problems. Ann Eugen 7(2):179–188
Friedman JH, Kohavi R, Yun Y (1996) Lazy decision trees. In: Proceedings of the 13th national conference on artificial intelligence (AAAI), Portland, OR, pp 717–724
Frome A, Singer Y, Malik J (2007) Image retrieval and classification using local distance functions. In: Scholkopf B, Platt JC, Hoffman T (eds) Advances in neural information processing systems 19 (NIPS). MIT Press, Cambridge, pp 417–424
Geng X, Zhan D-C, Zhou Z-H (2005) Supervised nonlinear dimensionality reduction for visualization and classification. IEEE Trans Syst Man Cybern-Part B: Cybern 35(6):1098–1107
Goldberger J, Hinton GE, Roweis ST, Salakhutdinov RR (2005) Neighbourhood components analysis. In: Saul LK, Weiss Y, Bottou L (eds) Advances in neural information processing systems 17 (NIPS). MIT Press, Cambridge, pp 513–520
Harden DR, Szedmak S, Shawe-Taylor J (2004) Canonical correlation analysis: an overview with application to learning methods. Neural Comput 16(12):2639–2664
He X, Niyogi P (2004) Locality preserving projections. In: Thrun S, Saul LK, Scholkopf B (eds) Advances in neural information processing systems 16 (NIPS). MIT Press, Cambridge, pp 153–160
Hotelling H (1936) Relations between two sets of variates. Biometrika 28(3–4):321–377
Kolda TG, Bader BW (2009) Tensor decompositions and applications. SIAM Rev 51(3):455–500
Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326
Schölkopf B, Smola A, Müller K-R (1998) Nonlinear component analysis as a kernel eigenvalue problem. Neural Comput 10(5):1299–1319
Tenenbaum JB, de Silva V, Langford JC (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290(5500):2319–2323
Wagstaff K, Cardie C, Rogers S, Schrödl S (2001) Constrained \(k\)-means clustering with background knowledge. In: Proceedings of the 18th international conference on machine learning (ICML), Williamstown, MA, pp 577–584
Weinberger KQ, Saul LK (2009) Distance metric learning for large margin nearest neighbor classification. J Mach Learn Res 10:207–244
Xing EP, Ng AY, Jordan MI, Russell S (2003) Distance metric learning, with application to clustering with side-information. In: Becker S, Thrun S, Obermayer K (eds) Advances in neural information processing systems 15 (NIPS). MIT Press, Cambridge, MA, pp 505–512
Yan S, Xu D, Zhang B, Zhang H-J (2007) Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Trans Pattern Anal Mach Intell 29(1):40–51
Yang J, Zhang D, Frangi AF, Yang J-Y (2004) Two-dimensional PCA: A new approach to appearance-based face representation and recognition. IEEE Trans Pattern Anal Mach Intell 26(1):131–137
Yang L, Jin R, Sukthankar R, Liu Y (2006) An efficient algorithm for local distance metric learning. In: Proceedings of the 21st national conference on artificial intelligence (AAAI), Boston, MA, pp 543–548
Ye J, Janardan R, Li Q (2005) Two-dimensional linear discriminant analysis. In: Saul LK, Weiss Y, Bottou L (eds) Advances in neural information processing systems 17 (NIPS). MIT Press, Cambridge, pp 1569–1576
Zhan D-C, Li Y-F, Zhou Z-H (2009) Learning instance specific distances using metric propagation. In: Proceedings of the 26th international conference on machine learning (ICML), Montreal, Canada, pp 1225–1232
Zhang D, Zhou Z-H (2005) (2D)\(^2\)PCA: 2-directional 2-dimensional PCA for efficient face representation and recognition. Neurocomputing 69(1–3):224–231
Zhang Z, Zha H (2004) Principal manifolds and nonlinear dimension reduction via local tangent space alignment. SIAM J Sci Comput 26(1):313–338
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Zhou, ZH. (2021). Dimensionality Reduction and Metric Learning. In: Machine Learning. Springer, Singapore. https://doi.org/10.1007/978-981-15-1967-3_10
Download citation
DOI: https://doi.org/10.1007/978-981-15-1967-3_10
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-1966-6
Online ISBN: 978-981-15-1967-3
eBook Packages: Computer ScienceComputer Science (R0)