Manifold Based Local Classifiers: Linear and Nonlinear Approaches
- 183 Downloads
In case of insufficient data samples in high-dimensional classification problems, sparse scatters of samples tend to have many ‘holes’—regions that have few or no nearby training samples from the class. When such regions lie close to inter-class boundaries, the nearest neighbors of a query may lie in the wrong class, thus leading to errors in the Nearest Neighbor classification rule. The K-local hyperplane distance nearest neighbor (HKNN) algorithm tackles this problem by approximating each class with a smooth nonlinear manifold, which is considered to be locally linear. The method takes advantage of the local linearity assumption by using the distances from a query sample to the affine hulls of query’s nearest neighbors for decision making. However, HKNN is limited to using the Euclidean distance metric, which is a significant limitation in practice. In this paper we reformulate HKNN in terms of subspaces, and propose a variant, the Local Discriminative Common Vector (LDCV) method, that is more suitable for classification tasks where the classes have similar intra-class variations. We then extend both methods to the nonlinear case by mapping the nearest neighbors into a higher-dimensional space where the linear manifolds are constructed. This procedure allows us to use a wide variety of distance functions in the process, while computing distances between the query sample and the nonlinear manifolds remains straightforward owing to the linear nature of the manifolds in the mapped space. We tested the proposed methods on several classification tasks, obtaining better results than both the Support Vector Machines (SVMs) and their local counterpart SVM-KNN on the USPS and Image segmentation databases, and outperforming the local SVM-KNN on the Caltech visual recognition database.
KeywordsAffine hull Common vector Convex hull Distance learning Image categorization Local classifier Manifold learning Object recognition
- 1.Simard, P., Le Cun, Y., Denker, J., & Victorri, B. (1998). Transformation invariance in pattern recognition—tangent distance and tangent propagation, lecture notes in computer science (vol. 1524, pp. 239–274). Berlin: Springer.Google Scholar
- 3.Hastie, T., & Tibshirani, R. (1996). Discriminant adaptive nearest neighbor classification. IEEE Trans. PAMI, 18(6), 607–616.Google Scholar
- 4.Vincent, P., & Bengio, Y. (2001). K-local hyperplane and convex distance nearest neighbor algorithms. Adv Neural Inf Process Syst, 14, 985–992.Google Scholar
- 5.Domeniconi, C., & Gunopulos, D. (2002). Efficient local flexible nearest neighbor classification. In Proceedings of the 2nd SIAM International Conference on Data Mining.Google Scholar
- 6.Zhang, H., Berg, a. C., Maire, M., & Malik, J. (2006). SVM-KNN: discriminative nearest neighbor classification for visual category recognition, in CVPR 2006 (pp. 2126–2136).Google Scholar
- 9.Olkun, O. (2004). Protein fold recognition with K-local hyperplane distance nearest neighbor algorithm. In Proceedings of the 2nd European Workshop on data Mining and Text Mining in Bioinformatics, pp. 51–57.Google Scholar
- 12.Verbeek, J. (2006). Learning non-linear image manifolds by global alignment of local linear models. IEEE Trans PAMI, 28, 1236–1250.Google Scholar
- 13.Cevikalp, H., Neamtu, M., & Wilkes, M. (2005). Discriminative common vectors for face recognition. IEEE Trans PAMI, 27, 4–13.Google Scholar
- 14.Kim, T.-K., & Kittler, J. (2005). Locally linear discriminant analysis for multimodally distributed classes for face recognition with a single model image. IEEE Trans PAMI, 27, 318–327.Google Scholar
- 15.Fitzgibbon, A. W., & Zisserman, A. (2003). Joint manifold distance: a new approach to appearance based clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
- 16.Zhang, J., Marszalek, M., Lazebnik, S., & Schmidt, C. (2006). Local features and kernels for classification of texture and object categories: a comprehensive study. In Proceedings of the Computer Vision and Pattern Recognition Workshop.Google Scholar
- 23.Fei-Fei, L. Fergus, R., & Perona, P. (2004) Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. In Proceedings of the IEEE CVPR Workshop of Generative Model Based Vision.Google Scholar
- 24.USPS dataset of handwritten characters created by the US Postal Service. Retrieved from ftp://ftp.kyb.tuebingen.mpg.de/pub/bs/data.Google Scholar
- 25.Keysers, D., Dohmen, J., Theiner, T., & Ney, H. (2000). Experiments with an extended tangent distance. In Proceedings of the 15th International Conference on Pattern Recognition, vol. 2, pp. 38–42.Google Scholar
- 26.C codes for computing tangent distances. Retrieved from http://www-i6.informatik.rwth-aachen.de/∼keysers/td/.
- 28.UCI—benchmark repository—a huge collection of artificial and real world data sets. University of California Irvine. Retrieved from http://www.ics.edu/∼mlearn/MLRepository.html.
- 29.Csurka, G., Dance, C., Fan, L., Willamowski, J., Bray, C. (2004). Visual categorization with bags of keypoints. In Proceedings of the ECCV Workshop on Statistical Learning for Computer Vision.Google Scholar
- 30.Lazebnik, S., Schmid, C., & Ponce, J. (2005). A sparse texture representation using local affine regions. IEEE Trans PAMI, 27(8), 1265–1278.Google Scholar
- 31.Fowlkes, C., Belogie, S., Chung, F., & Malik, J. (2004). Spectral grouping using the Nystrom method. IEEE Trans PAMI, 26, 1–12.Google Scholar
- 33.Levina, E., & Bickel, P. J. (2005). Maximum likelihood estimation of intrinsic dimension. In L. K. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing system, 17 (pp. 777–784). Cambridge, MA: MIT Press.Google Scholar