BoostML: An Adaptive Metric Learning for Nearest Neighbor Classification

  • Nayyar Abbas Zaidi
  • David McG. Squire
  • David Suter
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6118)

Abstract

A Nearest Neighbor (NN) classifier assumes class conditional probabilities to be locally smooth. This assumption is often invalid in high dimensions and significant bias can be introduced when using the nearest neighbor rule. This effect can be mitigated to some extent by using a locally adaptive metric. In this work we propose an adaptive metric learning algorithm that learns an optimal metric at the query point. We learn a distance metric using a feature relevance measure inspired by boosting. The modified metric results in a smooth neighborhood that leads to better classification results. We tested our technique on major UCI machine learning databases and compared the results to state of the art techniques. Our method resulted in significant improvements in the performance of the K-NN classifier and also performed better than other techniques on major databases.

Keywords

Adaptive Metric Learning Nearest Neighbor Bias-Variance analysis Curse-of-Dimensionality Feature Relevance Index 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Davis, J., Dhillon, I.: Structured metric learning for high dimensional problems. In: KDD (2008)Google Scholar
  2. 2.
    Weinberger, K., Blitzer, J., Saul, L.: Distance metric learning for large margin nearest neighbor classification. In: NIPS (2005)Google Scholar
  3. 3.
    Friedman, J.: Flexible metric nearest neighbor classification. Tech Report, Dept. of Statistics, Stanford University, Tech. Rep. (1994)Google Scholar
  4. 4.
    Shapire, R., Singer, Y.: Improved boosting algorithms using confidence rated predictions. In: Conf. on Computational Learning Theory (1998)Google Scholar
  5. 5.
    Hastie, T., Tibshirani, R.: Discriminative adaptive nearest neighbor classification. IEEE transactions on Pattern Analysis and Machine Intelligence (1996)Google Scholar
  6. 6.
    Domenciconi, C., Peng, J., Gunopulos, D.: An adaptive metric machine for pattern classification. In: NIPS (2000)Google Scholar
  7. 7.
    Peng, J., Heisterkamp, D., Dai, H.: Lda/svm driven nearest neighbor classification. In: CVPR (2001)Google Scholar
  8. 8.
    Janusz, K. (ed.): Support Vector Machines: Theory and Applications. Springer, Heidelberg (2005); ch. Adaptive Discriminant and Quasiconformal Kernel Nearest Neighbor ClassificationGoogle Scholar
  9. 9.
    Domenciconi, C., Peng, J., Gunopulos, D.: Large margin nearest neighbor classifiers. IEEE transactions on Pattern Analysis and Machine Intelligence (2005)Google Scholar
  10. 10.
    Zhou, Z., Dai, H.: Query-sensitive similarity measure for content-based image retrieval. In: ICDM (2006)Google Scholar
  11. 11.
    Mertz, C., Murphy, P.: Machine learning repository (2005), http://archive.ics.uci.edu/ml/

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Nayyar Abbas Zaidi
    • 1
  • David McG. Squire
    • 1
  • David Suter
    • 2
  1. 1.Clayton School of IT Monash UniversityClaytonAustralia
  2. 2.School of Computer Science University of AdelaideNorth TerraceAustralia

Personalised recommendations