Skip to main content
SpringerLink
Log in

On kernel difference-weighted k-nearest neighbor classification

  • Theoretical Advances
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

Nearest neighbor (NN) rule is one of the simplest and the most important methods in pattern recognition. In this paper, we propose a kernel difference-weighted k-nearest neighbor (KDF-KNN) method for pattern classification. The proposed method defines the weighted KNN rule as a constrained optimization problem, and we then propose an efficient solution to compute the weights of different nearest neighbors. Unlike traditional distance-weighted KNN which assigns different weights to the nearest neighbors according to the distance to the unclassified sample, difference-weighted KNN weighs the nearest neighbors by using both the correlation of the differences between the unclassified sample and its nearest neighbors. To take into account the effective nonlinear structure information, we further extend difference-weighted KNN to its kernel version KDF-KNN. Our experimental results indicate that KDF-WKNN is much better than the original KNN and the distance-weighted KNN methods, and is comparable to or better than several state-of-the-art methods in terms of classification accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Aha DW, Kibler D, Albert MK (1991) Instance-based learning algorithms. Mach Learn 6:37–66

    Google Scholar 

  2. Atiya AF (2005) Estimating the posterior probabilities using the K-nearest neighbor rule. Neural Comput 17:731–740

    Article  MATH  Google Scholar 

  3. Bailey T, Jain AK (1978) A note on distance-weighted k-nearest neighbor rules. IEEE Trans Syst Man Cybern 8:311–313

    Article  MATH  Google Scholar 

  4. Beygelzimer A, Kakade S, Langford J (2006) Cover trees for nearest neighbor. In: International conference on machine learning

  5. Blake CL, Merz CJ (1998) UCI repository of machine learning databases, Department of Information and Computer Sciences, University of California, Irvine. http://www.ics.uci.edu/mlearn/MLRepository.html

  6. Dasarathy BV (1991) Nearest neighbor (NN) norms: NN pattern classification techniques. IEEE Computer Society Press, Los Alamitos

    Google Scholar 

  7. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30

    MathSciNet  Google Scholar 

  8. Domeniconi C, Peng J, Gunopulos D (2002) Locally adaptive metric nearest neighbor classification. IEEE Trans Pattern Anal Mach Intell 24:1281–1285

    Article  Google Scholar 

  9. Duda RO, Hart PE, Stork DG (2001) Pattern classification, 2nd edn. Wiley, London

    Google Scholar 

  10. Dudani SA (1976) The distance-weighted k-nearest-neighbor rule. IEEE Trans Syst Man Cybern 6:325–327

    Google Scholar 

  11. Fix E, Hodges JL (1951) Discriminatory analysis:nonparametric discrimination:consistency properties. USAF School of Aviation Medicine, Project 21-49-004, Report No. 4:261–279

  12. Fukunaga K, Flick TE (1984) An optimal global nearest neighbor metric. IEEE Trans Pattern Anal Mach Intell 6:314–318

    MATH  Google Scholar 

  13. Hastie T, Tibshirani R (1996) Discriminant adaptive nearest neighbor classification. IEEE Trans Pattern Anal Mach Intell 18:607–616

    Article  Google Scholar 

  14. Hastie T, Tibshirani R, Friedman J (2001) The elements of statistical learning: data mining, inference, and prediction. Springer, New York

    MATH  Google Scholar 

  15. Herrero JR, Navarro JJ (2007) Exploiting computer resources for fast nearest neighbor classification. Pattern Anal Appl 10:265–275

    Article  MathSciNet  Google Scholar 

  16. Keller JM, Gray MR, Givens Jr. JA (1985) A fuzzy k-nearest neighbor algorithm. IEEE Trans Syst Man Cybern 15:580–585

    Google Scholar 

  17. Kuncheva LI, Bezdek JC (1998) Nearest prototype classification clustering, genetic algorithms, or random search. IEEE Trans Syst Man Cybern C 28:160–164

    Article  Google Scholar 

  18. Kuncheva LI, Bezdek JC (1999) Presupervised and postsupervised prototype classifier design. IEEE Trans Neural Netw 10:1142–1152

    Article  Google Scholar 

  19. Lam W, Keung CK, Liu D (2002) Discovering useful concept prototypes for classification based on filtering and abstraction. IEEE Trans Pattern Anal Mach Intell 24:1075–1090

    Article  Google Scholar 

  20. Macleod JES, Luk A, Titterington DM (1987) A re-examination of the distance-weighted k-nearest neighbor classification rule. IEEE Trans Syst Man Cybern 17:689–696

    Article  Google Scholar 

  21. Morin RL, Raeside DE (1981) A reappraisal of distance-weighted k-nearest neighbor classification for pattern recognition with missing data. IEEE Trans Syst Man Cybern 11:241–243

    Article  MathSciNet  Google Scholar 

  22. Müller KR, Mika S, Rätsch G, Tsuda K, Schölkopf B (2001) An introduction to kernel-based learning algorithms. IEEE Trans Neural Netw 12:181–202

    Article  Google Scholar 

  23. Paredes R, Vidal E (2006) Learning prototypes and distances: a prototype reduction technique based on nearest neighbor error minimization. Pattern Recognit 39:180–188

    Article  MATH  Google Scholar 

  24. Paredes R, Vidal E (2006) Learning weighted metrics to minimizing nearest-neighbor classification error. IEEE Trans Pattern Anal Mach Intell 28:1100–1110

    Article  Google Scholar 

  25. Peng J, Heisterkamp DR, Dai H (2004) Adaptive quasiconformal kernel nearest neighbor classification. IEEE Trans Pattern Anal Mach Intell 26:656–661

    Article  Google Scholar 

  26. Ricci F, Avesani P (1999) Data compression and local metrics for nearest neighbor classification. IEEE Trans Pattern Anal Mach Intell 21:380–384

    Article  Google Scholar 

  27. Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290:2323–2326

    Article  Google Scholar 

  28. Saul LK, Roweis ST (2003) Think globally, fit locally: unsupervised learning of low dimensional manifolds. J Mach Learn Res 4:119–155

    Article  MathSciNet  Google Scholar 

  29. Shakhnarovich G, Darrell T, Indyk P (2006) Nearest-neighbor methods in learning and vision: theory and practice. MIT press, Cambridge

    Google Scholar 

  30. Sheskin DJ (2004) Handbook of parametric and nonparametric statistical procedures, 3rd edn. Chapman & Hall/CRC, Boca Raton

    Google Scholar 

  31. Short RD, Fukunaga K (1984) The optimal distance measure for nearest neighbor classification. IEEE Trans Inf Theory 27:622–627

    Article  MathSciNet  Google Scholar 

  32. Toh KA, Tran QL, Srinivasan D (2004) Benchmarking a reduced multivariate polynomial pattern classifier. IEEE Trans Pattern Anal Mach Intell 26(6):740–755

    Article  Google Scholar 

  33. Tran QL, Toh KA, Srinivasan D, Wong KL, Low SQ (2005) An empirical comparison of nine pattern classifiers. IEEE Trans Syst Man Cybern B 35:1079–1091

    Article  Google Scholar 

  34. Vapnik VN (1998) Statistical learning theory. Wiley, New York

    MATH  Google Scholar 

  35. Zar JH (1999) Biostatistical analysis, 4th edn. Prentice Hall, Upper Saddle River

    Google Scholar 

  36. Zavrel J (1997) An empirical re-examination of weighted voting for K-NN. In: Daelemans W, Flach P, van den Bosch A (eds) Proceedings of the 7th Belgian-Dutch Conference on Machine Learning, Tilburg, pp 139–148

Download references

Acknowledgments

The work is supported in part by the UGC/CRC fund from the HKSAR Government, the central fund from the Hong Kong Polytechnic University and the National Natural Science Foundation of China (NSFC) under the contract No. 60332010, the 863 Project under the contract No. 2006AA01Z308 and No. 2006AA01Z193. Finally, the authors would like to thank the anonymous reviewers for their constructive suggestions.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Harbin Institute of Technology, Harbin, 150001, China

    Wangmeng Zuo & Kuanquan Wang

  2. Biometrics Research Centre, Department of Computing, Hong Kong Polytechnic University, Kowloon, Hong Kong

    David Zhang

Authors
  1. Wangmeng Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kuanquan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wangmeng Zuo.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zuo, W., Zhang, D. & Wang, K. On kernel difference-weighted k-nearest neighbor classification. Pattern Anal Applic 11, 247–257 (2008). https://doi.org/10.1007/s10044-007-0100-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-007-0100-z

Keywords

Search

Navigation