A Study of the Robustness of KNN Classifiers Trained Using Soft Labels

  • Neamat El Gayar
  • Friedhelm Schwenker
  • Günther Palm
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4087)


Supervised learning models most commonly use crisp labels for classifier training. Crisp labels fail to capture the data characteristics when overlapping classes exist. In this work we attempt to compare between learning using soft and hard labels to train K-nearest neighbor classifiers. We propose a new technique to generate soft labels based on fuzzy-clustering of the data and fuzzy relabelling of cluster prototypes. Experiments were conducted on five data sets to compare between classifiers that learn using different types of soft labels and classifiers that learn with crisp labels. Results reveal that learning with soft labels is more robust against label errors opposed to learning with crisp labels. The proposed technique to find soft labels from the data, was also found to lead to a more robust training in most data sets investigated.


Radial Basis Function Fuzzy Cluster Cluster Prototype Fuzzy Neural Network Model Fuzzy Label 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    El Gayar, N.F.: Fuzzy Neural Network Models for Unsupervised and Confidence-Based Learning. PhD thesis, University of Alexandria, Dept. of Comp. Sc. (1999)Google Scholar
  2. 2.
    Kuncheva, L.: Fuzzy classifier design. Physica-Verlag (2000)Google Scholar
  3. 3.
    Bezdek, J., Pal, N.: Two soft relatives of learning vector quantization. Neural Networks 8(5), 729–743 (1995)CrossRefGoogle Scholar
  4. 4.
    Villmann, T., Hammer, B., Schleif, F.M., Geweniger, T.: Fuzzy labeled neural gas for fuzzy classification. In: Proceedings of WSOM 2005, pp. 283–290 (2005)Google Scholar
  5. 5.
    Pal, S., Mitra, S.: Multilayer perceptron, fuzzy sets and classification. IEEE Transactions on Neural Networks 3, 683–697 (1992)CrossRefGoogle Scholar
  6. 6.
    Keller, J., Gray, M., Givens, J.: A fuzzy k-nearest algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence 7, 693–699 (1985)CrossRefGoogle Scholar
  7. 7.
    Angluin, D., Laird, D.: Learning from noisy examples. Machine Learning 2(4), 343–370 (1987)Google Scholar
  8. 8.
    Lam, P., Stork, D.: Evaluating classifiers by means of test data with noisy labels. In: Proceedings of the International Joint Conference on Artificial Intelligence, pp. 513–518 (2003)Google Scholar
  9. 9.
    Bernadt, J., Soh, L.: Authoritative citation knn learning with noisy training datasets. In: Proceedings of the International Conference on Artificial Intelligence, pp. 916–931 (2004)Google Scholar
  10. 10.
    Jin, R., Ghahramani, Z.: Learning with multiple labels. In: Advances in Neural Information Processing Systems, vol. 15 (2003)Google Scholar
  11. 11.
    Kuncheva, L., Bezdek, J.: An integrated frame work for generalized nearest prototype classifier design. International Journal of Uncertainty, Fuzziness and Knowledge-based System 6(5), 437–457 (1998)MATHCrossRefGoogle Scholar
  12. 12.
    Villmann, T., Schleif, F.M., Hammer, B.: Fuzzy labeled soft nearest neighbor classification with relevance learning. In: Proceedings of ICMLA 2005, pp. 11–15. IEEE Press, Los Alamitos (2005)Google Scholar
  13. 13.
    Karayiannis, N.: Repairs to GLVQ: A new family of competitive learning schemes. IEEE Transaction on Neural Networks 7, 1062–1071 (1996)CrossRefGoogle Scholar
  14. 14.
    Seo, S., Bode, M., Obermayer, K.: Soft nearest prototype classification. IEEE Transaction on Neural Networks 14, 390–398 (2003)CrossRefGoogle Scholar
  15. 15.
    Seo, S., Obermayer, K.: Soft learning vector quantization 15, 1589–1604 (2003)Google Scholar
  16. 16.
    Melin, P., Castillo, O.: Hybrid Intelligent Systems for Pattern Recognition Using Soft Computing. Springer, Heidelberg (2005)MATHGoogle Scholar
  17. 17.
    Xie, X., Beni, G.: A validity measure for fuzzy clustering. IEEE Trans. on Pattern Analysis and Machine Intelligence 13(8), 841–847 (1991)CrossRefGoogle Scholar
  18. 18.
    Czogala, E., Leski, J.: Fuzzy And Neuro-Fuzzy Intelligent Systems. Physica-Verlag (2000)Google Scholar
  19. 19.
    Van de Wouwer, P., Scheunders, P., van Dyck, M., De Bodt, M., Wuyts, F., Van de Heyning, P.: Wavelet-filvq classifier for speech analysis. In: Proc. of the Int. Conf. Pattern Recognition, pp. 214–218. IEEE Press, Los Alamitos (1996)CrossRefGoogle Scholar
  20. 20.
    Klir, G., Yuan, B.: Fuzzy Sets and Fuzzy Logic, Theory and Applications. Prentice-Hall of India (2002)Google Scholar
  21. 21.
    Nene, S.A., Nayar, S.K., Murase, H.: Columbia object image library (coil-20). Technical report, Technical Report CUCS-005-96 (1996)Google Scholar
  22. 22.
    ELENA: (, directory: pub/neural-nets/elena/databases)Google Scholar
  23. 23.
    Duda, R., Hart, P., Stork, D.: Pattern Classification. John Wiley and Sons, Chichester (2004)Google Scholar
  24. 24.
    Kuncheva, L.: Combining pattern classifiers. John Wiley and Sons, Chichester (2004)MATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Neamat El Gayar
    • 1
  • Friedhelm Schwenker
    • 2
  • Günther Palm
    • 2
  1. 1.Faculty of Computers and InformationCairo UniversityGizaEgypt
  2. 2.Department of Neural Information ProcessingUniversity of UlmUlmGermany

Personalised recommendations