Metric Learning for Prototype-Based Classification

  • Michael Biehl
  • Barbara Hammer
  • Petra Schneider
  • Thomas Villmann
Part of the Studies in Computational Intelligence book series (SCI, volume 247)


In this chapter, one of themost popular and intuitive prototype-based classification algorithms, learning vector quantization (LVQ), is revisited, and recent extensions towards automatic metric adaptation are introduced. Metric adaptation schemes extend LVQ in two aspects: on the one hand a greater flexibility is achieved since the metric which is essential for the classification is adapted according to the given classification task at hand. On the other hand a better interpretability of the results is gained since the metric parameters reveal the relevance of single dimensions as well as correlations which are important for the classification. Thereby, the flexibility of the metric can be scaled from a simple diagonal term to full matrices attached locally to the single prototypes. These choices result in a more complex form of the classification boundaries of the models, whereby the excellent inherent generalization ability of the classifier is maintained, as can be shown by means of statistical learning theory.


Support Vector Machine Cost Function Learning Rule Generalization Ability Learn Vector Quantization 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alegre, E., Biehl, M., Petkov, N., Sanchez, L.: Automatic classification of the acrosome status of boar spermatozoa using digital image processing and LVQ. Computers in Biology and Medicine 38, 461–468 (2008)CrossRefGoogle Scholar
  2. 2.
    Bartlett, P.L., Mendelson, S.: Rademacher and Gaussian complexities: risk bounds and structural risks. Journal of Machine Learning Research 3, 463–481 (2002)CrossRefMathSciNetGoogle Scholar
  3. 3.
    Biehl, M., Breitling, R., Li, Y.: Analysis of Tiling Microarray Data by Learning Vector Quantization and Relevance Learning. In: Yin, H., Tino, P., Corchado, E., Byrne, W., Yao, X. (eds.) IDEAL 2007. LNCS, vol. 4881, pp. 880–889. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  4. 4.
    Biehl, M., Caticha, N., Riegler, P.: Statistical mechanics of on-line learning. In: Biehl, M., Hammer, B., Verleysen, M., Villmann, T. (eds.) Similarity-based clustering, biomedical applications, and beyond. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  5. 5.
    Biehl, M., Gosh, A., Hammer, B.: Dynamics and generalization ability of LVQ algorithms. Journal of Machine Learning Research 8, 323–360 (2007)Google Scholar
  6. 6.
    Biehl, M., Hammer, B., Schneider, P.: Matrix Learning in Learning Vector Quantization, Technical Report Clausthal University of Technology, Department of Computer Science, IfI-06-14 (2006)Google Scholar
  7. 7.
    Bojer, T., Hammer, B., Koers, C.: Monitoring technical systems with prototype based clustering. In: Verleysen, M. (ed.) European Symposium on Artificial Neural Networks 2003, pp. 433–439. D-side publications (2003)Google Scholar
  8. 8.
    Bojer, T., Hammer, B., Schunk, D., Tluk von Toschanowitz, K.: Relevance determination in learning vector quantization. In: Proc. of European Symposium on Artificial Neural Networks (ESANN 2001), pp. 271–276. D facto publications (2001)Google Scholar
  9. 9.
    Bunte, K., Petkov, N., Bosman, H.H.W.J., Biehl, M., Jonkman, M.: Efficient color features for content based image retrieval in dermatolgoy (submitted, 2009)Google Scholar
  10. 10.
    Bunte, K., Schneider, P., Hammer, B., Schleif, F.-M., Villmann, T., Biehl, M.: Discriminative visualization by limited rank matrix learning. Machine Learning Reports MLR-03-2008 (2008),, ISSN:1865-3960
  11. 11.
    Crammer, K., Gilad-Bachrach, R., Navot, A., Tishby, A.: Margin analysis of the LVQ algorithm. In: Advances of Neural Information Processing Systems (2002)Google Scholar
  12. 12.
    Denecke, A., Wersing, H., Steil, J.J., Körner, E.: Robust object segmentation by adaptive metrics in Generalized LVQ (submitted, 2009)Google Scholar
  13. 13.
    Duda, R.O., Hart, P.E., Storck, D.G.: Pattern Classification. Wiley, Chichester (2001)zbMATHGoogle Scholar
  14. 14.
    Grandvalet, Y.: Anisotropic noise injection for input variable relevance determination. IEEE Transactions on Neural Networks 11(6), 1201–1212 (2000)CrossRefGoogle Scholar
  15. 15.
    Guyon, I., Elisseeff, A.: An Introduction to Variable and Feature Selection 3, 1157–1182 (2003)Google Scholar
  16. 16.
    Hammer, B., Strickert, M., Villmann, T.: On the generalization ability of GRLVQ networks. Neural Processing Letters 21(2), 109–120 (2005)CrossRefGoogle Scholar
  17. 17.
    Hammer, B., Strickert, M., Villmann, T.: Supervised neural gas with general similarity measure. Neural Processing Letters 21(1), 21–44 (2005)CrossRefGoogle Scholar
  18. 18.
    Hammer, B., Villmann, T.: Generalized relevance learning vector quantization. Neural Networks 15, 1059–1068 (2002)CrossRefGoogle Scholar
  19. 19.
    Kohonen, T.: Self-Organizing Maps. Springer, Heidelberg (2001)zbMATHGoogle Scholar
  20. 20.
    Lee, J.A., Verleysen, M.: Nonlinear dimensionality reduction. Springer, Heidelberg (2007)zbMATHCrossRefGoogle Scholar
  21. 21.
    Pregenzer, M., Pfurtscheller, G., Flotzinger, D.: Automated feature selection with distinction sensitive learning vector quantization. Neurocomputing 11, 19–29 (1996)CrossRefGoogle Scholar
  22. 22.
    Sato, A.S., Yamada, K.: An analysis of convergence in generalized LVQ. In: Niklasson, L., Boden, M., Ziemke, T. (eds.) ICANN 1998, pp. 172–176. Springer, Heidelberg (1998)Google Scholar
  23. 23.
    Sato, A.S., Yamada, K.: Generalized learning vector quantization. In: Tesauro, G., Touretzky, D., Leen, T. (eds.) Advances in Neural Information Processing Systems, vol. 7, pp. 423–429. MIT Press, Cambridge (1995)Google Scholar
  24. 24.
    Schleif, F.-M., Hammer, B., Kostrzewa, M., Villmann, T.: Exploration of Mass-Spectrometric Data in Clinical Proteomics Using Learning Vector Quantization Methods. Briefings in Bioinformatics 9(2), 129–143 (2007)CrossRefGoogle Scholar
  25. 25.
    Schneider, P., Biehl, M., Hammer, B.: Matrix adaptation in discriminative vector quantization. Technical Report Clausthal University of Technology, Department of Computer Science, IfI-08-08 (2008)Google Scholar
  26. 26.
    Schneider, P., Biehl, M., Hammer, B.: Adaptive relevance matrices in learning vector quantization (submitted, 2009)Google Scholar
  27. 27.
    Schneider, P., Bunte, K., Stiekema, H., Hammer, B., Villmann, T., Biehl, M.: Regularization in matrix relevance learning. Machine Learning Reports MLR-02-2008 (2008),, ISSN:1865-3960
  28. 28.
    Seo, S., Bode, M., Obermayer, K.: Soft nearest prototype classification. IEEE Transations on Neural Networks 14(2), 390–398 (2003)CrossRefGoogle Scholar
  29. 29.
    Seo, S., Obermayer, K.: Soft learning vector quantization. Neural Computation 15(7), 1589–1604 (2003)zbMATHCrossRefGoogle Scholar
  30. 30.
    Sontag, E.D.: Feedforward nets for interpolation and classification. Journal of Computer and System Sciences 45 (1992)Google Scholar
  31. 31.
    Strickert, M., Seiffert, U., Sreenivasulu, N., Weschke, W., Villmann, T., Hammer, B.: Generalized Relevance LVQ (GRLVQ) with Correlation Measures for Gene Expression Data. Neurocomputing 69, 651–659 (2006)CrossRefGoogle Scholar
  32. 32.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B 58, 267–288 (1996)zbMATHMathSciNetGoogle Scholar
  33. 33.
    Valiant, L.: A Theory of the Learnable. Communications of the ACM 27(11), 1134–1142 (1984)zbMATHCrossRefGoogle Scholar
  34. 34.
    Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)zbMATHGoogle Scholar
  35. 35.
    Villmann, T., Merenyi, E., Hammer, B.: Neural maps in remote sensing image analysis. Neural Networks 16(3-4), 389–403 (2003)CrossRefGoogle Scholar
  36. 36.
    Villmann, T., Schleif, F.-M., Hammer, B.: Comparison of Relevance Learning Vector Quantization with other Metric Adaptive Classification Methods. Neural Networks 19, 610–622 (2006)zbMATHCrossRefGoogle Scholar
  37. 37.
    Weinberger, K., Blitzer, J., Saul, L.: Distance metric learning for large margin nearest neighbor classification. In: Weiss, Y., Scholkopf, B., Platt, J. (eds.) Advances in Neural Information Processing Systems 18, pp. 1473–1480. MIT Press, Cambridge (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Michael Biehl
    • 1
  • Barbara Hammer
    • 2
  • Petra Schneider
    • 1
  • Thomas Villmann
    • 3
  1. 1.University of GroningenThe Netherlands
  2. 2.Clausthal University of TechnologyGermany
  3. 3.University of LeipzigGermany

Personalised recommendations