Divergence Based Online Learning in Vector Quantization

  • Thomas Villmann
  • Sven Haase
  • Frank-Michael Schleif
  • Barbara Hammer
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6113)


We propose the utilization of divergences in gradient descent learning of supervised and unsupervised vector quantization as an alternative for the squared Euclidean distance. The approach is based on the determination of the Fréchet-derivatives for the divergences, wich can be immediately plugged into the online-learning rules.


vector quantization divergence based learning information theory 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Amari, S.-I.: Differential-Geometrical Methods in Statistics. Springer, Heidelberg (1985)zbMATHGoogle Scholar
  2. 2.
    Banerjee, A., Merugu, S., Dhillon, I., Ghosh, J.: Clustering with bregman divergences. Journal of Machine Learning Research 6, 1705–1749 (2005)MathSciNetGoogle Scholar
  3. 3.
    Bauer, H.-U., Pawelzik, K.R.: Quantifying the neighborhood preservation of Self-Organizing Feature Maps. IEEE Trans. on Neural Networks 3(4), 570–579 (1992)CrossRefGoogle Scholar
  4. 4.
    Campbell, J.: Introduction to Remote Sensing. The Guilford Press, U.S.A. (1996)Google Scholar
  5. 5.
    Cichocki, A., Zdunek, R., Phan, A., Amari, S.-I.: Nonnegative Matrix and Tensor Factorizations. Wiley, Chichester (2009)CrossRefGoogle Scholar
  6. 6.
    Clark, R.N.: Spectroscopy of rocks and minerals, and principles of spectroscopy. In: Rencz, A. (ed.) Manual of Remote Sensing. John Wiley and Sons, Inc., New York (1999)Google Scholar
  7. 7.
    Cottrell, M., Hammer, B., Hasenfu, A., Villmann, T.: Batch and median neural gas. Neural Networks 19, 762–771 (2006)zbMATHCrossRefGoogle Scholar
  8. 8.
    Csiszr, I.: Information-type measures of differences of probability distributions and indirect observations. Studia Sci. Math. Hungaria 2, 299–318 (1967)Google Scholar
  9. 9.
    Frigyik, B.A., Srivastava, S., Gupta, M.: An introduction to functional derivatives. Technical Report UWEETR-2008-0001, Dept of Electrical Engineering, University of Washington (2008)Google Scholar
  10. 10.
    Fujisawa, H., Eguchi, S.: Robust parameter estimation with a small bias against heavy contamination. Journal of Multivariate Analysis 99, 2053–2081 (2008)zbMATHCrossRefMathSciNetGoogle Scholar
  11. 11.
    Heskes, T.: Energy functions for self-organizing maps. In: Oja, E., Kaski, S. (eds.) Kohonen Maps, pp. 303–316. Elsevier, Amsterdam (1999)CrossRefGoogle Scholar
  12. 12.
    Hulle, M.M.V.: Kernel-based topographic map formation achieved with an information theoretic approach. Neural Networks 15, 1029–1039 (2002)CrossRefGoogle Scholar
  13. 13.
    Jang, E., Fyfe, C., Ko, H.: Bregman divergences and the self organising map. In: Fyfe, C., Kim, D., Lee, S.-Y., Yin, H. (eds.) IDEAL 2008. LNCS, vol. 5326, pp. 452–458. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  14. 14.
    Kantorowitsch, I., Akilow, G.: Funktionalanalysis in normierten Rumen, 2nd edn. Akademie-Verlag, Berlin (1978) (revised edition)Google Scholar
  15. 15.
    Kohonen, T.: Self-Organizing Maps. Springer Series in Information Sciences, vol. 30. Springer, Heidelberg (1995) (Second Extended Edition 1997)Google Scholar
  16. 16.
    Kullback, S., Leibler, R.: On information and sufficiency. Annals of Mathematical Statistics 22, 79–86 (1951)zbMATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    Lee, J., Verleysen, M.: Generalization of the l p norm for time series and its application to self-organizing maps. In: Cottrell, M. (ed.) Proc. of Workshop on Self-Organizing Maps, WSOM 2005, Paris, Sorbonne, pp. 733–740 (2005)Google Scholar
  18. 18.
    Lehn-Schiler, T., Hegde, A., Erdogmus, D., Principe, J.: Vector quantization using information theoretic concepts. Natural Computing 4(1), 39–51 (2005)CrossRefMathSciNetGoogle Scholar
  19. 19.
    Linde, Y., Buzo, A., Gray, R.: An algorithm for vector quantizer design. IEEE Transactions on Communications 28, 84–95 (1980)CrossRefGoogle Scholar
  20. 20.
    Martinetz, T.M., Berkovich, S.G., Schulten, K.J.: ‘Neural-gas’ network for vector quantization and its application to time-series prediction. IEEE Trans. on Neural Networks 4(4), 558–569 (1993)CrossRefGoogle Scholar
  21. 21.
    Principe, J.C., FisherIII, J.W., Xu, D.: Information theoretic learning. In: Haykin, S. (ed.) Unsupervised Adaptive Filtering, Wiley, New York (2000)Google Scholar
  22. 22.
    Qin, A., Suganthan, P.: A novel kernel prototype-based learning algorithm. In: Proc. of the 17th Internat. Conf. on Pattern Recognition, ICPR 2004, vol. 4, pp. 621–624 (2004)Google Scholar
  23. 23.
    Renyi, A.: On measures of entropy and information. In: Proc. of the 4th Berkeley Symp. on Mathematical Statistics and Probability. Univ. of California Press, Berkeley (1961)Google Scholar
  24. 24.
    Renyi, A.: Probability Theory. North-Holland Publish. Company, Amsterdam (1970)Google Scholar
  25. 25.
    Sato, A., Yamada, K.: Generalized learning vector quantization. In: Touretzky, D.S., Mozer, M.C., Hasselmo, M.E. (eds.) Proc. of the 1995 Conf. on Advances in Neural Information Processing Systems, vol. 8, pp. 423–429. MIT Press, Cambridge (1996)Google Scholar
  26. 26.
    Villmann, T., Claussen, J.-C.: Magnification control in self-organizing maps and neural gas. Neural Computation 18(2), 446–469 (2006)zbMATHCrossRefMathSciNetGoogle Scholar
  27. 27.
    Villmann, T., Haase, S.: Mathematical aspects of divergence based vector quantization using frchet-derivatives - extended and revised version. Machine Learning Reports 4(MLR-01-2010), 1–35 (2010), ISSN:1865-3960, Google Scholar
  28. 28.
    Villmann, T., Merényi, E., Hammer, B.: Neural maps in remote sensing image analysis. Neural Networks 16(3-4), 389–403 (2003)CrossRefGoogle Scholar
  29. 29.
    Villmann, T., Schleif, F.-M.: Functional vector quantization by neural maps. In: Chanussot, J. (ed.) Proceedings of First Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS 2009), pp. 1–4. IEEE Press, Los Alamitos (2009)CrossRefGoogle Scholar
  30. 30.
    Zador, P.L.: Asymptotic quantization error of continuous signals and the quantization dimension. IEEE Transaction on Information Theory (28), 149–159 (1982)CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Thomas Villmann
    • 1
  • Sven Haase
    • 1
  • Frank-Michael Schleif
    • 2
  • Barbara Hammer
    • 2
  1. 1.Department of Mathematics/Natural Sciences/InformaticsUniversity of Applied Sciences Mittweida 
  2. 2.Institute of Computer ScienceClausthal University of TechnologyClausthal-ZellerfeldGermany

Personalised recommendations