Abstract
We propose the utilization of divergences in gradient descent learning of supervised and unsupervised vector quantization as an alternative for the squared Euclidean distance. The approach is based on the determination of the Fréchet-derivatives for the divergences, wich can be immediately plugged into the online-learning rules.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Amari, S.-I.: Differential-Geometrical Methods in Statistics. Springer, Heidelberg (1985)
Banerjee, A., Merugu, S., Dhillon, I., Ghosh, J.: Clustering with bregman divergences. Journal of Machine Learning Research 6, 1705–1749 (2005)
Bauer, H.-U., Pawelzik, K.R.: Quantifying the neighborhood preservation of Self-Organizing Feature Maps. IEEE Trans. on Neural Networks 3(4), 570–579 (1992)
Campbell, J.: Introduction to Remote Sensing. The Guilford Press, U.S.A. (1996)
Cichocki, A., Zdunek, R., Phan, A., Amari, S.-I.: Nonnegative Matrix and Tensor Factorizations. Wiley, Chichester (2009)
Clark, R.N.: Spectroscopy of rocks and minerals, and principles of spectroscopy. In: Rencz, A. (ed.) Manual of Remote Sensing. John Wiley and Sons, Inc., New York (1999)
Cottrell, M., Hammer, B., Hasenfu, A., Villmann, T.: Batch and median neural gas. Neural Networks 19, 762–771 (2006)
Csiszr, I.: Information-type measures of differences of probability distributions and indirect observations. Studia Sci. Math. Hungaria 2, 299–318 (1967)
Frigyik, B.A., Srivastava, S., Gupta, M.: An introduction to functional derivatives. Technical Report UWEETR-2008-0001, Dept of Electrical Engineering, University of Washington (2008)
Fujisawa, H., Eguchi, S.: Robust parameter estimation with a small bias against heavy contamination. Journal of Multivariate Analysis 99, 2053–2081 (2008)
Heskes, T.: Energy functions for self-organizing maps. In: Oja, E., Kaski, S. (eds.) Kohonen Maps, pp. 303–316. Elsevier, Amsterdam (1999)
Hulle, M.M.V.: Kernel-based topographic map formation achieved with an information theoretic approach. Neural Networks 15, 1029–1039 (2002)
Jang, E., Fyfe, C., Ko, H.: Bregman divergences and the self organising map. In: Fyfe, C., Kim, D., Lee, S.-Y., Yin, H. (eds.) IDEAL 2008. LNCS, vol. 5326, pp. 452–458. Springer, Heidelberg (2008)
Kantorowitsch, I., Akilow, G.: Funktionalanalysis in normierten Rumen, 2nd edn. Akademie-Verlag, Berlin (1978) (revised edition)
Kohonen, T.: Self-Organizing Maps. Springer Series in Information Sciences, vol. 30. Springer, Heidelberg (1995) (Second Extended Edition 1997)
Kullback, S., Leibler, R.: On information and sufficiency. Annals of Mathematical Statistics 22, 79–86 (1951)
Lee, J., Verleysen, M.: Generalization of the l p norm for time series and its application to self-organizing maps. In: Cottrell, M. (ed.) Proc. of Workshop on Self-Organizing Maps, WSOM 2005, Paris, Sorbonne, pp. 733–740 (2005)
Lehn-Schiler, T., Hegde, A., Erdogmus, D., Principe, J.: Vector quantization using information theoretic concepts. Natural Computing 4(1), 39–51 (2005)
Linde, Y., Buzo, A., Gray, R.: An algorithm for vector quantizer design. IEEE Transactions on Communications 28, 84–95 (1980)
Martinetz, T.M., Berkovich, S.G., Schulten, K.J.: ‘Neural-gas’ network for vector quantization and its application to time-series prediction. IEEE Trans. on Neural Networks 4(4), 558–569 (1993)
Principe, J.C., FisherIII, J.W., Xu, D.: Information theoretic learning. In: Haykin, S. (ed.) Unsupervised Adaptive Filtering, Wiley, New York (2000)
Qin, A., Suganthan, P.: A novel kernel prototype-based learning algorithm. In: Proc. of the 17th Internat. Conf. on Pattern Recognition, ICPR 2004, vol. 4, pp. 621–624 (2004)
Renyi, A.: On measures of entropy and information. In: Proc. of the 4th Berkeley Symp. on Mathematical Statistics and Probability. Univ. of California Press, Berkeley (1961)
Renyi, A.: Probability Theory. North-Holland Publish. Company, Amsterdam (1970)
Sato, A., Yamada, K.: Generalized learning vector quantization. In: Touretzky, D.S., Mozer, M.C., Hasselmo, M.E. (eds.) Proc. of the 1995 Conf. on Advances in Neural Information Processing Systems, vol. 8, pp. 423–429. MIT Press, Cambridge (1996)
Villmann, T., Claussen, J.-C.: Magnification control in self-organizing maps and neural gas. Neural Computation 18(2), 446–469 (2006)
Villmann, T., Haase, S.: Mathematical aspects of divergence based vector quantization using frchet-derivatives - extended and revised version. Machine Learning Reports 4(MLR-01-2010), 1–35 (2010), ISSN:1865-3960, http://www.uni-leipzig.de/~compint/mlr/mlr_01_2010.pdf
Villmann, T., Merényi, E., Hammer, B.: Neural maps in remote sensing image analysis. Neural Networks 16(3-4), 389–403 (2003)
Villmann, T., Schleif, F.-M.: Functional vector quantization by neural maps. In: Chanussot, J. (ed.) Proceedings of First Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS 2009), pp. 1–4. IEEE Press, Los Alamitos (2009)
Zador, P.L.: Asymptotic quantization error of continuous signals and the quantization dimension. IEEE Transaction on Information Theory (28), 149–159 (1982)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Villmann, T., Haase, S., Schleif, FM., Hammer, B. (2010). Divergence Based Online Learning in Vector Quantization. In: Rutkowski, L., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2010. Lecture Notes in Computer Science(), vol 6113. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-13208-7_60
Download citation
DOI: https://doi.org/10.1007/978-3-642-13208-7_60
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-13207-0
Online ISBN: 978-3-642-13208-7
eBook Packages: Computer ScienceComputer Science (R0)