SOM and Neural Gas as Graduated Nonconvexity Algorithms

  • Ana I. González
  • Alicia D’Anjou
  • M. Teresa García-Sebastian
  • Manuel Graña
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3982)


Convergence of the Self-Organizing Map (SOM) and Neural Gas (NG) is usually contemplated from the point of view of stochastic gradient descent. This class of algorithms is characterized by a very slow convergence rate. However we have found empirically that One-Pass realizations of SOM and NG provide good results or even improve over the slower realizations, when the performance measure is the distortion. One-Pass realizations use each data sample item only once, imposing a very fast reduction of the learning parameters that does not conform to the convergence requirements of stochastic gradient descent. That empirical evidence leads us to propose that the appropriate setting for the convergence analysis of SOM, NG and similar competitive clustering algorithms is the field of Graduated Nonconvexity algorithms. We show they can easily be put in this framework.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ahalt, S.C., Krishnamurthy, A.K., Chen, P., Melton, D.E.: Competitive Learning Algorithms for Vector Quantization. Neural Networks 3, 277–290 (1990)CrossRefGoogle Scholar
  2. 2.
    Allgower, E.L., Georg, K.: Numerical Continuation Methods. An Introduction. Springer Series in Computational Mathematics, vol. 13. Springer, Heidelberg (1990)zbMATHGoogle Scholar
  3. 3.
    Bodt, E., Cottrell, M., Letremy, P., Verleysen, M.: On the use of self-organizing maps to accelerate vector quantization. Neurocomputing 56, 187–203 (2004)CrossRefGoogle Scholar
  4. 4.
    Chan, C., Vetterli, M.: Lossy Compression of Individual Signals Based on String Matching and One-Pass Codebook Design. In: ICASSP 1995, Detroit, MI (1995)Google Scholar
  5. 5.
    Chinrungrueng, C., Séquin, C.: Optimal Adaptive K-Means Algorithm with Dynamic Adjustment of Learning Rate. IEEE Trans. on Neural Networks 6(1), 157–169 (1995)CrossRefGoogle Scholar
  6. 6.
    Fort, J.C., Letrémy, P., Cottrell, M.: Advantages and Drawbacks of the Batch Kohonen Algorithm. In: Verleysen, M. (ed.) Proc. of ESANN’2002, Brugge, pp. 223–230. Editions D Facto, Bruxelles (2002)Google Scholar
  7. 7.
    Fritzke, B.: The LBG-U method for vector quantization - an improvement over LBG inspired from neural networks. Neural Processing Letters 5(1), 35–45 (1997)CrossRefGoogle Scholar
  8. 8.
    Fukunaga, K.: Statistical Pattern Recognition. Academic Press, London (1990)zbMATHGoogle Scholar
  9. 9.
    Gersho, A.: On the structure of vector quantizers. IEEE Trans. Inf. Th. 28(2), 157–166 (1982)CrossRefMathSciNetGoogle Scholar
  10. 10.
    Gersho, A., Gray, R.M.: Vector Quantization and signal compression. Kluwer, Dordrecht (1992)zbMATHGoogle Scholar
  11. 11.
    Gonzalez, A.I., Graña, M., d’Anjou, A., Albizuri, F.X.: A near real-time evolutive strategy for adaptive Color Quantization of image sequences. In: Joint Conference Information Sciences, vol. 1, pp. 69–72 (1997)Google Scholar
  12. 12.
    Gray, R.M.: Vector Quantization. IEEE ASSP 1, 4–29 (1984)CrossRefGoogle Scholar
  13. 13.
    Hofmann, T., Buhmann, J.M.: Competitive Learning Algorithms for Robust Vector Quantization. IEEE Trans. Signal Processing 46(6), 1665–1675 (1998)zbMATHCrossRefMathSciNetGoogle Scholar
  14. 14.
    Kohonen, T.: Self-Organization and associative memory, (1988 2nd edn). Springer, Heidelberg (1984)zbMATHGoogle Scholar
  15. 15.
    Kohonen, T.: The self-organising map. Neurocomputing 21, 1–6 (1998)zbMATHCrossRefGoogle Scholar
  16. 16.
    Linde, Y., Buzo, A., Gray, R.M.: An algorithm for vector quantizer design. IEEE Trans. Comm. 28, 84–95 (1980)CrossRefGoogle Scholar
  17. 17.
    Martinetz, T., Berkovich, S., Schulten, K.: Neural-Gas network for vector quantization and his application to time series prediction. IEEE trans. Neural Networks 4(4), 558–569 (1993)CrossRefGoogle Scholar
  18. 18.
    Zhong, S., Ghosh, J.: A Unified Framework for Model-based Clustering. Journal of Machine Learning Research 4, 1001–1037 (2003)CrossRefMathSciNetGoogle Scholar
  19. 19.
    Blake, A., Zisserman, A.: Visual Reconstruction. MIT Press, Cambridge (1987)Google Scholar
  20. 20.
    Nielsen, M.: Graduated nonconvexity by functional focusing. IEEE Trans. Patt. Anal. Mach. Int. 19(5), 521–525 (1997)CrossRefGoogle Scholar
  21. 21.
    Gonzalez, A.I., Graña, M.: Controversial empirical results on batch versus one pass online algorithms. In: Proc. WSOM2005, Paris, France, September 2005, pp. 405–411 (2005)Google Scholar
  22. 22.
    Fort, J.C., Pagès, G.: About the Kohonen algorithm: strong or weak Self-organization? Neural Networks 9(5), 773–785 (1996)CrossRefGoogle Scholar
  23. 23.
    Gualtieri, J.A., Chettri, S.: Support vector machines for classification of hyperspectral data. In: Proc. Geosci. Rem. Sens. Symp., 2000, IGARSS, vol. 2, pp. 813–815 (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Ana I. González
    • 1
  • Alicia D’Anjou
    • 1
  • M. Teresa García-Sebastian
    • 1
  • Manuel Graña
    • 1
  1. 1.Grupo de Inteligencia Computacional, Facultad de InformáticaUPV/EHUSan SebastiánEspaña

Personalised recommendations