Neural Processing Letters

, Volume 18, Issue 3, pp 163–204 | Cite as

Information-Theoretic Competitive Learning with Inverse Euclidean Distance Output Units

  • Ryotaro Kamimura


In this paper, we propose a new information theoretic competitive learning method. We first construct a learning method in single-layered networks, and then we extend it to supervised multi-layered networks. Competitive unit outputs are computed by the inverse of Euclidean distance between input patterns and connection weights. As distance is smaller, competitive unit outputs are stronger. In realizing competition, neither the winner-take-all algorithm nor the lateral inhibition is used. Instead, the new method is based upon mutual information maximization between input patterns and competitive units. In maximizing mutual information, the entropy of competitive units is increased as much as possible. This means that all competitive units must equally be used in our framework. Thus, no under-utilized neurons or dead neurons are generated. When using multi-layered networks, we can improve noise-tolerance performance by unifying information maximization and minimization. We applied our method with single-layered networks to a simple artificial data problem and an actual road classification problem. In both cases, experimental results confirmed that the new method can produce the final solutions almost independently of initial conditions, and classification performance is significantly improved. Then, we used multi-layered networks, and applied them to a character recognition problem and a political data analysis. In these problem, we could show that noise-tolerance performance was improved by decreasing information content on input patterns to certain points.

competitive learning dead neurons inverse Euclidean distance mutual information maximization winner-take-all 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Linsker, R.: Self-organization in a perceptual network, Computer 21 (1988), 105–117.Google Scholar
  2. 2.
    Linsker, R.: How to generate ordered maps by maximizing the mutual information between input and output, Neural Computation 1 (1989), 402–411.Google Scholar
  3. 3.
    Atick, J. J. and Redlich, A. N.: Toward a theory of early visual processing, Neural Computation 2 (1990), 308–320.Google Scholar
  4. 4.
    Becker, S.: Mutual information maximization: models of cortical self-organization, Network: Computation in Neural Systems 7 (1996), 7–31.Google Scholar
  5. 5.
    Becker, S. and Hinton, G. E.: Learning mixture models of spatial coherence, Neural Computation 5 (1993), 267–277.Google Scholar
  6. 6.
    Kamimura, R. and Nakanishi, S.: Hidden information maximization for feature detection and rule discovery, Network 6 (1995), 577–622.Google Scholar
  7. 7.
    Kamimura, R. and Nakanishi, S.: Improving generalization performance by information minimization, IEICE Transactions on Information and Systems E78-D(2) (1995), 163–173.Google Scholar
  8. 8.
    Deco, G., Finnof, W. and Zimmermann, H. G.: Unsupervised mutual information criterion for elimination of overtraining in supervised multiplayer networks, Neural Computation 7 (1995), 86–107.Google Scholar
  9. 9.
    Akiyama, Y. and Furuya, T.: An extension of the back-propagation learning which performs entropy maximization as well as error minimization, Tech. Rep. NC91-6, IEICE Technical Report, 1991.Google Scholar
  10. 10.
    Kamimura, R., Kamimura, T. and Shultz, T. R.: Information theoretic competitive learning and linguistic rule acquistion, Transactions of the Japanese Society for Artificial Intelligence 16(2) (2001), 287–298.Google Scholar
  11. 11.
    Kamimura, R., Kamimura, T. and Uchida, O.: Flexible feature discovery and structural information, Connection Science 13(4) (2001), 323–347.Google Scholar
  12. 12.
    Kamimura, R., Kamimura, T. and Takeuchi, H.: Greedy information acquisition algorithm: A new information theoretic approach to dynamic information acquisition in neural networks, Connection Science 14(2) (2002), 137–162.Google Scholar
  13. 13.
    Kamimura, R.: Progressive feature extraction by greedy network-growing algorithm, to appear in Complex Systems, 2003.Google Scholar
  14. 14.
    Rumelhart, D. E. and Zipser, D.: Feature discovery by competitive learning, In: D. E. Rumelhart and G. E. H. et al. (eds), Parallel Distributed Processing, Cambridge: MIT Press, vol. 1, pp. 151–193, 1986.Google Scholar
  15. 15.
    Grossberg, S.: Competitive learning from interactive activation to adaptive resonance, Cognitive Science 11 (1987), 23–63.Google Scholar
  16. 16.
    DeSieno, D.: Adding a conscience to competitive learning, In: Proceedings of IEEE International Conference on Neural Networks, pp. 117–124, San Diego, IEEE, 1988.Google Scholar
  17. 17.
    Ahalt, S. C., Krishnamurthy, A. K., Chen, P. and Melton, D. E.: Competitive learning algorithms for vector quantization, Neural Networks 3 (1990), 277–290.Google Scholar
  18. 18.
    Xu, L.: Rival penalized competitive learning for clustering analysis, RBF net, and curve detection, IEEE Transaction on Neural Networks 4(4) (1993), 636–649.Google Scholar
  19. 19.
    Luk, A. and Lien, S.: Properties of the generalized lotto-type competitive learning, In: Proceedings of International conference on neural information processing, pp. 1180–1185, San Mateo: CA, Morgan Kaufmann Publishers, 2000.Google Scholar
  20. 20.
    Hulle, M. M. V.: The formation of topographic maps that maximize the average mutual information of the output responses to noiseless input signals, Neural Computation 9(3) (1997), 595–606.Google Scholar
  21. 21.
    Gatlin, L. L.: Information Theory and Living Systems, Columbia University Press, 1972.Google Scholar
  22. 22.
    Hecht-Nielsen, R.: Neurocomputing, Reading, MA: Addison-Wesley, 1990.Google Scholar
  23. 23.
    McClelland, C. A. and Hoggard, G. D.: Confliict patterns in the interactions among nations, In: R. Hayes (ed.), International Politics and Foreign Policy, p. 714, New York, Free Press, 1969.Google Scholar
  24. 24.
    Kamimura, R., Kamimura, T. and Shultz, T. R.: Self-organization by information control, In: Proceedings of IASTED International Conference on Applied Informatics, pp. 188–192, Anaheim, CA, ACTA Press, 2001.Google Scholar

Copyright information

© Kluwer Academic Publishers 2003

Authors and Affiliations

  • Ryotaro Kamimura
    • 1
  1. 1.Information Science Laboratory and Future Science and Technology Joint Research CenterTokai UniversityKanagawaJapan

Personalised recommendations