Associative Memory Networks and Sparse Similarity Preserving Codes

Abstract

This paper addresses the following main topics:

• Definition of the associative memory tasks, auto-association and hetero-association.

• Use of neural networks and local learning rules for the realization of associative memory.

• Derivation of the information capacity as an evaluation criterion.

• Comparision and optimization of local learning rules for associative memory.

• Sparse, distributed, similarity preserving data representation.

For auto-association and hetero-association the most plausible implementation of an associative memory by means of a neural network with modifiable connections is presented.

The evaluation of the memory’s performance in storing and retrieving information is discussed and the information storage capacity is defined. With respect to this criterion one can optimize the local rules for synaptic modification and the statistical format of the patterns to be stored.

The result is that the patterns should be sparse (i.e. they should contain mostly zeros) and the learning rule should be Hebbian. In the optimal case one can achieve a storage capacity of about 0.7 bit/synapse in hetero-association and 0.35 bit/synapse in auto-association. The second value can not be reached with realistic iterative retrieval procedures, but in practice values of 0.18 bit/synapse can be achieved for auto-association and above 0.6 bit/synapse for hetero-association. We derive some of these results mathematically and present some simulation studies illustrating the asymptotic mathematical calculations.

Finally we present some basic ideas on the construction of sparse, distributed, similarity preserving representation of data.