Stochastic Neurons

Abstract

We now consider a simple generalization of the neural networks discussed in the previous chapter which permits a more powerful theoretical treatment. For this purpose we replace the deterministic evolution law (3.5) for the neural activity by a stochastic law, which does not assign a definite value to s _i (t + 1), but only gives the probabilities that s _i (t + 1) takes one of the values +1 or -1. We request that the value s _i (t + 1) = ±1 will occur with probability f(±h _i ): where the activation function f(h) must have the proper limiting values f(h→ - ∞) = 0, f(h → +∞) = 1. Between these limits the activation function must rise monotonously, smoothly interpolating between 0 and 1. Such functions are often called sigmoidal functions. A standard choice, depicted in Fig. 4.1, is given by [Li74, Li78] which satisfies the condition f(h) + f(-h) = 1. Among physicists this function is commonly called the Fermi function, because it describes the thermal energy distribution in a system of identical fermions. In this case the parameter β has the meaning of an inverse temperature, T = β^-1. We shall also use this nomenclature in connection with neural networks, although this does not imply that the parameter β^-1 should denote a physical temperature at which the network operates. The rule (4.3) should rather be considered as a model for a stochastically operating network which has been conveniently designed to permit application of the powerful formal methods of statistical physics.¹