Stochastic Neurons

Abstract

We now consider a simple generalization of the neural networks discussed in the previous chapter which permits a more powerful theoretical treatment. For this purpose we replace the deterministic evolution law (3.5)

$$ {s_i}\left( {t\,\, + \,\,1} \right)\,\, = \,\,{\mathop{\rm sgn}} \left[ {{h_i}\left( t \right)} \right]\,\, \equiv \,\,{\mathop{\rm sgn}} \,\left[ {\sum\limits_{j\, = \,1}^N {{w_{ij}}{s_j}\left( t \right)} } \right] $$

(4.1)

for the neural activity by a stochastic law, which does not assign a definite value to s _i (t + 1), but only gives the probabilities that s _i (t + 1) takes one of the values +1 or -1. We request that the value s _i (t + 1) = ±1 will occur with probability f(±h _i ):

$$ \Pr \left[ {{s_i}\left( {t\,\, + \,\,1} \right)\, = \,\,1} \right]\,\, = \,\,f\left[ {{h_i}\left( t \right)} \right], $$

(4.2)

where the activation function f(h) must have the limiting values f(h → −∞) = 0, f(h → −∞) = 1. Between these limits the activation function must rise monotonously, smoothly interpolating between 0 and 1. Such functions are often called sigmoidal functions.