A Comparison of the Computational Power of Sigmoid and Boolean Threshold Circuits

Abstract

Research on neural networks has led to the investigation of massively parallel computational models that consist of analog computational elements. Usually these analog computational elements are assumed to be smooth threshold gates, i.e. γ-gates for some nondecreasing differentiate function γ: ℝ → ℝ. A γ-gate with weights w ₁,…, w _m ∈ ℝ; and threshold t ∈ ℝ is defined to be a gate that computes the function (x₁,...,x_m) $\mapsto \gamma \left( {\sum\nolimits_{i = 1}^m {{w_i}{x_i} - t} } \right)$ from ℝ^m into ℝ. A γ-circuit is defined as a directed acyclic circuit that consists of γ-gates. The most frequently considered special case of a smooth threshold circuit is the sigmoid threshold circuit, which is a σ-circuit for σ ℝ → ℝ defined by $\sigma (x) = \frac{1}{{1 + \exp ( - x)}}$