High-order Neural Networks and Networks with Composite Key Patterns

Abstract

The development of neural networks of order higher than one was approached independently by a number of research groups. Rumelhart et al examined high-order neural networks by introducing new types of units, different than the conventional ones, known as sigmapi units (Rumelhart and McClelland, 1986). Motivated by the relationship between first-order neural networks and linear discriminant functions, other groups of researchers emphasized the connection between neural networks of order higher than one and nonlinear discriminant functions (Chen et al., 1986; Psaltis and Park, 1986; Psaltis et al., 1988). Giles et al. used high-order recurrent neural networks to infer regular grammars from positive and negative strings of training samples (Giles et al., 1990; Giles et al., 1991). This chapter focuses on the architecture, training, and properties of neural networks of order higher than one (Karayiannis, 1991a; Karayiannis and Venetsanopoulos, 1992e). It is shown that highorder neural networks of any order can be trained by any of the learning algorithms developed for first-order neural networks. This chapter also evaluates the efficiency of the outer-product rule when applied to the training of neural networks of order higher than one. The investigation of the architecture of high-order neural networks leads to the development of neural networks with composite key patterns that are the essential generalization of high-order neural networks. The training, performance, and potential applications of neural networks with composite key patterns are also investigated (Karayiannis, 1991a; Karayiannis and Venetsanopoulos, 1992e).