Abstractive morphological learning with a recurrent neural network

Abstract

In traditional word-and-paradigm models of morphology, an inflectional system is represented via a set of exemplary paradigms. Novel wordforms are produced by analogy with previously encountered forms. This paper describes a recurrent neural network which can use this strategy to learn the paradigms of a morphologically complex language based on incomplete and randomized input. Results are given which show good performance for a range of typologically diverse languages.

Appendix

The network is structured to take as input a lexeme, a morphosyntactic feature set, and a partial wordform and to output a probability distribution over the next segment in the wordform. The input $x_{t}$ is a binary vector with as many dimensions as there are segments in the segment inventory of the language to be generated (plus designated start and end characters). The value of $x_{t}$ is one for the dimension corresponding to the previous character and zero for all other dimensions. This provides a localist, one-host representation of the immediate phonological context. The input $m_{t}$ is a localist input identifying a lexeme and a paradigm cell: one bit encodes the lexeme (say, walk) and another encodes the paradigm cell. Each combination of morphosyntactic features is given a unique representation: no generalizations are expressed at this level.

These inputs are mapped to a combined projection layer $z_{t}$ (Bengio et al. 2003):

$$\begin{aligned} z_{t}=\bigl({W^{x}}x_{t}+{b^{x}}\bigr) \oplus\bigl({W^{m}}m_{t}+{b^{m}}\bigr) \end{aligned}$$

where ⊕ is vector concatenation. The projection layer $z_{t}$ in turn is input for the recurrent layer, implemented via Long Short-Term Memory (LSTM) blocks (Hochreiter and Schmidhuber 1997; Jozefowicz et al. 2015). LSTMs avoid the problems with gradients exhibited by Elman-style simple recurrent networks and allow the model to more easily capture medium and long-distance temporal dependencies in the data (Hochreiter et al. 2001).¹⁴ The output of the recurrent layer $h_{t}$ is given by:

$$\begin{aligned} i =& \operatorname{\mbox{$\sigma$}}\bigl({W^{i}}z_{t}+{U^{i}}h_{t-1}+{b^{i}}\bigr)\\ f =& \operatorname{\mbox{$\sigma$}}\bigl({W^{f}}z_{t}+{U^{f}}h_{t-1}+{b^{f}}\bigr)\\ o =& \operatorname{\mbox{$\sigma$}}\bigl({W^{o}}z_{t}+{U^{o}}h_{t-1}+{b^{o}}\bigr)\\ c_{t} =& f\odot c_{t-1}+i\odot\tanh\bigl({W^{c}}z_{t}+{U^{c}}h_{t-1}+{b^{c}}\bigr)\\ h_{t} =& o\odot\tanh(c_{t}) \end{aligned}$$

where ⊙ denotes element-wise multiplication. For implementation purposes, the sigmoid function σ is evaluated using the ‘hard sigmoid’, a piecewise-linear approximation of the true sigmoid (Courbariaux et al. 2015):

$$\begin{aligned} \operatorname{\mbox{$\sigma$}}(x)=\max\biggl(0,\min\biggl(1,\frac{x}{5}+\frac{1}{2}\biggr)\biggr) \end{aligned}$$

Finally, $h_{t}$ is mapped to a vector with the same dimensionality as the input $x_{t}$ from which we can induce a probability distribution over output characters:

$$\begin{aligned} y_{t}={W^{y}}h_{t} \end{aligned}$$

The probability that the next character in the output $x_{t+1}$ is the j-th character in the character set is computed by applying the softmax function on the output layer:

$$\begin{aligned} p(x_{t+1}=j|x_{1}\ldots x_{t})=\frac{\operatorname{exp}(y^{j}_{t})}{\sum_{k}\operatorname{exp}(y^{k}_{t})} \end{aligned}$$

The probability of a wordform $p(x_{1}\ldots x_{n})$ given a lexeme and paradigm cell is the product of the probabilities of each character given the preceding context:

$$\begin{aligned} p(x_{1},\ldots,x_{n})=\prod_{t} p(x_{t}|x_{1}\ldots x_{t-1}) \end{aligned}$$

During training, the weights W and U and biases b are selected to maximize the log likelihood of the training data.