BLISS: an Artificial Language for Learnability Studies

Abstract

To explore neurocognitive mechanisms underlying the human language faculty, cognitive scientists use artificial languages to control more precisely the language learning environment and to study selected aspects of natural languages. Artificial languages applied in cognitive studies are usually designed ad hoc, to only probe a specific hypothesis, and they include a miniature grammar and a very small vocabulary. The aim of the present study is the construction of an artificial language incorporating both syntax and semantics, BLISS. Of intermediate complexity, BLISS mimics natural languages by having a vocabulary, syntax, and some semantics, as defined by a degree of non-syntactic statistical dependence between words. We quantify, using information theoretical measures, dependencies between words in BLISS sentences as well as differences between the distinct models we introduce for semantics. While modeling English syntax in its basic version, BLISS can be easily varied in its internal parametric structure, thus allowing studies of the relative learnability of different parameter sets.

Appendix

Full Grammar of BLISS

All the grammar rules and the lexicon of BLISS are shown in Table 4.

Table 4 The full BLISS PCFG (probabilistic context-free grammar)

Proof of Eq. 9

$$ \begin{aligned} P_{{\rm post}} &= \displaystyle\sum_{h}P(h) P(w_j | h)\\ &=\displaystyle\sum_{h}P(h)\left( (1-g) P_{{\rm prior}}(w_j) + g P_s(w_j | h) \right)\\ &=\displaystyle\sum_{h_{ie}} P(h_{ie}) P_{{\rm prior}}(w_j)\\ &+\displaystyle\sum_{h_{e}} P(h_{e}) \left( (1-g) P_{{\rm prior}}(w_j) + g P_s(w_j | h_e) \right)\\ &=\left(\displaystyle\sum_{h_{ie}} P(h_{ie}) + (1-g) \displaystyle\sum_{h_{e}} P(h_{e})\right)P_{{\rm prior}}(w_j)\\ &+ g \displaystyle\sum_{h_{ie}} P(h_{ie}) P_s(w_j | h_{ie}) \end{aligned} $$

(10)

After analyzing the relation between prior and posterior probabilities of a word in our models (Eq. 9), we have generated corpora with the desired overall word frequencies (the word frequencies of the semantics models were adjusted to the ones of the No-Semantics). Note, however, that there are some constraints regarding the possibility of generating sentences with arbitrary word frequencies. In Eq. 9, the numerator cannot be negative, because the output is a probability measure. Therefore, the possible posterior probability of a word is constrained by the parameter g and pairwise statistics in P _s (w _j | h _e ) , which here has been derived from Shakespeare.

Length of BLISS Sentences

The probability distribution of the length of sentences generated by the BLISS grammar is shown in Fig.  10. To calculate mutual information values between words appearing in different positions of a sentence, we need to work with sentences of the same length. Considering the fact that the average length of the sentences is about 5 and also to obtain at least 5 distinct measures of triple mutual information values, we chose length 7. Thus, in this paper, for the results involving mutual information and triple mutual information, we used sentences of at least 7 words.

Fig. 10

Probability distribution of sentence lengths in a corpus produced by the Subject–Verb model. Other models with grammar show similar distributions

Size of Model-generated Corpora

A technical question to be addressed is how many sentences are needed, to generate sufficient statistics. To answer this question, we produced up to 40 million sentences, and calculated several of the measures used in this paper.

In Fig.  11, the mutual information between words in neighboring positions in a sentence, averaged over positions, I(n;n−1), was measured for each model and with corpora of different size: 1, 10, 20, and 40 million sentences of at least 7 words. As shown, having 10 million sentences is enough for capturing pairwise statistics in the corpora, regardless of having syntax or semantics.

Fig. 11

Average mutual information I(n;n−1), also averaged over all positions, versus number of sentences in the corpus. The mutual information between words is measured for each model and with corpora of different size: 1, 10, 20, and 40 million sentences of at least 7 words. As shown, 10 million sentences are enough for capturing pairwise statistics in the corpora, regardless of syntax or semantics

Figure 12 shows the result for the average triple mutual information among words, I(n;n−2,n−1), which needs the largest samples among all measures we have applied. As illustrated, increasing the size of the corpus from 20 to 40 million sentences does not appreciably change results for the models with syntax (Subject–Verb, Verb–Subject, Exponential, and No-Semantics), while we see some changes for the models without syntax, which need larger samples.

Fig. 12

Average triple mutual information I(n;n−1,n−2), again averaged also across positions, versus number of sentences in the corpus. The three-way mutual information needs the largest sample among all measures we applied but, as illustrated, increasing the size from 20 to 40 million sentences does not change it considerably, for the models with syntax (Subject–Verb, Verb–Subject, Exponential, and No-Semantics). We see still some change for the models without syntax, which obviously need larger corpora, due to their larger word-to-word variability

In sum, corpora with 20 million sentences, of at least 7 words each, were used for the calculation of the mutual information and triple mutual information in this study.

For the KL-divergence results, we also need sufficient word-pair statistics, like for mutual information values. Hence, we have used corpora of at least 20 million sentences for the KL-divergence calculation as well, but without the 7-word constraint.