Selection criteria for word trigger pairs in language modeling

Abstract

In this paper, we study selection criteria for the use of word trigger pairs in statistical language modeling. A word trigger pair is defined as a long-distance word pair. To select the most significant trigger pairs, we need suitable criteria which are the topics of this paper. We extend a baseline language model by a single word trigger pair and use the perplexity of this extended language model as selection criterion. This extension is applied to all possible trigger pairs, the number of which is the square of the vocabulary size. When a unigram language model is applied as baseline model, this approach produces the mutual information criterion used in [7, 11]. The more interesting case is to use this criterion in the context of a more powerful model such as a bigram/trigram model with a cache. We study different variants of including word trigger pairs into such a language model. This approach produced better word trigger pairs than the conventional mutual information criterion. When used on the Wall Street Journal corpus, the trigger pairs selected reduced the perplexity of a trigram/cache language model from 138 to 128 for a 5-million word training set and from 92 to 87 for a 38-million word training set.