Evaluating Language Models Within a Predictive Framework: An Analysis of Ranking Distributions
Perplexity is a widely used criterion in order to compare language models without any task assumptions. However, the main drawback is that perplexity supposes probability distributions and hence cannot compare heterogeneous models. As an evaluation framework, we propose in this article to abandon perplexity and to extend the Shannon’s entropy idea which is based on model prediction performance using rank based statistics. Our methodology is able to predict joint word sequences being independent of the task or model assumptions. Experiments are carried out on the English language with different kind of language models. We show that long-term prediction language models are not more effective than the standard n-gram models. Ranking distributions follow exponential laws as already observed in predicting letter sequences. These distributions show a second mode not observed with letters and we propose to give some interpretation to this mode in this article.
KeywordsAverage Rank Maximum Rank Good Path Training Corpus Word Sequence
Unable to display preview. Download preview PDF.
- 4.Deligne, S., Bimbot, F.: Language modeling by variable length sequences: theoretical formulation and evaluation of multigrams. In: IEEE International Conference on Acoustics and Speech Signal Processing, pp. 169–172 (1995)Google Scholar
- 6.Garside, R., Geoffrey, L., Geoffrey, S.: The computational analysis of english. a corpus-based approach. Longman, London (1987)Google Scholar