An Efficient Minimum Vocabulary Construction Algorithm for Language Modeling
In learning a new word by a dictionary, we first need to know a set of “basic words” which are frequently appeared in word definitions. It often happens that you cannot understand the word you looked up because there are still some words you do not understand in its definitions or explanations provided by the dictionary. You can keep looking up these new words recursively till they all can be well explained by some basic words you already knew. How to automatically find a minimum set of such basic words to define (or recursively define) the entire vocabulary in a given dictionary is what are going to discuss in this paper. We propose an efficient algorithm to construct the Minimum Vocabulary (MV) using the word frequency information. The minimum vocabulary can be used for language modeling and experimental results demonstrate the effectiveness of using the minimum vocabulary as features in text classification.
KeywordsLanguage Modeling Word Frequency Basic Word Word Level Complex Word
Unable to display preview. Download preview PDF.
- 4.Gabrilovich, E., Markovitch: Text categorization with many redundant features: Using aggressive feature selection to make SVMs competitive with C4.5. In: ICML (2004)Google Scholar
- 6.Hirsh, D., Nation, P.: What vocabulary size is needed to read unsimplified texts for pleasure? Reading in a Foreign Language 8, 689–696 (1992)Google Scholar
- 8.Kilgarriff, A.: Using word frequency lists to measure corpus homogeneity and similarity between corpora. In: ACL-SIGDAT Workshop on Very Large Corpora, pp. 231-245 (1997)Google Scholar
- 9.Loper, E., Bird, S.: NLTK: The natural language toolkit. In: Proceedings of the ACL 2002 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, vol. 1, pp. 63–70 (2002)Google Scholar
- 11.Ramakrishnan, R., Gehrke, J.: Database Management Systems. McGraw-Hill, Inc. (1999)Google Scholar
- 13.West, M.P.: A General Service List of English Words. Longman (1976)Google Scholar