An Efficient Minimum Vocabulary Construction Algorithm for Language Modeling

  • Sina Lin
  • Zengchang Qin
  • Zehua Huang
  • Tao Wan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7345)


In learning a new word by a dictionary, we first need to know a set of “basic words” which are frequently appeared in word definitions. It often happens that you cannot understand the word you looked up because there are still some words you do not understand in its definitions or explanations provided by the dictionary. You can keep looking up these new words recursively till they all can be well explained by some basic words you already knew. How to automatically find a minimum set of such basic words to define (or recursively define) the entire vocabulary in a given dictionary is what are going to discuss in this paper. We propose an efficient algorithm to construct the Minimum Vocabulary (MV) using the word frequency information. The minimum vocabulary can be used for language modeling and experimental results demonstrate the effectiveness of using the minimum vocabulary as features in text classification.


Language Modeling Word Frequency Basic Word Word Level Complex Word 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Armstrong, W.W.: Dependency structures of data base relationships. Information Processing 74, 580–583 (1974)MathSciNetGoogle Scholar
  2. 2.
    Cancho, R.F.I., Solé, R.V.: Least effort and the origins of scaling in human language. Proceedings of the National Academy of Science 100, 788–791 (2003)zbMATHCrossRefGoogle Scholar
  3. 3.
    Chandrasekharan, N., Sridhar, R., Iyengar, S.: On the minimum vocabulary problem. Journal of the American Society for Information Science 38(4), 234–238 (1987)CrossRefGoogle Scholar
  4. 4.
    Gabrilovich, E., Markovitch: Text categorization with many redundant features: Using aggressive feature selection to make SVMs competitive with C4.5. In: ICML (2004)Google Scholar
  5. 5.
    Hazenberg, S., Hulstun, J.H.: Defining a minimal receptive second-language vocabulary for non- native university students: An empirical investigation. Applied Linguistics 17(2), 145–163 (1996)CrossRefGoogle Scholar
  6. 6.
    Hirsh, D., Nation, P.: What vocabulary size is needed to read unsimplified texts for pleasure? Reading in a Foreign Language 8, 689–696 (1992)Google Scholar
  7. 7.
    Kilgarriff, A.: Putting frequencies in the dictionary. International Journal of Lexicography 10(2), 135–155 (1997)CrossRefGoogle Scholar
  8. 8.
    Kilgarriff, A.: Using word frequency lists to measure corpus homogeneity and similarity between corpora. In: ACL-SIGDAT Workshop on Very Large Corpora, pp. 231-245 (1997)Google Scholar
  9. 9.
    Loper, E., Bird, S.: NLTK: The natural language toolkit. In: Proceedings of the ACL 2002 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, vol. 1, pp. 63–70 (2002)Google Scholar
  10. 10.
    Qin, Z., Thint, M., Huang, Z.: Ranking Answers by Hierarchical Topic Models. In: Chien, B.-C., Hong, T.-P., Chen, S.-M., Ali, M. (eds.) IEA/AIE 2009. LNCS (LNAI), vol. 5579, pp. 103–112. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  11. 11.
    Ramakrishnan, R., Gehrke, J.: Database Management Systems. McGraw-Hill, Inc. (1999)Google Scholar
  12. 12.
    Saiedian, H., Spencer, T.: An efficient algorithm to compute the candidate keys of a relational database schema. The Computer Journal 39(2), 124–132 (1996)zbMATHCrossRefGoogle Scholar
  13. 13.
    West, M.P.: A General Service List of English Words. Longman (1976)Google Scholar
  14. 14.
    Zhao, Q., Qin, Z., Wan, T.: What Is the Basic Semantic Unit of Chinese Language? A Computational Approach Based on Topic Models. In: Kanazawa, M., Kornai, A., Kracht, M., Seki, H. (eds.) MOL 12. LNCS (LNAI), vol. 6878, pp. 143–157. Springer, Heidelberg (2011)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Sina Lin
    • 1
  • Zengchang Qin
    • 1
    • 3
  • Zehua Huang
    • 1
    • 2
  • Tao Wan
    • 4
  1. 1.Intelligent Computing and Machine Learning Lab, School of ASEEBeihang UniversityBeijingChina
  2. 2.School of Advanced EngineeringBeihang UniversityChina
  3. 3.Robotics InstituteCarnegie Mellon UniversityPittsburghUSA
  4. 4.School of MedicineBoston UniversityBostonUSA

Personalised recommendations