Statistical Substring Reduction in Linear Time

  • Xueqiang Lü
  • Le Zhang
  • Junfeng Hu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3248)


We study the problem of efficiently removing equal frequency n-gram substrings from an n-gram set, formally called Statistical Substring Reduction (SSR). SSR is a useful operation in corpus based multi-word unit research and new word identification task of oriental language processing. We present a new SSR algorithm that has linear time (O(n)) complexity, and prove its equivalence with the traditional O(n 2) algorithm. In particular, using experimental results from several corpora with different sizes, we show that it is possible to achieve performance close to that theoretically predicated for this task. Even in a small corpus the new algorithm is several orders of magnitude faster than the O(n 2) one. These results show that our algorithm is reliable and efficient, and is therefore an appropriate choice for large scale corpus processing.


Statistical Unit Statistical String Extraction Task Radix Sort Oriental Language 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Chang, J.S.: Automatic Lexicon Acquisition and Precision-RecallMaximization for Untagged Text Corpora. PhD thesis, National Tsing-Hua University, National Tsing-Hua University Hsinchu, Taiwan 300, ROC. (1997) Google Scholar
  2. 2.
    Merkel, M., Andersson, M.: Knowledge-lite extraction of multi-word units with language filters and entropy thresholds. In: Proceedings of 2000 Conference on User-Oriented Content-Based Text and Image Handling, Paris, France, pp. 737–746 (2000)Google Scholar
  3. 3.
    Moon, K., Lee, J.H.: Translation of discontinuous multi-word translation units in a koreanto- japanese machine translation system. International Journal of Computer Processing of Oriental Languages 15, 79–99 (2002)CrossRefGoogle Scholar
  4. 4.
    Nagao, M., Mori, S.: A new method of n-gram statistics for large number of n and automatic extraction of words and phrases from large text data of japanese. In: The 15th International Conference on Computational Linguistics, vol. 1, pp. 611–615 (1994)Google Scholar
  5. 5.
    Han, K., Wang, Y., Chen, G.: Research on fast high-frequency extracting and statistics algorithm with no thesaurs. Journal of Chinese Information Processing 15, 23–30 (2001) (in Chinese)Google Scholar
  6. 6.
    Zhang, L., Lü, X., Shen, Y., Yao, T.: A statistical approach to extract chinese chunk candidates from large corpora. In: Proceeding of 20th International Conference on Computer Processing of Oriental Languages (ICCPOL 2003), pp. 109–117 (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Xueqiang Lü
    • 1
  • Le Zhang
    • 2
  • Junfeng Hu
    • 1
  1. 1.Institute of Computational LinguisticsPeking UniversityBeijing
  2. 2.Institute of Computer Software & TheoryNortheastern UniversityShenyang

Personalised recommendations