Statistical Substring Reduction in Linear Time
We study the problem of efficiently removing equal frequency n-gram substrings from an n-gram set, formally called Statistical Substring Reduction (SSR). SSR is a useful operation in corpus based multi-word unit research and new word identification task of oriental language processing. We present a new SSR algorithm that has linear time (O(n)) complexity, and prove its equivalence with the traditional O(n 2) algorithm. In particular, using experimental results from several corpora with different sizes, we show that it is possible to achieve performance close to that theoretically predicated for this task. Even in a small corpus the new algorithm is several orders of magnitude faster than the O(n 2) one. These results show that our algorithm is reliable and efficient, and is therefore an appropriate choice for large scale corpus processing.
KeywordsStatistical Unit Statistical String Extraction Task Radix Sort Oriental Language
Unable to display preview. Download preview PDF.
- 1.Chang, J.S.: Automatic Lexicon Acquisition and Precision-RecallMaximization for Untagged Text Corpora. PhD thesis, National Tsing-Hua University, National Tsing-Hua University Hsinchu, Taiwan 300, ROC. (1997) Google Scholar
- 2.Merkel, M., Andersson, M.: Knowledge-lite extraction of multi-word units with language filters and entropy thresholds. In: Proceedings of 2000 Conference on User-Oriented Content-Based Text and Image Handling, Paris, France, pp. 737–746 (2000)Google Scholar
- 4.Nagao, M., Mori, S.: A new method of n-gram statistics for large number of n and automatic extraction of words and phrases from large text data of japanese. In: The 15th International Conference on Computational Linguistics, vol. 1, pp. 611–615 (1994)Google Scholar
- 5.Han, K., Wang, Y., Chen, G.: Research on fast high-frequency extracting and statistics algorithm with no thesaurs. Journal of Chinese Information Processing 15, 23–30 (2001) (in Chinese)Google Scholar
- 6.Zhang, L., Lü, X., Shen, Y., Yao, T.: A statistical approach to extract chinese chunk candidates from large corpora. In: Proceeding of 20th International Conference on Computer Processing of Oriental Languages (ICCPOL 2003), pp. 109–117 (2003)Google Scholar