Entropy as an Indicator of Context Boundaries: An Experiment Using a Web Search Engine

  • Kumiko Tanaka-Ishii
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3651)


Previous works have suggested that the uncertainty of tokens coming after a sequence helps determine whether a given position is at a context boundary. This feature of language has been applied to unsupervised text segmentation and term extraction. In this paper, we fundamentally verify this feature. An experiment was performed using a web search engine, in order to clarify the extent to which this assumption holds. The verification was applied to Chinese and Japanese.


Search Engine Boundary Detection Indexing Strategy Word Boundary Word Segmentation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Kyoto University Text Corpus Version 3.0 (2003),
  2. 2.
    Ando, R.K., Lee, L.: Mostly-unsupervised statistical segmentation of japanese: Applications to kanji. In: ANLP-NAACL (2000)Google Scholar
  3. 3.
    Bell, T.C., Cleary, J.G., Witten, I.H.: Text Compression. Prentice-Hall, Englewood Cliffs (1990)Google Scholar
  4. 4.
    Creutz, M., Lagus, K.: Unsupervised discovery of morphemes. In: Workshop of the ACL Special Interest Group in Computational Phonology, pp. 21–30 (2002)Google Scholar
  5. 5.
    Frantzi, T.K., Ananiadou, S.: Extracting nested collocations. In: 16th COLING, pp. 41–46 (1996)Google Scholar
  6. 6.
    Harris, S.Z.: From phoneme to morpheme. Language, 190–222 (1955)Google Scholar
  7. 7.
    ICL. People daily corpus, beijing university, Institute of Computational Linguistics, Beijing University (1999),
  8. 8.
    Kempe, A.: Experiments in unsupervised entropy-based corpus segmentation. In: Workshop of EACL in Computational Natural Language Learning, pp. 7–13 (1999)Google Scholar
  9. 9.
    Nakagawa, H., Mori, T.: A simple but powerful automatic termextraction method. In: Computerm2: 2nd International Workshop on Computational Terminology, pp. 29–35 (2002)Google Scholar
  10. 10.
    Nobesawa, S., Tsutsumi, J., Jang, D.S., Sano, T., Sato, K., Nakanishi, M.: Segmenting sentences into linky strings using d-bigram statistics. In: COLING, pp. 586–591 (1998)Google Scholar
  11. 11.
    Saffran, J.R.: Words in a sea of sounds: The output of statistical learning. Cognition 81, 149–169 (2001)CrossRefGoogle Scholar
  12. 12.
    Sun, M., Dayang, S., Tsou, B.K.: Chinese word segmentation without using lexicon and hand-crafted training data. In: COLING-ACL (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Kumiko Tanaka-Ishii
    • 1
  1. 1.Graduate School of Information Science and TechnologyUniversity of Tokyo 

Personalised recommendations