Skip to main content
SpringerLink
Log in

Entropy as an Indicator of Context Boundaries: An Experiment Using a Web Search Engine

  • Conference paper
Natural Language Processing – IJCNLP 2005 (IJCNLP 2005)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 3651))

Included in the following conference series:

Abstract

Previous works have suggested that the uncertainty of tokens coming after a sequence helps determine whether a given position is at a context boundary. This feature of language has been applied to unsupervised text segmentation and term extraction. In this paper, we fundamentally verify this feature. An experiment was performed using a web search engine, in order to clarify the extent to which this assumption holds. The verification was applied to Chinese and Japanese.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Kyoto University Text Corpus Version 3.0 (2003), http://www.kc.t.u-tokyo.ac.jp/nl-resource/corpus.html

  2. Ando, R.K., Lee, L.: Mostly-unsupervised statistical segmentation of japanese: Applications to kanji. In: ANLP-NAACL (2000)

    Google Scholar 

  3. Bell, T.C., Cleary, J.G., Witten, I.H.: Text Compression. Prentice-Hall, Englewood Cliffs (1990)

    Google Scholar 

  4. Creutz, M., Lagus, K.: Unsupervised discovery of morphemes. In: Workshop of the ACL Special Interest Group in Computational Phonology, pp. 21–30 (2002)

    Google Scholar 

  5. Frantzi, T.K., Ananiadou, S.: Extracting nested collocations. In: 16th COLING, pp. 41–46 (1996)

    Google Scholar 

  6. Harris, S.Z.: From phoneme to morpheme. Language, 190–222 (1955)

    Google Scholar 

  7. ICL. People daily corpus, beijing university, Institute of Computational Linguistics, Beijing University (1999), http://162.105.203.93/Introduction/~corpustagging.htm

  8. Kempe, A.: Experiments in unsupervised entropy-based corpus segmentation. In: Workshop of EACL in Computational Natural Language Learning, pp. 7–13 (1999)

    Google Scholar 

  9. Nakagawa, H., Mori, T.: A simple but powerful automatic termextraction method. In: Computerm2: 2nd International Workshop on Computational Terminology, pp. 29–35 (2002)

    Google Scholar 

  10. Nobesawa, S., Tsutsumi, J., Jang, D.S., Sano, T., Sato, K., Nakanishi, M.: Segmenting sentences into linky strings using d-bigram statistics. In: COLING, pp. 586–591 (1998)

    Google Scholar 

  11. Saffran, J.R.: Words in a sea of sounds: The output of statistical learning. Cognition 81, 149–169 (2001)

    Article  Google Scholar 

  12. Sun, M., Dayang, S., Tsou, B.K.: Chinese word segmentation without using lexicon and hand-crafted training data. In: COLING-ACL (1998)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graduate School of Information Science and Technology, University of Tokyo,  

    Kumiko Tanaka-Ishii

Authors
  1. Kumiko Tanaka-Ishii
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Center for Language Technology, Macquarie University, 2019, Sydney, NSW, Australia

    Robert Dale

  2. Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong

    Kam-Fai Wong

  3. Institute for Infocomm Research, 21, Heng Mui Keng Terrace, 119613, Singapore

    Jian Su

  4. Language Information Sciences Research Centre, City University of Hong Kong, Tat Chee Avenue, Kowloon, Hong Kong

    Oi Yee Kwong

Rights and permissions

Reprints and permissions

Copyright information

© 2005 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Tanaka-Ishii, K. (2005). Entropy as an Indicator of Context Boundaries: An Experiment Using a Web Search Engine. In: Dale, R., Wong, KF., Su, J., Kwong, O.Y. (eds) Natural Language Processing – IJCNLP 2005. IJCNLP 2005. Lecture Notes in Computer Science(), vol 3651. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11562214_9

Download citation

  • DOI: https://doi.org/10.1007/11562214_9

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-29172-5

  • Online ISBN: 978-3-540-31724-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation