An Iterative Approach to Text Segmentation

  • Fei Song
  • William M. Darling
  • Adnan Duric
  • Fred W. Kroon
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6611)


We present divSeg, a novel method for text segmentation that iteratively splits a portion of text at its weakest point in terms of the connectivity strength between two adjacent parts. To search for the weakest point, we apply two different measures: one is based on language modeling of text segmentation and the other, on the interconnectivity between two segments. Our solution produces a deep and narrow binary tree – a dynamic object that describes the structure of a text and that is fully adaptable to a user’s segmentation needs. We treat it as a separate task to flatten the tree into a broad and shallow hierarchy either through supervised learning of a document set or explicit input of how a text should be segmented. The rich structure of our created tree further allows us to segment documents at varying levels such as topic, sub-topic, etc. We evaluated our new solution on a set of 265 articles from Discover magazine where the topic structures are unknown and need to be discovered. Our experimental results show that the iterative approach has the potential to generate better segmentation results than several leading baselines, and the separate flattening step allows us to adapt the results to different levels of details and user preferences.


Text Segmentation Language Modeling 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Beeferman, D., Berger, A., Lafferty, J.: Statistical models for text segmentation. Mach. Learn. 34(1-3), 177–210 (1999)CrossRefzbMATHGoogle Scholar
  2. 2.
    Choi, F.Y.Y.: Advances in domain independent linear text segmentation. In: Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, pp. 26–33. Morgan Kaufmann Publishers, San Francisco (2000)Google Scholar
  3. 3.
    Cieri, C., Graff, D., Liberman, M., Martey, N., Strassel, S.: Large, multilingual, broadcast news corpora for cooperative research in topic detection and tracking: The tdt-2 and tdt-3 corpus efforts. In: Proceedings of Language Resources and Evaluation Conference (2000)Google Scholar
  4. 4.
    Eisenstein, J.: Hierarchical text segmentation from multi-scale lexical cohesion. In: NAACL 2009: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 353–361. Association for Computational Linguistics, Morristown (2009)Google Scholar
  5. 5.
    Eisenstein, J., Barzilay, R.: Bayesian unsupervised topic segmentation. In: EMNLP 2008: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 334–343. Association for Computational Linguistics, Morristown (2008)CrossRefGoogle Scholar
  6. 6.
    Halliday, M.A.K., Hasan, R.: Cohesion in English (English Language). Longman Pub. Group, Harlow (1976)Google Scholar
  7. 7.
    Hearst, M.A.: Multi-paragraph segmentation of expository text. In: Proceedings of the 32nd Annual Meeting on Association for Computational Linguistics, pp. 9–16. Association for Computational Linguistics, Morristown (1994)CrossRefGoogle Scholar
  8. 8.
    Pevzner, L., Hearst, M.A.: A critique and improvement of an evaluation metric for text segmentation. Comput. Linguist. 28(1), 19–36 (2002)CrossRefGoogle Scholar
  9. 9.
    Reynar, J.C.: Topic Segmentation: Algorithms and Applications. PhD thesis, University of Pennsylvania (1998)Google Scholar
  10. 10.
    Skorochod’ko, E.F.: Adaptive method of automatic abstracting and indexing. In: Proceedings of the IFIP, vol. 71, pp. 1179–1182 (1972)Google Scholar
  11. 11.
    Utiyama, M., Isahara, H.: A statistical model for domain-independent text segmentation. In: ACL 2001: Proceedings of the 39th Annual Meeting on Association for Computational Linguistics, pp. 499–506. Association for Computational Linguistics, Morristown (2001)Google Scholar
  12. 12.
    Ye, N., Zhu, J., Zheng, Y., Ma, M.Y., Wang, H., Zhang, B.: A dynamic programming model for text segmentation based on min-max similarity. In: Li, H., Liu, T., Ma, W.-Y., Sakai, T., Wong, K.-F., Zhou, G. (eds.) AIRS 2008. LNCS, vol. 4993, pp. 141–152. Springer, Heidelberg (2008)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Fei Song
    • 1
  • William M. Darling
    • 1
  • Adnan Duric
    • 1
  • Fred W. Kroon
    • 2
  1. 1.School of Computer ScienceUniversity of GuelphGuelphCanada
  2. 2.PryLynx CorporationKitchenerCanada

Personalised recommendations