Chapter

Discovery Science

Volume 6926 of the series Lecture Notes in Computer Science pp 247-261

A Statistical Model for Topically Segmented Documents

  • Giovanni PontiAffiliated withENEA - Portici Research Center
  • , Andrea TagarelliAffiliated withDepartment of Electronics, Computer and Systems Sciences, University of Calabria
  • , George KarypisAffiliated withDepartment of Computer Science & Engineering, Digital Technology Center, University of Minnesota

* Final gross prices may vary according to local VAT.

Get Access

Abstract

Generative models for text data are based on the idea that a document can be modeled as a mixture of topics, each of which is represented as a probability distribution over the terms. Such models have traditionally assumed that a document is an indivisible unit for the generative process, which may not be appropriate to handle documents with an explicit multi-topic structure. This paper presents a generative model that exploits a given decomposition of documents in smaller text blocks which are topically cohesive (segments). A new variable is introduced to model the within-document segments: using this variable at document-level, word generation is related not only to the topics but also to the segments, while the topic latent variable is directly associated to the segments, rather than to the document as a whole. Experimental results have shown that, compared to existing generative models, our proposed model provides better perplexity of language modeling and better support for effective clustering of documents.