Abstract
The colossal growth of volatile online text data evokes the demand for automatic text analysis tools to identify worthwhile information. Documents, as well as text streams, can be structured beyond the concept of frequency distributions.
Here we introduce a novel method that provides a relative measure for information value over a time series that is mapped by a dynamic trie structure. We adapt the concept of entropy for textual data and employ a compression-based estimation method. The algorithm can perform in a real-time scenario because of its linear complexity and since it is based on a dynamic history of predefined size.
We show the suitability of our method with an experimental dataset and compare our results to an existing approach. Our results reveal structural properties of the texts and permit for deeper analysis of the presumably information peaks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Amati, G., Van Rijsbergen, C.J.: Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Trans. Inf. Syst. 20, 357–389 (2002)
Balinsky, A.A., Balinsky, H.Y., Simske, S.J.: On helmholtz’s principle for documents processing. In: Proc. of the 10th ACM Symp. on Document Engineering, DocEng 2010, p. 283. ACM, New York (2010)
Bohne, T., Borghoff, U.M.: Data fusion: Boosting performance in keyword extraction. In: 2013 20th IEEE International Conference and Workshops on Engineering of Computer Based Systems (ECBS), pp. 166–173 (2013)
Bohne, T., Rönnau, S., Borghoff, U.M.: Efficient keyword extraction for meaningful document perception. In: Proceedings of the 11th ACM Symposium on Document Engineering, DocEng 2011, pp. 185–194. ACM, New York (2011)
Gao, Y., Kontoyiannis, I., Bienenstock, E.: Estimating the entropy of binary time series: Methodology, some theory and a simulation study. Entropy 10(2), 71–99 (2008)
Horspool, R., Cormack, G.: Constructing word-based text compression algorithms. In: Data Compression Conference, DCC 1992, pp. 62–71 (March 1992)
Johnson, O., Sejdinovic, D., Cruise, J., Ganesh, A., Piechocki, R.J.: Non-parametric change-point detection using string matching algorithms. CoRR (2011)
Kontoyiannis, I., Algoet, P., Suhov, Y., Wyner, A.: Nonparametric entropy estimation for stationary processes and random fields, with applications to english text. IEEE Transactions on Information Theory 44(3), 1319–1327 (1998)
Mengle, S., Goharian, N.: Passage detection using text classification. J. Am. Soc. Inf. Sci. Technol. 60, 814–825 (2009)
Montemurro, M.A., Zanette, D.H.: Universal entropy of word ordering across linguistic families. PLoS ONE 6(5), e19875+ (2011)
Shannon, C.E., Weaver, W.: The mathematical theory of communication. University of Illinois Press (1949)
Wyner, A., Ziv, J.: The sliding-window lempel-ziv algorithm is asymptotically optimal. Proceedings of the IEEE 82(6), 872–877 (1994)
Ziv, J., Lempel, A.: A universal algorithm for sequential data compression. IEEE Transactions on Information Theory 23(3), 337 (1977)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bohne, T., Borghoff, U.M. (2013). Detecting Information Structures in Texts. In: Moreno-Díaz, R., Pichler, F., Quesada-Arencibia, A. (eds) Computer Aided Systems Theory - EUROCAST 2013. EUROCAST 2013. Lecture Notes in Computer Science, vol 8112. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-53862-9_59
Download citation
DOI: https://doi.org/10.1007/978-3-642-53862-9_59
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-53861-2
Online ISBN: 978-3-642-53862-9
eBook Packages: Computer ScienceComputer Science (R0)