Abstract
While eminently successful for the transmission of data, Shannon’s theory of information does not address semantic and subjective dimensions of data, such as relevance and surprise. We propose an observer-dependent computational theory of surprise where surprise is defined by the relative entropy between the prior and the posterior distributions of an observer. Surprise requires integration over the space of models in contrast with Shannon’s entropy, which requires integration over the space of data. We show how surprise can be computed exactly in a number of discrete and continuous cases using distributions from the exponential family with conjugate priors. We show that during sequential Bayesian learning, surprise decreases like 1/N and study how surprise differs and complements Shannon’s definition of information.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
J. Aczel and Z. Daroczy. On measures of information and their characterizations. Academic Press, New York, 1975.
P. Baldi and S. Brunak. Bioinformatics: the machine learning approach. MIT Press, Cambridge, MA, 2001. Second edition.
J. O. Berger. Statistical decision theory and Bayesian analysis. Springer-Verlag, New York, 1985.
R. E. Blahut. Principles and practice of information theory. Addison-Wesley, Reading, MA, 1987.
Bela Bollobas. Random Graphs. Academic Press, London, 1985.
G. E. P. Box and G. C. Tiao. Bayesian inference in statistical analysis. John Wiley and Sons, New York, 1992. (First Edition in 1973 ).
L. D. Brown. Fundamentals of Statistical Exponential Families. Institute of Mathematical Statistics, Hayward, CA, 1986.
T. M. Cover and J. A. Thomas. Elements of Information Theory. John Wiley, New York, 1991.
R. T. Cox. Probability, frequency and reasonable expectation. Am. J. Phys., 14: 1–13, 1964.
A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin. Bayesian Data Analysis. Chapman and Hall, London, 1995.
I. S. Gradshteyn and I. M. Ryzhik. Table of integrals, series, and products. Academic Press, New York, 1980.
S.F. Gull. Developments in maximum entropy data analysis. In J. Skilling, editor, Maximum entropy and Bayesian methods, pages 53–71. Kluwer, Dordrecht, 1989.
G. E. Hinton, P. Dayan, B. J. Frey, and R. M. Neal. The wake-sleep algorithm for unsupervised neural networks. Science, 268 (5214): 1158–1161, 1995.
E. T. Jaynes. Probability Theory: The Logic of Science. 1996. Unpublished.
E.T. Jaynes. Bayesian methods: General background. In J.H. Justice, editor, Maximum Entropy and Bayesian Methods in Statistics, pages 1–25. Cambridge University Press, Cambridge, 1986.
G. Jumarie. Relative information. Springer Verlag, New York, 1990.
S. Kullback. Information theory and statistics. Dover, New York, 1968. (First Edition in 1959 ).
R. J. McEliece. The Theory of Information and Coding. Addison-Wesley Publishing Company, Reading, MA, 1977.
L. J. Savage. The foundations of statistics. Dover, New York, 1972. (First Edition in 1954 ).
C. E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27: 379–423, 623–656, 1948.
N. Tishby, F. Pereira, and W. Bialek. The information bottleneck method. In B. Hajek and R. S. Sreenivas, editors, Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing, pages 368–377. University of Illinois, 1999.
V. Vapnik. The Nature of Statistical Learning Theory. Springer Verlag, New York, 1995.
A. Zellner. Optimal information processing and Bayes’s theorem. The American Statistician, 42 (4): 278–284, 1998.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer Science+Business Media New York
About this chapter
Cite this chapter
Baldi, P. (2002). A Computational Theory of Surprise. In: Blaum, M., Farrell, P.G., van Tilborg, H.C.A. (eds) Information, Coding and Mathematics. The Springer International Series in Engineering and Computer Science, vol 687. Springer, Boston, MA. https://doi.org/10.1007/978-1-4757-3585-7_1
Download citation
DOI: https://doi.org/10.1007/978-1-4757-3585-7_1
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4419-5289-9
Online ISBN: 978-1-4757-3585-7
eBook Packages: Springer Book Archive