Abstract
This paper outlines a theory of estimation, where optimality is defined for all sizes of data — not only asymptotically. Also one principle is needed to cover estimation of both real-valued parameters and their number. To achieve this we have to abandon the traditional assumption that the observed data have been generated by a “true” distribution, and that the objective of estimation is to recover this from the data. Instead, the objective in this theory is to fit ‘models’ as distributions to the data in order to find the regular statistical features. The performance of the fitted models is measured by the probability they assign to the data: a large probability means a good fit and a small probability a bad fit. Equivalently, the negative logarithm of the probability should be minimized, which has the interpretation of code length. There are three equivalent characterizations of optimal estimators, the first defined by estimation capacity, the second to satisfy necessary conditions for optimality for all data, and the third by the complete Minimum Description Length (MDL) principle.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Rissanen J. Modeling by shortest data description. Automatica, 1978, 14(5): 465–471
Shtarkov Yu M. Universal sequential coding of single messages. Translated from Problems of Information Transmission, 1987, 23(3): 3–17
Rissanen J. Optimal estimation. IEEE Information Theory Society Newsletter, 2009, 59(3): 1
Rissanen J. Information and Complexity in Statistical Modeling. New York, NY: Springer Verlag, 2007
Wallace C S, Boulton D M. An information measure for classification. Computer Journal, 1968, 11(2): 185–194
Rissanen J. Fisher information and stochastic complexity. IEEE Transactions on Information Theory, 1996, 42(1): 40–47
Grünwald P D. The Minimum Description Length Principle. Cambridge, MA: MIT Press, 2007
Cramér H. Mathematical Methods of Statistics. Princeton, NJ: Princeton University Press, 1946
Rissanen J. Stochastic complexity and modeling. Annals of Statistics, 1986, 14(3): 1080–1100
Author information
Authors and Affiliations
Corresponding author
Additional information
Jorma Rissanen was born on October 20, 1932 in Pielisjarvi, Finland. He received the Licentiate and the Doctor degrees of Technology in control theory and mathematics from the Technical University of Helsinki in 1960 and 1965, respectively. He is currently a fellow of Helsinki Institute for Information Technology, and Professor Emeritus in Technical University of Tampere, Finland. He has published over 100 papers and the Springer Verlag book Information and Complexity in Statistical Modeling.
He is the recipient of the IBM Corporate Award in 1991 for MDL principle and stochastic complexity; IEEE 1993 Richard W. Hamming Medal “For fundamental contributions to information theory, statistical inference, control theory, and the theory of complexity”; in 1998 an IEEE Information Theory Society Golden Jubilee Award for Technological Innovation for the invention of arithmetic coding; the 2007 Kolmogorov Medal from Computer Learning Research Center of University of London; and the 2009 Shannon Award from the IEEE Information Theory Society. He has also received two best paper awards from International Federation of Automatic Control in 1981 and the IEEE Information Theory Society in 1986, respectively.
He received Honorary Doctorate from the Technical University of Tampere, Finland, in 1992. He is a foreign member of Finland’s Academy of Science and Letters. He is an IEEE Fellow.
About this article
Cite this article
Rissanen, J. Basics of estimation. Front. Electr. Electron. Eng. China 5, 274–280 (2010). https://doi.org/10.1007/s11460-010-0104-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11460-010-0104-0