Abstract
A model-order determination procedure for time series is described which is based on computing the differential entropy of the time series' multivariate amplitude probability density. The procedure, termed the differential entropy method, applies to linear and nonlinear models for the time series. Simulations were performed to determine the performance characteristics of the differential entropy method on first-order autoregressive models with Gaussian and non-Gaussian inputs and on a first-order nonlinear model. It produces accurate model-order estimates for the linear model once the correlation coefficient becomes large enough. The threshold for accurate estimates is dependent on the amplitude distribution of the model's input. This technique estimates the model order of data having nonlinear dependence structure more accurately than thead hoc application of the AIC and MDL methods. Extension of the method to higher-order models is described, but, because of computation complexity issues, the differential entropy method is best used for estimating small model orders.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
H. Akaike, A New Look at the Statistical Model Identification,IEEE Transactions on Automatic Control, Vol. 19, no. 6, pp. 716–723, December 1974.
J. L. Bentley, Multidimensional Divide- and-Conquer,Communications of the ACM, Vol. 23, no. 4, pp. 214–229, April 1980.
J. L. Bentley and M. I. Shamos, Divide- and-Conquer in Multidimensional Space,Proceedings of the 8th Annual ACM Symposium on Theory of Computing, pp. 220–230, May 1976.
G. E. Box and G. M. Jenkins,Time Series Analysis Forecasting and Control. San Francisco, California: Holden-Day, 1970.
D. Cheng, A. Gersho, B. Ramamurthi, and Y. Shoham, Fast Search Algorithms for Vector Quantization and Pattern Matching,ICASSP Proceedings, March 19–21, 1984.
L. D. Davisson, The Prediction Error of Stationary Gaussian Time Series of Unknown Covariance,IEEE Transactions on Information Theory, Vol. 11, no. 4, pp. 527–532, October 1965.
R. G. Gallager,Information Theory and Reliable Communication. New York: Wiley, 1968.
H. Gish and J. N. Pierce, Asymptotically Efficient Quantizing,IEEE Transactions on Information Theory, Vol. 14, pp. 676–683, 1968.
I. S. Gradshteyn and I. M. Ryzhik,Tables of Integrals, Series and Products. New York: Academic Press, 1965.
R.M. Gray, Vector Quantization,IEEE ASSP Magazine, Vol. 1, no. 2, pp. 4–29, April 1984.
D. H. Johnson, C. Tsuchitani, D. A. Linebarger, and M. J. Johnson, Application of a Point Process Model to Response of Cat Lateral Superior Olive Units to Ipsilateral Tones,Hearing Research, Vol. 21, pp. 135–159, 1986.
R. L. Kashyap, Inconsistency of the AIC Rule of Estimating the Order of Autoregressive Models,IEEE Transactions on Automatic Control, Vol. 25, no. 5, pp. 996–998, October 1980.
A. R. Kumar, A Distribution-Free Model Order Estimation Technique Using Entropy, M.Sc. Thesis, Department of Electrical Engineering, Rice University, Houston, Texas, December 1985.
J. Makhoul, Linear Prediction: A Tutorial Review,Proceedings of the IEEE, Vol. 63, no. 4, pp. 561–580, April 1975.
H. Nakahama, M. Yamamoto, N. Ishii, H. Fujii, and K. Aya, Dependency as a Measure To Estimate the Order and the Values of Markov Processes,Biological Cybernetics, Vol. 25, pp. 209–226, 1977.
R. J. Niederjohn and D. G. Haworth, Speech Amplitude Statistics and the Gamma Density Function as a Theoretical Approximation,Journal of the Acoustical Society of America, Vol. 80, no. 6, pp. 1583–1588, December 1986.
A. V. Oppenheim and R. W. Schafer,Digital Signal Processing. Englewood Cliffs, New Jersey: Prentice-Hall, 1975.
A. Papoulis,Probability, Random Variables and Stochastic Processes, Second Edition. New York: McGraw-Hill, 1965.
E. Parzen,Stochastic Processes. San Francisco: Holden-Day, 1962.
M. S. Pinsker,Information and Information Stability of Random Variables and Processes. San Francisco: Holden-Day, 1964.
J. Rissanen, Modeling by Shortest Data Description,Automatica, Vol. 14, pp. 465–471, 1978.
G. Schwarz, Estimating the Dimension of a Model,The Annals of Statistics, Vol. 6, no. 2, pp. 461–464, 1978.
C. Shannon and W. Weaver,The Mathematical Theory of Communication. Urbana, Illinois: University of Illinois, 1949.
D. L. Snyder,Random Point Processes. New York: Wiley, 1975.
M. R. Soleymani and S. D. Morgera, An Efficient Nearest Neighbor Search Method,IEEE Transactions on Communications, Vol. 35, no. 6, pp. 677–679, June 1987.
P. M. Vaidya, An Optimal Algorithm for the All-Nearest-Neighbors Problem,Proceedings of the 21th Symposium on Foundations of Computer Science, pp. 117–122, October 27–29, 1986.
M. Wax, Order Selection for AR Models by Predictive Least Squares,IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 36, no. 4, pp. 581–588, April 1988.
Author information
Authors and Affiliations
Additional information
This work was supported by Grant NS20964 from the National Institute of Health. Most of this work is contained in A.R.K.'s Master's Thesis submitted to Rice University.
Rights and permissions
About this article
Cite this article
Kumar, A.R., Johnson, D.H. A distribution-free technique to estimate the order of markovian dependence using entropy. Circuits Systems and Signal Process 9, 31–54 (1990). https://doi.org/10.1007/BF01187720
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF01187720