# Generalized Principal Components Analysis and its Application in Approximate Stochastic Realization

## Abstract

The state of a linear system is an information interface betwen the past and the future, and approximate realization is essentially a problem of approximating an apparently high-dimensional interface by a low-order partial state. In this chapter, we generalize the ideas of principal components to the problem of approximating the information interface between two random vectors. Two such generalizations exist in the statistical literature [1,2]. Applications of these generalizations to the partial-state selection problem lead to three approximate stochastic realization methods. We discuss these methods and the partial-state selection criterion that each optimizes. We also study their relation to determinstic identification and balanced model reduction.

## Keywords

Mutual Information Random Vector Singular Value Decomposition Partial State Hankel Matrix## Preview

Unable to display preview. Download preview PDF.

## References

- 1.Hotelling, H., “Relations between two sets of Variates,”
*Biometrika***28**pp. 321–372 (1936).MATHGoogle Scholar - 2.Rao, C.R., “The use and Interpretation of Principal Component Analysis in Applied Research,”
*Sankhya Series A***26**pp. 329–358 (1964).MATHGoogle Scholar - 3.Chamberlain, J.E.,
*The principles of interferometric spectroscopy*, Wiley (1979).Google Scholar - 4.Bell, R.J.,
*Introductory Fourier transform spectroscopy*, Academic (1972).Google Scholar - 5.Burg, J. P., “Maximum Entropy Spectral Analysis,” Ph.D. Dissertation, Stanford University, Stanford California (1975).Google Scholar
- 6.Levinson, N., “The Wiener RMS (Root-Mean-Square) Error Criterion in Filter Design and Prediction,”
*J. Math. Phys.***25**pp. 261–278 (Jan 1947).MathSciNetGoogle Scholar - 7.Haykin, S. and Kesler, S., “Prediction-Error Filtering and Maximum-Entropy Spectral Estimation,” pp. 8–72 in
*Nonlinear methods of spectral analysis*, ed. Haykin S..Springer Verlag (1979).Google Scholar - 8.Moore, B. C., “Principal Component Analysis in Linear Systems: Controllability Observability and Model Reduction,”
*IEEE Transactions on Automatic Control***AC-26**(l)pp. 17–31 (February 1981).CrossRefGoogle Scholar - 9.Mullis, C. T. and Roberts. R. A., “Synthesis of Minimum Round-off Noise Fixed Point Digital Filters.”
*IEEE Transactions on Circuits and Systems***CAS-23**pp. 551–562 (1976).MathSciNetCrossRefGoogle Scholar - 10.Kailath, T.,
*Linear Systems*, Prentice Hall New York (1980).MATHGoogle Scholar - 11.Akaike, H., “Markovian Representation of Stochastic Processes by Canonical Variables,”
*SIAM Journal on Control***13**(1) pp. 162–173 (January 1975).MathSciNetMATHCrossRefGoogle Scholar - 12.Sorenson, H.,
*Parameter Estimation: Principles and Problems*, Marcel Dekker Inc. (1980).MATHGoogle Scholar - 13.Kailath, T.. “The Innovations Approach to Detection and Estimation Theory.”
*Proceedings of the IEEE***58**pp. 680–695 (May 1970).MathSciNetCrossRefGoogle Scholar - 14.Anderson, B. D. O. and Kailath, T., “The Choice of Signal Process Models,”
*Journal of Math. Analysis and Applications***35**(3) pp. 659–668 (September 1971).MathSciNetMATHCrossRefGoogle Scholar - 15.Gevers, M. and Kailath, T., “An Innovations Approach to Least Squares Estimation Part VI: Discrete-Time Innovations-Representations and Recursive Estimation,”
*IEEE Transactions on Automatic Control***AC-18**pp. 588–600 (December 1973).MathSciNetCrossRefGoogle Scholar - 16.Akaike, H., “Markovian Representation of Stochastic Processes and its Application to the Analysis of Autoregressive Moving Average Processes,”
*Annals of the Institute of Statistical Mathematics***26**pp. 363–387 (1974).MathSciNetMATHCrossRefGoogle Scholar - 17.Yaglom, A. M., “Outline of some topics in linear extrapolation of stationary random processes,” pp. 259–278 in
*Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability*, University of California Press, Berkeley CA (1965).Google Scholar - 18.Kailath, T., “A View of Three Decades of Linear Filtering Theory,”
*IEEE Trans. Inform. Theory***IT-20**(2) pp. 145–181 (Mar 1974).Google Scholar - 19.J.A. Cadzow., “Spectral Estimation: An Overdetermined Rational Model Equation Appraoach,”
*Proceedings of the IEEE***70**(9) pp. 907–939 (September 1982).CrossRefGoogle Scholar - 20.Prevosto, M., Benveniste, A., and Barnouin, B., “Identification of Vibrating Structures Subject to Nonstationary Excitation: A Nonstationary Stochastic Realization Problem,” pp. 252–255 in
*Proceedings of the Intl. Conf. on Acoustics Speech and Signal Processing*, IEEE, Paris France (May 1982).Google Scholar - 21.Benveniste, A. and Fuchs, J. J., “Single sample modal identification of a nonstationary stochastic process,”
*IEEE Transactions on Automatic Control***AC-30**(l) pp. 66–75 (January 1985).MathSciNetCrossRefGoogle Scholar - 22.Maciejowski, J. M., “The Use of Principal Components for Approximate Linearisation Stochastic Realisation and Spectral Factorisation,” Note Presented at the IEEE Colloquium on ‘Principal Components: Model Reduction and Control’ at City University London January 7 1983. (1983).Google Scholar
- 23.Kung, S.Y. and Arun, K.S., “A Novel Hankel Approximation Method for ARMA Pole Zero Estimation from Noisy Covariance Data,” pp. WA-19 in
*Technical Digest of the Topical Meeting on Signal Recovery and Synthesis with Incomplete Information and Partial Constraints*, Optical Society of America, Incline Village Nevada (January 1983).Google Scholar - 24.Gelfand, I.M. and Yaglom, A.M., “Calculation of the Amount of Information about a Random Function contained in other such Function.”
*American Mathematical Society Translations Series 2***12**pp. 199–246 (1959).MathSciNetGoogle Scholar - 25.Kullback, S. and Leibler, R.A., “On Information and Sufficiency.”
*Annals of Mathematical Statistics***22**pp. 79–86 (1951).MathSciNetMATHCrossRefGoogle Scholar - 26.Baram, Y., “Realization and Reduction of Markovian Models from Nonstationary Data,”
*IEEE Transactions Automatic Control***AC-26**(6) pp. 1225–1231 (December 1981).MathSciNetCrossRefGoogle Scholar - 27.Desai, U.B. and Pal. D., “A Realization Approach to Stochastic Model Reduction and Balanced Stochastic Realization,” pp. 613–620 in
*Proc. of the 16th Annual Conference on Information Sciences and Systems*, Princeton University, New Jersey (March 1982). Also to appear in Intl. Journal of Control. 1985.Google Scholar - 28.White, J., “Stochastic State Space Models from Emperical Data.” pp. 243–246 in
*International Conference on Acoustics Speech and Signal Processing*, IEEE, Boston MA (April 1983).Google Scholar - 29.Desai, U. B. and Pal, D., “A transformation approach to stochastic model reduction,”
*IEEE Transactions on Automatic Control***AC-29**pp. 1097–1099 (Dec. 1984).MathSciNetCrossRefGoogle Scholar - 30.Karalamangala, A. S., “A Principal Components Approach to Approximate Modeling and ARMA Spectral Estimation,” Ph.D. Dissertation, Dept. of Electr. Engg., Univ. of So. California, Los Angeles (1984).Google Scholar
- 31.Faurre, P., “Stochastic Realization Algorithms,” in
*System Identification: Advances and Case Studies*, ed. Mehra R.K. and Lainiotis D.G.,Academic Press (1976).Google Scholar - 32.Parlett, B. N.,
*The Symmetric Eigenvalue Problem*, Prentice-Hall Inc., Englewood Cliffs NJ (1980).MATHGoogle Scholar - 33.Wilkinson, J. H.,
*The algebraic eigenvalue problem*, Oxford University Press (1965).MATHGoogle Scholar - 34.Kung, S.Y., “A Toeplitz Approximation Method and some Applications,” pp. 262–266 in
*Proc. of the International Symposium on the Mathematical Theory of Networks and Systems*, Santa Monica CA (August 5–7 1981).Google Scholar - 35.Kung, S. Y., Arun. K. S., and BhaskarRao, D. V., “State-space and singular value decomposition based methods for harmonic retrieval problem,”
*Journal of the Optical Society of America*73 pp. 1799–1811 (December 1983).CrossRefGoogle Scholar - 36.Kung, S.Y., “A New Identification and Model Reduction Algorithm via Singular Value Decomposition,” pp. 705–714 in
*Proceedings of the 12th Asilomar Conference on Circuits Systems and Computers*, IEEE. Pacific Grove CA (November 1978).Google Scholar - 37.Hotelling, H., “Analysis of a Complex of Variables into Principal Components.”
*Journal of Educational Psychology***24**pp. 417–441 and 498–520 (1933).CrossRefGoogle Scholar - 38.Brown, Jr. J.L., “Mean Square Truncation Error in Series Expansions of Random Functions,”
*Journal of SIAM***8**(1) pp. 28–32 (November 1960).MATHGoogle Scholar - 39.Kung, S.Y. and Arun, K.S., “Approximate Realization Methods for ARMA Spectral Estimation,” pp. 105–109 in
*Proc. of the International Symposium on Circuits and Systems*, IEEE, Newport Beach CA (May 1983).Google Scholar - 40.Fujishige. S., Nagai. H., and Sawaragi. Y.. “System-theoretical Approach to Model Reduction and System-order Determination,”
*International Journal of Control***22**(6) pp. 807–819 (1975).MathSciNetMATHCrossRefGoogle Scholar