Generalized Principal Components Analysis and its Application in Approximate Stochastic Realization

  • K. S. Arun
  • S. Y. Kung


The state of a linear system is an information interface betwen the past and the future, and approximate realization is essentially a problem of approximating an apparently high-dimensional interface by a low-order partial state. In this chapter, we generalize the ideas of principal components to the problem of approximating the information interface between two random vectors. Two such generalizations exist in the statistical literature [1,2]. Applications of these generalizations to the partial-state selection problem lead to three approximate stochastic realization methods. We discuss these methods and the partial-state selection criterion that each optimizes. We also study their relation to determinstic identification and balanced model reduction.


Mutual Information Random Vector Singular Value Decomposition Partial State Hankel Matrix 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hotelling, H., “Relations between two sets of Variates,” Biometrika 28 pp. 321–372 (1936).MATHGoogle Scholar
  2. 2.
    Rao, C.R., “The use and Interpretation of Principal Component Analysis in Applied Research,” Sankhya Series A 26 pp. 329–358 (1964).MATHGoogle Scholar
  3. 3.
    Chamberlain, J.E., The principles of interferometric spectroscopy, Wiley (1979).Google Scholar
  4. 4.
    Bell, R.J., Introductory Fourier transform spectroscopy, Academic (1972).Google Scholar
  5. 5.
    Burg, J. P., “Maximum Entropy Spectral Analysis,” Ph.D. Dissertation, Stanford University, Stanford California (1975).Google Scholar
  6. 6.
    Levinson, N., “The Wiener RMS (Root-Mean-Square) Error Criterion in Filter Design and Prediction,” J. Math. Phys. 25 pp. 261–278 (Jan 1947).MathSciNetGoogle Scholar
  7. 7.
    Haykin, S. and Kesler, S., “Prediction-Error Filtering and Maximum-Entropy Spectral Estimation,” pp. 8–72 in Nonlinear methods of spectral analysis, ed. Haykin S..Springer Verlag (1979).Google Scholar
  8. 8.
    Moore, B. C., “Principal Component Analysis in Linear Systems: Controllability Observability and Model Reduction,” IEEE Transactions on Automatic Control AC-26(l)pp. 17–31 (February 1981).CrossRefGoogle Scholar
  9. 9.
    Mullis, C. T. and Roberts. R. A., “Synthesis of Minimum Round-off Noise Fixed Point Digital Filters.” IEEE Transactions on Circuits and Systems CAS-23 pp. 551–562 (1976).MathSciNetCrossRefGoogle Scholar
  10. 10.
    Kailath, T.,Linear Systems, Prentice Hall New York (1980).MATHGoogle Scholar
  11. 11.
    Akaike, H., “Markovian Representation of Stochastic Processes by Canonical Variables,” SIAM Journal on Control 13(1) pp. 162–173 (January 1975).MathSciNetMATHCrossRefGoogle Scholar
  12. 12.
    Sorenson, H., Parameter Estimation: Principles and Problems, Marcel Dekker Inc. (1980).MATHGoogle Scholar
  13. 13.
    Kailath, T.. “The Innovations Approach to Detection and Estimation Theory.” Proceedings of the IEEE 58 pp. 680–695 (May 1970).MathSciNetCrossRefGoogle Scholar
  14. 14.
    Anderson, B. D. O. and Kailath, T., “The Choice of Signal Process Models,” Journal of Math. Analysis and Applications 35(3) pp. 659–668 (September 1971).MathSciNetMATHCrossRefGoogle Scholar
  15. 15.
    Gevers, M. and Kailath, T., “An Innovations Approach to Least Squares Estimation Part VI: Discrete-Time Innovations-Representations and Recursive Estimation,” IEEE Transactions on Automatic Control AC-18 pp. 588–600 (December 1973).MathSciNetCrossRefGoogle Scholar
  16. 16.
    Akaike, H., “Markovian Representation of Stochastic Processes and its Application to the Analysis of Autoregressive Moving Average Processes,” Annals of the Institute of Statistical Mathematics 26 pp. 363–387 (1974).MathSciNetMATHCrossRefGoogle Scholar
  17. 17.
    Yaglom, A. M., “Outline of some topics in linear extrapolation of stationary random processes,” pp. 259–278 in Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, University of California Press, Berkeley CA (1965).Google Scholar
  18. 18.
    Kailath, T., “A View of Three Decades of Linear Filtering Theory,” IEEE Trans. Inform. Theory IT-20(2) pp. 145–181 (Mar 1974).Google Scholar
  19. 19.
    J.A. Cadzow., “Spectral Estimation: An Overdetermined Rational Model Equation Appraoach,” Proceedings of the IEEE 70(9) pp. 907–939 (September 1982).CrossRefGoogle Scholar
  20. 20.
    Prevosto, M., Benveniste, A., and Barnouin, B., “Identification of Vibrating Structures Subject to Nonstationary Excitation: A Nonstationary Stochastic Realization Problem,” pp. 252–255 in Proceedings of the Intl. Conf. on Acoustics Speech and Signal Processing, IEEE, Paris France (May 1982).Google Scholar
  21. 21.
    Benveniste, A. and Fuchs, J. J., “Single sample modal identification of a nonstationary stochastic process,” IEEE Transactions on Automatic Control AC-30(l) pp. 66–75 (January 1985).MathSciNetCrossRefGoogle Scholar
  22. 22.
    Maciejowski, J. M., “The Use of Principal Components for Approximate Linearisation Stochastic Realisation and Spectral Factorisation,” Note Presented at the IEEE Colloquium on ‘Principal Components: Model Reduction and Control’ at City University London January 7 1983. (1983).Google Scholar
  23. 23.
    Kung, S.Y. and Arun, K.S., “A Novel Hankel Approximation Method for ARMA Pole Zero Estimation from Noisy Covariance Data,” pp. WA-19 in Technical Digest of the Topical Meeting on Signal Recovery and Synthesis with Incomplete Information and Partial Constraints, Optical Society of America, Incline Village Nevada (January 1983).Google Scholar
  24. 24.
    Gelfand, I.M. and Yaglom, A.M., “Calculation of the Amount of Information about a Random Function contained in other such Function.” American Mathematical Society Translations Series 2 12 pp. 199–246 (1959).MathSciNetGoogle Scholar
  25. 25.
    Kullback, S. and Leibler, R.A., “On Information and Sufficiency.” Annals of Mathematical Statistics 22 pp. 79–86 (1951).MathSciNetMATHCrossRefGoogle Scholar
  26. 26.
    Baram, Y., “Realization and Reduction of Markovian Models from Nonstationary Data,” IEEE Transactions Automatic Control AC-26(6) pp. 1225–1231 (December 1981).MathSciNetCrossRefGoogle Scholar
  27. 27.
    Desai, U.B. and Pal. D., “A Realization Approach to Stochastic Model Reduction and Balanced Stochastic Realization,” pp. 613–620 in Proc. of the 16th Annual Conference on Information Sciences and Systems, Princeton University, New Jersey (March 1982). Also to appear in Intl. Journal of Control. 1985.Google Scholar
  28. 28.
    White, J., “Stochastic State Space Models from Emperical Data.” pp. 243–246 in International Conference on Acoustics Speech and Signal Processing, IEEE, Boston MA (April 1983).Google Scholar
  29. 29.
    Desai, U. B. and Pal, D., “A transformation approach to stochastic model reduction,” IEEE Transactions on Automatic Control AC-29 pp. 1097–1099 (Dec. 1984).MathSciNetCrossRefGoogle Scholar
  30. 30.
    Karalamangala, A. S., “A Principal Components Approach to Approximate Modeling and ARMA Spectral Estimation,” Ph.D. Dissertation, Dept. of Electr. Engg., Univ. of So. California, Los Angeles (1984).Google Scholar
  31. 31.
    Faurre, P., “Stochastic Realization Algorithms,” in System Identification: Advances and Case Studies, ed. Mehra R.K. and Lainiotis D.G.,Academic Press (1976).Google Scholar
  32. 32.
    Parlett, B. N.,The Symmetric Eigenvalue Problem, Prentice-Hall Inc., Englewood Cliffs NJ (1980).MATHGoogle Scholar
  33. 33.
    Wilkinson, J. H., The algebraic eigenvalue problem, Oxford University Press (1965).MATHGoogle Scholar
  34. 34.
    Kung, S.Y., “A Toeplitz Approximation Method and some Applications,” pp. 262–266 in Proc. of the International Symposium on the Mathematical Theory of Networks and Systems, Santa Monica CA (August 5–7 1981).Google Scholar
  35. 35.
    Kung, S. Y., Arun. K. S., and BhaskarRao, D. V., “State-space and singular value decomposition based methods for harmonic retrieval problem,” Journal of the Optical Society of America 73 pp. 1799–1811 (December 1983).CrossRefGoogle Scholar
  36. 36.
    Kung, S.Y., “A New Identification and Model Reduction Algorithm via Singular Value Decomposition,” pp. 705–714 in Proceedings of the 12th Asilomar Conference on Circuits Systems and Computers, IEEE. Pacific Grove CA (November 1978).Google Scholar
  37. 37.
    Hotelling, H., “Analysis of a Complex of Variables into Principal Components.” Journal of Educational Psychology 24 pp. 417–441 and 498–520 (1933).CrossRefGoogle Scholar
  38. 38.
    Brown, Jr. J.L., “Mean Square Truncation Error in Series Expansions of Random Functions,” Journal of SIAM 8(1) pp. 28–32 (November 1960).MATHGoogle Scholar
  39. 39.
    Kung, S.Y. and Arun, K.S., “Approximate Realization Methods for ARMA Spectral Estimation,” pp. 105–109 in Proc. of the International Symposium on Circuits and Systems, IEEE, Newport Beach CA (May 1983).Google Scholar
  40. 40.
    Fujishige. S., Nagai. H., and Sawaragi. Y.. “System-theoretical Approach to Model Reduction and System-order Determination,” International Journal of Control 22(6) pp. 807–819 (1975).MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Kluwer Academic Publishers, Boston 1986

Authors and Affiliations

  • K. S. Arun
    • 1
  • S. Y. Kung
    • 2
  1. 1.Coordinated Science LabUniv. of Illinois at Urbana-ChampaignUrbanaUSA
  2. 2.Signal and Image Processing InstUniv. of Southern CaliforniaLos AngelesUSA

Personalised recommendations