Principal Components and Factor Analysis
Suppose that observations are available on q variables. In practice, q is often quite large. If, for example, q = 25, it can be very difficult to grasp the relationships among the many variables. It might be convenient if the 25 variables could be reduced to a more manageable number Clearly, it is easier to work with 4 or 5 variables than with 25. Of course, one cannot reasonably expect to get a substantial reduction in dimensionality without some loss of information. We want to minimize that loss. Assuming that a reduction in dimensionality is desirable, how can it be performed efficiently? One reasonable method is to choose a small number of linear combinations of the variables based on their ability to reproduce the entire set of variables. In effect, we want to create a few new variables that are best able to predict the original variables. Principal component analysis finds linear combinations of the original variables that are best linear predictors of the full set of variables. This predictive approach to dimensionality reduction seems intuitively reasonable. We emphasize this interpretation of principal component analysis rather than the traditional motivation of finding linear combinations that account for most of the variability in the data. The predictive approach is mentioned in Rao (1973, p. 591). Seber (1984) takes an approach that is essentially predictive. Seber’s discussion is derived from Okamoto and Kanazawa (1968). Schervish (1986) gives an explicit derivation in terms of prediction. Other approaches, that are not restricted to linear combinations of the dependent variables, are discussed by Gnanadesikan (1977, Section 2.4) and Li and Chen (1985). Jolliffe (1986) gives a thorough discussion with many examples.
KeywordsCovariance Matrix Conditional Expectation Linear Predictor Factor Analysis Model Likelihood Ratio Test Statistic
Unable to display preview. Download preview PDF.