Spectral Properties of Correlation Matrices – Towards Enhanced Spectral Clustering

Abstract

This chapter compiles some properties of eigenvalues and eigenvectors of correlation and other matrices constructed from uncorrelated as well as systematically correlated Gaussian noise. All results are based on simulations. The situations depicted in the settings are found in time series analysis as one extreme variant and in gene/protein profile analysis with micro-arrays as the other extreme variant of the possible scenarios for correlation analysis and clustering where random matrix theory might contribute. The main difference between both is the number of variables versus the number of observations. To what extent the results can be transferred is yet unclear. While random matrix theory as such makes statements about the statistical properties of eigenvalues and eigenvectors, the expectation is that these statements, if used in a proper way, will improve the clustering of genes for the detection of functional groups. In the course of the scenarios, the relation and interchangeability between the concepts of time, experiment, and realisations of random variables play an important role. The mapping between a classical random matrix ensemble and the micro-array scenario is not yet obvious. In any case, we can make statements about pitfalls and sources of false conclusions. We also develop an improved spectral clustering algorithm that is based on the properties of eigenvalues and eigenvectors of correlation matrices. We found it necessary to rehearse and analyse these properties from the bottom up starting at one extreme end of scenarios and moving to the micro-array scenario.