Projection pursuit via white noise matrices

Abstract

Projection pursuit is a technique for locating projections from high- to low-dimensional space that reveal interesting non-linear features of a data set, such as clustering and outliers. The two key components of projection pursuit are the chosen measure of interesting features (the projection index) and its algorithm. In this paper, a white noise matrix based on the Fisher information matrix is proposed for use as the projection index. This matrix index is easily estimated by the kernel method. The eigenanalysis of the estimated matrix index provides a set of solution projections that are most similar to white noise. Application to simulated data and real data sets shows that our algorithm successfully reveals interesting features in fairly high dimensions with a practical sample size and low computational effort.

Appendix: Proof of proposition 1

In the standardized situation, we have

$$ J_{\!\!f}-I_d = E\left[\left(\nabla_x \log f(x)-x\right)\left(\nabla_x \log f(x)-x\right)^T \right]. $$

It follows that the jj diagonal entry will be zero if and only if $\frac{\partial}{\partial x_j} \log f(x)-x_j=0$ a.s. But $\frac{\partial}{\partial x_j} \log f(x)=\frac{\partial}{\partial x_j} \log f(x_j|x_{-j})$, which proves the result.

Discussion of “Projection pursuit via white noise matrices”

by G. Hui and B. Lindsay

Pranab K. Sen

Cary C. Boshamer Professor of Biostatistics, University of North Carolina

at Chapel Hill, Chapel Hill, NC 27599-7420, USA

e-mail: pksen@bios.unc.edu

The authors are to be commended for an excellent job with this projection pursuit by white noise matrices approach for locating high to low dimensional spaces with a view to preserving interesting features and appropriate algorithms to enhance statistical data analysis perspectives. A white noise matrix, based on the Fisher information matrix, is formulated wherein the classical kernel method is incorporated to estimate the effectiveness of the projection index. The main thrust of the paper is on the adaptability of the computational algorithm for fairly high dimensional data models with comparatively low computational efforts.

The basic feature of this interesting approach is the incorporation of the density f ₂(x) related to the basic density f(x) by the formula

$$ f_2({\bf x}) = f^2({\bf x}) \left/ \int_{\mathbb{R}^p} f^2({\bf y})d{\bf y}.\right. $$

(5)

With this, various properties of information matrices, some rather well known, have been incorporated to validate the computational aspects, albeit in an asymptotic setup. In this respect, we may keep in mind that in the general multivariate case, treating the sample mean vector as an estimator of the population mean vector, the Cramer–Rao information inequality stresses that

$$ {\bf V} - {\bf I}^{-1} = \text{positive semi-def\/inite}, $$

(6)

so that equivalently,

$$ {\bf I} - {\bf V}^{-1} = \text{positive semi-def\/inite,} $$

(7)

Thus, if any diagonal element of I is equal to the reciprocal of the corresponding diagonal element of V then by the Bhattacharya inequality, the corresponding element of V ^− 1 must be equal to the reciprocal of the corresponding element of V, and hence, the matrix V is diagonal, as was to be expected.

From the methodological point of view, the kernel method of course has the wider appeal of applicability even in the high-dimensional data models. However, the rate of convergence of the kernel estimators of densities in the univariate case is O(n ^− 2/5) not the conventional rate O(n ^− 1/2) while in the high-dimensional case, the rate becomes extremely slower. Therefore, from mathematical perspectives, there is a qualm: How large the sample size need be so as to suit exceedingly high-dimensional models? Current advances in multivariate density estimation clearly signal a drastically large sample size to claim asymptotic properties, as are needed in hypothesis testing problems.

Thirdly, there is a subtle emphasis on linear manifolds in projection pursuit methods of dimension reduction. However, in a general case, the lower-dimensional contours need not be hyperplanes. As such, how far the proposed methods can be used in such contour curves models?

Finally, even in the mixture models with Gaussian distributions, the usual parametric approach may not be very robust. In such a case, instead of the highly nonrobust covariance matrix V can some other robust competitors be used to robustify the procedures and the algorithms therein? Some Huber-type influence function might give a better impression although that may be harder to exhibit analytically.

Again, I would like to congratulate the authors for a commendable job.

Discussion of “Projection pursuit via white noise matrices”

by G. Hui and B. Lindsay

Surajit Ray

Department of Mathematics and Statistics, Boston University,

Boston, MA 02215, USA

e-mail: sray@math.bu.edu

1.1 Introduction

I would like to commend Guodong Hui and Bruce Lindsay for their excellent article on projection pursuit. The paper approaches projection pursuit from a “residual analysis” perspective. As in standard projection pursuit problem the goal is to separate out “interesting” and “uninteresting” dimensions in a multidimensional dataset. But unlike standard projection pursuit approach where one is focused on uncovering the possible dimensions of interest, this paper first looks for the uninteresting dimensions and then assigns the remaining dimensions as the interesting ones. There are several advantages to this approach. First it saves a lot of computational cost to first look for “null dimensions”, especially for data with “large magnitude” both in the dimension of data vectors and in the number of vectors. Statistical tests for “interesting directions” in projection pursuit approaches are hard to come by. In this paper, Hui and Lindsay provides a test statistic for choosing the “null dimensions”, based on the elements of the observed Fisher information matrix. As the observed fisher information is an easily computable quantity it has a huge computational advantage over other methods.

In this discussion I will mostly focus on the advantages of uncovering the “null directions” as the basis of statistical analyses and its possible extensions to solve challenging statistical problems. Hui and Lindsay have successfully used this technique in the context of projection pursuit approach, by defining the “uninteresting directions” as the directions closest to “normal distributions”. After discarding these “uninteresting directions” they are able to uncover the non-trivial linear and non-linear directions in high dimensional data. I think this bottom up approach is an excellent idea and there exists a large array of statistical problems where one can apply the technique of extraction of “white noise” developed in this paper. Many of the theoretical results and computational advantage developed in this paper should be readily applicable to these problems. I will focus on two interesting areas: one pertaining to the extension of the “white noise” concept for high dimensional low sample size scenario and second generalizing the concept of “white noise” to accommodate base distributions beyond normals.

1.2 Extension to high-dimensional low sample size framework

In recent time with the advancement of scientific instruments and techniques it is possible to generate huge amounts of data on a single unit, e.g. in gene expression data we can measure the expression of thousands of genes simultaneously. But in many cases the sample size is in the order of magnitude much lower than the order of magnitude of the dimensions, e.g while discriminating types of cancer the microrray gene expression data described in Golub et al. (1999) has a total of only 76 samples, but for each samples they record the expression values for 7129 genes. This datasets are often referred to as High Dimensional Low Sample Size (HDLSS) data and their characteristics are profoundly different from classical scenario where the number of samples are typically much larger than the number of dimensions. As a result methodologies, e.g. principal component analysis, clustering, developed under the classical framework are not always readily applicable under HDLSS framework.

As in classical scenario clustering is often an important statistical task for HDLLS datasets. There has been a lot of recent development in this research field of clustering and inference on number of clusters (see Fraley and Raftery 2002; Li et al. 2007; Ray and Lindsay 2008; Melnykov and Maitra 2010). In this discussion we will focus on clustering of HDLLS data and how we can make use of the notion of finding “white noise” to detect cluster membership and the number of clusters. We propose using available HDLSS asymptotic results to find structures in datasets which conforms to the dimensionality restriction of HDLSS datasets, i.e d ≫ n, d increases, while n is fixed. Here we use the word “structure” synonymous to homogeneous subsets or clusters of the data. So a dataset with no clusters, i.e. a data set which is analogous to white noise should be considered “structureless” by our definition.

1.2.1 Motivation

Hall et al. (2005) and later Ahn et al. (2007) show that in high dimension (d) and low sample size (n) the asymptotic kicks in in the direction of dimension and not the sample size. We start by inspecting the properties of gaussian data representing “white noise” in high dimensions.

The results in Hall et al. (2005) d-asymptotic approach showed that, as d increases, under some regularity conditions, the geometrical structure of data cloud becomes almost deterministic. In particular if we generate n standard normal variates Z ₁,Z ₂,...,Z _n of dimension d, where d ≫ n and d→ ∞,

• $\parallel Z \parallel =d^{\frac 1 2}+O_p(1)$, i.e. distance of a sample from the center is approximately $\sqrt{d}$. i.e. the data cloud lie on the surface of a sphere.

• The pairwise distance of the sample points $\parallel Z_1-Z_2 \parallel =2d^{\frac 1 2}+O_p(1) \sqrt(d)$.

• The pairwise angles of the sample points from the mean is $\angle(Z_1,Z_2) = \pi/2+ O_p(d^{-\frac 1 2})$, i.e. pairwise angles puts each pair of vectors perpendicular to the other.

In summary the above set of asymptotic results show that “white noise” HDLSS data acquires the randomness only from random rotation of the whole simplex generated by the above nearly deterministic relationship between the vectors of data. We will utilise these property to design diagnostic plots for white noise distributions and use it to extract clusters from HDLLS data.

Our goal in this note is to disucuss how the approach of Hui and Lindsay can be combined with the geometric structure of HDLSS data to develop a methodology for determining the number of clusters for high dimensional data. As in Hui and Lindsay we will keep on finding subclusters of the data, and remove that “selected” until we are left with only “white noise”. In the next section we demonstrate this concept using an example. Providing a formal framework to use it as a model selection tool requires further work and thus will be beyond this discussion.

1.2.2 Illustrative example

We will demonstrate our approach for model selection for the number of clusters in HDLLS scenario using a simulated example. This dataset was generated with d = 2,000 and n = 30 with four distinct clusters. A heatmap of the simulated data is given in the upper right hand corner of Fig. 13. The data follows a checkerboard pattern with the dimensions plotted along the y-axis and the samples plotted along the x-axis. Four sample clusters and five column clusters are easily visible. Here our main goal is to extract the sample clusters and arrive at the correct answer of four.

Fig. 13

Heat plot and distance plot of a HDLLS Data with four cluster data. The left panel from row 1 to 5 shows the structure of the most important directions, the second column shows the cumulative directions and finally the third column shows the residual structure of the data after removal of each of the directions. The right hand panel shows the distance statistics along with the correct asymptotic error bounds. Each row shows the effect of removing important directions from the data

Classical model selection approaches always predict much higher than four clusters as they fail to accommodate the special geometric structure of HDLSS data. This is easily explained by the fact that in higher dimensions data are so parse that datapoints forms spurious clusters.

But our approach of using the “white noise” of null direction as a model selection tool readily uses the geometric properties of the HDLSS data. Among the three properties of individual distance, pairwise distance and pairwise angles, presented in the beginning of this section, we demonstrate our findings for model selection tool only using the distance statistic. Note that the structures in this dataset are extracted using a Singular Value Decomposition (SVD) analysis of the data matrix. At each stage we remove the direction corresponding to the highest eigen value. One can definitely come up with an alternative method of clustering and achieve a clustering which will accommodate all the clusters in one step. In Fig. 13 the left panel from row 1 to 5 show the structure of the most important directions, the second column shows the cumulative directions and finally the third column shows the residual structure of the data after removal of each of the directions. The right hand panel shows the distance statistics along with the correct asymptotic error bounds. Each row shows the effect of removing important directions from the data. Finally, the fifth row shows the plots after removing all four important directions. The distance statistic shows that now more than 95% of the distance measures now fall within the bounds. In a typical model selection rule based on this distance measure one would stop here and arrive at the answer of four important directions. Note that the residual plot corresponding to direction 4 looks like a white noise and further extraction of directions just keeps the residual as noise but due to these extra dimension after adjusting the error bounds we find more distance outside the limit. This is an indication of overfitting of the model. Overall if we use the statistic of number of points outside the 95% CI of d we clearly see a global dip at four which is the true number of clusters.

As mentioned before, the above example just shows how one can use the null directions in a non-classical framework to extract important directions and design model selection tools based on the statistics. On the other hand Hui and Lindsay’s work provides a formal framework to use null directions in a Project Pursuit framework. The open questions is: Can one use the Hui and Lindsay’s framework to design a formal model selection method based on the HDLSS asymptotics, for clustering HDLSS data?

1.3 Uncovering white- T and skewed T noises

In this section we discuss another possible extension of Hui and Lindsay’s approach. There is growing interest in the literature on parametric families of multivariate distributions which represent, in some sense, departures from the multivariate normal family. In the recent past skew-normal (Azzalini and Valle 1996; Azzalini and Capitanio 1999; Lin et al. 2007) and skew-t distributions (Azzalini and Capitanio 2003; Frühwirth-Schnatter and Pyne 2010; Genton 2004) have proved to be useful for capturing skewness and kurtosis avoiding the need for transformations. For example finite mixtures of such distributions have been considered as a more general tool for handling heterogeneous data involving asymmetric behaviors across subpopulations. Multivariate versions of the skewed distributions are more challenging but they allow robust modeling of high-dimensional multimodal and asymmetric data generated by popular biotechnological platforms such as flow cytometry.

In notation a continuous random vector Y(p ×1) is said to have a multivariate skew normal distribution if its p.d.f. is given by

$$ f(y;\mu,\Sigma,D) = \frac 1 {\Phi(0;I+D \Sigma D')} \phi_p(y;\mu,\Sigma) \Phi[D(y-\mu)],\label{skewedt} $$

(8)

where $\mu\in \mathbb R$, Σ > 0; φ _p (.;μ,Σ) Φ(,;Σ), denote the p.d.f. and the c.d.f. of p-dimensional normal distribution with mean μ and covariance matrix Σ. Here D is the p ×p dimensional skewness parameter.

Hui and Lindsay’s current approach can uncover “non-normal” directions as the smoothing steps does not introduce normality and the test statistic is designed as a contrast against the “white noise” Fisher Info Matrix. Can the same set of tools be used to uncover the multivariate T or Multivariate skewed T, given in Eq. 8 as the “white noise” distribution?