On the Feasibility of Parsimonious Variable Selection for Hotelling’s $T^2$-test

Abstract

Hotelling’s $T^2$-test for the mean of a multivariate normal distribution is one of the triumphs of classical multivariate analysis. It is uniformly most powerful among invariant tests, and admissible, proper Bayes, and locally and asymptotically minimax among all tests. Nonetheless, investigators often prefer non-invariant tests, especially those obtained by selecting only a small subset of variables from which the $T^2$-statistic is to be calculated, because such reduced statistics are more easily interpretable for their specific application. Thus it is relevant to ask the extent to which power is lost when variable selection is limited to very small subsets of variables, e.g. of size one (yielding univariate Student-$t^2$ tests) or size two (yielding bivariate $T^2$-tests). This study presents preliminary evidence suggesting that in some cases, no power may be lost, in fact may be gained, over a wide range of alternatives.

Appendices

Appendix A Testing for additional information.

Variable selection for the $T^2$-test and related linear discriminant analysis was thoroughly studied in the 1970s and 1980s, an era of limited computer power, and subsequently by several authors with greater ability to consider all-subsets methods; a list of references appears below. Almost all of these studies were based on testing for additional information (= increased Mahalanobis distance), as now described.

For any two nested subsets $\omega \subset \omega '$ in $\Upomega _p$, in general $\Uplambda _\omega \le \Uplambda _{\omega '}$. The question of whether the power of the $T_{\omega '}$-test exceeds that of the $T_{\omega }$-test for the testing problem Eq. 9 usually was formulated as the problem of testing for additional information (TAI), namely, testing

$$\begin{aligned} \Uplambda _\omega '=\Uplambda _{\omega }\quad \mathrm {vs.}\quad \Uplambda _\omega '>\Uplambda _{\omega } \end{aligned}$$

(137)

based on a preliminary sample – see [Rao (1973)] §8c.4. This formulation was adopted by many researchers, even while citing the following result of Das Gupta and Perlman (1974) which implies that this standard formulation of TAI is inappropriate.

It was shown in [DGP, Theorem 2.1] that for fixed $\lambda >0$, the power function $\pi _\alpha (\lambda ;m,n)$ (recall Eq. 2) of the non-central f-test is strictly decreasing in m and strictly increasing in n.⁸ Therefore for any integer $1\le q\le n-1$ there exists a unique real number

$$\begin{aligned} g_\alpha (\lambda ):= g_\alpha (\lambda ; m,n,q)>0 \end{aligned}$$

(138)

such that

$$\begin{aligned} \pi _\alpha (\lambda ;m,n)=\pi _\alpha (\lambda +g_\alpha (\lambda );m+q, n-q). \end{aligned}$$

(139)

Here $g_\alpha (0)=0$ and $g_\alpha (\lambda )$ is strictly increasing in $\lambda$; cf. (Theorem 3.1, Das Gupta and Perlman, 1974). Thus the power is increased only if

$$\begin{aligned} \Uplambda _{\omega '}>\Uplambda _\omega +g_\alpha (\Uplambda _{\omega }; |\omega |,N-|\omega |,|\omega '\!\setminus \!\omega |). \end{aligned}$$

(140)

Therefore (Section 4, Das Gupta and Perlman, 1974) introduced the problem of testing for increased power (TIP), namely, testing

$$\begin{aligned}&H_1:\Uplambda _{\omega '}\le \Uplambda _\omega +g_\alpha (\Uplambda _{|\omega |}; |\omega |,N-|\omega |,|\omega '\!\setminus \!\omega |)\\ \mathrm {vs.}\ \ {}&K_1:\Uplambda _{\omega '}>\Uplambda _\omega +g_\alpha (\Uplambda _{|\omega |}; |\omega |,N-|\omega |,|\omega '\!\setminus \!\omega |), \end{aligned}$$

and proposed several (approximate) tests.

This proposal was noted by subsequent authors but never implemented for variable selection, possibly because of difficulties in computing the functions $g_\alpha (\cdot )$, especially if many pairs $(\omega ,\omega ')$ must be considered. However, if as suggested above, variable selection might be limited to very small subsets of variables in practical applications, then replacing the TAI by the TIP might be feasible.

Remark A1

The relation Eq. 92 in Conjecture 4.1 can be stated equivalently in terms of $g_\alpha$:

$$\begin{aligned} \textstyle 0<\alpha <\alpha _l^*(\lambda )\ \ \implies \ \ g_\alpha (\lambda ;1,2l,1)>\big (\frac{2l+1}{2l-1}\big )\lambda \quad \forall \ \lambda >0, \end{aligned}$$

(141)

with equality when $\alpha =\alpha _l^*(\lambda )$. Thus the relations Eqs. 118 and 124 in Proposition 4.3 also can be stated equivalently in terms of $g_\alpha$:

$$\begin{aligned} 0<\alpha <\alpha _1^*(\lambda )\ \ \implies \ \ g_\alpha (\lambda ;1,2,1)>3\lambda \quad \forall \ \lambda >0, \end{aligned}$$

(142)

with equality when $\alpha =\alpha _1^*(\lambda )$;

$$\begin{aligned} \textstyle 0<\alpha <\alpha _2^*(\lambda )\ \ \implies \ \ g_\alpha (\lambda ;1,4,1)>\frac{5}{3}\lambda \quad \forall \ \lambda >0, \end{aligned}$$

(143)

with equality when $\alpha =\alpha _2^*(\lambda )$.

Additional References for the Appendix

Hand, D. J. (1981). Discrimination and Classification, Wiley & Sons, New York. [Chapter 6]

Hawkins, D. M. (1976). The subset problem in multivariate analysis of variance. J. Royal Statist. Soc, Series B 38 132-139.

Jain, A. K. and Waller, W. G. (1978). On the optimal number of features in the classification of multivariate Gaussian data. Pattern Recognition 10 103-109.

Jiang, W., Wang, K., and Tsung, F. (2012). A variable-selection-based multivariate EWMA chart for process monitoring and diagnosis. J. Quality Technology 44 209-230.

McCabe, G. P. Jr. (1975). Computations for variable selection in discriminant analysis. Technometrics 17 259-263.

McKay, R. J. (1976). A graphical aid to selection of variables in two-group discriminant analysis. Appl. Statist. 27 259-263.

McLachlan, G. J. (1976). On the relationship between the F test and the overall error rate for variable selection in two-group discriminant function. Biometrics 36 501-510.

McLachlan, G. J. (1980). A criterion for selecting variables for the linear discriminant function. Biometrics 32 529-534.

McLachlan, G. J. (1992). Discriminant Analysis and Statistical Pattern Recognition, Wiley & Sons, New York. [Chapter 12]

Murray, G. D. (1977). A cautionary note on selection of variables in discriminant analysis. Appl. Statist. 26 246-250.

Nobuo, S. and Takahisa, I. (2016). A variable selection method for detecting abnormality base on the $T^2$ test. Comm. Statist. - Theory and Methods 46 501-510.

Rao, C. R. (1973). Linear Statistical Inference and its Applications, 2nd edition, Wiley & Sons, New York.

Schaafsma, W. (1982). In Handbook of Statistics, Vol. 2, P. R. Krishnaiah and L. N. Kanal, eds., 857-881. North Holland, Amsterdam.