Weighted least squares for archetypal analysis with missing data

Abstract

Archetypal analysis expresses observations in terms of a limited number of archetypes, defined as convex combinations of observed units. Such archetypes are found by minimizing a suitable loss function according to the ordinary least squares approach. The technique usually requires a data matrix without missing values. Obviously, this limits its applicability whenever at least one data entry is not available. For this purpose, extensions of archetypal analysis with missing data are developed. In line with recent advances in this domain, this is done by introducing a weighting system giving null weights to the missing entries that are imputed in order to determine the archetypes. This can be done by approaching the problem by means of weighted least squares. The effectiveness of the proposals, also in comparison with existing techniques, is explored by applications to simulated and real data sets.

Appendix 1

Let f(M | X, W) be the loss function to be minimized where X, W denote the data and weight matrices, respectively, and M is the model description for X. Assuming that the minimum of f(M | X, W) cannot be found by using a closed-form solution, f(M | X, W) can be minimized finding a monotonically decreasing sequence of loss function values, which converges because the loss function is bounded below by zero.

Majorization tools (see, e.g., de Leeuw and Heiser 1980), can be used to decrease the loss function value at every iteration. The idea of majorization is to find, at each iteration t and given the current estimate M^(t), a majorizing function g(M | M^(t), X, W) such that

$$\left\{ {\begin{array}{*{20}c} {{{g(\mathbf{M}|\mathbf{M}^{(t)}, \mathbf{X}, \mathbf{W}) }} \ge {{ f(\mathbf{M}|\mathbf{X}, \mathbf{W})}}} & {\forall {\mathbf{M}}}, \\ {{g(\mathbf{M}|\mathbf{M}^{(t)}, \mathbf{X}, \mathbf{W}) = f(\mathbf{M}|\mathbf{X}, \mathbf{W})}} & {{\text{if }}\mathbf{M} = {\mathbf{M}}^{(t)}{.}} \\ \end{array} } \right.$$

(41)

Therefore, the majorizing function is always higher than the loss function, except at the current estimate, where the two functions coincide.

The minimization of g is easier than that of f. If M^(t+1) is the minimizer of g, from (41) it is clear that the following inequalities hold:

$$\begin{array}{*{20}l} {f({\mathbf{M}}^{{(t)}} |{\mathbf{X}},{\text{ }}{\mathbf{W}}){\text{ }} = {\text{ }}g({\mathbf{M}}^{{(t)}} |{\mathbf{M}}^{{(t)}} ,{\text{ }}{\mathbf{X}},{\text{ }}{\mathbf{W}})\,{\mkern 1mu} \ge } \hfill \\ {g({\mathbf{M}}^{{(t + 1)}} |{\mathbf{M}}^{{(t)}} ,{\text{ }}{\mathbf{X}},{\text{ }}{\mathbf{W}}){\mkern 1mu} \ge f({\mathbf{M}}^{{(t + 1)}} |{\mathbf{X}},{\text{ }}{\mathbf{W}}).} \hfill \\ \end{array}$$

(42)

Thus, by minimizing g subsequently, we decrease the value of f (see also Fig. 

Fig. 9

Illustration of the majorization principle

9). Following Heiser (1987), in order to choose g, the majorizing function of f is

$$\begin{array}{*{20}l} {g({\mathbf{M}}|{\mathbf{M}}^{{(t)}} ,{\text{ }}{\mathbf{X}},{\text{ }}{\mathbf{W}}){\text{ }} = {\text{ }}{\mkern 1mu} {\mkern 1mu} } \hfill \\ {a{\text{ }} + {\text{ }}d^{2} ||{\mathbf{M}}^{{(t)}} - d^{{ - 2}} {\mathbf{W}}_{2} *{\mathbf{M}}^{{(t)}} + d^{{ - 2}} {\mathbf{W}}_{2} *{\mathbf{X}}{\text{ }} - {\mkern 1mu} {\mathbf{M}}{\mkern 1mu} ||^{2} = a{\text{ }} + {\text{ }}d^{2} f({\mathbf{M}}|{\mathbf{X}}^{{(t)}} ),} \hfill \\ \end{array}$$

(43)

where W₂ = W * W and d² = max(W₂). If the elements of W are either 1 or 0, it is clear that W₂ = W and d² = 1 and therefore (43) can be simplified as

$$g({\mathbf{M}}|{\mathbf{M}}^{{(t)}} ,{\text{ }}{\mathbf{X}},{\text{ }}{\mathbf{W}}){\text{ }} = a{\text{ }} + {\text{ }}||{\mathbf{M}}^{{(t)}} - {\mathbf{W}}*{\mathbf{M}}^{{(t)}} + {\mathbf{W}}*{\mathbf{X}}{\text{ }} - {\text{ }}{\mathbf{M}}{\mkern 1mu} ||^{2} = a{\text{ }} + {\text{ }}f({\mathbf{M}}|{\mathbf{X}}^{{(t)}} ).$$

(44)

In (44), ${\text{f(}}{\mathbf{M}}{\text{|}}{\mathbf{X}}^{{{\text{(t)}}}} {\text{) = }}$ || X^(t) – M ||², where X^(t) takes the form

$${\mathbf{X}}^{\left( t \right)} = {\mathbf{M}}^{(t)} {-}{\mathbf{W}}*{\mathbf{M}}^{(t)} + {\mathbf{W}}*{\mathbf{X}} = {\mathbf{V}}*{\mathbf{M}}^{(t)} + {\mathbf{W}}*{\mathbf{X}},$$

(45)

with V = 1_{n × p} – W.

The importance of Kiers (1997) is based on the result in (44) and (45).

The iterative procedure can be summarized as follows.

Step 0 (Inizialitation).

Initialize M as M^(t) with t = 0 and compute f(M^(t) | X, W). Note that f(M^(t) | X, W) is the sum of squared residuals for the non-missing entries.

Step 1 (Computation of the data matrix).

Compute X^(t) = V * M^(t) + W * X.

Step 2 (OLS estimation).

Update M as M^(t+1) that minimizes || X^(t) – M ||², i.e., the OLS problem using the data in X^(t).

Step 3 (Convergence).

Compute f(M^(t+1) | X, W). Letting ε be the pre-specified convergence criterion (e.g., 10^–6), if f(M^(t) | X, W) – f(M^(t+1) | X, W) > ε f(M^(t) | X, W), set t = t + 1 and go to Step 1; else consider the algorithm converged.