Degeneracy in Candecomp/Parafac and Indscal Explained For Several Three-Sliced Arrays With A Two-Valued Typical Rank

Abstract

The Candecomp/Parafac (CP) method decomposes a three-way array into a prespecified number R of rank-1 arrays, by minimizing the sum of squares of the residual array. The practical use of CP is sometimes complicated by the occurrence of so-called degenerate sequences of solutions, in which several rank-1 arrays become highly correlated in all three modes and some elements of the rank-1 arrays become arbitrarily large. We consider the real-valued CP decomposition of all known three-sliced arrays, i.e., of size p×q×3, with a two-valued typical rank. These are the 5×3×3 and 8×4×3 arrays, and the 3×3×4 and 3×3×5 arrays with symmetric 3×3 slices. In the latter two cases, CP is equivalent to the Indscal model. For a typical rank of {m,m+1}, we consider the CP decomposition with R=m of an array of rank m+1. We show that (in most cases) the CP objective function does not have a minimum but an infimum. Moreover, any sequence of feasible CP solutions in which the objective value approaches the infimum will become degenerate. We use the tools developed in Stegeman (2006), who considers p×p×2 arrays, and present a framework of analysis which is of use to the future study of CP degeneracy related to a two-valued typical rank. Moreover, our examples show that CP uniqueness is not necessary for degenerate solutions to occur.

1 1. Introduction

Carroll and Chang (1970) and Harshman (1970) independently proposed the same method for component analysis of three-way arrays, and named it Candecomp and Parafac, respectively. In the Candecomp/Parafac (CP) model, an I × J × K array X is decomposed into a prespecified number of R components Y^(r), r = 1, …, R, and a residual term E, all of the same order as X, i.e.,

$$ \underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle-}$}}{X} = \sum\limits_{r = 1}^R {\underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle-}$}}{Y} ^{(r)} + \underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle-}$}}{E} .} $$

(1.1)

. Each component Y^(r) is the outer product of three vectors a^(r), b^(r), and c^(r), i.e., y ^(r) _ijk = a ^(r) _i b ^(r) _j c ^(r) _k . For fixed R, the CP decomposition (1.1) is found by minimizing the sum of squares of E. Usually, an iterative algorithm is used for this purpose (see, e.g., Tomasi & Bro, 2006). In this paper we will denote column vectors as x, matrices as X, and three-way arrays as X.

We consider the real-valued CP model, i.e., we assume the array X and the component matrices A, B, and C to be real-valued. The real-valued CP model is used in a majority of applications in psychology and chemistry (see Kroonenberg, 1983; and Smilde, Bro, & Geladi, 2004). Complex-valued applications of CP occur in, e.g., signal processing and telecommunications research (see Sidiropoulos, 2004).

The concept of rank is the same for matrices and three-way arrays. The three-way rank of X is defined as the smallest number of rank-1 arrays whose sum equals X. A three-way array has rank 1 if it is the outer product of three vectors. Hence, in the CP decomposition (1.1) each of the R components Y ^(r) has rank 1. The three-way rank of X is equal to the smallest number of components for which a CP decomposition exists with perfect fit, i.e., with an all-zero residual term E. Since we consider the real-valued CP model, the rank of any array is assumed to be the rank over the real field.

A CP solution is usually expressed in terms of the component matrices A (I × R), B (J × R), and C (K × R), which have as columns the vectors a ^(r), b ^(r), and c ^(r), respectively. Let the kth slices of X and E be denoted by X _k (I × J) and E _k (I × J), respectively. Then (1.1) can be written as

$${{\rm{X}}_k} = {\rm{A}}{{\rm{C}}_k}{{\rm{B}}^{\rm{T}}} + {{\rm{E}}_k},k = 1,...K$$

(1.2)

, where C _k is the diagonal matrix with the kth row of C as its diagonal.

From the discussion on rank above, it follows that solving the CP model boils down to finding a best rank-R approximation of X, i.e.,

$$\eqalign{ & {\rm{Minimize }}{\left\| {{\rm{}} - {\rm{}}} \right\|^2}, \cr & {\rm<Superscript>bject to </Superscript>} \in {D_R} \cr} $$

(1.3)

, where ∥·∥ denotes the Frobenius norm and D _R is the set of I × J × K arrays of rank R or less, i.e.,

$$ \mathcal{D}_R = \left\{ {\underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle-}$}}{Y} : I \times J \times K with rank \leqslant R} \right\}. $$

(1.4)

The full rank-R decomposition (A,B,C) of the optimal solution of problem (1.3) is then the optimal CP solution.

To any set of component matrices (A,B,C) corresponds a fitted model array ^X = X − E. We will refer to a set (A,B,C) and the corresponding ^X which globally minimizes the sum of squares of E in (1.1), as a best rank-R approximation of X or as an optimal CP solution.

The uniqueness of a CP solution is usually studied for a given fitted model array ^X. It can be seen that the component matrices (A,B,C) corresponding to ^X can only be unique up to rescaling/counterscaling and jointly permuting columns of A, B, and C. Indeed, the fitted model array will be the same for the solution given by ¯A = AΠT _a , ¯B = BΠT _b , and ¯C = CΠT _c , for a permutation matrix Π and diagonal matrices T _a , T _b , and T _c with T _a T _b T _c = I _R . When, for a given fitted model array ^X, the CP solution (A,B,C) is unique up to these indeterminacies, it is called essentially unique. To mitigate the scaling indeterminacy, the columns of two component matrices can be normed at unit length (in this way, the diagonal elements of the corresponding diagonal matrices T _x can only be −1 or 1). If these constraints have been imposed, two component matrices are restricted, and one is unrestricted.

Kruskal (1977) has shown that essential uniqueness of the CP solution holds under relatively mild conditions. Kruskal’s condition relies on a particular concept of matrix rank that he introduced, which has been named k-rank after him. Specifically, the k-rank of a matrix is the largest number x such that every subset of x columns of the matrix is linearly independent. We denote the k-rank of a matrix A as k _A . For a CP solution (A,B,C), Kruskal (1977) proved that

$${k_{\rm{A}}} + {k_{\rm{B}}} + {k_{\rm{C}}} \ge 2R + 2$$

(1.5)

is a sufficient condition for essential uniqueness. This uniqueness property of CP is one of its most attractive features.

The practical use of CP, however, is sometimes complicated by the occurrence of so-called degenerate sequences of CP solutions. In such cases, convergence of the CP algorithm is extremely slow (it seems to be caught in a swamp, see Mitchell & Burdick, 1994) and some components of the CP solution become more and more correlated as the CP algorithm runs. Degenerate sequences of CP solutions were first reported in Harshman and Lundy (1984). In the majority of such cases, exactly two components, say Y ^(s) and Y ^(t), of the solution display the following pattern:

• In all three component matrices, the columns s and t become almost exactly equal up to a sign change, the product of these sign changes being −1.

• The magnitudes of the elements of columns s and t in the unrestricted component matrix become arbitrarily large.

This pattern is called a two-factor degeneracy (see Kruskal, Harshman, & Lundy, 1989). The contributions of Y ^(s) and Y ^(t) diverge in nearly opposite directions. However, their sum Y ^(s) + Y ^(t) still contributes to a better fit of the CP decomposition. Degenerate sequences of CP solutions can be avoided by imposing orthogonality constraints on the component matrices (see Harshman & Lundy, 1984). Of course, this will result in some loss of fit. Lim (2005) shows that if X is nonnegative and (A,B,C) are required to be nonnegative, then degeneracy does not occur.

Analogous to two-factor degeneracies, also three-factor degeneracies have been encountered, in which the three components Y ^(s), Y ^(t), and Y ^(u) display the following pattern:

• In two component matrices, the columns s, t, and u become almost exactly equal up to a sign change. In the third component matrix, the sum of the three columns (up to a sign change) becomes close to zero.

• The magnitudes of the elements of columns s, t, and u in the unrestricted component matrix become arbitrarily large.

The sign changes are such that the contribution of two of the factors together nearly cancels the contribution of the third factor, while the sum Y ^(s) +Y ^(t) +Y ^(u) still contributes to a better fit of the CP model.

Paatero (2000) has constructed degenerate sequences of CP solutions, where the degeneracies involve two, three, or four factors. Stegeman (2006) gives an example of a 3 × 3 × 2 array of rank 4 whose CP solution with R = 3 becomes a three-factor degeneracy as above. Also degeneracies involving five or more components can be constructed and encountered when fitting the CP model.

For clarity, by a sequence of degenerate CP solutions we mean a situation where the elements of those columns of the unrestricted component matrix which are involved in the degeneracy, increase without bound as the CP algorithm runs longer, while the CP objective value keeps decreasing (albeit extremely slowly). This is the case in all degeneracies discussed in this paper. This is different from the bounded degeneracy situation discussed in Mitchell and Burdick (1994) and Paatero (2000), wherein the associated elements of the columns in the unrestricted component matrix remain bounded, while the CP algorithm goes through a swamp and afterwards converges to the optimal solution (possibly after trying several different starting values).

2 2. Explaining Degenerate Sequences of CP Solutions

Kruskal et al. (1989) have argued that if degenerate sequences of CP solutions occur this is due to the fact that the CP objective function has no minimum, but an infimum. They reason that every sequence of CP solutions, of which the objective value is approaching the infimum, must fail to converge and displays the pattern of degeneracy as stated above. This explanation of sequences of degenerate CP solutions has recently been confirmed for two-sliced arrays by Stegeman (2006, 2007). Below we discuss the results of Stegeman (2006, 2007) and give an outline of the present paper, but first we establish in which cases the CP objective function may not have a minimum.

As stated above, solving the CP model boils down to solving problem (1.3), which minimizes the distance between X and Y ∈ D _R . If X has rank R + 1 or higher, i.e., X ∉ D _R , then an optimal solution of problem (1.3) must lie on the boundary of the set D _R .

It follows that if D _R is a closed set, i.e., if it contains all its boundary points, then problem (1.3) has an optimal solution and the CP objective function has a minimum. However, in general, the set D _R is not closed. The examples in Paatero (2000) show that D ₂, D ₄, and D ₅ are not closed. De Silva and Lim (2006) show that D ₁ is closed and D _R is not closed for any R ∈ {2, …, min(I,J,K)}, where the arrays have size I × J × K.

Unless D _R is an open set, the fact that it is not closed only implies that the CP objective function may not have a minimum. If an optimal boundary point is an element of D _R , however, then the CP objective function does have a minimum. Therefore, to explain the occurrence of degenerate CP solutions using the idea of Kruskal et al. (1989) above, we first need to show that situations where all optimal boundary points lie outside of D _R occur in practice and, second, that a sequence of CP solutions converging towards such a boundary point becomes degenerate.

In Stegeman (2007), this is done for generic p × q × 2 arrays X and all combinations of p, q, and R. This extends Stegeman (2006), who considers p × p × 2 arrays. In these papers it was found that a key role in the occurrence of degenerate sequences of CP solutions for p × q × 2 arrays is played by the two-valued typical rank of p × p × 2 arrays. The typical rank of a I × J × K array is defined as the set of rank values which have positive volume in the IJK-dimensional space of I × J × K arrays. For p × p × 2 arrays the typical rank is {p,p + 1} and for p × q × 2 arrays (with p > q) the typical rank is min(p, 2q) (see Ten Berge & Kiers, 1999). Note that rank values outside the typical rank set occur on sets of zero volume; equivalently, the sets of arrays with nontypical rank values have dimensionality lower than IJK. Consequently, generic arrays have rank values in the typical rank set only.

Hence, for p × p × 2 arrays the sets of rank-p arrays and rank-(p + 1) arrays both have dimensionality 2p ² and positive volume. In Stegeman (2006) it is shown that the boundary between these two sets does not contain rank-p arrays (except for a lower-dimensional subset which is immaterial in practice). Hence, if the p × p × 2 array X has rank p + 1 and the optimal CP solution with R = p is sought, then the optimal boundary points of D _p do not lie in D _p almost everywhere, which implies that the CP objective function does not have a minimum almost everywhere. Stegeman (2006) also shows that any sequence of CP updates, converging to such an optimal boundary point, becomes degenerate. The results in Stegeman (2006, 2007) confirm the idea of Kruskal et al. (1989) that degenerate sequences of CP solutions occur due to the fact that the CP objective function does not have a minimum.

For p × q × 2 arrays, the results of Stegeman (2006, 2007) imply that if a CP algorithm is designed to find an optimal CP solution and the sequence of CP updates converges to an optimal boundary point of D _R which does not belong to D _R , then it becomes degenerate. Hence, in this case, modified CP algorithms designed to avoid degenerate solutions, yet still trying to find an optimal CP solution (e.g., Rayens & Mitchell, 1997; Cao, Chen, Mo, & Yu, 2000) are no remedy. This is true for all degeneracies that are not bounded.

In this paper we further investigate how a two-valued typical rank is related to the occurrence of degenerate sequences of CP solutions. We consider all known three-way arrays with a two-valued typical rank {m, m+1}. For a generic array X of rank m+1 we fit the CP model with R = m and show whether and how degenerate sequences of CP solutions occur. Unfortunately, only few typical rank results are known for the real field (contrary to the complex field). Table 1 contains all known arrays which have a two-valued typical rank (other known typical ranks are single-valued). These results can be found in Ten Berge (2000, 2004) and Ten Berge, Sidiropoulos, and Rocci (2004). Table 1 also states whether degenerate sequences of CP solutions are encountered when fitting the CP model as described above, and whether the CP decomposition is essentially unique.

Table 1 Occurrence of degenerate sequences of CP solutions for all known arrays with a two-valued typical rank {m, m + 1}, where (sym.) indicates that the array has symmetric I × I slices. Also, it is stated whether the CP decomposition is essentially unique.

Case 1 of Table 1 has been treated in Stegeman (2006). Case 2 is completely analogous, since the criterion to distinguish p × p × 2 arrays of rank p from those of rank p + 1 does not depend on whether the two p × p slices are symmetric or not (see Ten Berge et al., 2004). Using the tools developed in Stegeman (2006), we will explain the occurrence of degenerate sequences of CP solutions in Cases 3, 4, 5, and 6 of Table 1. It will be shown that the two valued typical rank plays a similar role as in Cases 1 and 2, i.e., the boundary between the set of rank-m arrays and the set of rank-(m + 1) arrays consists almost everywhere of rank-(m + 1) arrays. Hence, the idea of Kruskal et al. (1989) is confirmed, i.e., the degenerate sequences of CP solutions occur due to the fact that the optimal boundary points of D _m in problem (1.3) do not lie in the set D _m itself. Moreover, any sequence of CP solutions converging to such an optimal boundary point will become degenerate. In Stegeman (2006, 2007) the essential uniqueness of the CP decomposition plays a key role in proving the latter point. As we can see from Table 1, in Cases 4 and 6 we do not have essential uniqueness but still the sequence of CP updates become degenerate. This shows that essential uniqueness is, in general, not a necessary condition for degenerate sequences of CP solutions to occur.

This paper is organized as follows. In Sections 4 through 7, the Cases 3 through 6 of Table 1 will be treated. For all cases, the proofs are conceptually similar to the proof of Stegeman (2006) and the basic ideas are explained in Section 3. For Case 5 a simulation experiment is conducted to determine how often degenerate solutions occur. Finally, Section 8 contains a discussion of the presented results.

3 3. Framework of Analysis

From Stegeman (2006) and the analysis in the current paper, it can be seen that the rank criteria in the cases of Table 1 are different for each case (except for Cases 1 and 2, as explained above). However, the approach to proving whether or not degenerate sequences of CP solutions occur is the same for all cases. The basic ideas are introduced in Stegeman (2006), and will be explained below. This framework of analysis will probably be of use for the study of other arrays having a two-valued typical rank (when they become known).

Suppose I × J × K arrays have typical rank {m, m + 1} and let X be a generic I × J × K array with rank m + 1. We consider fitting the CP model to X with R = m components. However, we do not consider the CP problem (1.3) and the set D _m in (1.4) of all arrays of rank m or less. Instead, we confine the analysis to arrays in a suitably chosen set R, such that the complement of R in the space of I × J × K arrays has zero volume. Moreover, R is chosen such that all arrays in R have at least rank m.

Note that the restriction to arrays in the set R is only virtual, since any array not lying in R can be approximated arbitrarily closely by arrays in R.

Consider the set D = D _m ∩ R, i.e., D consists of all arrays in R that have rank m. The problem we analyse is

$$ \begin{gathered} Minimize \left\| {\underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle-}$}}{X} - \underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle-}$}}{Y} } \right\|^2 , \hfill \\ subject \underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle-}$}}{Y} \in \mathcal{D} \hfill \\ \end{gathered} $$

(3.1)

. As explained in the previous section, since X ∉ D, any optimal solution of problem (3.1) is a boundary point of D. Next, define the set S as the closure of D within the set R, i.e., S contains D and all its boundary points which lie in R. Hence, by definition, S is a closed subset of R and the interior of S is contained in the set D. We also consider the following problem:

$$\eqalign{ & {\rm{Minimize }}{\left\| {{\rm{ - }}} \right\|^2}, \cr & {\rm<Superscript>bject to </Superscript>} \in {\rm{ }}S \cr} $$

(3.2)

. Since X ∉ D and S is closed, problem (3.2) always has an optimal solution ˜X which is a boundary point of S and of D. See Figure 1 for an illustration. The key question is whether ˜X lies in D or not. If it does, then problem (3.1) has an optimal solution ˜X and the objective function in (3.1) has a minimum. If ˜X does not lie in D and neither does any optimal solution of problem (3.2), then problem (3.1) has no optimal solution and the objective function in (3.1) has no minimum. For Cases 3, 4, 5, and 6 of Table 1, it will be shown that the latter is true almost everywhere, because the boundary points of D have rank larger than m almost everywhere. The next step is to show that any sequence of CP updates in D, which converges to ˜X, necessarily becomes degenerate.

Figure 1

Illustration of problems (3.1) and (3.2). The sets D (rank m) and R\S (rank ≥ m +1) have equal dimensionality and the boundary of D has lower dimensionality. The target array X lies in the set R\S. The boundary point ˜X is the optimal solution of problem (3.2).

Note that the sets D and R\S (which denotes all arrays in R not lying in S) both have positive IJK-dimensional volume, since they contain an IJK-dimensional set of rank-m and rank-(m + 1) arrays, respectively. The set S\D, containing the boundary points of D not lying in D itself, has zero IJK-dimensional volume. Since the target array X in (3.1) and (3.2) is a generic array of rank m + 1, it lies in the set R\S almost everywhere; see Figure 1.

In the analysis of p × p × 2 arrays of Stegeman (2006), the set R is given by the arrays Y with a nonsingular first p × p slice Y ₁. The set D is given by all Y ∈ R such that Y ₂ Y ⁻¹₁ has p real eigenvalues and p linearly independent eigenvectors. The set S is given by all Y ∈ R such that Y ₂ Y ⁻¹₁ has p real eigenvalues. Stegeman (2006) shows that the boundary of D consists of all Y ∈ R such that Y ₂ Y ⁻¹₁ has p real eigenvalues which are not all distinct. The same author shows that these boundary points (apart from a lower-dimensional subset which is immaterial in practice) do not lie in D. Hence, it follows that the objective function of problem (3.1) does not have a minimum almost everywhere.

In the following sections we treat Cases 3 through 6 of Table 1 using the framework described above. In each case, we define the sets R, D, and S and determine which boundary points of D lie in D and which do not. Moreover, it will be shown that any sequence of arrays in D, which converges to a boundary point of D of rank m + 1 or higher, will become degenerate as it converges to its limit point.

4 4. Fitting CP to Generic 3 × 3 × 4 Arrays with Symmetric Slices

Here we prove Case 3 of Table 1. Ten Berge et al. (2004) have shown that 3 × 3 × 4 arrays with symmetric 3 × 3 slices have a typical rank of {4, 5}. We consider fitting CP to a generic array X of rank 5, with R = 4. For the rank criterion of Ten Berge et al. (2004), we need the following definitions. For a 3 × 3 × 4 array Y with symmetric slices Y ₁, …,Y ₄, let the 9 × 4 matrix S contain the columnwise vecs of Y ₁, …, Y ₄. Let S ₄ be the 4 × 4 matrix consisting of rows 1, 2, 3, and 5 of S. Define the 4 × 1 vectors f and g as

$${{\rm{f}}^{_{^{\rm{T}}}}} = ({s_{61}}{\rm{ }}{s_{62}}{\rm{ }}{s_{63{\rm{ }}}}{s_{64}}){\rm{S}}_4^{ - 1}{\rm{ and }}{{\rm{g}}^{_{\rm{T}}}} = ({s_{91{\rm{ }}}}{s_{92{\rm{ }}}}{s_{93{\rm{ }}}}{s_{94}}){\rm{S}}_4^{ - 1}$$

(4.1)

, where s _ij denotes the (i, j)th element of S. Let the polynomial P be defined as

$$\eqalign{ & P(u) = {u^4}({g_4} - f_4^2) + {u^3}( - 2{f_2}{f_4} + {f_4}{g_3} - 2{f_3}{g_4} + {g_2}) \cr & + {u^2}( - f_2^2 - 2{f_1}{f_4} + {f_2}{g_3} - {f_3}{f_4}{g_3} + {g_1} - 2{f_3}{g_2} + f_3^2{g_4}) \cr & + u( - 2{f_1}{f_2} + {f_1}{g_3} - {f_2}{f_3}{g_3} - 2{f_3}{g_1} + f_3^2{g_2}) + ({g_1}f_3^2 - f_1^2 - {f_1}{f_3}{g_3}) \cr} $$

(4.2)

, where f _j and g _j denote the j th elements of f and g, respectively.

As discussed in the previous section, we now specify the sets R, D, and S for this particular case. Let

(4.3)

(4.4)

(4.5)

The following lemma states the properties of the sets R, D, and S that we need in our proof.

Lemma 4.1. Let the sets R, D, and S be given by (4.3–4.5).

• (i) A 3× 3 × 4 array with symmetric slices lies in R almost everywhere.

• (ii) The arrays in R have at least rank 4.

• (iii) The set D consists of the arrays in R that have rank 4.

• (iv) The set D is an open subset of R and S is the closure of D within the set R.

• (v) A 3 × 3 × 4 array with symmetric slices and rank 5 lies in R\S almost everywhere.

Proof: First, we prove (i). A singular S₄, the equality g₄ = f ²₄ , or P having a root f₃ all imply deterministic relations between the elements of Y, which are not implied by the symmetry of the slices. Hence, the complement of the set R has dimensionality lower than 24 (which is the dimensionality of the space of 3 × 3 × 4 arrays with symmetric slices). This implies (i).

The requirement that S ₄ is nonsingular implies that S has rank 4 and, hence, all arrays in R have at least rank 4. This proves (ii).

Ten Berge et al. (2004) show that an array Y ∈ R has rank 4 if P has four distinct real roots. If P has some complex roots or some identical real roots, then Y has at least rank 5. This proves (iii). The polynomial P has all roots different almost everywhere and all roots real on a set of positive volume. Also, complex roots of P occur on a set of positive volume. This implies (v).

The roots of P depend continuously on the coefficients of P and, hence, on the elements of the array Y. Therefore, the boundary points of D are those arrays for which P has four real roots, but not all distinct. Such arrays can be approximated arbitrarily closely from D, but also by arrays for which P has complex roots (see Lemma 2 in Stegeman, 2006). The boundary points of D do not lie in D itself and, hence, D is an open subset of R. This proves (iv).

We consider the CP problem (3.1) where D is given by (4.4). The following theorem states our results.

Theorem 4.2. LetXbe a generic 3 × 3 × 4 array with symmetric slices and rank 5. Let D be given by (4.4).

• (I) Problem (3.1) has no optimal solution; equivalently, the objective function of problem (3.1) has no minimum, only an infimum.

• (II) Any sequence of CP updates, in which the objective value of (3.1) converges to the infimum, will become degenerate.

Proof: Statement (I) follows from the fact that D is an open subset of R and X ∉ D, see (iv) and (v) of Lemma 4.1.

Next we prove (II). It follows from (iv) of Lemma 4.1 that the infimum of (3.1) is attained at a boundary point ˜X of D that is an optimal solution of problem (3.2), where S is given by (4.5). We have to show that any sequence of CP updates Y in D, converging to ˜X, will become degenerate. For an array Y ∈ D, let P have roots u ₁, …, u ₄. Ten Berge et al. (2004) show that Y has an essentially unique rank-4 CP decomposition (A,B,C) with

$${\rm{A}} = {\rm{B}} = \left[ {\matrix{ 1 & 1 & 1 & 1 \cr {{u_1}} & {{u_2}} & {{u_3}} & {{u_4}} \cr {q({u_1})} & {q({u_2})} & {q({u_3})} & {q({u_4})} \cr } } \right]{\rm{ and C}} = {\rm{S}}_4^{\rm{T}}{\left[ {\matrix{ 1 & {{u_1}} & {q({u_1})} & {u_1^2} \cr 1 & {{u_2}} & {q({u_2})} & {u_2^2} \cr 1 & {{u_3}} & {q({u_3})} & {u_3^2} \cr 1 & {{u_4}} & {q({u_4})} & {u_4^2} \cr } } \right]^{ - 1}}$$

(4.6)

, where q(u) = q(u|Y) is a well-defined continuous function of u, depending continuously on the elements of Y. Let the array ˜X have a polynomial ˜P with roots ũ _j . Since ˜X ∈ S\D, the roots ũ _j are real but not all distinct. The roots u _j of P depend continuously on Y. Hence, if Y converges to ˜X, the roots u _j will converge to the roots ũ _j . Since some of the roots ũ _j are identical, it follows from (4.6) that if Y is close to ˜X, then some columns in A = B will become more and more alike. Moreover, the corresponding columns in C will become arbitrarily large. Also, it can be shown that the sum of these columns remains small and does not blow up. Clearly, this is the pattern of a degenerate sequence of CP solutions as described in Section 1. This completes the proof of (II).

The Indscal model can be understood as CP for an I × I × K array with symmetric I × I slices, with the additional restriction A = B (see Carroll & Chang, 1970). Since the CP solution (4.6) is essentially unique and features A = B, the CP model is equivalent to the Indscal model in this case. Hence, Theorem 4.2 also explains the occurrence of degenerate sequences of Indscal solutions for 3 × 3 × 4 arrays of rank 5, with R = 4.

5 5. Fitting CP to Generic 3 × 3 × 5 Arrays with Symmetric Slices

Here we prove Case 4 of Table 1. Ten Berge et al. (2004) have shown that 3 × 3 × 5 arrays with symmetric 3 × 3 slices have a typical rank of {5, 6}. We consider fitting CP to a generic array X of rank 6, with R = 5. For the rank criterion of Ten Berge et al. (2004), we need the following definitions. For a 3 × 3 × 5 array Y with symmetric 3 × 3 slices Y ₁, …,Y ₅, let the 9 × 5 matrix S contain the columnwise vecs of Y ₁, …,Y ₅. Let S ₅ be the 5 × 5 matrix consisting of rows 1, 2, 3, 5 and 6 of S. Define the 5 × 1 vector f as

$${{\rm{f}}^{_{^{\rm{T}}}}} = ({s_{91{\rm{ }}}}{s_{92{\rm{ }}}}{s_{93{\rm{ }}}}{s_{94}}){\rm{S}}_5^{ - 1}$$

(5.1)

, where s _ij denotes the (i, j)th element of S. Let the quadratic function P be defined as

$$P(u,\upsilon ) = - {\upsilon^2} + \upsilon ({f_3} + u{f_5}) + ({f_1} + u{f_2} + {u^2}{f_4})$$

(5.2)

, where f _j denotes the jth element of f. The rank criterion of Ten Berge et al. (2004) involves the real-valued roots (u, υ) of P(u,υ). It can be seen that for a real-valued pair (u, υ) with P(u,υ) = 0, the discriminant D(u) must be nonnegative, where

$$\eqalign{ & D(u) = {({f_3} + u{f_5})^2} + 4({f_1} + u{f_2} + {u^2}{f_4}) \cr & = {u^2}(f_5^2 + 4{f_4}) + u(2{f_3}{f_5} + 4{f_2}) + (f_3^2 + 4{f_1}) \cr} $$

(5.3)

.

Let D* = D(u*), where u* is the point at which the derivative of D is zero. Then

$$D* = {{ - {{({f_3}{f_5} + 2{f_2})}^2}} \over {(f_5^2 + 4{f_4})}} + (f_3^2 + 4{f_1})$$

(5.4)

.

We distinguish the following cases with respect to D(u) and D*:

• (a) f ²₅ + 4f₄ > 0,

• (b) f ²₅ + 4f₄ < 0 and D* > 0,

• (c) f ²₅ + 4f₄ < 0 and D* < 0,

• (d) f ²₅ + 4f₄ < 0 and D* = 0.

Next, we specify the sets R, D, and S. Let

(5.5)

(5.6)

(5.7)

The following lemma states the properties of the sets R, D and S that we need in our proof.

Lemma 5.1. Let the sets R, D, and S be given by (5.5–5.7).

• (i) A 3 × 3 × 5 array with symmetric slices lies in R almost everywhere.

• (ii) The arrays in R have at least rank 5.

• (iii) The set D consists of the arrays in R that have rank 5.

• (iv) The set D is an open subset of R and S is the closure of D within the set R.

• (v) A 3 × 3 × 5 array with symmetric slices and rank 6 lies in R\S almost everywhere.

Proof: First, we prove (i). A singular S₅ or the equality f ²₅ =−4f₄ both imply deterministic relations between the elements of Y, which are not implied by the symmetry of the slices. Hence, the complement of the set R has dimensionality lower than 30 (which is the dimensionality of the space of 3 × 3 × 5 arrays with symmetric slices). This implies (i).

The requirement that S5 is nonsingular implies that S has rank 5 and, hence, all arrays in R have at least rank 5. This proves (ii).

Ten Berge et al. (2004) show that an array Y ∈ R has rank 5 if there exist five distinct real-valued pairs (u, υ) such that P(u,υ) = 0. If this is not the case, then Y has at least rank 6. For each u with D(u) > 0 there are two values for υ such that P(u,υ) = 0. The continuity of D(u) implies that if D(t) > 0 for some t, then we can find five different u close to this t with D(u) > 0. Hence, there exist five distinct real-valued pairs (u, υ) with P(u,υ) = 0 if D(t) > 0 for some t. The latter holds for cases (a) and (b) above, but not for (c) and (d). This proves (iii).

The sets of arrays which satisfy (a), (b), or (c) all have positive volume, while the set of arrays satisfying (d) has zero volume. Hence, a generic array of rank 6 satisfies (c). This proves (v).

Since the value of D* in (5.4) depends continuously on the elements of the array Y, it follows that the boundary points of D in R are given by the arrays for which (d) holds. Indeed, such arrays can be approximated arbitrarily closely from D but also by arrays for which (c) holds. The boundary points of D do not lie in D itself and, hence, D is an open subset of R. The set S is the union of D and its boundary points in R. This proves (iv).

We consider the CP problem (3.1) where D is given by (5.6). The following theorem states our results.

Theorem 5.2. LetXbe a generic 3 × 3 × 5 array with symmetric slices and rank 6. Let D be given by (5.6).

• (I) Problem (3.1) has no optimal solution; equivalently, the objective function of problem (3.1) has no minimum, only an infimum.

• (II) Any sequence of CP updates in which the objective value of (3.1) converges to the infimum will become degenerate.

Proof: Statement (I) follows from the fact that D is an open subset of R and X ∉ D, see (iv) and (v) of Lemma 5.1.

Next we prove (II). It follows from (iv) of Lemma 5.1 that the infimum of (3.1) is attained at a boundary point ˜X of D that is an optimal solution of problem (3.2), where S is given by (5.7).We have to show that any sequence of CP updates Y in D, converging to ˜X, will become degenerate. For an array Y ∈ D, let P(u _j , υ _j ) = 0 for the real-valued and distinct pairs (u ₁, υ ₁), …, (u ₅, υ ₅). Ten Berge et al. (2004) show that a rank-5 CP decomposition (A,B,C) of Y is of the form

$${\rm{A}} = {\rm{B}} = \left[ {\matrix{ 1 & 1 & 1 & 1 & 1 \cr {{u_1}} & {{u_2}} & {{u_3}} & {{u_4}} & {{u_5}} \cr {{\upsilon _1}} & {{\upsilon _2}} & {{\upsilon _3}} & {{\upsilon _4}} & {{\upsilon _5}} \cr } } \right]{\rm{ and C}} = {\rm{S}}_5^{\rm{T}}{\left[ {\matrix{ 1 & {{u_1}} & {{\upsilon _1}} & {u_1^2} & {{u_1}{\upsilon _1}} \cr 1 & {{u_2}} & {{\upsilon _2}} & {u_2^2} & {{u_2}{\upsilon _2}} \cr 1 & {{u_3}} & {{\upsilon _3}} & {u_3^2} & {{u_3}{\upsilon _3}} \cr 1 & {{u_4}} & {{\upsilon _4}} & {u_4^2} & {{u_4}{\upsilon _4}} \cr 1 & {{u_5}} & {{\upsilon _5}} & {u_5^2} & {{u_5}{\upsilon _5}} \cr } } \right]^{ - 1}}$$

(5.8)

. We denote the function D(u) corresponding to the boundary array ˜X by ˜D(u). Since ˜X satisfies (d), the function ˜D(u) is a second-degree polynomial with a negative leading coefficient and a maximum of ˜D(ũ*) = 0. The function D(u) depends continuously on the elements of Y. Hence, if Y converges to ˜X, the function D(u) will converge (pointwise) to the function ˜D(u). This implies that if Y is close to ˜X, then D(u) will only be positive in a small interval around ũ*. As a consequence, the values of u ₁, …, u ₅ in (5.8) will become more and more alike. Moreover, since D(u _j ) converges to ˜D(ũ*) = 0, also the values of υ ₁, …, υ ₅ will become more and more alike. Hence, it follows from (5.8) that if Y is close to ˜X, then the columns in A = B will become more and more alike and the columns in C will become arbitrarily large. Also, it can be shown that the sum of the columns in C remains small and does not blow up. This is the pattern of a five-factor degeneracy. This completes the proof of (II).

Contrary to the cases analyzed so far (i.e., in Stegeman, 2006, 2007, and in the previous section), the CP decomposition (5.8) is not essentially unique. Indeed, there is freedom to choose the u _j from an interval on the real line. For arrays close to the boundary of D, however, this interval decreases in size until only one point ũ* is left at the boundary itself. This shows that essential uniqueness of the CP decomposition is not necessary for the occurrence of degenerate sequences of CP solutions.

As in the previous section, the CP solution (5.8) necessarily features A = B (provided A and B are scaled such that their first rows consist only of ones) (see Ten Berge et al., 2004). Hence, the CP model is equivalent to the Indscal model in this case, and the analysis above also explains the occurrence of degenerate sequences of Indscal solutions for 3 × 3 × 5 arrays of rank 6, with R = 5.

6 6. Fitting CP to Generic 5 × 3 × 3 Arrays

Here we prove Case 5 of Table 1. Ten Berge (2004) has shown that 5 × 3 × 3 arrays have a typical rank of {5, 6}. We consider fitting CP to a generic array X of rank 6, with R = 5. For the rank criterion of Ten Berge (2004), we need the following definitions. Let a 5 × 3 × 3 array Y have 5 × 3 slices Y ₁, Y ₂, and Y ₃. Ten Berge and Kiers (1999) have shown that for a generic Y there exist nonsingular matrices S (5 × 5) and T (3 × 3) such that

$${\rm{S}}{{\rm{Y}}_{\rm{1}}}{\rm{T}} = \left[ {\matrix{ 1 & 0 & 0 \cr 0 & 1 & 0 \cr 0 & 0 & 1 \cr 0 & 0 & 0 \cr 0 & 0 & 0 \cr } } \right]{\rm{, S}}{{\rm{Y}}_{\rm{2}}}{\rm{T}} = \left[ {\matrix{ 0 & 0 & 0 \cr 0 & 0 & 0 \cr 1 & 0 & 0 \cr 0 & 1 & 0 \cr 0 & 0 & 1 \cr } } \right]{\rm{, S}}{{\rm{Y}}_{\rm{3}}}{\rm{T}} = \left[ {{\rm{f}}\left| {\rm{g}} \right|{\rm{h}}} \right]$$

(6.1)

, where the last slice can be treated as a generic 5 × 3 matrix. Moreover, for each array Y the matrices S and T can be uniquely determined. Note that the transformation (6.1) is rank-preserving. Let the polynomial P be given by

$$P(u) = {z_7}{u^7} + {z_6}{u^6} + {z_5}{u^5} + {z_4}{u^4} + {z_3}{u^3} + {z_2}{u^2} + {z_1}u + {z_0} = 0$$

(6.2)

, where the coefficients z _j depend continuously on f, g, and h and are given in the Appendix of Ten Berge (2004). If the transformation in (6.1) is possible and P is a seventh-degree polynomial, i.e., z ₇ ≠ 0, then P has one root −f ₂/f ₄, where f _j denotes the jth element of f, and six other roots u ₁, …, u ₆. Next, we specify the sets R, D, and S. Let

(6.3)

(6.4)

(6.5)

The following lemma states the properties of the sets R, D, and S that we need in our proof.

Lemma 6.1. Let the sets R, D, and S be given by (6.3–6.5).

• (i) A 5 × 3 × 3 array lies in R almost everywhere.

• (ii) The arrays in R have at least rank 5.

• (iii) The set D consists of the arrays in R that have rank 5.

• (iv) The set D is neither open nor closed in R and S is the closure of D in R.

• (v) A 5 × 3 × 3 array of rank 6 lies in R\S almost everywhere.

Proof: The transformation (6.1) is possible almost everywhere and z₇ = 0 in (6.2) implies a deterministic relation between the elements of Y. This proves (i).

In a full CP decomposition (A,B,C) of (6.1) the columns of the first two transformed slices are contained in the column space of the matrix A. Hence, A must have at least five columns and thus the array must have at least rank 5 if it lies in R. This proves (ii).

Ten Berge (2004) shows that the array Y has rank 5 if there are five distinct real roots of P among u₁, …, u₆. If this is not the case, then the array has at least rank 6, see the extension of the analysis of Ten Berge (2004) by Stegeman (2005). This proves (iii).

The polynomial P has all roots different almost everywhere and all roots real on a set of positive volume. Also, complex roots of P occur on a set of positive volume. Hence, for a generic array of rank 6, the polynomial P will have some complex roots. This proves (v).

The roots of P depend continuously on the coefficients of P, which depend continuously on the elements of f, g, and h, which in turn depend continuously on the elements of the array Y. Therefore, the boundary points of D are those arrays for which the roots u ₁, …, u ₆ of P are real, but not all distinct. Indeed, such arrays can be approximated arbitrarily closely from D and by arrays for which P has some complex roots, see Lemma 2 in Stegeman (2006). Some of the boundary points (those for which the roots u ₁, …, u ₆ contain only one identical pair) lie in D, while others (those for which there are no five distinct roots among u ₁, …, u ₆) do not lie in D. Hence, the set D is neither open nor closed. The set S is the union of D and its boundary points in R. This proves (iv).

We consider the CP problem (3.1) where D is given by (6.4). The following theorem states our results.

Theorem 6.2. LetXbe a generic 5 × 3 × 3 array with rank 6. Let D and S be given by (6.4) and (6.5), respectively. Suppose problem (3.2) has no optimal solution lying in D.

• (I) Problem (3.1) has no optimal solution; equivalently, the objective function of problem (3.1) has no minimum, only an infimum.

• (II) Any sequence of CP updates in which the objective value of (3.1) converges to the infimum will become degenerate.

Proof: Statement (I) follows from (iv) and (v) of Lemma 6.1 and the assumption that problem (3.2) has no optimal solution lying in D.

Next we prove (II). Let ˜X be an optimal solution of problem (3.2) which does not lie in D. We have to show that any sequence of CP updates Y in D, converging to ˜X, will become degenerate. For an array Y ∈ D, Ten Berge (2004) shows that a CP decomposition (A,B,C) of Y is of the form

(6.6)

, where u ₁, …, u ₅ are five distinct real roots of P in (6.2) and r(u) = r(u|Y) and q(u) = q(u|Y) are well-defined continuous functions of u, depending continuously on the elements of Y. Note that if the roots u ₁, …, u ₆ of P are real and distinct, then for each group of five roots a CP solution (6.7) is possible. Hence, there exist six possible CP solutions, each two of which share four roots of P. This partial uniqueness phenomenon is the main result of Ten Berge (2004).

Let the array ˜X have a polynomial ˜P with roots ũ ₁, …, ũ ₆. Since ˜X lies in S\D, the roots ũ ₁, …, ũ ₆ are real, but do not contain a group of five distinct roots. Let the array Y ∈ D converge to ˜X. The roots u ₁, …, u ₆ of P corresponding to Y depend continuously on the elements of Y. Therefore, the roots u ₁, …, u ₆ will converge to the roots ũ ₁, …, ũ ₆. Hence, when Y is close to ˜X, any group of five roots out of u ₁, …, u ₆ will contain at least two nearly identical roots. This implies that in the CP solution (6.7), the corresponding columns of B will be nearly identical, the corresponding columns of C are also nearly identical, and the corresponding columns in A become arbitrarily large. Also, it can be shown that the sum of these columns in A remains small and does not blow up. Clearly, this is the pattern of a degenerate sequence of CP solutions as described in Section 1. This completes the proof of (II).

If there exists an optimal solution ˜X of problem (3.2) which lies in D, then ˜X is also an optimal solution of problem (3.1) and the CP objective function in (3.1) has a minimum. However, as shown in Theorem 6.2, if no optimal solution of problem (3.2) lies in D, then problem (3.1) has no optimal solution and the CP objective function in (3.1) does not have a minimum. Both situations are true on sets of positive volume.

6.1 6.1. Simulation Results

Here we consider the proportion of degeneracies encountered when fitting CP with R = 5 to generic 5 × 3 × 3 arrays X of rank 6. We consider three categories of such arrays X, namely for which P in (6.2) has two, four, or six complex roots. We calculate the rank-5 approximation of 10 arrays of each category. For each array X we use 10 different (random) starting values for the component matrices A, B, and C. After the algorithm terminates, we use the optimal component matrices for each run to calculate the rank-5 arrays Y* closest to X. Since a 5 × 3 × 3 array of rank 5 has six possible rank-5 decompositions (see above), we focus on the optimal arrays Y* rather than on the component matrices. As a CP algorithm, we use the Multilinear Engine by Paatero (1999).

In a majority of cases (254 out of 300) all 10 runs for one array X yield approximately the same solution Y*. In the other 46 cases the algorithm terminates in a local minimum. We discard the outcomes of these 46 runs and will speak of the CP solution Y* for a certain array X. Notice that this indicates that, usually, problem (3.2) has a unique optimal solution ˜X.

As we expected, all solution arrays Y* lie in R, i.e., the transformation (6.1) is possible and the polynomial P* associated with Y* has degree seven. In Table 2, the frequencies of the different types of solutions can be found for each category of arrays X. All polynomials P* have some nearly identical roots, which is in agreement with our analysis above. If P* has only two nearly identical roots (apart from −f ₂/f ₄), then both nondegenerate and degenerate rank-5 decompositions of Y* are possible, and the latter occur about twice as often as the former. This corresponds to the case where ˜X ∈ D. If P* has more nearly identical roots, then only degenerate rank-5 decompositions of Y* exist. This corresponds to the case where ˜X ∉ D.

Table 2 Frequencies of different types of CP solutions resulting from fitting CP with R = 5 to generic 5 × 3 × 3 arrays X of rank 6. Of each category, 10 different arrays are considered.

7 7. Fitting CP to Generic 8 × 4 × 3 Arrays

Here we prove Case 6 of Table 1. Ten Berge (2000) has shown that 8 × 4 × 3 arrays have a typical rank of {8, 9}. We consider fitting CP to a generic array X of rank 9, with R = 8. For the rank criterion of Ten Berge (2000), we need the following definitions. Let the 8 × 4× 3 array Y have 8×4 slices Y ₁, Y ₂, and Y ₃. Consider the transformation

$${\left[ {{{\rm{Y}}_1}|{{\rm{Y}}_2}} \right]^{ - 1}}{{\rm{Y}}_1} = \left[ {\matrix{ {{{\rm{I}}_4}} \cr {\rm{O}} \cr } } \right],{\rm{ }}{\left[ {{{\rm{Y}}_1}|{{\rm{Y}}_2}} \right]^{ - 1}}{{\rm{Y}}_2} = \left[ {\matrix{ {\rm{O}} \cr {{{\rm{I}}_4}} \cr } } \right],{\rm{ }}{\left[ {{{\rm{Y}}_1}|{{\rm{Y}}_2}} \right]^{ - 1}}{{\rm{Y}}_3} = \left[ {\matrix{ {{\rm{W}}_1^{\rm{T}}} \cr {{\rm{W}}_2^{\rm{T}}} \cr } } \right]$$

(7.1)

, where W ₁ and W ₂ are 4 × 4 matrices. Next, we specify the sets R, D, and S. Let

$$ R = \left\{ {\underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle-}$}}{Y} : \left[ {\underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle-}$}}{Y} _1 |\underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle-}$}}{Y} _2 } \right] is nonsingular} \right\} $$

(7.2)

, and associated eigenvector k _j such that

(7.3)

,

(7.4)

.

The following lemma identifies the boundary points of D in R. For ease of presentation, its proof is postponed until the end of this section.

Lemma 7.1. The boundary points of D in R are those arrays for which (α _j W₁ + β _j W₂) has a real eigenvalue only for a finite number of nonproportional (α _j ,β _j ) ≠ (0, 0).

The following lemma states the properties of the sets R, D, and S that we need in our proof.

Lemma 7.2. Let the sets R, D, and S be given by (7.2–7.4).

• (i) An 8 × 4 × 3 array lies in R almost everywhere.

• (ii) The arrays in R have at least rank 8.

• (iii) The set D consists of the arrays in R that have rank 8.

• (iv) The set D is neither open nor closed in R and S is the closure of D in R.

• (v) An 8 × 4 × 3 array of rank 9 lies in R\S almost everywhere.

Proof: The proof of (i) is trivial. In a full CP decomposition (A,B,C) of (7.1) the columns of the first two transformed slices lie in the column space of the matrix A. Hence, A must have at least eight columns and thus the array must have at least rank 8 if it lies in R. This proves (ii).

Ten Berge (2000) shows that an array Y ∈ D has rank 8 and its full CP decomposition (A,B,C) is necessarily of the form

(7.5)

. This implies (iii).

Lemma 7.1 identifies the boundary points of D in R. Let Y be such a boundary point. Then the real eigenvalues of (α _j W ₁ + β _j W ₂) form identical pairs. Indeed, Lemma 2 of Stegeman (2006) shows that if this is not the case, then ((α _j + ε)W ₁ + (β _j + ε)W ₂) has a real eigenvalue for any ε small enough. Moreover, Lemma 2 of Stegeman (2006) shows that there is only one eigenvector associated with each pair of identical real eigenvalues almost everywhere on the boundary set. Hence, for the boundary array Y to lie in D, we need (almost everywhere) at least four nonproportional (α _j ,β _j ) such that (α _j W ₁ +β _j W ₂) has a real eigenvalue, see (7.5). It follows that not all boundary points necessarily lie in D and, thus, D is neither open nor closed. The set S in (7.4) is the union of D and its boundary points. This proves (iv).

The discussion of the boundary set of D above, implies that the boundary of D has zero volume. Hence, a generic array of rank 9 lies in the set R\S. This proves (v).

We consider the CP problem (3.1) where D is given by (7.3). The following theorem states our results.

Theorem 7.3. LetXbe a generic 8 × 4 × 3 array with rank 9. Let D and S be given by (7.3) and (7.4), respectively. Suppose problem (3.2) has no optimal solution lying in D.

• (I) Problem (3.1) has no optimal solution; equivalently, the objective function of problem (3.1) has no minimum, only an infimum.

• (II) Any sequence of CP updates in which the objective value of (3.1) converges to the infimum will become degenerate.

Proof: Statement (I) follows from (iv) and (v) of Lemma 7.2 and the assumption that problem (3.2) has no optimal solution lying in D.

Next we prove (II). Let ˜X be an optimal solution of problem (3.2) which does not lie in D.We have to show that any sequence of CP updates Y in D, converging to ˜X, will become degenerate. The array ˜X is a boundary point of D and satisfies the requirement of Lemma 7.1. Let thematrices ˜W ₁ and ˜W ₂ correspond to ˜X and the matrices W ₁ and W ₂ correspond to Y. Let (α ˜W ₁ + β ˜W ₂) have a real eigenvalue, in fact, a pair of real eigenvalues, only for (α,β) = γ (α ₁, β ₁), where γ is a nonzero constant. We assume α ₁ ≠ 0, set γ = α ⁻¹₁ and define ˜β ₁ = α ⁻¹₁ β ₁. Hence, γ (α ₁,β ₁) = (1, ˜β ₁). The (complex parts of the) eigenvalues of a matrix depend continuously on the elements of the matrix. Hence, for Y close enough to ˜X, the matrix (W ₁ + β W ₂) has a real eigenvalue only for β close to ˜β ₁. Therefore, in a rank-8 decomposition (7.5) of Y close to ˜X, all values of β _j are close to ˜β ₁. Hence, the corresponding columns of C are nearly identical. Moreover, since (˜W ₁ + ˜β ₁ ˜W ₂) has at most two linearly independent eigenvectors corresponding to the pair of identical real eigenvalues, the matrix B will consist of one or two groups of nearly identical columns (up to a sign change). As can be seen from (7.5), the corresponding groups of columns of A become arbitrarily large. Also, it can be shown that the sum of the columns in these groups (for each column up to a sign change) remains small and does not blow up. Clearly, this is the pattern of a degenerate sequence of CP solutions as described in Section 1.

For a general boundary array ˜X not lying in D (i.e., with more pairs (˜ _j ,˜β _j ) or with ˜α ₁ = 0), an analogous proof can be given to show that the rank-8 decomposition (7.5) of Y will feature one or several groups of degenerate components. This completes the proof of (II).

Note that, as in Case 4 of Table 1, the CP decomposition (7.5) is not essentially unique almost everywhere on D. Indeed, for Y ∈ D, we may (almost everywhere) pick the values for (α _j ,β _j ) from one or several continuous intervals. Hence, this is another example showing that essential uniqueness of the CP decomposition is not necessary for the occurrence of degenerate sequences of CP solutions.

If there exists an optimal solution ˜X of problem (3.2) which lies in D, then ˜X is also an optimal solution of problem (3.1) and the CP objective function in (3.1) has a minimum. However, as shown in Theorem 7.3, if no optimal solution of problem (3.2) lies in D, then problem (3.1) has no optimal solution and the CP objective function in (3.1) does not have a minimum. We would like to ascertain which of these two situations occurs in practice.

Hence, for generic 8 × 4 × 3 arrays X of rank 9, i.e., for which no linear combination of W ₁ and W ₂ has a real eigenvalue, we would like to determine how often the optimal boundary point ˜X has rank 8 and how often it has rank 9. Unfortunately, we were unable to answer this question by means of numerical experiments. The problem is the following. If the CP algorithm terminates with an array Y, then we may assume that Y is close to the unknown optimal boundary point ˜X. It is our experience that, for Y, the linear combination (W ₁ + β W ₂) has a real eigenvalue for β in one or several intervals on the real line. However, although such an interval is small, we do not know if it splits up into several smaller intervals as Y→˜X or if it remains one interval and converges to a single point ˜β _j . Therefore, we cannot say whether the linear combination (˜W ₁ + β ˜W ₂) corresponding to ˜X has a real eigenvalue for one, two, three, or even more values for β. As a consequence, we cannot say whether a rank-8 decomposition (7.5) is possible for ˜X or not.

In our numerical experiments, however, all runs did result in a degenerate sequence of CP solutions. Still, this does not mean that all optimal boundary points ˜X have rank 9 or higher. Indeed, suppose a nondegenerate rank-8 decomposition (7.5) is possible. Then the CP solution given by the CP algorithm may still be a degeneracy if it does not feature the right β’s for a nondegenerate rank-8 decomposition. This explains the question mark for Case 6 in Table 1.

It remains to give the proof of Lemma 7.1.

Proof of Lemma 7.1: Let Z be an array for which (αW₁ + βW₂) has a real eigenvalue only for (α, β) = γ (1,β₁), where we set γ = 1. Next, we prove that Z is a boundary point of D by showing that Z can be approximated arbitrarily closely from D and from its complement R\D.

The eigenvalues of a matrix depend continuously on the elements of the matrix. This implies that a sequence of matrices Y ⁽ⁿ⁾ in D can be constructed, which converges to Z. Indeed, the sequence can be chosen such that (W ⁽ⁿ⁾₁ + β W ⁽ⁿ⁾₂ ) has a real eigenvalue only for β in a small interval around β ₁. As Y ⁽ⁿ⁾→Z, the interval will become smaller and converge to the point β ₁. Moreover, the eigenvalues of (W ⁽ⁿ⁾₁ + β ₁ W ⁽ⁿ⁾₂ ) will converge to the eigenvalues of (W ₁+β ₁ W ₂).

The real eigenvalues of (W ₁+β ₁ W ₂) occur in identical pairs. Indeed, Lemma 2 in Stegeman (2006) shows that if this is not the case, then (W ₁ + (β ₁ + ε)W ₂) has a real eigenvalue for any ε small enough. Moreover, Lemma 2 in Stegeman (2006) shows that (W ₁ + β ₁ W ₂) can be approximated arbitrarily closely by 4 × 4 matrices having only complex eigenvalues. This implies that Z can be approximated arbitrarily closely by arrays in R\D for which (α W ₁ + β W ₂) has no real eigenvalues for all (α,β) ≠ (0, 0). Hence, Z is a boundary point of D in R. The proof for a general array Z satisfying the requirement of Lemma 7.1 is analogous.

The reasoning above can also be used to show that arrays for which (α W ₁ + β W ₂) has no real eigenvalues for all (α,β) ≠ (0, 0), and arrays for which (α W ₁ +β W ₂) has a real eigenvalue for all (α,β) in some continuous interval, cannot be boundary points of D. This completes the proof.

8 8. Discussion

We have presented results on fitting the CP model to all known arrays with a two-valued typical rank. For a typical rank of {m, m + 1}, we have considered the CP model with R = m for a generic array of rank m + 1. For 3 × 3 × 4 and 3 × 3 × 5 arrays with symmetric slices and for 8 × 4 × 3 arrays, this always results in a degenerate sequence of CP solutions. For 5 × 3 × 3 arrays, this is sometimes the case. We showed that all degenerate sequences of CP solutions are due to the fact that the set of rank-m arrays is not closed. In particular, the sequence of CP solutions converges to an array ˜X on the boundary between the sets of rank-m and rank-(m + 1) arrays. This array ˜X has rank m + 1 or higher and is an optimal solution of problem (3.2). This implies that the CP problem does not have an optimal solution and the CP objective function does not have a minimum. Moreover, we showed that if the sequence of CP solutions gets close to ˜X, then the CP decomposition necessarily becomes degenerate. This confirms the idea of Kruskal et al. (1989) about the occurrence of degenerate sequences of CP solutions and extends the analysis of Stegeman (2006, 2007).

Our approach to proving whether or not degenerate sequences of CP solutions occur is the same for all cases mentioned above. The basic ideas are introduced in Stegeman (2006). This framework of analysis is likely to be of use for the study of other arrays having a two-valued typical rank (when they become known).

We have restricted our analysis to arrays in a set R, the complement of which has zero volume. Theoretically, this is justifiable since any array not in R can be approximated arbitrarily closely by arrays in R. However, this does not exclude the possibility that the CP algorithm yields a CP solution not lying in R. It is our experience that for the cases of Table 1, the occurrence of degenerate sequences of CP solutions (for a generic X of rank m + 1) is as indicated in the table. Moreover, in all runs the CP algorithm yields CP solutions lying in the set R. Therefore, we feel that our theoretical results have straightforward practical implications.

Note that the occurrence of degenerate sequences of CP solutions in Table 1 does not depend on the algorithm used to minimize the CP objective function. If the CP algorithm yields a sequence of CP updates converging to an optimal boundary point of D not lying in D itself, the sequence becomes degenerate. This also holds for modified CP algorithms designed to avoid degeneracy.

The occurrence of degeneracy in the cases in Table 1 and their explanations are still valid when the Frobenius norm in the CP objective function is replaced by any other norm (e.g., weighted least squares or Gaussian maximum likelihood). This is because all norms on the finite-dimensional vector space are equivalent and induce the same (i.e., the Euclidean) topology.

Zijlstra and Kiers (2002) observed that two-factor degeneracies occur not only in CP but also in other variants of factor analysis. They show that two- and three-way factor analysis models which yield degenerate sequences of solutions, necessarily have rotationally unique components. For the cases examined in Stegeman (2006, 2007), degeneracy always occurs together with essential uniqueness of the CP solution. However, in Cases 4 and 6 of Table 1 the CP decomposition is not essentially unique, but still degenerate sequences of CP solutions occur. This shows that CP uniqueness is not a necessary condition for the occurrence of degeneracy in CP.

The degeneracies described in this paper are due to the two-valued typical rank {m, m + 1} of the arrays. In Cases 1, 2, 3, and 5 in Table 1, the criterion to distinguish between rank-m and rank-(m + 1) arrays involves a polynomial P(u|Y) depending continuously on the elements of the array Y. If the roots of P(u|Y) are real and distinct, then Y has rank m. If P(u|Y) has some complex roots, then Y has rank m + 1. In Cases 4 and 6 of Table 1, we have a polynomial P _θ (u|Y) depending continuously on some parameter θ and on the elements of the array Y. If sufficiently many θ exist such that (some of) the roots of P _θ (u|Y) are real, then Y has rank m. Otherwise, it has rank m + 1.

In the cases analyzed so far, a two-valued typical rank, a polynomial rank criterion as above, and the occurrence of degenerate sequences of CP solutions, are ultimately connected to the topological properties of the sets of real-valued I × J × K arrays with rank at most R. Therefore, it is plausible to expect that this is the case for all existing real-valued I × J × K arrays with a two-valued typical rank. In the complex-valued CP model, two-valued typical ranks of this type do not exist, since we do not have to distinguish between real and complex roots of a polynomial. Therefore, the degeneracies described in this paper do not occur in the complex-valued CP model. However, also in the complex case, the CP objective function does not always have a minimum. See the example in De Silva and Lim (2006, Proposition 4.6), which carries over to the complex case. Whether degenerate sequences of CP solutions occur on sets of positive volume in the complex-valued CP model is still an open problem.