On the interconnection between the higherorder singular values of real tensors
 943 Downloads
 4 Citations
Abstract
A higherorder tensor allows several possible matricizations (reshapes into matrices). The simultaneous decay of singular values of such matricizations has crucial implications on the lowrank approximability of the tensor via higherorder singular value decomposition. It is therefore an interesting question which simultaneous properties the singular values of different tensor matricizations actually can have, but it has not received the deserved attention so far. In this paper, preliminary investigations in this direction are conducted. While it is clear that the singular values in different matricizations cannot be prescribed completely independent from each other, numerical experiments suggest that sufficiently small, but otherwise arbitrary perturbations preserve feasibility. An alternating projection heuristic is proposed for constructing tensors with prescribed singular values (assuming their feasibility). Regarding the related problem of characterising sets of tensors having the same singular values in specified matricizations, it is noted that orthogonal equivalence under multilinear matrix multiplication is a sufficient condition for two tensors to have the same singular values in all principal, Tuckertype matricizations, but, in contrast to the matrix case, not necessary. An explicit example of this phenomenon is given.
Mathematics Subject Classification
15A18 15A21 15A691 Introduction and problem statement
A space \({\mathbb {R}}^{n_{1}}\otimes \dots \otimes {\mathbb {R}}^{n_{d}}\) of higherorder tensors is isomorphic to many different matrix spaces of the form \(( \bigotimes _{j\in t}{\mathbb {R}}^{n_{j}}) \otimes (\bigotimes _{i\notin t}{\mathbb {R}}^{n_{i}})\) where \(t\subsetneq \{1,\dots ,d\}\), \(t\ge 1\). Concretely, when identifying tensors with ddimensional arrays of coordinates with respect to an orthonormal tensor product basis, such an isomorphism is realized by reshaping the array into a matrix. The directions in t indicate the multiindices for the rows of the resulting matrix, while the other directions are used for the columns. All these different matricizations (also called unfoldings or reshapes in the literature) of the tensor carry some spectral information in form of their singular value decompositions.
For subsets of t that are part of a dimension partition tree, the column spaces of the corresponding matricizations satisfy certain nestedness properties that form the basis for important subspace based lowrank tensor decompositions like the Tucker format [22], the hierarchical Tucker (HT) format [7, 9], or the tensor train (TT) format [18, 19]. As a byproduct, the ranks \(r_t\) of the corresponding matricizations, that is, the number of nonzero singular values, are estimated as \(r_t \le r_{t_1} \cdots r_{t_s} \), where \(t = t_1 \cup \dots \cup t_s\) is a disjoint partition. In contrast, the interconnections between the singular values themselves have not been studied so far.
At first sight, the singular values of different matricizations could be considered as unnatural or artificial characteristics for tensors, as they ignore their multilinear nature. However, as it turns out, they provide crucial measures for the approximability of tensors in the aforementioned lowrank subspace formats. In the pioneering work [3] the higherorder singular value decomposition has been defined, and it has been shown how it can be practically used to obtain quasioptimal lowrank approximations in the Tucker format with full error control. The approximation is obtained by an orthogonal projection on the tensor product of subspaces spanned by the dominating singular vectors of the corresponding matricizations in \({\mathbb {R}}^{n_{j}}\otimes ( \bigotimes _{i\ne j}{ \mathbb {R}}^{n_{i}}) \) (i.e. corresponding to the choices \(t=\{j\}\) for \(j=1,\dots ,d\)). An upper bound of the squared error is then given by the sum of squares of all deleted singular values in all directions. Later, variants of such truncation procedures have been obtained for the TT format [16, 17] and the HT format [7] with similar error bounds, but involving singular values of some other matricizations of the tensor.
Building on these available, quasioptimal bounds for lowrank approximations via higherorder versions of SVD truncation, it is understandable that quite some theorems have been stated making simultaneous assumptions on the singular values of certain matricizations of a tensor. This concerns stability of lowrank ODE integrators [12, 14], local convergence of optimization algorithms [20], or approximability by lowrank tensor formats [1], to name a few. Assumed properties of interest are decay rate of and gaps between the singular values, for instance. A principal task would then be to give alternative descriptions of classes of tensors satisfying such assumptions to prevent tautological results or, in worst case, void statements. But this task has turned out to be notoriously difficult. For tensors arising from function discretization, some qualitative statements about the decay of singular values can be obtained from their regularity using explicit analytic approximation techniques by tensor products of (trigonometric) polynomials or wavelets, exponential sum, or cross approximation; see [8, 21] and references therein. But these qualitative implications on the decay of singular values obtained from explicit separable approximations can rarely be made quantitatively precise, for instance, if they contain unknown constants, and also provide little insight on the actual interconnection between different matricizations.
In its purest form the question we are interested in is very simple to state. Given prescribed singular values for some matricizations (having, e.g., some favourable properties), does there exist at all a tensor having these singular values? For a matrix this is of course very easy to answer by simply constructing a diagonal matrix. For tensors it turns out to be quite difficult, and seems to depend on how many matricizations are simultaneously considered.

We show that not all configurations of higherorder singular values can be feasible. The proof is nonconstructive (Sect. 3.1).

However, conducted numerical experiments suggest that the singular values for different matricizations are, except for degenerate situations, locally independent from each other. That is, in the neighbourhood of a tensor it is possible to slightly perturb, say, only the singular values of the first matricization, while maintaining the singular values of the other ones. This is fundamentally different from the matrix case, since the singular values of a matrix are always the same as the ones of its transpose. However, currently this remains an unproved conjecture Sect. 3.2).

We propose the method of alternating projections as a heuristic to construct (approximately) tensors with prescribed singular values in certain matricizations (Sect. 3.3).

The higherorder SVD (HOSVD) is a generalization of the SVD from matrices to tensors. The role of the diagonal matrix of singular values is replaced by the core tensor in the HOSVD, representing the normal form under orthogonal equivalence, and characterized by slicewise orthogonality properties. We show manifold properties of the set of these core tensors (called HOSVD tensors) in the case of strictly decreasing and positive higherorder singular values (Sect. 2.2).

We provide an example of two \(2\times 2\times 2\) tensors having the same singular values in all three principal matricizations without being orthogonally equivalent (Sect. 2.4).
The rest of this section is devoted to the precise statements of the considered problems. They require some amount of definitions and notation, which will be introduced first.
1.1 Preliminaries, definitions, notation
Definition 1.1
 1.For \(j=1,\dots ,d\), the vectoris called the vector of modej singular values. The tuple$$\begin{aligned} {{\varvec{\sigma }}}^{(j)}_{\mathbf {X}} = {{\mathrm{{diag}}}}\left( \Sigma ^{(j)}_{\mathbf {X}}\right) = \left( \sigma ^{(j)}_{1},\dots ,\sigma ^{(j)}_{n_{j}}\right) \in {\mathbb {R}}^{n_{j}}_{+} \end{aligned}$$is called the set of higherorder singular values of the tensor \({\mathbf {X}}\).$$\begin{aligned} {\varvec{\Sigma }}_{\mathbf {X}} = \left( {\varvec{\sigma }}^{(1)}_{\mathbf {X} },\dots ,{\varvec{\sigma }}^{(d)}_{\mathbf {X}}\right) \in {\mathbb {R}}^{n_{1}}_{+} \times \dots \times {\mathbb {R}}^{n_{d}}_{+} \end{aligned}$$
 2.Correspondingly, for \(j=1,\dots ,d\), the vectoris called the vector of modej Gramian eigenvalues. The tuple$$\begin{aligned} {{\varvec{\lambda }}}^{(j)}_{\mathbf {X}} = {{\mathrm{{diag}}}}\left( \Lambda ^{(j)}_{\mathbf {X} }\right) = \left( \left( \sigma ^{(j)}_{1}\right) ^{2},\dots ,\left( \sigma ^{(j)}_{n_{j}}\right) ^{2}\right) \in {\mathbb {R} }^{n_{j}}_{+} \end{aligned}$$is called the set of higherorder Gramian eigenvalues of the tensor \({\mathbf {X}}\).$$\begin{aligned} {\varvec{\Lambda }}_{\mathbf {X}} = \left( {\varvec{\lambda }}^{(1)} _{\mathbf {X}},\dots ,{\varvec{\lambda }}^{(d)}_{\mathbf {X}}\right) \end{aligned}$$
 3.
The multilinear rank of the tensor \({\mathbf {X}}\) is the tuple \({\mathbf {r} }_{\mathbf {X}} = (r^{(1)},\dots ,r^{(d)})\) with \(r^{(j)} = {{\mathrm{{rank}}}}( M_{{\mathbf {X}}}^{(j)} ) = {{\mathrm{{rank}}}}( G_{{\mathbf {X}}}^{(j)} )\) being equal to the number of nonzero entries of \({\varvec{\sigma }}^{(j)}_{\mathbf {X}}\).
 4.
The tensor \({\mathbf {X}}\) is called nonsingular, if \({\mathbf {r}}_{\mathbf {X}} = (n_{1},\dots ,n_{d})\).
We note that for matrices the definition of ‘nonsingular’ coincides with the usual definition (in particular, it enforces \(n_{1} = n_{2}\)). In general, the following is true.
Proposition 1.2
Proof
Consider j fixed. By isomorphy and known results on matrices, it is clear that the set of all \({\mathbf {X}}\) with \( M_{{\mathbf {X}}}^{(j)} \) being of rank \(n_j\) is not empty, open, and dense if and only if \(n_j \le n_j^c\). The set of nonsingular tensors is the intersection of these sets for \(j=1,\dots ,d\). As such, it is also open and dense. \(\square \)
The following two facts are useful to know, and follow immediately from the matrix case.
Proposition 1.3
The function \({\mathbf {X}} \mapsto {\varvec{\Sigma }}_{\mathbf {X}}\) is continuous on \({\mathscr {S}}\). Assuming (1.2), the set \({\mathscr {S} }^{*}\) is relatively open and dense in \({\mathscr {S}}\).
Proof
The continuity of \({\varvec{\Sigma }}_{\mathbf {X}}\) as a function of \({\mathbf {X}}\) follows by isomorphy to \({\mathbb {R}}^{n_j \times n_j^c}\) from the continuity of each \({\varvec{\sigma }}^{(j)}_{\mathbf {X}}\) as a function of \( M_{{\mathbf {X}}}^{(j)} \). The proof that \({\mathscr {S}}^{*}\) is relatively open and dense in \({\mathscr {S}}\) is analogous to the proof of Proposition 1.2. \(\square \)
1.2 Problem statement
Regarding the higherorder singular values of tensors a principle question of interest is the following one.
Problem 1.4
(Feasible higherorder singular values) Given \({\varvec{\Sigma }} \in {\mathfrak {S}}_{\ge }\), does there exist a tensor \({\mathbf {X}} \in {\mathscr {S}}\) such that \({\varvec{\Sigma }}_{\mathbf {X}} = {\varvec{\Sigma }}\)?
A relaxed question of a more qualitative nature is the following one.
Problem 1.5
(Properties of \({\mathfrak {F}}\)) What are the topological properties of the set \({\mathfrak {F}}\) as a subset of \({\mathfrak {S}}_{\ge }\)? Does it, for instance, have positive (relative) Lebesgue measure?
Numerical experiments with random tensors seem to indicate that the answer to the second question could be positive when \(d\ge 3\), but we are not able to prove it. So it remains a conjecture. In fact, we conjecture that for every \(\mathbf {X}\) in \(\mathscr {S^{*}}\) it holds that \(\varvec{\Sigma }_{\mathbf {X}}\) is an interior point of \(\mathfrak {F}\), see Sect. 3.2. A striking implication of this conjecture is that given \({\mathbf {X}} \in \mathscr {S}^{*}\), its highorder singular values in different directions can be perturbed independently from each other without loosing feasibility (local independence of highorder singular values). In Sect. 3.3 we will present a heuristic approach to do this using an alternating projection method, which seems to work quite reliably for small perturbations, although we are currently neither able to prove its convergence nor that limit points must have the desired property.
To approach Problems 1.4 and 1.5, it seems useful to also study the following problem, which is of some interest in itself.
Problem 1.6
(Tensors with same higherorder singular values) Given \({\mathbf {X}} \in {\mathscr {S}}\), characterize sets of tensors having the same singular values \({\varvec{\Sigma }}_{\mathbf {X}}\) as \({\mathbf {X}}\).
2 Tensors with the same higherorder singular values
In this section we focus on equivalence classes of tensors having the same higherorder singular values.
2.1 Orthogonally equivalent tensors
Definition 2.1
(see [5]) Two tensors \({\mathbf {X}},{\mathbf {Y}} \in {\mathbb {R}}^{n_{1} \times \dots \times n_{d}}\) are called orthogonally equivalent, if there exists \((U^{(1)},\dots ,U^{(d)}) \in {\mathrm {O}}(n_{1} \times \dots \times n_{d})\) such \({\mathbf {Y}} {=} (U^{(1)},\dots ,U^{(d)})~\cdot {\mathbf {X}}\).
From (2.2), we draw a trivial but important conclusion.
Proposition 2.2
If two tensors are orthogonally equivalent, then they have the same higherorder singular values.
In particular, the orbit of each \({\mathbf {X}}\) under the group action contains only tensors with identical higherorder singular values.
Proposition 2.3
Proof
We write \({\mathrm {O}}\) instead of \({\mathrm {O}}(n_1 \times \dots \times n_d)\). Consider the canonical map \(\theta _{\mathbf {X}}:{\mathrm {O}}\rightarrow {\mathscr {S}}^{*}\), \((U^{(1)},\dots ,U^{(d)}) \mapsto (U^{(1)},\dots ,U^{(d)}) \cdot {\mathbf {X}}\), whose image is \({\mathrm {O}}\cdot {\mathbf {X}}\). Since \(\theta _{\mathbf {X}}\) is of constant rank [6, §16.10.2] and easily shown to be locally injective (uniqueness of left singular vectors up to sign flipping for \({\mathbf {X}}\in {\mathscr {S}}^{*}\)), it is already an immersion [6, §16.8.8.(iv)]. The result is now standard, see, e.g., [6, §16.8.8.(ii)]. \(\square \)
For \({\mathbf {X}} \in {\mathscr {S}} \setminus {\mathscr {S}}^{*}\) the dimension of \({\mathrm {O}}(n_{1} \times \dots \times n_{d}) \cdot {\mathbf {X}}\) can be smaller than \(\dim {\mathrm {O}}(n_{1} \times \dots \times n_{d})\). Note that we did not attempt to prove or disprove that the orbits are globally embedded submanifolds.
2.2 HOSVD tensors
The compact Lie group \({\mathrm {O}}(n_{1} \times \dots \times n_{d})\) acts freely on \({\mathscr {S}}^{*}\). It also acts properly (since it is compact and acts continuously). By a general theorem (e.g. [6, § 16.10.3]), the quotient manifold \({\mathscr {S}}^{*} / {\mathrm {O}} (n_{1} \times \dots \times n_{d})\) of equivalence classes exists, and the canonical mapping \({ \mathscr {S}}^{*} \rightarrow {\mathscr {S}}^{*} / {\mathrm {O}}(n_{1} \times \dots \times n_{d})\) is a submersion. A concrete realization of this abstract quotient manifold is the set \({\mathscr {H}}^{*}\) of regular HOSVD tensors which is now introduced.
Definition 2.4
Tensors satisfying (2.3) are called HOSVD tensors. The subset of HOSVD tensors in \({\mathscr {S}}\) is denoted by \({\mathscr {H}}\), and the subset of HOSVD tensors in \(\mathscr {S}^{*}\) by \({\mathscr {H}}^{*} = {\mathscr {H}} \cap {\mathscr {S}}^{*}\).
HOSVD tensors can be regarded as representatives of orbits \({\mathrm {O}} (n_{1} \times \dots \times n_{d}) \cdot {\mathbf {X}}\) of orthogonally equivalent tensors. For \({\mathbf {X}} \in {\mathscr {S}}^{*}\), the representatives are essentially unique as stated next. Here it is instructive to note that for square matrices, the set \(\mathscr {H}^{*}\) consists of regular diagonal matrices with strictly decreasing diagonal entries.
Proposition 2.5
Let \({\mathbf {X}}, {\mathbf {Y}} \in {\mathscr {H}}^{*}\) be two HOSVD tensors. If \({\mathbf {X}}\) and \({\mathbf {Y}}\) are orthogonally equivalent, that is, \({\mathbf {Y}} = (U^{(1)},\dots ,U^{(d)}) \cdot {\mathbf {X}}\), then the \(U^{(j)}\) must be diagonal orthogonal matrices (i.e. with values \(\pm 1\) on the diagonal).
The proof is immediate from (2.2), (2.3), and the uniqueness of orthogonal diagonalization up to sign flipping in the case of mutually distinct eigenvalues. Comparing with the explicit form (2.1), we see that the action of \((U^{(1)}, \dots , U^{(d)}) \cdot {\mathbf {X}}\) with diagonal \(U^{(j)}\) with \(\pm 1\) entries results in some sign flipping pattern for the entries of \({\mathbf {X}} \). This provides the following, sometimes useful necessary condition.
Proposition 2.6
If two HOSVD tensors \({\mathbf {X}}, {\mathbf {Y}} \in {\mathscr {H}}^{*}\) are orthogonally equivalent, then \(X_{i_{1},\dots ,i_{d}}= Y_{i_{1},\dots ,i_{d}}\). In particular, \({\mathbf {X}}\) and \({\mathbf {Y}}\) have the same zero pattern.
We now turn to the manifold properties of \({\mathscr {H}}^{*}\).
Theorem 2.7
Proof
Remark 2.8
In our definition (2.3) of HOSVD tensors we required the diagonal elements of \(G_{\mathbf {X}}^{(j)}\) to be decreasing. This has advantages and drawbacks. One advantage are the narrower uniqueness properties leading to the practical condition in Proposition 2.6. A disadvantage is that it is more difficult to design HOSVD tensors “by hand” as in Sect. 2.4. Alternatively, one may define a set \(\tilde{\mathscr {H}}\) by just requiring the \(G_{\mathbf {X}}^{(j)}\) to be diagonal. Then for every \({\mathbf {X}}\in \tilde{\mathscr {H}}\) we have \((P^{(1)},\dots ,P^{(d)}) \cdot {\mathbf {X}}\in \mathscr {H}\), where \(P^{(j)}\) are permutation matrices that sort the diagonal entries of \(G_{\mathbf {X}}^{(j)}\) accordingly. For mutually distinct eigenvalues the choice of \(P^{(j)}\) is unique. The corresponding set \(\tilde{\mathscr {H}}^{*}\) is therefore the finite disjoint union of sets \((P^{(1)},\dots ,P^{(d)}) \cdot \mathscr {H}^{*}\) over all \(P^{(j)}\), and as such also an embedded submanifold of \(\mathscr {S}^{*}\).
2.3 Degrees of freedom
A principal challenge in understanding the interconnection between higherorder singular values of tensors arises from the fact that, in contrast to the matrix case, the converse statement of Proposition 2.2 is in general not true when \(d \ge 3\). Tensors may have the same higherorder singular values without being orthogonally equivalent. This can be seen from the following heuristic.
2.4 A nonequivalent example
The previous considerations suggest that there must exist tensors having the same higherorder singular values without being orthogonally equivalent. We construct here an example of size \(2 \times 2 \times 2\) using Proposition 2.6. Let us shortly count the degrees of freedom in this situation. The Euclidean unit sphere \({\mathscr {S} }\) is of dimension seven, the set \({\mathfrak {S}}_{\ge }\) of potential tuples of higherorder singular values is of dimension three, while orbits \({ \mathrm {O}}(2 \times 2 \times 2) \cdot {\mathbf {X}}\) of orthogonally equivalent tensors are of dimension at most three, too. This indicates for every \({ \mathbf {X}} \in {\mathscr {S}}\) an at least onedimensional set of nonequivalent tensor with same higherorder singular values.
3 The set of feasible configurations
The set \({\mathfrak {F}} = {\mathfrak {F}}(n_{1},\dots ,n_{d}) \subseteq {\mathfrak {S}} _{\ge }\) of feasible configurations has been defined in (1.2). In this section we investigate this set. A simple observation worth to mention is that \({\mathfrak {F}}\) is closed. This follows from Proposition 1.3 and the compactness of \({\mathscr {S}}\).
3.1 Not all configurations are feasible
Based on this fact, we can first give trivial examples of singular tensors for which the nonzero singular values in different directions are not independent of each other.
Lemma 3.1
Let \({\mathbf {X}}\) have multilinear rank \(\mathbf {r} = (r_{1},\dots ,r_{d})\). Assume \(r_{j} = 1\) for \(j \ge 3\). Then \(r_{1} = r_{2}\) and \(\{ \sigma ^{(1)}_{1},\dots , \sigma ^{(1)}_{r_{1}} \} = \{ \sigma ^{(2)}_{1},\dots , \sigma ^{(2)}_{r_{2}} \}\).
Proof
Let \(\mathbf {C} \in {\mathbb {R}}^{r_1 \times \dots \times r_d}\) be the economic HOSVD core tensor of \({\mathbf {X}}\). The matricizations \( M_{\mathbf {C}}^{(j)} \) for \(j \ge 3\) are just row vectors and have only one singular value which equals the Frobenius norm of \(\mathbf {X}\). On the other hand, we have \( M_{\mathbf {C}}^{(1)} = ( M_{\mathbf {C}}^{(2)} )^{\mathsf {T}}\) (up to possible permutations), which implies the result. \(\square \)
Since tensors with \(r_{j} = 1\) for \(j \ge 3\) considered in the previous lemma are naturally identified as elements of \({\mathbb {R}}^{n_{1}} \otimes { \mathbb {R}}^{n_{2}}\), that is, as matrices, the previous statement may appear rather odd at first. However, using a perturbation argument, it leads to a nonconstructive proof that nonfeasible configurations for higherorder singular values do exist even in the nonsingular case. In fact, these configurations are of positive volume within \({\mathfrak {S}}_{\ge }\).
Theorem 3.2
Proof
Assume to the contrary that for every n there exists a tensor \({\mathbf {X}}_n \in {\mathscr {S}}\) such that \({\varvec{\Sigma }}_{{\mathbf {X}}_n} \in \mathscr {O}^{(1)}_{1/n} \times \dots \times \mathscr {O}^{(d)}_{1/n}\). The sequence of \({\mathbf {X}}_n\) has a convergent subsequence with a limit \({\mathbf {X}}\in {\mathscr {S}}\). By Lemma 1.3, \({\mathbf {X}}\) has higherorder singular values \({\varvec{\Sigma }}_{{\mathbf {X}}} = ({\varvec{\sigma }}^{(1)},{\varvec{\sigma }}^{(2)},\mathbf {e}^{(3)}_1,\dots ,\mathbf {e}^{(d)}_1)\). Now Lemma 3.1 applies, but is in contradiction to (3.1). \(\square \)
Remark 3.3
The condition (3.1) can hold in two cases: (i) the number of nonzero singular values in direction one and two are the same (\(r_{1} = r_{2}\)), but the singular values themselves are not, or (ii) \(r_{1} \ne r_{2}\). The second case has some interesting implications for rectangular tensors. Assume for instance \(n_{1} \ne n_{2}\). Then by Theorem 3.2 there cannot exist normalized nonsingular tensors in \({\mathbb {R}}^{n_{1} \times \dots \times n_{d}}\) for which the singular value vectors \({\varvec{\sigma }}^{(j)}\) in directions \(j = 3,\dots ,d\) are arbitrarily close to the corresponding unit vector \(\mathbf {e}^{(j)}\). This surprising connection between mode sizes of the tensor and location of the singular value vectors is not obvious, especially given the fact that almost every tensor is nonsingular (assuming (1.2)).
3.2 A conjecture on interior points
For \(d = 2\) we have seen that \({\mathfrak {F}}(n_{1},n_{2})\) is a set of measure zero within \({\mathfrak {S}}_{\ge }(n_{1},n_{2})\), even when \(n_{1} = n_{2}\). One question is whether this is also true for higherorder tensors. Remarkably, the following experiment suggests that this does not need to be the case.
As the resulting point cloud appears threedimensional, we suppose that the set of feasible configurations is also threedimensional. But one can also verify in the plot that not all configurations are feasible. Above we made use of the fact that \({\varvec{\sigma }}^{(j)} = (1,0)\) (Tucker rank in the direction j equals one) implies \({\varvec{\sigma }}^{(i)} = {\varvec{\sigma }}^{(k)}\) for \(i,k \ne j\). This can be seen in the picture as the convex polytope intersects the hyperplanes \(x=1\), \(y=1\) and \(z=1\) in single onedimensional facets of 45 degree.
We are led to the following conjecture.
Conjecture 3.4
When \(d \ge 3\), and given the compatibility condition (1.2), the set \({\mathfrak {F}}(n_{1},\dots ,n_{d})\) has positive (relative) volume in \({\mathfrak {S}}_{\ge }(n_{1},\dots ,n_{d})\).
In fact, the following seems likely (under the same assumptions).
Conjecture 3.5
For generic \({\mathbf {X}} \in {{\mathscr {S}}}\), \({\varvec{\Sigma }}_{\mathbf {X}}\) is a (relative) interior point of \({\mathfrak {F}}(n_{1},\dots ,n_{d})\) within \({\mathfrak {S}}_{\ge }(n_{1},\dots ,n_{d})\).
Remark 3.6
During revision of the paper, a possible strategy to prove this conjecture has been revealed. It is based on the observation that \(\varvec{\Sigma }_{{\mathbf {X}}}\) is a relative interior point of \(\mathfrak {F}(n_{1},\dots ,n_{d})\) if and only if the map \(g({\mathbf {X}}) = ( G_{{\mathbf {X}}}^{(1)} ,\dots , G_{{\mathbf {X}}}^{(d)} )\) (that has already been considered in (2.4)) is locally surjective when regarded as a map from the unit sphere \({\mathscr {S}}\) to the Cartesian product of hyperplanes \(\{A^{(j)} \in {\mathbb {R}}^{n_j \times n_j}_{\text {sym}} :{{\mathrm{{tr}}}}(A^{(j)}) = 1\}\). In other words, one has to show that the rank of the derivative \(g'({\mathbf {X}})\), when restricted to the tangent space \(T_{{\mathbf {X}}} {\mathscr {S}}\), equals the maximum possible value \(\alpha = \left( \sum _{j=1}^d \frac{1}{2} n_j(n_j + 1) \right)  d\). A sufficient condition for this is that \(g'({\mathbf {X}})\) is of rank \(\alpha + 1\) on \({\mathbb {R}}^{n_1 \times \dots \times n_d}\). However, as \(g'({\mathbf {X}})\) depends polynomially on the entries of \({\mathbf {X}}\), the function \({\mathbf {X}}\mapsto {{\mathrm{{rank}}}}(g'({\mathbf {X}}))\) achieves its maximum value for almost all \({\mathbf {X}}\). Since it is bounded by \(\alpha + 1\), it is therefore enough to find a single tensor \({\mathbf {X}}\) for which rank \(\alpha + 1\) is achieved. In this way, one can validate Conjecture 3.5 for different configurations of \(n_1,\dots ,n_d\) by constructing random \({\mathbf {X}}\) and evaluating the rank of \(g'({\mathbf {X}})\) numerically. A rigorous proof would have to confirm this numerical rank for “simple” candidates \({\mathbf {X}}\), which we were able to do for \(2 \times 2 \times 2\) tensors so far. This approach shall be subject of a future work.
3.3 Alternating projection method
Even in the case that one would be given the information that a configuration \({\varvec{\Sigma }}=({\varvec{\sigma }}^{(1)},\dots ,{ \varvec{\sigma }}^{(d)})\in {\mathfrak {S}}_{\ge }\) is feasible, the question remains how to construct a corresponding tensor. Note that the suggested strategy to prove Conjecture 3.5 by showing full rank of (2.4) may not provide an explicit way for perturbing singular values in single directions.
Although the interpretation as an alternating projection method is nice, we remark that the multiplication by \(U_{\mathbf {Y}}^{(j)}\) in (3.4) could be omitted in practice. It is an easy induction to show that in this case an orthogonally equivalent sequence of tensors would be produced.
Even assuming that intersection points exist, we are currently not able to provide local or global convergence results for the alternating projection method (3.3). Instead, we confine ourselves with three numerical illustrations.
Recovering a feasible configuration
To obtain a feasible configuration \({\varvec{\Sigma }}\), we create a normone tensor \({\mathbf {X}}\) and take its higherorder singular values, \({ \varvec{\Sigma }}={\varvec{\Sigma }}_{\mathbf {X}}\). Then we run the iteration (3.3) starting from a random initialization, and measure the errors \(\Vert {\varvec{\sigma }}_{{ \mathbf {X}}_{k+1}}^{(j)}{\varvec{\sigma }}_{\mathbf {X}}^{(j)}\Vert _{2}\) (Euclidean norm) after every full cycle of projections. Since \(\Pi _{{ \varvec{\sigma }}^{(3)}}^{(3)}\) is applied last, the singular values in direction three are always correct at the time the error is measured. The question is whether also the singular values in the other directions converge to the desired target values. The left plot in Fig. 2 shows one typical example of error curves observed in this kind of experiment in \({ \mathbb {R}}^{30\times 30\times 30}\). We see that the sequence \({\varvec{ \Sigma }}_{{\mathbf {X}}_{k}}\) converges to \({\varvec{\Sigma }}\), hence every cluster point of the sequence \({\mathbf {X}}_{k}\) will have the desired higherorder singular values. So far, we have no theoretical explanation for the shifted peaks occurring in the curves.
Since our initial guess is random, we do not expect that the generating \({ \mathbf {X}}\) or an orthogonally equivalent tensor will be recovered. To verify this, we make use of Proposition 2.6 and measure the error \(\max _{i_{1},\dots ,i_{d}} \vert \hat{X} _{i_{1},\dots ,i_{d}} \vert  \vert (\hat{X}_{k+1})_{i_{1},\dots ,i_{d}} \vert \) after every loop, where \(\hat{{\mathbf {X}}} _{k+1}\) and \(\hat{{\mathbf {X}}}\) are HOSVD representatives in the corresponding orbits of orthogonal equivalence. The right plot in Fig. shows this error curve, and we can see it does not tend to zero. By Proposition 2.6, the limiting tensor is hence not orthogonally equivalent to \(\mathbf {X }\). Since this behaviour was observed being typical, the alternating projection method can be suggested as a practical procedure to construct tensors having the same higherorder singular values without being orthogonally equivalent.
Perturbation of a feasible configuration
Infeasible configuration
When conducting our experiments with the alternating projection method, we made the experience that with high probability even a randomly generated configuration will be feasible. Indeed, Fig. 1 supports this in the \(2 \times 2 \times 2\) case, as the feasible configurations seem to make up a rather large fraction in \({\mathfrak {S}}_{\ge }(2,2,2)\).
To construct an infeasible configuration we therefore mimic the proof of Theorem 3.2: we generate \({\varvec{ \sigma }}^{(j)}\) as \((1,0,\dots ,0)+\mathscr {O}(\epsilon _{j})\) (as described in Footnote 3), where we use very small \(\epsilon _{j}\) for \(j\ge 3\), e.g., \(\epsilon _j = 10^{6}\). By the arguments presented above this should also enforce \({\varvec{\sigma }}^{(1)}\) to be close to \({\varvec{\sigma }}^{(2)}\) to ensure feasibility. To impede this, we use larger \(\epsilon _1\) and \(\epsilon _2\) instead, e.g., \(\epsilon _{1}=\epsilon _{2}=10^{3}\) (an alternative would be to generate \({ \varvec{\sigma }}^{(1)}\) and \({\varvec{\sigma }}^{(2)}\) completely random). Our results suggest that this indeed results in an infeasible configuration. Accordingly, the alternating projection method fails. The right plot in Fig. 3 shows the outcome of one experiment, again in \(\mathbb {R}^{10 \times 10 \times 10 \times 10}\).
Footnotes
 1.
For \(d=2\), i.e., matrices, it is the case: as (1.2) is assumed, we have \(n_{1}=n_{2}=n\), and two square matrices have the same singular values if and only if they are orthogonally equivalent. The formula gives \(n^{2}2n+1\) which, however, only equals \( n^{2}n(n1)=\dim ({\mathrm {O}}(n\times n))(n1)\). The reason is that in the matrix case we know that the singular values of \({\mathbf {X}}\) and \({ \mathbf {X}}^{\mathsf {T}}\) are the same. Hence the feasible set is only of dimension \(n1\), and not of dimension \(2(n1)\) (the argument will be repeated in Sect. 3.1). For tensors, however, we conjecture that the dimension of \({\mathfrak {F}}\) is indeed \( (n_{1}+\dots +n_{d})d\), see Sect. 3.2.
 2.
For our experiments we made use of the Tensor Toolbox [2] in Matlab.
 3.
Practically, \({\varvec{\Sigma }}_\epsilon \) was generated by normalizing and sorting the perturbed \(\left\sigma ^{(j)} + \mathscr {O}(\epsilon )\right\).
Notes
Acknowledgments
Open access funding provided by Max Planck Society.
References
 1.Bachmayr, M., Dahmen, W.: Adaptive nearoptimal rank tensor approximation for highdimensional operator equations. Found. Comput. Math. 15(4), 839–898 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
 2.Bader, B.W., Kolda, T.G., et al.: Matlab tensor toolbox version 2.6 (2015). http://www.sandia.gov/~tgkolda/TensorToolbox/
 3.De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl. 21(4), 1253–1278 (2000, electronic)Google Scholar
 4.de Sá, E.M.: Exposed faces and duality for symmetric and unitarily invariant norms. Linear Algebra Appl. 197(198), 429–450 (1994)MathSciNetzbMATHGoogle Scholar
 5.de Silva, V., Lim, L.H.: Tensor rank and the illposedness of the best lowrank approximation problem. SIAM J. Matrix Anal. Appl. 30(3), 1084–1127 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
 6.Dieudonné, J.: Treatise on Analysis, vol. III. Academic Press, New York (1972)zbMATHGoogle Scholar
 7.Grasedyck, L.: Hierarchical singular value decomposition of tensors. SIAM J. Matrix Anal. Appl. 31(4), 2029–2054 (2009/2010)Google Scholar
 8.Hackbusch, W.: Tensor Spaces and Numerical Tensor Calculus. Springer, Heidelberg (2012)CrossRefzbMATHGoogle Scholar
 9.Hackbusch, W., Kühn, S.: A new scheme for the tensor representation. J. Fourier Anal. Appl. 15(5), 706–722 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
 10.Hitchcock, F.L.: The expression of a tensor or a polyadic as a sum of products. J. Math. Phys. 6, 164–189 (1927)CrossRefzbMATHGoogle Scholar
 11.Hitchcock, F.L.: Multiple invariants and generalized rank of a \(p\)way matrix or tensor. J. Math. Phys. 7, 39–79 (1927)CrossRefzbMATHGoogle Scholar
 12.Koch, O., Lubich, C.: Dynamical tensor approximation. SIAM J. Matrix Anal. Appl. 31(5), 2360–2375 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
 13.Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455–500 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
 14.Lubich, C., Rohwedder, T., Schneider, R., Vandereycken, B.: Dynamical approximation by hierarchical Tucker and tensortrain tensors. SIAM J. Matrix Anal. Appl. 34(2), 470–494 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
 15.Mirsky, L.: A trace inequality of John von Neumann. Monatsh. Math. 79(4), 303–306 (1975)MathSciNetCrossRefzbMATHGoogle Scholar
 16.Oseledets, I., Tyrtyshnikov, E.: TTcross approximation for multidimensional arrays. Linear Algebra Appl. 432(1), 70–88 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
 17.Oseledets, I.V.: Tensortrain decomposition. SIAM J. Sci. Comput. 33(5), 2295–2317 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
 18.Oseledets, I.V., Tyrtyshnikov, E.E.: Breaking the curse of dimensionality, or how to use SVD in many dimensions. SIAM J. Sci. Comput. 31(5), 3744–3759 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
 19.Oseledets, I.V., Tyrtyshnikov, E.E.: Recursive decomposition of multidimensional tensors. Dokl. Akad. Nauk 427(1), 14–16 (2009, in Russian) [English translation in: Dokl. Math. 80(1), 460–462 (2009)]Google Scholar
 20.Rohwedder, T., Uschmajew, A.: On local convergence of alternating schemes for optimization of convex problems in the tensor train format. SIAM J. Numer. Anal. 51(2), 1134–1162 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
 21.Schneider, R., Uschmajew, A.: Approximation rates for the hierarchical tensor format in periodic Sobolev spaces. J. Complex. 30(2), 56–71 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
 22.Tucker, L.R.: Some mathematical notes on threemode factor analysis. Psychometrika 31(3), 279–311 (1966)MathSciNetCrossRefGoogle Scholar
 23.von Neumann, J.: Some matrixinequalities and metrization of matrixspace. Tomsk. Univ. Rev. 1, 286–300 (1937)Google Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.