Abstract
We provide moment bounds for expressions of the type \((X^{(1)} \otimes \cdots \otimes X^{(d)})^T A (X^{(1)} \otimes \cdots \otimes X^{(d)})\) where \(\otimes \) denotes the Kronecker product and \(X^{(1)}, \ldots , X^{(d)}\) are random vectors with independent, mean 0, variance 1, subgaussian entries. The bounds are tight up to constants depending on d for the case of Gaussian random vectors. Our proof also provides a decoupling inequality for expressions of this type. Using these bounds, we obtain new, improved concentration inequalities for expressions of the form \(\Vert B (X^{(1)} \otimes \cdots \otimes X^{(d)})\Vert _2\).
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
1.1 Background and studied objects
Given a matrix \(A \in {\mathbb {R}}^{n \times n}\) and a random vector \(X \in {\mathbb {R}}^n\), the Hanson–Wright inequality provides a tail bound for the chaos \(X^T A X - {\mathbb {E}} X^T A X\). In the original work [1], X was assumed to have independent subgaussian entries whose distributions are symmetric about 0.
This result has been improved and adapted to various settings in a number of works, for example [2] gives a version which holds for vectors with general subgaussian entries without the symmetry assumption of the distribution:
Theorem 1
(Theorem 1.1 from [2]) Let \(A \in {\mathbb {R}}^{n \times n}\). Let \(X \in {\mathbb {R}}^n\) be a random vector with independent entries such that \({\mathbb {E}} X = 0\) and such that X has a subgaussian norm of at most K. Then for every \(t \ge 0\),
where \(\Vert A\Vert _F\) is the Frobenius and \(\Vert A\Vert _{2 \rightarrow 2}\) the spectral norm of A.
Today, the Hanson–Wright inequality is an important probabilistic tool and can be found in various textbooks covering the basics of signal processing and probability theory, such as [3, 4]. It has found numerous applications, in particular it has been a key ingredient for the construction of fast Johnson–Lindenstrauss embeddings [5].
For subgaussian \(X \in {\mathbb {R}}^n\), linear expressions \(\sum _{k = 1}^n a_k X_k\) can be controlled by Hoeffding’s inequality, while quadratic (order 2) expressions \(X^T A X = \sum _{j, k = 1}^n A_{j, k} X_j X_k\) can be controlled by the Hanson–Wright inequality. Thus, it is natural to wonder to what extent such control extends to a higher-order subgaussian chaos of the form
Expressions of this type for subgaussian vectors have been considered in [6] where they are controlled using specific tensor norms of the arrays of all expected partial derivatives of certain degree with respect to the entries in X.
In contrast, for independent random vectors \(X^{(1)}, \ldots , X^{(d)}\), the decoupled chaos
can be controlled with simpler bounds and has been considered in multiple previous works for numerous different distributions of the random vectors [7,8,9].
In the course of adapting fast Johnson–Lindenstrauss embeddings to data with Kronecker structure as introduced in [10] (see also [11, 12]), one encounters expressions of the form \((X^{(1)} \otimes \cdots \otimes X^{(d)})^T A (X^{(1)} \otimes \cdots \otimes X^{(d)})\) which are somewhat intermediate between (1) and (2), as they can be expanded as
In the case that A is of the form \(B^T B\) for a matrix B (suitably reindexed), this expression can be rewritten as the square of
a quantity of interest studied by Roman Vershynin in the context of random tensors [13]. Hence, understanding (3) can also help understanding (4), see Sect. 3.4 below.
Even though (3) can be cast as a specific case of (1) for which [6] provides optimal bounds, these bounds are not straightforward to use in this specific situation since they are given in terms of partial derivatives and not in terms of the coefficients \(A_{i_1, \ldots , i_{2 d}}\).
The main results of this paper provide moment estimates for the semi-decoupled chaos process (3) that are easier to use as they are explicitly given in terms of the coefficients \(A_{i_1, \ldots , i_{2 d}}\). Our bounds imply improved estimates for (4) and lay the foundations for an order-optimal analysis of fast Kronecker-structured Johnson–Lindenstrauss embeddings. We refer the reader to our companion paper [14] for a discussion of the implications in this regard. We nevertheless expect that our results should find broader use beyond these specific applications.
1.2 Previous work
For the case where \(X^{(1)}, \ldots , X^{(d)}\) are independent Gaussian vectors, the concentration of (2) has been studied in [7] which provides upper and lower moment bounds which match up to a constant factor depending only on the order d. We will obtain our main results for subgaussian vectors by careful reduction to the Gaussian bounds.
Higher order chaos expressions have also been studied for distributions beyond Gaussian. Specifically, [15, Section 9], considers (1) for the case of Rademacher vectors. However, the bounds are more intricate than in [7] and the coefficient array \({\varvec{A}} = (A_{i_1, \ldots , i_{d}})_{i_1, \ldots , i_{d}=1}^n\) must satisfy a symmetry condition and be diagonal-free, i.e., \(A_{i_1, \ldots , i_d} = 0\) if any two of the indices \(i_1, \ldots , i_d\) coincide.
Upper and lower bounds on the moments of (2) are shown in [8, 9] for the case of symmetric random variables with logarithmically concave and convex tails, meaning that for a random variable \(X \in {\mathbb {R}}\), the function \(t \mapsto - \log {\mathbb {P}}(|X |\ge t)\) is convex or concave, respectively. However, for general subgaussian random variables, neither of these has to be the case. In addition, these works only consider the decoupled chaos (2) and provide a decoupling inequality to control (1) for diagonal-free \({\varvec{A}}\).
Upper moment bounds for general polynomials of independent subgaussian random variables are provided in [6]. Similar to our work, the authors utilize the decoupling techniques of [16]. Since (3) is a polynomial in the entries of \(X^{(1)}, \ldots , X^{(d)}\), it can also be controlled using the results from [6]. Because the aforementioned work also shows that these moment bounds are tight for the case of Gaussian vectors, one of the main results (Theorem 3) of our work can also be shown using their results. However, their result bounds the corresponding \(L_p\) norms in terms of norms of the array of all \(d' \le 2 d\) expected partial derivatives, meaning that significant additional work would be required to relate these derivatives to the expressions in Theorem 3. We believe, that our approach is not much longer but more insightful. In addition, it provides the decoupling result Theorem 4 which will be of independent interest.
More work on related topics include [17, 18] where upper and lower bounds for the case of random variables satisfying the moment condition \(\Vert X\Vert _{2 p} \le \alpha \Vert X\Vert _p\) are considered for the case of positive variables of order 2. The recent work [19] provides similar bounds to [6] for distributions of bounded \(\psi _\alpha \) norm for \(\alpha \in (0, 1]\) (or \(\alpha \in (0, 2]\) for some fo their results), such as subexponential distributions. Like in [6], their bounds are given in terms of partial derivatives, not directly in terms of the coefficients.
The decoupling technique used in many proofs of the standard Hanson–Wright inequality relates \(X^T A X\) to \(X^T A {\bar{X}}\) where \({\bar{X}}\) is an independent copy of X. This approach was first introduced in [20], already in a general higher-dimensional form. The general idea is to upper bound convex functions (e.g. moments) of (1) by the corresponding expressions of (2), up to a constant. Beside independent, symmetrically distributed entries of the random vectors, the result also requires the coefficient array to be symmetric and diagonal free.
The subsequent work [21] has also shown the reverse decoupling bound, up to constant factors, proving that through (2), one can also provide lower bounds on the moments of (1) with the same assumptions on the coefficient array. However, in some applications it can be interesting to consider non-diagonal-free coefficient arrays. For example, in the scenario of \(\Vert B (X^{(1)} \otimes \dots \otimes X^{(d)})\Vert _2^2\), the coefficient array \(B^T B\) cannot be expected to fulfill the diagonal-free condition in general. The work in [16] lifts the restriction of a diagonal-free coefficient array and bounds the tails of slight modifications of (2) and (1) by each other up to certain constants in the case of Gaussian random variables.
The concentration of the norm (4) has recently been studied for the subgaussian case in [13]. It is shown that
for an absolute constant c and for \(0 \le t \le 2 n^\frac{d}{2} \Vert B\Vert _{2 \rightarrow 2}\). This bound suggests that techniques like the chaos moment bounds in [7] could be applied to this problem, which is what we do in this work and leads to Theorem 6 below.
1.3 Overview of our contribution
The goal of this work is to provide upper and lower bounds for the moments of the deviation of (3) from its expectation for vectors with independent subgaussian entries (Theorem 3 below). Key steps of the proof include a decoupling inequality for expressions of the form (3), Theorem 4, and a comparison to Gaussian random vectors. Finally, based on our results for (3), we provide a concentration inequality for (4) as stated in Theorem 6 which extends previous results of [13].
Possible applications of such results include recent developments in norm-preserving maps for vectors with tensor structure in the context of machine learning methods using the kernel trick [10,11,12].
1.4 Notation
Our results on \(X^T A X\) where X is a Kronecker product of d random vectors will depend crucially on the structure of the coefficient matrix A rearranged as a higher-order (specifically order 2d) array. As such, we must establish sophisticated notation for such arrays and their indices.
Consider a vector of dimensions \({\varvec{n}} = (n_1, n_2, \ldots , n_d)\) and a subset \(I \subset [d]\). We call a function \({\varvec{i}}: I \rightarrow {\mathbb {N}}\) a partial index of order d on I if for all \(l \in I\), \({\varvec{i}}_l := {\varvec{i}}(l) \in [n_l]\). Assume there is exactly one such function if \(I = \emptyset \). If \(I = [d]\), then \({\varvec{i}}\) is called an index of order d. We denote the set of all partial indices of order d on I as \({\varvec{J}}^{\varvec{n}}(I)\); the set of all indices of order d is denoted by \({\varvec{J}}^{\varvec{n}}:= {\varvec{J}}^{\varvec{n}}([d])\). \({\varvec{J}}^{\varvec{n}}\) can be identified with \([n_1] \times \cdots \times [n_d]\).
A function \({\varvec{B}}: {\varvec{J}}^{\varvec{n}}\rightarrow {\mathbb {R}}\) is called an array of order d. Because of the aforementioned identification, we also write \({\varvec{B}} \in {\mathbb {R}}^{n_1 \times \cdots \times n_d} =: {\mathbb {R}}^{{\varvec{n}}}\). For \(I \subset [d]\), we define \({\mathbb {R}}^{{\varvec{n}}}(I)\) to be the set of partial arrays \({\varvec{B}}: {\varvec{J}}^{\varvec{n}}(I) \rightarrow {\mathbb {R}}\). For \(I = [d]\), this is just the aforementioned array definition.
We denote
for the Frobenius norm of the (partial) array where \(B_{{\varvec{i}}} := {\varvec{B}}({\varvec{i}})\) are its entries.
For disjoint sets \(I,\,J \subset [d]\) and corresponding partial indices \({\varvec{i}} \in {\varvec{J}}^{\varvec{n}}(I)\), \({\varvec{j}} \in {\varvec{J}}^{\varvec{n}}(J)\), define the partial index \({\varvec{i}} {\dot{\times }} {\varvec{j}} \in {\varvec{J}}^{\varvec{n}}(I \cup J)\) by
We will often work with arrays of order 2d whose dimensions along the first d axes are the same as the dimensions along the remaining d ones. We use the notation \({\varvec{n}}^{\times 2} = (n_1, \ldots , n_d, n_1, \ldots , n_d)\) to denote such arrays.
For sets \(I \subset [2 d]\), \(J \subset [d]\) such that \(I \cap (J + d) = \emptyset \) and for corresponding partial indices \({\varvec{i}} \in {\varvec{J}}^{\varvec{n}}(I)\), \({\varvec{j}} \in {\varvec{J}}^{\varvec{n}}(J)\), define the partial index \({\varvec{i}} \dot{+} {\varvec{j}} \in {\varvec{J}}^{{\varvec{n}}^{\times 2}}(I \cup (J + d))\) by
For \({\varvec{i}} \in {\varvec{J}}^{\varvec{n}}(I)\) and \(J \subset I\), define \({\varvec{i}}_J \in {\varvec{J}}^{\varvec{n}}(J)\) to be the restriction of \({\varvec{i}}\) to J, i.e., \(({\varvec{i}}_J)_l = {\varvec{i}}_l\) for all \(l \in J\).
As suggested by the explanations above, our convention is to use bold letters for higher order arrays (e.g., \({\varvec{A}}\)) while their entries are denoted in non-bold letters (e.g., \(A_{{\varvec{i}}}\)). For some of our results, we will convert matrices into higher-order arrays by rearranging their entries. In these cases, we will denote the matrices in non-bold letters and use the same letter in bold for the array, e.g., A and \({\varvec{A}}\). For the entries, it will be clear from the indices which object is being referred to. Besides that, we will also always use bold letters for array indices (e.g., \({\varvec{i}}\)), for vectors of array dimensions (e.g. \({\varvec{n}}\)), and for the set \({\varvec{J}}^{\varvec{n}}\).
We denote \(Id_n \in {\mathbb {R}}^{n \times n}\) for the identity matrix, \(\Vert A\Vert _F\) for the Frobenius norm of a matrix, and \(\Vert A\Vert _{2 \rightarrow 2}\) for the spectral norm of a matrix.
For a random variable \(Y \in {\mathbb {R}}\), we define \(\Vert Y\Vert _{L_p} := ({\mathbb {E}}|Y |^p)^{1 / p}\) and we define the subgaussian norm \(\Vert Y\Vert _{\psi _2} := \sup _{p \ge 1} \Vert Y\Vert _{L_p} / \sqrt{p}\). For a random vector \(X \in {\mathbb {R}}^n\), we define the subgaussian norm \(\Vert X\Vert _{\psi _2} := \sup _{v \in {\mathbb {R}}^n, \Vert v\Vert _2 = 1} \Vert \langle X, v \rangle \Vert _{\psi _2}\), and we call X isotropic if \({\mathbb {E}} X X^T = Id_n\).
1.5 Previous relevant results
Since our result is based on the bounds given by Latala in [7], we also consider the following norms which are also used in that result. In our notation, the norms of interest are stated as follows.
Definition 1
For \({\varvec{n}} \in {\mathbb {N}}^d\) and an array \({\varvec{A}} \in {\mathbb {R}}^{{\varvec{n}}}\), we define the following norms for any partition \(I_1, \dots , I_\kappa \) of [d].
For example, when \(d=2\), the array \({\varvec{A}}\) is a matrix and \(\Vert \cdot \Vert _{\{1,2\}}\) coincides with the Frobenius and \(\Vert \cdot \Vert _{\{1\}, \{2\}}\) with the spectral norm. Latala [7] proved the following upper and lower moment bounds for a decoupled Gaussian chaos of arbitrary order. Even though it is only shown for \(p \ge 2\) in [7], it holds for all \(p \ge 1\) as explained in Remark 1 below.
Theorem 2
(Theorem 1 in [7]) Let \({\varvec{n}} \in {\mathbb {N}}^d\), \({\varvec{A}} \in {\mathbb {R}}^{{\varvec{n}}}\), \(p \ge 1\).
Let \(S(d, \kappa )\) denote the set of partitions of [d] into \(\kappa \) nonempty disjoint subsets. Define
Consider independent Gaussian random vectors \(g^{(1)} \sim N(0, Id_{{\varvec{n}}_1}), \ldots , g^{(d)} \sim N(0, Id_{{\varvec{n}}_d})\). Then
where \(C(d) > 0\) is a constant that only depends on d.
Remark 1
Theorem 1 in [7] only shows this statement for \(p \ge 2\). However, by a small adjustment, we can see that it also holds for \(1 \le p \le 2\) with a possibly different C(d). Let \(X := \sum _{{\varvec{i}} \in {\varvec{J}}^{\varvec{n}}} A_{{\varvec{i}}} \prod _{l \in [d]} g^{(l)}_{{\varvec{i}}_l}\). For the upper bound we have for \(1 \le p \le 2\),
For the lower bound, we consider the recent work [22] about a generalized Gaussian chaos with values in an arbitrary Banach space. Theorem 2.1 in their work states the lower bound
for all \(p \ge 1\), where \({\mathcal {P}}(J)\) is defined as the set of all partitions of J (into non-empty, pairwise disjoint sets) and \({\left| \left| \left| {\varvec{A}} \right| \right| \right| }_{{\mathcal {P}}}\), defined in (2.2) of [22], is a non-negative expression that coincides with our definition of \(\Vert {\varvec{A}}\Vert _{I_1, \ldots , I_\kappa }\) if \({\mathcal {P}} = (I_1, \ldots , I_\kappa )\) is a partition of the entire set [d]. Therefore we can restrict the sum over J in (9) to the term \(J = [d]\) and obtain
2 Main results
The main contribution of our work is the following result which gives a generalization of the Hanson–Wright inequality (Theorem 1) in terms of upper and lower moment bounds. Note that the operators \({\dot{\times }}\) and \(\dot{+}\) are defined in (6) and (7).
Theorem 3
For \(d \ge 1\), let \({\varvec{n}} = (n_1, \ldots , n_d)\) be a vector of dimensions, and let \(N = n_1 \ldots n_d\).
Let \(A \in {\mathbb {R}}^{N \times N}\) and \(X^{(1)} \in {\mathbb {R}}^{n_1}, \ldots , X^{(d)} \in {\mathbb {R}}^{n_d}\) be random vectors with independent, mean 0, variance 1 entries with subgaussian norms bounded by \(L \ge 1\). Define \(X := X^{(1)} \otimes \cdots \otimes X^{(d)}\). There exists a constant C(d), depending only on d, such that for all \(p \ge 1\),
The numbers \(m_p\) are defined as follows. By rearranging its entries, regard A as an array \({\varvec{A}} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}\) of order 2d such that
For any \(I \subset [d]\) and for \(I^c = [d] \backslash I\), define \({\varvec{A}}^{(I)} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}(I^c \cup (I^c + d))\) by
for all \({\varvec{i}}, {\varvec{i}}' \in {\varvec{J}}^{\varvec{n}}(I^c)\).
For \(T \subset [2 d]\) and \(1 \le \kappa \le 2d\), denote by \(S(T, \kappa )\) the set of partitions of T into \(\kappa \) sets. Then for any \(p \ge 1\), define
If in addition, \(X^{(1)} \sim N(0, Id_{n_1}), \dots , X^{(d)} \sim N(0, Id_{n_d})\) are normally distributed (i.e. L is constant), and \({\varvec{A}}\) satisfies the symmetry condition that for all \(l \in [d]\) and any \({\varvec{i}}, {\varvec{i}}' \in {\varvec{J}}^{\varvec{n}}([d] \backslash \{l\})\), \({\varvec{j}}, {\varvec{j}}' \in {\varvec{J}}^{\varvec{n}}(\{l\})\),
then also the lower bound
holds for all \(p \ge 1\). Here, \({\tilde{C}}(d) > 0\) only depends on d.
Note that these upper bounds can directly be converted to tail bounds in the style of Theorems 1 or 6 using Lemma 13. After introducing the required tools, the proof of Theorem 3 will be split up into two parts. We will prove the upper bound in Sect. 3.2.2 and then the lower bound in Sect. 3.3.2.
Remark 2
The symmetry condition required for the lower bound is not satisfied for all matrices. However, for any matrix A, we can find a matrix \({\tilde{A}}\) satisfying the symmetry condition and such that \(X^T A X = X^T {\tilde{A}} X\) always holds. To do this, in the array notation we can define \(\tilde{{\varvec{A}}}\) by transposing \({\varvec{A}}\) along all possible sets of axes and then taking the mean \({\tilde{A}}_{{\varvec{i}} \dot{+} {\varvec{i}}'} = \frac{1}{2^d} \sum _{I \subset [d]} A_{({\varvec{i}}_{I^c} {\dot{\times }} {\varvec{i}}'_{I}) \dot{+} ({\varvec{i}}_{I} {\dot{\times }} {\varvec{i}}'_{I^c})}\) for any \({\varvec{i}}, {\varvec{i}}' \in {\varvec{J}}^{\varvec{n}}\). This is a generalization of taking \({\tilde{A}} = \frac{1}{2} (A + A^T)\) for \(d = 1\). Note however, that \({\tilde{A}}\) might have significantly smaller norms than A which is why the lower moment bounds in Theorem 3 might not hold for A directly.
A central part of our argument is the following specialized decoupling result for expressions as in (3) which might be of independent interest.
Theorem 4
Let \({\varvec{n}} = (n_1, \dots , n_d) \in {\mathbb {N}}^d\), \({\varvec{A}} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}\), \(X^{(1)} \in {\mathbb {R}}^{n_1}, \dots , X^{(d)} \in {\mathbb {R}}^{n_d}\) random vectors with independent mean 0, variance 1 entries and \({\bar{X}}^{(1)}, \dots , {\bar{X}}^{(d)}\) corresponding independent copies. Then for all \(p \ge 1\),
Remark 3
Consider the special case in Theorem 4 of \(X^{(1)}, \dots , X^{(d)}\) being Rademacher vectors, i.e., having independent entries that are \(\pm 1\) with a probability of \(\frac{1}{2}\) each. Then any squared entry is 1 almost surely. This implies that the factor \(\prod _{l \in J} \left[ (X^{(l)}_{{\varvec{i}}_l})^2 - 1 \right] \) is 0 unless \(J = \emptyset \). So on the right hand side of the inequality in Theorem 4, only the terms with \(J = \emptyset \) need to be considered.
Theorem 4 combines two aspects. On the one hand, there is the probabilistic aspect that the factors \(X^{(l)}_{{\varvec{i}}_l} X^{(l)}_{{\varvec{i}}'_l}\) are replaced by \(X^{(l)}_{{\varvec{k}}_l} {\bar{X}}^{(l)}_{{\varvec{k}}'_l}\), using the independent copy \({\bar{X}}^{(l)}_{{\varvec{k}}'_l}\). On the other hand, there is the arithmetic one that the quadratic factors \((X^{(l)}_{{\varvec{i}}_l})^2\) arising on the left hand side for \({\varvec{i}}_l = {\varvec{i}}'_l\) are expressed by factors \(\left[ (X^{(l)}_{{\varvec{i}}_l})^2 - 1 \right] \), which have mean 0. A crucial ingredient for the proof of Theorem 4 is the following theorem, which summarizes the aforementioned arithmetic aspect. Note that it does not take any randomness in the vectors into account.
Theorem 5
Let \({\varvec{n}} \in {\mathbb {N}}^d\), \({\varvec{A}} \in {\mathbb {R}}^{{\varvec{n}}}\), \(X^{(1)} \in {\mathbb {R}}^{n_1}, \ldots , X^{(d)} \in {\mathbb {R}}^{n_d}\). Then
where for any \(I \subset [d]\) and \({\varvec{i}} \in {\varvec{J}}^{\varvec{n}}([d] \backslash I)\),
Theorem 3 also leads to the following new tail bound for \(\Vert A (X^{(1)} \otimes \dots \otimes X^{(d)})\Vert _2\). Note that it contains the deviation of the non-squared norm. This improves upon the previous result by Vershynin [13] as described in (5), up to the constant C(d). By comparison, our result provides a strictly stronger bound for matrices with smaller Frobenius norm and holds for all \(t \ge 0\).
Theorem 6
Let \(B \in {\mathbb {R}}^{n_0 \times n^d}\) be a matrix, \(X^{(1)}, \dots , X^{(d)} \in {\mathbb {R}}^{n}\) independent random vectors with independent, mean 0, variance 1 entries with subgaussian norm bounded by \(L \ge 1\), and let \(X := X^{(1)} \otimes \dots \otimes X^{(d)} \in {\mathbb {R}}^{n^d}\). Then for a constant C(d) depending only on d and for any \(t > 0\),
Note that the third interval intersects the first two intervals. In any interval of intersection, both bounds hold. For slightly more complicated but provably optimal moment bounds, we refer the reader to Corollary 22.
Remark 4
In addition to extending the previous result in (5) from [13] to all \(t \ge 0\), our result provides a strict improvement of that result for matrices with stable rank \((\Vert B\Vert _F / \Vert B\Vert _{2 \rightarrow 2})^2 \in (1, n^{\frac{d - 1}{2}} )\).
As an example, consider a square matrix \(B \in {\mathbb {R}}^{n^d \times n^d}\) with mildly exponentially decreasing singular values \(\sigma _j = e^{-\frac{1}{2} n^{-\frac{d}{4}} (j - 1) }\) for \(1 \le j \le r\). Then by direct calculations, one can check that \(\Vert B\Vert _{2 \rightarrow 2} = \sigma _1 = 1\) and
So the stable rank is \(\in [\frac{1}{2} n^\frac{d}{4}, 2 n^\frac{d}{4}]\). Indeed, for at least the (for large enough n non-empty) interval \(n^{\frac{1}{4}d - \frac{1}{4}} \le t \le \frac{1}{2} n^{\frac{3}{8}d - \frac{1}{4}}\), the third line in Theorem 6 provides a probability bound \(\le e^2 \exp \left( - C(d) \frac{t^2}{2 n^{\frac{3}{4}d - \frac{1}{2}}} \right) \) while the first line only provides a bound of \(e^2 \exp \left( - C(d) \frac{t^2}{n^{d - 1}} \right) \), i.e., there is an improvement for \(d \ge 3\).
3 Main proofs
3.1 Preliminaries
The classical symmetrization theorem for normed spaces, such as Lemma 6.4.2 in [23], can be extended to increasing convex functions of norms as the following result from [24] shows.
Lemma 7
(Special case of Lemma A1 in [24]) Let \(X_1, \ldots , X_n\) be independent, mean 0 real-valued random variables and \(p \ge 1\). Let \(\xi _1, \ldots , \xi _n\) be independent Rademacher variables that are independent of \(X_1, \ldots , X_n\). Then
Decoupling theorems for quadratic forms relate double sums \(\sum _{j, k = 1}^n A_{j, k} X_j X_k\) over random variables \((X_j)_{j \in [n]}\) to a “decoupled” expression \(\sum _{j, k = 1}^n A_{j, k} X_j {\bar{X}}_k\) where the \({\bar{X}}_k\) are independent copies of the \(X_k\). Different versions have been used in probability theory for a long time and we refer to Section 3.6 in [25] for an overview of their history. The following version for convex functions, Theorem 8.11 in the textbook [3], is an adaptation of Proposition 1.9 in [26].
Theorem 8
Let \(A \in {\mathbb {R}}^{n \times n}\) be a matrix, \(X \in {\mathbb {R}}^n\) a vector with independent mean 0 entries, and \({\bar{X}}\) and independent copy of X. Let \(F: {\mathbb {R}} \rightarrow {\mathbb {R}}\) be a convex function. Then
Also the following elementary result will be used.
Lemma 9
Let T be a finite set. Then
Proof
By grouping all \(S \subset T\) of the same size and applying the binomial theorem,
\(\square \)
Although this is a very elementary statement and consequence of the binomial theorem, we are not aware of any previous usages of precisely this identity. One somewhat similar tool is given by Mazur-Orlicz formula ((11) in [27]), which has also been used in a problem related to decoupling inequalities in [28]. It is stated as
With \(v_1 = \cdots = v_k = 0\), this becomes
For \(k = |T |> 0\), the \(\{0, 1\}\)-tuples \((\epsilon _1, \ldots , \epsilon _k)\) can be identified with the subsets \(S \subset T\) such that \(|S |= \epsilon _1 + \cdots + \epsilon _k\) and then this identity implies Lemma 9 for \(T \ne \emptyset \).
For the norms in Definition 1, we need the following property about restricting arrays to some diagonal entries. This can be obtained directly from a repeated application of Lemma 5.2 in [6] (where \(K = \{l, l + d\}\) for each \(l \in I\)). Here again, we use the notation of \({\dot{\times }}\) and \(\dot{+}\) from (6) and (7).
Lemma 10
Let \({\varvec{A}} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}\), \(I \subset [d]\) and define \({\varvec{A}}^{[I]} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}\) by
for all \({\varvec{i}}, {\varvec{i}}' \in {\varvec{J}}^{\varvec{n}}\). Then for any partition \(I_1, \dots , I_\kappa \) of [2d], we have
For comparisons between functions of subgaussian and of Gaussian variables, we will use the concept of strong domination of random variables. See, e.g., [29] for the following definition and further explanations.
Definition 2
(Definition 3.2.1 in[29]) Let \(X, Y \in {\mathbb {R}}\) be random variables and \(\kappa , \lambda > 0\). We say that X is \((\kappa , \lambda )\)-strongly dominated by Y (\(X \prec _{(\kappa , \lambda )} Y\)) if for every \(t > 0\),
It can be shown that linear combinations of independent, strongly dominated random variables are again strongly dominated which in turn implies the following statement about expectations of convex functions of these linear combinations.
Theorem 11
(Corollary 3.2.1 in [29]) Let \(X_1, \ldots , X_n, Y_1, \ldots , Y_n \in {\mathbb {R}}\) be independent symmetric random variables and \(a_1, \ldots , a_n \in {\mathbb {R}}\) fixed coefficients such that \(X_i \prec _{(\kappa , \lambda )} Y_i\). Then for any nondecreasing \(\varphi : {\mathbb {R}}^{+} \rightarrow {\mathbb {R}}^{+}\),
Statements similar to the following lemma have been used in multiple works to establish a relation between \(\left|\Vert A x\Vert _2 - a \right|\) and \(\left|\Vert A x\Vert _2^2 - a^2 \right|\), for example in the proof of Lemma 5.36 in [30]. For completeness, we state it as a separate result with its proof here.
Lemma 12
For real numbers \(a, b \ge 0\), \(b \ne 0\), it holds that
Proof
We obtain
and since \(a, b \ge 0\), i.e., \(|a - b |\le |a |+ |b |= |a + b |\), it follows that \(|a - b |^2 \le |a - b ||a + b |= |a^2 - b^2 |\), proving the second inequality.
For the first inequality, first assume the case \(a \le 2 b\). Then \(a + b \le 3 b\) such that
In the case that \(a \ge 2 b\), i.e., \(a - b \ge b \ge 0\), we obtain
\(\square \)
Relations between moments and tail bounds have also been well-known in the field. For an overview see, e.g., Chapter 7.3 in [3]. In this spirit, we state and prove the following small tool for the case of mixed tails which we encounter in this work.
Lemma 13
(Moments and tail bounds) Let T be a finite set and X an \({\mathbb {R}}\) valued random variable such that for all \(p \ge p_0 \ge 0\),
for values \(\gamma _{k, l} > 0\), \(e_{k, l} > 0\).
Then for all \(t > 0\),
Proof
Fix any \(u > 0\). For any \(k \in [d]\), define \(l'(k) := {{\,\mathrm{argmax}\,}}_{l \in T} \left( \frac{u}{\gamma _{k, l}} \right) ^{\frac{1}{e_{k, l}}}\), then choose \(k' := {{\,\mathrm{argmin}\,}}_{k \in [d]} \left( \frac{u}{\gamma _{k, l'(k)}} \right) ^{ \frac{1}{e_{k, l'(k)}} }\), and \(p := \left( \frac{u}{\gamma _{k', l'(k')}} \right) ^{\frac{1}{e_{k', l'(k')}}}\), such that \(p = \min _{k \in [d]} \max _{l \in T} \left( \frac{u}{\gamma _{k, l}} \right) ^{\frac{1}{e_{k, l}}}\).
If \(p < p_0\), then \({\mathbb {P}}(|X |> e d u) \le 1 = e^{p_0} \exp (-p_0) \le e^{p_0} \exp (-p)\).
If \(p \ge p_0\), then by the choice of p,
So by Markov’s inequality,
In all cases, we obtain
The result follows by taking \(u := \frac{t}{e d}\). \(\square \)
3.2 Proof of the upper bound
3.2.1 Required tools
Lemma 14
There is an absolute constant C such that the following holds. Let \(X \in {\mathbb {R}}^n\) be random with mean 0 and \(\Vert X\Vert _{\psi _2} \le L\). Take a Gaussian vector \(g \sim N(0, Id_n)\) and \(a \in {\mathbb {R}}^n\). Then for all \(p \ge 1\),
Proof
By the assumption on X, \(\sum _{k = 1}^n a_k X_k = \langle a, X \rangle \) is mean 0 with \(\Vert \langle a, X \rangle \Vert _{\psi _2} \le L \Vert a\Vert _2\), implying that for any \(p \ge 1\),
On the other hand, \(\langle a, g \rangle \sim N(0, \Vert a\Vert _2^2)\), so by the known absolute moments of the normal distribution and Stirling’s approximation,
implying that \({\mathbb {E}}|\langle a, X \rangle |^p \le \left( \frac{C_1 e}{\sqrt{2}} L \right) ^p {\mathbb {E}} |\langle a, g \rangle |^p\). \(\square \)
In order to control arbitrary chaoses, we will derive a similar result as Lemma 14 for squared subgaussian and Gaussian variables. To achieve this, we make use of strong domination. The following theorem states that this can be used to compare squared subgaussian and Gaussian variables.
Lemma 15
There exist absolute constants \(\kappa , \lambda > 0\) such that the following holds. Let X be a random variable with \({\mathbb {E}} X^2 = 1\) and \(\Vert X\Vert _{\psi _2} \le L\), \(L \ge 1\) and \(g \sim N(0, 1)\). Let \(\xi ,\,\xi ' \in \{\pm 1\}\) be Rademacher variables that are independent of X and g. Then \(\xi (X^2 - 1) \prec _{(\kappa , \lambda L^2)} \xi ' (g^2 - 1)\) in the sense of Definition 2.
Proof
For any \(t > 0\),
For a constant \(c \ge 1\), the first term can be bounded by
The second term is 0 if \(t \ge 1\) since \(-(X^2 - 1) \le 1\). For \(t \le 1\), \(e^{- \frac{t}{c^2 L^2}} \ge e^{- \frac{1}{c^2 L^2}} \ge e^{- 1}\). Then it holds that \({\mathbb {P}}( - (X^2 - 1) > t) \le 1 \le e \cdot e^{- \frac{t}{c^2 L^2}}\), and altogether we obtain
On the other hand, for any \(\lambda > 0\),
To bound this, we use the following properties of the normal distribution: (see Proposition 7.5 in [3])
For \(0 < u \le \frac{1}{4}\), the second inequality in (12) yields
For \(u \ge \frac{1}{4}\), the first inequality in (12) gives \({\mathbb {P}} \left( |g |\ge \sqrt{1 + u} \right) \ge \frac{1}{5} \sqrt{\frac{2}{\pi }} \frac{1}{\sqrt{1 + u}} e^{- \frac{1 + u}{2} }\). Using that \(\frac{1}{\sqrt{1 + u}} \ge e^{-\frac{1}{2} u}\) for all \(u > 0\), we obtain for \(u \ge \frac{1}{4}\),
So for any \(u > 0\), \({\mathbb {P}}(|g |> \sqrt{1 + u}) \ge \frac{1}{17} e^{-u}\). By choosing \(\lambda = c^2\) and combining,
\(\square \)
Theorem 16
There is an absolute constant \(C > 0\) such that the following holds. Let \(X \in {\mathbb {R}}^n\) have independent entries that have mean 0 and variance 1 and are subgaussian with \(\psi _2\) norm \(\le L\) for an \(L \ge 1\). Take a Gaussian vector \(g \sim N(0, Id_n)\) and \(a \in {\mathbb {R}}^n\). Then
Proof
Consider independent Rademacher variables \(\xi _1, \ldots , \xi _n, {\bar{\xi }}_1, \ldots , {\bar{\xi }}_n \in \{\pm 1\}^n\) that are also independent of X and g. By the symmetrization Lemma 7, it holds that
Using that \(\xi _k (X^2 - 1) \prec _{(\kappa , \lambda L^2)} {\bar{\xi }}_k (g^2 - 1)\) by Lemma 15 and that \(|\cdot |^p\) is a convex nondecreasing function \({\mathbb {R}}^{+} \rightarrow {\mathbb {R}}^{+}\), Theorem 11 implies that there is a constant \({\tilde{C}} > 0\) such that
\(\square \)
Theorem 5 is an important tool for the proof of our decoupling result (Theorem 4). Its purpose is to rearrange a chaos in such a way that—under some changes—the quadratic factors that occur (here \((X^{(l)}_{{\varvec{i}}_l})^2\)) are replaced by corresponding mean 0 factors of the type \(\left[ (X^{(l)}_{{\varvec{i}}_l})^2 - 1 \right] \), which also occur in Theorem 4.
Rearranging the terms with this theorem enables an iterative application of the standard decoupling Theorem 8 in the proof of Theorem 4. Furthermore, the factors \(\left[ (X^{(l)}_{{\varvec{i}}_l})^2 - 1 \right] \) are 0 in the Rademacher case (Remark 3). In the general case, after the comparison with Gaussians, they will be turned into a product of two independent factors with the subsequent Lemma 17 in the proof of Theorem 3.
As the next step, we prove this Theorem 5.
Proof of Theorem 5
Observing that for any \(I \subset [d]\), \({\varvec{i}} \in {\varvec{J}}^{\varvec{n}}(I)\),
we obtain
This implies the claim using Lemma 9. \(\square \)
A key to the proof of the upper moment bound in our main result (Theorem 3) is the decoupling technique of Theorem 4. With the above auxiliary results, we can give the proof of it here.
Proof of Theorem 4
since each summand \({\varvec{i}}, {\varvec{i}}'\) is precisely considered in the sum for \(I = \{l \in [d]: {\varvec{i}}_l = {\varvec{i}}'_l\}\) and no other I.
Now applying Theorem 5 yields
Because of
and the triangle inequality, we obtain
For any fixed \(l_0 \in I^c\), we obtain that \(\Vert S_{I, J}\Vert _{L_p} =\)
We can apply the decoupling Theorem 8 to this for the convex function \(|\cdot |^p\) and the expectation conditioned on all variables except \(X^{(l_0)}\). This leads to \(\Vert S_{I, J}\Vert _{L_p} \le \)
Repeating this procedure iteratively for all other \(l \in I^c\), we obtain
Substituting this into (14) completes the proof. \(\square \)
The works in [16, 21] have investigated polynomials with higher powers of Gaussian variables. Since in our scenario, we only have two occurrences of every vector, thus we can repeatedly apply their result for the case of two coinciding indices. Considering that \(H_2(x) = x^2 - 1\) is the Hermite polynomial of degree 2 and leading coefficient 1, equation (2.9) in [16] in our setup can be written as follows. Note that as suggested there, the case \(p \ge 1\) can also be shown using Jensen’s inequality which can be used to show this inequality with coefficient 2.
Lemma 17
Let \(a \in {\mathbb {R}}^n\), \(g, {\bar{g}} \sim N(0, Id_n)\), \(p \ge 1\). Then
Combining the previous lemmas, now we can prove the upper bound in the main Theorem 3.
3.2.2 Proof of Theorem 3, upper bound
Step 1: Decoupling
Let \(\alpha := \Vert X^T A X - {\mathbb {E}} X^T A X\Vert _{L_p}\). By Theorem 4, \(\alpha \le \)
Step 2: Replacing the subgaussian factors by Gaussians
In (15), we can repeatedly apply Lemma 14 to replace all the linear subgaussian factors by Gaussian ones. Afterwards, Theorem 16 allows the same for the quadratic terms. Together, this yields that \(\alpha \le \)
Step 3: Decoupling of squared Gaussians In an analogous fashion as in step 2, we can successively replace all the factors \(\left[ (g^{(l)}_{{\varvec{i}}_l})^2 - 1 \right] \) in (16) by \(g^{(l)}_{{\varvec{i}}_l} {\bar{g}}^{(l)}_{{\varvec{i}}_l}\) using Lemma 17. This leads to
where for all \({\varvec{i}}, {\varvec{i}}' \in {\varvec{J}}^{\varvec{n}}(J \cup I^c)\),
Step 4: Completing the proof Then Theorem 2 yields that
where for \(S((J \cup I^c) \cup ((J \cup I^c) + d), \kappa )\) being the set of all partitions of \((J \cup I^c) \cup ((J \cup I^c) + d)\) into \(\kappa \) sets,
By Lemma 10, \(\Vert {\varvec{A}}^{(I, J)}\Vert _{I_1, \ldots , I_\kappa } \le \Vert {\varvec{A}}^{(I)}\Vert _{I_1, \ldots , I_\kappa }\) where \({\varvec{A}}^{(I)} = {\varvec{A}}^{(I, \emptyset )}\) as given in the statement of Theorem 3. Together with this, the upper bound in Theorem 3 follows.
3.3 Proof of the lower bound
3.3.1 Required tools
In this section, we will prove the lower bound in Theorem 3. Unlike the upper bound, we will only prove this for the case of Gaussian vectors. Indeed, for arbitrary subgaussian distributions, the lower bound fails to hold as the following simple example for the case \(d = 1\) shows: Consider the identity matrix \(Id_n\) and a Rademacher vector \(\xi \in \{\pm 1\}^n\). Then the object of interest in Theorem 3 is \(\xi ^T Id_n \xi - {\mathbb {E}}[\xi ^T Id_n \xi ] = 0\) even though the moment bounds \(m_p\) would be \(> 0\).
We follow the approach of reversing all steps in the proof of the upper bound, without the Gaussian comparison steps. This is why also the two decoupling steps before and after the Gaussian comparison can be performed together.
As mentioned before, Gaussian decoupling, with upper as well as lower bounds, has been studied in [16] where central ideas of [21] have been used. [16] provides a decoupling inequality for Gaussian chaos with an arbitrary number of coinciding indices. Similarly to Lemma 17, we can adapt the result of Equation (2.9) in [16] to our situation as follows.
Lemma 18
Let \(A \in {\mathbb {R}}^{n \times n}\) be a symmetric matrix, \(g, {\bar{g}} \sim N(0, Id_n)\) be independent, and \(p \ge 1\).
To generalize this to cases of multiple axes, we iteratively apply Lemma 18 to obtain the following corollary.
Corollary 19
Let \({\varvec{n}} \in {\mathbb {N}}^d\), \({\varvec{A}} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}\) such that \({\varvec{A}}\) satisfies the symmetry condition that for all \(l \in [d]\) and any \({\varvec{i}}, {\varvec{i}}' \in {\varvec{J}}^{\varvec{n}}([d] \backslash \{l\})\), \({\varvec{j}}, {\varvec{j}}' \in {\varvec{J}}^{\varvec{n}}(\{l\})\),
Let \(g^{(1)}, {\bar{g}}^{(1)} \sim N(0, Id_{{\varvec{n}}_1}), \ldots , g^{(d)}, {\bar{g}}^{(d)} \sim N(0, Id_{{\varvec{n}}_d})\) be independent. Then for any set \(I \subset [d]\), \(p \ge 1\),
Independently of the Gaussian decoupling approach, the following two lemmas provide a tool to reverse the application of the rearrangement result Theorem 5 in the proof of the upper bound.
Lemma 20
Let \({\varvec{A}} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}\) be an array of order 2d and \(X^{(1)} \in {\mathbb {R}}^{n_1}, \ldots X^{(d)} \in {\mathbb {R}}^{n_d}\) vectors. Then
Proof
Note that
Using this, we obtain
Observing that
we can conclude
Lemma 9 yields
such that
\(\square \)
Lemma 21
Let \({\varvec{A}} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}\) be an array of order 2d and \(X^{(1)} \in {\mathbb {R}}^{n_1}, \dots X^{(d)} \in {\mathbb {R}}^{n_d}\) independent random vectors with mean 0, variance 1 entries. Then for any subset \(\emptyset \ne I \subset [d]\), \(p \ge 1\),
where \(C(|I |)\) is a constant only depending on \(|I |\).
Proof
By the assumptions on the vectors \(X^{(l)}\),
Since this is exactly the term for \(I = \emptyset \) in Lemma 20, we obtain for the term on the right hand side of (19),
Using these terms, we need to show that \(\Vert S_I\Vert _{L_p} \le C(|I |) \Vert b\Vert _{L_p}\) for all \(\emptyset \ne I \subset [d]\).
Now we prove this by induction over \(|I |\). First assume \(I = \{l_0\}\). For any \(J \ne \emptyset , I\), there exists an \(l \in J \backslash I\) and then
since there is at least one factor whose conditional expectation is 0.
We conclude
where we used Jensen’s inequality on the conditional expectation in the last step.
Now assume that we have already shown (19) for all \(\emptyset \ne I' \subset [d]\) with \(|I' |< |I |\).
For all \(J \subset [d]\) such that \(J \ne \emptyset , I\), one of the following holds.
-
\(J \backslash I = \emptyset \), i.e., \(J \subset I\): Because \(J \ne I\), \(|J |< |I |\), so by induction
$$\begin{aligned} \Vert S_J\Vert _{L_p} \le C(|J |) \Vert b\Vert _{L_p}. \end{aligned}$$(20) -
\(J \backslash I \ne \emptyset \). Since there is an \(l' \in J \backslash I\),
$$\begin{aligned} {\mathbb {E}} \left[ \prod _{l \in J} \left[ X^{(l)}_{{\varvec{i}}_{l}} X^{(l)}_{{\varvec{i}}'_l} - \mathbbm {1}_{{\varvec{i}}_l = {\varvec{i}}'_l} \right] \,\bigg \vert \, (X^{(l)})_{l \in I} \right] = 0. \end{aligned}$$(21)
The triangle inequality yields together with (20), that \(\Vert S_I\Vert _{L_p} \le \)
The first term on the right hand side can be controlled with (21) and Jensen’s inequality,
So altogether \(\Vert S_I\Vert _{L_p} \le C(|I |) \Vert b\Vert _{L_p}\) where \(C(|I |) := \sum _{J \subset I: J \ne \emptyset , I} C(|J |) + 1\) depends only on \(|I |\). \(\square \)
Now we introduced all the necessary tools and can prove the lower bound of the main result, Theorem 3.
3.3.2 Proof of Theorem 3, lower bound
For any \(J \subset I \subset [d]\), define the array \({\varvec{A}}^{(I, J)}\) as in the proof of the upper bound (17) and
Step 1: Adding off-diagonal terms
Define independent Rademacher vectors \((\xi ^{(l)})_{l \in J}\) which are also independent of the \(g^{(1)}, \ldots g^{(d)}\), \({\bar{g}}^{(1)}, \ldots , {\bar{g}}^{(d)}\).
Noting that \({\mathbb {E}}_\xi [\xi ^{(l)}_{{\varvec{i}}_l} \xi ^{(l)}_{{\varvec{i}}'_l}] = \mathbbm {1}_{{\varvec{i}}_l = {\varvec{i}}'_l}\), we obtain
Substituting into (22) and applying Jensen’s inequality and Fubini’s theorem yields
By the symmetry of the normal distribution, conditioned on \((\xi ^{(l)})_{l \in J}\), \((\xi ^{(l)}_{{\varvec{i}}_l} g^{(l)}_{{\varvec{i}}_l}, \xi ^{(l)}_{{\varvec{i}}'_l} {\bar{g}}^{(l)}_{{\varvec{i}}'_l})\) and \((g^{(l)}_{{\varvec{i}}_l}, {\bar{g}}^{(l)}_{{\varvec{i}}'_l})\) have the same distribution. So we can conclude
Step 2: Inverse Gaussian decoupling
For every \(J \subset I \subset [d]\), we obtain then by the symmetry of \({\varvec{A}}\) and Corollary 19,
Step 3: Removing the mean subtractions in every factor
Since \(I \backslash J \ne [d]\), \(J \cup I^c \ne \emptyset \) and Lemma 21 provides that \(\alpha ^{(I, J)} \le \)
Adding this up over all \(J \subset I \subset [d]\), \(I \backslash J \ne [d]\) yields
where \(C(d) := \sum _{\begin{array}{c} J \subset I \subset [d]: I \backslash J \ne [d] \end{array}} C_1(|J \cup I^c |)\) depends only on d.
Step 4: Completing the proof
Restricting the left hand side in (23) to the terms in which \(J = \emptyset \). The remaining terms \(\alpha ^{(I, \emptyset )}\) only contain the arrays \({\varvec{A}}^{(I, \emptyset )}\) which are equal to the \({\varvec{A}}^{(I)}\) from the theorem statement. Subsequently, we can bound the \(\alpha ^{(I, \emptyset )}\) from below using Theorem 2 (similarly to the upper bound) to obtain the lower bound in Theorem 3.
3.4 Concentration of \(\Vert B X \Vert _2\)
In this section, we apply our main results to the concentration of \(\Vert B X\Vert _2\) where \(X = X^{(1)} \otimes \dots \otimes X^{(d)}\) is a Kronecker product of independent vectors with subgaussian entries. The following statement is a direct consequence from Theorem 3 and Lemma 12.
Corollary 22
Let \(B \in {\mathbb {R}}^{n_0 \times N}\) be a matrix where \(N = n_1 \cdots n_d\) and \(X := X^{(1)} \otimes \cdots \otimes X^{(d)} \in {\mathbb {R}}^N\) a random vector as in Theorem 3.
Let \({\varvec{A}} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}\) be the rearrangement of the matrix \(A = B^* B\) as an array with 2d axes. For any \(I \subset [d]\), define the array \({\varvec{A}}^{(I)}\) as in (10).
For \(T \subset [2 d]\), \(1 \le \kappa \le 2d\), denote \(S(T, \kappa )\) for the set of partitions of T into \(\kappa \) sets and \(I^c = [d] \backslash I\). Define for any \(p \ge 1\) and any \(\kappa \in [2d]\),
Then there is a constant \(C(d) > 0\), depending only on d, such that for all \(p \ge 1\),
If in addition, \(X^{(1)} \sim N(0, Id_{n_1}), \ldots , X^{(d)} \sim N(0, Id_{n_d})\) are normally distributed (i.e., L is constant) and \({\varvec{A}}\) satisfies the symmetry condition (11), then also the lower bound
holds for all \(p \ge 1\). Above, \({\tilde{C}}(d) > 0\) that depends only on d.
Lemma 23
Let \({\varvec{B}} \in {\mathbb {R}}^{n_1 \times \dots \times n_d}\). Assume that \(I_1, \ldots , I_\kappa \) is a partition of [d]. Let \({\bar{I}}_{\kappa } \cup {\bar{I}}_{\kappa + 1} = I_\kappa \) be a partition into two subsets. Then
Proof
Take arrays \({\varvec{\alpha }}^{(1)} \in {\mathbb {R}}^{{\varvec{n}}}(I_1), \ldots , {\varvec{\alpha }}^{(\kappa - 1)} \in {\mathbb {R}}^{{\varvec{n}}}(I_{\kappa - 1}), \bar{{\varvec{\alpha }}}^{(\kappa )} \in {\mathbb {R}}^{{\varvec{n}}}({\bar{I}}_{\kappa }), \bar{{\varvec{\alpha }}}^{(\kappa + 1)} \in {\mathbb {R}}^{{\varvec{n}}}({\bar{I}}_{\kappa + 1})\), with Frobenius norm 1 each, such that \(\Vert {\varvec{B}}\Vert _{I_1, \ldots , I_{\kappa - 1}, {\bar{I}}_{\kappa }, {\bar{I}}_{\kappa + 1}} = \sum _{{\varvec{i}} \in {\varvec{J}}^{\varvec{n}}} B_{{\varvec{i}}} \alpha ^{(1)}_{{\varvec{i}}_{I_1}} \cdots \alpha ^{(\kappa - 1)}_{{\varvec{i}}_{I_{\kappa - 1}}} {\bar{\alpha }}^{(\kappa )}_{{\varvec{i}}_{{\bar{I}}_{\kappa }}} {\bar{\alpha }}^{(\kappa + 1)}_{{\varvec{i}}_{{\bar{I}}_{\kappa + 1}}}\). Now define \({\varvec{\alpha }}^{(\kappa )} \in {\mathbb {R}}^{{\varvec{n}}}(I_\kappa )\) by \(\alpha ^{(\kappa )}_{{\varvec{i}}} = {\bar{\alpha }}^{(\kappa )}_{{\varvec{i}}_{{\bar{I}}_{\kappa }}} {\bar{\alpha }}^{(\kappa + 1)}_{{\varvec{i}}_{{\bar{I}}_{\kappa + 1}}}\) for every \({\varvec{i}} \in {\varvec{J}}^{\varvec{n}}(I_\kappa )\). Then \(\Vert {\varvec{\alpha }}^{(\kappa )}\Vert _{2} = 1\) and by the definition of \(\Vert \cdot \Vert _{I_1, \ldots , I_\kappa }\) as the supremum over \({\varvec{\alpha }}^{(1)}, \ldots , {\varvec{\alpha }}^{(\kappa )}\), we obtain
which proves the first inequality.
To prove the second inequality, take arrays \({\varvec{\alpha }}^{(1)} \in {\mathbb {R}}^{{\varvec{n}}}(I_1), \ldots , {\varvec{\alpha }}^{(\kappa )} \in {\mathbb {R}}^{{\varvec{n}}}(I_\kappa )\) such that
Now define \(\tilde{{\varvec{B}}} \in {\mathbb {R}}^{{\varvec{n}}}(I_\kappa )\) such that for all \({\varvec{i}} \in {\varvec{J}}^{\varvec{n}}({\bar{I}}_\kappa ), {\varvec{j}} \in {\varvec{J}}^{\varvec{n}}({\bar{I}}_{\kappa + 1})\),
For \(N_1 := \prod _{l \in {\bar{I}}_{\kappa }} n_l\) and \(N_2 := \prod _{l \in {\bar{I}}_{\kappa + 1}} n_l\), we can interpret \(\tilde{{\varvec{B}}}\) as a matrix \({\tilde{B}} \in {\mathbb {R}}^{N_1 \times N_2}\) with rows indexed by \({\varvec{i}} \in {\varvec{J}}^{\varvec{n}}({\bar{I}}_\kappa )\) and columns indexed by \({\varvec{j}} \in {\varvec{J}}^{\varvec{n}}({\bar{I}}_{\kappa + 1})\).
Then
such that
where by definition the maximum is attained at \({\varvec{\beta }} = {\varvec{\alpha }}^{(\kappa )}\), implying
For the spectral norm, we obtain from the definition of \(\Vert \cdot \Vert _{I_1, \ldots , I_{\kappa - 1}, {\bar{I}}_{\kappa }, {\bar{I}}_{\kappa + 1}}\),
The second inequality now follows from (24), (25) and the general property of matrices that
\(\square \)
Lemma 24
Let \({\varvec{A}} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}\), \(I \subset [d]\). Define \({\varvec{A}}^{(I)}\) as in (10).
Let \(I_1, \ldots , I_{\kappa }\) be a partition of \(([d] \backslash I) \cup (d + ([d] \backslash I))\). Let \(I_{\kappa +1}, \ldots , I_{\kappa + |I |}\) be the sets \(\{j, j+d\}\) for every \(j \in I\). Then \(I_1, \ldots , I_{\kappa + |I |}\) is a partition of [2d] and
Proof
Take \({\varvec{\alpha }}^{(1)} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}(I_1), \dots , {\varvec{\alpha }}^{(\kappa )} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}(I_\kappa )\), all having a Frobenius norm of 1, such that
Now define \({\varvec{\alpha }}^{(\kappa + 1)} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}(\{j_1, j_1 + d\}), \ldots , {\varvec{\alpha }}^{(\kappa + |I |)} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}(\{j_{|I |}, j_{|I |} + d\})\) (where \(I = \{j_1, \ldots , j_{|I |}\}\)) such that for all \(r \in [|I |]\) and \({\varvec{i}} \in {\varvec{J}}^{{\varvec{n}}^{\times 2}}(\{j_r, j_r + d\})\),
Then for \({\varvec{i}} \in {\varvec{J}}^{{\varvec{n}}^{\times 2}}(I \cup (I + d))\)
Substituting this into (26) yields
\(\square \)
Using the aforementioned results, we can give the proof of Theorem 6 about \(\Vert B (X^{(1)} \otimes \cdots \otimes X^{(d)})\Vert _2\) in which we find suitable bounds for all the tensor norms of \(B^* B\) in terms of \(\Vert B\Vert _{2 \rightarrow 2}\) and \(\Vert B\Vert _F\).
Proof of Theorem 6
Let \(A := B^* B \in {\mathbb {R}}^{n^d \times n^d}\) and \({\varvec{A}} \in {\mathbb {R}}^{{\varvec{n}}^{\times 2}}\) be the corresponding array of order 2d obtained by rearranging A for \({\varvec{n}} = (n, \ldots , n)\). Note that here the dimensions along all axes are equal. For \(I \subset [2 d]\), define \({\varvec{A}}^{(I)}\) as in Corollary 22.
Step 1: Showing the norm inequalities
In both cases, we start by extending \(I_1, \ldots , I_\kappa \) to \(I_1, \dots , I_{\kappa + |I |}\) as in Lemma 24, obtaining
Then the first inequality of (27) follows by repeatedly joining all the sets \(I_1, \ldots , I_{\kappa + |I |}\) in the sense of Lemma 23 (first inequality) yielding \(\Vert {\varvec{A}}\Vert _{I_1, \ldots , I_{\kappa + |I |}} \le \Vert {\varvec{A}}\Vert _{[2 d]} = \Vert A\Vert _F\).
For the second inequality in (27), we distinguish two cases. First assume that \(\kappa \le d - |I |\). Then \(|I |\le d - \kappa \). Since A is a matrix in \({\mathbb {R}}^{n^d \times n^d}\), \(\Vert A\Vert _{2 \rightarrow 2} \le n^\frac{d}{2} \Vert A\Vert _F\) and with the first inequality in (27), we obtain
In the other case that \(\kappa > d - |I |\), denote \(\kappa '\) for the number of sets among \(I_1, \ldots , I_\kappa \) that only contain one element. Since each of the other sets must contain at least two elements, this leads to the inequality
This implies that among \(I_1, \ldots , I_\kappa \), there must be at least \(\kappa - d + |I |\) sets with exactly one element that are all contained in [d] or all contained in \([2 d] \backslash [d]\). Without loss of generality, we can assume that these are \(I_1, \ldots , I_{\kappa - d + |I |}\). Now take the unions \({\bar{I}}_1 := I_1 \cup \cdots \cup I_{\kappa - d + |I |}\) and \({\bar{I}}_2 := I_{\kappa - d + |I |+ 1} \cup \dots \cup I_{\kappa + |I |}\). With (28) and the first inequality of Lemma 23, we obtain
Now split up \({\bar{I}}_2\) into \({\bar{I}}_{2, 1} := {\bar{I}}_2 \cap [d]\) and \({\bar{I}}_{2, 2} := {\bar{I}}_2 \cap ([2 d] \backslash [d])\). If neither \({\bar{I}}_{2, 1}\) nor \({\bar{I}}_{2, 2}\) is empty, then with the second inequality of Lemma 23, we obtain
where in the last step we used the first inequality in Lemma 23 with the fact that \({\bar{I}}_1 \cup {\bar{I}}_{2, 1} \cup {\bar{I}}_{2, 2} = [2 d]\) and each of these three sets is contained in either [d] or \([2 d] \backslash [d]\). Note that the inequality between the first and the third term still holds in the case that \({\bar{I}}_{2, 1}\) or \({\bar{I}}_{2, 2}\) is empty and thus Lemma 8.4 cannot be applied in the first step.
Now assume \({\bar{I}}_1 \subset [d]\) (otherwise \({\bar{I}}_1 \subset [2 d] \backslash [d]\) and the proof works analogously). Then \({\bar{I}}_1 \cup {\bar{I}}_{2, 1} = [d]\) and \({\bar{I}}_{2, 1} = [2 d] \backslash [d]\). So \(\min \{|{\bar{I}}_{2, 1} |, |{\bar{I}}_{2, 2} |\} = |{\bar{I}}_{2, 1} |= d - |{\bar{I}}_1 |= d - (\kappa - d + |I |) = 2d - \kappa - |I |\). This implies
This completes the proof of (27).
Step 2: Moment and tail bounds
Now, use Corollary 22 and its notation of \(m_{p, \kappa }\) and \(m_p\). The number of terms in the sum of the definition of \(m_{p, \kappa }\) only depends on d. This fact together with (27) leads to
where \(C_1(d)\) is a constant depending only on d. Furthermore, we obtain
Since this is an upper bound on the \(L_p\) norm of \(\Vert B X\Vert _2 - \Vert B\Vert _F\), Lemma 13 implies
where
Now, for each of multiple different ranges of t, we select one of the four terms in (29).
Step 3: Bound for \(t \le n^\frac{d}{2} \Vert B\Vert _{2 \rightarrow 2}\)
For \(\kappa = 1\), we obtain using the first term in (29), \(\beta _1 \ge \left( {t} / {(n^{\frac{d - 1}{2}} \Vert B\Vert _{2 \rightarrow 2})} \right) ^2\).
For \(\kappa \ge 2\), we can use the fourth term in (29) to show the same bound because
This implies that
Step 5: Bound for \(t \ge n^\frac{d}{2} \Vert B\Vert _{2 \rightarrow 2}\)
For all \(\kappa \in [2 d]\), using the fourth term in (29) yields
such that
Step 6: Bound for \(n^{\frac{d - 1}{4}} \Vert B\Vert _{2 \rightarrow 2} \le t \le n^{\frac{d - 1}{4}} \Vert B\Vert _F\)
Using the third term in (29), we obtain that
implying
\(\square \)
4 Discussion
In total, for a chaos of the type
we have shown moment bounds that are tight (up to dependence on d) for the Gaussian case. Along with this, we have also shown a specific decoupling inequality for the above expression and improved moment and tail bounds for \(\Vert B(X^{(1)} \otimes \dots \otimes X^{(d)})\Vert _2\).
The application [14] generalizes the result in [5] on constructing Johnson–Lindenstrauss embeddings from matrices satisfying the restricted isometry property to Johnson–Lindenstrauss embeddings with a fast transformation of Kronecker products. This leads to expressions of the type \(\Vert \Phi D_\xi x\Vert _2^2\), where \(\Phi \in {\mathbb {R}}^{m \times N}\) is a matrix, \(x \in {\mathbb {R}}^N\) a vector, and \(D_\xi \in {\mathbb {R}}^{N \times N}\) is a diagonal matrix with entries from \(\xi = \xi ^{(1)} \otimes \dots \otimes \xi ^{(d)}\), where \(\xi ^{(1)}, \ldots , \xi ^{(d)}\) are independent Rademacher vectors. Then \(\Vert \Phi D_\xi x\Vert _2^2\) can be rewritten as a chaos of the above type with the Rademacher vectors \(\xi ^{(l)}\) as \(X^{(l)}\). This chaos is controlled with the decoupling statement of Theorem 4, more specifically the Rademacher case (Remark 3), in which the terms significantly simplify. After some necessary intermediate steps, the upper moment bounds of Theorem 3 are applied to the resulting decoupled chaos.
References
Hanson, D.L., Wright, F.T.: A bound on tail probabilities for quadratic forms in independent random variables. Ann. Math. Stat. 42(3), 1079–1083 (1971)
Rudelson, M., Vershynin, R.: Hanson–Wright inequality and sub-gaussian concentration. Electron. Commun. Probab. 18, 9 (2013)
Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Applied and Numerical Harmonic Analysis. Birkhäuser, Basel (2013). https://doi.org/10.1007/978-0-8176-4948-7
Vershynin, R.: High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge (2018)
Krahmer, F., Ward, R.: New and improved Johnson–Lindenstrauss embeddings via the restricted isometry property. SIAM J. Math. Anal. 43(3), 1269–1281 (2011)
Adamczak, R., Wolff, P.: Concentration inequalities for non-Lipschitz functions with bounded derivatives of higher order. Probab. Theory Relat. Fields 162(3), 531–586 (2015)
Latała, R.: Estimates of moments and tails of Gaussian chaoses. Ann. Probab. 34(6), 2315–2331 (2006)
Adamczak, R.A., Latała, R.: Tail and moment estimates for chaoses generated by symmetric random variables with logarithmically concave tails. Ann. l’I.H.P. Probab. Stat. 48(4), 1103–1136 (2012)
Kolesko, K., Latała, R.: Moment estimates for chaoses generated by symmetric random variables with logarithmically convex tails. Stat. Probab. Lett. 107, 210–214 (2015)
Battaglino, C., Ballard, G., Kolda, T.G.: A practical randomized CP tensor decomposition. SIAM J. Matrix Anal. Appl. 39(2), 876–901 (2018)
Ahle, T.D., Kapralov, M., Knudsen, J.B.T., Pagh, R., Velingker, A., Woodruff, D.P., Zandieh, A.: Oblivious sketching of high-degree polynomial kernels. In: SODA, pp. 141–160 (2020)
Jin, R., Kolda, T.G., Ward, R.: Faster Johnson–Lindenstrauss transforms via Kronecker products. Inf. Inference (2020). https://doi.org/10.1093/imaiai/iaaa028
Vershynin, R.: Concentration inequalities for random tensors. Bernoulli 26(4), 3139–3162 (2020)
Bamberger, S., Krahmer, F., Ward, R.: Johnson–Lindenstrauss embeddings with Kronecker structure (2021). arXiv:2106.13349
Boucheron, S., Bousquet, O., Lugosi, G., Massart, P.: Moment inequalities for functions of independent random variables. Ann. Probab. 33(2), 514–560 (2005)
Arcones, M.A., Giné, E.: On decoupling, series expansions, and tail behavior of chaos processes. J. Theor. Probab. 6, 101–122 (1993)
Meller, R.: Two-sided moment estimates for a class of nonnegative chaoses. Stat. Probab. Lett. 119, 213–219 (2016)
Meller, R.: Tail and moment estimates for a class of random chaoses of order two. Stud. Math. 249, 1–32 (2019)
Götze, F., Sambale, H., Sinulis, A.: Concentration inequalities for polynomials in \(\alpha \)-sub-exponential random variables. Electron. J. Probab. 26, 1–22 (2021)
McConnell, T.R., Taqqu, M.S.: Decoupling inequalities for multilinear forms in independent symmetric random variables. Ann. Probab. 14(3), 943–954 (1986)
Kwapien, S.: Decoupling inequalities for polynomial chaos. Ann. Probab. 15(3), 1062–1071 (1987)
Adamczak, R., Latała, R., Meller, R.: Moments of gaussian chaoses in Banach spaces. Electron. J. Probab. 26, 1–36 (2021)
Vershynin, R.: Quadratic Forms, Symmetrization, and Contraction. Cambridge Series in Statistical and Probabilistic Mathematics, pp. 127–146. Cambridge University Press, Cambridge (2018)
de la Pena, V.H.: Decoupling and Khintchine’s inequalities for \(u\)-statistics. Ann. Probab. 20(4), 1877–1892 (1992)
de la Peña, V., Giné, E.: Decoupling. Probability and Its Applications. Springer, New York (1999)
Bourgain, J., Tzafriri, L.: Invertibility of ‘large’ submatrices with applications to the geometry of Banach spaces and harmonic analysis. Isr. J. Math. 57(2), 137–224 (1987)
Mazur, S., Orlicz, W.: Grundlegende eigenschaften der polynomischen operationen. Erste mitteilung. Stud. Math. 1(5), 50–68 (1934)
de la Peña, V.H., Montgomery-Smith, S.J.: Decoupling inequalities for the tail probabilities of multivariate u-statistics. Ann. Probab. 806–816 (1995)
Kwapien, S., Woyczynski, W.: Random Series and Stochastic Integrals: Single and Multiple. Birkhäuser, Boston (1992)
Vershynin, R.: In: Eldar, Y.C., Kutyniok, G. (eds.) Introduction to the Non-asymptotic Analysis of Random Matrices, pp. 210–268. Cambridge University Press, Cambridge (2012)
Acknowledgements
R.W. is supported by Air Force Office of Scientific Research (AFOSR) MURI FA9550-19-1-0005, National Science Foundation (NSF) DMS 1952735, and NSF IFML 2019844. S.B. and F.K. have been supported by the German Science Foundation (DFG) in the context of the Emmy-Noether Junior Research Group KR4512/1-2. R.W. and F.K. gratefully acknowledge support from the Institute for Advanced Study, where this project was initiated.
Funding
Open Access funding enabled and organized by Projekt DEAL. R.W. is supported by AFOSR MURI FA9550-19-1-0005, NSF DMS 1952735, and NSF IFML 2019844. S.B. and F.K. have been supported by the German Science Foundation (DFG) in the context of the Emmy-Noether Junior Research Group KR4512/1-2.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Communicated by Johannes Maly.
This article is part of the topical collection “Recent advances in computational harmonic analysis” edited by Dae Gwan Lee, Ron Levie, Johannes Maly and Hanna Veselovska.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Bamberger, S., Krahmer, F. & Ward, R. The Hanson–Wright inequality for random tensors. Sampl. Theory Signal Process. Data Anal. 20, 14 (2022). https://doi.org/10.1007/s43670-022-00028-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s43670-022-00028-4