Notes on Computational Hardness of Hypothesis Testing: Predictions Using the Low-Degree Likelihood Ratio

Abstract

These notes survey and explore an emerging method, which we call the low-degree method, for understanding statistical-versus-computational tradeoffs in high-dimensional inference problems. In short, the method posits that a certain quantity—the second moment of the low-degree likelihood ratio—gives insight into how much computational time is required to solve a given hypothesis testing problem, which can in turn be used to predict the computational hardness of a variety of statistical inference tasks. While this method originated in the study of the sum-of-squares (SoS) hierarchy of convex programs, we present a self-contained introduction that does not require knowledge of SoS. In addition to showing how to carry out predictions using the method, we include a discussion investigating both rigorous and conjectural consequences of these predictions. These notes include some new results, simplified proofs, and refined conjectures. For instance, we point out a formal connection between spectral methods and the low-degree likelihood ratio, and we give a sharp low-degree lower bound against subexponential-time algorithms for tensor PCA.

Appendices

Appendix 1: Omitted Proofs

Neyman-Pearson Lemma

We include here, for completeness, a proof of the classical Neyman–Pearson lemma [87].

Proof of Lemma 1

Note first that a test f is completely determined by its rejection region, $R_f = \{\boldsymbol Y: f(\boldsymbol Y) = \mathbb {P}\}$. We may rewrite the power of f as

$$\displaystyle \begin{aligned} 1 - \beta(f) = \mathbb{P}[f(\boldsymbol Y) = \mathbb{P}] = \int_{R_f}d\mathbb{P}(\boldsymbol Y) = \int_{R_f}L(\boldsymbol Y)d\mathbb{Q}(\boldsymbol Y). \end{aligned}$$

On the other hand, our assumption on α(f) is equivalent to

$$\displaystyle \begin{aligned} \mathbb{Q}[R_f] \leq \mathbb{Q}[L(\boldsymbol Y) > \eta]. \end{aligned}$$

Thus, we are interested in solving the optimization

$$\displaystyle \begin{aligned} \begin{array}{ll} \text{maximize} & \int_{R_f}L(\boldsymbol Y)d\mathbb{Q}(\boldsymbol Y) \\ \text{subject to} & R_f \in \mathscr{F}, \\ & \mathbb{Q}[R_f] \leq \mathbb{Q}[L(\boldsymbol Y) > \eta]. \end{array} \end{aligned}$$

From this form, let us write , then the difference of powers is

$$\displaystyle \begin{aligned} (1 - \beta(L_\eta)) - (1 - \beta(f)) &= \int_{R_\star}L(\boldsymbol Y)d\mathbb{Q}(\boldsymbol Y) - \int_{R_f}L(\boldsymbol Y)d\mathbb{Q}(\boldsymbol Y) \\ &= \int_{R_\star \setminus R_f}L(\boldsymbol Y)d\mathbb{Q}(\boldsymbol Y) - \int_{R_f \setminus R_\star}L(\boldsymbol Y)d\mathbb{Q}(\boldsymbol Y) \\ &\geq \eta\left(\mathbb{Q}[R_\star \setminus R_f] - \mathbb{Q}[R_f \setminus R_\star]\right) \\ &= \eta\left(\mathbb{Q}[R_\star] - \mathbb{Q}[R_f]\right) \\ &\geq 0, \end{aligned} $$

completing the proof.

Equivalence of Symmetric and Asymmetric Noise Models

For technical convenience, in the main text we worked with an asymmetric version of the spiked Wigner model (see Sect. 3.2), Y  = λxx ^⊤ + Z where Z has i.i.d. $\mathscr {N}(0,1)$ entries. A more standard model is to instead observe $\widetilde {\boldsymbol Y} = \frac {1}{2}(\boldsymbol Y + \boldsymbol Y^\top ) = \lambda \boldsymbol x \boldsymbol x^\top + \boldsymbol W$, where W is symmetric with $\mathscr {N}(0,1)$ diagonal entries and $\mathscr {N}(0,1/2)$ off-diagonal entries, all independent. These two models are equivalent, in the sense that if we are given a sample from one then we can produce a sample from the other. Clearly, if we are given Y , we can symmetrize it to form $\widetilde {\boldsymbol Y}$. Conversely, if we are given $\widetilde {\boldsymbol Y}$, we can draw an independent matrix G with i.i.d. $\mathscr {N}(0,1)$ entries, and compute $\widetilde {\boldsymbol Y} + \frac {1}{2}(\boldsymbol G - \boldsymbol G^\top )$; one can check that the resulting matrix has the same distribution as Y (we are adding back the “skew-symmetric part” that is present in Y but not $\widetilde {\boldsymbol Y}$).

In the spiked tensor model (see Sect. 3.1), our asymmetric noise model is similarly equivalent to the standard symmetric model defined in [93] (in which the noise tensor Z is averaged over all permutations of indices). Since we can treat each entry of the symmetric tensor separately, it is sufficient to show the following one-dimensional fact: for unknown $x \in \mathbb {R}$, k samples of the form $y_i = x + \mathscr {N}(0,1)$ are equivalent to one sample of the form $\tilde y = x + \mathscr {N}(0,1/k)$. Given {y _i}, we can sample $\tilde y$ by averaging: $\frac {1}{k}\sum _{i=1}^k y_i$. For the converse, fix unit vectors a ₁, …, a _k at the corners of a simplex in $\mathbb {R}^{k-1}$; these satisfy $\langle \boldsymbol a_i,\boldsymbol a_j \rangle = -\frac {1}{k-1}$ for all i≠j. Given $\tilde y$, draw $\boldsymbol u \sim \mathscr {N}(0,{\boldsymbol I}_{k-1})$ and let $y_i = \tilde y + \sqrt {1-1/k} \,\langle \boldsymbol a_i,\boldsymbol u \rangle$; one can check that these have the correct distribution.

Low-Degree Analysis of Spiked Wigner Above the PCA Threshold

Proof of Theorem 6

We follow the proof of Theorem 2(ii) in Sect. 3.1.2. For any choice of d ≤ D, using the standard bound $\binom {2d}{d} \ge 4^d/(2\sqrt {d})$,

$$\displaystyle \begin{aligned} \|L_n^{\le D}\|{}^2 &\ge \frac{\lambda^{2d}}{d!} \operatorname*{\mathbb{E}}_{\boldsymbol x^1,\boldsymbol x^2}[\langle \boldsymbol x^1,\boldsymbol x^2 \rangle^{2d}] \\ &\ge \frac{\lambda^{2d}}{d!} \binom{n}{d} \frac{(2d)!}{2^{d}} \\ &= \frac{\lambda^{2d}}{d!} \frac{n!}{d!(n-d)!} \frac{(2d)!}{2^{d}} \\ &= \lambda^{2d} \binom{2d}{d} \frac{n!}{(n-d)! 2^d} \\ &\ge \lambda^{2d} \frac{4^d}{2\sqrt{d}} \frac{(n-d)^d}{2^d} \\ &= \frac{1}{2\sqrt{d}} \left(2\lambda^2 (n-d)\right)^d \\ &= \frac{1}{2\sqrt{d}} \left(\hat\lambda^2 \left(1 - \frac{d}{n}\right)\right)^d.\end{aligned} $$

(using the moment bound12 from Section3.1.2)

Since $\hat \lambda > 1$, this diverges as n →∞ provided we choose d ≤ D with ω(1) ≤ d ≤ o(n).

Appendix 2: Omitted Probability Theory Background

Hermite Polynomials

Here we give definitions and basic facts regarding the Hermite polynomials (see, e.g, [101] for further details), which are orthogonal polynomials with respect to the standard Gaussian measure.

Definition 15

The univariate Hermite polynomials are the sequence of polynomials $h_k(x) \in \mathbb {R}[x]$ for k ≥ 0 defined by the recursion

$$\displaystyle \begin{aligned} h_0(x) &= 1, \\ h_{k + 1}(x) &= xh_k(x) - h_k^\prime(x).\end{aligned} $$

The normalized univariate Hermite polynomials are $\widehat {h}_k(x) = h_k(x) / \sqrt {k!}$.

The following is the key property of the Hermite polynomials, which allows functions in $L^2(\mathscr {N}(0, 1))$ to be expanded in terms of them.

Proposition 10

The normalized univariate Hermite polynomials form a complete orthonormal system of polynomials for $L^2(\mathscr {N}(0, 1))$.

The following are the multivariate generalizations of the above definition that we used throughout the main text.

Definition 16

The N-variate Hermite polynomials are the polynomials for $\boldsymbol \alpha \in \mathbb {N}^N$. The normalizedN-variate Hermite polynomials inNvariables are the polynomials for $\boldsymbol \alpha \in \mathbb {N}^N$.

Again, the following is the key property justifying expansions in terms of these polynomials.

Proposition 11

The normalized N-variate Hermite polynomials form a complete orthonormal system of (multivariate) polynomials for $L^2(\mathscr {N}(\boldsymbol 0, \boldsymbol I_N))$.

For the sake of completeness, we also provide proofs below of the three identities concerning univariate Hermite polynomials that we used in Sect. 2.3 to derive the norm of the LDLR under the additive Gaussian noise model. It is more convenient to prove these in a different order than they were presented in Sect. 2.3, since one identity is especially useful for proving the others.

Proof of Proposition 8 , Integration by Parts

Recall that we are assuming a function $f: \mathbb {R} \to \mathbb {R}$ is k times continuously differentiable and f and its derivatives are $O(\exp (|x|{ }^\alpha ))$ for α ∈ (0, 2), and we want to show the identity

$$\displaystyle \begin{aligned} \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_k(y) f(y)] = \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}\left[ \frac{d^k f}{dy^k}(y)\right]. \end{aligned}$$

We proceed by induction. Since h ₀(y) = 1, the case k = 0 follows immediately. We also verify by hand the case k = 1, with h ₁(y) = y:

$$\displaystyle \begin{aligned} \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}\left[yf(y) \right] &= \frac{1}{\sqrt{2\pi}} \int_{-\infty}^\infty f(y) \cdot ye^{-y^2 / 2}dy \\ &= \frac{1}{\sqrt{2\pi}} \int_{-\infty}^\infty f^\prime(y) e^{-y^2 / 2}dy \\ &= \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}\left[f^\prime(y) \right], \end{aligned} $$

where we have used ordinary integration by parts.

Now, suppose the identity holds for all degrees smaller than some k ≥ 2, and expand the degree k case according to the recursion:

$$\displaystyle \begin{aligned} \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_k(y) f(y)] &= \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[y h_{k - 1}(y) f(y)] - \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_{k - 1}^\prime(y) f(y)] \\ &= \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_{k - 1}^\prime(y)f(y)] + \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_{k - 1}(y)f^\prime(y)] \\ &\hspace{3.5cm} - \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_{k - 1}^\prime(y) f(y)] \\ &= \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_{k - 1}(y)f^\prime(y)] \\ &= \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}\left[\frac{d^k f}{dy^k}(y)\right], \end{aligned} $$

where we have used the degree 1 and then the degree k − 1 hypotheses.

Proof of Proposition 7 , Translation Identity

Recall that we want to show, for all k ≥ 0 and $\mu \in \mathbb {R}$, that

$$\displaystyle \begin{aligned} \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(\mu, 1)}[h_k(y)] = \mu^k. \end{aligned}$$

We proceed by induction on k. Since h ₀(y) = 1, the case k = 0 is immediate. Now, suppose the identity holds for degree k − 1, and expand the degree k case according to the recursion:

$$\displaystyle \begin{aligned} \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(\mu, 1)}[h_k(y)] &= \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_k(\mu + y)] \\ &= \mu \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_{k - 1}(\mu + y)] + \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[y h_{k - 1}(\mu + y)] \\ &\hspace{3.75cm} - \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_{k - 1}^\prime(\mu + y)] \end{aligned} $$

which may be simplified by the Gaussian integration by parts to

$$\displaystyle \begin{aligned} &= \mu \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_{k - 1}(\mu + y)] + \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_{k - 1}^\prime(\mu + y)] \\ &\hspace{3.75cm} - \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_{k - 1}^\prime(\mu + y)] \\ &= \mu \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}[h_{k - 1}(\mu + y)], \end{aligned} $$

and the result follows by the inductive hypothesis.

Proof of Proposition 9 , Generating Function

Recall that we want to show the series identity for any $x, y \in \mathbb {R}$,

$$\displaystyle \begin{aligned} \exp\left(xy - \frac{1}{2}x^2\right) = \sum_{k = 0}^\infty \frac{1}{k!}x^k h_k(y). \end{aligned}$$

For any fixed x, the left-hand side belongs to $L^2(\mathscr {N}(0, 1))$ in the variable y. Thus this is merely a claim about the Hermite coefficients of this function, which may be computed by taking inner products. Namely, let us write

then using Gaussian integration by parts,

$$\displaystyle \begin{aligned} \langle f_x, \widehat{h}_k \rangle &= \frac{1}{\sqrt{k!}}\operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}\left[f_x(y) h_k(y)\right] \\ &= \frac{1}{\sqrt{k!}}\operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}\left[\frac{d^k f_x}{dy^k}(y) \right] \\ &= \frac{1}{\sqrt{k!}}x^k \operatorname*{\mathbb{E}}_{y \sim \mathscr{N}(0, 1)}\left[f_x(y) \right]. \end{aligned} $$

A simple calculation shows that $\mathbb {E}_{y \sim \mathscr {N}(0, 1)}[f_x(y)] = 1$ (this is an evaluation of the Gaussian moment-generating function that we have mentioned in the main text), and then by the Hermite expansion

$$\displaystyle \begin{aligned} f_x(y) = \sum_{k = 0}^\infty \langle f_x, \widehat{h}_k \rangle \widehat{h}_k(y) = \sum_{k = 0}^\infty \frac{1}{k!}x^k h_k(y), \end{aligned}$$

giving the result.

Subgaussian Random Variables

Many of our rigorous arguments rely on the concept of subgaussianity, which we now define. See, e.g., [92] for more details.

Definition 17

For σ ² > 0, we say that a real-valued random variable π is σ ²-subgaussian if $\mathbb {E}[\pi ] = 0$ and for all $t \in \mathbb {R}$, the moment-generating function $M(t) = \mathbb {E}[\exp (t \pi )]$ of π exists and is bounded by $M(t) \le \exp (\sigma ^2 t^2 / 2)$.

Here σ ² is called the variance proxy, which is not necessarily equal to the variance of π (although it can be shown that σ ² ≥Var[π]). The name subgaussian refers to the fact that $\exp (\sigma ^2 t^2 / 2)$ is the moment-generating function of $\mathscr {N}(0,\sigma ^2)$.

The following are some examples of (laws of) subgaussian random variables. Clearly, $\mathscr {N}(0,\sigma ^2)$ is σ ²-subgaussian. By Hoeffding’s lemma, any distribution supported on an interval [a, b] is (b − a)²∕4-subgaussian. In particular, the Rademacher distribution Unif({±1}) is 1-subgaussian. Note also that the sum of n independent σ ²-subgaussian random variables is σ ²n-subgaussian.

Subgaussian random variables admit the following bound on their absolute moments; see Lemmas 1.3 and 1.4 of [92].

Proposition 12

If π is σ ² -subgaussian then

$$\displaystyle \begin{aligned}\mathbb{E}[|\pi|{}^k] \le (2\sigma^2)^{k/2} k \varGamma(k/2)\end{aligned}$$

for every integer k ≥ 1.

Here Γ(⋅) denotes the gamma function which, recall, is defined for all positive real numbers and satisfies Γ(k) = (k − 1)! when k is a positive integer. We will need the following property of the gamma function.

Proposition 13

For all x > 0 and a > 0,

$$\displaystyle \begin{aligned}\frac{\varGamma(x+a)}{\varGamma(x)} \le (x+a)^a.\end{aligned}$$

Proof

This follows from two standard properties of the gamma function. The first is that (similarly to the factorial) Γ(x + 1)∕Γ(x) = x for all x > 0. The second is Gautschi’s inequality, which states that Γ(x + s)∕Γ(x) < (x + s)^s for all x > 0 and s ∈ (0, 1).

In the context of the spiked Wigner model (Sect. 3.2), we now prove that subgaussian spike priors admit a local Chernoff bound (Definition 14).

Proposition 14

Suppose π is σ ²-subgaussian (for some constant σ ² > 0) with $\mathbb {E}[\pi ] = 0$and $\mathbb {E}[\pi ^2] = 1$. Let $(\mathscr {X}_n)$be the spike prior that draws each entry ofxi.i.d. from π (where π does not depend on n). Then $(\mathscr {X}_n)$admits a local Chernoff bound.

Proof

Since π is subgaussian, π ² is subexponential, which implies $\mathbb {E}[\exp (t \pi ^2)] < \infty$ for all |t|≤ s for some s > 0 (see e.g., Lemma 1.12 of [92]).

Let π, π ^′ be independent copies of π, and set Π = ππ ^′. The moment-generating function of Π is

$$\displaystyle \begin{aligned} M(t) = \mathbb{E}[\exp(t \varPi)] = \mathbb{E}_\pi \mathbb{E}_{\pi'}[\exp(t \pi \pi')] \le \mathbb{E}_\pi\left[\exp\left(\sigma^2 t^2 \pi^2/2\right)\right] < \infty \end{aligned}$$

provided $\frac {1}{2}\sigma ^2 t^2 < s$, i.e. $|t| < \sqrt {2s/\sigma ^2}$. Thus M(t) exists in an open interval containing t = 0, which implies $M'(0) = \mathbb {E}[\varPi ] = 0$ and $M''(0) = \mathbb {E}[\varPi ^2] = 1$ (this is the defining property of the moment-generating function: its derivatives at zero are the moments).

Let η > 0 and . Since M(0) = 1, M′(0) = 0, M″(0) = 1 and, as one may check, $f(0) = 1, f'(0) = 0, f''(0) = \frac {1}{1-\eta } > 1$, there exists δ > 0 such that, for all t ∈ [−δ, δ], M(t) exists and M(t) ≤ f(t).

We then apply the standard Chernoff bound argument to $\langle \boldsymbol x^1,\boldsymbol x^2 \rangle = \sum _{i=1}^n \varPi _i$ where Π ₁, …, Π _n are i.i.d. copies of Π. For any α > 0,

$$\displaystyle \begin{aligned} \Pr\left\{\langle \boldsymbol x^1,\boldsymbol x^2 \rangle \ge t\right\} &= \Pr\left\{\exp(\alpha \langle \boldsymbol x^1,\boldsymbol x^2 \rangle) \ge \exp(\alpha t)\right\}\\ &\le \exp(-\alpha t) \mathbb{E}[\exp(\alpha \langle \boldsymbol x^1,\boldsymbol x^2 \rangle)] \end{aligned} $$

(byMarkov’s inequality)

$$\displaystyle \begin{aligned} &= \exp(-\alpha t) \mathbb{E}\left[\exp\left(\alpha \sum_{i=1}^n \varPi_i\right)\right]\\ &= \exp(-\alpha t) [M(\alpha)]^n\\ &\le \exp(-\alpha t) [f(\alpha)]^n \\ &= \exp(-\alpha t) \exp\left(\frac{\alpha^2 n}{2(1-\eta)}\right). \end{aligned} $$

(provided α ≤ δ)

Taking α = (1 − η)t∕n,

$$\displaystyle \begin{aligned}\Pr\left\{\langle \boldsymbol x^1,\boldsymbol x^2 \rangle \ge t\right\} \le \exp\left(-\frac{1}{n}(1-\eta)t^2 + \frac{1}{2n}(1-\eta)t^2\right) = \exp\left(-\frac{1}{2n}(1-\eta)t^2\right)\end{aligned}$$

as desired. This holds provided α ≤ δ, i.e. t ≤ δn∕(1 − η). A symmetric argument with − Π in place of Π holds for the other tail, $\Pr \left \{\langle \boldsymbol x^1,\boldsymbol x^2 \rangle \le -t\right \}$.

Hypercontractivity

The following hypercontractivity result states that the moments of low-degree polynomials of i.i.d. random variables must behave somewhat reasonably. The Rademacher version is the Bonami lemma from [88], and the Gaussian version appears in [53] (see Theorem 5.10 and Remark 5.11 of [53]). We refer the reader to [88] for a general discussion of hypercontractivity.

Proposition 15 (Bonami Lemma)

Letx = (x ₁, …, x _n) have either i.i.d. $\mathscr {N}(0,1)$or i.i.d. Rademacher (uniform ± 1) entries, and let $f: \mathbb {R}^n \to \mathbb {R}$be a polynomial of degree k. Then

$$\displaystyle \begin{aligned}\mathbb{E}[f(x)^4] \le 3^{2k} \,\mathbb{E}[f(x)^2]^2.\end{aligned}$$

We will combine this with the following standard second moment method.

Proposition 16 (Paley-Zygmund Inequality)

If Z ≥ 0 is a random variable with finite variance, and 0 ≤ θ ≤ 1, then

$$\displaystyle \begin{aligned}\mathrm{Pr}\left\{Z > \theta\, \mathbb{E}[Z]\right\} \ge (1-\theta)^2 \frac{\mathbb{E}[Z]^2}{\mathbb{E}[Z^2]}.\end{aligned}$$

By combining Propositions 16 and 15, we immediately have the following.

Corollary 2

Letx = (x ₁, …, x _n) have either i.i.d. $\mathscr {N}(0,1)$or i.i.d. Rademacher (uniform ± 1) entries, and let $f: \mathbb {R}^n \to \mathbb {R}$be a polynomial of degree k. Then, for 0 ≤ θ ≤ 1,

$$\displaystyle \begin{aligned}\Pr\left\{f(x)^2 > \theta\, \mathbb{E}[f(x)^2]\right\} \ge (1-\theta)^2 \frac{\mathbb{E}[f(x)^2]^2}{\mathbb{E}[f(x)^4]} \ge \frac{(1-\theta)^2}{3^{2k}}.\end{aligned}$$

Remark 5

One rough interpretation of Corollary 2 is that if f is degree k, then $\mathbb {E}[f(x)^2]$ cannot be dominated by an event of probability smaller than roughly 3^−2k.