Sparse Spectral Methods for Solving High-Dimensional and Multiscale Elliptic PDEs

Abstract

In his monograph Chebyshev and Fourier Spectral Methods, John Boyd claimed that, regarding Fourier spectral methods for solving differential equations, “[t]he virtues of the Fast Fourier Transform will continue to improve as the relentless march to larger and larger [bandwidths] continues” [Boyd in Chebyshev and Fourier spectral methods, second rev ed. Dover Publications, Mineola, NY, 2001, pg. 194]. This paper attempts to further the virtue of the Fast Fourier Transform (FFT) as not only bandwidth is pushed to its limits, but also the dimension of the problem. Instead of using the traditional FFT however, we make a key substitution: a high-dimensional, sparse Fourier transform paired with randomized rank-1 lattice methods. The resulting sparse spectral method rapidly and automatically determines a set of Fourier basis functions whose span is guaranteed to contain an accurate approximation of the solution of a given elliptic PDE. This much smaller, near-optimal Fourier basis is then used to efficiently solve the given PDE in a runtime which only depends on the PDE’s data compressibility and ellipticity properties, while breaking the curse of dimensionality and relieving linear dependence on any multiscale structure in the original problem. Theoretical performance of the method is established herein with convergence analysis in the Sobolev norm for a general class of non-constant diffusion equations, as well as pointers to technical extensions of the convergence analysis to more general advection–diffusion–reaction equations. Numerical experiments demonstrate good empirical performance on several multiscale and high-dimensional example problems, further showcasing the promise of the proposed methods in practice.

Appendices

Appendix A: Stamp Set Cardinality Bound

We begin by proving the following combinatorial upper bound for the cardinality of a stamp set.

Lemma 8

Suppose that $\textbf{ 0 } \in {{\,\textrm{supp}\,}}(\hat{ a })$, ${{\,\textrm{supp}\,}}(\hat{ a }) = -{{\,\textrm{supp}\,}}(\hat{ a })$, and $\vert {{\,\textrm{supp}\,}}(\hat{ a }) \vert = s$. Then

$$\begin{aligned} \vert \mathcal {S}^N[\hat{ a }]({{\,\textrm{supp}\,}}(\hat{ f })) \vert \le \vert {{\,\textrm{supp}\,}}(\hat{ f }) \vert \sum _{ n = 0 }^N \sum _{ t = 0 }^{ \min (n, (s-1)/2) } 2^{ t } \left( {\begin{array}{c}(s-1)/2\\ t\end{array}}\right) \left( {\begin{array}{c}n - 1\\ t - 1\end{array}}\right) . \end{aligned}$$

(A1)

Proof

We begin by separating $\mathcal {S}^N$ into the disjoint pieces

$$\begin{aligned} \mathcal {S}^N = \bigsqcup _{ n = 0 }^{ N } \left( \mathcal {S}^n \setminus \left( \bigcup _{ i = 0 }^{ n - 1 } \mathcal {S}^i \right) \right) \end{aligned}$$

and computing the cardinality of each of these sets (where we take $S^{ -1 } = \emptyset$). If $\textbf{ k } \in \mathcal {S}^n {\setminus } \left( \cup _{ i = 0 }^{ n - 1 } \mathcal {S}^i \right)$, then we are able to write $\textbf{ k }$ as

$$\begin{aligned} \textbf{ k } = \textbf{ k }_f + \sum _{ m = 1 }^n \textbf{ k }_a^m \end{aligned}$$

(A2)

where $\textbf{ k }_f \in {{\,\textrm{supp}\,}}(\hat{ f })$ and $\textbf{ k }_a^m \in {{\,\textrm{supp}\,}}(\hat{ a }) {\setminus } \{ 0 \}$ for all $m = 1, \ldots , n$. Additionally, since $\textbf{ k }$ is not in any earlier stamping sets, this is the smallest n for which this is possible. In particular, it is not possible for any two frequencies in the sum to be negatives of each other resulting in pairs of cancelled terms.

With this summation in mind, arbitrarily split ${{\,\textrm{supp}\,}}( \hat{ a } ) {\setminus } \{ \textbf{ 0 } \}$ into $A \sqcup -A$ (i.e., place all frequencies which do not negate each other into A and their negatives in $-A$). By collecting like frequencies that occur as a $\textbf{ k }_a^m$ term in (A2), we can rewrite this sum as

$$\begin{aligned} \textbf{ k } = \textbf{ k }_f + \sum _{ \textbf{ k }_a \in A } s(\textbf{ k }, \textbf{ k }_a) m(\textbf{ k }, \textbf{ k }_a) \textbf{ k }_a, \end{aligned}$$

(A3)

where the sign function $s(\textbf{ k }, \textbf{ k }_a)$ is given by

$$\begin{aligned} s(\textbf{ k }, \textbf{ k }_a):= {\left\{ \begin{array}{ll} 1 &{}\text {if } \textbf{ k }_a \text { is a term in the summation }(A2)\\ -1 &{}\text {if } -\textbf{ k }_a \text { is a term in the summation }(A2)\\ 0 &{}\text {otherwise} \end{array}\right. } \end{aligned}$$

and the multiplicity function $m(\textbf{ k }, \textbf{ k }_a)$ is defined as the number of times that $\textbf{ k }_a$ or $-\textbf{ k }_a$ appears as a $\textbf{ k }_a^m$ term in (A2). Letting $\textbf{ s }(\textbf{ k }):= (s(\textbf{ k }, \textbf{ k }_a))_{ k_a \in A }$ and $\textbf{ m }(\textbf{ k }):= (m(\textbf{ k }, \textbf{ k }_a))_{ k_a \in A }$, we can then identify any $\textbf{ k } \in \mathcal {S}^n {\setminus } \left( \cup _{ i = 0 }^{ n - 1 } \mathcal {S}^i \right)$ with the tuple

$$\begin{aligned} (\textbf{ k }_f, \textbf{ s }(\textbf{ k }), \textbf{ m }(\textbf{ k })) \in {{\,\textrm{supp}\,}}(\textbf{ f }) \times \{-1, 0, 1\}^{ A } \times \{0, \ldots , n\}^A. \end{aligned}$$

Upper bounding the number of these tuples that can correspond to a value of $\textbf{ k } \in \mathcal {S}^n {\setminus } \left( \cup _{ i = 0 }^{ n - 1 } \mathcal {S}^i \right)$ will then upper bound the cardinality of this set.

Since any $\textbf{ k }_f \in {{\,\textrm{supp}\,}}(\hat{ f })$ can result in a valid $\textbf{ k }$ value, we will focus on the pairs of sign and multiplicity vectors. Define by $T_n \subseteq \{-1, 0, 1\}^A \times \{0, \ldots , n\}^A$ the set of valid sign and multiplicity pairs that can correspond to a $\textbf{ k } \in \mathcal {S}^n {\setminus } \left( \cup _{ i = 0 }^{ n - 1 } \mathcal {S}^i \right)$. In particular, for $(\textbf{ s }, \textbf{ m }) \in T_n$, $\vert \vert \textbf{ m } \vert \vert _1 = n$ and ${{\,\textrm{supp}\,}}(\textbf{ s } ) = {{\,\textrm{supp}\,}}( \textbf{ m } )$. Thus, we can write

$$\begin{aligned} \begin{aligned} T_n \subseteq \bigsqcup _{ t = 0 }^{ \min (n, |A|) }&\left\{ (\textbf{ s }, \textbf{ m }) \in \{-1, 0, 1\}^A \times \{0, \ldots , n\}^A \right. \\ {}&\qquad \left. \mid \vert \vert \textbf{ m } \vert \vert _1 = n \text { and } |{{\,\textrm{supp}\,}}(\textbf{ s })| = |{{\,\textrm{supp}\,}}(\textbf{ m })| = t \right\} . \end{aligned} \end{aligned}$$

This inner set then corresponds to the t-partitions of the integer n spread over the |A| entries of $\textbf{ m }$ where each non-zero term is assigned a sign $-1$ or 1. The cardinality is therefore $2^t \left( {\begin{array}{c} |A| \\ t \end{array}}\right) \left( {\begin{array}{c}n - 1\\ t - 1\end{array}}\right)$: the first factor is from the possible sign options, the second is the number of ways to choose the entries of $\textbf{ m }$ which are nonzero, and the last is the number of t-partitions of n which will fill the nonzero entries of $\textbf{ m }$. Noting that $|A| = \frac{s - 1}{2}$, our final cardinality estimate is

$$\begin{aligned} \vert \mathcal {S}^N\vert&= \sum _{ n = 0 }^{ N } \vert \mathcal {S}^n \setminus \left( \bigcup _{ i = 0 }^{ n - 1 } \mathcal {S}^i \right) \vert \\&\le \sum _{ n = 0 }^N \vert {{\,\textrm{supp}\,}}(\hat{ f })\vert |T_n|\\&\le \vert {{\,\textrm{supp}\,}}(\hat{ f }) \vert \sum _{ n = 0 }^N \sum _{ t = 0 }^{ \min (n, (s-1)/2) } 2^{ t } \left( {\begin{array}{c}(s-1)/2\\ t\end{array}}\right) \left( {\begin{array}{c}n - 1\\ t - 1\end{array}}\right) \end{aligned}$$

as desired. $\square$

Though this upper bound is much tighter than the one given in the main text, it is harder to parse. As such, we simplify it to the bound presented in Lemma 2, restated here for convenience.

Lemma 2

Suppose that $\textbf{ 0 } \in {{\,\textrm{supp}\,}}(\hat{ a })$, ${{\,\textrm{supp}\,}}(\hat{ a }) = -{{\,\textrm{supp}\,}}(\hat{ a })$, and $\vert {{\,\textrm{supp}\,}}(\hat{ a }) \vert = s$. Then

$$\begin{aligned} \vert \mathcal {S}^N[\hat{ a }]({{\,\textrm{supp}\,}}(\hat{ f }))\vert \le 6 \vert {{\,\textrm{supp}\,}}(\hat{ f }) \vert \left( \frac{\max (s, 2N)}{\sqrt{2}} \right) ^{ \min (s, 2N) }. \end{aligned}$$

Proof

Let $r = (s - 1) / 2$. We consider two cases:

Case 1::

($s \ge 2N \implies r \ge N$) We estimate the innermost sum of (A1). Since $r \ge N \ge n$, $\min (n, (s - 1) / 2) = n$. By upper bounding the binomial coefficients with powers of r, we obtain

$$\begin{aligned} \sum _{ t = 0 }^{ n } 2^t \left( {\begin{array}{c}r\\ t\end{array}}\right) \left( {\begin{array}{c}n-1\\ t-1\end{array}}\right)&\le \sum _{ t = 0 }^n 2^t (r^t)^2 \\&\le 2(2r^2)^{ n } \end{aligned}$$

where the second estimate follows from approximating the geometric sum. Again, bounding the next geometric sum by double the largest term, we have

$$\begin{aligned} \vert \mathcal {S}^N \vert \le \vert {{\,\textrm{supp}\,}}(\hat{ f }) \vert \sum _{ n = 0 }^N 2(2r^2)^n \le \vert {{\,\textrm{supp}\,}}(\hat{ f }) \vert 4(2r^2)^N \le 4 \vert {{\,\textrm{supp}\,}}(\hat{ f }) \vert \left( \frac{ s }{\sqrt{ 2 }}\right) ^{ 2N }. \end{aligned}$$

Case 2::

($s< 2N \implies r < N$) Bounding the innermost sum of (A1) proceeds much the same way as Case 1, but we must first split the outermost sum into the first $r + 1$ terms and last $N - r$ terms. Working with the first terms, we find

$$\begin{aligned} \sum _{ n = 0 }^r \sum _{ t = 0 }^{ n } 2^t \left( {\begin{array}{c}r\\ t\end{array}}\right) \left( {\begin{array}{c}n-1\\ t-1\end{array}}\right) \le 4(2r^2)^r \end{aligned}$$

using the argument in Case 1. Now, we bound

$$\begin{aligned} \sum _{ n = r + 1 }^N \sum _{ t = 0 }^{ r } 2^t \left( {\begin{array}{c}r\\ t\end{array}}\right) \left( {\begin{array}{c}n-1\\ t-1\end{array}}\right)&\le \sum _{ n = r + 1 }^N 2(2(n - 1)^2)^r \\&\le 2N(2(N - 1)^2)^r \\&\le \sqrt{ 2 } (\sqrt{ 2 }N)^{ 2r + 1 }. \end{aligned}$$

Thus,

$$\begin{aligned} \vert \mathcal {S}^N \vert \le \vert {{\,\textrm{supp}\,}}(\hat{ f }) \vert \left[ 4(2r^2)^r + \sqrt{ 2 } (\sqrt{ 2 }N)^{ 2r + 1 } \right] \le (4 + \sqrt{ 2 })\vert {{\,\textrm{supp}\,}}(\hat{ f }) \vert \left( \frac{2N}{\sqrt{ 2 }} \right) ^s. \end{aligned}$$

Combining the two cases gives the desired upper bound.

$\square$

Appendix B: Proof of SFT Recovery Guarantees

We restate the theorem for convenience.

Theorem 2

([26], Corollary 2) Let $\mathcal {I} \subseteq \mathbb {Z}^d$ be a frequency set of interest with expansion defined as $K:= \max _{ j \in \{1, \ldots , d\} } (\max _{ \textbf{ k } \in \mathcal {I} } k_j - \min _{ \textbf{ l } \in \mathcal {I}} l_j ) + 1$ (i.e., the sidelength of the smallest hypercube containing $\mathcal {I}$), and $\Lambda ( \textbf{ z }, M)$ be a reconstructing rank-1 lattice for $\mathcal {I}$.

There exists a fast, randomized SFT which, given $\Lambda (\textbf{ z }, M)$, sampling access to $g \in L^2$, and a failure probability $\sigma \in (0, 1]$, will produce a 2s-sparse approximation $\hat{ \textbf{ g } }^s$ of ${\hat{g}}$ and function $g^s:= \sum _{ \textbf{ k } \in {{\,\textrm{supp}\,}}( \hat{ \textbf{ g }}^s) } {\hat{g}}_{ \textbf{ k } }^s e_\textbf{ k }$ approximating g satisfying

$$\begin{aligned} \vert \vert g - g^s \vert \vert _{ L^2 } \le \vert \vert {\hat{g}} - \hat{ \textbf{ g } }^s \vert \vert _{ \ell ^2 }&\le (25 + 3K) \left[ \frac{\vert \vert \hat{ g }|_{ \mathcal {I} } - (\hat{ g }|_{ \mathcal {I} })_s^\textrm{opt}\vert \vert _1}{\sqrt{ s }} + \sqrt{ s } \vert \vert \hat{ g } - \hat{ g }|_{ \mathcal {I} } \vert \vert _1 \right] \end{aligned}$$

with probability exceeding $1 - \sigma$. If $g \in L^\infty$, then we additionally have

$$\begin{aligned} \vert \vert g - g^s \vert \vert _{ L^\infty } \le \vert \vert {\hat{g}} - \hat{ \textbf{ g } }^s \vert \vert _{ \ell ^1 } \le (33 + 4K) \left[ \vert \vert \hat{ g } |_{ \mathcal {I} } - (\hat{ g }|_{ \mathcal {I} })_s^\textrm{opt} \vert \vert _1 + \vert \vert \hat{ g } - \hat{ g }|_{ \mathcal {I} } \vert \vert _1 \right] \end{aligned}$$

with the same probability estimate. The total number of samples of g and computational complexity of the algorithm can be bounded above by

$$\begin{aligned} \mathcal {O} \left( d s \log ^3( d K M) \log \left( \frac{d K M }{\sigma } \right) \right) . \end{aligned}$$

Proof

The $L^2$ upper bound is mostly the same as the original result. We are not considering noisy measurements here which removes the $\sqrt{ s } e_\infty$ term from that result (though, this could be added back in if desired). Additionally, we have upper bounded $\vert \vert \hat{ g } - \hat{ g }|_{ \mathcal {I} } \vert \vert _2$ by $\sqrt{s} \vert \vert \hat{ g } - \hat{ g }|_{ \mathcal {I} } \vert \vert _1$ adding one to the constant.

The $L^\infty$ / $\ell ^1$ bound was not given in the original paper, but can be proven using the same techniques. In particular, replacing the $\ell ^2$ norm by the $\ell ^1$ norm in [26, Lemma 4] has the effect of replacing all $\ell ^2$ norms with $\ell ^1$ norms and replacing $\sqrt{ 2s }$ by 2s. This small change cascades through the proof of Property 3 in [26, Theorem 2] (again, with $\ell ^2$ norms replaced by $\ell ^1$ norms) to produce the univariate $\ell ^1$ upper bound (in the language of the original paper)

$$\begin{aligned} \vert \vert \hat{ \textbf{ a } } - \textbf{ v } \vert \vert _1 \le \vert \vert \hat{ \textbf{ a } } - \hat{ \textbf{ a } }_{ 2s }^\textrm{opt} \vert \vert _1 + (16 + 6 \sqrt{ 2 }) \left( \vert \vert \hat{ \textbf{ a } } - \hat{ \textbf{ a } } _{ s }^\textrm{opt} \vert \vert _1 + s (\vert \vert \hat{ a } - \hat{ \textbf{ a } } \vert \vert _1 + \vert \vert \mu \vert \vert _\infty )\right) =: \eta _1. \end{aligned}$$

A similar logic applies to revising the proof of [26, Lemma 1]. Equation (4) with all $\ell ^2$ norms replaced by $\ell ^1$ norms is derived the same way, and the first term is upper bounded by the maximal entry of the vector multiplied by the number of elements without the square root. The remainder of the proof carries through without change which leads to a final error estimate of

$$\begin{aligned} \vert \vert \textbf{ b } - c \vert \vert _{ \ell ^2 } \le (\beta + \eta _\infty ) \max (s - \vert \mathcal {S}_\beta \vert , 0 ) + \eta _1 + \vert \vert c|_{ \mathcal {I} } - c|_{ \mathcal {S}_\beta } \vert \vert _{ 1 } + \vert \vert c - c|_{ \mathcal {I} } \vert \vert _1. \end{aligned}$$

Finally, the proof of [26, Corollary 2] follows using the same logic as the original substituting these revised upper bounds. $\square$