Smolyak’s Algorithm: A Powerful Black Box for the Acceleration of Scientific Computations

Abstract

We provide a general discussion of Smolyak’s algorithm for the acceleration of scientific computations. The algorithm first appeared in Smolyak’s work on multidimensional integration and interpolation. Since then, it has been generalized in multiple directions and has been associated with the keywords: sparse grids, hyperbolic cross approximation, combination technique, and multilevel methods. Variants of Smolyak’s algorithm have been employed in the computation of high-dimensional integrals in finance, chemistry, and physics, in the numerical solution of partial and stochastic differential equations, and in uncertainty quantification. Motivated by this broad and ever-increasing range of applications, we describe a general framework that summarizes fundamental results and assumptions in a concise application-independent manner.

Appendix

Lemma 1

Let γ _j  > 0, β _j  > 0, and t _j  > 0 for j ∈{1, …, n}. Then

$$\displaystyle \begin{aligned} \sum_{(\boldsymbol{{\beta}}+\boldsymbol{\gamma})\cdot\boldsymbol{{k}}\leq L}\exp(\boldsymbol{\gamma}\cdot\boldsymbol{{k}})(\boldsymbol{{k}}+\mathbf{1})^{\boldsymbol{t}}\leq C(\boldsymbol{\gamma},\boldsymbol{t},n)\exp(\mu L)(L+1)^{n^*-1+t^*}, \end{aligned}$$

where $\rho :=\max _{j=1}^{n}\gamma _j/{\beta }_j$ , $\mu :=\frac {\rho }{1+\rho }$ , J := {j ∈{1, …, n} : γ _j ∕β _j  = ρ}, n ^∗ := |J|, t ^∗ :=∑_{j ∈ J} t _j , and $(\boldsymbol {{k}}+\mathbf {1})^{\boldsymbol {t}}:=\prod _{j=1}^{n}({k}_j+1)^{t_j}$.

Proof

First, we assume without loss of generality that the dimensions are ordered according to whether they belong to J or J ^c := {1, …, n}∖ J. To avoid cluttered notation we then separate dimensions by plus or minus signs in the subscripts; for example, we write $\boldsymbol {t}=(\boldsymbol {t}_J,\boldsymbol {t}_{J^{c}})=:(\boldsymbol {t}_+,\boldsymbol {t}_-)$.

Next, we may replace the sum by an integral over {(β + γ) ⋅x ≤ L}. Indeed, by monotonicity we may do so if we replace L by L + |β + γ|₁, but looking at the final result we observe that a shift of L only affects the constant C(γ, t, n).

Finally, using a change of variables y _j  := (β _j  + γ _j )x _j and the shorthand μ := γ∕(β + γ) (with componentwise division) we obtain

$$\displaystyle \begin{aligned} \begin{array}{rcl} &\displaystyle &\displaystyle \int_{(\boldsymbol{{\beta}}+\boldsymbol{\gamma})\cdot \boldsymbol{x}\leq L}\exp(\boldsymbol{\gamma}\cdot \boldsymbol{x})(\boldsymbol{x}+\mathbf{1})^{\boldsymbol{t}}\;d\boldsymbol{x}\\&\displaystyle &\displaystyle \quad \leq C \int_{|\boldsymbol{y}|{}_1\leq L}\exp(\boldsymbol{\mu}\cdot \boldsymbol{y})(\boldsymbol{y}+\mathbf{1})^{\boldsymbol{t}}\;d\boldsymbol{y}\\ &\displaystyle &\displaystyle \quad =C\int_{|\boldsymbol{y}_{+}|{}_1\leq L}\exp(\boldsymbol{\mu}_{+}\cdot \boldsymbol{y}_{+})(\boldsymbol{y}_{+}+\mathbf{1})^{\boldsymbol{t}_{+}}\\&\displaystyle &\displaystyle \qquad \times\int_{|\boldsymbol{y}_{-}|{}_{1}\leq L-|\boldsymbol{y}_{+}|{}_1}\exp(\boldsymbol{\mu}_{-}\cdot\boldsymbol{y}_{-})(\boldsymbol{y}_{-}+\mathbf{1})^{\boldsymbol{t}_{-}}\;d\boldsymbol{y}_{-}\;d\boldsymbol{y}_{+}\\ &\displaystyle &\displaystyle \quad \leq C\int_{|\boldsymbol{y}_{+}|{}_1\leq L}\exp(\mu|\boldsymbol{y}_{+}|{}_1)(\boldsymbol{y}_{+}+\mathbf{1})^{\boldsymbol{t}_{+}}\\&\displaystyle &\displaystyle \qquad \times\int_{|\boldsymbol{y}_{-}|{}_{1}\leq L-|\boldsymbol{y}_{+}|{}_1}\exp(\mu_{-}|\boldsymbol{y}_-|{}_{1})(\boldsymbol{y}_{-}+\mathbf{1})^{\boldsymbol{t}_{-}}\;d\boldsymbol{y}_{-}\;d\boldsymbol{y}_{+}=(\star), \end{array} \end{aligned} $$

where the last equality holds by definition of $\mu =\max \{\boldsymbol {\mu }_{+}\}$ and $\mu _{-}:=\max \{\boldsymbol {\mu }_{-}\}$. We use the letter C here and in the following to denote quantities that depend only on γ, t and n but may change value from line to line. Using $(\boldsymbol {y}_{ +}+\mathbf {1})^{\boldsymbol {t}_{+}}\leq (|\boldsymbol {y}_{+}|{ }_1+1)^{|\boldsymbol {t}_{+}|{ }_1}$ and $(\boldsymbol {y}_{-}+\mathbf {1})^{\boldsymbol {t}_{-}}\leq (|\boldsymbol {y}_{-}|{ }_1+1)^{|\boldsymbol {t}_{-}|{ }_1}$ and the linear change of variables y↦(|y|₁, y ₂, …, y _n ) in both integrals, we obtain

where we used supremum bounds for both integrals for the third inequality, the change of variables w := L − u for the penultimate equality, and the fact that μ > μ ₋ for the last inequality.

Lemma 2

Let γ _j  > 0, β _j  > 0, and s _j  > 0 for j ∈{1, …, n}. Then

$$\displaystyle \begin{aligned} \sum_{(\boldsymbol{{\beta}}+\boldsymbol{\gamma})\cdot\boldsymbol{{k}}>L}\exp(-\boldsymbol{{\beta}}\cdot\boldsymbol{{k}})(\boldsymbol{{k}}+\mathbf{1})^{\boldsymbol{s}} \leq C(\boldsymbol{{\beta}},\boldsymbol{s},n)\exp(-\nu L)(L+1)^{n^{*}-1+s^*}, \end{aligned}$$

where $\rho :=\max _{j=1}^{n}\gamma _j/{\beta }_j$ , $\nu :=\frac {1}{1+\rho }$ , J := {j ∈{1, …, n} : γ _j ∕β _j  = ρ}, n ^∗ := |J|, s ^∗ :=∑_{j ∈ J} t _j , and $(\boldsymbol {{k}}+\mathbf {1})^{\boldsymbol {s}}:=\prod _{j=1}^{n}({k}_j+1)^{s_j}$.

Proof

First, we assume without loss of generality that the dimensions are ordered according to whether they belong to J or J ^c. To avoid cluttered notation we then separate dimensions by plus or minus signs in the subscripts; for example, we write $\boldsymbol {s}=(\boldsymbol {s}_J,\boldsymbol {s}_{J^{c}})=:(\boldsymbol {s}_+,\boldsymbol {s}_-)$.

Next, we may replace the sum by an integral over {(β + γ) ⋅x > L}. Indeed, by monotonicity we may do so if we replace L by L −|β + γ|₁, but looking at the final result we observe that a shift of L only affects the constant C(β, s, n).

Finally, using a change of variables y _j  := (β _j  + γ _j )x _j and the shorthand ν := β∕(β + γ) (with componentwise division) we obtain

$$\displaystyle \begin{aligned} \begin{aligned} &\int_{(\boldsymbol{{\beta}}+\boldsymbol{\gamma})\cdot \boldsymbol{x}>L}\exp(-\boldsymbol{{\beta}}\cdot \boldsymbol{x})(\boldsymbol{x}+\mathbf{1})^{\boldsymbol{s}}\;d\boldsymbol{x}\leq C\int_{|\boldsymbol{y}|{}_1>L}\exp(-\boldsymbol{\nu}\cdot \boldsymbol{y})(\boldsymbol{y}+\mathbf{1})^{\boldsymbol{s}}\;d\boldsymbol{y}\\ &=C\int_{|\boldsymbol{y}_{+}|{}_1> L}\exp(-\boldsymbol{\nu}_{+}\cdot \boldsymbol{y}_{+})(\boldsymbol{y}_{+}+\mathbf{1})^{\boldsymbol{s}_{+}}\int_{|\boldsymbol{y}_{-}|{}_{1}>(L-|\boldsymbol{y}_{+}|{}_1)^{+}}\exp(-\boldsymbol{\nu}_{-}\cdot\boldsymbol{y}_{-})(\boldsymbol{y}_{-}+\mathbf{1})^{\boldsymbol{s}_{-}}d\boldsymbol{y}_{-}d\boldsymbol{y}_{+}\\ &\leq C\int_{|\boldsymbol{y}_{+}|{}_1> L}\exp(-\nu|\boldsymbol{y}_{+}|{}_1)(\boldsymbol{y}_{+}+\mathbf{1})^{\boldsymbol{s}_{+}}\int_{|\boldsymbol{y}_{-}|{}_{1}>(L-|\boldsymbol{y}_{+}|{}_1)^{+}}\exp(-\nu_{-}|\boldsymbol{y}_-|{}_{1})(\boldsymbol{y}_{-}+\mathbf{1})^{\boldsymbol{s}_{-}}d\boldsymbol{y}_{-}d\boldsymbol{y}_{+}\\ &=:(\star), \end{aligned} \end{aligned}$$

where the last equality holds by definition of $\nu =\max \{\boldsymbol {\nu }_{+}\}$ and $\nu _{-}:=\max \{\boldsymbol {\nu }_{-}\}$. We use the letter C here and in the following to denote quantities that depend only on β, s and n but may change value from line to line. Using $(\boldsymbol {y}_{ +}+\mathbf {1})^{\boldsymbol {s}_{+}}\leq (|\boldsymbol {y}_{+}|{ }_1+1)^{|\boldsymbol {s}_{+}|{ }_1}$ and $(\boldsymbol {y}_{-}+\mathbf {1})^{\boldsymbol {s}_{-}}\leq (|\boldsymbol {y}_{-}|{ }_1+1)^{|\boldsymbol {s}_{-}|{ }_1}$ and the linear change of variables y↦(|y|₁, y ₂, …, y _n ) in both integrals, we obtain

To bound (⋆⋆), we estimate the inner integral using the inequality $\int _{a}^{\infty }\exp (-b v)(v+1)^{c}\,dv\leq C\exp (-b a)(a+1)^c$ [21, (8.11.2)], which is valid for all positive a, b, c:

$$\displaystyle \begin{aligned} \begin{aligned} (\star\star)&\leq C\int_0^{L}\exp(-\nu u)(u+1)^{|\boldsymbol{s}_{+}|{}_{1}+|J|-1}\exp(-\nu_{-}(L-u))(L-u+1)^{|\boldsymbol{s}_{-}|{}_1+|J^{c}|-1}\;du\\ &\leq C(L+1)^{|\boldsymbol{s}_{+}|{}_{1}+|J|-1}\int_0^{L}\exp(-\nu(L-w))\exp(-\nu_{-}w)(w+1)^{|\boldsymbol{s}_{-}|{}_1+|J^{c}|-1}\;dw\\ &=C(L+1)^{|\boldsymbol{s}_{+}|{}_{1}+|J|-1}\exp(-\nu L)\int_0^{L}\exp(-(\nu_{-}-\nu)w)(w+1)^{|\boldsymbol{s}_{-}|{}_1+|J^{c}|-1}\;dw\\ &\leq C(L+1)^{|\boldsymbol{s}_{+}|{}_{1}+|J|-1}\exp(-\nu L), \end{aligned} \end{aligned}$$

where we used a supremum bound and the change of variables w := L − u for the second inequality, and the fact that ν ₋ > ν for the last inequality. Finally, to bound (⋆ ⋆ ⋆), we observe that the inner integral is independent of L, and bound the outer integral in the same way we previously bounded the inner integral. This shows

$$\displaystyle \begin{aligned} (\star\star\star)\leq C\exp(-\nu L)(L+1)^{|\boldsymbol{s}_{+}|{}_{1}+|J|-1}. \end{aligned}$$