Abstract
We study the approximation capacity of some variation spaces corresponding to shallow ReLU\(^k\) neural networks. It is shown that sufficiently smooth functions are contained in these spaces with finite variation norms. For functions with less smoothness, the approximation rates in terms of the variation norm are established. Using these results, we are able to prove the optimal approximation rates in terms of the number of neurons for shallow ReLU\(^k\) neural networks. It is also shown how these results can be used to derive approximation bounds for deep neural networks and convolutional neural networks (CNNs). As applications, we study convergence rates for nonparametric regression using three ReLU neural network models: shallow neural network, over-parameterized neural network, and CNN. In particular, we show that shallow neural networks can achieve the minimax optimal rates for learning Hölder functions, which complements recent results for deep neural networks. It is also proven that over-parameterized (deep or shallow) neural networks can achieve nearly optimal rates for nonparametric regression.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
Neural networks generate very popular function classes used in machine learning algorithms [2, 28]. The fundamental building blocks of neural networks are ridge functions (also called neurons) of the form \(x \in \mathbb {R}^d \mapsto \rho ((x^\intercal ,1)v)\), where \(\rho :\mathbb {R}\rightarrow \mathbb {R}\) is a continuous activation function and \(v\in \mathbb {R}^{d+1}\) is a trainable parameter. It is well-known that a shallow neural network with non-polynomial activation
is universal in the sense that it can approximate any continuous functions on any compact set with desired accuracy when the number of neurons N is sufficiently large [11, 23, 48]. The approximation and statistical properties of neural networks with different architectures have also been widely studied in the literature [6, 52, 64, 67], especially when \(\rho \) is a sigmoidal activation or \(\rho \) is the ReLU\(^k\) function \(\sigma _k(t) = \max \{0,t\}^k\), the k-power of the rectified linear unit (ReLU) with \(k\in \mathbb {N}_0:=\mathbb {N}\cup \{0\}\).
The main focus of this paper is rates of approximation by neural networks. For classical smooth function classes, such as Hölder functions, Mhaskar [39] (see also [48, Theorem 6.8]) presented approximation rates for shallow neural networks, when the activation function \(\rho \in C^\infty (\Omega )\) is not a polynomial on some open interval \(\Omega \) (ReLU\(^k\) does not satisfy this condition). It is known that the rates obtained by Mhaskar are optimal if the network weights are required to be continuous functions of the target function. Recently, optimal rates of approximation have also been established for deep ReLU neural networks [31, 54, 64, 65], even without the continuity requirement on the network weights. All these approximation rates are obtained by using the idea that one can construct neural networks to approximate polynomials efficiently. There is another line of works [4, 25, 34, 56, 57] studying the approximation rates for functions of certain integral forms (such as (1.2)) by using a random sampling argument due to Maurey [49]. In particular, Barron [4] derived dimension independent approximation rates for sigmoid type activations and functions h, whose Fourier transform \(\widehat{h}\) satisfies \(\int _{\mathbb {R}^d} |\omega | |\widehat{h}(\omega )| d\omega <\infty \). This result has been improved and generalized to ReLU activation in recent articles [25, 56, 57].
In this paper, we continue the study of these two lines of approximation theories for neural networks (i.e. the constructive approximation of smooth functions and the random approximation of integral representations). Our main result shows how well integral representations corresponding to ReLU\(^k\) neural networks can approximate smooth functions. By combining this result with the random approximation theory of integral forms, we are able to establish the optimal rates of approximation for shallow ReLU\(^k\) neural networks. Specifically, we consider the following function class defined on the unit ball \(\mathbb {B}^d\) of \(\mathbb {R}^d\) induced by vectors on unit sphere \(\mathbb {S}^d\) of \(\mathbb {R}^{d+1}\) as
which can be regarded as a shallow ReLU\(^k\) neural network with infinite width [3]. The restriction on the total variation \(\Vert \mu \Vert :=|\mu |(\mathbb {S}^d) \le M\) gives a constraint on the size of the weights in the network. We study how well \(\mathcal {F}_{\sigma _k}(M)\) approximates the unit ball of Hölder class \(\mathcal {H}^\alpha \) with smoothness index \(\alpha >0\) as \(M\rightarrow \infty \). Roughly speaking, our main theorem shows that, if \(\alpha > (d+2k+1)/2\), then \(\mathcal {H}^\alpha \subseteq \mathcal {F}_{\sigma _k}(M)\) for some constant M depending on \(k,d,\alpha \), and if \(\alpha < (d+2k+1)/2\), we obtain the approximation bound
where for two quantities X and Y, \(X \lesssim Y\) (or \(Y \gtrsim X\)) denotes the statement that \(X\le CY\) for some constant \(C>0\) (we will also denote \(X \asymp Y\) when \(X \lesssim Y \lesssim X\)). In other words, sufficiently smooth functions are always contained in the shallow neural network space \(\mathcal {F}_{\sigma _k}(M)\). And, for less smooth functions, we can characterize the approximation error by the variation norm. Furthermore, combining our result with the random approximation bounds from [3, 55, 57], we are able to prove that shallow ReLU\(^k\) neural network of the form (1.1) achieves the optimal approximation rate \(\mathcal {O}(N^{-\alpha /d})\) for \(\mathcal {H}^\alpha \) with \(\alpha < (d+2k+1)/2\), which generalizes the result of Mhaskar [39] to ReLU\(^k\) activation.
In addition to shallow neural networks, we can also apply our results to derive approximation bounds for multi-layer neural networks and convolutional neural networks (CNNs) when \(k=1\) (ReLU activation \(\sigma :=\sigma _1\)). These approximation bounds can then be used to study the performances of machine learning algorithms based on neural networks [2]. Here, we illustrate the idea by studying the nonparametric regression problem. The goal of this problem is to learn a function h from a hypothesis space \(\mathcal {H}\) from its noisy samples
where \(X_i\) is sampled from an unknown probability distribution \(\mu \) and \(\eta _i\) is Gaussian noise. One popular algorithm for solving this problem is the empirical least square minimization
where \(\mathcal {F}_n\) is an appropriately chosen function class. For instance, in deep learning, \(\mathcal {F}_n\) is parameterized by deep neural networks and one solves the minimization by (stochastic) gradient descent methods. Assuming that we can compute a minimizer \(f_n^* \in \mathcal {F}_n\), the performance of the algorithm is often measured by the square loss \(\Vert f_n^*-h\Vert _{L^2(\mu )}^2\). A fundamental question in learning theory is to determine the convergence rate of the error \(\Vert f_n^*-h\Vert _{L^2(\mu )}^2 \rightarrow 0\) as the sample size \(n\rightarrow \infty \). The error can be decomposed into two components: approximation error and generalization error (also called estimation error). For neural network models \(\mathcal {F}_n\), our results provide bounds for the approximation errors, while the generalization errors can be bounded by the complexity of the models [40, 53]. We study the cases \(\mathcal {H}= \mathcal {H}^\alpha \) with \(\alpha < (d+3)/2\) or \(\mathcal {H}=\mathcal {F}_{\sigma }(1)\) for three ReLU neural network models: shallow neural network, over-parameterized neural network, and CNN. The models and our contributions are summarized as follows:
-
1.
Shallow ReLU neural network \(\mathcal {F}_{\sigma }(N,M)\), where N is the number of neurons and M is a bound for the variation norm that measures the size of the weights. We prove optimal approximation rates (in terms of N) for this model. It is also shown that this model can achieve the optimal convergence rates for learning \(\mathcal {H}^\alpha \) and \(\mathcal {F}_{\sigma }(1)\), which complements the recent results for deep neural networks [27, 52].
-
2.
Over-parameterized (deep or shallow) ReLU neural network \(\mathcal{N}\mathcal{N}(W,L,M)\) studied in [24], where W, L are the width and depth respectively, and M is a constraint on the weight matrices. For fixed depth L, the generalization error for this model can be controlled by M [24]. When \(\mathcal {H}=\mathcal {H}^\alpha \), we characterize the approximation error by M, and allow the width W to be arbitrary large so that the model can be over-parameterized (the number of parameters is larger than the number of samples). When \(\mathcal {H}=\mathcal {F}_{\sigma }(1)\), we can simply increase the width to reduce the approximation error so that the model can also be over-parameterized. Our result shows that this model can achieve nearly optimal convergence rates for learning \(\mathcal {H}^\alpha \) and \(\mathcal {F}_{\sigma }(1)\). Both the approximation and convergence rates improve the results of [24].
-
3.
Sparse convolutional ReLU neural network \(\mathcal {CNN}(s,L)\) introduced by [67], where L is the depth and \(s\ge 2\) is a fixed integer that controls the filter length. This model is shown to be universal for approximation [67] and universal consistent for regression [30]. We improve the approximation bound in [67] and give new convergence rates of this model for learning \(\mathcal {H}^\alpha \) and \(\mathcal {F}_{\sigma }(1)\).
The approximation rates and convergence rates of nonparametric regression for these models are summarized in Table 1, where we use the notation \(a\vee b:= \max \{a,b\}\).
The rest of the paper is organized as follows. In Sect. 2, we present our approximation results for shallow neural networks. Section 3 gives a proof of our main theorem. In Sect. 4, we apply our approximation results to study these neural network models and derive convergence rates for nonparametric regression using these models. Section 5 concludes this paper with a discussion on possible future directions of research.
2 Approximation Rates for Shallow Neural Networks
Let us begin with some notations for function classes. Let \(\mathbb {B}^d= \{x\in \mathbb {R}^d:\Vert x\Vert _2\le 1\}\) and \(\mathbb {S}^{d-1}=\{x\in \mathbb {R}^d:\Vert x\Vert _2=1\}\) be the unit ball and the unit sphere of \(\mathbb {R}^d\). We are interested in functions of the integral form
where \(\mu \) is a signed Radon measure on \(\mathbb {S}^d\) with finite total variation \(\Vert \mu \Vert :=|\mu |(\mathbb {S}^d)<\infty \) and \(\sigma _k(t):= \max \{t,0\}^k\) with \(k\in \mathbb {N}_0:=\mathbb {N}\cup \{0\}\) is the ReLU\(^k\) function (when \(k=0\), \(\sigma _0(t)\) is the Heaviside function). For simplicity, we will also denote the ReLU function by \(\sigma :=\sigma _1\). The variation norm \(\gamma (f)\) of f is the infimum of \(\Vert \mu \Vert \) over all decompositions of f as (2.1) [3]. By the compactness of \(\mathbb {S}^d\), the infimum can be attained by some signed measure \(\mu \). We denote \(\mathcal {F}_{\sigma _k}(M)\) as the function class that contains all functions f in the form (2.1) whose variation norm \(\gamma (f) \le M\), see (1.2). The class \(\mathcal {F}_{\sigma _k}(M)\) can be thought of as an infinitely wide neural network with a constraint on its weights. The variation spaces corresponding to shallow neural networks have been studied by many researchers. We refer the reader to [8, 44,45,46, 51, 55,56,57,58] for several other definitions and characterizations of these spaces.
We will also need the notion of classical smoothness of functions on Euclidean space. Given a smoothness index \(\alpha >0\), we write \(\alpha =r+\beta \) where \(r\in \mathbb {N}_0\) and \(\beta \in (0,1]\). Let \(C^{r,\beta }(\mathbb {R}^d)\) be the Hölder space with the norm
where \(s=(s_1,\dots ,s_d) \in \mathbb {N}_0^d\) is a multi-index and
Here we use \(\Vert \cdot \Vert _{L^\infty }\) to denote the supremum norm, since we mainly consider continuous functions. We write \(C^{r,\beta }(\mathbb {B}^d)\) for the Banach space of all restrictions to \(\mathbb {B}^d\) of functions in \(C^{r,\beta }(\mathbb {R}^d)\). The norm is given by \(\Vert f\Vert _{C^{r,\beta }(\mathbb {B}^d)} = \inf \{ \Vert g\Vert _{C^{r,\beta }(\mathbb {R}^d)}: g\in C^{r,\beta }(\mathbb {R}^d) \text{ and } g=f \text{ on } \mathbb {B}^d\}\). For convenience, we will denote the unit ball of \(C^{r,\beta }(\mathbb {B}^d)\) by
Note that, for \(\alpha =1\), \(\mathcal {H}^\alpha \) is a class of Lipschitz continuous functions.
Due to the universality of shallow neural networks [48], \(\mathcal {F}_{\sigma _k}(M)\) can approximate any continuous functions on \(\mathbb {B}^d\) if M is sufficiently large. Our main theorem estimates the rate of this approximation for Hölder class.
Theorem 2.1
Let \(k\in \mathbb {N}_0\), \(d\in \mathbb {N}\) and \(\alpha >0\). If \(\alpha > (d+2k+1)/2\) or \(\alpha = (d+2k+1)/2\) is an even integer, then \(\mathcal {H}^\alpha \subseteq \mathcal {F}_{\sigma _k}(M)\) for some constant M depending on \(k,d,\alpha \). Otherwise,
where the implied constants only depend on \(k,d,\alpha \).
The proof of Theorem 2.1 is deferred to the next section. Our proof uses similar ideas as [3, Proposition 3], which obtained the same approximation rate for \(\alpha =1\) (with an additional logarithmic factor). The conclusion is more complicated for the critical value \(\alpha =(d+2k+1)/2\). We think this is due to the proof technique and conjecture that \(\mathcal {H}^\alpha \subseteq \mathcal {F}_{\sigma _k}(M)\) for all \(\alpha \ge (d+2k+1)/2\), see Remark 3.4. Nevertheless, in practical applications of machine learning, the dimension d is large and it is reasonable to expect that \(\alpha <(d+2k+1)/2\).
In order to apply Theorem 2.1 to shallow neural networks with finite neurons, we can approximate \(\mathcal {F}_{\sigma _k}(M)\) by the subclass
where we restrict the measure \(\mu \) to be a discrete one supported on at most N points. The next proposition shows that any function in \(\mathcal {F}_{\sigma _k}(M)\) is the limit of functions in \(\mathcal {F}_{\sigma _k}(N,M)\) as \(N\rightarrow \infty \).
Proposition 2.2
For \(k\in \mathbb {N}_0\), \(\mathcal {F}_{\sigma _k}(1)\) is the closure of \(\cup _{N\in \mathbb {N}} \mathcal {F}_{\sigma _k}(N,1)\) in \(L^\infty (\mathbb {B}^d)\).
Proof
Let us denote the closure of \(\cup _{N\in \mathbb {N}} \mathcal {F}_{\sigma _k}(N,1)\) in \(L^\infty (\mathbb {B}^d)\) by \(\widetilde{\mathcal {F}}_{\sigma _k}(1)\). We first show that \(\mathcal {F}_{\sigma _k}(1) \subseteq \widetilde{\mathcal {F}}_{\sigma _k}(1)\). For any \(f\in \mathcal {F}_{\sigma _k}(1)\) with the integral form \(f(x) = \int _{\mathbb {S}^d} \sigma _k((x^\intercal ,1)v) d\mu (v)\), we can decompose f as
where \(\mu _+\) and \(\mu _-\) are the positive and negative parts of \(\mu \). If \(f_+, f_-\in \widetilde{\mathcal {F}}_{\sigma _k}(1)\), then \(f \in \widetilde{\mathcal {F}}_{\sigma _k}(1)\). Hence, without loss of generality, we can assume \(\mu \) is a probability measure. We are going to approximate f by uniform laws of large numbers. Let \(\{v_i\}_{i=1}^N\) be N i.i.d. samples from \(\mu \). By symmetrization argument (see [60, Theorem 4.10] for example), we can bound the expected approximation error by Rademacher complexity [7]:
where \((\epsilon _1,\dots ,\epsilon _N)\) is an i.i.d. sequence of Rademacher random variables. For \(k\in \mathbb {N}\), the Lipschitz constant of \(\sigma _k\) on \([-\sqrt{2},\sqrt{2}]\) is \(k2^{(k-1)/2}\). By the contraction property of Rademacher complexity [29, Corollary 3.17],
For \(k=0\), the VC dimension of the function class \(\{f_x(v)=\sigma _0((x^\intercal ,1)v): x\in \mathbb {B}^d\}\) is at most d [60, Proposition 4.20]. Thus, we have the bound \(\mathcal {E}_0(N) \lesssim \sqrt{d/N}\) by [60, Example 5.24]. Hence, f is in the closure of \(\cup _{N\in \mathbb {N}} \mathcal {F}_{\sigma _k}(N,1)\).
Next, we show that \(\widetilde{\mathcal {F}}_{\sigma _k}(1) \subseteq \mathcal {F}_{\sigma _k}(1)\) for \(k\in \mathbb {N}_0\). Since \(\cup _{N\in \mathbb {N}} \mathcal {F}_{\sigma _k}(N,1) \subseteq \mathcal {F}_{\sigma _k}(1)\), we only need to show that \(\mathcal {F}_{\sigma _k}(1)\) is closed in \(L^\infty (\mathbb {B}^d)\). Let \(f_n(x)= \int _{\mathbb {S}^d} \sigma _k((x^\intercal ,1)v) d\mu _n(v)\), where \(\Vert \mu _n\Vert \le 1\), be a convergent sequence with limit \(f \in L^\infty (\mathbb {B}^d)\). It remains to show that \(f\in \mathcal {F}_{\sigma _k}(1)\).
For \(k\in \mathbb {N}\), by the compactness of \(\mathbb {S}^d\) and Prokhorov’s theorem, there exists a weakly convergent subsequence \(\mu _{n_i} \rightarrow \mu \). In particular, \(\Vert \mu \Vert \le 1\) and for any \(x\in \mathbb {B}^d\),
By the compactness of \(\mathbb {B}^d\), \(f_{n_i}\) converges uniformly to \(\widetilde{f}\). Hence \(f= \widetilde{f} \in \mathcal {F}_{\sigma _k}(1)\).
For \(k=0\), we use the idea from [58, Lemma 3]. We can view \(f_n\) as a Bochner integral \(\int _\mathbb {D}i_{\mathbb {D}\rightarrow L^2(\mathbb {B}^d)} d\mu _n\) of the inclusion map \(i_{\mathbb {D}\rightarrow L^2(\mathbb {B}^d)}\), where \(\mathbb {D}:=\{g_v(x)=\sigma _0((x^\intercal ,1)v): v\in \mathbb {S}^d \}\). Notice that the set \(\mathbb {D}\subseteq L^2(\mathbb {B}^d)\) is compact, because the mapping \(v\mapsto g_v\) is continuous. By Prokhorov’s theorem, there exists a weakly convergent subsequence \(\mu _{n_i} \rightarrow \mu \). Let us denote \(\widetilde{f}= \int _\mathbb {D}i_{\mathbb {D}\rightarrow L^2(\mathbb {B}^d)} d\mu \), then \(\widetilde{f} \in \mathcal {F}_{\sigma _k}(1)\) by viewing the Bochner integral as an integral over \(\mathbb {S}^d\). If we choose a countable dense sequence \(\{g_j\}_{j=1}^\infty \) of \(L^2(\mathbb {B}^d)\), then the weak convergence implies that
for all j. The strong convergence \(f_{n_i} \rightarrow f\) in \(L^\infty (\mathbb {B}^d)\) implies that the same equality for f replacing \(\widetilde{f}\). Therefore, \(\langle g_j, f \rangle _{L^2(\mathbb {B}^d)} = \langle g_j, \widetilde{f} \rangle _{L^2(\mathbb {B}^d)}\) for all j, which shows \(f= \widetilde{f} \in \mathcal {F}_{\sigma _k}(1)\). \(\square \)
The proof of Proposition 2.2 actually shows the approximation rate \(\mathcal {O}(N^{-1/2})\) for the subclass \(\mathcal {F}_{\sigma _k}(N,1)\). This rate can be improved if we take into account the smoothness of the activation function. For ReLU activation, Bach [3, Proposition 1] showed that approximating \(f\in \mathcal {F}_{\sigma }(1)\) by neural networks with finitely many neurons is essentially equivalent to the approximation of a zonoid by zonotopes [10, 37]. Using this equivalence, he obtained the rate \(\mathcal {O}(N^{-\frac{1}{2}-\frac{3}{2d}})\) for ReLU neural networks. Similar idea was applied to the Heaviside activation in [32, Theorem 4], which proved the rate \(\mathcal {O}(N^{-\frac{1}{2}-\frac{1}{2d}})\) for such an activation function. For ReLU\(^k\) neural networks, the general approximation rate \(\mathcal {O}(N^{-\frac{1}{2} - \frac{2k+1}{2d}})\) was established in \(L^2\) norm by [57], which also showed that this rate is sharp. The recent work [55] further proved that this rate indeed holds in the uniform norm. We summarize their results in the following lemma.
Lemma 2.3
( [55]) For \(k\in \mathbb {N}_0\) and \(d\in \mathbb {N}\), it holds that
Combining Theorem 2.1 and Lemma 2.3, we can derive the rate of approximation by shallow neural network \(\mathcal {F}_{\sigma _k}(N,M)\) for Hölder class \(\mathcal {H}^\alpha \). Recall that we use the notation \(a\vee b:= \max \{a,b\}\).
Corollary 2.4
Let \(k\in \mathbb {N}_0\), \(d\in \mathbb {N}\) and \(\alpha >0\).
-
1.
If \(\alpha > (d+2k+1)/2\) or \(\alpha = (d+2k+1)/2\) is an even integer, then there exists a constant M depending on \(k,d,\alpha \) such that
$$\begin{aligned} \sup _{h\in \mathcal {H}^\alpha } \inf _{f\in \mathcal {F}_{\sigma _k}(N,M)} \Vert h-f\Vert _{L^\infty (\mathbb {B}^d)} \lesssim N^{-\frac{1}{2} - \frac{2k+1}{2d}}. \end{aligned}$$ -
2.
If \(\alpha = (d+2k+1)/2\) is not an even integer, then there exists \(M \asymp \sqrt{\log N}\) such that
$$\begin{aligned} \sup _{h\in \mathcal {H}^\alpha } \inf _{f\in \mathcal {F}_{\sigma _k}(N,M)} \Vert h-f\Vert _{L^\infty (\mathbb {B}^d)} \lesssim N^{-\frac{1}{2} - \frac{2k+1}{2d}} \sqrt{\log N}. \end{aligned}$$ -
3.
If \(\alpha < (d+2k+1)/2\), then
$$\begin{aligned} \sup _{h\in \mathcal {H}^\alpha } \inf _{f\in \mathcal {F}_{\sigma _k}(N,M)} \Vert h-f\Vert _{L^\infty (\mathbb {B}^d)} \lesssim N^{-\frac{\alpha }{d}} \vee M^{-\frac{2\alpha }{d+2k+1-2\alpha }}. \end{aligned}$$Thus, the rate \(\mathcal {O}(N^{-\alpha /d})\) holds when \(M\gtrsim N^{(d+2k+1-2\alpha )/(2d)}\).
Proof
We only present the proof for part (3), since other parts can be derived similarly. If \(\alpha < (d+2k+1)/2\), then by Theorem 2.1, for any \(h\in \mathcal {H}^\alpha \), there exists \(g\in \mathcal {F}_{\sigma _k}(K)\) such that \(\Vert h-g\Vert _{L^\infty (\mathbb {B}^d)} \lesssim K^{-\frac{2\alpha }{d+2k+1-2\alpha }}\). By Lemma 2.3, then there exists \(f\in \mathcal {F}_{\sigma _k}(N,K)\) such that \(\Vert g-f\Vert _{L^\infty (\mathbb {B}^d)} \lesssim KN^{-\frac{1}{2} - \frac{2k+1}{2d}}\). If \(M \ge N^{\frac{d+2k+1-2\alpha }{2d}}\), we choose \(K=N^{\frac{d+2k+1-2\alpha }{2d}}\) then \(f\in \mathcal {F}_{\sigma _k}(N,K) \subseteq \mathcal {F}_{\sigma _k}(N,M)\) and
If \(M \le N^{\frac{d+2k+1-2\alpha }{2d}}\), we choose \(K=M\), then
Combining the two bounds gives the desired result. \(\square \)
We make some comments on the approximation rate for \(\mathcal {H}^\alpha \) with \(\alpha < (d+2k+1)/2\). As shown by [48, Corollary 6.10], the rate \(\mathcal {O}(N^{-\alpha /d})\) in the \(L^2\) norm is already known for \(\alpha =1,2,\dots ,(d+2k+1)/2\). For ReLU activation, the recent paper [36] obtained the rate \(\mathcal {O}(N^{-\frac{\alpha }{d} \frac{d+2}{d+4}})\) in the supremum norm. Corollary 2.4 shows that the rate \(\mathcal {O}(N^{-\alpha /d})\) holds in the supremum norm for all ReLU\(^k\) activations. And more importantly, we also provide an explicit control on the network weights to ensure that this rate can be achieved, which is useful for estimating generalization errors (see Sect. 4.2). It is well-known that the optimal approximation rate for \(\mathcal {H}^\alpha \) is \(\mathcal {O}(N^{-\alpha /d})\), if we approximate \(h\in \mathcal {H}^\alpha \) by a function class with N parameters and the parameters are continuously dependent on the target function h [13]. However, this result is not directly applicable to neural networks, because we do not have guarantee that the parameters in the network depend continuously on the target function (in fact, this is not true for some constructions [31, 63, 65]). Nevertheless, one can still prove that the rate \(\mathcal {O}(N^{-\alpha /d})\) is optimal for shallow ReLU\(^k\) neural networks by arguments based on pseudo-dimension as done in [31, 63, 64].
We describe the idea of proving approximation lower bounds through pseudo-dimension by reviewing the result of Maiorov and Ratsaby [33] (see also [1]). Recall that the pseudo-dimension \(\,\textrm{Pdim}\,(\mathcal {F})\) of a real-valued function class \(\mathcal {F}\) defined on \(\mathbb {B}^d\) is the largest integer n for which there exist points \(x_1,\dots ,x_n \in \mathbb {B}^d\) and constants \(c_1,\dots ,c_n\in \mathbb {R}\) such that
Maiorov and Ratsaby [33] introduced a nonlinear n-width defined as
where \(p\in [1,\infty ]\) and \(\mathcal {F}_n\) runs over all the classes in \(L^p(\mathbb {B}^d)\) with \(\,\textrm{Pdim}\,(\mathcal {F}_n)\le n\). They constructed a well-separated subclass of \(\mathcal {H}^\alpha \) such that if a function class \(\mathcal {F}\) can approximate this subclass with small error, then \(\,\textrm{Pdim}\,(\mathcal {F})\) should be large. In other words, the approximation error of any class \(\mathcal {F}_n\) with \(\,\textrm{Pdim}\,(\mathcal {F}_n)\le n\) can be lower bounded. Consequently, they proved that
By [6], we can upper bound the pseudo-dimension of shallow ReLU\(^k\) neural networks as \(n:= \,\textrm{Pdim}\,(\mathcal {F}_{\sigma _k}(N,M)) \lesssim N \log N\). Hence,
which shows that the rate \(\mathcal {O}(N^{-\alpha /d})\) in Corollary 2.4 is optimal in the \(L^p\) norm (ignoring logarithmic factors). This also implies the optimality of Theorem 2.1 (otherwise, the proof of Corollary 2.4 would give a rate better than \(\mathcal {O}(N^{-\alpha /d})\)).
3 Proof of Theorem 2.1
Following the idea of [3], we first transfer the problem to approximation on spheres. Let us begin with a brief review of harmonic analysis on spheres [12]. For \(n\in \mathbb {N}_0\), the spherical harmonic space \(\mathbb {Y}_n\) of degree n is the linear space that contains the restrictions of real harmonic homogeneous polynomials of degree n on \(\mathbb {R}^{d+1}\) to the sphere \(\mathbb {S}^d\). The dimension of \(\mathbb {Y}_n\) is \(N(d,n):= \frac{2n+d-1}{n} \left( {\begin{array}{c}n+d-2\\ d-1\end{array}}\right) \) if \(n\ne 0\) and \(N(d,n):=1\) if \(n=0\). Spherical harmonics are eigenfunctions of the Laplace-Beltrami operator:
where in the coordinates \(u=(u_1,\dots ,u_{d+1}) \in \mathbb {S}^d\),
Spherical harmonics of different degrees are orthogonal with respect to the inner product \(\langle f,g \rangle = \int _{\mathbb {S}^d} f(u)g(u) d\tau _d(u)\), where \(\tau _d\) is the surface area measure of \(\mathbb {S}^d\) (normalized by the surface area \(\omega _d:= 2\pi ^{(d+1)/2}/\Gamma ((d+1)/2)\) so that \(\tau _d(\mathbb {S}^d)=1\)).
Let \(\mathcal {P}_n:L^2(\mathbb {S}^d)\rightarrow \mathbb {Y}_n\) denote the orthogonal projection operator. For any orthonormal basis \(\{Y_{nj}:1\le j\le N(d,n)\}\) of \(\mathbb {Y}_n\), the addition formula [12, Theorem 1.2.6] shows
where \(P_n\) is the Gegenbauer polynomial
with normalization \(P_n(1)=1\). Applying the Cauchy-Schwarz inequality to (3.1), we get \(|P_n(t)|\le 1\). For \(n \ne 0\), \(P_n(t)\) is odd (even) if n is odd (even). Note that, for \(d=1\) and \(n\ne 0\), \(N(d,n)=2\) and \(P_n(t)\) is the Chebyshev polynomial such that \(P_n(\cos \theta )=\cos (n \theta )\). We can write the projection \(\mathcal {P}_n\) as
This motivates the following definition of a convolution operator on the sphere.
Definition 3.1
(Convolution) Let \(\varrho \) be the probability distribution with density \(c_d (1-t^2)^{(d-2)/2}\) on \([-1,1]\), with the constant \(c_d =( \int _{-1}^1 (1-t^2)^{(d-2)/2}dt)^{-1} = \omega _{d-1}/\omega _d\). For \(f\in L^1(\mathbb {S}^d)\) and \(g\in L^1_\varrho ([-1,1])\), define
The convolution on the sphere satisfies Young’s inequality [12, Theorem 2.1.2]: for \(p,q,r \ge 1\) with \(p^{-1}= q^{-1}+r^{-1}-1\), it holds
where the norm is the uniform one when \(r=\infty \). Observe that the projection \(\mathcal {P}_n f = f*(N(d,n) P_n)\) is a convolution operator with \(\Vert N(d,n) P_n\Vert _{L^\infty ([-1,1])} \le N(d,n)\). Furthermore, for \(g\in L^1_\varrho ([-1,1])\), let \(\widehat{g}(n)\) denote the Fourier coefficient of g with respect to the Gegenbauer polynomials,
By the Funk-Hecke formula, one can show that [12, Theorem 2.1.3]
This identity is analogous to the Fourier transform of ordinary convolution.
One of the key steps in our proof of Theorem 2.1 is the observation that functions of the form \(f(u)= \int _{\mathbb {S}^d} \phi (v) \sigma _k(u^\intercal v) d\tau _d(v)\) are convolutions \(\phi *\sigma _k\) with the activation function \(\sigma _k\in L^\infty ([-1,1])\). [3, Appendix D.2] has computed the Fourier coefficients \(\widehat{\sigma _k}(n)\) explicitly. We summarize the result in the following.
Proposition 3.2
For \(k\in \mathbb {N}_0\), \(\widehat{\sigma _k}(n)=0\) if and only if \(n\ge k+1\) and \(n \equiv k\bmod 2\). If \(n=0\),
If \(n\ge k+1\) and \(n+1 \equiv k\bmod 2\),
By the Stirling formula \(\Gamma (x) = \sqrt{2\pi } x^{x-1/2}e^{-x}(1+\mathcal {O}(x^{-1}))\), we have \(\widehat{\sigma _k}(n) \asymp n^{-(d+2k+1)/2}\) for \(n\in \mathbb {N}\) satisfying \(\widehat{\sigma _k}(n)\ne 0\).
Next, we introduce the smoothness of functions on the sphere. For \(0\le \theta \le \pi \), the translation operator \(T_\theta \), also called spherical mean operator, is defined by
where \(\mathbb {S}^\perp _u:= \{v\in \mathbb {S}^d:u^\intercal v =0\}\) is the equator in \(\mathbb {S}^d\) with respect to u (hence \(\mathbb {S}^\perp _u\) is isomorphic to the sphere \(\mathbb {S}^{d-1}\)). We note that the translation operator satisfies \(\mathcal {P}_n(T_\theta f) = P_n(\cos \theta ) \mathcal {P}_n(f)\). For \(\alpha >0\) and \(0<\theta <\pi \), we define the \(\alpha \)-th order difference operator
where \(\left( {\begin{array}{c}\alpha \\ j\end{array}}\right) =\frac{\alpha (\alpha -1)\cdots (\alpha -j+1)}{j!}\), in a distributional sense by \(\mathcal {P}_n(\Delta _\theta ^\alpha f) = (1-P_n(\cos \theta ))^{\alpha /2} \mathcal {P}_n f\), \(n\in \mathbb {N}_0\). For \(f\in L^p(\mathbb {S}^d)\) and \(1\le p<\infty \) or \(f\in C(\mathbb {S}^d)\) and \(p=\infty \), the \(\alpha \)-th order modulus of smoothness is defined by
For even integers \(\alpha =2s\), one can also use combinations of \(T_{j\theta }\) and obtain [15, 50]
Another way to characterize the smoothness is through the K-functionals. We first introduce the fractional Sobolev space induced by the Laplace-Beltrami operator. We say a function \(f\in L^p(\mathbb {S}^d)\) belong to the Sobolev space \(\mathcal {W}^{\alpha ,p}(\mathbb {S}^d)\) if there exists a function in \(L^p(\mathbb {S}^d)\), which will be denoted by \((-\Delta )^{\alpha /2}f\), such that
where we assume \(f,(-\Delta )^{\alpha /2}f\in C(\mathbb {S}^d)\) for \(p=\infty \). Then we can define the \(\alpha \)-th K-functional of \(f\in L^p(\mathbb {S}^d)\) as
It can be shown [12, Theorem 10.4.1] that the moduli of smoothness and the K-functional are equivalent:
To prove Theorem 2.1, we denote the function class
as the corresponding function class of \(\mathcal {F}_{\sigma _k}(M)\) on \(\mathbb {S}^d\). Abusing the notation, we will also denote \(\gamma (g) =\inf _{\mu } \Vert \mu \Vert \) as the variation norm of \(g\in \mathcal {G}_{\sigma _k}(M)\). The next proposition transfers our approximation problem on the unit ball \(\mathbb {B}^d\) to that on the sphere \(\mathbb {S}^d\).
Proposition 3.3
Let \(k\in \mathbb {N}_0\), \(d\in \mathbb {N}\) and \(\alpha =r+\beta \) where \(r\in \mathbb {N}_0\) and \(\beta \in (0,1]\). Denote \(\Omega := \{(u_1,\dots ,u_{d+1})^\intercal \in \mathbb {S}^d: u_{d+1}\ge 1/\sqrt{2} \}\) and define an operator \(S_k:L^\infty (\Omega ) \rightarrow L^\infty (\mathbb {B}^d)\) by
The operator \(S_k\) satisfies: (1) If \(g\in \mathcal {G}_{\sigma _k}(M)\), then \(S_kg\in \mathcal {F}_{\sigma _k}(M)\). (2) For any \(h\in \mathcal {H}^\alpha \), there exists \(\widetilde{h}\in C(\mathbb {S}^d)\) such that \(S_k \widetilde{h} =h\), \(\Vert \widetilde{h}\Vert _{L^\infty (\mathbb {S}^d)}\le C\) and \(\omega _{2s^*}(\widetilde{h},t)_\infty \le C t^{\alpha }\), where \(s^*\in \mathbb {N}\) is the smallest integer such that \(\alpha \le 2s^*\) and C is a constant independent of h. Furthermore, \(\widetilde{h}\) can be chosen to be odd or even.
Proof
-
1.
If \(g(u) = \int _{\mathbb {S}^d} \sigma _k(u^\intercal v) d\mu (v)\) for some \(\Vert \mu \Vert \le M\), then for \(x\in \mathbb {B}^d\),
$$\begin{aligned} S_kg(x)&= (\Vert x\Vert _2^2+1)^{k/2} \int _{\mathbb {S}^d} \sigma _k\left( (\Vert x\Vert _2^2+1)^{-1/2}(x^\intercal ,1) v\right) d\mu (v) \\&= \int _{\mathbb {S}^d} \sigma _k\left( (x^\intercal ,1) v\right) d\mu (v). \end{aligned}$$Hence, \(S_kg \in \mathcal {F}_{\sigma _k}(M)\) by definition.
-
2.
Given \(h\in \mathcal {H}^\alpha \), for any \(u=(u_1,\dots ,u_{d+1})^\intercal \in \Omega \), we define \(\widetilde{h}(u):= u_{d+1}^k h(u_{d+1}^{-1}u')\), where \(u'=(u_1,\dots ,u_d)^\intercal \). It is easy to check that \(S_k \widetilde{h} =h\). Note that h is completely determined by the function values of \(\widetilde{h}\) on \(\Omega \). Observe that the smoothness of \(\widetilde{h}\) on \(\Omega \) can be controlled by the smoothness of h. We can extend \(\widetilde{h}\) to \(\mathbb {R}^{d+1}\) so that \(\Vert \widetilde{h}\Vert _{C^{r,\beta }(\mathbb {R}^{d+1})} \le C_0\) for some constant \(C_0\) independent of h, by using (refined version of) Whitney’s extension theorem [17,18,19]. It remains to show that \(\omega _{2\,s^*}(\widetilde{h},t)_\infty \lesssim t^{\alpha }\). By the equivalence (3.3),
$$\begin{aligned} \omega _{2s^*}(\widetilde{h},t)_\infty&\lesssim \sup _{0<\theta \le t} \sup _{u\in \mathbb {S}^d} \left| \sum _{j=0}^{2s^*} (-1)^j \left( {\begin{array}{c}2s^*\\ j\end{array}}\right) \int _{\mathbb {S}^\perp _u} \widetilde{h}(u\cos j\theta + v\sin j\theta ) d\tau _{d-1}(v) \right| \\&\le \sup _{0<\theta \le t} \sup _{u\in \mathbb {S}^d} \sup _{v\in \mathbb {S}^\perp _u} \left| \sum _{j=0}^{2s^*} (-1)^j \left( {\begin{array}{c}2s^*\\ j\end{array}}\right) \widetilde{h}(u\cos j\theta + v\sin j\theta ) \right| \\&=: \sup _{0<\theta \le t} \sup _{u\in \mathbb {S}^d} \sup _{v\in \mathbb {S}^\perp _u} |H(u,v,\theta )|. \end{aligned}$$Next, we estimate \(\sup _{0<\theta \le t}|H(u,v,\theta )|\) for small \(t>0\) and fixed u, v. One can check that the function \(f(\cdot ):= \widetilde{h}(u\cos (\cdot ) + v\sin (\cdot ))\) is in \(C^{r,\beta }([0,t_0])\) for small \(t_0>0\), and \(\Vert f\Vert _{C^{r,\beta }([0,t_0])} \lesssim \Vert \widetilde{h}\Vert _{C^{r,\beta }(\mathbb {R}^{d+1})}\). Let \(\widetilde{\Delta }_\theta f(\cdot ):= f(\cdot +\theta ) - f(\cdot )\) be the difference operator and \(\widetilde{\Delta }_\theta ^{n+1}:= \widetilde{\Delta }_\theta \widetilde{\Delta }_\theta ^n\) for \(n\in \mathbb {N}\). The binomial theorem shows \(H(u,v,\theta ) = \widetilde{\Delta }_\theta ^{2\,s^*}f(0)\). Then, the classical theory of moduli of smoothness [14, Chapter 2.6-\(-\)2.9] implies
$$\begin{aligned} \sup _{0<\theta \le t}|H(u,v,\theta )| \le \sup _{0<\theta \le t}|\widetilde{\Delta }_\theta ^{2s^*}f(0)| \lesssim \Vert f\Vert _{C^{r,\beta }([0,t_0])} t^\alpha . \end{aligned}$$Consequently, we get the desired bound \(\omega _{2s^*}(\widetilde{h},t)_\infty \lesssim t^{\alpha }\). Finally, in order to ensure that \(\widetilde{h}\) is odd or even, we can multiply \(\widetilde{h}\) by an infinitely differentiable function, which is equal to one on \(\Omega \) and zero for \(u_{d+1}\le 1/(2\sqrt{2})\), and extend \(\widetilde{h}\) to be odd or even. These operations do not decrease the smoothness of \(\widetilde{h}\). \(\square \)
By Proposition 3.3, for any \(h\in \mathcal {H}^\alpha \) and \(g\in \mathcal {G}_{\sigma _k}(M)\), we have
for some \(\widetilde{h}\in C(\mathbb {S}^d)\). Since \(S_kg\in \mathcal {F}_{\sigma _k}(M)\), we can derive approximation bounds for \(\mathcal {F}_{\sigma _k}(M)\) by studying the approximation capacity of \(\mathcal {G}_{\sigma _k}(M)\). Now, we are ready to prove Theorem 2.1.
Proof of Theorem 2.1
By Proposition 3.3, for any \(h\in \mathcal {H}^\alpha \), there exists \(\widetilde{h}\in C(\mathbb {S}^d)\) such that \(S_k \widetilde{h} =h\), \(\Vert \widetilde{h}\Vert _{L^\infty (\mathbb {S}^d)}\le C\) and \(\omega _{2s^*}(\widetilde{h},t)_\infty \le Ct^{\alpha }\), where \(s^*\in \mathbb {N}\) is the smallest integer such that \(\alpha \le 2s^*\). We choose \(\widetilde{h}\) to be odd (even) if k is even (odd). Using \(\omega _s(\widetilde{h},t)_2\le 2^{s-2\,s^*+2}\omega _{2\,s^*}(\widetilde{h},t)_2\) for \(s>2\,s^*\) [12, Proposition 10.1.2] and the Marchaud inequality [15, Eq.(9.6)]
we have
We study how well \(g\in \mathcal {G}_{\sigma _k}(M)\) approximates \(\widetilde{h}\). It turns out that it is enough to consider a subset of \(\mathcal {G}_{\sigma _k}(M)\) that contains functions of the form
for some \(\phi \in L^2(\mathbb {S}^d)\). Note that \(\gamma (g)\le \inf _{\phi } \Vert \phi \Vert _{L^1(\mathbb {S}^d)} \le \inf _{\phi } \Vert \phi \Vert _{L^2(\mathbb {S}^d)}\), where the infimum is taken over all \(\phi \in L^2(\mathbb {S}^d)\) satisfy the integral representation of g. Observing that \(g=\phi * \sigma _k\) is a convolution, by identity (3.2), \(\mathcal {P}_n g = \widehat{\sigma _k}(n) \mathcal {P}_n\phi \). Hence, we have the Fourier decomposition
which converges in \(L^2(\mathbb {S}^d)\). This implies that \(g\in \mathcal {G}_{\sigma _k}(M)\) if g is continuous, \(\mathcal {P}_n g=0\) for any \(n\in \mathbb {N}_0\) satisfying \(\widehat{\sigma _k}(n)= 0\) and
By Proposition 3.2, we know that \(\widehat{\sigma _k}(n)=0\) if and only if \(n\ge k+1\) and \(n \equiv k\bmod 2\). For \(\widehat{\sigma _k}(n)\ne 0\), we have \(\widehat{\sigma _k}(n) \asymp n^{-(d+2k+1)/2}\).
We consider the convolutions \(g_m:=\widetilde{h}*L_m = \int _{\mathbb {S}^d} \widetilde{h}(u)L_m(u^\intercal v)d\tau _d(v)\) with
where \(\eta \) is a \(C^\infty \)-function on \([0,\infty )\) such that \(\eta (t)=1\) for \(0\le t\le 1\) and \(\eta (t)=0\) for \(t\ge 2\). Since \(\eta \) is supported on [0, 2], the summation can be terminated at \(n=2m-1\), so that \(g_m\) is a polynomial of degree at most \(2m-1\). Since \(\widetilde{h}\) is odd (even) if k is even (odd), \(\mathcal {P}_n g_m= \eta (n/m) \mathcal {P}_n \widetilde{h}=0\) for any \(n \equiv k\bmod 2\). Furthermore, [12, Theorem 10.3.2] shows that
By the equivalence (3.4) and \(\omega _{2s^*}(\widetilde{h},m^{-1})_\infty \lesssim m^{-\alpha }\), the equivalence (3.6) for \(p=\infty \) implies that we can bound the approximation error as
Applying the estimate (3.5) to the equivalence (3.6) with \(p=2\), we get
Using \(\mathcal {P}_n((-\Delta )^{s/2}g_m) = (n(n+d-1))^{s/2}\mathcal {P}_n g_m\), we can estimate the norm \(\gamma (g_m)\) as follows
where we choose \(s=(d+2k+1)/2\) in the last inequality. We continue the proof in three different cases.
Case I: \(\alpha > (d+2k+1)/2\) or \(\alpha = (d+2k+1)/2\) is an even integer. In this case, \(s<\alpha \) or \(s=\alpha =2\,s^*\). Thus,
Since \(\mathcal {P}_n g_m=\eta (n/m)\mathcal {P}_n \widetilde{h} = \mathcal {P}_n \widetilde{h}\) for \(n\le m\), we have
This shows that \(\widetilde{h} \subseteq \mathcal {G}_{\sigma _k}(M)\) for some constant M. Hence, \(h=S_k \widetilde{h} \in \mathcal {F}_{\sigma _k}(M)\) by Proposition 3.3.
Case II: \(\alpha = (d+2k+1)/2\) is not an even integer. We have \(s=\alpha \ne 2s^*\) and
This shows that \(g_m \in \mathcal {G}_{\sigma _k}(M)\) with \(M\lesssim \sqrt{\log m}\). Therefore,
By Proposition 3.3, \(f:=S_k g_m \in \mathcal {F}_{\sigma _k}(M)\) and
Case III: \(\alpha < (d+2k+1)/2\). In this case, \(s>\alpha \) and
This shows that \(g_m \in \mathcal {G}_{\sigma _k}(M)\) with \(M\lesssim m^{(d+2k+1-2\alpha )/2}\). Therefore,
By Proposition 3.3, \(f:=S_k g_m \in \mathcal {F}_{\sigma _k}(M)\) and
which finishes the proof. \(\square \)
Remark 3.4
Since we are only able to estimate the smoothness \(\omega _{2s^*}(\widetilde{h},t)_\infty \) for even integer \(2s^*\), we have an extra logarithmic factor for the bound \(\omega _s(\widetilde{h},t)_2 \lesssim t^\alpha \sqrt{\log (1/t)}\) in (3.5) when \(s=\alpha \ne 2s^*\), due to the Marchaud inequality. Consequently, we can only obtain exponential convergence rate when \(\alpha = (d+2k+1)/2\) is not an even integer. We conjecture the bound \(\omega _s(\widetilde{h},t)_2 \lesssim t^\alpha \) holds for all \(s\ge \alpha \). If this is the case, then the proof of Theorem 2.1 implies \(\mathcal {H}^\alpha \subseteq \mathcal {F}_{\sigma _k}(M)\) for some constant M when \(\alpha \ge (d+2k+1)/2\).
4 Nonparametric Regression
In this section, we apply our approximation results to nonparametric regression using neural networks. For simplicity, we will only consider ReLU activation function (\(k=1\)), which is the most popular activation in deep learning.
We study the classical problem of learning a d-variate function \(h\in \mathcal {H}\) from its noisy samples, where we will assume \(\mathcal {H}= \mathcal {H}^\alpha \) with \(\alpha <(d+3)/2\) or \(\mathcal {H}= \mathcal {F}_{\sigma }(1)\). Note that, due to Theorem 2.1, the results for \(\mathcal {F}_{\sigma }(1)\) can be applied to \(\mathcal {H}^\alpha \) with \(\alpha >(d+3)/2\) by scaling the variation norm. Suppose we have a data set of \(n\ge 2\) samples \(\mathcal {D}_n = \{(X_i,Y_i)\}_{i=1}^n \subseteq \mathbb {B}^d \times \mathbb {R}\) which are independently and identically generated from the regression model
where \(\mu \) is the marginal distribution of the covariates \(X_i\) supported on \(\mathbb {B}^d\), and \(\eta _i\) is an i.i.d. Gaussian noise independent of \(X_i\) (we will treat the variance \(V^2\) as a fixed constant). We are interested in the empirical risk minimizer (ERM)
where \(\mathcal {F}_n\) is a function class parameterized by neural networks. For simplicity, we assume here and in the sequel that the minimum above indeed exists. The performance of the estimation is measured by the expected risk
It is equivalent to evaluating the estimator by the excess risk
In the statistical analysis of learning algorithms, we often require that the hypothesis class is uniformly bounded. We define the truncation operator \(\mathcal {T}_B\) with level \(B>0\) for real-valued functions f as
Since we always assume the regression function h is bounded, truncating the output of the estimator \(f_n^*\) appropriately dose not increase the excess risk. We will estimate the convergence rate of \(\mathbb {E}_{\mathcal {D}_n} \Vert \mathcal {T}_{B_n}f_n^*-h\Vert _{L^2(\mu )}^2\), where \(B_n \lesssim \log n\), as the number of samples \(n\rightarrow \infty \).
4.1 Shallow Neural Networks
The rate of convergence of neural network regression estimates has been analyzed by many papers [9, 26, 27, 38, 41, 52]. It is well-known that the optimal minimax rate of convergence for learning a regression function \(h\in \mathcal {H}^\alpha \) is \(n^{-2\alpha /(d+2\alpha )}\) [59]. This optimal rate has been established (up to logarithmic factors) for two-hidden-layers neural networks with certain squashing activation functions [26] and for deep ReLU neural networks [27, 52]. For shallow networks, [38] proved a rate of \(n^{-2\alpha /(2\alpha +d+5)+\epsilon }\) with \(\epsilon >0\) for a certain cosine squasher activation function. However, to the best of our knowledge, it is unknown whether shallow neural networks can achieve the optimal rate. In this section, we provide an affirmative answer to this question by proving that shallow ReLU neural networks can achieve the optimal rate for \(\mathcal {H}^\alpha \) with \(\alpha <(d+3)/2\).
We will use the following lemma to analyze the convergence rate. It decomposes the error of the ERM into generalization error and approximation error, and bounds the generalization error by the covering number of the hypothesis class \(\mathcal {F}_n\).
Lemma 4.1
( [27]) Let \(f_n^*\) be the estimator (4.2) and set \(B_n = c_1\log n\) for some constant \(c_1>0\). Then,
for \(n>1\) and some constant \(c_2>0\) (independent of n and \(f_n^*\)), where \(X_{1:n} =(X_1,\dots ,X_n)\) denotes a sequence of sample points in \(\mathbb {B}^d\) and \(\mathcal {N}(\epsilon , \mathcal {T}_{B_n}\mathcal {F}_n,\Vert \cdot \Vert _{L^1(X_{1:n})})\) denotes the \(\epsilon \)-covering number of the function class \(\mathcal {T}_{B_n}\mathcal {F}_n:=\{\mathcal {T}_{B_n}f,f\in \mathcal {F}_n\}\) in the metric \(\Vert f-g\Vert _{L^1(X_{1:n})} = \frac{1}{n}\sum _{i=1}^n|f(X_i)-g(X_i)|\).
For shallow neural network model \(\mathcal {F}_n= \mathcal {F}_{\sigma }(N_n,M_n)\), Lemma 2.3 and Corollary 2.4 provide bounds for the approximation errors. The covering number of the function class \(\mathcal {T}_{B_n}\mathcal {F}_n\) can be estimated by using the pseudo-dimension of \(\mathcal {T}_{B_n}\mathcal {F}_n\) [22]. Choosing \(N_n,M_n\) appropriately to balance the approximation and generalization errors, we can derive convergence rates for the ERM.
Theorem 4.2
Let \(f_n^*\) be the estimator (4.2) with \(\mathcal {F}_n = \mathcal {F}_{\sigma }(N_n,M_n)\) and set \(B_n = c_1\log n\) for some constant \(c_1>0\).
-
1.
If \(\mathcal {H}= \mathcal {H}^\alpha \) with \(\alpha <(d+3)/2\), we choose
$$\begin{aligned} N_n \asymp n^{\frac{d}{d+2\alpha }}, \quad M_n \gtrsim n^{\frac{d+3-2\alpha }{2d+4\alpha }}, \end{aligned}$$then
$$\begin{aligned} \mathbb {E}_{\mathcal {D}_n} \Vert \mathcal {T}_{B_n}f_n^*-h\Vert _{L^2(\mu )}^2 \lesssim n^{-\frac{2\alpha }{d+2\alpha }} (\log n)^4. \end{aligned}$$ -
2.
If \(\mathcal {H}= \mathcal {F}_{\sigma }(1)\), we choose
$$\begin{aligned} N_n \asymp n^{\frac{d}{2d+3}}, \quad M_n \ge 1, \end{aligned}$$then
$$\begin{aligned} \mathbb {E}_{\mathcal {D}_n} \Vert \mathcal {T}_{B_n}f_n^*-h\Vert _{L^2(\mu )}^2 \lesssim n^{-\frac{d+3}{2d+3}} (\log n)^4. \end{aligned}$$
Proof
To apply the bound in Lemma 4.1, we need to estimate the covering number \(\mathcal {N}(\epsilon , \mathcal {T}_{B_n}\mathcal {F}_n,\Vert \cdot \Vert _{L^1(X_{1:n})})\). The classical result of [22, Theorem 6] showed that the covering number can be bounded by pseudo-dimension:
where \(\,\textrm{Pdim}\,(\mathcal {T}_{B_n}\mathcal {F}_n)\) is the pseudo-dimension of the function class \(\mathcal {T}_{B_n}\mathcal {F}_n\), see (2.2). For ReLU neural networks, [6] showed that
Consequently, we have
Applying Lemma 4.1 and Corollary 2.4, if \(\mathcal {H}= \mathcal {H}^\alpha \) with \(\alpha <(d+3)/2\), then
By choosing \(N_n \asymp n^{d/(d+2\alpha )}\) and \(M_n \gtrsim N_n^{(d+3-2\alpha )/(2d)}\), we get \(\mathbb {E}_{\mathcal {D}_n} \Vert \mathcal {T}_{B_n}f_n^*-h\Vert _{L^2(\mu )}^2 \lesssim n^{-2\alpha /(d+2\alpha )} (\log n)^4\). Similarly, by Lemmas 4.1 and 2.3, if \(\mathcal {H}= \mathcal {F}_{\sigma }(1)\) and \(M_n \ge 1\), then
We choose \(N_n \asymp n^{d/(2d+3)}\), then \(\mathbb {E}_{\mathcal {D}_n} \Vert \mathcal {T}_{B_n}f_n^*-h\Vert _{L^2(\mu )}^2 \lesssim n^{-(d+3)/(2d+3)} (\log n)^4\). \(\square \)
Remark 4.3
The Gaussian noise assumption on the model (4.1) can be weaken for Lemma 4.1 and hence for Theorem 4.2. We refer the reader to [27, Appendix B, Lemma 18] for more details. Theorem 4.2 can be easily generalized to shallow ReLU\(^k\) neural networks for \(k\ge 1\) by using the same proof technique. For example, one can show that, if \(h\in \mathcal {H}^\alpha \) with \(\alpha <(d+2k+1)/2\), then we can choose \(\mathcal {F}_n = \mathcal {F}_{\sigma _k}(N_n,M_n)\) with \(N_n \asymp n^{\frac{d}{d+2\alpha }}\) and \(M_n \gtrsim n^{\frac{d+2k+1-2\alpha }{2d+4\alpha }}\), such that \(\mathbb {E}_{\mathcal {D}_n} \Vert \mathcal {T}_{B_n}f_n^*-h\Vert _{L^2(\mu )}^2 \lesssim n^{-\frac{2\alpha }{d+2\alpha }} (\log n)^4\).
Theorem 4.2 shows that least square minimization using shallow ReLU neural networks can achieve the optimal rate \(n^{-\frac{2\alpha }{d+2\alpha }}\) for learning functions in \(\mathcal {H}^\alpha \) with \(\alpha <(d+3)/2\). For the function class \(\mathcal {F}_{\sigma }(1)\), the rate \(n^{-\frac{d+3}{2d+3}}\) is also minimax optimal as proven by [46, Lemma 25] (they studied a slightly different function class, but their result also holds for \(\mathcal {F}_{\sigma }(1)\)). Specifically, [57, Theorem 4 and Theorem 8] give a sharp estimate for the metric entropy
Combining this estimate with the classical result of Yang and Barron (see [61, Proposition 1] and [60, Chapter 15]), we get
where the infimum taken is over all estimators based on the samples \(\mathcal {D}_n\), which are generated from the model (4.1).
4.2 Deep Neural Networks and Over-Parameterization
There is a direct way to generalize the analysis in the last section to deep neural networks: we can implement shallow neural networks by sparse multi-layer neural networks with the same order of parameters, and estimate the approximation and generalization performance of the constructed networks. Since the optimal convergence rates of deep neural networks have already been established in [27, 52], we do not pursue in this direction. Instead, we study the convergence rates of over-parameterized neural networks by using the idea discussed in [24]. The reason for studying such networks is that, in modern applications of deep learning, the number of parameters in the networks is often much larger than the number of samples. However, in the convergence analysis of [27, 52], the network that achieves the optimal rate is under-parameterized (see also the choice of \(N_n\) in Theorem 4.2). Hence, the analysis can not explain the empirical performance of deep learning models used in practice.
Following [24], we consider deep neural networks with norm constraints on weight matrices. For \(W,L\in \mathbb {N}\), we denote by \(\mathcal{N}\mathcal{N}(W,L)\) the set of functions that can be parameterized by ReLU neural networks in the form
where \(A^{(\ell )} \in \mathbb {R}^{N_{\ell +1}\times N_{\ell }}\), \(b^{(\ell )}\in \mathbb {R}^{N_{\ell +1}}\) with \(N_0 =d,N_{L+1} =1\) and \(\max \{N_1,\dots ,N_L\} =W\). The numbers W and L are called the width and depth of the neural network, respectively. Let us use the notation \(f_\theta \) to emphasize that the neural network function is parameterized by \(\theta =((A^{(0)},b^{(0)}),\dots ,(A^{(L)},b^{(L)}))\). We can define a norm constraint on the weight matrices as follows
where we use \(\Vert A\Vert := \sup _{\Vert x\Vert _\infty \le 1} \Vert Ax\Vert _\infty \) to denote the operator norm (induced by the \(\ell ^\infty \) norm) of a matrix \(A = (a_{i,j}) \in \mathbb {R}^{m\times n}\). It is well-known that \(\Vert A\Vert \) is the maximum 1-norm of the rows of A:
The motivation for such a definition of \(\kappa (\theta )\) is discussed in [24]. For \(M\ge 0\), we denote by \(\mathcal{N}\mathcal{N}(W,L,M)\) as the set of functions \(f_\theta \in \mathcal{N}\mathcal{N}(W,L)\) that satisfy \(\kappa (\theta ) \le M\). It is shown by [24, Proposition 2.5] that, if \(W_1\le W_2,L_1\le L_2, M_1\le M_2\), then \(\mathcal{N}\mathcal{N}(W_1,L_1,M_1) \subseteq \mathcal{N}\mathcal{N}(W_2,L_2,M_2)\). (Strictly speaking, [24] use the convention that the bias \(b^{(L)}=0\) in the last layer. But the results can be easily generalized to the case \(b^{(L)}\ne 0\), see [62, Section 2.1] for details.)
To derive approximation bounds for deep neural networks, we consider the relationship of \(\mathcal {F}_{\sigma }(N,M)\) and \(\mathcal{N}\mathcal{N}(N,1,M)\). The next proposition shows the function classes \(\mathcal{N}\mathcal{N}(N,1,M)\) and \(\mathcal {F}_{\sigma }(N,M)\) have essentially the same approximation power.
Proposition 4.4
For any \(N\in \mathbb {N}\) and \(M>0\), we have \(\mathcal {F}_{\sigma }(N,M)\subseteq \mathcal{N}\mathcal{N}(N,1,\sqrt{d+1}M)\) and \(\mathcal{N}\mathcal{N}(N,1,M) \subseteq \mathcal {F}_{\sigma }(N+1,M)\).
Proof
Each function \(f(x) =\sum _{i=1}^N a_i\sigma ((x^\intercal ,1)v_i)\) in \(\mathcal {F}_{\sigma }(N,M)\) can be parameterized in the form (4.4) with \(W=N,L=1\) and
Since \(v_i\in \mathbb {S}^d\), it is easy to see that \(\kappa (\theta ) \le \sqrt{d+1} M\). Hence, \(\mathcal {F}_{\sigma }(N,M)\subseteq \mathcal{N}\mathcal{N}(N,1,\sqrt{d+1}M)\).
On the other side, let \(f_\theta \in \mathcal{N}\mathcal{N}(N,1,M)\) be a function parameterized in the form (4.4) with \((A^{(0)}, b^{(0)}) = (a^{(0)}_1,\dots ,a^{(0)}_N)^\intercal \) and \((A^{(1)}, b^{(1)}) = (a^{(1)}_1,\dots ,a^{(1)}_N,b^{(1)})\), where \(a^{(0)}_i\in \mathbb {R}^{d+1}\) and \(a^{(1)}_i, b^{(1)}\in \mathbb {R}\). Then, \(f_\theta \) can be represented as
where we assume \(\Vert a^{(0)}_i\Vert _2 \ne 0\) without loss of generality. Since
we conclude that \(\mathcal{N}\mathcal{N}(N,1,M) \subseteq \mathcal {F}_{\sigma }(N+1,M)\). \(\square \)
As a corollary of Theorem 2.1 and Lemma 2.3, we get the following approximation bounds for deep neural networks.
Corollary 4.5
For \(\mathcal {H}^\alpha \) with \(0<\alpha <(d+3)/2\), we have
For \(\mathcal {F}_{\sigma }(1)\), there exists a constant \(M\ge 1\) such that
Proof
The first part is a direct consequence of Corollary 2.4 and the inclusion \(\mathcal {F}_{\sigma }(W,M) \subseteq \mathcal{N}\mathcal{N}(W,L,\sqrt{d+1}M)\). The second part follows from Lemma 2.3 and we can choose \(M=\sqrt{d+1}\). \(\square \)
In the first part of Corollary 4.5, if we allow the width W to be arbitrary large, say \(W\gtrsim M^{2d/(d+3-2\alpha )}\), then we can bound the approximation error by the size of weights. Hence, this result can be applied to over-parameterized neural networks. (Note that, in Theorem 4.2, we use a different regime of the bound.) For the approximation of \(\mathcal {F}_{\sigma }(1)\), the size of weights is bounded by a constant. We will show that this constant can be used to control the generalization error. Since the approximation error is bounded by W and is independent of M, we do not have trade-off in the error decomposition of ERM and only need to choose W sufficiently large to reduce the approximation error. Hence, it can also be applied to over-parameterized neural networks.
The approximation rate \(M^{-\frac{2\alpha }{d+3-2\alpha }}\) for \(\mathcal {H}^\alpha \) in Corollary 4.5 improves the rate \(M^{-\frac{\alpha }{d+1}}\) proven by [24]. Using the upper bound for Rademacher complexity of \(\mathcal{N}\mathcal{N}(W,L,M)\) (see Lemma 4.6), [24] also gave an approximation lower bound \((M\sqrt{L})^{-\frac{2\alpha }{d-2\alpha }}\). For fixed depth L, our upper bound is very close to this lower bound. We conjecture that the rate in Corollary 4.5 is optimal with respect to M (for fixed depth L). The discussion of optimality at the end of Sect. 2 implies that the conjecture is true for shallow neural networks (i.e. \(L=1\)).
To control the generalization performance of over-parameterized neural networks, we need to have size-independent sample complexity bounds for such networks. Several methods have been applied to obtain such kind of bounds in recent works [5, 21, 42, 43]. Here, we will use the result of [21], which estimates the Rademacher complexity of deep neural networks [7]. For a set \(S\subseteq \mathbb {R}^n\), let us denote its Rademacher complexity by
where \(\xi _{1:n} = (\xi _1,\dots , \xi _n)\) is a sequence of i.i.d. Rademacher random variables. The following lemma is from [21, Theorem 3.2] and [24, Lemma 2.3].
Lemma 4.6
For any \(x_1,\dots ,x_n \in [-1,1]^d\), let \(S:= \{(f(x_1),\dots ,f(x_n)):f \in \mathcal{N}\mathcal{N}(W,L,M) \} \subseteq \mathbb {R}^n\), then
Now, we can estimate the convergence rates of the ERM based on over-parameterized neural networks. As usual, we decompose the excess risk of the ERM into approximation error and generalization error, and bound them by Corollary 4.5 and Lemma 4.6, respectively. Note that the convergence rates in the following theorem are worse than the optimal rates in Theorem 4.2.
Theorem 4.7
Let \(f_n^*\) be the estimator (4.2) with \(\mathcal {F}_n = \{ \mathcal {T}_{B_n}f: f\in \mathcal{N}\mathcal{N}(W_n,L,M_n) \}\), where \(L\in \mathbb {N}\) is a fixed constant, \(1\le B_n \lesssim \log n\) in case (1) and \(\sqrt{2} \le B_n \lesssim \log n\) in case (2).
-
1.
If \(\mathcal {H}= \mathcal {H}^\alpha \) with \(\alpha <(d+3)/2\), we choose
$$\begin{aligned} W_n \gtrsim n^{\frac{d}{d+3+2\alpha }}, \quad M_n \asymp n^{\frac{1}{2} - \frac{2\alpha }{d+3+2\alpha }}, \end{aligned}$$then
$$\begin{aligned} \mathbb {E}_{\mathcal {D}_n} \Vert f_n^*-h\Vert _{L^2(\mu )}^2 \lesssim n^{-\frac{2\alpha }{d+3+2\alpha }} \log n. \end{aligned}$$ -
2.
If \(\mathcal {H}= \mathcal {F}_{\sigma }(1)\), we choose a large enough constant M and let
$$\begin{aligned} W_n \gtrsim n^{\frac{d}{2d+6}}, \quad M_n=M, \end{aligned}$$then
$$\begin{aligned} \mathbb {E}_{\mathcal {D}_n} \Vert f_n^*-h\Vert _{L^2(\mu )}^2 \lesssim n^{-\frac{1}{2}} \log n. \end{aligned}$$
Proof
The proof is essentially the same as [24, Theorem 4.1]. Observe that, for any \(f\in \mathcal {F}_n\),
Using \(\mathbb {E}_{\mathcal {D}_n} [\mathcal {L}_n(f)] = \mathcal {L}(f)\) and taking the infimum over \(f\in \mathcal {F}_n\), we get
Let us denote the collections of sample points and noises by \(X_{1:n} =(X_1,\dots ,X_n)\) and \(\eta _{1:n}=(\eta _1,\dots ,\eta _n)\). We can bound the generalization error as follows
where we denote \(\Phi _n:= \{f-h:f\in \mathcal {F}_n\}\). By a standard symmetrization argument (see [60, Theorem 4.10]), we can bound the first term in (4.6) by the Rademacher complexity:
where \(\Phi _n^2(X_{1:n}):= \{ (\phi ^2(X_1),\dots ,\phi ^2(X_n)) \in \mathbb {R}^n: \phi \in \Phi _n \} \subseteq \mathbb {R}^n\) is the set of function values on the sample points. Recall that we assume \(B_n\ge 1\) in case (1) and \(B_n\ge \sqrt{2}\) in case (2). Hence, we always have \(\Vert h\Vert _{L^\infty (\mathbb {B}^d)} \le \sqrt{2}\) and \(\Vert \phi \Vert _{L^\infty (\mathbb {B}^d)} \le 2B_n\) for any \(\phi \in \Phi _n\). By the structural properties of Rademacher complexity [7, Theorem 12],
where we apply Lemma 4.6 in the last inequality. Note that the second term in (4.6) is a Gaussian complexity. We can also bound it by the Rademacher complexity [7, Lemma 4]:
In summary, we conclude that
If \(\mathcal {H}= \mathcal {H}^\alpha \) with \(\alpha <(d+3)/2\), by Corollary 4.5, we have
Combining with (4.5) and (4.7), we know that if we choose \(M_n \asymp n^{\frac{1}{2} - \frac{2\alpha }{d+3+2\alpha }}\) and \(W_n \gtrsim n^{\frac{d}{d+3+2\alpha }}\), then
Similarly, if \(\mathcal {H}= \mathcal {F}_{\sigma }(1)\), by Corollary 4.5, then there exist a constant \(M\ge 1\) such that, if \(M_n=M\),
Thus, for any \(W_n \gtrsim n^{d/(2d+6)}\), we get \(\mathbb {E}_{\mathcal {D}_n} \Vert f_n^*-h\Vert _{L^2(\mu )}^2 \lesssim n^{-1/2} \log n\), \(\square \)
4.3 Convolutional Neural Networks
In contrast to the vast amount of theoretical studies on fully connected neural networks, there are only a few papers analyzing the performance of convolutional neural networks [16, 20, 30, 35, 47, 66,67,68]. The recent work [30] showed the universal consistency of CNNs for nonparametric regression. In this section, we show how to use our approximation results to analyze the convergence rates of CNNs.
Following [67], we introduce a sparse convolutional structure on deep neural networks. Let \(s\ge 2\) be a fixed integer, which is used to control the filter length. Given a sequence \(w=(w_i)_{i\in \mathbb {Z}}\) on \(\mathbb {Z}\) supported on \(\{0,1,\dots ,s\}\), the convolution of the filter w with another sequence \(x=(x_i)_{i\in \mathbb {Z}}\) supported on \(\{1,\dots ,d\}\) is a sequence \(w*x\) given by
Regarding x as a vector of \(\mathbb {R}^d\), this convolution induces a \((d+s) \times d\) Toeplitz type convolutional matrix
Note that the number of rows of \(A^{w}\) is s greater than the number of columns. This leads us to consider deep neural networks of the form (4.4) with linearly increasing widths \(\{N_\ell = d + \ell s\}_{\ell =0}^L\). We denote by \(\mathcal {CNN}(s,L)\) the set of functions that can be parameterized in the form (4.4) such that \(A^{(\ell )} = A^{w^{(\ell )}}\) for some filter \(w^{(\ell )}\) supported on \(\{0,1,\dots ,s\}\), \(0\le \ell \le L-1\), and the biases \(b^{(\ell )}\) take the special form
with \(N_\ell -2s\) repeated components in the middle. By definition, it is easy to see that \(\mathcal {CNN}(s,L) \subseteq \mathcal{N}\mathcal{N}(d+Ls,L)\). The assumption on the special form (4.8) of biases is used to reduce the free parameters in the network. As in [67], one can compute that the number of free parameters in \(\mathcal {CNN}(s,L)\) is \((5s+2)L+2d-2s\), which grows linearly on L.
The next proposition shows that all functions in \(\mathcal{N}\mathcal{N}(N,1)\) can be implemented by CNNs.
Proposition 4.8
[67] If \(N,L\in \mathbb {N}\) satisfy \(L\ge \lfloor \frac{Nd}{s-1}+1 \rfloor \), then \(\mathcal{N}\mathcal{N}(N,1) \subseteq \mathcal {CNN}(s,L)\).
Proof
This result is proven in [67, Proof of Theorem 2]. We only give a sketch of the construction for completeness. Any function \(f\in \mathcal{N}\mathcal{N}(N,1)\) can be written as \(f(x) = \sum _{i=1}^N c_i \sigma (a_i^\intercal x+b_i) + c_0\), where \(a_i\in \mathbb {R}^d\) and \(b_i,c_i\in \mathbb {R}\). Define a sequence v supported on \(\{0,\dots ,Nd-1\}\) by stacking the vectors \(a_1,\dots , a_N\) (with components reversed) by
Applying [67, Theorem 3] to the sequence v, we can construct filters \(\{w^{(\ell )}\}_{\ell =0}^{L-1}\) supported on \(\{0,1,\dots ,s\}\) such that \(v=w^{(L-1)}*w^{(L-2)}*\dots *w^{(0)}\), which implies \(A^{w^{(L-1)}} \cdots A^{w^{(0)}} = A^v \in \mathbb {R}^{(d+Ls)\times d}\). Note that, by definition, for \(i=1,\dots ,N\), the id-th row of \(A^v\) is exactly \(a_i^\intercal \). Then, for \(\ell =0,\dots ,L-2\), we can choose \(b^{(\ell )}\) satisfying (4.8) such that \(f^{(\ell +1)}(x) = A^{w^{(\ell )}} \cdots A^{w^{(0)}}x+B^{(\ell )}\), where \(B^{(\ell )}>0\) is a sufficiently large constant that makes the components of \(f^{(\ell +1)}(x)\) positive for all \(x\in \mathbb {B}^d\). Finally, we can construct \(b^{(L-1)}\) such that \(f^{(L)}_k(x)=\sigma (a_i^\intercal x+b_i)\) for \(i=1,\dots ,N\) and \(k=id\), which implies \(f\in \mathcal {CNN}(s,L)\). \(\square \)
Note that Proposition 4.8 shows each shallow neural network can be represent by a CNN, with the same order of number of parameters. As a corollary, we obtain approximation rates for CNNs.
Corollary 4.9
Let \(s\ge 2\) be an integer.
-
1.
For \(\mathcal {H}^\alpha \) with \(0<\alpha <(d+3)/2\), we have
$$\begin{aligned} \sup _{h\in \mathcal {H}^\alpha } \inf _{f\in \mathcal {CNN}(s,L)} \Vert h-f\Vert _{L^\infty (\mathbb {B}^d)} \lesssim L^{-\frac{\alpha }{d}}. \end{aligned}$$ -
2.
For \(\mathcal {F}_{\sigma }(1)\), we have
$$\begin{aligned} \sup _{h \in \mathcal {F}_{\sigma }(1)} \inf _{f\in \mathcal {CNN}(s,L)} \Vert h-f\Vert _{L^\infty (\mathbb {B}^d)} \lesssim L^{-\frac{1}{2} - \frac{3}{2d}}. \end{aligned}$$
Proof
For any \(N\in \mathbb {N}\), we take \(L=\lfloor \frac{Nd}{s-1}+1 \rfloor \asymp N\), then \(\mathcal {F}_{\sigma }(N,M) \subseteq \mathcal{N}\mathcal{N}(N,1) \subseteq \mathcal {CNN}(s,L)\) for any \(M>0\), by Proposition 4.8. (1) follows from Corollary 2.4 and (2) is from Lemma 2.3. \(\square \)
Since the number of parameters in \(\mathcal {CNN}(s,L)\) is approximately L, the rate \(\mathcal {O}(L^{-\alpha /d})\) in part (1) of Corollary 4.9 is the same as the rate in [64] for fully connected neural networks. However, [31, 65] showed that this rate can be improved to \(\mathcal {O}(L^{-2\alpha /d})\) for fully connected neural networks by using the bit extraction technique [6]. It would be interesting to see whether this rate also holds for \(\mathcal {CNN}(s,L)\).
As in Theorem 4.2, we use Lemma 4.1 to decompose the error and bound the approximation error by Corollary 4.9. The covering number is bounded again by pseudo-dimension.
Theorem 4.10
Let \(f_n^*\) be the estimator (4.2) with \(\mathcal {F}_n = \mathcal {CNN}(s,L_n)\), where \(s\ge 2\) is a fixed integer, and set \(B_n = c_1\log n\) for some constant \(c_1>0\).
-
1.
If \(\mathcal {H}= \mathcal {H}^\alpha \) with \(\alpha <(d+3)/2\), we choose
$$\begin{aligned} L_n \asymp n^{\frac{d}{2d+2\alpha }}, \end{aligned}$$then
$$\begin{aligned} \mathbb {E}_{\mathcal {D}_n} \Vert \mathcal {T}_{B_n}f_n^*-h\Vert _{L^2(\mu )}^2 \lesssim n^{-\frac{\alpha }{d+\alpha }} (\log n)^4. \end{aligned}$$ -
2.
If \(\mathcal {H}= \mathcal {F}_{\sigma }(1)\), we choose
$$\begin{aligned} L_n \asymp n^{\frac{d}{3d+3}}, \end{aligned}$$then
$$\begin{aligned} \mathbb {E}_{\mathcal {D}_n} \Vert \mathcal {T}_{B_n}f_n^*-h\Vert _{L^2(\mu )}^2 \lesssim n^{-\frac{d+3}{3d+3}} (\log n)^4. \end{aligned}$$
Proof
The proof is the same as Theorem 4.2 and [68]. We can use (4.3) to bound the covering number by the pseudo-dimension. For convolutional neural networks, [6] gave the following estimate of the pseudo-dimension:
where \(p(s,L_n)=(5\,s+2)L_n+2d-2\,s \lesssim L_n\) and \(q(s,L_n)\le L_n (d+sL_n) \lesssim L_n^2\) are the numbers of parameters and neurons of the network \(\mathcal {CNN}(s,L_n)\), respectively. Therefore,
Applying Lemma 4.1 and Corollary 4.9, if \(\mathcal {H}= \mathcal {H}^\alpha \) with \(\alpha <(d+3)/2\), then
We choose \(L_n \asymp n^{d/(2d+2\alpha )}\), then \(\mathbb {E}_{\mathcal {D}_n} \Vert \mathcal {T}_{B_n}f_n^*-h\Vert _{L^2(\mu )}^2 \lesssim n^{-\alpha /(d+\alpha )} (\log n)^4\). Similarly, if \(\mathcal {H}= \mathcal {F}_{\sigma }(1)\), then
We choose \(L_n \asymp n^{d/(3d+3)}\), then \(\mathbb {E}_{\mathcal {D}_n} \Vert \mathcal {T}_{B_n}f_n^*-h\Vert _{L^2(\mu )}^2 \lesssim n^{-(d+3)/(3d+3)} (\log n)^4\). \(\square \)
Finally, we note that the recent paper [68] also studied the convergence of CNNs and proved the rate \(\mathcal {O}(n^{-1/3}(\log n)^2)\) for \(\mathcal {H}^\alpha \) with \(\alpha >(d+4)/2\). The convergence rate we obtained in Theorem 4.10 for \(\mathcal {F}_{\sigma }(1)\), which includes \(\mathcal {H}^\alpha \) with \(\alpha >(d+3)/2\) by Theorem 2.1, is slightly better than their rate.
5 Conclusion
This paper has established approximation bounds for shallow ReLU\(^k\) neural networks. We showed how to use these bounds to derive approximation rates for (deep or shallow) neural networks with constraints on the weights and convolutional neural networks. We also applied the approximation results to study the convergence rates of nonparametric regression using neural networks. In particular, we established the optimal convergence rates for shallow neural networks and showed that over-parameterized neural networks can achieve nearly optimal rates.
There are a few interesting questions we would like to propose for future research. First, for approximation by shallow neural networks, we establish the optimal rate in the supremum norm by using the results of [55] (Lemma 2.3). The paper [55] actually showed that approximation bounds similar to Lemma 2.3 also hold in Sobolev norms. We think it is a promising direction to extend our approximation results in the supremum norm (Theorem 2.1 and Corollary 2.4) to the Sobolev norms. Second, it is unclear whether over-parameterized neural networks can achieve the optimal rate for learning functions in \(\mathcal {H}^\alpha \). It seems that refined generalization error analysis is needed. Finally, it would be interesting to extend the theory developed in this paper to general activation functions and study how the results are affected by the activation functions.
References
Achour, E.M., Foucault,A., Gerchinovitz, S., Malgouyres, F.: a general approximation lower bound in \({L}^{}\)p norm, with applications to feed-forward neural networks. In: Advances in Neural Information Processing Systems, vol. 35, pp. 22396–22408. Curran Associates, Inc. (2022)
Anthony, M., Bartlett, P.L.: Neural Network Learning: Theoretical Foundations. Cambridge University Press, Cambridge (2009)
Bach, F.: Breaking the curse of dimensionality with convex neural networks. J. Mach. Learn. Res. 18(19), 1–53 (2017)
Barron, A.R.: Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inf. Theory 39(3), 930–945 (1993)
Bartlett, P.L., Foster, D.J., Telgarsky, M.: Spectrally-normalized margin bounds for neural networks. In: Advances in Neural Information Processing Systems, vol. 30, pp. 6240–6249. Curran Associates, Inc. (2017)
Bartlett, P.L., Harvey, N., Liaw, C., Mehrabian, A.: Nearly-tight VC-dimension and Pseudodimension bounds for piecewise linear neural networks. J. Mach. Learn. Res. 20(63), 1–17 (2019)
Bartlett, P.L., Mendelson, S.: Rademacher and Gaussian complexities: risk bounds and structural results. J. Mach. Learn. Res. 3, 463–482 (2002)
Bartolucci, F., De Vito, E., Rosasco, L., Vigogna, S.: Understanding neural networks with reproducing kernel Banach spaces. Appl. Comput. Harmon. Anal. 62, 194–236 (2023)
Bauer, B., Kohler, M.: On deep learning as a remedy for the curse of dimensionality in nonparametric regression. Ann. Stat. 47(4), 2261–2285 (2019)
Bourgain, J., Lindenstrauss, J., Milman, V.: Approximation of zonoids by zonotopes. Acta Math. 162, 73–141 (1989)
Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 2(4), 303–314 (1989)
Dai, F., Xu, Y.: Approximation Theory and Harmonic Analysis on Spheres and Balls, vol. 23. Springer, Berlin (2013)
DeVore, R.A., Howard, R., Micchelli, C.: Optimal nonlinear approximation. Manuscr. Math. 63(4), 469–478 (1989)
DeVore, R.A., Lorentz, G.G.: Constructive Approximation, vol. 303. Springer, Berlin (1993)
Ditzian, Z.: Measures of smoothness on the sphere. In: Frontiers in Interpolation and Approximation, pp. 75–91. Chapman and Hall/CRC (2006)
Fang, Z., Feng, H., Huang, S., Zhou, D.-X.: Theory of deep convolutional neural networks II: spherical analysis. Neural Netw. 131, 154–162 (2020)
Fefferman, C.: Whitney’s extension problem for \(C^m\). Ann. Math. 164(1), 313–359 (2006)
Fefferman, C.: Extension of \(C^{m,\omega }\)-smooth functions by linear operators. Revista Matemática Iberoamericana 25(1), 1–48 (2009)
Fefferman, C., Israel, A.: Fitting Smooth Functions to Data. American Mathematical Society, Providence (2020)
Feng, H., Huang, S., Zhou, D.-X.: Generalization analysis of CNNs for classification on spheres. IEEE Trans. Neural Netw. Learn. Syst. 34(9), 6200–6213 (2023)
Golowich, N., Rakhlin, A., Shamir, O.: Size-independent sample complexity of neural networks. Inf. Inference: J. IMA 9(2), 473–504 (2020)
Haussler, D.: Decision theoretic generalizations of the PAC model for neural net and other learning applications. Inf. Comput. 100(1), 78–150 (1992)
Hornik, K.: Approximation capabilities of multilayer feedforward networks. Neural Netw. 4(2), 251–257 (1991)
Jiao, Y., Wang, Y., Yang, Y.: Approximation bounds for norm constrained neural networks with applications to regression and GANs. Appl. Comput. Harmon. Anal. 65, 249–278 (2023)
Klusowski, J.M., Barron, A.R.: Approximation by combinations of ReLU and squared ReLU ridge functions with \(l^1\) and \(l^0\) controls. IEEE Trans. Inf. Theory 64(12), 7649–7656 (2018)
Kohler, M., Krzyżak, A.: Adaptive regression estimation with multilayer feedforward neural networks. J. Nonparametr. Stat. 17(8), 891–913 (2005)
Kohler, M., Langer, S.: On the rate of convergence of fully connected deep neural network regression estimates. Ann. Stat. 49(4), 2231–2249 (2021)
LeCun, Y., Bengio, Y., Hinton, G.E.: Deep learning. Nature 521(7553), 436–444 (2015)
Ledoux, M., Talagrand, M.: Probability in Banach Spaces: Isoperimetry and Processes. Springer, Berlin (1991)
Lin, S.-B., Wang, K., Wang, Y., Zhou, D.-X.: Universal consistency of deep convolutional neural networks. IEEE Trans. Inf. Theory 68(7), 4610–4617 (2022)
Lu, J., Shen, Z., Yang, H., Zhang, S.: Deep network approximation for smooth functions. SIAM J. Math. Anal. 53(5), 5465–5506 (2021)
Ma, L., Siegel, J.W., Jinchao, X.: Uniform approximation rates and metric entropy of shallow neural networks. Res. Math. Sci. 9(3), 46 (2022)
Maiorov, V., Ratsaby, J.: On the degree of approximation by manifolds of finite pseudo-dimension. Constr. Approx. 15(2), 291–300 (1999)
Makovoz, Y.: Random approximants and neural networks. J. Approx. Theory 85(1), 98–109 (1996)
Mao, T., Shi, Z., Zhou, D.-X.: Approximating functions with multi-features by deep convolutional neural networks. Anal. Appl. 21(01), 93–125 (2023)
Mao, T., Zhou, D.-X.: Rates of approximation by ReLU shallow neural networks. J. Complex. 79, 101784 (2023)
Matoušek, J.: Improved upper bounds for approximation by zonotopes. Acta Math. 177(1), 55–73 (1996)
McCaffrey, D.F., Ronald, G.A.: Convergence rates for single hidden layer feedforward networks. Neural Netw. 7(1), 147–158 (1994)
Mhaskar, H.N.: Neural networks for optimal approximation of smooth and analytic functions. Neural Comput. 8(1), 164–177 (1996)
Mohri, M., Rostamizadeh, A., Talwalkar, A.: Foundations of Machine Learning. MIT Press, Cambridge (2018)
Nakada, R., Imaizumi, M.: Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. J. Mach. Learn. Res. 21(174), 1–38 (2020)
Neyshabur, B., Bhojanapalli, S., Srebro, N.: A PAC-Bayesian approach to spectrally-normalized margin bounds for neural networks. In: 6th International Conference on Learning Representations (2018)
Neyshabur, B., Tomioka, R., Srebro, N.: Norm-based capacity control in neural networks. In: Proceedings of the 28th Conference on Learning Theory, vol. 40, pp. 1376–1401. PMLR (2015)
Ongie, G., Willett, R., Soudry, D., Srebro, N.: a function space view of bounded norm infinite width ReLU nets: the multivariate case. In: 8th International Conference on Learning Representations (2020)
Parhi, R., Nowak, R.D.: What kinds of functions do deep neural networks learn? insights from variational spline theory. SIAM J. Math. Data Sci. 4(2), 464–489 (2022)
Parhi, R., Nowak, R.D.: Near-minimax optimal estimation with shallow ReLU neural networks. IEEE Trans. Inf. Theory 69(2), 1125–1140 (2023)
Petersen, P., Voigtlaender, F.: Equivalence of approximation by convolutional neural networks and fully-connected networks. Proc. Am. Math. Soc. 148(4), 1567–1581 (2020)
Pinkus, A.: Approximation theory of the MLP model in neural networks. Acta Numer. 8, 143–195 (1999)
Pisier, G.: Remarques sur un résultat non publié de B. Maurey. Séminaire d’Analyse fonctionnelle (dit“ Maurey-Schwartz”) pp. 1–12 (1981)
Rustamov, K.P.: On equivalence of different moduli of smoothness on the sphere. Trudy Matematicheskogo Instituta im V. A. Steklova 204, 274–304 (1993)
Savarese, P., Evron, I., Soudry, D., Srebro, N.: How do infinite width bounded norm networks look in function space? In: Proceedings of the 32nd Conference on Learning Theory, vol. 99, pp. 2667–2690. PMLR (2019)
Schmidt-Hieber, J.: Nonparametric regression using deep neural networks with ReLU activation function. Ann. Stat. 48(4), 1875–1897 (2020)
Shalev-Shwartz, S., Ben-David, S.: Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, Cambridge (2014)
Shen, Z., Yang, H., Zhang, S.: Deep network approximation characterized by number of neurons. Commun. Comput. Phys. 28(5), 1768–1811 (2020)
Siegel, J.W.: Optimal approximation of zonoids and uniform approximation by shallow neural networks (2023). arXiv:2307.15285
Siegel, J.W., Jinchao, X.: Approximation rates for neural networks with general activation functions. Neural Netw. 128, 313–321 (2020)
Siegel, J.W., Xu, J.: Sharp bounds on the approximation rates, metric entropy, and n-widths of shallow neural networks. Found. Comput. Math. (2022). https://doi.org/10.1007/s10208-022-09595-3
Siegel, J.W.: Characterization of the variation spaces corresponding to shallow neural networks. Construct. Approx. 57, 1109–1132 (2023)
Stone, C.J.: Optimal global rates of convergence for nonparametric regression. Ann. Stat. 10(4), 1040–1053 (1982)
Wainwright, M.J.: High-Dimensional Statistics: A Non-asymptotic Viewpoint, vol. 48. Cambridge University Press, Cambridge (2019)
Yang, Y., Barron, A.: Information-theoretic determination of minimax rates of convergence. Ann. Stat. 27(5), 1564–1599 (1999)
Yang, Y.: Learning distributions by generative adversarial networks: approximation and generalization. PhD thesis, The Hong Kong University of Science and Technology (2022)
Yang, Y., Li, Z., Wang, Y.: Approximation in shift-invariant spaces with deep ReLU neural networks. Neural Netw. 153, 269–281 (2022)
Yarotsky, D.: Error bounds for approximations with deep ReLU networks. Neural Netw. 94, 103–114 (2017)
Yarotsky, D.: Optimal approximation of continuous functions by very deep ReLU networks. In: Proceedings of the 31st Conference on Learning Theory, vol. 75, pp. 639–649. PMLR (2018)
Zhou, D.-X.: Theory of deep convolutional neural networks: downsampling. Neural Netw. 124, 319–327 (2020)
Zhou, D.-X.: Universality of deep convolutional neural networks. Appl. Comput. Harmon. Anal. 48(2), 787–794 (2020)
Zhou, T.-Y., Huo, X.: Learning ability of interpolating deep convolutional neural networks. Appl. Comput. Harmon. Anal. 68, 101582 (2024)
Acknowledgements
The work described in this paper was partially supported by InnoHK initiative, The Government of the HKSAR, Laboratory for AI-Powered Financial Technologies, the Research Grants Council of Hong Kong [Projects No. CityU 11306220 and 11308020] and National Natural Science Foundation of China [Project No. 12371103] when the second author worked at City University of Hong Kong. We thank the referees for their helpful comments and suggestions on the paper.
Funding
Open access publishing enabled by City University of Hong Kong Library’s agreement with Springer Nature.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Wolfgang Dahmen.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Yang, Y., Zhou, DX. Optimal Rates of Approximation by Shallow ReLU\(^k\) Neural Networks and Applications to Nonparametric Regression. Constr Approx (2024). https://doi.org/10.1007/s00365-024-09679-z
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s00365-024-09679-z