A unifying representer theorem for inverse problems and machine learning

The standard approach for dealing with the ill-posedness of the training problem in machine learning and/or the reconstruction of a signal from a limited number of measurements is regularization. The method is applicable whenever the problem is formulated as an optimization task. The standard strategy consists in augmenting the original cost functional by an energy that penalizes solutions with undesirable behavior. The effect of regularization is very well understood when the penalty involves a Hilbertian norm. Another popular configuration is the use of an $\ell_1$-norm (or some variant thereof) that favors sparse solutions. In this paper, we propose a higher-level formulation of regularization within the context of Banach spaces. We present a general representer theorem that characterizes the solutions of a remarkably broad class of optimization problems. We then use our theorem to retrieve a number of known results in the literature---e.g., the celebrated representer theorem of machine leaning for RKHS, Tikhonov regularization, representer theorems for sparsity promoting functionals, the recovery of spikes---as well as a few new ones.


Introduction.
A recurrent problem in science and engineering is the reconstruction of a multidimensional signal f : R d → R from a finite number of (possibly noisy) linear measurements y = (y m ) = ν(f ) ∈ R M where the operator ν = (ν m ) : f → ν(f ) = ( ν 1 , f , . . . , ν M , f ) symbolizes the linear measurement process. The machine learning version of the problem is the determination of a function f : R d → R from a finite number of samples y m = f (x m ) + ǫ m where ǫ m is small perturbation term; it is a special case of the former with ν m = δ(·−x m ). Since a function that takes values over the continuum is an infinite-dimensional entity, the reconstruction problem is inherently ill-posed. The standard remedy is to impose an additional minimum-energy requirement which, in effect, "regularizes" the solution. A natural choice of regularization is a "smoothness" norm associated with some function space X ′ (typically, a Sobolev space), which results in the prototypical formulation of the problem as arg min f ∈X ′ f X ′ s.t. ν m , f = y m , m = 1, . . . , M.
An alternative version that is better suited for noisy data is arg min with an adequate choice of hyper-parameters λ ∈ R + and p ∈ [1, ∞). We note that the second unconstrained form is a generalization of the first: the latter is recovered in the limit by taking λ → 0. The term "representer theorem" is typically used to designate a parametric formula-preferably, a linear expansion in terms of some basis functions-that spans the whole range of solutions of this type of problem, irrespective of the value of the data y ∈ R M . Such theorems are valued by practitioners because they indicate the way in which the initial problem can be recast as a finitedimensional optimization, making it amenable to standard numerical computations.
The other benefit is that, by describing the manifold of possible solutions, they provides us with a better understanding of the effect of regularization. The best known example is the representer theorem for reproducing kernel Hilbert spaces (RKHS), which states that the solution of (2) with ν m , f = f (x m ) and a Hilbertian regularization norm necessarily lives in a subspace of dimension M spanned by kernels centered on the data points x m [14,28,30,37]. This theorem, in its extended version [36], is the foundation for the majority of kernel-based methods for machine learning, including regression, radial basis functions and support-vector machines [38,21,41]. More recently, motivated by the success of ℓ 1 and total-variation regularization for compressed sensing [17,9,8], researchers have derived corresponding representer theorems in order to explain the sparsifying effect of such penalties and their robustness to missing data [24,45,25,6]. A representer theorem for measures has also been invoked to justify the use of the total-variation norm for the super-resolution localization of spikes [10,15,19,31] (see, Section 3.5 for details).
In this paper, we present a unifying treatment of regularization by considering the problem from the abstract perspective of optimization in Banach spaces. The first part (Section 2) is devoted to the theory. Our formulation heavily relies on the notion of Banach conjugates which is explained in Section 2.1. We then immediately proceed with the presentation of our key result: a generalized representer theorem (Theorem 5) that is valid for arbitrary convex data terms and Banach spaces in general, including the non-reflexive ones. The proof that is developed Section 2.2 is rather soft (or "high-level"), as it exclusively relies on the powerful machinery of duality mappings and the Hahn-Banach theorem-in other words, there is no need for Gâteaux derivatives nor subdifferentials, which are often invoked in such contexts. The resulting form of the solution in Theorem 5 is enlightening because it separates out the effect of the measurement operator from that of the regularization topology. Specifically, the functionals ν 1 , . . . , ν M specify a linear solution manifold that is then isometrically mapped into primary space via the conjugate map J : X → X ′ , which may or may not be linear, depending on wether the regularization norm is Hilbertian or not.
In the second part of the paper, we illustrate the power of the framework by using it to retrieve a number classical results: Schölkopf's generalized representer theorem for RKHS (Section 3.1), the closed-form solution of continuous-domain Tikhonov regularization with a Hilbertian norm (Section 3.2), as well as representer theorems for the non-reflexive spaces X ′ = ℓ 1 (Z d ) (Section 3.4) and X ′ = M(Ω)-the space of signed Radon measures on a compact domain-(Section 3.5) that are relevant to compressed sensing and super-resolution localization, respectively. In addition, we present a novel representer theorem for ℓ p -norm regularization (Section 3.3) as well as a representer theorem for generalized total-variation (Section 3.6)-in the spirit of [46]-that justifies the uses of sparse kernel expansions for machine learning in line with the generalized LASSO [33].

Banach spaces and duality mappings.
Definition 1. A normed vector space X is a linear space equipped with a norm, henceforth denoted by · X . It is called a Banach space if it is complete; that is, if every Cauchy sequence in (X , · X ) converges to an element of X . It is said to be strictly convex if, for all f 1 , f 2 ∈ X such that f 1 X = f 2 X = 1 and f 1 = f 2 , one has λf 1 + (1 − λ)f 2 X < 1 for any λ ∈ (0, 1). Finally, a Hilbert space is a Banach space whose norm is induced by an inner product.
We recall that X ′ is the space of linear functionals g : f → g, f △ = g(f ) ∈ R that are continuous on X . It is a Banach space equipped with the dual norm A direct implication of this definition is the generic duality bound for any f ∈ X , g ∈ X ′ . In fact, (3) can be interpreted as the Banach generalization of the Cauchy-Schwarz inequality for Hilbert spaces. By invoking the Hahn-Banach theorem, one can also prove that the duality bound is sharp for any dual pair (X , X ′ ) of Banach spaces [35]. This remarkable property inspired Beurling and Livingston to introduce the notion of duality mapping and to identify conditions of uniqueness [4]. We like to view the latter as the generalization of the classical Riesz map R : H ′ → H, or rather its inverse J = R −1 : H → H ′ that describes the isometric isomorphism between a Hilbert space H and its continuous dual H ′ [32]. The caveat with Banach spaces is that the duality mapping is not necessarily single-valued nor bijective.
Definition 2 (Duality mapping). Let (X , X ′ ) be a dual pair of Banach spaces. Then, the elements f * ∈ X ′ and f ∈ X form a conjugate pair if . For any given f ∈ X , the set of admissible conjugates defines the duality mapping which is a non-empty subset of X ′ . Whenever the duality mapping is single-valued (for instance, when X ′ is strictly convex), one also defines the duality operator J : X → X ′ , which is such that f * = J(f ).
We now list the properties of the duality mapping that are relevant for our purpose (see [4], [ Theorem 3 (Properties of duality mapping). Let (X , X ′ ) be a dual pair of Banach spaces. Then, the following holds: 1. Every f ∈ X admits at least one conjugate f * ∈ X ′ . 2. J(λf ) = λJ(f ) for any λ ∈ R (homogeneity). 3. For every f ∈ X , the set J(f ) is convex and weak * closed in X ′ . 4. The duality mapping is single-valued if X ′ is strictly convex; the latter condition is also necessary if X is reflexive. 5. When X is reflexive, then the duality map is bijective if and only if both X and X ′ are strictly convex.
The most favorable scenario is covered by Item 5. In that case, the duality map is invertible with f = (f * ) * = J −1 J(f ) in conformity with the property that X ′′ = X .
We now prove that the duality map is linear if and only if X = H is a Hilbert space. In that case, the unitary operator J : H → H ′ is precisely the inverse of the Riesz map R : H ′ → H. Proposition 4. Let (X , X ′ ) be a dual pair of Banach spaces such that X ′ is strictly convex. Then, the duality map J : X → X ′ : x → J(x) = x * is linear if and only if X is a Hilbert space.
Proof. First, we recall that all Hilbert spaces are strictly convex. Consequently, the indirect part of the statement is Riesz's celebrated representation theorem, which identifies the canonical linear isometry J = R −1 between a Hilbert space and its dual [35]. As for the converse implication, we show that the underlying inner product is Its bilinearity follows from the bilinearity of the duality product and the linearity of J, while the symmetry in x and y is obvious. Finally, the definition of the conjugate yields x, , which confirms that the bilinear form ·, · X is positive-definite, and hence the inner product that induces the · X -norm.
As an example, we provide the expression of the (unique) Banach conjugate This formula is intimately connected to Hölder's inequality. In particular, the L 2 conjugation map with p = q = 2 is an identity.

General representer theorem.
We now make use of the powerful tool of conjugation to characterize the solution of a broad class of unconstrained optimization problems in Banach space.
Theorem 5 (General Banach representer theorem). Let us consider the following setting: • (X , X ′ ) is a dual pair of Banach spaces.

{+∞} is a proper and strictly-convex loss functional.
• ψ : R + → R + is some arbitrary increasing convex function. Then, for any fixed y ∈ R M , the solution set of the generic optimization problem is non-empty, convex and weak * -compact and such that any We note that the condition of unicity requires the strict convexity of both ψ : R + → R and f → f X ′ . This applies to Banach spaces such as with ψ(t) = λ|t| p strictly convex. While the solution of (4) also exists for Banach spaces such as M( , the uniqueness is usually lost in such non-reflexive scenarios. Proof. The proof uses standard arguments in convex analysis together with a dual reformulation of the problem inspired from the interpretation of best interpolation given by Carl de Boor in [13]. (i) Existence and reformulation as a generalized interpolation problem. First, we recall that the basic properties of (weak lower semi-) continuity, (strict) convexity and coercivity 1 are preserved though functional composition. The functional f → f X ′ is convex, (norm-)continuous and coercice on X ′ from the definition of a norm. Since ψ : R + → R + is convex and its domain is finite-dimensional, it is necessarily continuous and coercive. This ensures that f → ψ ( f X ′ ) is endowed with the same three basic properties. The linear measurement operator ν : X ′ → R N is continuous on X ′ by assumption (i.e. ν m ∈ X ⇒ ν m ∈ X ′′ because of the canonical embedding of a Banach space it its bidual) and trivially convex. The final ingredient is the (strict) convexity of E : R M × R M → R, which implies the lower semi-continuity of z → E y, z on R d (since it is proper) and hence, by composition, the convexity and lower semi continuity is (weakly) lower semi-continuous, convex, and coercive on X ′ , which guarantees the existence of the solution (as well as the convexity and closedness of the solution set) by a standard argument in convex analysis [20]. Likewise, the unicity is ensured when ψ ( f X ′ ) is strictly convex, in which case f → F (f ) is strictly convex as well.
For the general (not necessarily unique) scenario, we take advantage of the strict convexity of E : R M × R M → R + ∪ {+∞} to show that all minimizers of F (f ) share a common measurement vector z 0 = ν(f 0 ) ∈ R M (the argument is that the existence of distinct z's would contradict the assumption of strict convexity). Although z 0 = ν(f 0 ) ∈ R M is usually not known before hand, this property provides us with a convenient parametric characterization of the solution set as where z ranges over R M .
(ii) Explicit resolution of the generalized interpolation problem (5). For Problem (5) to be well-defined for any z ∈ R M , we need the functionals ν m to be linearly independent. (If this is not the case, we simply reduce the set accordingly.) This ensures that any ν ∈ N ν has a unique expansion ν = M m=1 a m ν m . Based on this representation, we define the linear functional Moreover, the Hahn-Banach theorem ensures the existence of a continuous, norm-preserving extension of λ to the whole Banach space X ; that is, an element f 0 ∈ X ′ such that The connection between the above statement and the generalized interpolation problem (5) is that the complete set of continuous extensions of λ to X ⊃ N ν is given by with the property that The next fundamental observation is that N ν = N ′ ν ′ because both spaces are of finite dimension N 0 and hence reflexive. Consequently, for any we have ν 0 X = λ and λ(ν 0 ) = ν 0 2 X , as well as ν 0 X = f 0 X ′ for all f 0 ∈ S z 0 because of (6). Since f 0 ∈ U ⊂ X ′ and ν 0 ∈ N ν ⊂ X , this yields We have just shown that S z 0 ⊆ J(ν 0 ) for any extremal element ν 0 ∈ {g ∈ N ν : λ(g) = λ g X , g X = λ }. In the case where the duality mapping J(λ) is multivalued, there is little hope that the inclusion goes the other way around because one generally has that J(ν 0 ) = J(ν ′ 0 ) for ν 0 = ν ′ 0 . However, we suspect that S z 0 = ν0∈J(λ) J(ν 0 ) . Finally, we easily deduce that S z 0 is weak * -compact since it is included in the closed ball in X ′ of radius f 0 X ′ < ∞, which is itself weak * -compact (by the Banach-Alaoglu theorem).
When X ′ is strictly convex, the situation is simpler because the duality mapping from X to X ′ is single-valued and the solution f 0 ∈ X ′ unique. Likewise, ν 0 -the Banach conjugate of λ in (N ν , · X )-is unique if the · X -norm is strictly convex. Whenever X is reflexive, this ensures that there is bijection between f 0 and ν 0 with f 0 = ν * 0 and ν 0 = f * 0 = (ν * 0 ) * . Moreover, the latter conjugate map is linear if and only if X is a Hilbert space by Proposition 4.
Note that the existence of the conjugate of ν 0 ∈ N ν ⊂ X is essential to the argumentation. This is the reason why the problem is formulated with f ∈ X ′ subject to the hypothesis that ν 1 , . . . , ν M ∈ X (weak * continuity). These considerations are inconsequential in the simpler reflexive scenario where the role of the two spaces is interchangeable since X = X ′′ . The hypothesis of linear independence of the ν m in Theorem 5 is only made for convenience: it can be dropped as explained in the proof, which then leads to a corresponding reduction in the number of degrees of freedom (M ) of the solution.
3.1. Kernel/RKHS methods in machine learning. Here, the search space X ′ is a reproducing kernel Hilbert space on R d denoted by H with f 2 H = f, f H where ·, · H is the underlying inner product. The predual space is X = H ′′ = H (reflexive scenario). The RKHS property [3] is equivalent to the existence of a (unique) positive definite kernel h : R d × R d → R (the reproducing kernel of H) such that for all f ∈ H and any x m ∈ R d .
In the context of machine learning, the loss function E is usually chosen to be additive: E(y, z) = M m=1 E m y m , z m [37,26]. Given a series of data points x m , y m , m = 1, . . . , M with x m ∈ R d , the learning problem is then to estimate a function where λ ∈ R + is an adjustable regularization parameter. In functional terms, the reproducing kernel represents the Schwartz kernel [40] of the Riesz map R : . The application of Theorem 5 with X ′ = H then immediately yields the parametric form of the solution which is a linear kernel expansion. The optimality of such kernel expansions is precisely the result stated in Schölkopf's representer theorem for RKHS [36]. Moreover, by invoking the reproducing kernel property (7) with f = h(·, x n ) ∈ H, one readily finds that f 0 2 H = a T Ga where the Gram matrix G ∈ R M×M is specified by [G] m,n = h(x m , x n ). By re-injecting the parametric form of the solution into the cost functional in (8), we then end up with the equivalent finite-dimensional minimization task which yields the exact solution of the original infinite-dimensional optimization problem. In short, (10) is the optimal dicretization of the functional optimization problem (9), which is then readily transcribable into a numerical implementation using standard (finite-dimensional) techniques.

Tikhonov regularization.
Tikhonov regularization is a classical approach for dealing with ill-posed linear inverse problems [43,27]. The goal there is to recover a function f : R d → R from a noisy or imprecise series of linear measurements y m = ν m , f + ǫ m where ǫ m is the disturbance term. By using the same functional framework as in Section 3.1 with ν 1 , · · · , ν M ∈ H ′ = X and X ′ = H ′′ = H, one formulates the recovery problem as The application of Theorem 5 then yields the parametric form of the solution with ϕ m = R{ν m } where R is the Riesz map H ′ = X → H = X ′ . The next fundamental observation is that the bilinear form (ν m , ν n ) → ν m , R{ν n } is actually the inner product for the dual space H ′ ; that is, ν m , ϕ n = ν m , ν n H ′ . In fact, by using the property that ν m and ϕ m = ν * m are Hilbert conjugates, we have that ν m , ϕ n = ν m , ν n H ′ = ν * m , ν * n H = ϕ m , ϕ n H , which, somewhat remarkably, shows that the underlying system matrix is equal to the Gram matrix of the basis {ϕ m }.
Therefore, by re-injecting (12) into the cost functional in (11), we are able to reformulate the initial optimization problem as a finite-dimensional minimisation under the implicit assumption that H is invertible. We note that the latter is equivalent to the linear independence of the ϕ m (resp., the linear independence of the ν m due to the Riesz pairing).
3.3. Towards compressed sensing: ℓ p -norm regularization. A classical problem in signal processing is to recover an unknown discrete signal s = (s n ) ∈ R N from a set of corrupted linear measurements y m = h T m s+ǫ m , m = 1, . . . , M . The measurements vectors h 1 , . . . , h M ∈ R N specify the system matrix H = [h 1 h 2 · · · h M ] T ∈ R M×N . When M (the number of measurements) is less than N (the size of the unknown signal s), the reconstruction problem is a priori ill-posed and even much more so when M ≪ N (compressed sensing scenario). However, if the original signal is known to be sparse-i.e., s 0 ≤ K 0 with K 0 < 2M -and the system matrix H satisfies some "incoherence" properties, then the theory of compressed sensing provides general guarantees for a stable recovery [24,9,17]. The computational strategy then is to impose an ℓ p regularization (with p small to favor sparsity) on the solution and to formulate the reconstruction problem as . The standard choice for CS is p = 1, which is the smallest exponent that still results in a convex optimization problem.
We now show how we can use Theorem 5 to characterize the effect of such a regularization for p ∈ (1, ∞). The corresponding Banach space is X ′ = (R N , · ℓp ) whose predual is X = (R N , · ℓq ) with 1 p + 1 q = 1. Moreover, the underlying norms are strictly convex for p > 1 which guarantees that the solution is unique, irrespective of M and H. By introducing the dual signal ν 0 = H T a ∈ X and by using the known form of the corresponding Banach q-to-p duality map J : X → X ′ , we then readily deduce that the solution can be represented as for a suitable value of the (dual) parameter vector a ∈ R M . While the exact value of a is data dependent, the above formula provides us with the description of the solution manifold of intrinsic dimension M . Another way to put it is that the fact that s minimizes (14) induces a non-linear pairing between the data vector y ∈ R M and the dual variable a ∈ R M in (15). In particular, for p = 2, we have that s = H T a = M m=1 h m a m , which confirms the well-known result that s ∈ span{h m }. The latter also explains why classical quadratic/Tikhonov regularization performs poorly when M is much smaller than N .
3.4. Sparsity-promoting regularization. The limit case 2 of the previous scenario is p = 1 (CS) for which the norm is no longer strictly convex. To deal with such cases where the solution is potentially non-unique, we first recall the Krein-Milman theorem [35, p. 75] which allows us to describe the weak * -compact solution set S in Theorem 5 as the convex hull of its extreme points. We then invoke a recent result by Boyer et al. that yields the following characterization of the extremal points of Problem (4).
Theorem 6. All extremal points f 0 of the solution set S of Problem (4) can be expressed as a k e k for some 1 ≤ K 0 ≤ M where the e k are some extremal points of the unit "regularization" ball B X ′ = {f ∈ X ′ : f X ′ ≤ 1} and a = (a 1 , · · · , a K0 ) ∈ R K0 is a vector of appropriate weights.
The above is a direct corollary of [6, Theorem 1 with j = 0] applied to an extreme point of the equivalent generalized interpolation problem (5).
The characterization in Theorem 6 is highly relevant for the scenario X ′ = ℓ 1 (Z) whose extreme vectors are intrinsically sparse; i.e, e k = (±δ[n−n k ]) n∈Z for some fixed offset n k ∈ Z. Here, δ[·] denotes the Kronecker impulse which is such that δ[0] = 1 and δ[n] = 0 for n = 0. Hence, the outcome is that the use of the ℓ 1 penalty (e.g., (14) with p = 1) has a tendency to induce sparse solutions with f 0 = K 0 ≤ M , which is more or less the flavor of the representer theorem(s) in [45].
It should be pointed out, however, that the result in Theorem 6 is not particularly informative for strictly-convex spaces such as ℓ p (Z) or L p (R d ) with p ∈ (1, ∞) for which all unit vectors-i.e., e ∈ X ′ with e X ′ = 1-are extremal points of the unit ball. Indeed, since the corresponding solution is unique (by Theorem 5), we trivially have that f 0 = f 0 X ′ e 1 with K 0 = 1 and e 1 = f 0 / f 0 X ′ .  f, ϕ < +∞} 2 Our analysis is not applicable to p < 1 because the corresponding metric does no longer fulfil the properties of a norm; in other words, ℓp(Z) fails to be a Banach space for p ∈ (0, 1).
is the Banach space of signed Radon measures on Ω (by the Riesz-Markov representation theorem [34]). Moreover, it is well known that the extreme points of the unit ball in M(Ω) are point measures (a.k.a. Dirac impulses) of the form e k = ±δ(· − x k ) for some x k ∈ Ω with the property that for any ϕ ∈ C(Ω). For any given series of (independent) analysis functions ν 1 , . . . , ν M ∈ C(Ω) (e.g., Fourier exponentials), we can therefore invoke Theorems 5 and 6 with X ′ = M(Ω) to deduce that the extreme points of the problem are inherently sparse. This means that there necessarily exists some minimizer(s) of the form with K 0 ≤ M , a 1 , . . . , a K0 ∈ R K0 and x 1 , . . . , x K0 ∈ Ω. The fact that Problem (16) admits a global solution whose representation is given by (17) is a result that can be traced back to the work of Fisher and Jerome in [23,Theorem 1]. This optimality result is the foundation for a recent variational method for super-resolution localization that was investigated by a number of authors [7,10,22]. Besides the development of grid-free optimization schemes, researchers have worked out the conditions on x k and ν m (analysis functionals) under which the minimization (16) can provide a perfect recovery of spike trains of the form given by (17) with a small K 0 [11,15,31]. The remarkable finding is that there are many configurations for which super-resolution recovery is guaranteed with the accuracy only being dependent on the signal-to-noise ratio and the minimal spacing between neighboring spikes.
3.6. Sparse kernel expansions. Schwartz's space of smooth and rapidly decaying functions on R d is denoted by S(R d ). Its continuous dual is S ′ (R d ): the space of tempered distributions. In this section, L : We also assume that the generalized impulse response of L −1 is a bivariate function of slow growth h : R d × R d → R. In other words, the inverse operator L −1 has the explicit integral representation for any ϕ ∈ S(R d ). In conformity with the nomenclature of [46], the native Banach space for L, M(R d ) is which is isometrically isomorphic to M(R d ) (the space of Radon measures on R d ). This is to say that the operators L, L −1 have restrictions L : − − → M(R d ) and − − → M L (R d ) that are isometries. Consequently, we can apply Theorem 5 to deduce that the generic learning problem arg min admits a solution, albeit not necessarily a unique one since the underlying search space M L (R d )-or, equivalent, the parent space M(R d )-is neither reflexive nor strictly convex.
In order to refine the above statement with the help of Theorem 6, we first observe that the extreme points of the unit ball in M(R d ) are of the form e k = ±δ(· − τ k ) with τ k ∈ R d , which is consistent with the result in Section 3.5 for M(Ω). Since the map L −1 : is isometric, this allows us to identify the extreme points of the unit ball in M L (R d ) as where h : R d × R d → R is the kernel of the operator in (18). Consequently, we can invoke Theorem 6 to prove that the extreme points of Problem (19) are all expressible as with parameters K 0 ≤ M , τ 1 , . . . , τ K0 ∈ R d and a = (a k ) ∈ R K0 . Moreover, since L{h(·, τ k )} = δ(· − τ k ) and δ(· − τ k ) M = e k M = 1, the optimal regularization cost is Lf 0 M = K0 k=1 |a k | = a ℓ1 , which makes an interesting connection with ℓ 1norm minimization and the generalized LASSO [42,33]. To sum up, the solution (20) has a kernel expansion that is similar to (9) with the important twist that the kernel centers τ k are adaptive, meaning that their location as well as their cardinality K 0 is data-dependent. In effect, it is the underlying ℓ 1 -norm penalty that helps reducing the number K 0 of active kernels, thereby producing a sparse solution.
When L : where the operator F −1 : The overarching message in the above optimality result is that the choice of the regularization operator L in (19) predetermines the parametric form of the kernel in (20). Now, in light of (21), we can also turn the argument around by specifying a desired form of kernel h LSI : R d → R and inferring the frequency response of the corresponding regularization operator Now, the necessary and sufficient condition for the continuity of L : S ′ (R d ) → S ′ (R d ) is that the function L : R d → R be smooth and slowly growing (see [40]). A parametric class of kernels that meets this admissibility requirement is the super-exponential family h LSI (x) = exp (− x α ) with α ∈ (0, 2). The limit case with α = 2 (Gaussian) is excluded because the corresponding frequency response in (22) (inverse of a Gaussian) is not slowly increasing anymore.

Conclusion.
The main point of this paper has been to show that the general issue of regularization can be investigated through a common (abstract) umbrella, with the main result being expressed by Theorem 6, which is valid for Banach spaces in general. We have shown that the fundamental ingredient in the quest for a representer theorem is the identification and characterization of a corresponding dual pair of Banach spaces. The proposed formulation provides a definite answer for the reflexive and strictly convex scenario-in which case the solution is also known to be unique-whenever the duality mapping is known (e.g., for the classical L p (R d ) or ℓ p (Z) spaces with p ∈ (1, ∞)). While it also offers interesting insights for certain non-strictly-convex and sparsity promoting norms such as · ℓ1 and its continuousdomain counterpart-the total variation · M and generalization thereof-it raises many additional questions concerning the potential unicity of such solutions and the necessity to develop some corresponding numerical optimization schemes.
We have made the link with the existing literature in machine learning (regression) and the resolution of ill-posed inverse problems by considering several concrete cases, including RKHS, and compressed sensing. The conciseness and self-containedness of the proposed derivations is a good indication of the power of the approach. Since the concept of Banach space is remarkably general, one can easily conceive of other variations around the common theme of regularization and representer theorems. Potential topics for further research include the use of non-standard norms, the deployment of hybrid regularization schemes, vector-valued functions or feature maps [1], and the consideration of direct-sum spaces and semi-norms, as in the theory of splines [14,18,47,16,29,46]. In short, there is ample room for additional theoretical and practical investigations, in direct analogy with what has been accomplished during the past few decades in the simpler-but more restrictive-context of reproducing kernel Hilbert spaces [2,1]. Interestingly, there also appears to be a link with deep neural/kernel networks, as has been demonstrated recently [5,44].