A fractional Hadamard formula and applications

Abstract

We derive a shape derivative formula for the family of principal Dirichlet eigenvalues $\lambda _s(\Omega )$ of the fractional Laplacian $(-\Delta )^s$ associated with bounded open sets $\Omega \subset \mathbb {R}^N$ of class $C^{1,1}$. This extends, with a help of a new approach, a result in Dalibard and Gérard-Varet (Calc. Var. 19(4):976–1013, 2013) which was restricted to the case $s=\frac{1}{2}$. As an application, we consider the maximization problem for $\lambda _s(\Omega )$ among annular-shaped domains of fixed volume of the type $B\setminus \overline{B}'$, where B is a fixed ball and $B'$ is ball whose position is varied within B. We prove that $\lambda _s(B\setminus \overline{B}')$ is maximal when the two balls are concentric. Our approach also allows to derive similar results for the fractional torsional rigidity. More generally, we will characterize one-sided shape derivatives for best constants of a family of subcritical fractional Sobolev embeddings.

1 Introduction

Let $s\in (0,1)$ and $\Omega \subset \mathbb {R}^N$ be a bounded open set. The present paper is devoted to the study of best constants $\lambda _{s,p}(\Omega )$ in the family of subcritical Sobolev inequalities

$$\begin{aligned} \lambda _{s,p}(\Omega ) \Vert u\Vert _{L^p(\Omega )}^2 \le [u]_{s}^2 \qquad \text {for all }u \in {{\mathcal {H}}}^s_0(\Omega ), \end{aligned}$$

(1.1)

where $p\in [1, \frac{2N}{N-2s})$ if $2s<N$ and $p\in [1, \infty )$ if $2s\ge N=1$. Here, the Sobolev space ${{\mathcal {H}}}^s_0(\Omega )$ is given as completion of $C^\infty _c(\Omega )$ with respect to the norm $[\,\cdot \,]_{s}$ defined by

$$\begin{aligned}{}[u]_{s}^2= \frac{ c_{N,s}}{2}\int _{\mathbb {R}^{N}}\! \int _{\mathbb {R}^{N}}\!\frac{(u(x)-u(y))^2}{|x-y|^{N+2s}}dx dy \quad \qquad \text {with} \quad c_{N,s}= \pi ^{-\frac{N}{2}}s 4^s\frac{\Gamma (\frac{N}{2}+s)}{\Gamma (1-s)}.\nonumber \\ \end{aligned}$$

(1.2)

The normalization constant $c_{N,s}$ is chosen such that $[u]_{s}^2=\int _{\mathbb {R}^N}|\xi |^{2s}|\hat{u}(\xi )|^2d\xi$ for $u \in {{\mathcal {H}}}^s_0(\Omega )$, where $\hat{u}$ denotes the Fourier transform of u. The best (i.e., largest possible) constant in (1.1) is given by

$$\begin{aligned} \lambda _{s,p}(\Omega ):=\inf \left\{ [u]_s^2\,:\, u\in {{\mathcal {H}}}^s_0(\Omega ),\quad \Vert u\Vert _{L^p(\Omega )}=1 \right\} . \end{aligned}$$

(1.3)

As a consequence of the subcriticality assumption on p and the boundedness of $\Omega$, the space ${{\mathcal {H}}}^s_0(\Omega )$ compactly embeds into $L^p(\Omega )$. Therefore a direct minimization argument shows that $\lambda _{s,p}(\Omega )$ admits a nonnegative minimizer $u\in {{\mathcal {H}}}^s_0(\Omega )$ with $\Vert u\Vert _{L^p(\Omega )}=1$. Moreover, every such minimizer solves, in the weak sense, the semilinear problem

$$\begin{aligned} (-\Delta )^su = \lambda _{s,p}(\Omega )u^{p-1}\quad \text { in }\Omega ,\qquad \quad u = 0\quad \text { in }\mathbb {R}^N\setminus \Omega . \end{aligned}$$

(1.4)

where $(-\Delta )^s$ stands for the fractional Laplacian. It therefore follows from regularity theory and the strong maximum principle for $(-\Delta )^s$ that u is strictly positive in $\Omega$, see Lemma 2.3 below. We recall that, for functions $\varphi \in C^{1,1}_c(\mathbb {R}^N)$, the fractional Laplacian is given by

$$\begin{aligned} (-\Delta )^s\varphi (x)=c_{N,s}\,PV \!\!\int _{\mathbb {R}^N} \frac{\varphi (x)-\varphi (y)}{|x-y|^{N+2s}}\, dy=\frac{c_{N,s}}{2}\int _{\mathbb {R}^N}\frac{2\varphi (x)-\varphi (x+y)-\varphi (x-y)}{|y|^{N+2s}}\, dy. \end{aligned}$$

Of particular interest are the cases $p=1$ and $p=2$ which correspond to the fractional torsion problem

$$\begin{aligned} (-\Delta )^su = \lambda _{s,1}(\Omega )\quad \text { in }\Omega ,\qquad \quad u = 0\quad \text { in }\mathbb {R}^N\setminus \Omega , \end{aligned}$$

(1.5)

and the eigenvalue problem

$$\begin{aligned} (-\Delta )^su = \lambda _{s,2}(\Omega )u\quad \text { in }\Omega ,\qquad \quad u = 0\quad \text { in }\mathbb {R}^N\setminus \Omega , \end{aligned}$$

(1.6)

associated with the first Dirichlet eigenvalue of the fractional Laplacian, respectively. In these cases, the minimization problem for $\lambda _{s,p}(\Omega )$ in (1.3) possesses a unique positive minimizer. Indeed, it is a well-known consequence of the fractional maximum principle that (1.5) admits a unique solution, and that (1.6) has a unique positive eigenfunction with $\Vert u\Vert _{L^2(\Omega )}=1$. Incidentally, the uniqueness of positive minimizers extends to the full range $1 \le p \le 2$, as we shall show in Lemma A.1 in the appendix of this paper.

Our first goal in this paper is to analyze the dependence of the best constants on the underlying domain $\Omega$. For this we shall derive a formula for a one-sided shape derivative of the map $\Omega \mapsto \lambda _{s,p}(\Omega )$. We assume from now on that $\Omega \subset \mathbb {R}^N$ is a bounded open set of class $C^{1,1}$, and we consider a family of deformations $\{\Phi _\varepsilon \}_{\varepsilon \in (-1,1)}$ with the following properties:

$$\begin{aligned} \begin{aligned}&\Phi _\varepsilon \in C^{1,1}(\mathbb {R}^N; \mathbb {R}^N) \text { for }\varepsilon \in (-1,1), \Phi _0= {{\,\mathrm{id}\,}}_{\mathbb {R}^N}, \text {and}\\&\text {the map }(-1,1) \rightarrow C^{0,1}(\mathbb {R}^N,\mathbb {R}^N), \varepsilon \rightarrow \Phi _\varepsilon \text {is of class }C^2. \end{aligned} \end{aligned}$$

(1.7)

We note that (1.7) implies that $\Phi _\varepsilon :\mathbb {R}^N\rightarrow \mathbb {R}^N$ is a global diffeomorphism if $|\varepsilon |$ is small enough, see e.g. [7, Chapter 4.1]. To clarify, we stress that we only need the $C^2$-dependence of $\Phi _\varepsilon$ on $\varepsilon$ with respect to Lipschitz-norms, while $\Phi _\varepsilon$ is assumed to be a $C^{1,1}$-function for $\varepsilon \in (-1,1)$ to guarantee $C^{1,1}$-regularity of the perturbed domains $\Phi _\varepsilon (\Omega )$.

From the variational characterization of $\lambda _{s,p}(\Omega )$ it is not difficult to see that the map $\varepsilon \mapsto \lambda _{s,p}(\Phi _\varepsilon (\Omega ))$ is continuous. However, since $\lambda _{s,p}(\Omega )$ may not have a unique positive minimizer, we cannot expect this map to be differentiable. We therefore rely on determining the right derivative of $\varepsilon \mapsto \lambda _{s,p}(\Phi _\varepsilon (\Omega ))$ from which we derive differentiability whenever $\lambda _{s,p}(\Omega )$ admits a unique positive minimizer, thereby extending the classical Hadamard shape derivative formula for the first Dirichlet eigenvalue of the Laplacian $-\Delta$.

Throughout this paper, we consider a fixed function $\delta \in C^{1,1}(\mathbb {R}^N)$ which coincides with the signed distance function $\mathrm{dist}(\cdot ,\mathbb {R}^{N}\setminus \Omega )-\mathrm{dist}(\cdot , \Omega )$ in a neighborhood of the boundary $\partial \Omega$. We note here that, since we assume that $\Omega$ is of class $C^{1,1}$, the signed distance function is also of class $C^{1,1}$ in a neighborhood of $\partial \Omega$ but not globally on $\mathbb {R}^N$. We also suppose that $\delta$ is chosen with the property that $\delta$ is positive in $\Omega$ and negative in $\mathbb {R}^N\setminus \overline{\Omega }$, as it is the case for the signed distance function.

Our first main result is the following.

Theorem 1.1

Let $\lambda _{s,p}(\Omega )$ be given by (1.3) and consider a family of deformations $\Phi _\varepsilon$ satisfying (1.7). Then the map $\varepsilon \mapsto \theta (\varepsilon ):=\lambda _{s,p}(\Phi _\varepsilon (\Omega ))$ is right differentiable at $\varepsilon =0$. Moreover,

$$\begin{aligned} \partial _+\theta (0)=\min \left\{ \Gamma (1+s)^2 \int _{\partial \Omega }(u/\delta ^s)^2 X \cdot \nu \,dx\,:\, u\in \mathcal {H}\right\} , \end{aligned}$$

(1.8)

where $\nu$ denotes the interior unit normal on $\partial \Omega$, $\mathcal {H}$ is the set of positive minimizers for $\lambda _{s,p}(\Omega )$ and $X := \partial _\varepsilon \big |_{\varepsilon =0} \Phi _\varepsilon$.

Here the function $u/\delta ^s$ is defined on $\partial \Omega$ as a limit. Namely, for $x_0\in \partial \Omega$, the limit

$$\begin{aligned} \frac{u}{\delta ^s}(x_0)=\lim _{{\mathop {x\in \Omega }\limits ^{x\rightarrow x_0 }}}\frac{u}{\delta ^s}(x) \end{aligned}$$

(1.9)

exists, as the function $u/\delta ^s$ extends to a function in $C^\alpha (\overline{\Omega })$ for some $\alpha >0$, see [20]. In addition, the function $\delta ^{1-s}\nabla u$ also admits a Hölder continuous extension on $\overline{\Omega }$ satisfying $\delta ^{1-s}\nabla u\cdot \nu =s u/\delta ^s$ on $\partial \Omega$, see [8]. As a consequence, the expression $u/\delta ^s$, restricted on $\partial \Omega$, plays the role of an inner fractional normal derivative. Note that, for $s = 1$, the limit on the RHS of (1.9) coincides with the classical inner normal derivative of u at $x_0$.

We observe that the constant $\Gamma (1+s)^2$ appears also in the fractional Pohozaev identity, see e.g. [21]. This is, to some extend, not surprising at least in the classical case since Pohozaev’s identity can be obtained using techniques of domain variation, see e.g. [23].

We also remark that one-sided derivatives naturally arise in the analysis of parameter-dependent minimization problems, see e.g. [7, Section 10.2.3] for an abstract result in this direction. Related to this, they also appear in the analysis of the domain dependence of eigenvalue problems with possible degeneracy, see e.g. [11] and the references therein.

A natural consequence of Theorem 1.1 is that the map $\varepsilon \mapsto \theta (\varepsilon )=\lambda _{s,p}(\Phi _\varepsilon (\Omega ))$ is differentiable at $\varepsilon = 0$ whenever $\lambda _{s,p}(\Omega )$ admits a unique positive minimizer. Indeed, applying Theorem 1.1 to the map $\varepsilon \mapsto \widetilde{\theta }(\varepsilon ):=\lambda _{s,p}(\Phi _{-\varepsilon }(\Omega ))$ yields

$$\begin{aligned} \partial _-\theta (0)=-\partial _+\widetilde{\theta }(0)=\max \left\{ \Gamma (1+s)^2 \int _{\partial \Omega }(u/\delta ^s)^2 X\cdot \nu \,dx\,:\, u\in \mathcal {H}\right\} , \end{aligned}$$

where ${{\mathcal {H}}}$ is given as in Theorem 1.1. As a consequence, we obtain the following result.

Corollary 1.2

Let $\lambda _{s,p}(\Omega )$ be given by (1.3) and consider a family of deformations $\Phi _\varepsilon$ satisfying (1.7). Suppose that $\lambda _{s,p}(\Omega )$ admits a unique positive minimizer $u\in {{\mathcal {H}}}^s_0(\Omega )$. Then the map $\varepsilon \mapsto \theta (\varepsilon )=\lambda _{s,p}(\Phi _\varepsilon (\Omega ))$ is differentiable at $\varepsilon =0$. Moreover

$$\begin{aligned} \theta '(0)=\Gamma (1+s)^2 \int _{\partial \Omega }(u/\delta ^s)^2 X\cdot \nu \,dx, \end{aligned}$$

(1.10)

where $X:=\partial _\varepsilon \big |_{\varepsilon =0}\Phi _\varepsilon$.

As mentioned earlier, $\lambda _{s,p}(\Omega )$ admits a unique positive minimizer $u\in {{\mathcal {H}}}^s_0(\Omega )$ for $1 \le p \le 2$, see Lemma A.1 in the appendix. Therefore Corollary 1.2 extends, in particular, the classical Hadamard formula, for the first Dirichlet eigenvalue $\lambda _{1,2}(\Omega )$ of $-\Delta$, to the fractional setting. We recall, see e.g. [15], that the classical Hadamard formula is given by

$$\begin{aligned} \frac{d}{d\varepsilon }\Big |_{\varepsilon = 0}\lambda _{1,2}(\Phi _\varepsilon (\Omega )) = \int _{\partial \Omega } |\nabla u|^2 X\cdot \nu \,dx. \end{aligned}$$

(1.11)

An analogue of Corollary 1.2 for the case of the local r-Laplace operator was obtained in [2, 12]. We also point out that, prior to this paper, a Hadamard formula in the fractional setting of the type (1.10) was obtained in [6] for the special case $p=1$, $s= \frac{1}{2}$, $N = 2$ and $\Omega$ of class $C^\infty$. We are not aware of any other previous work related to Theorem 1.1 or 1.2 in the fractional setting.

Our next result provides a characterization of constrained local minima of $\lambda _{s,p}$. Here and in the following, we call a bounded open subset $\Omega$ of class $C^{1,1}$ a constrained local minimum for $\lambda _{s,p}$ if for all families of deformations $\Phi _\varepsilon$ satisfying (1.7) and the volume invariance condition $|\Phi _\varepsilon (\Omega )|= |\Omega |$ for $\varepsilon \in (-1,1)$, there exists $\varepsilon _0 \in (0,1)$ with $\lambda _{s,p}(\Phi _\varepsilon (\Omega )) \ge \lambda _{s,p}(\Omega )$ for $\varepsilon \in (-\varepsilon _0,\varepsilon _0)$. Our classification result reads as follows.

Corollary 1.3

Let $p\in \{1\}\cup [2,\infty )$. If an open subset $\Omega$ of $\mathbb {R}^N$ of class $C^{3}$ is a volume constrained local minimum for $\Omega \mapsto \lambda _{s,p}(\Omega )$, then $\Omega$ is a ball.

Corollary 1.3 is a consequence of Theorem 1.1, from which we derive that if $\Omega$ is a constrained local minimum then any element $u\in \mathcal {H}$ satisfies the overdetermined condition $u/\delta ^s\equiv constant$ on $\partial \Omega$. Therefore by the rigidity result in [9] we find that $\Omega$ must be a ball. We point out that we are not able to include the case $p \in (1,2)$ in Corollary 1.3, since the rigidity result in [9] is based on the moving plane method and therefore requires the nonlinearity in (1.4) to be Lipschitz. The case $p \in (1,2)$ therefore remains an open problem in Corollary 1.3.

We note that the authors in [6] considered the shape minimization problem for $\lambda _{s,p}(\Omega )$ in the case $p=1$, $s= \frac{1}{2}$, $N = 2$ among domains $\Omega$ of class $C^\infty$ of fixed volume. They showed in [6] that such minimizers are discs.

Next we consider the optimization problem of $\Omega \mapsto \lambda _{s,p}(\Omega )$ for $p\in \{1,2\}$ and $\Omega$ a punctured ball, with the hole having the shape of ball. We show that, as the hole moves in $\Omega$ then $\lambda _{s,p}(\Omega )$ is maximal when the two balls are concentric. In the local case $s=1$ and $N=2$, this is a classical result by Hersch [16]. For subsequent generalizations in the case of the local problem, see [5, 14, 18].

Theorem 1.4

Let $p\in \{1,2\}$, $B_1(0)$ be the unit centered ball and $\tau \in (0,1)$. Define

$$\begin{aligned} {{\mathcal {A}}}:=\{a\in B_{1}(0)\,:\, B_\tau (a) \subset B_1(0)\}. \end{aligned}$$

Then the map ${{\mathcal {A}}}\rightarrow \mathbb {R}$, $\,a\mapsto \lambda _{s,p}(B_1(0)\setminus \overline{ B_\tau (a)})$ takes its maximum at $a=0$.

The proof of Theorem 1.4 is inspired by the argument given in [14, 18] for the local case $s=1$. It uses the fractional Hadamard formula in Corollary 1.2 and maximum principles for anti-symmetric functions. Our proof also shows that the map $a\mapsto \lambda _{s,p}(B_1(0)\setminus \overline{ B_\tau (a)})$ takes its minimum when the boundary of the ball $B_{\tau }(a)$ touches the one of $B_{1}(0)$, see Sect. 5 below.

The proof of Theorem 1.1 is based on the use of test functions in the variational characterization of $\lambda _{s,p}(\Omega )$ and $\lambda _{s,p}(\Phi _\varepsilon (\Omega ))$. The general strategy is inspired by the direct approach in [11], which is related to a Neumann eigenvalue problem on manifolds. In the case of $\lambda _{s,p}(\Phi _\varepsilon (\Omega ))$, it is important to make a change of variables so that $\lambda _{s,p}(\Phi _\varepsilon (\Omega ))$ is determined by minimizing an $\varepsilon$-dependent family of seminorms among functions $u\in {{\mathcal {H}}}^s_0(\Omega )$, see Sect. 2 below. An obvious choice of test functions are minimizers u and $v_\varepsilon$ for $\lambda _{s,p}( \Omega )$ and $\lambda _{s,p}(\Phi _\varepsilon (\Omega ))$, respectively. However, due to the fact that u is only of class $C^s$ up to the boundary, we cannot obtain a boundary integral term directly from the divergence theorem. In particular, the integration by parts formula given in [21, Theorem 1.9] does not apply to general vector fields X which appear in (1.8). Hence, we need to replace u with $\zeta _k u$, where $\zeta _k$ is a cut-off function vanishing in a $\frac{1}{k}$-neighborhood of $\partial \Omega$. This leads to upper and lower estimates of $\lambda _{s,p}(\Phi _\varepsilon (\Omega ))$ up to order $o(\varepsilon )$, where the first order term is given by an integral involving $(-\Delta )^s(\zeta _k u)$ and $\nabla (\zeta _k u)$. We refer the reader to Sect. 4 below for more precise information. A highly nontrivial task is now to pass to the limit as $k\rightarrow \infty$ in order to get boundary integrals involving $\psi := u/\delta ^s$. This is the most difficult part of the paper. We refer to Proposition 2.4 and Sect. 6 below for more details.

The paper is organized as follows. In Sect. 2, we provide preliminary results on convergence properties of integral functional, inner approximations of functions in ${{\mathcal {H}}}^s_0(\Omega )$ and on properties of minimizers of (1.3). In Sect. 3, we introduce notation related to domain deformations and related quantities. In Sect. 4 we establish a preliminary variant of Theorem 1.1, which is given in Proposition 4.1. In this variant, the constant $\Gamma (1+s)^2$ in (1.8) is replaced by an implicitly given value which still depends on cut-off data. The proofs of the main results, as stated in this introduction, are then completed in Sect. 5. Finally, Sect. 6 is devoted to the proof of the main technical ingredient of the paper, which is given by Proposition 2.4.

2 Notations and preliminary results

Throughout this section, we fix a bounded open set $\Omega \subset \mathbb {R}^N$. As noted in the introduction, we define the space ${{\mathcal {H}}}^s_0(\Omega )$ as completion of $C^\infty _c(\Omega )$ with respect to the norm $[\,\cdot \,]_{s}$ given in (1.2). Then ${{\mathcal {H}}}^s_0(\Omega )$ is a Hilbert space with scalar product

$$\begin{aligned} (u,v) \mapsto [u,v]_{s}= \frac{ c_{N,s}}{2}\int _{\mathbb {R}^{N}}\! \int _{\mathbb {R}^{N}}\!\frac{(u(x)-u(y))(v(x)-v(y))}{|x-y|^{N+2s}}dx dy, \end{aligned}$$

where $c_{N,s}$ is given in (1.2). It is well known and easy to see that ${{\mathcal {H}}}^s_0(\Omega )$ coincides with the closure of $C^\infty _c(\Omega )$ in the standard fractional Sobolev space $H^s(\mathbb {R}^N)$. Moreover, if $\Omega$ has a continuous boundary, then ${{\mathcal {H}}}^s_0(\Omega )$ admits the highly useful characterization

$$\begin{aligned} {{\mathcal {H}}}^s_0(\Omega )= \bigl \{ w \in L^1_{loc}(\mathbb {R}^N)\,:\, [w]_s^2 < \infty ,\quad w \equiv 0\; \text {on }\mathbb {R}^N \setminus \Omega \bigr \}, \end{aligned}$$

(2.1)

see e.g. [13, Theorem 1.4.2.2]. We start with an elementary but useful observation.

Lemma 2.1

Let $\mu \in L^\infty (\mathbb {R}^N\times \mathbb {R}^N)$, and let $(v_k)_k$ be a sequence in ${{\mathcal {H}}}^s_0(\Omega )$ with $v_k \rightarrow v$ in ${{\mathcal {H}}}^s_0(\Omega )$ as $k \rightarrow \infty$. Then we have

$$\begin{aligned} \lim _{k\rightarrow \infty }\int _{\mathbb {R}^{2N}} \frac{(v_k(x)-v_k(y))^2\mu (x,y)}{|x-y|^{N+2s}} dxdy = \int _{\mathbb {R}^{2N}} \frac{(v(x)-v(y))^2\mu (x,y)}{|x-y|^{N+2s}} dxdy. \\ \end{aligned}$$

Proof

We have

$$\begin{aligned}&\Bigl |\int _{\mathbb {R}^{2N}} \frac{(v_k(x)-v_k(y))^2-(v(x)-v(y))^2\mu (x,y)}{|x-y|^{N+2s}} dxdy\Bigr |\\&\quad \le \Vert \mu \Vert _{L^\infty } \int _{\mathbb {R}^{2N}} \frac{|(v_k(x)-v_k(y))^2-(v(x)-v(y))^2|}{|x-y|^{N+2s}} dxdy, \end{aligned}$$

where

$$\begin{aligned}&\int _{\mathbb {R}^{2N}} \frac{|(v_k(x)-v_k(y))^2-(v(x)-v(y))^2|}{|x-y|^{N+2s}} dxdy\\&\quad =\int _{\mathbb {R}^{2N}} \frac{|[(v_k(x)-v(x))-(v_k(y)-v(y))][(v_k(x)+v(x))-(v_k(y)+v(y))]|}{|x-y|^{N+2s}} dxdy\\&\quad \le \frac{2}{c_{N,s}}[v_k-v]_s [v_k+v]_s\, \rightarrow \, 0 \qquad \text {as } k \rightarrow \infty . \end{aligned}$$

$\square$

Throughout the remainder of this paper, we fix $\rho \in C^\infty _c(-2,2)$ with $0 \le \rho \le 1$, $\rho \equiv 1$ on $(-1,1)$, and we define

$$\begin{aligned} \zeta \in C^\infty (\mathbb {R}),\qquad \zeta (t)=1-\rho (t). \end{aligned}$$

(2.2)

Moreover, for $k \in \mathbb {N}$, we define the functions

$$\begin{aligned} \rho _k,\, \zeta _k\, \in \, C^{1,1}(\mathbb {R}^N), \quad \qquad \rho _k(x)= \rho (k \delta (x)), \;\quad \zeta _k(x)= \zeta (k \delta (x)). \end{aligned}$$

(2.3)

We note that the function $\rho _k$ is supported in the $\frac{2}{k}$-neighborhood of the boundary, while the function $\zeta _k$ vanishes in the $\frac{1}{k}$-neighborhood of the boundary.

Lemma 2.2

Let $\Omega \subset \mathbb {R}^N$ be a bounded Lipschitz domain and let $u \in {{\mathcal {H}}}^s_0(\Omega )$. Moreover, for $k \in \mathbb {N}$, let $u_k := u \zeta _k \in {{\mathcal {H}}}^s_0(\Omega )$ denote inner approximations of u. Then we have

$$\begin{aligned} u_k \rightarrow u \qquad \text {in }{{\mathcal {H}}}^s_0(\Omega ). \end{aligned}$$

Proof

In the following, the letter $C>0$ stands for various constants independent of k. Since $\rho _k= 1 -\zeta _k$, it suffices to show that

$$\begin{aligned} u \rho _k \in {{\mathcal {H}}}^s_0(\Omega )\; \text {for }k \text {sufficiently large}\qquad \text {and}\qquad [u \rho _k]_s \rightarrow 0 \quad \text {as }k \rightarrow \infty . \end{aligned}$$

(2.4)

For $\varepsilon >0$, we put $A_\varepsilon = \{x \in \Omega \,:\, \delta (x)< \varepsilon \}$. Since $u \rho _k$ vanishes in $\mathbb {R}^N \setminus A_{\frac{2}{k}}$, $0 \le \rho _k \le 1$ on $\mathbb {R}^N$ and $|{\rho _k}(x)-{\rho _k}(y)| \le C \min \{k |x-y|,1\}$ for $x,y \in \mathbb {R}^N$, we observe that

$$\begin{aligned}&\frac{1}{c_{N,s}} [{\rho _k} u]^2_s =\frac{1}{2}\int _{\mathbb {R}^N}\int _{\mathbb {R}^N} \frac{[u(x){\rho _k}(x)-u(y){\rho _k}(y)]^2}{|x-y|^{N+2s}} dydx \nonumber \\&\quad = \frac{1}{2}\int _{A_{\frac{4}{k}}}\int _{A_{\frac{4}{k}}} \frac{[u(x){\rho _k}(x)-u(y){\rho _k}(y)]^2}{|x-y|^{N+2s}}\ dydx\nonumber \\&\quad \quad +\int _{A_{{\frac{2}{k}}}} u(x)^2 {\rho _k}(x)^2 \int _{\mathbb {R}^N \setminus A_{\frac{4}{k}}} |x-y|^{-N-2s}\ dydx\nonumber \\&\quad \le \frac{1}{2}\int _{A_{\frac{4}{k}}}\int _{A_{\frac{4}{k}}} \frac{\bigl [u(x)\bigl ({\rho _k}(x)-{\rho _k}(y)\bigr )+ {\rho _k}(y)\bigl (u(x)-u(y)\bigr )\bigr ]^2}{|x-y|^{N+2s}}\ dydx\nonumber \\&\quad \quad + C \int _{A_{{\frac{2}{k}}}} u(x)^2 \mathrm{dist}(x,\mathbb {R}^N \setminus A_{\frac{4}{k}})^{-2s} dx \nonumber \\&\quad \le \int _{A_{\frac{4}{k}}} u^2(x) \int _{A_{\frac{4}{k}}}\frac{({\rho _k}(x)-{\rho _k}(y))^2}{|x-y|^{N+2s}}\ dydx + \int _{A_{\frac{4}{k}}}\int _{A_{\frac{4}{k}}}\frac{(u(x)-u(y))^2}{|x-y|^{N+2s}}dydx \nonumber \\&\quad \quad + C\int _{A_{{\frac{2}{k}}}} u(x)^2 \delta ^{-2s}(x)dx\nonumber \\&\quad \le C k^2 \int _{A_{\frac{4}{k}}} u^2(x)\int _{B_{\frac{1}{k}}(x)} |x-y|^{2-2s-N} dydx \nonumber \\&\quad \quad + C \int _{A_{\frac{4}{k}}} u^2(x)\int _{\mathbb {R}^N \setminus B_{\frac{1}{k}}(x)} |x-y|^{-N-2s} dydx\nonumber \\&\quad \quad + \int _{A_{\frac{4}{k}}}\int _{A_{\frac{4}{k}}}\frac{(u(x)-u(y))^2}{|x-y|^{N+2s}}dydx + C \int _{A_{{\frac{2}{k}}}} u(x)^2 \delta ^{-2s}(x) dx \nonumber \\&\quad \le C k^{2s} \int _{A_{\frac{4}{k}}} u^2(x)dx + \int _{A_{\frac{4}{k}}}\int _{A_{\frac{4}{k}}}\frac{(u(x)-u(y))^2}{|x-y|^{N+2s}}dydx + C \int _{A_{{\frac{2}{k}}}} u(x)^2 \delta ^{-2s}(x) dx\nonumber \\&\quad \le C \int _{A_{\frac{4}{k}}} u^2(x)\delta ^{-2s}(x)dx + \int _{A_{\frac{4}{k}}}\int _{A_{\frac{4}{k}}}\frac{(u(x)-u(y))^2}{|x-y|^{N+2s}}dydx. \end{aligned}$$

(2.5)

Now, since $\Omega$ has a Lipschitz boundary, using $\int _{\mathbb {R}^N \setminus \Omega }|x-y|^{-N-2s}\,dy\sim \delta ^{-2s}(x)$ see e.g [3], we get

$$\begin{aligned} \int _{\Omega } u^2(x)\delta ^{-2s}(x)dx \le C \int _{\Omega }u^2(x) \int _{\mathbb {R}^N \setminus \Omega }|x-y|^{-N-2s}\,dy dx \le C[u]_s^2, \end{aligned}$$

and therefore

$$\begin{aligned} \int _{A_{\frac{4}{k}}} u^2(x)\delta ^{-2s}(x)dx \rightarrow 0 \qquad \text {as } k \rightarrow \infty . \end{aligned}$$

(2.6)

Moreover, since also

$$\begin{aligned} \int _{\Omega } \int _{\Omega }\frac{(u(x)-u(y))^2}{|x-y|^{N+2s}}dydx \le \frac{2}{c_{N,s}}[u]_s^2, \end{aligned}$$

we have

$$\begin{aligned} \int _{A_{\frac{4}{k}}}\int _{A_{\frac{4}{k}}}\frac{(u(x)-u(y))^2}{|x-y|^{N+2s}}dydx \rightarrow 0 \qquad \text {as }k \rightarrow \infty . \end{aligned}$$

(2.7)

Combining (2.5)–(2.7), we obtain (2.4), as required. $\square$

From now on, we fix a bounded $C^{1,1}$-domain $\Omega \subset \mathbb {R}^N$. We also let

$$\begin{aligned} C^s_0(\overline{\Omega }) = \left\{ w\in C^s(\overline{\Omega }): w = 0\quad \text {in}\quad \mathbb {R}^N\setminus \Omega \right\} , \end{aligned}$$

and we recall the following regularity and positivity properties of nonnegative minimizers for $\lambda _{s,p}(\Omega )$ as defined in (1.3).

Lemma 2.3

Let $u \in {{\mathcal {H}}}_0^s(\Omega )$ be a nonnegative minimizer for $\lambda _{s,p}(\Omega )$. Then $u\in C^s_0(\overline{\Omega }) \cap C^\infty _{loc}(\Omega )$. Moreover, $\psi :=\frac{u}{\delta ^s} \in C^{\alpha }(\overline{\Omega })$ for some $\alpha \in (0,1)$, and there exists a constant $c=c(N,s,\Omega ,\alpha ,p)>0$ with the property that

$$\begin{aligned} \Vert \psi \Vert _{C^{\alpha }(\overline{\Omega })}\le c \end{aligned}$$

(2.8)

and

$$\begin{aligned} | \nabla \psi (x) | \le c \delta ^{\alpha -1}(x) \qquad {\text { for all }}x\in \Omega . \end{aligned}$$

(2.9)

Moreover, $\psi >0$ on $\overline{\Omega }$, so in particular $u>0$ in $\Omega$.

Proof

By standard arguments in the calculus of variations, u is a weak solution of (1.4). By [22, Proposition 1.3] we have that $u\in L^\infty (\Omega )$, and therefore the RHS of (1.4) is a function in $L^\infty (\Omega )$. Thus the regularity up to the boundary $u \in C^s_0(\overline{\Omega })$ is proved in [20], where also the $C^\alpha$-bound (2.8) for the function $\psi = \frac{u}{\delta ^s}$ is established for some $\alpha >0$. Moreover, (2.9) is proved in [8]. It also follows from (1.4), the strong maximum principle and the Hopf lemma for the fractional Laplacian that $\psi$ is a strictly positive function on $\overline{\Omega }$. In particular, $u>0$ in $\Omega$, Therefore $u \in C^\infty _{loc}(\Omega )$ follows by interior regularity theory (see e.g. [19]) and the fact that the function $t \mapsto t^{p-1}$ is of class $C^\infty$ on $(0,\infty )$. $\square$

The computation of one-sided shape derivatives as given in Theorem 1.1 will be carried out in Sect. 4, and it requires the following key technical proposition. Since its proof is long and quite involved, we postpone the proof to Sect. 6 below.

Proposition 2.4

Let $X \in C^{0}(\overline{\Omega },\mathbb {R}^N)$, let $u \in C^s_0(\overline{\Omega })\cap C^1(\Omega )$, and assume that $\psi :=\frac{u}{\delta ^s}$ extends to a function on $\overline{\Omega }$ satisfying (2.8) and (2.9). Moreover, put $U_k:=u \zeta _k \in C^{1,1}_c(\Omega )$, where $\zeta _k$ is defined in (2.3). Then

$$\begin{aligned} \lim _{k\rightarrow \infty }\int _{\Omega }\nabla U_k \cdot X \Bigl ( u (-\Delta )^s\zeta _k -I(u,\zeta _k)\Bigr )\,dx= -\kappa _s \int _{\partial \Omega } \psi ^2 X\cdot \nu \, dx, \end{aligned}$$

where

$$\begin{aligned} \kappa _s:= - \int _{\mathbb {R}}h'(r) (-\Delta )^sh(r)\, dr \qquad \text {with}\quad h(r):=r_+^s \zeta (r)=\max (r,0)^s\zeta (r) \end{aligned}$$

(2.10)

and $\zeta$ given in (2.2), and where we use the notation

$$\begin{aligned} I(u,v)(x):= c_{N,s}\int _{\mathbb {R}^N}\frac{(u(x)-u(y))(v(x)-v(y))}{|x-y|^{N+2s}}\, dy \end{aligned}$$

(2.11)

for $u\in C^s_c(\mathbb {R}^N)$, $v\in C^{0,1}(\mathbb {R}^N)$ and $x \in \mathbb {R}^N$.

Remark 2.5

The minus sign in the definition of the constant $\kappa _s$ in (2.10) might appear a bit strange at first glance. We shall see later that, defined in this way, $\kappa _s$ has a positive value. A priori it is not clear that the value of $\kappa _s$ does not depend on the particular choice of the function $\zeta$. This follows a posteriori once we have established in Proposition 4.1 below that this constant appears in Theorem 1.1. This will then allow us to show that $\kappa _s = \frac{\Gamma (1+s)^2}{2}$ by applying the resulting shape derivative formula to a one-parameter family of concentric balls, see Sect. 5 below. A more direct, but somewhat lengthy computation of $\kappa _s$ is possible via the logarithmic Laplacian, which has been introduced in [4].

3 Domain perturbation and the associated variational problem

Here and in the following, we define $\Omega _\varepsilon : = \Phi _\varepsilon (\Omega )$. In order to study the dependence of $\lambda _{s,p}(\Omega _\varepsilon )$ on $\varepsilon$, it is convenient to pull back the problem on the fixed domain $\Omega$ via a change of variables. For this we let $\text {Jac}_{\Phi _\varepsilon }$ denote the Jacobian determinant of the map $\Phi _\varepsilon \in C^{1,1}(\mathbb {R}^N)$, and we define the kernels

$$\begin{aligned} K_\varepsilon (x,y):= { c_{N,s}}\frac{\text {Jac}_{\Phi _\varepsilon }(x)\text {Jac}_{\Phi _\varepsilon }(y)}{|\Phi _\varepsilon (x)-\Phi _\varepsilon (y)|^{N+2s}}\qquad \text {and} \qquad K_0(x,y)= { c_{N,s}}\frac{1}{|x-y|^{N+2s}}. \end{aligned}$$

(3.1)

Then (1.7) gives rise to the well known expansions

$$\begin{aligned} \text {Jac}_{\Phi _\varepsilon }(x)=1+\varepsilon \text {div}X(x)+ O(\varepsilon ^2),\qquad \partial _\varepsilon \text {Jac}_{\Phi _\varepsilon }(x) = \text {div}X(x)+ O(\varepsilon ) \end{aligned}$$

(3.2)

uniformly in $x \in \mathbb {R}^N$, where $X:=\partial _\varepsilon \big |_{\varepsilon =0} \Phi _\varepsilon \in C^{0,1}(\mathbb {R}^N;\mathbb {R}^N)$ and therefore ${\text {div}}X$ is a.e. defined on $\mathbb {R}^N$. From (1.7), we also get

$$\begin{aligned} {|\Phi _\varepsilon (x)-\Phi _\varepsilon (y)|^{-N-2s}}=|x-y|^{-N-2s}\left( 1+ 2\varepsilon \frac{x-y}{|x-y|} \cdot P_{X}(x,y)+O(\varepsilon ^2) \right) ^{-\frac{N+2s}{2}}, \end{aligned}$$

and

$$\begin{aligned} \partial _\varepsilon {|\Phi _\varepsilon (x)-\Phi _\varepsilon (y)|^{-N-2s}}=|x-y|^{-N-2s}\left( - (N+2s) \frac{x-y}{|x-y|} \cdot P_{X}(x,y)+O(\varepsilon )\right) , \end{aligned}$$

uniformly in $x,y \in \mathbb {R}^N$, $x \not = y$ with

$$\begin{aligned} P_X \in L^\infty (\mathbb {R}^N \times \mathbb {R}^N), \qquad P_{X}(x,y)=\frac{X(x)-X(y)}{|x-y|}. \end{aligned}$$

Moreover by (3.2) and the fact that $\partial _\varepsilon \Phi _\varepsilon$, $X\in C^{0,1}(\mathbb {R}^N)$, we have that

$$\begin{aligned} K_\varepsilon (x,y)=K_0(x,y)+\varepsilon \partial _\varepsilon \Big |_{\varepsilon =0} K_\varepsilon (x,y)+O(\varepsilon ^2) K_0(x,y) , \end{aligned}$$

(3.3)

and

$$\begin{aligned} \partial _\varepsilon K_\varepsilon (x,y)=\partial _\varepsilon \Big |_{\varepsilon =0}K_\varepsilon (x,y) +O(\varepsilon )K_0(x,y), \end{aligned}$$

(3.4)

uniformly in $x,y \in \mathbb {R}^N$, $x \not = y$, where

$$\begin{aligned} \partial _\varepsilon \Big |_{\varepsilon =0} K_\varepsilon (x,y)=- \Bigl [(N+2s)\frac{x-y}{|x-y|} \cdot P_{X}(x,y)-&(\text {div}X(x)+ \text {div}X(y))\Bigr ] K_0(x,y). \end{aligned}$$

(3.5)

In particular, it follows from (3.3) and (3.5) that there exist $\varepsilon _0,C>0$ with the property that

$$\begin{aligned} \frac{1}{C}K_0(x,y)\le K_\varepsilon (x,y)\le C K_0(x,y) \qquad \text { for all }x, y\in \mathbb {R}^N, x \not =y \text { and }\varepsilon \in (-\varepsilon _0,\varepsilon _0).\nonumber \\ \end{aligned}$$

(3.6)

For $v\in {{\mathcal {H}}}^s_0(\Omega )$ and $\varepsilon \in (-\varepsilon _0,\varepsilon _0)$, we now define

$$\begin{aligned} {{\mathcal {V}}}_{v}(\varepsilon ):= \frac{1}{2}\int _{\mathbb {R}^{2N}} {(v(x)-v(y))^2} K_\varepsilon (x,y) dxdy . \end{aligned}$$

(3.7)

Then, by (1.3), (1.7) and a change of variables, we have the following variational characterization for $\lambda _{s,p}(\Omega _\varepsilon )$:

$$\begin{aligned} \lambda _{s,p}^\varepsilon&:= \lambda _{s,p}(\Omega _\varepsilon )=\inf \left\{ [u]_s^2\,:\, u\in {{\mathcal {H}}}^s_0(\Omega _\varepsilon ), \quad \int _{\Omega _\varepsilon } |u|^p\, dx=1 \right\} \nonumber \\&=\inf \left\{ {{\mathcal {V}}}_{v}(\varepsilon ) \,:\, v\in {{\mathcal {H}}}^s_0(\Omega ),\quad \int _{\Omega } |v|^p\text {Jac}_{\Phi _\varepsilon }(x)\, dx=1 \right\} \quad \text {for }\varepsilon \in (-\varepsilon _0,\varepsilon _0). \end{aligned}$$

(3.8)

As mentioned earlier, we prefer to use (3.8) from now on where the underlying domain is fixed and the integral terms depend on $\varepsilon$ instead. It follows from (3.3) and (3.4) that, for given $v \in {{\mathcal {H}}}^s_0(\Omega )$, the function ${{\mathcal {V}}}_v: (-\varepsilon _0,\varepsilon _0) \rightarrow \mathbb {R}$ is of class $C^1$ with

$$\begin{aligned} {{\mathcal {V}}}_v'(0) = \frac{1}{2}\int _{\mathbb {R}^{2N}} {(v(x)-v(y))^2} \partial _\varepsilon \big |_{\varepsilon =0} K_\varepsilon (x,y) dxdy, \end{aligned}$$

(3.9)

where $\partial _\varepsilon \big |_{\varepsilon =0} K_\varepsilon (x,y)$ is given in (3.5),

$$\begin{aligned} |{{\mathcal {V}}}_v'(0)| \le C [v]_s^2 \qquad \text {with a constant }C >0 \end{aligned}$$

(3.10)

and we have the expansions

$$\begin{aligned} {{\mathcal {V}}}_v(\varepsilon ) = {{\mathcal {V}}}_v(0)+ \varepsilon {{\mathcal {V}}}_v'(0) + O(\varepsilon ^2)[v]_s^2, \qquad {{\mathcal {V}}}_v'(\varepsilon ) = {{\mathcal {V}}}_v'(0) + O(\varepsilon )[v]_s^2 \end{aligned}$$

(3.11)

with $O(\varepsilon )$, $O(\varepsilon ^2)$ independent of v. From (3.2), (3.6) and the variational characterization (3.8), it is easy to see that

$$\begin{aligned} \frac{1}{C} \le \lambda _{s,p}^\varepsilon \le C \qquad \text {for all }\varepsilon \in (-\varepsilon _0,\varepsilon _0) \text {with some constant }C>0. \end{aligned}$$

Using this and (3.2), (3.6) once more, we can show that

$$\begin{aligned} \frac{1}{C}\le \Vert v_\varepsilon \Vert _{L^p(\Omega )}\le C\qquad \text {and}\qquad \frac{1}{C} \le [v_\varepsilon ]_s\le C . \end{aligned}$$

(3.12)

for every $\varepsilon \in (-\varepsilon _0,\varepsilon _0)$ and every minimizer $v_\varepsilon \in {{\mathcal {H}}}^s_0(\Omega )$ for (3.8) with a constant $C>0$.

The following lemma is essentially a corollary of Lemma 2.1.

Lemma 3.1

Let $(v_k)_k$ be a sequence in ${{\mathcal {H}}}^s_0(\Omega )$ with $v_k \rightarrow v$ in ${{\mathcal {H}}}^s_0(\Omega )$. Then we have

$$\begin{aligned} \lim _{k\rightarrow \infty }{{\mathcal {V}}}_{v_{k}}(0)= {{\mathcal {V}}}_{v}(0)\qquad \text {and}\qquad \lim _{k\rightarrow \infty }{{\mathcal {V}}}_{v_k}'(0)= {{\mathcal {V}}}_{v}'(0). \end{aligned}$$

Proof

The first limit is trivial since ${{\mathcal {V}}}_v(0)= [v]_s^2$ for $v \in {{\mathcal {H}}}^s_0(\Omega )$. The second limit follows from Lemma 2.1, (3.5) and (3.9) by noting that $\mu \in L^\infty (\mathbb {R}^N \times \mathbb {R}^N)$ for the function

$$\begin{aligned} \mu (x,y)=- (N+2s) \frac{x-y}{|x-y|} \cdot P_{X}(x,y) + (\text {div}X(x)+ \text {div}X(y)). \end{aligned}$$

$\square$

4 One-sided shape derivative computations

We keep using the notation of the previous sections, and we recall in particular the variational characterization of $\lambda _{s,p}^\varepsilon =\lambda _{s,p}(\Omega _\varepsilon )$ given in (3.8). The aim of this section is to prove the following result.

Proposition 4.1

We have

$$\begin{aligned} \partial _\varepsilon ^+ \Big |_{\varepsilon =0}\lambda _{s,p}^\varepsilon =\min&\left\{ 2 \kappa _s \int _{\partial \Omega } (u/\delta ^s)^2X\cdot \nu \, dx\,:\, u\in \mathcal {H}\right\} , \end{aligned}$$

where $\mathcal {H}$ is the set of positive minimizers for $\lambda _{s,p}^0:= \lambda _{s,p}(\Omega )$, $X:=\partial _\varepsilon \big |_{\varepsilon =0} \Phi _\varepsilon$ and $\kappa _s$ is given by (2.10).

The proof of Proposition 4.1 requires several preliminary results. We start with a formula for the derivative of the function given by (3.7).

Lemma 4.2

Let $U \in C^{1,1}_c(\Omega )$. Then

$$\begin{aligned}&{{\mathcal {V}}}_{U}'(0)= -2 \int _{\mathbb {R}^N} \nabla U \cdot X (-\Delta )^s U dx. \end{aligned}$$

(4.1)

Proof

By (3.5), (3.11) and Fubini’s theorem, we have

$$\begin{aligned} {{\mathcal {V}}}_{U}'(0)&= \frac{-(N+2s) c_{N,s}}{2}\int _{\mathbb {R}^{2N}} {(U(x)-U(y))^2}\frac{(x-y)\cdot (X(x)-X(y))}{|x-y|^{N+2s+2}} dx dy \nonumber \\&+ \frac{1}{2}\int _{\mathbb {R}^{2N}} {(U(x)-U(y))^2} K_0(x,y) (\text {div}X(x)+ \text {div}X(y))dxdy\\&= \frac{-(N+2s) c_{N,s}}{2} \lim _{\mu \rightarrow 0} \int _{|x-y|>\mu } {(U(x)-U(y))^2}\frac{(x-y)\cdot (X(x)-X(y))}{|x-y|^{N+2s+2}} dxdy \nonumber \\&+ \int _{\mathbb {R}^{2N}} {(U(x)-U(y))^2} K_0(x,y)\text {div}X(x) dxdy\\&=-(N+2s) c_{N,s} \lim _{\mu \rightarrow 0} \int _{\mathbb {R}^N} \int _{\mathbb {R}^N \setminus \overline{B_\mu (y)}} {(U(x)-U(y))^2}\frac{(x-y)\cdot X(x)}{|x-y|^{N+2s+2}} dxdy \nonumber \\&+ \int _{\mathbb {R}^{2N}} {(U(x)-U(y))^2} K_0(x,y)\text {div}X(x) dxdy \end{aligned}$$

Applying, for fixed $y \in \mathbb {R}^N$ and $\mu >0$, the divergence theorem in the domain $\{x \in \mathbb {R}^{N}\,:\, |x-y| > \mu \}$ and using that $\nabla _{x} |x-y|^{-N-2s}=-(N+2s)\frac{x-y}{|x-y|^{N+2s+2}}$, we obtain

$$\begin{aligned} {{\mathcal {V}}}_{U}'(0)&=c_{N,s} \lim _{\mu \rightarrow 0} \int _{\mathbb {R}^N} \int _{\mathbb {R}^N \setminus \overline{B_\mu (y)}} {(U(x)-U(y))^2} \nabla _{x} |x-y|^{-N-2s} \cdot X(x) dxdy \nonumber \\&+ \int _{\mathbb {R}^{2N}} {(U(x)-U(y))^2} K_0(x,y)\text {div}X(x) dxdy \nonumber \\&=- \lim _{\mu \rightarrow 0} \int _{\mathbb {R}^N} \int _{\mathbb {R}^N \setminus \overline{B_\mu (y)}} {(U(x)-U(y))^2} K_0(x,y) \text {div}X(x)dxdy \nonumber \\&- \lim _{\mu \rightarrow 0}\int _{\mathbb {R}^N} \int _{\mathbb {R}^N \setminus \overline{B_\mu (y)}} {(U(x)-U(y))\nabla U(x) \cdot X(x) } K_0(x,y) dxdy \nonumber \\&+ \lim _{\mu \rightarrow 0} \int _{\mathbb {R}^N} \int _{\partial B_\mu (y)} {(U(x)-U(y))^2} \frac{y-x}{|x-y|}\cdot X(x) K_0(x,y) \, d\sigma (y)\,dx \nonumber \\&+ \int _{\mathbb {R}^{2N}} {(U(x)-U(y))^2} K_0(x,y) \text {div}X(x)dxdy\nonumber \\&= - \lim _{\mu \rightarrow 0}\int _{|x-y|>\mu } (U(x)-U(y))\nabla U(x) \cdot X(x) K_0(x,y) d(x,y) \nonumber \\&+ \lim _{\mu \rightarrow 0}\mu ^{-N-1-2s} \int _{|x-y|=\mu } {(U(x)-U(y))^2} (y-x) \cdot X(x) \, d\sigma (x,y)\nonumber \\&=- \frac{c_{N,s}}{2} \lim _{\mu \rightarrow 0}\int _{\mathbb {R}^N}\nabla U(x) \cdot X(x) \int _{\mathbb {R}^N \setminus \overline{B_\mu (0)}} \frac{2 U(x)-U(x+z)-U(x-z)}{|z|^{N+2s}} dz dx \nonumber \\&+ \frac{1}{2} \lim _{\mu \rightarrow 0}\mu ^{-N-1-2s} \int _{|x-y|= \mu } {(U(x)-U(y))^2} (y-x) \cdot (X(x)-X(y)) \, d\sigma (x,y) \end{aligned}$$

(4.2)

Since $U \in C^{1,1}_c(\Omega )$, we have that

$$\begin{aligned}&\frac{c_{N,s}}{2} \lim _{\mu \rightarrow 0}\int _{\mathbb {R}^N} \nabla U(x) \cdot X(x) \int _{\mathbb {R}^N \setminus \overline{B_\mu (0)}} \frac{2 U(x)-U(x+z)-U(x-z)}{|z|^{N+2s}} dz dx\nonumber \\&\quad = \frac{c_{N,s}}{2} \int _{\mathbb {R}^N}\nabla U(x) \cdot X(x) \int _{\mathbb {R}^N} \frac{2 U(x)-U(x+z)-U(x-z)}{|z|^{N+2s}} dz dx\nonumber \\&\quad = \int _{\mathbb {R}^N} (-\Delta )^sU(x) \nabla U(x) \cdot X(x) dx. \end{aligned}$$

(4.3)

Moreover, since U is compactly supported, we may fix $R>0$ large enough such that $(U(x)-U(y))^2 = 0$ for all $x,y \in B_R(0)$ with $|x-y|<1$. Setting $N_\mu := \{(x,y) \in B_R(0) \times B_R(0) \,:\, |x-y|= \mu \}$ for $0<\mu <1$ and using that $U, X \in C^{0,1}(\mathbb {R}^N)$, we thus deduce that

$$\begin{aligned}&\mu ^{-N-1-2s} \int _{|x-y|= \mu } {(U(x)-U(y))^2} (y-x) \cdot (X(x)-X(y)) \, d\sigma (x,y)\nonumber \\&\quad = \mu ^{-N-1-2s} \int _{N_\mu } {(U(x)-U(y))^2} (y-x) \cdot (X(x)-X(y))\, d\sigma (x,y) = O(\mu ^{3-1-2s}) \rightarrow 0, \end{aligned}$$

(4.4)

as $\mu \rightarrow 0$, since the $2N-1$-dimensional measure of the set $N_\mu$ is of order $O(N-1)$ as $\mu \rightarrow 0$. The claim now follows by combining (4.2)–(4.4). $\square$

We cannot apply Lemma 4.2 directly to minimizers $u \in {{\mathcal {H}}}^s_0(\Omega )$ of $\lambda _{s,p}(\Omega )$ since these are not contained in $C^{1,1}_c(\Omega )$. The aim is therefore to apply Lemma 4.2 to $U_k:=u \zeta _k \in C^{1,1}_c(\Omega )$ with $\zeta _k$ given in (2.3), and to use Proposition 2.4. This leads to the following derivative formula which plays a key role in the proof of Proposition 4.1.

Lemma 4.3

Let $u\in {{\mathcal {H}}}^s_0(\Omega )$ be a solution to (1.4). Then we have

$$\begin{aligned} {{\mathcal {V}}}_u'(0)= \frac{2\lambda _{s,p}(\Omega ) }{p}\int _{\Omega } u^p \text {div}\,X \,dx + 2 \kappa _s \int _{\partial \Omega } (u/\delta ^s)^2 X\cdot \nu \, dx. \end{aligned}$$

Proof

By Lemma 2.3 and since $\Omega$ is of class $C^{1,1}$, we have $U_k:=u \zeta _k\in C^{1,1}_c(\Omega ) \subset {{\mathcal {H}}}^s_0(\Omega )$ for $k \in \mathbb {N}$, and $U_k \rightarrow u$ in ${{\mathcal {H}}}^s_0(\Omega )$ by Lemma 2.2. Consequently, ${{\mathcal {V}}}_u'(0)= \lim \limits _{k \rightarrow \infty }{{\mathcal {V}}}_{U_k}'(0)$ by Corollary 3.1, so it remains to show that

$$\begin{aligned} \lim _{k \rightarrow \infty }{{\mathcal {V}}}_{U_k}'(0) = \frac{2\lambda _{s,p}(\Omega ) }{p}\int _{\Omega } u^p \text {div}X \,dx + 2 \kappa _s \int _{\partial \Omega } (u/\delta ^s)^2 X\cdot \nu \, dx. \end{aligned}$$

(4.5)

Applying Lemma 4.2 to $U_k$, we find that

$$\begin{aligned} {{\mathcal {V}}}_{U_k}'(0)= -2 \int _{\mathbb {R}^N} \nabla U_k \cdot X (-\Delta )^s U_k dx \qquad \text {for }k \in \mathbb {N}. \end{aligned}$$

By the standard product rule for the fractional Laplacian, we have $(-\Delta )^sU_k=u(-\Delta )^s\zeta _k + \zeta _k (-\Delta )^su-I(u, \zeta _k)$ with $I(u, \zeta _k)$ given by (2.11). We thus obtain

$$\begin{aligned}&{{\mathcal {V}}}_{U_k}'(0)=- 2\int _{\mathbb {R}^N} \nabla U_k \cdot X \zeta _k (-\Delta )^su\, dx-2\int _{\mathbb {R}^N} [\nabla U_k \cdot X] u (-\Delta )^s\zeta _k \, dx \nonumber \\&\qquad + 2 \int _{\mathbb {R}^N} \nabla U_k \cdot X I(u, \zeta _k)\, dx \nonumber \\&\quad =-2\lambda _{s,p}(\Omega ) \int _{\Omega } \nabla U_k \cdot X \zeta _k u^{p-1}\, dx -2 \int _{\mathbb {R}^N}\nabla U_k \cdot X \Bigl ( u (-\Delta )^s\zeta _k - I(u, \zeta _k)\Bigr )\,dx, \end{aligned}$$

(4.6)

where we used that $(-\Delta )^su=\lambda _{s,p}(\Omega ) u^{p-1}$ in $\Omega$. Consequently, Proposition 2.4 yields that

$$\begin{aligned} \lim _{k \rightarrow \infty }{{\mathcal {V}}}_{U_k}'(0)= -2\lambda _{s,p}(\Omega ) \lim _{k \rightarrow \infty } \int _{\Omega } \nabla U_k\cdot X \zeta _k u^{p-1}\, dx + 2 \kappa _s \int _{\partial \Omega } \psi ^2 X\cdot \nu \, dx. \end{aligned}$$

(4.7)

Moreover, integrating by parts, we obtain, for $k \in \mathbb {N}$,

$$\begin{aligned}&\int _{\Omega } [\nabla U_k\cdot X] \zeta _k u^{p-1}\, dx = \frac{1}{p} \int _{\Omega } [\nabla u^{p} \cdot X] \zeta _k^2 \, dx+ \int _{\Omega } [\nabla \zeta _k \cdot X] \zeta _k u^{p}\, dx \nonumber \\&\quad =-\frac{1}{p}\int _{\Omega } u^p \text {div}X \zeta _k^2 \,dx -\frac{2}{p}\int _{\Omega } u^p \zeta _k [X\cdot \nabla \zeta _k] \,dx + \int _{\Omega } u^{p} \zeta _k [X\cdot \nabla \zeta _k] \, dx. \end{aligned}$$

(4.8)

Since $u^p\in C^s_0(\overline{\Omega })$ by Lemma 2.3, it is easy to see from the definition of $\zeta _k$ that the last two terms in (4.8) tend to zero as $k\rightarrow \infty$, whereas

$$\begin{aligned} \lim _{k\rightarrow \infty }\int _{\Omega } u^p\text {div} X \zeta _k^2 \,dx= \int _{\Omega } u^p\text {div} X \,dx. \end{aligned}$$

Hence

$$\begin{aligned} \lim _{k \rightarrow \infty } \int _{\Omega } \nabla U_k\cdot X \zeta _k u^{p-1} \, dx = - \frac{1}{p} \int _{\Omega } u^p\text {div} X \,dx. \end{aligned}$$

Plugging this into (4.7), we obtain (4.5), as required. $\square$

Our next lemma provides an upper estimate for $\partial _\varepsilon ^+ \Big |_{\varepsilon =0}\lambda _{s,p}^\varepsilon$.

Lemma 4.4

Let $u\in {{\mathcal {H}}}$ be a positive minimizer for $\lambda _{s,p}^0 = \lambda _{s,p}(\Omega )$. Then

$$\begin{aligned} \limsup _{\varepsilon \rightarrow 0^+}\frac{\lambda _{s,p}^\varepsilon -\lambda _{s,p}^0}{\varepsilon } \le&2 \kappa _s \int _{\partial \Omega } (u/\delta ^s)^2 X\cdot \nu \, dx . \end{aligned}$$

(4.9)

Proof

For $\varepsilon \in (-\varepsilon _0,\varepsilon _0)$, we define

$$\begin{aligned} j(\varepsilon ):=\frac{{{\mathcal {V}}}_{u}(\varepsilon )}{\tau (\varepsilon )}\quad \text {for }k \in \mathbb {N}\text {with} \quad \tau (\varepsilon ):= \left( \int _{\Omega }|u|^p \text {Jac}_{\Phi _\varepsilon }(x)\, dx \right) ^{2/p}. \end{aligned}$$

By (3.8), we then have $\lambda _{s,p}^\varepsilon \le j(\varepsilon )$ for $\varepsilon \in (-\varepsilon _0,\varepsilon _0)$. Moreover,

$$\begin{aligned} \tau (0)=\Vert u\Vert _{L^p(\Omega )}^{2/p}=1, \quad {{\mathcal {V}}}_{u}(0)= [u]_s^2 = \lambda _{s,p}(\Omega )\quad \text {and}\quad j(0)= \frac{{{\mathcal {V}}}_{u}(0)}{\tau (0)} = \lambda _{s,p}^0, \end{aligned}$$

which implies that

$$\begin{aligned} \partial _\varepsilon ^+ \Big |_{\varepsilon =0}\lambda _{s,p}^\varepsilon \le j'(0) = 2 \kappa _s \int _{\partial \Omega } (u/\delta ^s)^2 X\cdot \nu \, dx, \end{aligned}$$

by Lemma 4.3 and (3.2), as claimed. $\square$

Next, we shall prove a lower estimate for $\partial _\varepsilon ^+ \Big |_{\varepsilon =0}\lambda _{s,p}^\varepsilon$.

Lemma 4.5

We have

$$\begin{aligned} \liminf _{\varepsilon \searrow 0^+}\frac{\lambda _{s,p}^\varepsilon -\lambda _{s,p}^0}{\varepsilon }\ge \inf \left\{ 2\kappa _s \int _{\partial \Omega }(u/\delta ^s)^2 X\cdot \nu \,dx\,:\, u\in {{\mathcal {H}}}\right\} . \end{aligned}$$

Proof

Let $(\varepsilon _n)_n$ be a sequence of positive numbers converging to zero and with the property that

$$\begin{aligned} \lim _{n\rightarrow \infty }\frac{ \lambda _{s,p}^{\varepsilon _n}- \lambda _{s,p}^0}{\varepsilon _n} = \liminf _{\varepsilon \searrow 0^+}\frac{\lambda _{s,p}^\varepsilon -\lambda _{s,p}^0}{\varepsilon }. \end{aligned}$$

(4.10)

For $n\in \mathbb {N}$, we let $v_{\varepsilon _n}$ be a positive minimizer corresponding to the variational characterization of $\lambda _{s,p}^{\varepsilon _n}$ given in (3.8), i.e. we have

$$\begin{aligned} \lambda _{s,p}^{\varepsilon _n} = {{\mathcal {V}}}_{v_{\varepsilon _n}}(\varepsilon _n) \qquad \text {and}\qquad \int _{\Omega } v_{\varepsilon _n}^p\text {Jac}_{\Phi _{\varepsilon _n}}dx = 1. \end{aligned}$$

(4.11)

Since $v_{\varepsilon _n}$ remains bounded in ${{\mathcal {H}}}^s_0(\Omega )$ by (3.12), we may pass to a sub-sequence with the property that $v_{\varepsilon _n} \rightharpoonup u$ in ${{\mathcal {H}}}^s_0(\Omega )$ for some $u\in {{\mathcal {H}}}^s_0(\Omega )$. Moreover, $v_{\varepsilon _n} \rightarrow u$ in $L^p(\Omega )$ as $n \rightarrow \infty$ since the embedding ${{\mathcal {H}}}^s_0(\Omega ) \rightarrow L^p(\Omega )$ is compact. In the following, to keep the notation simple, we write $\varepsilon$ in place of $\varepsilon _n$. By (3.10), (3.11) and (4.11), we have

$$\begin{aligned} {{\mathcal {V}}}_{v_{\varepsilon }}(0)= {{\mathcal {V}}}_{v_\varepsilon }(\varepsilon )- \varepsilon {{\mathcal {V}}}_{v_\varepsilon }'(0) + O(\varepsilon ^2)[v_\varepsilon ]_s^2 = \lambda _{s,p}^\varepsilon - \varepsilon {{\mathcal {V}}}_{v_\varepsilon }'(0) + O(\varepsilon ^2)=\lambda _{s,p}^\varepsilon + O(\varepsilon )\nonumber \\ \end{aligned}$$

(4.12)

and therefore

$$\begin{aligned} {{\mathcal {V}}}_{u}(0) =[u]_s^2 \le \liminf _{\varepsilon \rightarrow 0} [v_\varepsilon ]_s^2 = \liminf _{\varepsilon \rightarrow 0}{{\mathcal {V}}}_{v_{\varepsilon }}(0) \le \limsup _{\varepsilon \rightarrow 0}\lambda _{s,p}^\varepsilon \le \lambda _{s,p}^0,\nonumber \\ \end{aligned}$$

(4.13)

where the last inequality follows from Lemma 4.4. In view of (3.2) and the strong convergence $v_\varepsilon \rightarrow u$ in $L^p(\Omega )$, we see that

$$\begin{aligned} 1 = \int _{\Omega } v_\varepsilon ^p\text {Jac}_{\Phi _\varepsilon }dx= \int _{\Omega } v_\varepsilon ^p(1+\varepsilon \text {div}X )dx+O(\varepsilon ^2) = \int _{\Omega } u^p\,dx +o(1) \end{aligned}$$

(4.14)

as $\varepsilon \rightarrow 0$, and hence $\Vert u\Vert _{L^p(\Omega )}=1$. Combining this with (4.13), we see that $u \in {{\mathcal {H}}}$ is a minimizer for $\lambda _{s,p}^0$, and that equality must hold in all inequalities of (4.13). From this we deduce that

$$\begin{aligned} v_\varepsilon \rightarrow u \text {strongly in} {{\mathcal {H}}}^s_0(\Omega ). \end{aligned}$$

(4.15)

Now (4.12) and the variational characterization of $\lambda _{s,p}^0$ imply that

$$\begin{aligned} \lambda _{s,p}^0 \left( \int _{\Omega } v_{\varepsilon }^pdx \right) ^{2/p}\le {{\mathcal {V}}}_{v_{\varepsilon }}(0)= \lambda _{s,p}(\Omega _\varepsilon ) - \varepsilon {{\mathcal {V}}}_{v_\varepsilon }'(0) + O(\varepsilon ^2) \end{aligned}$$

(4.16)

whereas by (4.14) we have

$$\begin{aligned} \int _{\Omega } v_\varepsilon ^p\,dx = 1-\varepsilon \int _{\Omega } v_\varepsilon ^p \text {div}X dx+O(\varepsilon ^2) = 1-\varepsilon \int _{\Omega } u^p \text {div}X dx+o(\varepsilon ) \end{aligned}$$

and therefore

$$\begin{aligned} \left( \int _{\Omega } v_{\varepsilon }^pdx \right) ^{2/p}=1- \frac{2 \varepsilon }{p}\int _{\Omega } u^p \text {div}X dx +o(\varepsilon ). \end{aligned}$$

(4.17)

Plugging this into (4.16), we get the inequality

$$\begin{aligned} \lambda _{s,p}^\varepsilon \ge \Bigl (1- \frac{2 \varepsilon }{p}\int _{\Omega } u^p \text {div}X dx\Bigr ) \lambda _{s,p}^0 + \varepsilon {{\mathcal {V}}}_{v_\varepsilon }'(0)+o(\varepsilon ). \end{aligned}$$

Since, moreover, ${{\mathcal {V}}}_{v_\varepsilon }'(0) \rightarrow {{\mathcal {V}}}_u'(0)$ as $\varepsilon \rightarrow 0$ by Lemma 3.1 and (4.15), it follows that

$$\begin{aligned} \lambda _{s,p}^\varepsilon -\lambda _{s,p}^0 \ge \varepsilon \Bigl ({{\mathcal {V}}}_{u}'(0) -\frac{2 \lambda _{s,p}^0}{p}\int _{\Omega } u^p \text {div}X dx\Bigr )+o(\varepsilon ) \end{aligned}$$

and therefore

$$\begin{aligned} \lambda _{s,p}^\varepsilon -\lambda _{s,p}^0 \ge 2 \varepsilon \kappa _s \int _{\partial \Omega }(u/\delta ^s)^2 X\cdot \nu \,dx+o(\varepsilon ) \end{aligned}$$

by Lemma 4.3. We thus conclude that

$$\begin{aligned} \lim _{\varepsilon \rightarrow 0^+}\frac{\lambda _{s,p}^\varepsilon -\lambda _{s,p}^0}{\varepsilon } \ge 2\kappa _s \int _{\partial \Omega }(u/\delta ^s)^2 X\cdot \nu \,dx. \end{aligned}$$

Taking the infinimum over $u\in {{\mathcal {H}}}$ in the RHS of this inequality and using (4.10), we get the result. $\square$

Proof of Proposition 4.1 (completed)

Proposition 4.1 is a consequence of Lemmas 4.4 and 4.5. Indeed, let

$$\begin{aligned} A_{s,p}(\Omega ):= \inf \left\{ 2\kappa _s \int _{\partial \Omega }(u/\delta ^s)^2 X\cdot \nu \,dx \,:\, u\in {{\mathcal {H}}}\right\} . \end{aligned}$$

Thanks to (2.8) the infinimum $A_{s,p}(\Omega )$ is attained.

Finally by Lemma 4.4 and Lemma 4.5 we get

$$\begin{aligned} A_{s,p}(\Omega ) \ge \partial _\varepsilon ^+ \Big |_{\varepsilon =0}\lambda _{s,p}^\varepsilon \ge \liminf _{\varepsilon \searrow 0}\frac{\lambda _{s,p}^\varepsilon -\lambda _{s,p}^0}{\varepsilon }\ge A_{s,p}(\Omega ). \end{aligned}$$

$\square$

5 Proof of the main results

In this section we complete the proofs of the main results stated in the introduction.

Proof of Theorem 1.1 (completed)

In view of Proposition 4.1, the proof of Theorem 1.1 is complete once we show that

$$\begin{aligned} 2\kappa _s=\Gamma (1+s)^2, \end{aligned}$$

(5.1)

where $\Gamma$ is the usual Gamma function. In view of (2.10), the constant $\kappa _s$ does not depend on N, p and $\Omega$, we consider the case $N=p=1$ and the family of diffeomorphisms $\Phi _\varepsilon$ on $\mathbb {R}^N$ given by $\Phi _\varepsilon (x)=(1+\varepsilon )x$, $\varepsilon \in (-1,1)$, so that $X:= \partial _\varepsilon \big |_{\varepsilon = 0} \Phi _\varepsilon$ is simply given by $X(x)=x$. Letting $\Omega _0:=(-1,1)$, we define $\Omega _{\varepsilon }=\Phi _\varepsilon (\Omega _0)=(-1-\varepsilon ,1+\varepsilon )$. Moreover, we consider $w_\varepsilon \in {{\mathcal {H}}}^s_0(\Omega _\varepsilon ) \cap C^s_0([-1-\varepsilon ,1+\varepsilon ])$ given by

$$\begin{aligned} w_\varepsilon (x)=\ell _s((1+\varepsilon )^2-|x|^2)^s_+ \qquad \text { with }\qquad \ell _s:= \frac{2^{-2s}\Gamma (1/2)}{\Gamma (s+1/2)\Gamma (1+s)}. \end{aligned}$$

(5.2)

It is well known that $w_\varepsilon$ is the unique solution of the problem

$$\begin{aligned} (-\Delta )^sw_\varepsilon =1 \quad \text {in }\Omega _\varepsilon ,\qquad w_\varepsilon \equiv 0 \quad \text {on }\mathbb {R}^N \setminus \Omega _\varepsilon , \end{aligned}$$

see e.g. [21] or [9]. Recalling (1.4), we thus deduce that $u_\varepsilon = \lambda _{s,1}(\Omega _\varepsilon )w_\varepsilon$ is the unique positive minimizer corresponding to (1.3) in the case $N=p=1$, which implies that $\Vert u_\varepsilon \Vert _{L^1(\mathbb {R})}=1$ and therefore

$$\begin{aligned} \lambda _{s,1}(\Omega _\varepsilon )= \Vert w_\varepsilon \Vert _{L^1(\mathbb {R})}^{-1}= (1+\varepsilon )^{-(2s+1)} \Vert w_0\Vert _{L^1(\mathbb {R})}^{-1}. \end{aligned}$$

(5.3)

Moreover, by standard properties of the Gamma function,

$$\begin{aligned} \Vert w_0\Vert _{L^1(\mathbb {R})}&=\ell _s \int _{-1}^1 (1-|x|^2)^s\, dx=2 \ell _s \int _0^1(1-r^2)^s\, dr= \ell _s \int _0^1 t^{-1/2} (1-t)^s\, dt\\&= \ell _s \frac{\Gamma (1/2)\Gamma (s+1)}{\Gamma (s+3/2)}= \ell _s \frac{\Gamma (1/2)\Gamma (s+1)}{(s+1/2)\Gamma (s+1/2)}= \frac{2^{2s}\,\ell _s^2\,\Gamma (s+1)^2}{s+1/2}. \end{aligned}$$

By differentiating (5.3), we get

$$\begin{aligned} \partial _\varepsilon \Big |_{\varepsilon =0}\lambda _{s,1}(\Omega _\varepsilon ) =-\frac{2s+1}{\Vert w_0\Vert _{L^1(\mathbb {R})}}. \end{aligned}$$

(5.4)

On the other hand, by Proposition 4.1 and the fact that $u_0$ is the unique positive minimizer for $\lambda _{s,1}$, we deduce that

$$\begin{aligned} \partial _\varepsilon ^+\Big |_{\varepsilon =0}\lambda _{s,1}(\Omega _\varepsilon )= & {} -2\kappa _s [(u_0/\delta ^s)^2(1)+(u_0/\delta ^s)^2(-1)]\\= & {} -2^{2+2s} \kappa _s \,\ell _s^2 \,\lambda _{s,1}(\Omega _0)^2= -\frac{2^{2+2s} \kappa _s\, \ell _s^2}{\Vert w_0\Vert _{L^1(\mathbb {R})}^2}. \end{aligned}$$

We thus conclude that

$$\begin{aligned} 2 \kappa _s = \frac{(2s+1)\Vert w_0\Vert _{L^1(\mathbb {R})}}{2^{1+2s} \ell _s^2} = \Gamma (s+1)^2. \end{aligned}$$

Thus, by Proposition 4.1, we get the result as stated in the theorem. $\square$

Proof of Corollary 1.3

Let $h\in C^{3}(\partial \Omega )$, with $\int _{\partial \Omega }h\, dx=0$. Then it is well known (see e.g. [11, Lemma 2.2]) that there exists a family of diffeomorphisms $\Phi _\varepsilon : \mathbb {R}^N \rightarrow \mathbb {R}^N$, $\varepsilon \in (-1,1)$ satisfying (1.7) and having the following properties:

$$\begin{aligned} |\Phi _\varepsilon (\Omega )|= |\Omega | \text { for }\varepsilon \in (-1,1), \text {and }X:= \partial _\varepsilon \big |_{\varepsilon =0}\Phi _\varepsilon \text {equals }h \nu \text {on }\partial \Omega . \end{aligned}$$

(5.5)

By assumption, there exists $\varepsilon _0 \in (0,1)$ with $\lambda _{s,p}(\Phi _\varepsilon (\Omega )) \ge \lambda _{s,p}(\Omega )$ for $\varepsilon \in (-\varepsilon _0,\varepsilon _0)$. Applying Theorem 1.1 and noting that $X \cdot \nu \equiv h$ on $\partial \Omega$ by (5.5), we get

$$\begin{aligned} \min \left\{ \Gamma (1+s)^2 \int _{\partial \Omega }(u/\delta ^s)^2 h\,dx\,:\, u\in \mathcal {H}\right\} \ge 0. \end{aligned}$$

By the same argument applied to $-h$, we get

$$\begin{aligned} \max \left\{ \Gamma (1+s)^2 \int _{\partial \Omega }(u/\delta ^s)^2 h\,dx\,:\, u\in \mathcal {H}\right\} \le 0. \end{aligned}$$

(5.6)

We thus conclude that

$$\begin{aligned} \int _{\partial \Omega }(u/\delta ^s)^2 h\,dx = 0 \qquad \text { for every }u\in \mathcal {H}~\text {and for all } h\in C^3(\partial \Omega ), \text {with }\int _{\partial \Omega }h\, dx=0. \end{aligned}$$

By a standard argument, this implies that $u/\delta ^s$ is constant on $\partial \Omega$. Now, since u solves (1.4) and $p\in \{1\}\cup [2,\infty )$, we deduce from [9, Theorem 1.2] that $\Omega$ is a ball. $\square$

Proof of Theorem 1.4

Consider the unit centered ball $B_1=B_1(0)$. For $\tau \in (0,1)$ and $t\in (\tau -1,1-\tau )$, we define $B^t:=B_{\tau }(te_1)$, where $e_1$ is the first coordinate direction. To prove Theorem 1.4, we can take advantage of the invariance under rotations of the problem and may restrict our attention to domains of the form $\Omega (t) = B_1\setminus \overline{B^t}$. We define

$$\begin{aligned} \theta :(\tau -1,1-\tau )\rightarrow \mathbb {R}, \qquad \theta (t):=\lambda _{s,p}(\Omega (t)). \end{aligned}$$

(5.7)

We claim that $\theta$ is differentiable and satisfies

$$\begin{aligned} \theta '(t)<0 \qquad \text { for }t\in (0,1-\tau ). \end{aligned}$$

(5.8)

For this we fix $t \in (\tau -1,1-\tau )$ and a vector field $X: \mathbb {R}^N \rightarrow \mathbb {R}^N$ given by $X(x) = \rho (x)e_{1}$, where $\rho \in C^{\infty }_{c}(B_{1})$ satisfies $\rho \equiv 1$ in a neighborhood of $B^t$. For $\varepsilon \in (-1,1)$, we then define $\Phi _\varepsilon : \mathbb {R}^N \rightarrow \mathbb {R}^N$ by $\Phi _\varepsilon (x)=x + \beta \varepsilon X(x)$, where $\beta >0$ is chosen sufficiently small to guarantee that $\Phi _\varepsilon$, $\varepsilon \in (-1,1)$ is a family of diffeomorphisms satisfying (1.7) and satisfying $\Phi _\varepsilon (B_1) = B_1$ for $\varepsilon \in (-1,1)$. Then, by construction, we have

$$\begin{aligned} \Phi _\varepsilon (\Omega (t))= \Phi _{\varepsilon }\left( B_1\setminus \overline{B^t}\right) = B_1\setminus \overline{\Phi _{\varepsilon }(B^t)}= B_1\setminus \overline{B^{t+\beta \varepsilon }} = \Omega (t+\beta \varepsilon ). \end{aligned}$$

(5.9)

Next we recall that, since $p \in \{1,2\}$, there exists a unique positive minimizer $u \in {{\mathcal {H}}}^s_0(\Omega (t))$ corresponding to the variational characterization (1.3) of $\lambda _{s,p}(\Omega (t))$. Hence, by Corollary 1.2, the map $\varepsilon \mapsto \lambda _{s,p}(\Phi _\varepsilon (\Omega (t)))$ is differentiable at $\varepsilon = 0$. In view of (5.9), we thus find that the map $\theta$ in (5.7) is differentiable at t, and

$$\begin{aligned} \theta '(t)= & {} \frac{1}{\beta } \frac{d}{d\varepsilon } \Big |_{\varepsilon =0}\lambda _{s,p}(\Phi _\varepsilon (\Omega (t))) = \Gamma (1+s)^2\int _{\partial \Omega (t)}\left( \frac{u}{\delta ^{s}}\right) ^{2} X\cdot \nu \, dx \nonumber \\= & {} \Gamma (1+s)^2\int _{\partial B^t}\left( \frac{u}{\delta ^{s}}\right) ^{2} \nu _1\, dx \end{aligned}$$

(5.10)

by (1.10). Here $\nu$ denotes the interior unit normal on $\partial \Omega (t)$ which coincides with the exterior unit normal to $B^t$ on $\partial B^t$, and we used that

$$\begin{aligned} X \equiv e_1\quad \text {on }\partial B^t,\qquad X \equiv 0 \quad \text {on }\partial B_1 = \partial \Omega (t) \setminus \partial B^t \end{aligned}$$

to get the last equality in (5.10). Next, for fixed $t\in (0,1-\tau )$, let H be the half space defined by $H= \{x\in \mathbb {R}^N: x\cdot e_1> t\}$ and let $\Theta = H \cap \Omega (t)$. We also let $r_H:\mathbb {R}^N\rightarrow \mathbb {R}^N$ be the reflection map with respect to he hyperplane $\partial H:=\{x\in \mathbb {R}^N: x\cdot e_1= t\}$. For $x\in \mathbb {R}^N$, we denote ${\bar{x}}:= r_{H}(x)$, $\overline{u} (x):= u({\overline{x}})$. Using these notations, we have

$$\begin{aligned}&\theta '(t) = \Gamma (1+s)^2\int _{\partial B^{t}}\left( \frac{u}{\delta ^{s}}\right) ^{2}\nu _{1}\,dx \nonumber \\&\quad =\Gamma (1+s)^2\int _{\partial B^{t}\cap \Theta }\left( \left( \frac{ u}{\delta ^{s}}\right) ^{2}(x)-\left( \frac{\overline{u}}{\delta ^{s}} \right) ^{2}( x)\right) \nu _{1}\, dx. \end{aligned}$$

(5.11)

Let $w = \overline{u} -u \in H^s(\mathbb {R}^N)$. Then w is a (weak) solution of the problem

$$\begin{aligned} (-\Delta )^sw= \lambda _{s,p}(\Omega ({t})) \overline{u}^{p-1}- \lambda _{s,p}(\Omega ({t})) u^{p-1}= c_p w \qquad \text {in}\quad \Theta , \end{aligned}$$

(5.12)

where

$$\begin{aligned} {\left\{ \begin{array}{ll} c_p:= \lambda _{s,p}(\Omega ({t}))&{}\qquad \text { for }p=2,\\ c_p=0 &{}\qquad \text { for }p=1. \end{array}\right. } \end{aligned}$$

Moreover, by definition, $w \equiv \overline{u}\ge 0$ in $H\setminus \overline{\Theta }$, and $w \equiv \overline{u} >0$ in the subset $[r_H(B_1) \cap H] \setminus \overline{\Theta }$ which has positive measure since $t>0$. Using that w is anti-symmetric with respect to H and the fact that $\lambda _{s,p}(\Theta )> c_p$ (which follows since $\Theta$ is a proper subdomain of $\Omega (t)$), we can apply the weak maximum principle for antisymmetric functions (see [9, Proposition 3.1] or [17, Proposition 3.5]) to deduce that $w\ge 0$ in $\Theta$. Moreover, since $w \not \equiv 0$ in $\mathbb {R}^N$, it follows from the strong maximum principle for antisymmetric functions given in [17, Proposition 3.6] that $w>0$ in $\Theta$. Now by the fractional Hopf lemma for antisymmetric functions (see [9, Proposition 3.3]) we conclude that

$$\begin{aligned} 0< \frac{w}{\delta ^s}= \frac{\overline{u}}{\delta ^s}- {\frac{u}{\delta ^s}}\quad \text {and therefore}\quad {\frac{\overline{u}}{\delta ^s}} > \frac{u}{\delta ^s} \ge 0 \qquad \text {on }\partial B^{t} \cap \Theta . \end{aligned}$$

From this and (5.11) we get (5.8), since $\nu _1>0$ on $\partial B^{t}\cap \Theta$.

To conclude, we observe that the function $t\mapsto \lambda _{s,p}(t)=\lambda _{s,p}(\Omega (t))$ is even, thanks to the invariance of the problem under rotations. Therefore the function $\theta$ attains its maximum uniquely at $t=0$. $\square$

6 Proof of Proposition 2.4

The aim of this section is to prove Proposition 2.4. For the readers convenience, we repeat the statement here.

Proposition 6.1

Let $X \in C^{0}(\overline{\Omega },\mathbb {R}^N)$, let $u \in C^s_0(\overline{\Omega })\cap C^1(\Omega )$, and assume that $\psi :=\frac{u}{\delta ^s}$ extends to a function on $\overline{\Omega }$ satisfying (2.8) and (2.9). Moreover, put $U_k:=u \zeta _k \in C^{1,1}_c(\Omega )$, where $\zeta _k$ is defined in (2.3). Then

$$\begin{aligned} \lim _{k\rightarrow \infty }\int _{\Omega }\nabla U_k \cdot X \Bigl ( u (-\Delta )^s\zeta _k -I(u,\zeta _k)\Bigr )\,dx= - \kappa _s \int _{\partial \Omega } \psi ^2 X\cdot \nu \, dx, \end{aligned}$$

(6.1)

where

$$\begin{aligned} \kappa _s:= -\int _{\mathbb {R}} h'(r) (-\Delta )^sh(r)\, dr \qquad \text {with}\quad h(r):=r^s_+ \zeta (r) \end{aligned}$$

(6.2)

and $\zeta$ given in (2.2), and where we use the notation

$$\begin{aligned} I(u,v)(x):= \int _{\mathbb {R}^N}{(u(x)-u(y))(v(x)-v(y))}K_0(x,y)\, dy \end{aligned}$$

(6.3)

for $u\in C^s_c(\mathbb {R}^N)$, $v\in C^{0,1}(\mathbb {R}^N)$ and $x \in \mathbb {R}^N$.

The remainder of this section is devoted to the proof of this proposition. For $k \in \mathbb {N}$, we define

$$\begin{aligned} g_k:= \nabla U_k \cdot X \Bigl ( u (-\Delta )^s\zeta _k -I(u,\zeta _k)\Bigr ) \quad : \quad \Omega \rightarrow \mathbb {R}. \end{aligned}$$

(6.4)

For $\varepsilon >0$, we put

$$\begin{aligned}&\Omega ^\varepsilon = \{x \in \mathbb {R}^N\,:\, |\delta (x)|< \varepsilon \} \qquad \text {and}\qquad \\&\Omega ^\varepsilon _+= \{x \in \mathbb {R}^N\,:\, 0< \delta (x)< \varepsilon \}= \{x \in \Omega \,:\, \delta (x)< \varepsilon \}. \end{aligned}$$

For every $\varepsilon >0$, we then have

$$\begin{aligned} \lim _{k\rightarrow \infty }\int _{\Omega \setminus \Omega ^\varepsilon } g_k\,dx =0. \end{aligned}$$

(6.5)

To see this, we first note that $\zeta _k \rightarrow 1$ pointwise on $\mathbb {R}^N \setminus \partial \Omega$, and therefore a.e. on $\mathbb {R}^N$. Moreover, choosing a compact neighborhood $K \subset \Omega$ of $\Omega \setminus \Omega ^\varepsilon$, we have

$$\begin{aligned} (-\Delta )^s\zeta _k(x)= c_{N,s}\int _{\mathbb {R}^N \setminus K}\frac{1- \zeta _k(y)}{|x-y|^{N+2s}}dy \qquad \text {for }x \in \Omega \setminus \Omega ^\varepsilon \text { and }k \text {sufficiently large,} \end{aligned}$$

where $\frac{|1- \zeta _k(y)|}{|x-y|^{N+2s}}\le \frac{C}{1+|y|^{N+2s}}$ for $x \in \Omega \setminus \Omega ^\varepsilon ,\, y\in \mathbb {R}^N \setminus K$ and $C>0$ independent of x and y. Consequently, $\Vert (-\Delta )^s\zeta _k\Vert _{L^\infty (\Omega \setminus \Omega ^\varepsilon )}$ remains bounded independently of k and $(-\Delta )^s\zeta _k \rightarrow 0$ pointwise on $\Omega \setminus \Omega ^\varepsilon$ by the dominated convergence theorem. Similarly, we see that $\Vert I(u,\zeta _k)\Vert _{L^\infty (\Omega \setminus \Omega ^\varepsilon )}$ remains bounded independently of k and $I(u,\zeta _k) \rightarrow 0$ pointwise on $\Omega \setminus \Omega ^\varepsilon$. Consequently, we find that

$$\begin{aligned} \Vert g_k\Vert _{L^\infty (\Omega \setminus \Omega ^\varepsilon )} \text {is bounded independently of }k \text {and} g_k \rightarrow 0 \text {pointwise on} \Omega \setminus \Omega ^\varepsilon . \end{aligned}$$

Hence (6.5) follows again by the dominated convergence theorem. As a consequence,

$$\begin{aligned} \lim _{k\rightarrow \infty }\int _{\Omega } g_k(x) \,dx = \lim _{k\rightarrow \infty }\int _{\Omega ^\varepsilon _+} g_k(x)\,dx \qquad \text {for every } \varepsilon >0. \end{aligned}$$

(6.6)

Let, as before, $\nu : \partial \Omega \rightarrow \mathbb {R}^N$ denotes the unit interior normal vector field on $\Omega$. Since we assume that $\partial \Omega$ is of class $C^{1,1}$, the map $\nu$ is Lipschitz, which means that the derivative $d \nu : T \partial \Omega \rightarrow \mathbb {R}^N$ is a.e. well defined and bounded. Moreover, we may fix $\varepsilon >0$ from now on with the property that the map

$$\begin{aligned} \Psi : \partial \Omega \times (-\varepsilon , \varepsilon ) \rightarrow \Omega ^{\varepsilon },\qquad (\sigma ,r) \mapsto {\Psi }(\sigma ,r)= \sigma + r \nu (\sigma ) \end{aligned}$$

(6.7)

is a bi-Lipschitz map with $\Psi (\partial \Omega \times (0, \varepsilon ))= \Omega ^{\varepsilon }_+$. In particular, $\Psi$ is a.e. differentiable, and the variable r is precisely the signed distance of the point ${\Psi }(\sigma ,r)$ to the boundary $\partial \Omega$, i.e.,

$$\begin{aligned} \delta (\Psi (\sigma ,r)) = r \qquad \text {for }\sigma \in \partial \Omega ,\, 0 \le r < \varepsilon . \end{aligned}$$

(6.8)

Moreover, for $0<\varepsilon ' \le \varepsilon$, it follows from (6.6) that

$$\begin{aligned} \lim _{k\rightarrow \infty }\int _{\Omega } g_k\,dx&= \lim _{k\rightarrow \infty }\int _{\Omega ^{\varepsilon '}_+} g_k\,dx = \lim _{k\rightarrow \infty }\int _{\partial \Omega } \int _{0}^{\varepsilon '}\mathrm{Jac}_{{\Psi }}(\sigma ,r) g_k({\Psi }(\sigma ,r))\,dr d\sigma \nonumber \\&= \lim _{k\rightarrow \infty }\frac{1}{k} \int _{\partial \Omega } \int _{0}^{k \varepsilon '}j_k(\sigma ,r) G_k(\sigma ,r) \,dr d\sigma , \end{aligned}$$

(6.9)

where we define

$$\begin{aligned} j_k(\sigma ,r) = \mathrm{Jac}_{{\Psi }}(\sigma ,\frac{r}{k}) \quad \text {and}\quad G_k(\sigma ,r)=g_k({\Psi }(\sigma ,\frac{r}{k})) \qquad \text {for a.e. }\sigma \in \partial \Omega ,\, 0 \le r < k \varepsilon .\nonumber \\ \end{aligned}$$

(6.10)

We note that

$$\begin{aligned} \begin{aligned}&\Vert j_k\Vert _{L^\infty (\partial \Omega \times [0,k\varepsilon ))} \le \Vert \mathrm{Jac}_{{\Psi }}\Vert _{L^\infty (\Omega _\varepsilon )}< \infty \quad \text {for all } k, \text { and}\\&\lim _{k \rightarrow \infty } j_k(\sigma ,r)= \mathrm{Jac}_{{\Psi }}(\sigma ,0)=1 \quad \text {for a.e. }\sigma \in \partial \Omega , r>0. \end{aligned} \end{aligned}$$

(6.11)

By definition of the functions $g_k$ in (6.4), we may write

$$\begin{aligned} G_k(\sigma ,r) = G_{k}^0(\sigma ,r)[G_{k}^1(\sigma ,r) - G_{k}^2(\sigma ,r)] \qquad \text {for }\sigma \in \partial \Omega ,\, 0 \le r < k \varepsilon \end{aligned}$$

(6.12)

with

$$\begin{aligned} \begin{aligned} G_k^0(\sigma ,r)&= [\nabla U_k \cdot X]({\Psi }(\sigma ,\frac{r}{k}))\\ G_k^1(\sigma ,r)&= [u (-\Delta )^s\zeta _k]({\Psi }(\sigma ,\frac{r}{k}))&\qquad \text {and}\\ G_k^2(\sigma ,r)&= I(u,\zeta _k)({\Psi }(\sigma ,\frac{r}{k})). \end{aligned} \end{aligned}$$

(6.13)

In order to analyze the limit in (6.9) for suitable $\varepsilon ' \in (0,\varepsilon ]$, we provide estimates for the functions $G_k^0, G_k^1, G_k^2$ separately in the following. We start with an estimate for $G_k^0$ given by the following lemma.

Lemma 6.2

Let $\alpha \in (0,1)$ be given by Lemma 2.3. Then we have

$$\begin{aligned} k^{s-1}|G_k^0(\sigma ,r)| \le C(r^{s-1}+r^{s-1+\alpha }) \qquad \text {for } k \in \mathbb {N}, 0 \le r < k\varepsilon \end{aligned}$$

(6.14)

with a constant $C>0$, and

$$\begin{aligned} \lim _{k \rightarrow \infty } k^{s-1} G_k^0(\sigma ,r) = h'(r) \psi (\sigma ) [X(\sigma ) \cdot \nu (\sigma )] \quad \text {for }\sigma \in \partial \Omega , r>0 \end{aligned}$$

(6.15)

with the function $r \mapsto h(r)= r^s_+ \zeta (r)$ given in (6.2).

Proof

Since $u= \psi \delta ^s$, we have

$$\begin{aligned} \nabla u = s \delta ^{s-1} \psi \nabla \delta + \delta ^s \nabla \psi = s \delta ^{s-1} \psi \nabla \delta + O(\delta ^{s-1+\alpha })\qquad \text {in }\Omega \end{aligned}$$

by Lemma 2.3, and therefore, since $\zeta _k = \zeta \circ (k \delta )$ by (2.3),

$$\begin{aligned}&\nabla U_k = \nabla \Bigl (u \zeta _k \Bigr )= \Bigl (s \zeta \circ (k \delta ) + k \delta \zeta ' \circ (k \delta )\Bigr ) \psi \delta ^{s-1}\nabla \delta \\&\quad + O(\delta ^{s-1+\alpha }) \qquad \text {in }\Omega . \end{aligned}$$

Consequently, by (6.8) we have

$$\begin{aligned}&\bigl [\bigl (\nabla U_k\bigr ) \circ {\Psi }\bigr ](\sigma ,\frac{r}{k}) = \Bigl (s \zeta (r) + r \zeta '(r)\Bigr ) \psi (\sigma + \frac{r}{k}\nu (\sigma )) \bigl (\frac{r}{k}\bigr )^{s-1} \nabla \delta (\sigma + \frac{r}{k}\nu (\sigma )) \\&\quad + O\Bigl (\bigl (\frac{r}{k}\bigr )^{s-1+\alpha }\Bigr ) \end{aligned}$$

for $\sigma \in \partial \Omega$, $0 \le r < \varepsilon$ with $O(r^{s-1+\alpha })$ independent of k, and therefore

$$\begin{aligned} G_k^0(\sigma ,r)&= \Bigl (s \zeta (r) + r \zeta '(r)\Bigr ) \psi (\sigma + \frac{r}{k}\nu (\sigma )) \nabla \delta (\sigma + \frac{r}{k}\nu (\sigma )) \cdot X(\sigma + \frac{r}{k}\nu (\sigma ))k^{1-s} r^{s-1}\\&+ k^{1-s-\alpha } O(r^{s-1+\alpha }) \qquad \text {for }\sigma \in \partial \Omega , 0 \le r < k \varepsilon . \end{aligned}$$

Since $\alpha >0$, we deduce that

$$\begin{aligned} k^{s-1}G_k^0(\sigma ,r)&\rightarrow \Bigl (s \zeta (r) + r \zeta '(r)\Bigr ) \psi (\sigma ) \nabla \delta (\sigma ) \cdot X(\sigma )r^{s-1}\\&=h'(r) \psi (\sigma ) X(\sigma ) \cdot \nu (\sigma ) \qquad \text {as }k \rightarrow \infty \end{aligned}$$

for $\sigma \in \partial \Omega$, $r>0$, while

$$\begin{aligned} k^{s-1}|G_k^0(\sigma ,r)| \le C(r^{s-1}+r^{s-1+\alpha }) \qquad \text {for } k \in \mathbb {N}, 0 \le r < k\varepsilon \end{aligned}$$

with a constant $C>0$ independent of k and r, as claimed. $\square$

Next we consider the functions $G_k^1$ defined in (6.13), and we first state the following estimate.

Proposition 6.3

There exists $\varepsilon '>0$ with the property that

$$\begin{aligned} |k^{-2s} (-\Delta )^s\zeta _k({\Psi }(\sigma , \frac{r}{k}))|\le \frac{C}{1+r^{1+2s}}\qquad \text {for }k \in \mathbb {N}, 0 \le r < k \varepsilon ' \end{aligned}$$

(6.16)

with a constant $C>0$. Moreover,

$$\begin{aligned} \lim _{k \rightarrow \infty }k^{-2s} (-\Delta )^s\zeta _k({\Psi }(\sigma , \frac{r}{k})) = (-\Delta )^s \zeta (r) \quad \text {for }\sigma \in \partial \Omega , r>0. \end{aligned}$$

(6.17)

Before giving the somewhat lengthy proof of this proposition, we infer the following corollary related to the functions $G_k^1$.

Corollary 6.4

There exists $\varepsilon '>0$ with the property that

$$\begin{aligned} |k^{-s}G_k^1(\sigma ,r)|\le \frac{C r^s}{1+r^{1+2s}}\qquad \hbox {for} k \in \mathbb {N}\hbox {,} 0 \le r < k \varepsilon ' \end{aligned}$$

(6.18)

with a constant $C>0$. Moreover,

$$\begin{aligned} \lim _{k \rightarrow \infty }k^{-s}G_k^1(\sigma ,r) = \psi (\sigma ) r^s (-\Delta )^s \zeta (r) \quad \text {for }\sigma \in \partial \Omega , r>0. \end{aligned}$$

(6.19)

Proof

Since $u = \psi \delta ^s$ we have $u({\Psi }(\sigma ,\frac{r}{k}))= k^{-s} \psi (\sigma + \frac{r}{k} \nu (\sigma )) r^s$ for $k \in \mathbb {N}$, $0 \le r < k\varepsilon$, and

$$\begin{aligned} \lim _{k \rightarrow \infty }k^{s}u({\Psi }(\sigma ,\frac{r}{k})) = \psi (\sigma ) r^s \quad \text {for }\sigma \in \partial \Omega , r>0. \end{aligned}$$

Since moreover $\Vert \psi \Vert _{L^\infty (\Omega _\varepsilon )}<\infty$, the claim now follows from Proposition 6.3 by recalling the definition in $G_k^1$ in (6.13). $\square$

We now turn to the proof of Proposition 6.3, and we need some preliminary considerations. Since $\partial \Omega$ is of class $C^{1,1}$ by assumption, there exists an open ball $B \subset \mathbb {R}^{N-1}$ centered at the origin and, for every $\sigma \in \partial \Omega$, a parametrization $f_\sigma : B \rightarrow \partial \Omega$ of class $C^{1,1}$ with the property that $f_\sigma (0)=\sigma$ and $d f_\sigma (0): \mathbb {R}^{N-1} \rightarrow \mathbb {R}^N$ is a linear isometry. For $z \in B$ we then have

$$\begin{aligned} f_\sigma (z)-f_\sigma (0)= df_\sigma (0)z + O(|z|^2) \end{aligned}$$

and therefore

$$\begin{aligned}&|f_\sigma (0)-f_\sigma (z)|^2 = |df_\sigma (0)z|^2 + O(|z|^3) = |z|^2 + O(|z|^3), \end{aligned}$$

(6.20)

$$\begin{aligned}&(f_\sigma (0)-f_\sigma (z))\cdot \nu (\sigma ) = - df_\sigma (0)z \cdot \nu (\sigma )+ O(|z|^2)= O(|z|^2), \end{aligned}$$

(6.21)

where we used in (6.21) that $d f_\sigma (0)z$ belongs to the tangent space $T_\sigma \partial \Omega = \{\nu (\sigma )\}^\perp$. Here and in the following, the term $\mathcal {O}(\tau )$ stands for a function depending on $\tau$ and possibly other quantities but satisfying $|\mathcal {O}(\tau )| \le C \tau$ with a constant $C>0$.

Recalling the definition of the map $\Psi$ in (6.7) and writing $\nu _\sigma (z):= \nu (f_\sigma (z))$ for $z \in B$, we now define

$$\begin{aligned} \Psi _\sigma : (-\varepsilon ,\varepsilon ) \times B \rightarrow \Omega ^\varepsilon ,\qquad \Psi _\sigma (r,z) = \Psi (f_\sigma (z),r) = f_\sigma (z) + r \nu _\sigma (z). \end{aligned}$$

(6.22)

Then $\Psi _\sigma$ is a bi-Lipschitz map which maps $(-\varepsilon ,\varepsilon ) \times B$ onto a neighborhood of $\sigma$. Consequently, there exists $\varepsilon ' \in (0,\frac{\varepsilon }{2})$ with the property that

$$\begin{aligned} |\sigma -y| \ge 3\varepsilon ' \qquad \text {for all }y \in \mathbb {R}^N \setminus \Psi _\sigma ((-\varepsilon ,\varepsilon ) \times B). \end{aligned}$$

(6.23)

Moreover, $\varepsilon '$ can be chosen independently of $\sigma \in \partial \Omega$.

Coming back to the proof of Proposition 6.3, we now write, for $\sigma \in \partial \Omega$ and $r \in [0,k\varepsilon ')$,

$$\begin{aligned} (-\Delta )^s\zeta _k(\Psi (\sigma ,\frac{r}{k})) = c_{N,s} \Bigl ( A_k(\sigma ,r)+ B_k(\sigma ,r) \Bigr ) \end{aligned}$$

(6.24)

with

$$\begin{aligned} A_{k}(\sigma ,r) := \int _{ \Psi _\sigma ((-\varepsilon ,\varepsilon ) \times B)} \frac{\zeta (r)-\zeta _k(y)}{|\Psi (\sigma ,\frac{r}{k}) - y|^{N+2s}}\, dy \end{aligned}$$

and

$$\begin{aligned} B_k(\sigma ,r) := \int _{\mathbb {R}^N \setminus \Psi _\sigma ((-\varepsilon ,\varepsilon ) \times B)} \frac{\zeta (r)-\zeta _k(y)}{|\Psi (\sigma ,\frac{r}{k})- y|^{N+2s}}\, dy. \end{aligned}$$

Here we used that $\zeta _k(\Psi (\sigma ,\frac{r}{k}))= \zeta (r)$ for $\sigma \in \partial \Omega$, $r \in [0,k \varepsilon ')$ by (6.8) and the definition of $\zeta _k$. We first provide a rather straightforward estimate for the functions $B_k$.

Lemma 6.5

We have

$$\begin{aligned} k^{-2s}|B_k(\sigma ,r)| \le \frac{C}{1+r^{1+2s}} \qquad \quad \text {for }k \in \mathbb {N}, 0 \le r < k \varepsilon ', \sigma \in \partial \Omega \end{aligned}$$

(6.25)

with a constant $C>0$ and

$$\begin{aligned} \lim _{k \rightarrow \infty }k^{-2s}|B_k(\sigma ,r)|=0 \qquad \text {for every } \sigma \in \Omega , r\ge 0. \end{aligned}$$

(6.26)

Proof

By (6.23) and since $r < k\varepsilon '$, we have

$$\begin{aligned} |\Psi (\sigma ,\frac{r}{k})- y|= & {} |\sigma - y + \frac{r}{k}\nu (\sigma )|\ge |\sigma - y|- \frac{r}{k} \\\ge & {} \frac{|\sigma - y|}{3} +\varepsilon ' \quad \text {for }y \in \mathbb {R}^N \setminus \Psi _\sigma ((-\varepsilon ,\varepsilon ) \times B). \end{aligned}$$

Recalling that $\zeta = 1 -\rho$, $\,\zeta _k = 1-\rho _k$ and that $\rho _k$ is supported in $\Omega ^{{\tiny {\frac{2}{k}}}}$, we thus estimate

$$\begin{aligned} |B_k(\sigma ,r)|&\le \int _{\mathbb {R}^N \setminus \Psi _\sigma ((-\varepsilon ,\varepsilon ) \times B)} \frac{|\rho (r)-\rho _k(y)|}{|\Psi (\sigma ,\frac{r}{k})- y|^{N+2s}}\, dy \\&\le 3^{N+2s} |\rho (r)| \int _{\mathbb {R}^N} \Bigl (|\sigma - y| +3 \varepsilon '\Bigr )^{-N-2s}dy + \bigl (\varepsilon '\bigr )^{-N-2s} \int _{\mathbb {R}^N} |\rho _k(y)| \, dy\\&\le C \Bigl (|\rho (r)| +|\Omega ^{{\tiny {\frac{2}{k}}}}|\Bigr ) \le C \Bigl (|\rho (r)| + k^{-1}\Bigr ). \end{aligned}$$

Here and in the following, the letter C stands for various positive constants. This estimate readily yields (6.26). Moreover,

$$\begin{aligned} k^{-2s}|B_k(\sigma ,r)| \le C k^{-2s}\Bigl (|\rho (r)| + k^{-1}\Bigr ) \le \frac{C}{1+r^{1+2s}} + k^{-1-2s} \le \frac{C}{1+r^{1+2s}} \end{aligned}$$

for $k \in \mathbb {N}$, $0 \le r < k \varepsilon '$, $\sigma \in \partial \Omega$, as claimed in (6.25). $\square$

To complete the proof of Proposition 6.3, it thus remains to consider the functions $A_{k}$ in the following. For this, we need the following additional estimates for the maps $\Psi _\sigma$, $\sigma \in \partial \Omega$. We note here that $\Psi _\sigma$ is a.e. differentiable since it is Lipschitz, so the Jacobian determinant $\mathrm{Jac}_{{\Psi _\sigma }}$ is a.e. well-defined on $(-\varepsilon ,\varepsilon ) \times B$.

Lemma 6.6

There exists a constant $C_0$ with the property that for every $\sigma \in \partial \Omega$ we have the following estimates:

• (i) $\quad |\mathrm{Jac}_{{\Psi _\sigma }}(r,z)| \le C_0$ for a.e. $r \in (-\varepsilon ,\varepsilon )$, $z \in B$;

• (ii) $\quad |\mathrm{Jac}_{{\Psi _\sigma }}(r,z)-1| \le C_0(|r|+|z|)$ for a.e. $r \in (-\varepsilon ,\varepsilon )$, $z \in B$;

• (iii) $\quad |\mathrm{Jac}_{{\Psi _\sigma }}(r+t,z)-\mathrm{Jac}_{{\Psi _\sigma }}(r-t,z)| \le C_0|t|$ for a.e. $r \in (-\varepsilon ,\varepsilon )$, $z \in B$, $t \in (-\varepsilon -r,\varepsilon -r)$. Moreover, for $\sigma \in \partial \Omega$, $r \in (-\varepsilon ,\varepsilon )$, $z \in B$, $t \in (-\varepsilon -r,\varepsilon -r)$ we have

• (iv) $\quad \frac{1}{C_0} \bigl (t^2 + |z|^2\bigr )^{\frac{1}{2}} \le |{\Psi _\sigma }(r,0)- {\Psi _\sigma }(r+t,z)| \le C_0 \bigl (t^2 + |z|^2\bigr )^{\frac{1}{2}}$, and for $\sigma \in \partial \Omega$, $r \in (-\varepsilon ,\varepsilon )$, $t \in (-\varepsilon -r,\varepsilon -r) \setminus \{0\}$ and $z \in \frac{1}{|t|}B$ we have

• (v) $\quad \Bigl |\frac{|{\Psi _\sigma }(r,0)- {\Psi _\sigma }(r+t ,|t| z)|^{2}}{t^2}-(1+|z|^2)\Bigr | \le C_0 (|t|+ |r| + |tz|)|z|^2$;

• (vi)
  $$\begin{aligned}&\Bigl |\bigl |{\Psi _\sigma }(r,0)- {\Psi _\sigma }(r+t ,|t| z)\bigr |^{-N-2s}- \bigl |{\Psi _\sigma }(r,0)- {\Psi _\sigma }(r-t ,|t| z)\bigr |^{-N-2s}\Bigr |\\&\quad \quad \le C_0 |t|^{1-N-2s}(1+|z|^2)^{-\frac{N+2s}{2}}. \end{aligned}$$

Proof

The inequalities (i) and (iv) are direct consequences of the fact that $\Psi _\sigma$ is bi-Lipschitz. In particular, if $C_0$ is a Lipschitz constant for $\Psi _\sigma ^{-1}$, we have

$$\begin{aligned} \bigl (t^2 + |z|^2\bigr )^{\frac{1}{2}} = |(-t,z)|= |(r,0) -(r+t,z)| \le C_0 |{\Psi _\sigma }(r,0)- {\Psi _\sigma }(r+t,z)| \end{aligned}$$

for $\sigma \in \partial \Omega$, $r \in (-\varepsilon ,\varepsilon )$, $z \in B$ and $t \in (-\varepsilon -r,\varepsilon -r)$, so the first inequality in (iv) follows. By making $C_0$ larger if necessary so that it is also a Lipschitz constant for $\Psi _\sigma$, we then deduce the second inequality in (iv).

To see (ii) and (iii), we note that $d \Psi _\sigma$ is a.e. given by

$$\begin{aligned} d \Psi _\sigma (r,z)(r',z') = [d f_\sigma (z)+ r d \nu _\sigma (z)]z' + r' \nu _\sigma (z) \end{aligned}$$

for $(r,z) \in (-\varepsilon ,\varepsilon ) \times B$, $(r',z') \in \mathbb {R}\times \mathbb {R}^{N-1}$, which implies that

$$\begin{aligned}{}[d \Psi _\sigma (r,z)-d \Psi _\sigma (0,0)] (r',z') = [d f_\sigma (z)-df_\sigma (0)]z'+ r d \nu _\sigma (z) +r' \bigl (\nu _\sigma (z)-\nu _\sigma (0)\bigr ) \end{aligned}$$

and

$$\begin{aligned}{}[d \Psi _\sigma (r+t,z)-d \Psi _\sigma (r-t,z)] (r',z') = 2t d \nu _\sigma (z) z'. \end{aligned}$$

Since $d f_\sigma$, $\nu _\sigma$ are Lipschitz functions on B, $d\nu _\sigma$ is a bounded function on B and the determinant is a locally Lipschitz continuous function on the space of linear endomorphisms of $\mathbb {R}^N$, it follows that

$$\begin{aligned}&|\mathrm{Jac}_{{\Psi _\sigma }}(r,z)-\mathrm{Jac}_{{\Psi _\sigma }}(0,0)| \le C_0(|r|+|z|) \quad \text {and}\quad |\mathrm{Jac}_{{\Psi _\sigma }}(r+t,z)\\&\quad -\mathrm{Jac}_{{\Psi _\sigma }}(r-t,z)| \le C_0|t| \end{aligned}$$

for a.e. $r \in (-\varepsilon ,\varepsilon )$, $z \in B$, $t \in (-\varepsilon -r,\varepsilon -r)$. Moreover, $\mathrm{Jac}_{{\Psi _\sigma }}(0,0) = 1$ since the map

$$\begin{aligned} \mathbb {R}\times \mathbb {R}^{N-1} \rightarrow \mathbb {R}^N, \qquad (r',z') \mapsto d{\Psi _\sigma }(0,0)(r',z') = d f_\sigma (0)z' + r' \nu _\sigma (0) \end{aligned}$$

is an isometry. Hence (ii) and (iii) follow.

To see (v) and (vi), we note that by definition of $\Psi _\sigma$ we have

$$\begin{aligned} {\Psi _\sigma }(r,0)- {\Psi _\sigma }(r+t,z)=f_\sigma (0)-f_\sigma (z)-t \nu _\sigma (0)+(r+t)(\nu _\sigma (0)-\nu _\sigma (z)) \end{aligned}$$

for $z \in B$, $r \in (0,\varepsilon ')$ and $t \in (-\varepsilon -r,\varepsilon -r)$. Using moreover that $(\nu _\sigma (0)-\nu _\sigma (z))\cdot \nu _\sigma (0)=\frac{1}{2}|\nu _\sigma (0)-\nu _\sigma (z)|^2$, we get

$$\begin{aligned}&|{\Psi _\sigma }(r,0)- {\Psi _\sigma }(r+t,z)|^2 =t^2+ |f_\sigma (0)-f_\sigma (z)|^2 +(r+t)^2|\nu _\sigma (0)-\nu _\sigma (z)|^2 \nonumber \\&\quad \quad - 2t(f_\sigma (0)-f_\sigma (z))\cdot \nu _\sigma (0) -t(r+t)|\nu _\sigma (0)-\nu _\sigma (z)|^2 \nonumber \\&\quad \quad +2(r+t)(f_\sigma (0)-f_\sigma (z))\cdot (\nu _\sigma (0)-\nu _\sigma (z)) \nonumber \\&\quad =t^2+ |f_\sigma (0)-f_\sigma (z)|^2 +r(r+t)|\nu _\sigma (0)-\nu _\sigma (z)|^2 \nonumber \\&\qquad - 2t(f_\sigma (0)-f_\sigma (z))\cdot \nu _\sigma (0) +2(r+t)(f_\sigma (0)-f_\sigma (z))\cdot (\nu _\sigma (0)-\nu _\sigma (z)) \nonumber \\&\quad =t^2+ |z|^2 + \bigl [|z| m_\sigma (z) +r(r+t) n_\sigma (z)-2t p_\sigma (z) +2(r+t)q_\sigma (z) \bigr ]|z|^2 \end{aligned}$$

(6.27)

for $z \in B$, $r \in (-\varepsilon ,\varepsilon )$ and $t \in (-\varepsilon -r,\varepsilon -r)$ with the functions

$$\begin{aligned} m_\sigma (z)= & {} \frac{|f_\sigma (0)-f_\sigma (z)|^2-|z|^2}{|z|^3}, \quad n_\sigma (z)= \frac{|\nu _\sigma (0)-\nu _\sigma (z)|^2}{|z|^2},\quad \\ p_\sigma (z)= & {} \frac{(f_\sigma (0)-f_\sigma (z))\cdot \nu _\sigma (0)}{|z|^2} \end{aligned}$$

and

$$\begin{aligned} q_\sigma (z) = \frac{(f_\sigma (0)-f_\sigma (z))\cdot (\nu _\sigma (0)-\nu _\sigma (z))}{|z|^2}, \qquad z \in B \setminus \{0\}, \end{aligned}$$

which are all bounded as a consequence of the Lipschitz continuity of $f_\sigma$ and $\nu _\sigma$ and of (6.20) and (6.21). We deduce that

$$\begin{aligned}&\Bigl |\frac{|{\Psi _\sigma }(r,0)- {\Psi _\sigma }(r+t ,|t| z)|^{2}}{t^2}-(1+|z|^2)\Bigr |\\&\quad = \Bigl ||tz| m_\sigma (|t|z) +r(r+t) n_\sigma (|t|z)-2t p_\sigma (|t|z) +2(r+t)q_\sigma (|t|z) \Bigr | |z|^2 \\&\quad \le C_0(|tz|+ |r| + |t|)|z|^2 \end{aligned}$$

for $\sigma \in \partial \Omega$, $r \in (-\varepsilon ,\varepsilon )$, $t \in (-\varepsilon -r,\varepsilon -r) \setminus \{0\}$ and $z \in \frac{1}{|t|}B$ if $C_0$ is chosen sufficiently large, as claimed in (v).

For the proof of (vi), we now set $w_\sigma (r,t,z):= \frac{1}{t^2}|{\Psi _\sigma }(r,0)- {\Psi _\sigma }(r+t,|t|z)|^2$, and we note that

$$\begin{aligned} w_\sigma (r,t,z) \ge \frac{1+|z|^2}{C_0^2}\quad \text {for }\sigma \in \partial \Omega , r \in (-\varepsilon ,\varepsilon ), t \in (-\varepsilon -r,\varepsilon -r) \setminus \{0\}, z \in \frac{1}{|t|}B \end{aligned}$$

by (iv). Moreover, from (6.27) we infer that

$$\begin{aligned} \Bigl |w_\sigma (r,t,z) -w_\sigma (r,-t,z) \Bigr |= \Bigl | 2 r t n_\sigma (|t|z) +4t\bigl ( q_\sigma (|t|z)-p_\sigma (|t|z)\bigr ) \Bigr | |z|^2 \le C_0 |t||z|^2 \end{aligned}$$

for $\sigma \in \partial \Omega$, $r \in (-\varepsilon ,\varepsilon )$, $t \in (-\varepsilon -r,\varepsilon -r) \setminus \{0\}$ and $z \in \frac{1}{|t|}B$ if $C_0$ is made larger if necessary. Using these estimates together with the mean value theorem, we get that, for some $\tau = \tau (\sigma ,r,t,z)$ with $-t< \tau < t$,

$$\begin{aligned}&\Bigl | |{\Psi _\sigma }(r,0)- {\Psi _\sigma }(r+t ,|t|z)|^{-N-2s}-|{\Psi _\sigma }(r,0)- {\Psi _\sigma }(r-t ,|t|z)|^{-N-2s}\Bigr |\\&\quad = |t|^{-N-2s}\Bigl |w_\sigma (r,t,z)^{-\frac{N+2s}{2}} -w_\sigma (r,-t,z)^{-\frac{N+2s}{2}} \Bigr |\\&\quad = \frac{(N+2s)|t|^{-N-2s} }{2} w_\sigma (r,\tau ,z)^{-\frac{N+2s+2}{2}} \Bigl |w_\sigma (r,t,z) -w_\sigma (r,-t,z) \Bigr |\\&\quad \le C_0|t|^{1-N-2s} (1+|z|^2)^{-\frac{N+2s+2}{2}}|z|^2 \le C_0|t|^{1-N-2s} (1+|z|^2)^{-\frac{N+2s}{2}} \end{aligned}$$

for $z\in B$, $r\in (0,\varepsilon ')$ and $t \in (-\varepsilon +r,\varepsilon -r)$ after making $C_0$ larger if necessary, as claimed in (vi). $\square$

We now have all the tools to study the quantity $A_{k}(\sigma ,r)$ in (6.24).

Lemma 6.7

We have

$$\begin{aligned} k^{-2s}|A_k(\sigma ,r)| \le \frac{C}{1+r^{1+2s}} \qquad \quad \text {for }k \in \mathbb {N}, 0 \le r < k \varepsilon ', \sigma \in \partial \Omega \end{aligned}$$

(6.28)

with a constant $C>0$ and

$$\begin{aligned} \lim _{k \rightarrow \infty }k^{-2s}A_k(\sigma ,r)=\frac{(-\Delta )^s\zeta (r)}{c_{N,s}} \qquad \text {for every }\sigma \in \Omega , r\ge 0. \end{aligned}$$

(6.29)

Proof

For $\sigma \in \partial \Omega$ and $0< r < k \varepsilon '$, we write, with a change of variables,

$$\begin{aligned}&A_{k}(\sigma ,r) \nonumber \\&\quad = \int _{ \Psi _\sigma ((-\varepsilon ,\varepsilon ) \times B)} \frac{\zeta (r)-\zeta _k(y)}{|\Psi (\sigma ,\frac{r}{k}) - y|^{N+2s}}\, dy\nonumber \\&\quad = \int _{-\varepsilon }^{\varepsilon } \int _{B}\mathrm{Jac}_{\Psi _\sigma }({\tilde{r}},z)\, \frac{\zeta (r)-\zeta (k {\tilde{r}})}{|\Psi _\sigma (\frac{r}{k},0) - \Psi _\sigma ({\tilde{r}},z)|^{N+2s}}\,d z d {\tilde{r}} \nonumber \\&\quad = \frac{1}{k}\int _{-k\varepsilon -r}^{k\varepsilon -r} \int _{B}\mathrm{Jac}_{\Psi _\sigma }(\frac{r+t}{k},z)\, \frac{\zeta (r)-\zeta (r+t)}{|\Psi _\sigma (\frac{r}{k},0) - \Psi _\sigma (\frac{r+t}{k},z)|^{N+2s}}\,d z d t \nonumber \\&\quad = \int _{-k\varepsilon -r}^{k\varepsilon -r} \frac{|t|^{N-1}}{k^N} \int _{\frac{k}{|t|}B}\mathrm{Jac}_{\Psi _\sigma }(\frac{r+t}{k},\frac{|t|z}{k})\, \frac{\zeta (r)-\zeta (r+t)}{|\Psi _\sigma (\frac{r}{k},0) - \Psi _\sigma (\frac{r+t}{k},\frac{|t|z}{k})|^{N+2s}}\,d z d t\nonumber \\&\quad = k^{2s}\int _{\mathbb {R}}\frac{\zeta (r)-\zeta (r+t)}{|t|^{1+2s}}{{\mathcal {K}}}_k(r,t) dt \end{aligned}$$

(6.30)

with the kernels ${{\mathcal {K}}}_k: (0,k\varepsilon ') \times \mathbb {R}\rightarrow \mathbb {R}$ defined by

$$\begin{aligned} {{\mathcal {K}}}_k(r,t)= \left\{ \begin{aligned}&\Bigl (\frac{|t|}{k}\Bigr )^{N+2s} \int _{\frac{k}{|t|}B} \frac{ \mathrm{Jac}_{{\Psi _\sigma }}(\frac{r+t}{k},\frac{|t|z}{k})}{\bigl |{\Psi _\sigma }(\frac{r}{k},0)-{\Psi _\sigma }(\frac{r+t}{k},\frac{|t|z}{k}) \bigr |^{N+2s}}dz,&\quad t\in (-k\varepsilon -r,k\varepsilon -r), \\&0,&\quad t \not \in (-k\varepsilon -r,k\varepsilon -r). \end{aligned} \right. \end{aligned}$$

Consequently,

$$\begin{aligned} A_{k}(\sigma ,r)= k^{2s}\Bigl ( J^1_k(\sigma ,r)+ J^2_k(\sigma ,r)\Bigr ) \end{aligned}$$

(6.31)

with

$$\begin{aligned} J^1_k(\sigma ,r):=\frac{1}{4 } \int _{\mathbb {R}} \frac{2\zeta (r)-\zeta (r+t)-\zeta (r-t)}{|t|^{1+2s}} \bigl ({{\mathcal {K}}}_k(r,t) +{{\mathcal {K}}}_k(r,-t)\bigr ) dt \end{aligned}$$

and

$$\begin{aligned} J^2_k(\sigma ,r):=-\frac{1}{4} \int _{\mathbb {R}} \frac{\zeta (r+t)-\zeta (r-t)}{|t|^{2s}}\,\frac{{{\mathcal {K}}}_k(r,t)- {{\mathcal {K}}}_k(r,-t)}{|t|} dt. \end{aligned}$$

By Lemma 6.6(i), (iv) and the definition of ${{\mathcal {K}}}_k$, we have

$$\begin{aligned} |{{\mathcal {K}}}_k(r,t)| \le C_0^{N+2s+1} \int _{\frac{k}{|t|}B} \Bigl (1+ \Bigl |z|^2\Bigr )^{-\frac{N+2s}{2}}dz \le C_0^{N+2s+1} a_{N,s} \end{aligned}$$

(6.32)

for $r \in (-k\varepsilon ',k\varepsilon ')$ and $t \in \mathbb {R}\setminus \{0\}$ with

$$\begin{aligned} a_{N,s}:=\int _{\mathbb {R}^{N-1}}(1+|z|^2)^{-\frac{N+2s}{2}}dz < \infty . \end{aligned}$$

(6.33)

Moreover, by Lemma 6.6(i)(ii),(iv),(v) and the dominated convergence theorem, we have

$$\begin{aligned} \lim _{k\rightarrow \infty }{{\mathcal {K}}}_k(r,t)= \int _{\mathbb {R}^{N-1}}(1+|z|^2)^{-\frac{N+2s}{2}}\,dz = a_{N,s} \qquad \text {for every }r \ge 0, t \in \mathbb {R}\setminus \{0\}.\nonumber \\ \end{aligned}$$

(6.34)

Using (6.32) and the fact that $\rho =1-\zeta \in C^\infty _c(\mathbb {R})$, we obtain the estimate

$$\begin{aligned} | J^1_k(\sigma ,r)|&\le C \int _{\mathbb {R}} {\frac{|2\zeta ( r)-\zeta (r+t)-\zeta (r-t)|}{|t|^{1+2s}}} \,dt \nonumber \\&= C \int _{\mathbb {R}} {\frac{|2\rho (r)-\rho (r+t)-\rho (r-t)|}{|t|^{1+2s}}} \,dt \le \frac{C}{1+r^{1+2s}} \end{aligned}$$

(6.35)

for $k \in \mathbb {N}$, $r\in (0,k\varepsilon ')$ and $\sigma \in \partial \Omega$. Here and in the following, the letter $C>0$ stands for different positive constants. Moreover, by (6.32), (6.34) and the dominated convergence theorem, we find that

$$\begin{aligned}&\lim _{k \rightarrow \infty } J^1_k(\sigma ,r) = \frac{a_{N,s}}{2} \int _{\mathbb {R}}\frac{2\zeta (r)-\zeta (r+t)-\zeta (r-t)}{|t|^{1+2s}}dt \nonumber \\&\quad = \frac{a_{N,s}}{c_{1,s}} (-\Delta )^s\zeta (r)= \frac{(-\Delta )^s\zeta (r)}{c_{N,s}}. \end{aligned}$$

(6.36)

Here we have used the fact that

$$\begin{aligned} c_{N,s}a_{N,s}=c_{1,s}, \end{aligned}$$

(6.37)

see e.g. [10].

Next we deal with $J^2_k(\sigma ,r)$, and for this we have to estimate the kernel differences $|{{\mathcal {K}}}_k(r,t)- {{\mathcal {K}}}_k(r,-t)|$. By Lemma 6.6(i), (iii), (iv) and (vi), we have

$$\begin{aligned}&\Bigl | \frac{ \mathrm{Jac}_{{\Psi _\sigma }}(\frac{r+t}{k},\frac{|t|z}{k})}{\bigl |{\Psi _\sigma }(\frac{r}{k},0)-{\Psi _\sigma }(\frac{r+t}{k},\frac{|t|z}{k}) \bigr |^{N+2s}}-\frac{ \mathrm{Jac}_{{\Psi _\sigma }}(\frac{r-t}{k},\frac{|t|z}{k})}{\bigl |{\Psi _\sigma }(\frac{r}{k},0)-{\Psi _\sigma }(\frac{r-t}{k},\frac{|t|z}{k}) \bigr |^{N+2s}}\Bigr |\\&\quad \le C \Bigl (\frac{|t|}{k}\Bigr )^{1-N-2s}(1+|z|^2)^{-\frac{N+2s}{2}} \end{aligned}$$

for $z\in \frac{k}{|t|}B$, $r\in (0,k\varepsilon ')$ and $t \in (-k\varepsilon +r,k\varepsilon -r)$ and therefore

$$\begin{aligned}&\frac{|{{\mathcal {K}}}_k(r,t)- {{\mathcal {K}}}_k(r,-t)|}{|t|} \le \frac{C}{k} \int _{\frac{k}{|t|}B}(1+|z|^2)^{-\frac{N+2s}{2}} \,dz \nonumber \\&\quad \le \frac{C}{k} \int _{\mathbb {R}^{N-1}}(1+|z|^2)^{-\frac{N+2s}{2}} \,dz \le \frac{C}{k} \ \end{aligned}$$

(6.38)

for $r\in (0,k\varepsilon ')$ and $t \in (-k\varepsilon +r,k\varepsilon -r)$. Moreover, by definition we have

$$\begin{aligned} |{{\mathcal {K}}}_k(r,t)- {{\mathcal {K}}}_k(r,-t)|= 0 \qquad \text {for }t \in \mathbb {R}\setminus (-k\varepsilon -r,k\varepsilon +r), \end{aligned}$$

(6.39)

while for $t \in (-k\varepsilon -r,-k\varepsilon +r) \cup (k\varepsilon -r,k\varepsilon +r)$ we have $|t|\ge k\varepsilon -\varepsilon ' \ge \frac{k\varepsilon }{2}$ and therefore, similarly as in (6.32),

$$\begin{aligned} \frac{|{{\mathcal {K}}}_k(r,t)|}{|t|} \le \frac{C}{|t|} \int _{\frac{k}{|t|}B} \bigl (1+ |z|^2\bigr )^{-\frac{N+2s}{2}} dz \le \frac{C}{k} \int _{\frac{2}{\varepsilon }B} \bigl (1+ |z|^2\bigr )^{-\frac{N+2s}{2}} dz \le \frac{C}{k}. \end{aligned}$$

(6.40)

Note here that the constant $C>0$ on the RHS depends on $\varepsilon$, but this is not a problem. Combining (6.38)–(6.40) and using that $\rho =1-\zeta \in C^\infty _c(\mathbb {R})$, we get

$$\begin{aligned}&|J^2_k(\sigma ,r)| \le \frac{1}{4 } \int _{\mathbb {R}} \frac{|\zeta (r+t)-\zeta (r-t)|}{|t|^{2s}} \frac{|{{\mathcal {K}}}_k(r,t)- {{\mathcal {K}}}_k(r,-t)|}{|t|} dt\\&\quad \le \frac{C}{k} \int _{\mathbb {R}} \frac{|\zeta (r+t)-\zeta (r-t)|}{t^{2s}}d t = \frac{C}{k} \int _{\mathbb {R}} \frac{|\rho (r+t)-\rho (r-t)|}{t^{2s}} d t \le \frac{C(1+r)^{-2s}}{k} \end{aligned}$$

for $k \in \mathbb {N}$, $\sigma \in \partial \Omega$ and $0 \le r <k \varepsilon '$. Hence

$$\begin{aligned} |J^2_k(\sigma ,r)| \le \frac{C}{1+r^{1+2s}}\qquad \text {for }k \in \mathbb {N}, 0 \le r < k \varepsilon ' \end{aligned}$$

(6.41)

and

$$\begin{aligned} \lim _{k \rightarrow \infty }|J^2_k(\sigma ,r)| = 0 \qquad \text {for all }r \ge 0. \end{aligned}$$

(6.42)

Now (6.28) follows by combining (6.31), (6.35) and (6.41). Moreover, (6.29) follows by combining (6.31), (6.36) and (6.42). $\square$

Proof of Proposition 6.3

The proof is completed by combining (6.24) with Lemmas 6.5 and 6.7. $\square$

It finally remains to estimate the function $G_k^2$ in (6.12).

Lemma 6.8

There exists $\varepsilon '>0$ with the property that the function $G_k^2$ defined in (6.13) satisfies

$$\begin{aligned} |k^{-s} G_k^2(\sigma ,r)| \le \frac{C}{1+r^{1+s}}\qquad \text {for }k \in \mathbb {N}, 0 \le r < k \varepsilon ', \sigma \in \partial \Omega \end{aligned}$$

(6.43)

with a constant $C>0$. Moreover,

$$\begin{aligned} \lim _{k \rightarrow \infty } k^{-s} G_k^2(\sigma ,r) = \psi (\sigma ){\tilde{I}}(r) \end{aligned}$$

(6.44)

with

$$\begin{aligned} {\tilde{I}}(r) = c_{1,s} \int _{\mathbb {R}}\frac{ \bigl (r^s_+ - (r+t)^s_+ \bigr )\bigl (\zeta (r)-\zeta (r+t)\bigr )}{|t|^{1+2s}}dt. \end{aligned}$$

Proof

The proof is similar to the one of Proposition 6.3, but there are some differences we need to deal with. First, as in the proof of Proposition 6.3, we choose $\varepsilon ' \in (0,\frac{\varepsilon }{2})$ small enough, so that (6.23) holds. Similarly as in (6.24) we can then write

$$\begin{aligned} G_k^2(\sigma ,r)= c_{N,s} \Bigl ( \widetilde{A}_k(\sigma ,r)+ {\widetilde{B}}_k(\sigma ,r) \Bigr ) \end{aligned}$$

(6.45)

with

$$\begin{aligned} {\widetilde{A}}_{k}(\sigma ,r) := \int _{ \Psi _\sigma ((-\varepsilon ,\varepsilon ) \times B)} \frac{(u({\Psi }(\sigma ,\frac{r}{k}))-u(y))(\zeta (r)-\zeta _k(y))}{|\Psi (\sigma ,\frac{r}{k}) - y|^{N+2s}}\, dy \end{aligned}$$

and

$$\begin{aligned} {\widetilde{B}}_k(\sigma ,r) = \int _{\mathbb {R}^N \setminus \Psi _\sigma ((-\varepsilon ,\varepsilon ) \times B)} \frac{(u({\Psi }(\sigma ,\frac{r}{k}))-u(y))(\zeta (r)-\zeta _k(y))}{|\Psi (\sigma ,\frac{r}{k}) - y|^{N+2s}}\, dy. \end{aligned}$$

As noted in the proof of Lemma 6.5, we have

$$\begin{aligned} |\Psi (\sigma ,\frac{r}{k})- y| \ge \frac{|\sigma - y|}{3} +\varepsilon ' \quad \text {for }y \in \mathbb {R}^N \setminus \Psi _\sigma ((-\varepsilon ,\varepsilon ) \times B), 0<r< k \varepsilon '. \end{aligned}$$

Therefore, since $u \in L^\infty (\mathbb {R}^N)$, we may estimate as in the proof of Lemma 6.5 to get

$$\begin{aligned} |{\widetilde{B}}_k(\sigma ,r)| \le 2 \Vert u\Vert _{L^\infty } \int _{\mathbb {R}^N \setminus \Psi _\sigma ((-\varepsilon ,\varepsilon ) \times B)}\frac{|\rho (r)-\rho _k(y)|}{|\Psi (\sigma ,\frac{r}{k}) - y|^{N+2s}}dy \le C \Bigl (|\rho (r)| + k^{-1}\Bigr ). \end{aligned}$$

Here, as before, the letter C stands for various positive constants. Consequently,

$$\begin{aligned} \lim _{k \rightarrow \infty }k^{-s}|{\widetilde{B}}_k(\sigma ,r)|=0 \qquad \text {for every }\sigma \in \Omega , r\ge 0, \end{aligned}$$

(6.46)

since $\rho$ has compact support in $\mathbb {R}$, and

$$\begin{aligned} k^{-s}|\widetilde{B}_k(\sigma ,r)| \le C k^{-s}\Bigl (|\rho (r)| + k^{-1}\Bigr ) \le \frac{C}{1+r^{1+s}} \qquad \text {for }k \in \mathbb {N}, 0 \le r < k \varepsilon ', \sigma \in \partial \Omega .\nonumber \\ \end{aligned}$$

(6.47)

Hence it remains to estimate ${\widetilde{A}}_{k}(\sigma ,r)$. For this we note that, by the same change of variables as in (6.30), we have

$$\begin{aligned}&{\widetilde{A}}_{k}(\sigma ,r)= \int _{-\varepsilon }^{\varepsilon } \int _{B}\mathrm{Jac}_{\Psi _\sigma }(z,{\tilde{r}})\, \frac{(u(\Psi (\frac{r}{k},0))-u(\Psi _\sigma ({\tilde{r}},z))) (\zeta (r)-\zeta (k {\tilde{r}}))}{|\Psi _\sigma (\frac{r}{k},0) - \Psi _\sigma ({\tilde{r}},z)|^{N+2s}}\,d z d {\tilde{r}} \nonumber \\&\quad = k^s \int _{\mathbb {R}}\frac{\zeta (r)-\zeta (r+t)}{|t|^{1+s}} \widetilde{{{\mathcal {K}}}}_k(r,t)dt \end{aligned}$$

(6.48)

with the kernel

$$\begin{aligned}&{\widetilde{{{\mathcal {K}}}}}_k(r,t)\\&= \left\{ \begin{aligned}&\Bigl (\frac{|t|}{k}\Bigr )^{N+s} \int _{\frac{k}{|t|}B} \frac{\bigl (u(\Psi _\sigma (\frac{r}{k},0))-u(\Psi _\sigma (\frac{r+t}{k},\frac{|t|}{k}z))\bigr ) \mathrm{Jac}_{{\Psi _\sigma }}(\frac{r+t}{k},\frac{|t|z}{k})}{\bigl |{\Psi _\sigma }(\frac{r}{k},0)-{\Psi _\sigma }(\frac{r+t}{k},\frac{|t|z}{k}) \bigr |^{N+2s}}dz,&\quad t\in (-k\varepsilon -r,k\varepsilon -r), \\&0,&\quad t \not \in (-k\varepsilon -r,k\varepsilon -r). \end{aligned} \right. \end{aligned}$$

Since $u\in C^s(\mathbb {R}^N)$ and $\Psi _\sigma$ is Lipschitz, we have

$$\begin{aligned} \bigl |u(\Psi _\sigma (\frac{r}{k},0))-u(\Psi _\sigma (\frac{r+t}{k},\frac{|t|}{k}z)\bigr | \le C \Bigl (\bigl (\frac{|t|}{k}\bigr )^2 +\bigl (\frac{|t z|}{k}\bigr )^{2}\Bigr )^{\frac{s}{2}} \le C\Bigl (\frac{|t|}{k}\Bigr )^s(1+|z|^s), \end{aligned}$$

for $\sigma \in \partial \Omega$, $r \in (-k\varepsilon ,k\varepsilon )$, $t \in (-k\varepsilon -r,k\varepsilon -r) \setminus \{0\}$ and $z \in \frac{k}{|t|}B$. Therefore, by using Lemma 6.6(i), (iv) as in (6.32),

$$\begin{aligned} |{\widetilde{{{\mathcal {K}}}}}_k(r,t)| \le C \int _{\mathbb {R}^{N-1}}(1+|z|^s) (1+|z|^2)^{-\frac{N+2s}{2}}dz \le C \int _{\mathbb {R}^{N-1}}(1+|z|)^{-N-s}dz < \infty .\nonumber \\ \end{aligned}$$

(6.49)

Inserting this estimate in (6.48), we conclude that

$$\begin{aligned} k^{-s} |{\widetilde{A}}_{k}(\sigma ,r)|\le C \int _{\mathbb {R}}\frac{|\zeta (r)-\zeta (r+t)|}{|t|^{1+s}}\,dt = C \int _{\mathbb {R}}\frac{|\rho (r)-\rho (r+t)|}{|t|^{1+s}}\,dt \le \frac{C}{1+r^{1+s}}. \end{aligned}$$

for $k \in \mathbb {N}$, $0 \le r < k \varepsilon '$, $\sigma \in \partial \Omega$. Combining this inequality with (6.47), we obtain (6.43). Moreover, since $u \in C^s_0(\overline{\Omega })$ and $\psi = \frac{u}{\delta ^s} \in C^0(\overline{\Omega })$, we have

$$\begin{aligned} \lim _{k \rightarrow \infty } k^s \left[ u(\Psi _\sigma (\frac{r}{k},0))-u(\Psi _\sigma (\frac{r+t}{k},\frac{|t|}{k}z))\right] = \psi (\sigma )(r^s_+ - (r+t)^s_+ ) \end{aligned}$$

(6.50)

for $\sigma \in \partial \Omega$, $r> 0$ and $t \in \mathbb {R}$ and $z \in \mathbb {R}^{N-1}$. Consequently, arguing as for (6.34) with Lemma 6.6(i)(ii),(iv),(v) and the dominated convergence theorem, we find that

$$\begin{aligned}&\lim _{k \rightarrow \infty } \widetilde{{{\mathcal {K}}}}_k(r,t) = \psi (\sigma )\frac{(r^s_+ - (r+t)^s_+)}{|t|^s}\int _{\mathbb {R}^{N-1}} (1+|z|^2)^{-\frac{N+2s}{2}} dz \nonumber \\&\quad = a_{N,s}\psi (\sigma )\frac{(r^s_+ - (r+t)^s_+)}{|t|^s} \end{aligned}$$

(6.51)

for $\sigma \in \partial \Omega$, $r> 0$ and $t \in \mathbb {R}$ with $a_{N,s}$ given in (6.33). Hence, by (6.48), (6.49), (6.51) and the dominated convergence theorem,

$$\begin{aligned}&\lim _{k \rightarrow \infty } k^{-s} {\widetilde{A}}_{k}(\sigma ,r)= a_{N,s} \psi (\sigma ) \int _{\mathbb {R}}\frac{ (r^s_+ - (r+t)^s_+ )(\zeta (r)-\zeta (r+t))}{|t|^{1+2s}}dt \\&\quad = \frac{a_{N,s}}{c_{1,s}} \psi (\sigma ) {\tilde{I}}(r) = \frac{\psi (\sigma ) {\tilde{I}}(r)}{c_{N,s}}, \end{aligned}$$

where we used again (6.37) for the last equality. Combining this with (6.45) and (6.46), we obtain (6.44). $\square$

We are now ready to complete the

Proof of Proposition 6.1

Combining (6.14), (6.18) and (6.43), we see that there exists $\varepsilon '>0$ with the property that the functions $G_k$ defined in (6.4) satisfy

$$\begin{aligned} \frac{G_k(\sigma ,r)}{k} \le C \frac{r^{s-1} + r^{s-1+\alpha }}{1+r^{1+s}}\qquad \text {for }k \in \mathbb {N}, 0 \le r < k \varepsilon ' \end{aligned}$$

(6.52)

with a constant $C>0$ independent of k and r. Since $s, \alpha \in (0,1)$, the RHS of this inequality is integrable over $[0,\infty )$. Moreover, by (6.15), (6.19) and (6.44),

$$\begin{aligned} \frac{1}{k}G_k(\sigma ,r) \rightarrow [X(\sigma ) \cdot \nu (\sigma )] \psi ^2(\sigma )h'(r)\bigl (r^{s}(-\Delta )^s \zeta (r)-\tilde{I}(r)\bigr ) \end{aligned}$$

(6.53)

for every $r>0$, $\sigma \in \partial \Omega$ as $k \rightarrow \infty$. Next we note that, by a standard computation,

$$\begin{aligned} (-\Delta )^s h(r)= & {} (-\Delta )^s [r^{s}_+ \zeta (r)] = \zeta (r)(-\Delta )^s r^s_+ + r^s_+ (-\Delta )^s \zeta (r)-{\tilde{I}}(r) \nonumber \\= & {} r^s_+ (-\Delta )^s \zeta (r)-{\tilde{I}}(r) \end{aligned}$$

(6.54)

for $r>0$ since $r^s_+$ is an s-harmonic function on $(0,\infty )$ see e.g [1]. Hence, by (6.9), (6.9), (6.52), (6.53), (6.54) and the dominated convergence theorem, we conclude that

$$\begin{aligned} \lim _{k \rightarrow \infty }\int _{\Omega }g_k dx&= \int _0^\infty h'(r)(-\Delta )^s h(r)dr \int _{\partial \Omega }[X(\sigma ) \cdot \nu (\sigma )] \psi ^2(\sigma )d\sigma \\&= \int _{\mathbb {R}} h'(r)(-\Delta )^s h(r)dr \int _{\partial \Omega }[X(\sigma ) \cdot \nu (\sigma )] \psi ^2(\sigma )d\sigma , \end{aligned}$$

as claimed in (6.1). $\square$

Appendix A

Here we give a short proof of the uniqueness of positive minimizers of the problem (1.3) for $1 \le p \le 2$.

Lemma A.1

Let $\Omega \subset \mathbb {R}^N$ be a bounded open set of class $C^{1,1}$, let $p\in [1,2]$, and let $u_1$ and $u_2$ be two positive minimizers of (1.3). Then $u_1=u_2$.

Proof

Suppose by contradiction that there are two different positive minimizers $u_1,u_2$ for the minimization problem. Then, since $\Vert u_1\Vert _{L^p(\Omega )} = \Vert u_2\Vert _{L^p(\Omega )} = 1$, the difference $u_1-u_2$ changes sign. Since moreover $\frac{u_1}{\delta ^s}$ and $\frac{u_2}{\delta ^s}$ are continuous positive functions on $\overline{\Omega }$ by Lemma 2.3, there exists a maximal $\tau \in (0,1)$ with

$$\begin{aligned} \tau u_1 \le u_2 \qquad \text {on }\overline{\Omega }. \end{aligned}$$

Moreover, $\tau u_1 \not \equiv u_2$ since $u_1-u_2$ changes sign. Consequently, $v:= u_2 -\tau u_1$ satisfies $v \ge 0$ on $\overline{\Omega }$ and $v \not \equiv 0$. Moreover, using that $p-1 \in [0,1]$ and $\tau \in (0,1)$, we find that

$$\begin{aligned} (-\Delta )^s v = \lambda \bigl (u_2^{p-1}-\tau u_1^{p-1}\bigr ) \ge \lambda \bigl (u_2^{p-1}-(\tau u_1)^{p-1}\bigr ) \ge 0 \quad \text {in } \Omega ,\qquad v= 0 \qquad \text {in }\mathbb {R}^N \setminus \Omega \end{aligned}$$

with $\lambda := \lambda _{s,p}(\Omega )>0$. Now the strong maximum principle for the fractional Laplacian and the fractional Hopf lemma implies that $v= u_2- \tau u_1$ is strictly positive in $\Omega$ and $\frac{v}{\delta ^s}>0$ on $\partial \Omega$. This contradicts the maximality of $\tau$. Hence uniqueness holds. $\square$