Symmetry via antisymmetric maximum principles in nonlocal problems of variable order

Abstract

We consider the nonlinear problem

$$\begin{aligned} (P)\qquad \left\{ \begin{array}{ll} I u=f(x,u)&{} \quad \text { in } \ \Omega ,\\ u=0 &{}\quad \text { on } \ \mathbb {R}^{N}{\setminus }\Omega \\ \end{array}\right. \end{aligned}$$

in an open bounded set $\Omega \subset \mathbb {R}^{N}$, where $I$ is a nonlocal operator, which may be anisotropic and may have varying order. We assume mild symmetry and monotonicity assumptions on $I, \Omega$ and the nonlinearity $f$ with respect to a fixed direction, say $x_1$, and we show that any nonnegative weak solution $u$ of $(P)$ is symmetric in $x_1$. Moreover, we have the following alternative: Either $u\equiv 0$ in $\Omega$, or $u$ is strictly decreasing in $|x_1|$. The proof relies on new maximum principles for antisymmetric supersolutions of an associated class of linear problems.

1 Introduction

In this work, we study the following class of nonlocal and semilinear Dirichlet problems in a bounded open set $\Omega \subset \mathbb {R}^N$:

$$\begin{aligned} (P)\qquad \left\{ \begin{array}{ll} I u=f(x,u)&{} \quad \text { in } \ \Omega ;\\ u=0&{}\quad \text { on } \ \mathbb {R}^{N}{\setminus }\Omega . \end{array}\right. \end{aligned}$$

Here, the nonlinearity $f:\Omega \times \mathbb {R}\rightarrow \mathbb {R}$ is a measurable function with properties to be specified later, and $I$ is a nonlocal linear operator. Due to various applications in physics, biology and finance with anomalous diffusion phenomena, nonlocal problems have gained enormous attention recently. In particular, problem $(P)$ has been studied with $I=(-\Delta )^{\frac{\alpha }{2}}$, the fractional Laplacian of order $\alpha \in (0,2)$. In this case, special properties of the fractional Laplacian have been used extensively to study existence, regularity and symmetry of solutions to $(P)$. In particular, some approaches rely on available Green function representations associated with $(-\Delta )^{\frac{\alpha }{2}}$, (see, e.g., [6, 7, 10–12, 14]), whereas other techniques are based on a representation of $(-\Delta )^{\frac{\alpha }{2}}$ as a Dirichlet-to-Neumann map (see, e.g., [8, 9, 17]). These useful features of the fractional Laplacian are closely linked to its isotropy and its scaling laws. However, in the modeling of anisotropic diffusion phenomena and of processes which do not exhibit similar properties, it is necessary to study more general nonlocal operators $I$. In this spirit, general classes of nonlocal operators have been considered, e.g., in [2, 3, 15, 16, 19, 25].

In the present work, we consider $(P)$ for a class of nonlocal operators $I$, which includes the fractional Laplacian but also more general operators which may be anisotropic and may have varying order. More precisely, the class of operators $I$ in $(P)$ is related to nonnegative nonlocal bilinear forms of the type

$$\begin{aligned} {\fancyscript{J}}(u,v)=\frac{1}{2}\int \limits _{\mathbb {R}^N}\int \limits _{\mathbb {R}^{N}}(u(x)-u(y))(v(x)-v(y))J(x-y)\ dxdy \end{aligned}$$

(1.1)

with a measurable function $J:\mathbb {R}^N {\setminus } \{0\} \rightarrow [0,\infty )$. We assume that $J$ is even, i.e., $J(-z)=J(z)$ for $z \in \mathbb {R}^{N} {\setminus } \{0\}$. Moreover, we assume the following integral condition:

$$\begin{aligned} (J1) \qquad \quad \quad \int \limits _{\mathbb {R}^{N}} \min (1,|z|^2)\, J(z)\, dz < \infty \qquad \quad \text {and} \qquad \quad \int \limits _{\mathbb {R}^{N}}J(z)\ dz = \infty . \end{aligned}$$

By similar arguments as in the recent paper [16], we shall see in Sect. 2 below that this assumption ensures that ${\fancyscript{J}}$ is a closed and symmetric quadratic form in $L^2(\Omega )$ with a dense domain given by

$$\begin{aligned} {\fancyscript{D}}(\Omega ):=\left\{ u:\mathbb {R}^{N}\rightarrow \mathbb {R}\hbox { measurable}\;:\; {\fancyscript{J}}(u,u)<\infty \quad \text {and}\quad u\equiv 0\hbox { on }\mathbb {R}^{N}{\setminus } \Omega \right\} . \end{aligned}$$

(1.2)

Here, and in the following, we identify $L^2(\Omega )$ with the space of functions $u \in L^2(\mathbb {R}^N)$ with $u \equiv 0$ on $\mathbb {R}^N {\setminus } \Omega$. Consequently, ${\fancyscript{J}}$ is the quadratic form of a unique self-adjoint operator $I$ on $L^2(\Omega )$, which also satisfies

$$\begin{aligned}{}[I u](x)= \lim _{\varepsilon \rightarrow 0} \int \limits _{|y-x|\ge \varepsilon } [u(x)-u(y)]J(x-y)\,dy \qquad \text {for } \quad u \in {\fancyscript{C}}^2_c(\Omega ),\quad x \in \mathbb {R}^N \end{aligned}$$

see Corollary 2.4 below. One may study solutions $u$ of $(P)$ in strong sense, requiring that $u$ is contained in the domain of the operator $I$. However, it is more natural to consider the weaker notion of solutions given by the quadratic form ${\fancyscript{J}}$ itself. More precisely, we call a function $u\in {\fancyscript{D}}(\Omega )$ a solution of $(P)$ if the integral $\int \nolimits _{\Omega }f(x,u(x))\varphi (x)\ dx$ exists for all $\varphi \in {\fancyscript{D}}(\Omega )$ and

$$\begin{aligned} {\fancyscript{J}}(u,\varphi )=\int \limits _{\Omega }f(x,u(x))\varphi (x)\ dx \qquad \text {for all } \quad \varphi \in {\fancyscript{D}}(\Omega ). \end{aligned}$$

We note that the fractional Laplacian $I:= (-\Delta )^{\alpha /2}$ corresponds to the kernel $J(z)= c_{N,\alpha } |z|^{-N-\alpha }$ with $c_{N,\alpha }= \alpha (2-\alpha )\pi ^{-N/2}2^{\alpha -2}\frac{\Gamma (\frac{N+\alpha }{2})}{\Gamma (2-\frac{\alpha }{2})}$. Our paper is motivated by recent symmetry results for nonlinear equations involving the fractional Laplacian (see [4, 6, 10, 11, 14, 20]). More precisely, we present a general approach, based on maximum principles for antisymmetric functions, to investigate symmetry properties of bounded nonnegative solutions of $(P)$ in bounded Steiner symmetric open sets $\Omega$. We claim that this approach is simpler and more general than the techniques applied in the papers cited above. In particular, it also applies to anisotropic operators and operators of variable order. To state our main symmetry result, we first introduce the following geometric assumptions on $J$ and the set $\Omega$.

$(D)$ :

$\Omega \subset \mathbb {R}^N$ is an open bounded set which is Steiner symmetric in $x_1$, i.e., for every $x\in \Omega$ and $s \in [-1,1]$, we have $(sx_1,x_2,\ldots ,x_N) \in \Omega$.

$(J2)$ :

The kernel $J$ is strictly monotone in $|x_1|$, i.e., for all $z' \in \mathbb {R}^{N-1}$, $s,t \in \mathbb {R}$ with $|s| <|t|$, we have $J(s,z') > J(t,z')$.

Note that $(J2)$ in particular implies that $J$ is positive on $\mathbb {R}^N {\setminus } \{0\}$. We may now state our main symmetry result.

Theorem 1.1

Let $(J1), (J2)$ and $(D)$ be satisfied, and assume that the nonlinearity $f$ has the following properties.

$(F1)$ :

$f: \Omega \times \mathbb {R}\rightarrow \mathbb {R}$, $(x,u)\mapsto f(x,u)$ is a Carathéodory function such that for every bounded set $K \subset \mathbb {R}$, there exists $L=L(K)>0$ with

$$\begin{aligned} \sup \limits _{x\in \Omega } |f(x,u)-f(x,v)|\le L |u-v| \quad \text { for } \quad u,v \in K. \end{aligned}$$

$(F2)$ :

$f$ is symmetric in $x_1$ and monotone in $|x_1|$, i.e., for every $u \in \mathbb {R}$, $x \in \Omega$ and $s \in [-1,1]$, we have $f(s x_{1},x_2,\ldots ,x_N,u) \ge f(x,u)$.

Then, every nonnegative solution $u \in L^\infty (\Omega ) \cap {\fancyscript{D}}(\Omega )$ of $(P)$ is symmetric in $x_1$. Moreover, either $u \equiv 0$ in $\mathbb {R}^{N}$, or $u$ is strictly decreasing in $|x_1|$ and therefore satisfies

$$\begin{aligned} \mathop {{{\mathrm{essinf}}}}\limits _{K}\, u>0\qquad \text {for every compact set } K \subset \Omega . \end{aligned}$$

(1.3)

Here, and in the following, if $\Omega$ satisfies $(D)$ and $u: \Omega \rightarrow \mathbb {R}$ is measurable, we say that $u$ is

• symmetric in $x_1$ if $u(-x_{1},x')=u(x_{1},x')$ for almost every $x= (x_{1},x')\in \Omega$.

• strictly decreasing in $|x_1|$ if for every $\lambda \in \mathbb {R}{\setminus } \{0\}$ and every compact set $K \subset \{x \in \Omega \,:\, \frac{x_1}{\lambda } > 1\}$ we have

  $$\begin{aligned} \mathop {{{\mathrm{essinf}}}}\limits _{x \in K}\, \bigl [u(2\lambda -x_1, x_2,\ldots ,x_N) -u(x)\bigr ] >0. \end{aligned}$$

Remark 1.2

We wish to single out a particular class of operators satisfying $(J1)$ and $(J2)$. Let $\alpha ,\beta \in (0,2)$, $c>1$ and consider a measurable map $k:(0,\infty )\rightarrow (0,\infty )$ such that

$$\begin{aligned} \frac{\rho ^{-N}}{c} \le k(\rho )\le c\rho ^{-N-\alpha }\quad \text {for } \rho \le 1\hbox { and}\qquad k(\rho ) \le c\rho ^{-N-\beta }\quad \text {for }\quad \rho >1. \end{aligned}$$

Suppose moreover that $k$ is strictly decreasing on $(0,\infty )$, and let $|\cdot |_\sharp$ denote a norm on $\mathbb {R}^N$ with the property that $|(s,z')|_\sharp <|(t,z')|_\sharp$ for every $s,t \in \mathbb {R}$ with $|s|<|t|$ and $z' \in \mathbb {R}^{N-1}$. Then, the kernel

$$\begin{aligned} J: \mathbb {R}^N {\setminus } \{0\} \rightarrow \mathbb {R},\qquad J(z)= k(|z|_\sharp ) \end{aligned}$$

(1.4)

satisfies $(J1)$ and $(J2)$. As remarked before, the case where $|\cdot |_{\sharp }=|\cdot |$ is the euclidean norm on $\mathbb {R}^N$ and $k(\rho )= c_{N,\alpha } \rho ^{-N-\alpha }$ corresponds to the fractional Laplacian $I=(-\Delta )^{\alpha /2}$. The class defined here also includes operators of order varying between 0 and $\alpha \in (0,2)$. In particular, zero-order operators are admissible. Moreover, the choice of non-Euclidean norms $|\cdot |_{\sharp }$ leads to anisotropic operators. In particular, for $1 \le p < \infty$, the norm

$$\begin{aligned} |x|_\sharp = |x|_p:= \left( \sum \limits _{i=1}^N |x_i|^p\right) ^{1/p} \qquad \text {for }\quad x \in \mathbb {R}^N \end{aligned}$$

(1.5)

has the required properties. We also remark that the very recent work [19] of Kassmann and Mimica shows that harmonic functions with respect to the operator in (1.4) are continuous (a probabilistic definition of harmonicity is used in [19]). We therefore conjecture that in this case, solutions $u \in L^\infty (\Omega ) \cap {\fancyscript{D}}(\Omega )$ of $(P)$ are also continuous if we assume in addition that $f$ is locally bounded in $u$ uniformly in $x \in \Omega$. Up to now, this seems open.

As a direct consequence of Theorem 1.1, we have the following. Here, $e_j \in \mathbb {R}^N$ denotes the $j$-th coordinate vector for $j=1,\ldots ,N$.

Corollary 1.3

Let $J(z)=k(|z|_p)$, where $k$ is as in Remark 1.2, $1 \le p< \infty$ and $|\cdot |_p$ is given in (1.5).

• (i) Let $\Omega \subset \mathbb {R}^{N}$ be Steiner symmetric in $x_1,\ldots ,x_N$, i.e., for every $x\in \Omega$, $j=1,\ldots ,N$ and $s \in [0,2]$, we have $x- s x_j e_j \in \Omega$. Moreover, let $f$ fulfill $(F1)$ and be symmetric and monotone in $x_1,\ldots ,x_N$, i.e., for every $u \in \mathbb {R}$, $x \in \Omega$, $j=1,\ldots ,N$ and $s \in [0,2]$, we have $f(x-s x_j e_j, u) \ge f(x,u)$. Then, every nonnegative solution $u\in L^\infty (\Omega ) \cap {\fancyscript{D}}(\Omega )$ of $(P)$ is symmetric in $x_1,\ldots ,x_N$. Moreover, either $u \equiv 0$ in $\mathbb {R}^{N}$ or $u$ is strictly decreasing in $|x_1|,\ldots ,|x_N|$ and therefore satisfies (1.3).

• (ii) If $p=2$, $\Omega \subset \mathbb {R}^{N}$ is a ball centered in 0 and $f$ fulfills $(F1)$, $(F2)$ and is radial in $x$, i.e., $f(x,u)=f(|x|e_1,u)$ for $x\in \Omega$, then every nonnegative solution $u\in L^\infty (\Omega ) \cap {\fancyscript{D}}(\Omega )$ of $(P)$ is radially symmetric. Moreover, either $u \equiv 0$ in $\mathbb {R}^{N}$ or $u$ is strictly decreasing in $|x|$ and therefore satisfies (1.3).

In the special case where $I=(-\Delta )^{\frac{\alpha }{2}}$, $\alpha \in (0,2)$, Theorem 1.1 has been obtained by the authors in [20, Corollary 1.2] as a corollary of result on asymptotic symmetry for the corresponding parabolic problem. While some of the parabolic estimates in [20] are not available for the class of nonlocal operators considered here, we will be able to formulate elliptic counterparts of some of the tools from [20] in the present setting. Independently from our work [20], a weaker variant of Theorem 1.1 in the special case $I=(-\Delta )^{\frac{\alpha }{2}}$, restricted to strictly positive solutions, is proved in the very recent preprint [4, Theorem 1.2], where also related problems for the fractional Laplacian with singular local linear terms are considered. Corollary 1.3(ii) for $I=(-\Delta )^{\frac{\alpha }{2}}$, $\alpha \in (0,2)$ has been proved first by Birkner, López-Mimbela and Wakolbinger [6] for $I=(-\Delta )^{\frac{\alpha }{2}}$ and a nonlinearity $f=f(u)$ which is nonnegative and increasing. In the very recent papers [10, 14], Corollary 1.3(ii) is proved for strictly positive solutions in the case $I=(-\Delta )^{\frac{\alpha }{2}}$ under different assumptions on $f$. The proofs in these papers rely on the explicit form of the Green function associated with $(-\Delta )^{\frac{\alpha }{2}}$ in balls.

In order to explain the difference between considering nonnegative or positive solutions, we point out that the conclusion (1.3) can be seen as a strong maximum principle for bounded solutions of $(P)$ in open sets satisfying $(D)$, which has no counterpart in the context of the corresponding Dirichlet problem

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u=f(x,u)&{} \quad \text { in } \quad \Omega ;\\ u=0&{}\quad \text { on }\quad \partial \Omega . \end{array}\right. \end{aligned}$$

(1.6)

Note that we do not assume $\Omega$ to be connected in Theorem 1.1, but even in domains $\Omega \subset \mathbb {R}^N$ the assumptions $(D)$ and $(F1)$, $(F2)$ do not guarantee that nonnegative solutions of (1.6) are either strictly positive or identically zero in $\Omega$, see, e.g., [21] for examples for nonnegative solutions of (1.6) with interior zeros. The positivity property (1.3) can be seen as a consequence of the long-range nonlocal interaction enforced by $(J2)$. Note that $(J2)$ is not satisfied for kernels of the form

$$\begin{aligned} z \mapsto J(z)= 1_{B_r(0)}|z|^{-N-\alpha }\quad \text {with } \quad \alpha \in (0,2),\quad r>0. \end{aligned}$$

(1.7)

It is therefore natural to ask whether a result similar to Theorem 1.1 also holds for kernels of the type (1.7), which vanish outside a compact set and therefore model short-range nonlocal interaction. Related to this case, we have to follow result for a.e. positive solutions of $(P)$ in $\Omega$.

Theorem 1.4

Let $\Omega \subset \mathbb {R}^{N}$ satisfy $(D)$, and let the even kernel $J: \mathbb {R}^N {\setminus } \{0\} \rightarrow [0,\infty )$ satisfy $(J1)$ and

$(J2)'$ :

For all $z' \in \mathbb {R}^{N-1}$, $s,t \in \mathbb {R}$ with $|s| \le |t|$ we have $J(s,z') \ge J(t,z')$. Moreover, there is $r_0>0$ such that

$$\begin{aligned} J(s,z') > J(t,z') \qquad \text {for all } z' \in \mathbb {R}^{N-1}\hbox { and } s,t \in \mathbb {R},\hbox { with } |z'| \le r_0\hbox { and } |s| <|t|\le r_0. \end{aligned}$$

Furthermore, suppose that the nonlinearity satisfies $(F1)$ and $(F2)$. Then, every a.e. positive solution $u\in L^\infty (\Omega ) \cap {\fancyscript{D}}(\Omega )$ of $(P)$ is symmetric in $x_1$ and strictly decreasing in $|x_1|$ on $\Omega$. Consequently, it satisfies (1.3).

Note that the kernel class given by (1.7) satisfies $(J1)$ and $(J2)'$. We recall that Gidas, Ni and Nirenberg [18] proved the corresponding symmetry result for strictly positive solutions of (1.6) under some restrictions on $\Omega$ which were then removed in [5]. These results rely on the moving plane method which, in other variants, had already been introduced in [1, 24]. For nonlocal problems involving the fractional Laplacian, the moving plane method was used in a stochastic framework by Birkner, López-Mimbela and Wakolbinger in the above-mentioned paper [6]. Chen, Li and Ou [11] used the explicit form of the inverse of the fractional Laplacian to prove symmetry results for $I=(-\Delta )^{\frac{\alpha }{2}}$ and $f(u)=u^{(N+\alpha )/(N-\alpha )}$ in $\mathbb {R}^{N}$. For this, they developed a variant of the moving plane method for integral equations. Similar methods were used in the above-mentioned papers [10, 14].

The results on the present paper rely on a different variant of the moving plane method which partly extends recent techniques of [13, 20, 23] and, independently, [4]. More precisely, we show that $(J1)$ and $(J2)$—or, alternatively, $(J2)'$—are sufficient assumptions for the bilinear form ${\fancyscript{J}}$ to provide maximum principles for antisymmetric solutions of associated linear operator inequalities in weak form, see Sect. 3. Here, antisymmetry refers to a reflection at a given hyperplane. Combining different (weak and strong) versions of these maximum principles, we then develop a framework for the moving plane method for nonnegative solutions of $(P)$, which are not necessarily strictly positive. The approach seems more direct and more flexible than the ones in [10, 11, 14] since it does not depend on Green function representations.

The paper is organized as follows. In Sect. 2, we collect useful properties of the nonlocal bilinear forms which we consider. Section 3 is devoted to classes of linear problems related to $(P)$ and hyperplane reflections. In particular, we prove a small volume type maximum principle and a strong maximum principle for antisymmetric supersolutions of these problems. In Sect. 4, we complete the proof of Theorem 1.1, and in Sect. 5, we complete the proof of Theorem 1.4.

2 Preliminaries

We fix some notation. For subsets $D,U \subset \mathbb {R}^N$, we write $dist (D,U):= \inf \{|x-y|\,:\, x \in D,\, y \in U\}$. If $D= \{x\}$ is a singleton, we write $dist (x,U)$ in place of $dist (\{x\},U)$. For $U\subset \mathbb {R}^{N}$ and $r>0$, we consider $B_{r}(U):=\{x\in \mathbb {R}^{N}\;:\; dist (x,U)<r\}$, and we let, as usual $B_r(x)=B_{r}(\{x\})$ be the open ball in $\mathbb {R}^{N}$ centered at $x \in \mathbb {R}^N$ with radius $r>0$. For any subset $M \subset \mathbb {R}^N$, we denote by $1_M: \mathbb {R}^N \rightarrow \mathbb {R}$ the characteristic function of $M$ and by $diam (M)$ the diameter of $M$. If $M$ is measurable, $|M|$ denotes the Lebesgue measure of $M$. Moreover, if $w: M \rightarrow \mathbb {R}$ is a function, we let $w^+= \max \{w,0\}$ and $w^-=-\min \{w,0\}$ denote the positive and negative part of $w$, respectively.

Throughout the remainder of the paper, we assume that $J:\mathbb {R}^N {\setminus } \{0\}\rightarrow [0,\infty )$ is even and satisfies $(J1)$. We let ${\fancyscript{J}}$ be the corresponding quadratic form defined in (1.1), and for an open set $\Omega \subset \mathbb {R}$, we consider ${\fancyscript{D}}(\Omega )$ as defined in (1.2). It follows from $(J1)$ that $J$ is positive on a set of positive measure. Thus, by [16, Lemma 2.7], we have ${\fancyscript{D}}(\Omega ) \subset L^2(\Omega )$ and

$$\begin{aligned} \Lambda _{1}(\Omega ):=\inf _{u\in {\fancyscript{D}}(\Omega )}\frac{{\fancyscript{J}}(u,u)}{\Vert u\Vert ^2_{L^{2}(\Omega )}} \, >\,0 \qquad \text {for every open bounded set } \Omega \subset \mathbb {R}^{N}, \end{aligned}$$

(2.1)

which amounts to a Poincaré–Friedrichs type inequality. We will need lower bounds for $\Lambda _1(\Omega )$ in the case where $|\Omega |$ is small. For this, we set

$$\begin{aligned} \Lambda _1(r):= \inf \{ \Lambda _{1}(\Omega ): \; \Omega \subset \mathbb {R}^N\hbox { open}, |\Omega |=r\} \qquad \text {for }\quad r>0. \end{aligned}$$

Lemma 2.1

We have $\Lambda _{1}(r) \rightarrow \infty$ as $r \rightarrow 0$.

Proof

Let

$$\begin{aligned} J_c:= \{ z \in \mathbb {R}^N {\setminus } \{0\}\,:\, J(z) \ge c\} \qquad \text {and}\qquad J^c:= \{ z \in \mathbb {R}^N {\setminus } \{0\}\,:\, J(z) < c\} \end{aligned}$$

for $c \in [0,\infty ]$. We also consider the decreasing rearrangement $d:(0,\infty ) \rightarrow [0,\infty ]$ of $J$ given by $d(r)= \sup \{c \ge 0 \,:\, |J_c| \ge r \}$. We first note that

$$\begin{aligned} |J_{d(r)}| \ge r \qquad \text {for every } \quad r>0 \end{aligned}$$

(2.2)

Indeed, this is obvious if $d(r)=0$, since $J_0= \mathbb {R}^N {\setminus } \{0\}$. If $d(r)>0$, we have $|J_c| \ge r$ for every $c < d(r)$ by definition, whereas $|J_c| < \infty$ for every $c>0$ as a consequence of the fact that $J \in L^{1}(\mathbb {R}^{N}{\setminus } B_1(0))$ by $(J1)$. Consequently, since $J_{d(r)}= \mathop {\bigcap }\nolimits _{c < d(r)} J_c$, we have $|J_{d(r)}| = \inf \nolimits _{c < d(r)} |J_c| \ge r.$ Next we claim that

$$\begin{aligned} \Lambda _1(r) \ge \int \limits _{J^{d(r)}} J(z)\,dz \qquad \text {for } \quad r>0. \end{aligned}$$

(2.3)

Indeed, let $r>0$ and $\Omega \subset \mathbb {R}^N$ be measurable with $|\Omega |=r$. For $u\in {\fancyscript{D}}(\Omega )$, we have

$$\begin{aligned} {\fancyscript{J}}(u,u)&=\frac{1}{2}\int \limits _{\mathbb {R}^{N}}\int \limits _{\mathbb {R}^{N}}(u(x)-u(y))^2J(x-y)\ dxdy \nonumber \\&=\frac{1}{2}\int \limits _{\Omega }\int \limits _{\Omega }(u(x)-u(y))^2 J(x-y)\ dxdy+\int \limits _{\Omega }u^2(x)\int \limits _{\mathbb {R}^{N}{\setminus } \Omega } J(x-y)\ dy \ dx \nonumber \\&\ge \inf _{x\in \Omega }\biggl (\;\int \limits _{\;\mathbb {R}^{N}{\setminus } \Omega _x} J(y)\ dy\biggr )\Vert u\Vert ^2_{L^{2}(\Omega )} \end{aligned}$$

(2.4)

with $\Omega _x:=x+\Omega$. Let $d:=d(r)$. Since $|J_{d}| \ge r = |\Omega |$ by (2.2), we have $|J_d {\setminus } \Omega _x| \ge |\Omega _x {\setminus } J_d|$, and thus, for every $x \in \Omega$,

$$\begin{aligned} \int \limits _{\mathbb {R}^{N}{\setminus } \Omega _x} J(y)\ dy&=\int \limits _{\mathbb {R}^N {\setminus } J_d} J(y)\ dy +\int \limits _{J_d{\setminus } \Omega _x}J(y)\ dy- \int \limits _{\Omega _{x}{\setminus } J_d}J(y)\ dy\\&\ge \int \limits _{J^d} J(y)\ dy + \Bigl (|J_d{\setminus } \Omega _x| - |\Omega _{x}{\setminus } J_d|\Bigr )d \ge \int \limits _{J^d} J(y)\ dy. \end{aligned}$$

Combining this with (2.4), we obtain (2.3), as claimed. As a consequence of the second property in $(J1)$, the decreasing rearrangement of $J$ satisfies $d(r) \rightarrow \infty$ as $r \rightarrow 0$ and

$$\begin{aligned} \int \limits _{J^{d(r)}} J(y)\ dy \rightarrow \infty \qquad \text {as }\quad r \rightarrow 0. \end{aligned}$$

Together with (2.3), this shows the claim. $\square$

Proposition 2.2

Let $\Omega \subset \mathbb {R}^N$ be open and bounded. Then, ${\fancyscript{D}}(\Omega )$ is a Hilbert space with the scalar product ${\fancyscript{J}}$.

Proof

We argue similarly as in the proof of [16, Lemma 2.3]. Let $(u_n)_n\subset {\fancyscript{D}}(\Omega )$ be a Cauchy sequence. By (2.1) and the completeness of $L^2(\Omega )$, we have that $u_n\rightarrow u\in L^{2}(\Omega )$ for a function $u\in L^{2}(\Omega )$. Hence, there exists a subsequence such that $u_{n_k}\rightarrow u$ almost everywhere in $\Omega$ as $k \rightarrow \infty$. By Fatou’s Lemma, we therefore have that

$$\begin{aligned} {\fancyscript{J}}(u,u)\le \liminf _{k\rightarrow \infty }{\fancyscript{J}}(u_{n_k},u_{n_k})\le \sup _{k\in \mathbb {N}}{\fancyscript{J}}(u_{n_k},u_{n_k})<\infty , \end{aligned}$$

so that $u \in {\fancyscript{D}}(\Omega )$. Applying Fatou’s Lemma again, we find that

$$\begin{aligned} {\fancyscript{J}}(u_{n_k}-u,u_{n_k}-u)&\le \liminf _{j \rightarrow \infty }{\fancyscript{J}}(u_{n_k}-u_{n_j},u_{n_k}-u_{u_j})\\&\le \sup _{j \ge k}{\fancyscript{J}}(u_{n_k}-u_{n_j},u_{n_k}-u_{u_j}) \quad \text {for } \quad k \in \mathbb {N}. \end{aligned}$$

Since $(u_{n})_n$ is a Cauchy sequence with respect to the scalar product ${\fancyscript{J}}$, it thus follows that $\lim \limits _{k \rightarrow \infty }u_{n_k} = u$ and therefore also $\lim \limits _{n \rightarrow \infty }u_{n} = u$ in ${\fancyscript{D}}(\Omega )$. This shows the completeness of ${\fancyscript{D}}(\Omega )$. $\square$

Proposition 2.3

• (i) We have ${\fancyscript{C}}^{0,1}_{c}(\mathbb {R}^{N}) \subset {\fancyscript{D}}(\mathbb {R}^N)$.

• (ii) Let $v \in {\fancyscript{C}}_c^2(\mathbb {R}^N)$. Then, the principle value integral

  $$\begin{aligned}{}[Iv](x):= P.V. \int \limits _{\mathbb {R}^N} (v(x)-v(y)) J(x-y)\,dy = \lim _{\varepsilon \rightarrow 0} \int \limits _{|x-y|\ge \varepsilon } (v(x)-v(y)) J(x-y)\,dy \end{aligned}$$

  (2.5)

  exists for every $x \in \mathbb {R}^N$. Moreover, $I v \in L^\infty (\mathbb {R}^N)$, and for every bounded open set $\Omega \subset \mathbb {R}^N$ and every $u \in {\fancyscript{D}}(\Omega )$, we have

  $$\begin{aligned} {\fancyscript{J}}(u,v)= \int \limits _{\mathbb {R}^N} u(x) [Iv](x)\,dx. \end{aligned}$$

Proof

(i) Let $u\in {\fancyscript{C}}^{0,1}_{c}(\mathbb {R}^{N})$, and let $K>0$, $R>2$ be such that $supp (u) \subset B_{R-2}(0)$,

$$\begin{aligned} |u(x)|\le K \quad \text {and}\quad |u(x)-u(y)|\le K|x-y| \qquad \text {for }\quad x,\quad y \in \mathbb {R}^N,\quad x \not =y. \end{aligned}$$

Then, as a consequence of $(J1)$,

$$\begin{aligned} 2{\fancyscript{J}}(u,u)&=\int \limits _{ B_{R}(0)}\int \limits _{ B_{R}(0)}(u(x)-u(y))^2 J(x-y)\ dxdy\\&\quad +\,2\int \limits _{ B_{R}(0)}u^{2}(x)\int \limits _{\mathbb {R}^{N}{\setminus } B_{R}(0)}J(x-y)\ dydx\\&\le K^2 \int \limits _{ B_{R}(0)}\int \limits _{ B_{R}(0)}|x-y|^{2} J(x-y)\ dx dy\\&\quad +\,2K^2 \int \limits _{ B_{R-2}(0)}\;\int \limits _{\mathbb {R}^{N}{\setminus } B_{R}(0)}J(x-y)\ dydx\\&\le 2 K^2|B_R(0)|\left( \,\, \int \limits _{ B_{2R}(0)}|z|^{2} J(z)\ dz +\int \limits _{\mathbb {R}^{N}{\setminus } B_{1}(0)}J(z)\ dz \right) <\infty \end{aligned}$$

and thus $u \in {\fancyscript{D}}(\mathbb {R}^N)$.

(ii) Since $v\in {\fancyscript{C}}^{2}_{c}(\mathbb {R}^{N})$, there exist a constant $K>0$ such that

$$\begin{aligned} |2v(x)-v(x+z)-v(x-z)|\le K \min (|z|^2,1) \qquad \text {for all }\quad x,\quad z \in \mathbb {R}^N. \end{aligned}$$

(2.6)

Hence, by the first inequality in $(J1)$, we have

$$\begin{aligned} \int \limits _{\mathbb {R}^N} |2 v(x)-v(x+z)-v(x-z)|J(z)\,dz \;\le \;\tilde{K}:=K \int \limits _{\mathbb {R}^N} \min (|z|^2,1)\,J(z)\,dz \, < \, \infty .\qquad \end{aligned}$$

(2.7)

Put $h(x,y):= (v(x)-v(y))J(x-y)$ for $x,y \in \mathbb {R}^N$, $x \not = y$. For every $x \in \mathbb {R}^N$, $\varepsilon >0$, we then have, since $J$ is even,

$$\begin{aligned} \int \limits _{|y-x|\ge \varepsilon } h(x,y)\,dy&= \int \limits _{|z|\ge \varepsilon } [v(x)-v(x+z)] J(z)\,dz= \int \limits _{|z|\ge \varepsilon } [v(x)-v(x-z)] J(z)\,dz \nonumber \\&= \frac{1}{2} \int \limits _{|z|\ge \varepsilon } [2 v(x)-v(x+z)-v(x-z)]J(z)\,dz. \end{aligned}$$

(2.8)

By (2.7) and Lebesgue’s theorem, we thus conclude the existence of the limit

$$\begin{aligned}{}[Iv](x)=\lim _{\varepsilon \rightarrow 0} \int \limits _{|z|\ge \varepsilon }h(x,y)\,dy =\frac{1}{2} \int \limits _{\mathbb {R}^N} [2 v(x)-v(x+z)-v(x-z)]J(z)\,dz, \end{aligned}$$

whereas $|[Iv](x)|\le \frac{\tilde{K}}{2}$ for every $x \in \mathbb {R}^N$. Hence, $Iv \in L^\infty (\mathbb {R}^N)$. Next, let $\Omega \subset \mathbb {R}^N$ be open and bounded and $u\in {\fancyscript{D}}(\Omega )$, so that also $u \in L^2(\Omega )$. Since, by (2.7) and (2.8),

$$\begin{aligned} \Bigl |\int \limits _{|y-x|\ge \varepsilon } h(x,y)\,dy \Bigr | \le \frac{\tilde{K}}{2} \qquad \text {for every }\quad x \in \mathbb {R}^N,\quad \varepsilon >0, \end{aligned}$$

Lebesgue’s Theorem implies that

$$\begin{aligned} {\fancyscript{J}}(u,v)&= \frac{1}{2} \lim _{\varepsilon \rightarrow 0} \int \limits _{|x-y|\ge \varepsilon } (u(x)-u(y))h(x,y) \,dx \,dy\\&= \lim _{\varepsilon \rightarrow 0} \int \limits _{\mathbb {R}^N} u(x) \int \limits _{|y-x| \ge \varepsilon } h(x,y) \,dy dx = \int \limits _{\mathbb {R}^N} u(x) \left[ \, \lim _{\varepsilon \rightarrow 0} \int \limits _{|y-x| \ge \varepsilon } h(x,y) \,dy\right] dx \\&=\int \limits _{\mathbb {R}^N} u(x) [Iv](x)\,dx. \end{aligned}$$

The proof is finished.$\square$

Corollary 2.4

Let $\Omega \subset \mathbb {R}^{N}$ be open and bounded. Then, ${\fancyscript{J}}$ is a closed quadratic form with dense form domain ${\fancyscript{D}}(\Omega )$ in $L^{2}(\Omega )$. Consequently, ${\fancyscript{J}}$ is the quadratic form of a unique self-adjoint operator $I$ in $L^2(\Omega )$. Moreover, $C_c^2(\Omega )$ is contained in the domain of $I$, and for every $v \in {\fancyscript{C}}_c^2(\Omega )$, the function $Iv \in L^2(\Omega )$ is a.e. given by (2.5).

Proof

Since ${\fancyscript{C}}^{0,1}_{c}(\Omega ) \subset L^2(\Omega )$ is dense, ${\fancyscript{D}}(\Omega )$ is a dense subset of $L^2(\Omega )$ by Proposition 2.3(i). Moreover, the quadratic form ${\fancyscript{J}}$ is closed in $L^2(\Omega )$ as a consequence of (2.1) and Lemma 2.2. Hence, ${\fancyscript{J}}$ is the quadratic form of a unique self-adjoint operator $I$ in $L^2(\Omega )$ (see, e.g., [22, Theorem VIII.15, p. 278]). Moreover, for every $v \in {\fancyscript{C}}_c^2(\Omega )$, $u \in {\fancyscript{D}}(\Omega )$, we have $|J(u,v)| \le |\Omega | \Vert Iv\Vert _{L^\infty (\Omega )} \Vert u\Vert _{L^2(\Omega )}$ by Proposition 2.3(ii). Consequently, $v$ is contained in the domain of $I$ and satisfies $J(u,v) = \int _{\mathbb {R}^N}u [Iv]\,dx$ for every $u \in {\fancyscript{D}}(\Omega )$. From Proposition 2.3(ii), it then follows that $Iv$ is a.e. given by (2.5).$\square$

Next, we wish to extend the definition of ${\fancyscript{J}}(v,\varphi )$ to more general pairs of functions $(v,\varphi )$. In the following, for a measurable subset $U' \subset \mathbb {R}^N$, we define ${\fancyscript{H}}(U')$ as the space of all functions $v \in L^2(\mathbb {R}^N)$ such that

$$\begin{aligned} \rho (v,U'):= \int \limits _{U'}\int \limits _{U'}(v(x)-v(y))^2J(x-y)\ dxdy<\infty . \end{aligned}$$

(2.9)

Note that ${\fancyscript{D}}(\mathbb {R}^N) \cap L^2(\mathbb {R}^N) \subset {\fancyscript{H}}(U')$ for any measurable subset $U' \subset \mathbb {R}^N$, and thus also ${\fancyscript{D}}(U) \subset {\fancyscript{H}}(U')$ for any bounded open set $U \subset \mathbb {R}^N$ by (2.1).

Lemma 2.5

Let $U' \subset \mathbb {R}^N$ be an open set and $v,\varphi \in {\fancyscript{H}}(U')$. Moreover, suppose that $\varphi \equiv 0$ on $\mathbb {R}^N{\setminus } U$ for some subset $U \subset U'$ with $dist (U,\mathbb {R}^N {\setminus } U')>0$. Then,

$$\begin{aligned} \int \limits _{\mathbb {R}^N} \int \limits _{\mathbb {R}^N} |v(x)-v(y)| |\varphi (x)-\varphi (y)|J(x-y) \,dx dy < \infty , \end{aligned}$$

(2.10)

and thus,

$$\begin{aligned} {\fancyscript{J}}(v,\varphi ) := \frac{1}{2} \int \limits _{\mathbb {R}^N} \int \limits _{\mathbb {R}^N} (v(x)-v(y)) (\varphi (x)-\varphi (y))J(x-y) \,dx dy \end{aligned}$$

is well defined.

Proof

Since $J$ satisfies $(J1)$, we have $K:= \int _{\mathbb {R}^N {\setminus } B_r(0)}J(z)\,dz < \infty$ with $r:= dist (U,\mathbb {R}^N {\setminus } U')>0$. As a consequence,

$$\begin{aligned} \int \limits _{\mathbb {R}^N}&\int \limits _{\mathbb {R}^N} |v(x)-v(y)| |\varphi (x)-\varphi (y)|J(x-y) \,dx dy\\&= \int \limits _{U'}\int \limits _{U'}|v(x)-v(y)||\varphi (x)-\varphi (y)|J(x-y)\ dxdy\\&\quad + 2 \int \limits _{U}\int \limits _{\mathbb {R}^{N}{\setminus } U'} |v(x)-v(y)| |\varphi (x)|J(x-y)\ dydx\\&\le \frac{1}{2} \bigl [\rho (v,U') + \rho (\varphi ,U')\bigr ] + \int \limits _{U}\int \limits _{\mathbb {R}^{N}{\setminus } U'} \Bigl [2 \bigl (|v(x)|^2+ |v(y)|^2\bigr ) + |\varphi (x)|^2\Bigr ] J(x-y)\ dydx\\&\le \frac{1}{2} \bigl [\rho (v,U') + \rho (\varphi ,U')\bigr ] + K \Bigl ( 4 \Vert v\Vert _{L^2(\mathbb {R}^N)}^2+ \Vert \varphi \Vert _{L^2(\mathbb {R}^N)}^2\Bigr )<\infty . \end{aligned}$$

$\square$

Lemma 2.6

If $U' \subset \mathbb {R}^N$ is open and $v\in {\fancyscript{H}}(U')$, then $v^\pm \in {\fancyscript{H}}(U')$ and $\rho (v^\pm ,U') \le \rho (v,U')$.

Proof

We have $v^\pm \in L^2(\mathbb {R}^N)$ since $v \in L^2(\mathbb {R}^N)$. Moreover, $v^+(x) v^-(x) =0$ for $x \in \mathbb {R}^N$, and thus,

$$\begin{aligned} \rho (v,U')&= \rho (v^+,U') \!+\! \rho (v^-,U') \!-\! 2\int \limits _{U'}\int \limits _{U'} (v^+(x)\!-\!v^+(y))(v^-(x)\!-\!v^-(y))J(x\!-\!y)\ dxdy\\&=\rho (v^+,U') + \rho (v^-,U') +2 \int \limits _{U'}\int \limits _{U'}[v^{+}(x)v^{-}(y)+v^{+}(y)v^{-}(x)] J(x-y)\ dxdy\\&\ge \rho (v^+,U') + \rho (v^-,U'). \end{aligned}$$

The claim follows.$\square$

We close this section with a remark on assumption $(J2)$.

Remark 2.7

Suppose that $(J2)$ is satisfied. Then, for every fixed $z' \in \mathbb {R}^N$, the function $t \mapsto J(t,z')$ is strictly decreasing in $|t|$ and therefore coincides a.e. on $\mathbb {R}$ with the function $t \mapsto \tilde{J}(t,z'):= \lim \limits _{s \rightarrow t^-}J(s,z')$. Hence, $J$ and the function $\tilde{J}$ differ only on a set of measure zero in $\mathbb {R}^N$. Replacing $J$ by $\tilde{J}$ if necessary, we may therefore deduce from $(J2)$ the symmetry property

$$\begin{aligned} J(-t,z')= J(t,z') \qquad \text {for every }\quad z' \in \mathbb {R}^{N-1},\quad t\in \mathbb {R}. \end{aligned}$$

(2.11)

This will be used in the following section.

3 The linear problem associated with a hyperplane reflection

In the following, we consider a fixed open affine half space $H \subset \mathbb {R}^N$, and we let $Q: \mathbb {R}^N \rightarrow \mathbb {R}^N$ denote the reflection at $\partial H$. For the sake of brevity, we sometimes write $\bar{x}$ in place of $Q(x)$ for $x \in \mathbb {R}^N$. A function $v:\mathbb {R}^{N}\rightarrow \mathbb {R}^{N}$ is called antisymmetric (with respect to $Q$) if $v(\bar{x})=-v(x)$ for $x \in \mathbb {R}^{N}$. As before, we consider an even kernel $J: \mathbb {R}^N {\setminus } \{0\} \rightarrow [0,\infty )$ satisfying $(J1)$. We also assume the following symmetry and monotonicity assumptions on $J$:

$$\begin{aligned} J(\bar{x}-\bar{y})&=J(x-y) \qquad \text {for all } \quad x,y \in \mathbb {R}^N; \end{aligned}$$

(3.1)

$$\begin{aligned} J(x-y)&\ge J(x- \bar{y}) \qquad \text {for all } \quad x,y \in H. \end{aligned}$$

(3.2)

Remark 3.1

If $(J1)$, $(J2)$ and (2.11) are satisfied and

$$\begin{aligned} H= \{x \in \mathbb {R}^N\,:\, x_1 > \lambda \} \qquad \text {or}\qquad H= \{x \in \mathbb {R}^N\,:\, x_1 < -\lambda \} \end{aligned}$$

for some $\lambda \in \mathbb {R}$, then (3.1) and (3.2) hold. In this case, $J$ even satisfies the following strict variant of (3.2):

$$\begin{aligned} J(x-y) > J(x- \bar{y}) \qquad \text {for all } \quad x,y \in H. \end{aligned}$$

(3.3)

We will need this property in Proposition 3.6 below.

Lemma 3.2

Let $J$ satisfy $(J1)$, (3.1) and (3.2). Moreover, let $U' \subset \mathbb {R}^N$ be an open set with $Q(U')=U'$, and let $v \in {\fancyscript{H}}(U')$ be an antisymmetric function such that $v \ge 0$ on $H {\setminus } U$ for some open bounded set $U \subset H$ with $\overline{U} \subset U'$. Then, the function $w:= 1_H\, v^-$ is contained in ${\fancyscript{D}}(U)$ and satisfies

$$\begin{aligned} {\fancyscript{J}}(w,w) \le - {\fancyscript{J}}(v,w) \end{aligned}$$

(3.4)

Proof

We first show that $w \in {\fancyscript{H}}(U')$. Clearly, we have $w \in L^2(\mathbb {R}^N)$, since $v \in L^2(\mathbb {R}^N)$. Moreover, by (3.1), the symmetry of $U'$, the antisymmetry of $v$ and (3.2), we have

$$\begin{aligned} \rho (v,U')&=\int \limits _{U'\cap H}\int \limits _{U'\cap H}(v(x)-v(y))^2 J(x-y)\ dxdy \nonumber \\&\quad +\int \limits _{U'{\setminus } H}\int \limits _{U'{\setminus } H}(v(x)-v(y))^2 J(x-y)\ dxdy\nonumber \\&\quad +2\int \limits _{U'{\setminus } H}\int \limits _{U'\cap H}(v(x)-v(y))^2 J(x-y)\ dxdy\nonumber \\&=2\int \limits _{U'\cap H}\int \limits _{U'\cap H}\Bigl [(v(x)-v(y))^2 J(x-y) + (v(x)+v(y))^2 J(x-\bar{y})\Bigr ]\ dxdy\nonumber \\&\ge \int \limits _{U'\cap H}\int \limits _{U'\cap H} \Bigl [(v(x)-v(y))^2 J(x-y) + [(v(x)-v(y))^2 \nonumber \\&\quad + (v(x)+v(y))^2] J(x-\bar{y})\Bigr ]\ dxdy\nonumber \\&\ge \int \limits _{U'\cap H}\int \limits _{U'\cap H} \Bigl [(v(x)-v(y))^2 J(x-y) + 2v^2(x) J(x-\bar{y})\Bigr ]\ dxdy\nonumber \\&= \int \limits _{U'}\int \limits _{U'}(1_{H}v(x)-1_{H}v(y))^2 J(x-y)\ dxdy = \rho (1_{H}\,v, U') \end{aligned}$$

(3.5)

and thus $\rho (1_{H}\,v,U') < \infty$. Hence, $1_{H}\,v \in {\fancyscript{H}}(U')$ and thus also $w \in {\fancyscript{H}}(U')$ by Lemma 2.6. Since $w \equiv 0$ in $\mathbb {R}^N {\setminus } U$, the right-hand side of (3.4) is well defined and finite by Lemma 2.5. To show (3.4), we first note that

$$\begin{aligned}{}[w+v]w= [ 1_H v^+ + 1_{\mathbb {R}^N {\setminus } H} v]1_H v^- \equiv 0 \qquad \text {on } \mathbb {R}^N \end{aligned}$$

and therefore

$$\begin{aligned}&[w(x)-w(y)]^2 + [v(x)-v(y)] [w(x)-w(y)] \\&\quad = - \Bigl (w(x)[w(y)+v(y)] + w(y)[w(x)+v(x)]\Bigr ) \end{aligned}$$

for $x,y \in \mathbb {R}^N$. Using this identity in the following together with the antisymmetry of $v$, the symmetry properties of $J$ and the fact that $w \equiv 0$ on $\mathbb {R}^N {\setminus } H$, we find that

$$\begin{aligned} {\fancyscript{J}}(w,w) + {\fancyscript{J}}(v,w)&= -\int \limits _{H} \int \limits _{\mathbb {R}^N}w(x) [w(y)+v(y)] J(x-y)\,dy dx\\&=-\int \limits _{H} \int \limits _{\mathbb {R}^N}w(x) [1_H(y)v^+(y) +1_{\mathbb {R}^N {\setminus } H} v(y)] J(x-y)\,dy dx\\&=-\int \limits _{H} \int \limits _{H}w(x) [v^+(y)J(x-y) -v(y) J(x-\bar{y})] \,dy dx \,\le \, 0, \end{aligned}$$

where in the last step, we used the fact that $v^+(y) \ge v(y)$ and $J(x-y) \ge J(x-\bar{y}) \ge 0$ for $x,y \in H$. Hence, (3.4) is true, and in particular, we have ${\fancyscript{J}}(w,w)< \infty$. Since $w \equiv 0$ on $\mathbb {R}^N {\setminus } U$, it thus follows that $w \in {\fancyscript{D}}(U)$.$\square$

In order to implement the moving plane method, we have to deal with the class of antisymmetric supersolutions of a class of linear problems. A related notion was introduced in [20] in a parabolic setting related to the fractional Laplacian.

Definition 3.3

Let $U\subset H$ be an open bounded set and let $c\in L^{\infty }(U)$. We call an antisymmetric function $v: \mathbb {R}^N \rightarrow \mathbb {R}^N$ an antisymmetric supersolution of the problem

$$\begin{aligned} Iv= c(x)v\quad \text { in }\quad U, \qquad v \equiv 0 \quad \text {on }\quad H {\setminus } U \end{aligned}$$

(3.6)

if $v \in {\fancyscript{H}}(U')$ for some open bounded set $U' \subset \mathbb {R}^N$ with $Q(U')=U'$ and $\overline{U} \subset U'$, $v\ge 0$ on $H{\setminus } U$ and

$$\begin{aligned} {\fancyscript{J}}(u,\varphi )\ge \int \limits _{U}c(x)u(x)\varphi (x)\ dx \qquad \text { for all }\quad \varphi \in {\fancyscript{D}}(U), \varphi \ge 0. \end{aligned}$$

(3.7)

Remark 3.4

Assume $(J1)$ and (3.1), and let $\Omega \subset \mathbb {R}^{N}$ be an open bounded set such that $Q(\Omega \cap H)\subset \Omega$. Furthermore, let $f: \Omega \times \mathbb {R}\rightarrow \mathbb {R}$ be a Carathéodory function satisfying $(F1)$ and such that

$$\begin{aligned} f(\bar{x},\tau ) \ge f(x,\tau )\qquad \text {for every }\quad \tau \in \mathbb {R},\quad x \in H\cap \Omega . \end{aligned}$$

(3.8)

If $u \in {\fancyscript{D}}(\Omega )$ is a nonnegative solution of $(P)$, then $v:=u\circ Q-u$ is an antisymmetric supersolution of (3.6) with $U:= \Omega \cap H$ and $c \in L^\infty (U)$ defined by

$$\begin{aligned} c(x)= \left\{ \begin{array}{ll} \frac{f(x,u(\bar{x}))-f(x,u(x))}{v(x)}&{}\quad \text {if } \quad v(x) \not = 0;\\ 0 &{}\quad \text {if } \quad v(x)= 0. \end{array} \right. \end{aligned}$$

Indeed, since $u\in {\fancyscript{D}}(\Omega )$, we have $v \in {\fancyscript{D}}(\mathbb {R}^{N}) \cap L^2(\mathbb {R}^N)$ and thus $v \in {\fancyscript{H}}(U')$ for any open set $U' \subset \mathbb {R}^N$. Moreover, $v \ge 0$ on $H {\setminus } U$ since $u$ is nonnegative and $u \equiv 0$ on $H {\setminus } U$. Furthermore, if $\varphi \in {\fancyscript{D}}(U)$, then $\varphi \circ Q-\varphi \in {\fancyscript{D}}(\Omega )$ by the symmetry properties of $J$ and since $Q(U)\subset \Omega$. If, in addition, $\varphi \ge 0$, then we have, using (3.1),

$$\begin{aligned} {\fancyscript{J}}(v,\varphi )&= {\fancyscript{J}}(u \circ Q - u,\varphi )={\fancyscript{J}}(u,\varphi \circ Q - \varphi )= \int \limits _{\Omega }f(x,u)[\varphi \circ Q - \varphi ]\,dx\\&=\!\!\!\int \limits _{ Q(U)}\!\!\!f(x,u(x))\varphi \circ Q\,dx - \int \limits _{U}\!\!f(x,u(x))\varphi \,dx\\&=\!\!\int \limits _{U}[f(\bar{x},u(\bar{x}))-f(x,u(x))]\varphi (x)\,dx \ge \int \limits _{U}c(x) v \varphi \,dx. \end{aligned}$$

Here, (3.8) was used in the last step. The boundedness of $c$ follows from $(F1)$.

We now have all the tools to establish maximum principles for antisymmetric supersolutions of (3.6).

Proposition 3.5

Assume that $J$ satisfies $(J1)$, (3.1) and (3.2), and let $U \subset H$ be an open bounded set. Let $c\in L^{\infty }(U)$ with $\Vert c^+\Vert _{L^\infty (U)} <\Lambda _1(U)$, where $\Lambda _1(U)$ is given in (2.1).

Then, every antisymmetric supersolution $v$ of (3.6) in $U$ satisfies $v\ge 0$ a.e. in $H$.

Proof

By Lemma 3.2, we have that $w:=1_{H} v^{-} \in {\fancyscript{D}}(U)$ and ${\fancyscript{J}}(w,w) \le -{\fancyscript{J}}(v,w)$. Consequently,

$$\begin{aligned} \Lambda _{1}(U) \Vert w\Vert _{L^{2}(U)}^{2}&\le {\fancyscript{J}}(w,w) \le -{\fancyscript{J}}(v,w) \le - \int \limits _{U}c(x) v(x)w(x) \ dx\\&= \int \limits _{U}c(x) w^2(x) \ dx \le \Vert c^+\Vert _{L^\infty (U)}\Vert w\Vert _{L^{2}(U)}^{2}. \end{aligned}$$

Since $\Vert c^+\Vert _{L^\infty (U)}< \Lambda _{1}(U)$ by assumption, we conclude that $\Vert w\Vert _{L^{2}(U)}=0$ and hence $v\ge 0$ a.e. in $H$.$\square$

We note that a combination of Proposition 3.5 with Lemma 2.1 gives rise to an “antisymmetric” small volume maximum principle which generalizes the available variants for the fractional Laplacian, see [13, Proposition 3.3 and Corollary 3.4] and [23, Lemma 5.1]. Next, we prove a strong maximum principle which requires the strict inequality (3.3).

Proposition 3.6

Assume that $J$ satisfies $(J1)$, (3.1) and (3.3). Moreover, let $U \subset H$ be an open bounded set and $c \in L^\infty (U)$. Furthermore, let $v$ be an antisymmetric supersolution of (3.6) such that $v\ge 0$ a.e. in $H$. Then, either $v \equiv 0$ a.e. in $\mathbb {R}^{N}$, or

$$\begin{aligned} \mathop {{{\mathrm{essinf}}}}\limits _{K}\, v >0 \qquad \text {for every compact subset } K \subset U. \end{aligned}$$

Proof

We assume that $v \not \equiv 0$ in $\mathbb {R}^N$. For given $x_0 \in U$, it then suffices to show that $\mathop {{{\mathrm{essinf}}}}\limits _{B_r(x_0)}\, v >0$ for $r>0$ sufficiently small. Since $v\not \equiv 0$ in $\mathbb {R}^N$ and $v$ is antisymmetric with $v \ge 0$ in $H$, there exists a bounded set $M \subset H$ of positive measure with $x_0 \not \in \overline{M}$ and such that

$$\begin{aligned} \delta := \inf _{M} v >0. \end{aligned}$$

(3.9)

By Lemma 2.1, we may fix $0<r< \frac{1}{4} dist (x_0,[\mathbb {R}^N {\setminus } H] \cup M)$ such that $\Lambda _1(B_{2r}(x_0)) > \Vert c\Vert _{L^{\infty }(U) }$. Next, we fix a function $f\in {\fancyscript{C}}^{2}_{c}(\mathbb {R}^{N})$ such that $0 \le f \le 1$ on $\mathbb {R}^N$ and

$$\begin{aligned} f(x):=\left\{ \begin{array}{ll} 1, &{}\text { for }\quad |x-x_0|\le r,\\ 0, &{}\text { for }\quad |x-x_0|\ge 2r.\\ \end{array}\right. \end{aligned}$$

Moreover, we define

$$\begin{aligned} w: \mathbb {R}^N \rightarrow \mathbb {R}, \qquad w(x):=f(x)-f(\bar{x})+ a\bigl [1_M(x)-1_{M} (\bar{x}))\bigr ], \end{aligned}$$

where $a>0$ will be fixed later. We also put $U_0:= B_{2r}(x_0)$ and $U_0':= B_{3r}(x_0) \cup Q(B_{3r}(x_0))$. Note that the function $w$ is antisymmetric and satisfies

$$\begin{aligned} w\equiv 0 \quad \text {on }\quad H {\setminus } (U_0 \cup M), \quad \qquad w \equiv a \quad \text {on } M. \end{aligned}$$

(3.10)

We claim that $w \in {\fancyscript{H}}(U_0')$. Indeed, by Proposition 2.3(i), we have $f - f \circ Q \in {\fancyscript{D}}(\mathbb {R}^N) \cap L^2(\mathbb {R}^N) \subset {\fancyscript{H}}(U_0')$, whereas $1_M - 1_{Q(M)} \in {\fancyscript{H}}(U_0')$ since $M$ is bounded and $[M \cup Q(M)] \cap U_0'= \emptyset$.

Next, let $\varphi \in {\fancyscript{D}}(U_0)$, $\varphi \ge 0$. By Proposition 2.3(ii), we have

$$\begin{aligned} {\fancyscript{J}}(f,\varphi ) \le C \int \limits _{U_0}\varphi (x)\ dx \end{aligned}$$

(3.11)

with $C=C(f)>0$ independent of $\varphi$. Since

$$\begin{aligned} f(\bar{x}) \varphi (x) = 1_{M}(x) \varphi (x)= 1_{Q(M)}(x) \varphi (x) = 0 \qquad \text {for every } x \in \mathbb {R}^N, \end{aligned}$$

we have

$$\begin{aligned} {\fancyscript{J}}(w,\varphi )&={\fancyscript{J}}(f,\varphi ) - {\fancyscript{J}}(f \circ Q,\varphi ) +a\bigl [{\fancyscript{J}}(1_M, \varphi ) -{\fancyscript{J}}(1_{Q(M)}, \varphi )\bigr ]\\&\le C \int \limits _{U_0}\varphi (x)\ dx +\int \limits _{U_0}\int \limits _{Q(U_0)} \varphi (x)f(\overline{y}) J(x-y)\ dydx\\&\quad -a\bigl [ \int \limits _{U_0}\int \limits _{M}\varphi (x)J(x-y)\ dydx -\int \limits _{U_0}\int \limits _{Q(M)}\varphi (x)J(x-y)\ dydx \bigr ] \\&\le \Bigl ( C + \sup _{x \in U_0} \int \limits _{Q(U_0)} J(x-y)\,dy \Bigr ) \int \limits _{U_0}\varphi (x)\ dx\\&\quad -a\int \limits _{U_0}\varphi (x) \int \limits _{M}[J(x-y)-J(x-\bar{y})]\ dydx\\&\le C_a \int \limits _{U_0} \varphi (x)\,dx \end{aligned}$$

with

$$\begin{aligned} C_a := C + \sup _{x \in U_0} \int \limits _{Q(U_0)} J(x-y)\,dy - a \inf _{x \in U_0} \int \limits _M (J(x-y)-J(x-\bar{y}))\ dy\in \mathbb {R}\end{aligned}$$

Since $\overline{U}_0 \subset H$, (3.3) and the continuity of the function $x \mapsto \int _M (J(x-y)-J(x-\bar{y}))\ dy$ on $\overline{U}_0$ imply that

$$\begin{aligned} \inf _{x \in U_0} \int \limits _M (J(x-y)-J(x,\bar{y}))\ dy>0 \end{aligned}$$

Consequently, we may fix $a>0$ sufficiently large such that $C_a \le -\Vert c\Vert _{L^\infty (U_0)}$. Since $0 \le w \le 1$ in $U_0$, we then have

$$\begin{aligned} {\fancyscript{J}}(w,\varphi ) \le -\Vert c\Vert _{L^\infty (U_0)} \int \limits _{U_0}\varphi (x)\,dx \le \int \limits _{U_0}c(x)w(x)\varphi (x)\ dx. \end{aligned}$$

(3.12)

We now consider the function $\tilde{v}:=v-\frac{\delta }{a} w \in {\fancyscript{H}}(U_0')$, which by (3.9) and (3.10) satisfies $\tilde{v} \ge 0$ on $H {\setminus } U_0$. Hence, by assumption and (3.12), $\tilde{v}$ is an antisymmetric supersolution of the problem

$$\begin{aligned} I\tilde{v} = c(x)\tilde{v}\quad \text { in }\quad U_0, \qquad \tilde{v} \equiv 0 \quad \text {on }\quad H {\setminus } U_0 \end{aligned}$$

(3.13)

Since $\Vert c\Vert _{L^{\infty }(U_0)}<\Lambda _1(U_0)$, Proposition 3.5 implies that $\tilde{v} \ge 0$ a.e. in $U_0$, so that $v \ge \frac{\delta }{a} w = \frac{\delta }{a} >0$ a.e. in $B_{r}(x_0)$. This ends the proof.$\square$

4 Proof of the main symmetry result

In this section, we complete the proof of Theorem 1.1. So throughout this section, we assume that $J:\mathbb {R}^N {\setminus } \{0\} \rightarrow [0,\infty )$ is even and satisfies $(J1)$ and $(J2)$, $\Omega \subset \mathbb {R}^N$ satisfies $(D)$ and the nonlinearity $f$ satisfies $(F_1)$ and $(F_2)$. Moreover, we let $u \in L^\infty (\Omega ) \cap {\fancyscript{D}}(\Omega )$ be a nonnegative solution of $(P)$. For $\lambda \in \mathbb {R}$, we consider the open affine half space

$$\begin{aligned} H_{\lambda }:=\left\{ \begin{array}{ll} \{x\in \mathbb {R}^{N}\;:\; x_{1}>\lambda \}&{}\quad \text {if } \quad \lambda \ge 0;\\ \{x\in \mathbb {R}^{N}\;:\; x_{1}<\lambda \}&{}\quad \text {if } \quad \lambda < 0.\\ \end{array} \right. \end{aligned}$$

Moreover, we let $Q_{\lambda }:\mathbb {R}^{N}\rightarrow \mathbb {R}^{N}$ denote the reflection at $\partial H_{\lambda }$, i.e. $Q_{\lambda }(x)=(2\lambda -x_{1},x')$. By Remark 2.7, we may assume without loss of generality that (2.11) holds. As noted in Remark 3.1, $J$ therefore satisfies the symmetry and monotonicity conditions (3.1) and (3.3) with $H$ replaced by $H_\lambda$. Let $\ell :=\sup \nolimits _{x\in \Omega }x_1$. Setting $\Omega _{\lambda }:= H_\lambda \cap \Omega$ for $\lambda \in \mathbb {R}$, we note that $Q_{\lambda }(\Omega _\lambda )\subset \Omega$ for all $\lambda \in (-\ell ,\ell )$ and $Q_{0}(\Omega )=\Omega$ as a consequence of assumption $(D)$. Then, for all $\lambda \in (-\ell ,\ell )$, Remark 3.4 implies that $v_{\lambda }:=u \circ Q_\lambda -u \in {\fancyscript{D}}(\mathbb {R}^N) \cap L^2(\mathbb {R}^N)$ is an antisymmetric supersolution of the problem

$$\begin{aligned} Iv = c_\lambda (x)v \quad \text { in }\quad \Omega _\lambda ,\quad v \equiv 0 \quad \text {on }\quad H_\lambda {\setminus } \Omega _\lambda \end{aligned}$$

(4.1)

with

$$\begin{aligned} c_\lambda \in L^{\infty }(\Omega _\lambda )\quad \text {given by}\quad c_\lambda (x)= \left\{ \begin{array}{ll} \frac{f(x,u(Q_\lambda (x)))-f(x,u(x))}{v_\lambda (x)},&{}\quad v_\lambda (x) \not = 0;\\ 0, &{}\quad v_\lambda (x)= 0. \end{array} \right. \end{aligned}$$

Note that, as a consequence of $(F1)$ and since $u \in L^\infty (\Omega )$, we have

$$\begin{aligned} c_\infty := \sup _{\lambda \in (-\ell ,\ell )}\Vert c_\lambda \Vert _{L^\infty (\Omega _\lambda )} < \infty . \end{aligned}$$

We now consider the statement

$$\begin{aligned} (S_{\lambda })\qquad \mathop {{{\mathrm{essinf}}}}\limits _{K} \,v_{\lambda }> 0 \qquad \text {for every compact subset } K \subset \Omega _{\lambda }. \end{aligned}$$

Assuming that $u \not \equiv 0$ from now on, we will show $(S_{\lambda })$ for all $\lambda \in (0,\ell )$. Since $|\Omega _{\lambda }| \rightarrow 0$ as $\lambda \rightarrow \ell$, Lemma 2.1 implies that there exists $\epsilon \in (0,\ell )$ such that $\Lambda _1(\Omega _{\lambda })>c_\infty$ for all $\lambda \in [\epsilon ,\ell )$. Applying Proposition 3.5, we thus find that

$$\begin{aligned} v_\lambda \ge 0 \quad \text {a.e. in }\quad H_\lambda \quad \hbox {for all }\quad \lambda \in [\epsilon ,\ell ). \end{aligned}$$

(4.2)

We now show

Claim 1

If $v_\lambda \ge 0$ a.e. in $H_\lambda$ for some $\lambda \in (0,\ell )$, then $(S_{\lambda })$ holds.

To prove this, by Proposition 3.6, it suffices to show that $v_{\lambda }\not \equiv 0$ in $\mathbb {R}^N$. If, arguing by contradiction, $v_{\lambda }\equiv 0$ in $\mathbb {R}^N$, then $\partial H_{\lambda }$ is a symmetry hyperplane of $u$. Since $\lambda \in (0,\ell )$ and $u \equiv 0$ in $\mathbb {R}^N {\setminus } \Omega$, we then have $u\equiv 0$ in the nonempty set $\Omega _{-\ell +2\lambda }$. Setting $\lambda '= -\ell +\lambda$, we thus infer that $v_{\lambda '} \equiv 0$ in $\Omega _{\lambda '}$. Consequently, $v_{\lambda '}\equiv 0$ in $\mathbb {R}^{N}$ by Proposition 3.6. Thus, $u$ has the two different parallel symmetry hyperplanes $\partial H_{\lambda }$ and $\partial H_{\lambda '}$. Since $u$ vanishes outside a bounded set, this implies that $u\equiv 0$, which is a contradiction. Thus, Claim 1 is proved.

Next, we show

Claim 2

If $(S_{\lambda })$ holds for some $\lambda \in (0,\ell )$, then there is $\delta \in (0,\lambda )$ such that $(S_{\mu })$ holds for all $\mu \in (\lambda -\delta ,\lambda )$.

To prove this, suppose that $(S_{\lambda })$ holds for some $\lambda \in (0,\ell )$. Using Lemma 2.1, we fix $s \in (0, |\Omega _\lambda |)$ such that $\Lambda _1(s)> c_\infty$, which implies that $\Lambda _1(U)>c_\infty$ for all open sets $U\subset \mathbb {R}^N$ with $|U|\le s$. Since $\Omega$ is bounded, we may also fix $\delta _0 >0$ such that

$$\begin{aligned} |\Omega _{\mu } {\setminus } \Omega _{\mu +\delta _0}|<s/2\qquad \text {for all } \mu \ge 0. \end{aligned}$$

By Lusin’s Theorem, there exists a compact subset $K \subset \Omega$ such that $|\Omega {\setminus } K|< s/4$ and such that the restriction $u|_{K}$ is continuous. For $\mu \ge 0$, we now consider the compact set

$$\begin{aligned} K_\mu := \overline{\Omega }_{\mu +\delta _0} \cap K \cap Q_\mu (K) \,\subset \,K \cap \Omega _\mu \end{aligned}$$

and the open set $U_\mu :=\Omega _\mu {\setminus } K_\mu$. Note that

$$\begin{aligned} |U_\mu | \!\le \! |\Omega _\mu {\setminus } \Omega _{\mu +\delta _0}|\!+\! |\Omega _\mu {\setminus } K| \!+\! |\Omega _\mu {\setminus } Q_\mu (K)| \le \frac{s}{2} + 2 |\Omega {\setminus } K| < s \qquad \text {for } \mu \ge 0. \end{aligned}$$

(4.3)

As a consequence, for $0 \le \mu \le \lambda$, we have $|K_\mu | > |\Omega _\mu |-s \ge |\Omega _\lambda |-s>0$, and thus, $K_\mu \not = \varnothing$. Property $(S_\lambda )$ and the continuity of $u|_K$ imply that $\min \nolimits _{K_\lambda } v_\lambda >0$. Thus, again by the continuity of $u|_K$, there exists $\delta \in (0,\min \{\lambda ,\delta _0\})$ such that

$$\begin{aligned} \min _{K_\mu } v_\mu >0 \qquad \text {for all } \mu \in [\lambda -\delta ,\lambda ]. \end{aligned}$$

Consequently, for $\mu \in (\lambda -\delta ,\lambda )$, the function $v_{\mu }$ is an antisymmetric supersolution of the problem

$$\begin{aligned} Iv=c_{\mu }(x)v \quad \text {in } U_\mu , \qquad v \equiv 0 \quad \text {on } H_{\mu } {\setminus } U_\mu , \end{aligned}$$

whereas $\Lambda _1(U_\mu )>c_\infty$ by (4.3) and the choice of $s$. Hence, $v_{\mu }\ge 0$ in $H_{\mu }$ by Proposition 3.5, and thus, $(S_{\mu })$ holds by Claim 1. This proves Claim 2.

To finish the proof, we consider

$$\begin{aligned} \lambda _{0}:=\inf \{\tilde{\lambda } \in (0,\ell ) \;:\; (S_{\lambda }) \hbox { holds for all } \lambda \in (\tilde{\lambda },\ell ) \} \quad \in \;[0,\ell ). \end{aligned}$$

We then have $v_{\lambda _0} \ge 0$ in $H_{\lambda _0}$. Hence, Claim 1 and Claim 2 imply that $\lambda _{0}=0$. Since the procedure can be repeated in the same way starting from $-\ell$, we find that $v_{0} \equiv 0$. Hence, the function $u$ has the asserted symmetry and monotonicity properties.

It remains to show (1.3). So let $K \subset \Omega$ be compact. Replacing $K$ by $K \cup Q_0(K)$ if necessary, we may assume that $K$ is symmetric with respect to $Q_0$. Let $K':= \{x \in K\,:\, x_1 \le 0\}$. Since for $\lambda >0$ sufficiently small $Q_\lambda (K')$ is a compact subset of $\Omega _\lambda$, the property $(S_\lambda )$ and the symmetry of $u$ then imply that

$$\begin{aligned} \mathop {{{\mathrm{essinf}}}}\limits _{K}\, u = \mathop {{{\mathrm{essinf}}}}\limits _{K'}\, u \ge \mathop {{{\mathrm{essinf}}}}\limits _{Q_\lambda (K')}\, v_\lambda >0, \end{aligned}$$

as claimed in (1.3).

5 Proof of a variant symmetry result

In this section, we prove Theorem 1.4, which is concerned with the class of even kernel functions satisfying $(J2)'$ in place of $(J2)$. Throughout this section, we consider a symmetric kernel $J: \mathbb {R}^N {\setminus } \{0\} \rightarrow [0,\infty )$ satisfying $(J1)$. We fix an open affine half space $H \subset \mathbb {R}^N$, and we consider the notation of Sect. 3. Moreover, we assume the symmetry and monotonicity assumptions (3.1) and (3.2), so that Lemma 3.2 and Proposition 3.5 are available. In order to derive a variant of the strong maximum principle given in Proposition 3.6, we introduce the following strict monotonicity condition:

$$\begin{aligned} \text {There exists } r_0\!>\!0 \hbox { such that } J(x-y) \!>\! J(x- \bar{y})\hbox { for all } x,y \!\in \! H \hbox { with } |x-y| \le r_0 \end{aligned}$$

(5.1)

We then have the following.

Proposition 5.1

Assume that $J$ satisfies $(J1)$, (3.1), (3.2) and (5.1). Moreover, let $U \subset H$ be a subdomain and $c \in L^\infty (U)$. Furthermore, let $v$ be an antisymmetric supersolution of (3.6) such that $v\ge 0$ a.e. in $H$.

Then, either $v \equiv 0$ a.e. in a neighborhood of $\overline{U}$, or

$$\begin{aligned} \mathop {{{\mathrm{essinf}}}}\limits _{K}\, v >0 \qquad \text {for every compact subset } K \subset U. \end{aligned}$$

We stress that in contrast to Proposition 3.6, we require connectedness of $U$ here.

Proof

Let $W$ denote the set of points $y \in U$ such that $\mathop {{{\mathrm{essinf}}}}\limits {B_{r}(y)}\, v >0$ for $r>0$ sufficiently small, and let $r_0>0$ be as in (5.1). We claim the following.

$$\begin{aligned} \text {If } x_0 \in U \hbox { is such that } v \not \equiv 0 \hbox { in } B_{\frac{r_0}{2}}(x_0),\hbox { then } x_0 \in W. \end{aligned}$$

(5.2)

To prove this, let $x_0 \in U$ be such that $v \not \equiv 0$ in $B_{\frac{r_0}{2}}(x_0)$. Then, there exists a bounded set $M \subset H \cap B_{\frac{r_0}{2}}(x_0)$ of positive measure with $x_0 \not \in \overline{M}$ and such that

$$\begin{aligned} \delta := \inf _{M} v >0 \end{aligned}$$

(5.3)

By Lemma 2.1, we may fix $0<r< \frac{1}{4}\min \{r_0\,,\, dist (x_0,[\mathbb {R}^N {\setminus } H] \cup M)\}$ such that $\Lambda _1(B_{2r}(x_0)) > \Vert c\Vert _{L^{\infty }(U) }$. Next, we put $U_0:= B_{2r}(x_0)$ and $U_0':= B_{3r}(x_0) \cup Q(B_{3r}(x_0))$. Moreover, we define the functions $f\in C^{2}_{c}(\mathbb {R}^{N})$ and $w \in {\fancyscript{H}}(U_0')$, depending on $a>0$, as in the proof of Proposition 3.6. As noted there, $w$ is antisymmetric and satisfies

$$\begin{aligned} w\equiv 0 \quad \text {on } H {\setminus } (U_0 \cup M), \quad \qquad w \equiv a \quad \text {on } M. \end{aligned}$$

(5.4)

As in the proof of Proposition 3.6, we also see that

$$\begin{aligned} {\fancyscript{J}}(w,\varphi ) \le C_a \int \limits _{U_0} \varphi (x)\,dx \qquad \text {for all }\quad \varphi \in {\fancyscript{D}}(U_0), \varphi \ge 0 \end{aligned}$$

with

$$\begin{aligned} C_a := C + \sup _{x \in U_0} \int \limits _{Q(U_0)} J(x-y)\,dy - a \inf _{x \in \overline{U}_0} \int \limits _M (J(x-y)-J(x-\bar{y}))\ dy \end{aligned}$$

Since $\overline{U}_0 \subset H \cap B_{\frac{r_0}{2}}(x_0)$ and $M \subset H \cap B_{\frac{r_0}{2}}(x_0)$, (5.1) and the continuity of the function $x \mapsto \int _M (J(x-y)-J(x-\bar{y}))\ dy$ on $\overline{U}_0$ imply that

$$\begin{aligned} \inf _{x \in \overline{U}_0} \int \limits _M (J(x-y)-J(x,\bar{y}))\ dy>0 \end{aligned}$$

Hence, we may proceed precisely as in the proof of Proposition 3.6 to prove that $v \ge \frac{\delta }{a} >0$ a.e. in $B_{r}(x_0)$ for $a>0$ sufficiently large, so that $x_0 \in W$. Hence, (5.2) is true.

From (5.2), it immediately follows that $W$ is both open and closed in $U$. Moreover, if $v \not \equiv 0$ in $\{x \in H\,:\, dist (x,U) < \frac{r_0}{2}\}$, then $W$ is nonempty and therefore $W=U$ by the connectedness of $U$. This ends the proof.$\square$

Next, we complete the proof of Theorem 1.4. So, throughout the remainder of this section, we assume that $J:\mathbb {R}^N {\setminus } \{0\} \rightarrow [0,\infty )$ is even and satisfies $(J1)$ and $(J2)'$, $\Omega \subset \mathbb {R}^N$ satisfies $(D)$ and the nonlinearity $f$ satisfies $(F_1)$ and $(F_2)$. Moreover, we let $u \in L^\infty (\Omega ) \cap {\fancyscript{D}}(\Omega )$ denote an a.e. positive solution of $(P)$. For $\lambda \in \mathbb {R}$, we let $H_\lambda$, $Q_\lambda$, $\Omega _\lambda$, $c_\lambda$ and $v_\lambda$ be defined as in Sect. 4, and again, we put $\ell :=\sup \nolimits _{x\in \Omega }x_1$. As a consequence of $(J1)$ and $(J2)'$, we may assume that $J$ satisfies (3.1) (3.2) and (5.1) with $H$ replaced by $H_\lambda$ (the argument of Remark 3.1 still applies). As in Sect. 4, we then consider the statement

$$\begin{aligned} (S_{\lambda })\qquad \mathop {{{\mathrm{essinf}}}}\limits _{K} \,v_{\lambda }> 0 \qquad \text {for every compact subset } K \subset \Omega _{\lambda }. \end{aligned}$$

We wish to show $(S_{\lambda })$ for all $\lambda \in (0,\ell )$. As in Sect. 4, we find $\epsilon \in (0,\ell )$ such that

$$\begin{aligned} v_\lambda \ge 0 \quad \text {a.e. in } H_\lambda \quad \hbox { for all } \lambda \in [\epsilon ,\ell ). \end{aligned}$$

(5.5)

We now show

Claim 1

If $v_{\lambda } \ge 0$ a.e. in $H_{\lambda }$ for some $\lambda \in (0,\ell )$, then $(S_{\lambda })$ holds.

To prove this, we argue by contradiction. If $(S_\lambda )$ does not hold, then, by Proposition 5.1, there exists a connected component $\Omega '$ of $\Omega _\lambda$ and a neighborhood $N$ of $\overline{\Omega '}$ such that $v_{\lambda }\equiv 0$ in $N$. However, since $\lambda \in (0,\ell )$, the set $\tilde{N}:= Q_\lambda (N {\setminus } \Omega ) \cap \Omega$ has positive measure and $v_\lambda \equiv 0$ in $\tilde{N}$ by the antisymmetry of $v_\lambda$. However, $v \equiv -u$ on $\tilde{N}$, so $u \equiv 0$ a.e. on $\tilde{N}$, contrary to the assumption that $u >0$ a.e. in $\Omega$. Thus, Claim 1 is proved.

Precisely as in Sect. 4, we may now show

Claim 2

If $(S_{\lambda })$ holds for some $\lambda \in (0,\ell )$, then there is $\delta \in (0,\lambda )$ such that $(S_{\mu })$ holds for all $\mu \in (\lambda -\delta ,\lambda )$.

Moreover, based on (5.5), Claim 1 and Claim 2, we may now finish the proof of Theorem 1.4 precisely as in the end of Sect. 4.