Mountain pass and linking type sign-changing solutions for nonlinear problems involving the fractional Laplacian

Abstract

In this paper we study the existence of sign-changing solutions for nonlinear problems involving the fractional Laplacian

$$ \left \{ \textstyle\begin{array}{l@{\quad}l} (-\Delta)^{s}u-\lambda u=f(x,u),&x\in\Omega, \\ u=0,&x\in\mathbb{R}^{n}\setminus\Omega, \end{array}\displaystyle \right . $$

(0.1)

where $\Omega\subset\mathbb{R}^{n}$ ($n\geq2$) is a bounded smooth domain, $s\in(0,1)$, $(-\Delta)^{s}$ denotes the fractional Laplacian, λ is a real parameter, the nonlinear term f satisfies superlinear and subcritical growth conditions at zero and at infinity. When $\lambda\leq0$, we prove the existence of a positive solution, a negative solution and a sign-changing solution by combing minimax method with invariant sets of descending flow. When $\lambda\geq \lambda_{1}^{s}$ (where $\lambda_{1}^{s}$ denotes the first eigenvalue of the operator $(-\Delta)^{s}$ in Ω with homogeneous Dirichlet boundary data), we prove the existence of a sign-changing solution by using a variation of linking type theorems.

1 Introduction

This paper is concerned with the existence of sign-changing solutions for nonlinear problems involving the fractional Laplacian

$$ \left \{ \textstyle\begin{array}{l@{\quad}l} (-\Delta)^{s}u-\lambda u=f(x,u), & x\in\Omega, \\ u=0, & x\in\mathbb{R}^{n}\setminus\Omega, \end{array}\displaystyle \right . $$

(1.1)

where $\Omega\subset\mathbb{R}^{n}$ ($n\geq2$) is a bounded domain with smooth boundary ∂Ω, f is a Carathéodory function, $0 < s< 1$, $(-\Delta)^{s}$ is the fractional Laplacian operator. The fractional Laplace $(-\Delta)^{s}$ of a rapidly decaying test function u is defined as

$$ (-\Delta)^{s}u(x)=C_{n,s}\mathrm{P.V.} \int_{\mathbb{R}^{n}}\frac {u(x)-u(y)}{|x-y|^{n+2s}}\,dy, $$

where P.V. denotes the principal value of the singular integral, and

$$C_{n,s}^{-1}= \int_{\mathbb{R}^{n}}\frac{1-\cos x_{1}}{|x|^{n+2s}}\,dx. $$

It is possible to prove that $(-\Delta)^{s}$ is a pseudo-differential operator, and more precisely that

$$ (-\Delta)^{s}u(x)=\mathscr{F}^{-1} \bigl(| \xi|^{2s}\mathscr{F}(u) (\xi) \bigr), \quad\forall s\in(0,1), $$

where $\mathscr{F}$ is the Fourier transform.

The fractional Laplacian operator $(-\Delta)^{s}$ arises in many different applications, such as continuum mechanics, phase transition phenomena, population dynamics and game theory, as they are the typical outcome of stochastically stabilization of Lévy processes; see [1] and the references therein.

The problem (1.1) is a nonlinear equation involving the fractional Laplacian which has been studied extensively for many years. Chang and González [2] studied this problem in conformal geometry. Caffarelli et al. [3, 4] investigated free boundary problems of the fractional Laplacian. Silvestre [5] obtained some regularity results of the obstacle problem of the fractional Laplacian.

In the past few years, some existence results on sign-changing solutions of nonlinear elliptic equations have been obtained by combing minimax method with invariant sets of descending flow (see [6, 7]). In this paper, we generalize the method to study the sign-changing solution for the nonlinear equation involving the fractional Laplacian $(-\Delta)^{s}$.

The problem (1.1) has different variational structures when λ take different values. This paper is divided into two parts. In Section 3, we discuss the case $\lambda\leq0$, a positive and a negative and a sign-changing solution have been found by constructing different invariant sets on which the functional is bounded below; In Section 4, we discuss the case $\lambda>\lambda_{1}^{s}$ (where $\lambda_{1}^{s}$ denotes the first eigenvalue of the operator $(-\Delta)^{s}$ in Ω with homogeneous Dirichlet boundary data), by using a variation of the linking theorem.

To state our results, we suppose that the Carathéodory function f: $\bar{\Omega} \times\mathbb{R}^{n} \to\mathbb{R}$ verifying the following conditions:

($f_{1}$):

$f(x,0)=0$;

($f_{2}$):

$|f(x,t)|\leq C(1+|t|^{p-1})$ for some $C>0$ and $p\in (2,2^{*}_{s})$, where $2^{*}_{s}=\frac{2n}{n-2s}$;

($f_{3}$):

there exists a constant $\mu>2$ such that

$$0 < \mu F(x, t) \leq tf(x, t),\quad \forall x\in\bar{\Omega}, t\in \mathbb{R} \setminus\{0\}, $$

where $F(x,t)=\int_{0}^{t}f(x,\tau)\,d\tau$;

($f_{4}$):

$\lim_{t\to0}\frac{f(x,t)}{t}=0$, uniformly in $x\in\bar{\Omega}$;

($f_{5}$):

for every $x\in\bar{\Omega}$ the function $t\mapsto \frac{f(x,t)}{|t|}$ is nondecreasing on $\mathbb{R}$.

Now, we are ready to state the main results of this paper.

Theorem 1.1

Suppose that ($f_{1}$)-($f_{4}$) and $\lambda\leq0$ hold. Then

• (i) problem (1.1) has at least a positive solution $u_{+}$ and a negative solution $u_{-}$;

• (ii) problem (1.1) has a sign-changing solution ū. If in addition ($f_{5}$) holds then ū is a least energy sign-changing solution with precisely two nodal domains.

Theorem 1.2

Suppose that ($f_{1}$)-($f_{4}$) and $\lambda\geq\lambda_{1}^{s}$ hold. Then problem (1.1) has a sign-changing solution.

2 Preliminaries

Recall that the fractional Sobolev space $H^{s}(\Omega)$ is defined as

$$ H^{s}(\Omega)= \biggl\{ g\in L^{2}(\Omega): \frac {|g(x)-g(y)|}{|x-y|^{\frac{n+2s}{2}}}\in L^{2}(\Omega\times\Omega ) \biggr\} , $$

endowed with the natural norm

$$ \|g\|_{H^{s}(\Omega)}:= \biggl( \int_{\Omega}|g|^{2}\,dx+ \int_{\Omega} \int_{\Omega}\frac {|g(x)-g(y)|^{2}}{|x-y|^{n+2s}}\,dx\,dy \biggr)^{1/2}, $$

(2.1)

where the term

$$ [g]_{H^{s}(\Omega)}:= \biggl( \int_{\Omega} \int_{\Omega}\frac {|g(x)-g(y)|^{2}}{|x-y|^{n+2s}}\,dx\,dy \biggr)^{1/2}, $$

is the so-called Gagliardo (semi)norm of g. $H_{0}^{s}(\Omega)$ denotes the closure of $C_{0}^{\infty}(\Omega)$ in the norm

$$\|g\|_{H_{0}^{s}(\Omega)}= \biggl( \int_{\Omega} \int_{\Omega}\frac {|g(x)-g(y)|^{2}}{|x-y|^{n+2s}}\,dx\,dy \biggr)^{1/2}, $$

which is equivalent with the norm defined in (2.1). For the basic properties of fractional Sobolev spaces, we refer the interested reader to [8].

One can also define $(-\Delta)^{s}$ by using the method of bilinear Dirichlet forms, that is, $(-\Delta)^{s}$ is the closed selfadjoint operator on $L^{2}(\mathbb{R}^{n})$ associated with the bilinear symmetric closed form

$$\ell(u,v)=\frac{C_{n,s}}{2} \int_{\mathbb{R}^{n}\times\mathbb {R}^{N}}\frac{(u(x)-u(y))(v(x)-v(y))}{|x-y|^{n+2s}}\,dx\,dy ,\quad u,v\in H^{s} \bigl( \mathbb{R}^{n} \bigr), $$

in the sense that

$$D \bigl((-\Delta)^{s} \bigr)= \bigl\{ u\in H^{s} \bigl( \mathbb{R}^{n} \bigr), (-\Delta)^{s}u\in L^{2} \bigl( \mathbb {R}^{n} \bigr) \bigr\} $$

and

$$\ell(u,\varphi)= \bigl((-\Delta)^{s}u,\varphi \bigr)= \int_{\mathbb{R}^{n}}\varphi (-\Delta)^{s}u\,dx, \quad\forall u\in D \bigl((-\Delta)^{s} \bigr),\varphi \in H^{s} \bigl( \mathbb{R}^{n} \bigr). $$

Remark that there are maybe some distinctions between the fractional Laplacians by different definitions. In this paper, we define it as follows.

Definition 2.1

see [9–12]

Let $e_{i}$, $\lambda_{i}$ be the eigenfunctions and the eigenvalues of the Laplace operator −Δ in Ω with Dirichlet homogeneous boundary data, $a_{i}$ represents the projection of u on the direction $e_{i}$. Then

$$(-\Delta)^{s}u=\sum_{i\in\mathbb{N}}a_{i} \lambda_{i}^{s}e_{i}. $$

For the reader’s convenience, we review the main result for the following eigenvalue problem of the Laplace operator $(-\Delta)$ with Dirichlet homogeneous boundary data.

Proposition 2.2

see [13, 14]

Let Ω be an open smooth bounded set of $\mathbb{R}^{n}$.

$$ \left \{ \textstyle\begin{array}{l@{\quad}l} (-\Delta)u=\lambda u & \textit{in } \Omega,\\ u=0 & \textit{on } \partial\Omega. \end{array}\displaystyle \right . $$

(2.2)

Then,

• (a) the set of the eigenvalues of problem (2.2) consists of a sequence $\{\lambda_{k}\}_{k\in\mathbb{N}}$ with

  $$0< \lambda_{1}< \lambda_{2}\leq\cdots\leq \lambda_{k}\leq \lambda_{k+1}\leq\cdots $$

  and

  $$\lambda_{k}\to+\infty \quad\textit{as } k\to+\infty; $$

• (b) the sequence $\{e_{k}\}_{k\in\mathbb{N}}$ of eigenfunctions corresponding to $\lambda_{k}$ is an orthonormal basis of $L^{2}(\Omega)$ and an orthogonal basis of $H_{0}^{s}(\Omega)$, and each eigenvalue $\lambda_{k}$ has finite multiplicity;

• (c) the first eigenvalue $\lambda_{1}$ is associated with the only positive eigenfunctions $e_{1}$. The higher eigenvalues $\{\lambda_{k}\}$ ($k\geq2$) are associated with sign-changing eigenfunctions $\{e_{k}\}$ ($k\geq2$).

For the reader’s convenience, we review the main embedding result for fractional Sobolev spaces.

Proposition 2.3

Let $s\in(0, 1)$ and $\Omega\subseteq\mathbb{R}^{n}$ be an open set of class $C^{0,1}$ with bounded boundary. Then the following assertions hold true:

• (a) (see [8]) $H^{s}(\Omega)$ is continuously embedded in $H^{s}(\mathbb{R}^{n})$, namely for any $u\in H^{s}(\Omega)$ there exists $\tilde{u}\in H^{s}(\mathbb{R}^{n})$ such that $\tilde{u}|_{\Omega}=u$ and

  $$\|\tilde{u}\|_{H^{s}(\mathbb{R}^{n})} \leq C_{1} \|u\|_{H^{s}(\Omega)}, $$

  where $C_{1} = C(n,s,\Omega)$.

• (b) (see [10]) The embedding $H_{0}^{s}(\Omega )\hookrightarrow L^{r}(\Omega)$ is continuous for any $r\in[2,2^{*}_{s}]$, and compact for any $r\in[2,2^{*}_{s})$.

For the proof of Theorem 1.1 and Theorem 1.2, we observe that problem (1.1) has a variational structure, indeed it is the Euler-Lagrange equation of the functional $I : H_{0}^{s}(\Omega)\to \mathbb{R}$ defined as follows:

$$ I(u)=\frac{1}{2} \int_{\Omega}\big|(-\Delta)^{\frac{s}{2}}u\big|^{2}\,dx- \frac {1}{2} \int_{\Omega} \lambda|u|^{2} \,dx- \int_{\Omega} F(x,u)\,dx. $$

Note that the functional I is Fréchet differentiable in $u\in H_{0}^{s}(\Omega)$ and for any $\phi\in H_{0}^{s}(\Omega)$

$$ \bigl\langle I'(u),\phi \bigr\rangle = \int_{\Omega}\phi(-\Delta)^{s}u \,dx- \int _{\Omega} \bigl(\lambda u+ f(x,u) \bigr)\phi \,dx. $$

Thus, critical points of I are weak solutions to problem (1.1). In order to find these critical points, we will make use of the minimax method. For this purpose, we have to check that the functional I satisfies the (PS) condition.

Definition 2.4

see [15]

Let $c\in\mathbb {R}$ and $\{u_{j}\}$ be a sequence in $H_{0}^{s}(\Omega)$ such that

$$ \begin{aligned}[c] & I(u_{j})\to c, \\ &\sup_{\varphi\in H_{0}^{s}(\Omega),\|\varphi\|_{H_{0}^{s}(\Omega )}=1}\bigl\{ \big|\bigl\langle I'(u_{j}), \varphi\bigr\rangle \big|\bigr\} \to0, \end{aligned} $$

(2.3)

as $j\to+\infty$, then $\{u_{j}\}$ is a $(\mathit{PS})_{c}$ sequence. Moreover, if there exists $u_{0}\in H_{0}^{s}(\Omega)$ such that, up to a subsequence, $\|u_{j}-u_{0}\| _{H_{0}^{s}(\Omega)}\to0$ as $j\to+\infty$, we said that I satisfies the (PS) condition.

Lemma 2.5

Let f be a function satisfying conditions ($f_{1}$)-($f_{4}$). Let $\{u_{j}\}$ be a $(\mathit{PS})_{c}$ sequence in $H_{0}^{s}(\Omega)$ such that $\{ u_{j}\}$ is bounded in $H_{0}^{s}(\Omega)$. Then I satisfies the (PS) condition.

Proof

Since $\{u_{j}\}$ is bounded in $H_{0}^{s}(\Omega)$ and $H_{0}^{s}(\Omega)$ is a reflexive space (being a Hilbert space), up to a subsequence, still denoted by $\{u_{j}\}$, there exists $u_{0}\in H_{0}^{s}(\Omega)$ such that $u_{j}\rightharpoonup u_{0}$ weakly in $H_{0}^{s}(\Omega)$, that is,

$$ (u_{j},\varphi)_{H_{0}^{s}(\Omega)}\to (u_{0}, \varphi)_{H_{0}^{s}(\Omega)}, \quad\forall\varphi\in H_{0}^{s}(\Omega), $$

(2.4)

as $j\to+\infty$. Moreover, by Proposition 2.3(b), up to a subsequence

$$ \begin{aligned} &u_{j}\to u_{0} \quad\text{in } L^{r}(\Omega), r\in\bigl[2,2^{*}_{s}\bigr); \\ &u_{j}\to u_{0}\quad \text{a.e. in } \Omega, \end{aligned} $$

(2.5)

as $j\to+\infty$ and there exists $h\in L^{r}(\Omega)$ such that

$$ \big|u_{j}(x)\big|\leq h(x) \quad\text{a.e. in } \Omega, \forall j \in \mathbb{N}. $$

(2.6)

By ($f_{2}$), (2.4)-(2.6), the fact that the map $t\mapsto f(\cdot, t)$ is continuous in $t\in\mathbb{R}$ and the dominated convergence theorem we get

$$ \int_{\Omega}f \bigl(x,u_{j}(x) \bigr)u_{j}(x)\,dx \to \int_{\Omega}f \bigl(x,u_{0}(x) \bigr)u_{0}(x)\,dx $$

(2.7)

and

$$ \int_{\Omega}f \bigl(x,u_{j}(x) \bigr)u_{0}(x)\,dx \to \int_{\Omega}f \bigl(x,u_{0}(x) \bigr)u_{0}(x)\,dx $$

(2.8)

as $j\to+\infty$. By $I'(u_{j})\to0$, we have

$$\bigl\langle I'(u_{j}),u_{j} \bigr\rangle \leq \big\| I'(u_{j})\big\| _{(H^{s}_{0}(\Omega))^{*}}\|u_{j}\| _{H^{s}_{0}(\Omega)}\to0, \quad\text{as } j\to+\infty. $$

Moreover, by (2.5)-(2.8), we have

$$ \begin{aligned} (u_{j},u_{j})_{H_{0}^{s}(\Omega)}&= \bigl\langle I'(u_{j}),u_{j} \bigr\rangle +\lambda \int_{\Omega}\big|u_{j}(x)\big|^{2}\,dx+ \int_{\Omega}f \bigl(x,u_{j}(x) \bigr)u_{j}(x)\,dx \\ &\to\lambda \int_{\Omega}\big|u_{0}(x)\big|^{2}\,dx+ \int_{\Omega}f \bigl(x,u_{0}(x) \bigr)u_{0}(x)\,dx \end{aligned} $$

and

$$ \begin{aligned} (u_{j},u_{0})_{H_{0}^{s}(\Omega)}&= \bigl\langle I'(u_{j}),u_{0} \bigr\rangle +\lambda \int_{\Omega}u_{j}(x)u_{0}(x)\,dx+ \int_{\Omega }f \bigl(x,u_{j}(x) \bigr)u_{0}(x)\,dx \\ &\to\lambda \int_{\Omega}\big|u_{0}(x)\big|^{2}\,dx+ \int_{\Omega}f \bigl(x,u_{0}(x) \bigr)u_{0}(x)\,dx. \end{aligned} $$

Consequently, recalling also (2.4), we deduce that

$$ \|u_{j}\|^{2}_{H_{0}^{s}(\Omega)}\to \|u_{0}\|^{2}_{H_{0}^{s}(\Omega)} \quad\text{as } j\to+\infty. $$

(2.9)

Finally we have

$$ \begin{aligned} \|u_{j}-u_{0}\|^{2}_{H_{0}^{s}(\Omega)}&= \|u_{j}\|^{2}_{H_{0}^{s}(\Omega)}+\|u_{0}\| ^{2}_{H_{0}^{s}(\Omega)} -2(u_{j},u_{0})_{H_{0}^{s}(\Omega)} \\ &\to 2\|u_{0}\|^{2}_{H_{0}^{s}(\Omega)}-2(u_{0},u_{0})_{H_{0}^{s}(\Omega)}=0, \end{aligned} $$

as $j\to+\infty$, thanks to (2.4) and (2.9). Then the assertion of Lemma 2.5 is proved. □

We define the operator $A: H_{0}^{s}(\Omega)\to H_{0}^{s}(\Omega)$ by

$$A(u)=(-\Delta)^{-s}\circ h(u), \quad\forall u\in H_{0}^{s}( \Omega), $$

where $h(u)(x):=\lambda u(x)+f(x,u(x))$, $\forall u\in H_{0}^{s}(\Omega)$, is the corresponding Nemytskii operator. $(-\Delta)^{-s}$ is equal to the inverse of the operator $(-\Delta )^{s}$ (see [7, 10, 16]). $A(u)$ is uniquely determined by the relation

$$\bigl(A(u),\phi \bigr)_{H_{0}^{s}(\Omega)}= \int_{\Omega}(-\Delta)^{s}A(u)\phi \,dx= \int_{\Omega} \bigl(\lambda u(x)+f \bigl(x,u(x) \bigr) \bigr)\phi \,dx, \quad\forall\phi\in H_{0}^{s}(\Omega). $$

Lemma 2.6

The operator $A: H_{0}^{s}(\Omega)\to H_{0}^{s}(\Omega)$ is compact and $I'(u)=u-A(u)$, $\forall u\in H_{0}^{s}(\Omega)$.

Proof

The proof is similarly to Lemma 5.1 in [7]. Indeed, together with ($f_{2}$) and the compact embedding $H_{0}^{s}(\Omega )\hookrightarrow L^{r}(\Omega)$ for $r\in[2,2^{*}_{s})$, it is easy to see that A is compact. Note that

$$ \begin{aligned} \bigl(I'(u),\phi \bigr)&= \int_{\Omega}(-\Delta)^{s}u(x)\phi(x) \,dx- \int_{\Omega} \bigl(\lambda u+f(x,u) \bigr)\phi(x) \,dx \\ &= \int_{\Omega}(-\Delta)^{s}u(x)\phi(x) \,dx- \int_{\Omega} (-\Delta )^{s}A(u)\phi(x) \,dx \\ &= \bigl(u-A(u),\phi \bigr)_{H_{0}^{s}(\Omega)}, \quad\forall u,\phi\in H_{0}^{s}(\Omega), \end{aligned} $$

which implies that $I'(u)=u-A(u)$. □

In order to apply the method of invariant set of descending flow (see [6]), we need the following concepts and conclusions given by Liu and Sun [6].

Let X be a Banach space, f a functional defined on X. Let $K=\{u: u\in X,f'(u)=0\}$ and $X_{0}=X\setminus K$.

Definition 2.7

see [6]

A Lipschitz continuous map $W: X_{0}\rightarrow X$ is called a pseudogradient vector field for f if it satisfies:

• (i) $\langle f'(u),W(u)\rangle\geq\frac{1}{2}\|f'(u)\|^{2}$, for all $u\in X_{0}$;

• (ii) $\|W(u)\|\leq2\|f'(u)\|$, for all $u\in X_{0}$.

Let W be a pseudogradient vector field for f and $u_{0}\in X_{0}$, consider the following initial value problems in $X_{0}$:

$$ \left \{ \textstyle\begin{array}{l} \frac{du(t)}{dt}=-W(u(t)), \quad t\geq0,\\ u(0)=u_{0}. \end{array}\displaystyle \right . $$

(2.10)

By the theory of ordinary differential equations in Banach spaces, problem (2.10) has a unique solution, denoted by $\varphi (t, u_{0})$ with maximal interval of existence $[0,\tau(u_{0}))$. By Definition 2.7(i),

$$\frac{d}{dt}f \bigl(\varphi(t,u_{0}) \bigr)= \bigl\langle f'(u),-W(u) \bigr\rangle \leq-\frac {1}{2} \big\| f'(u) \big\| ^{2}, $$

which implies that $I(\varphi(t,u_{0}))$ is strictly decreasing in $[0,\tau(u))$ and therefore $\varphi(t, u_{0})$ ($0\leq t<\tau(u)$) is called a descending flow curve associated with W.

Definition 2.8

see [6]

A nonempty subset M of X is called an invariant set of descending flow for f determined by W if

$$\bigl\{ \varphi(t,u_{0}): 0\leq t< \tau(u_{0}) \bigr\} \subset M $$

for all $u_{0}\in M\setminus K$.

By the definition, it is easy to see that X is an invariant set of descending flow. Let $M_{1}$ and $M_{2}$ be the invariant set of descending flow, both $M_{1}\cap M_{2}$ and $M_{1}\cup M_{2}$ are invariant sets of descending flow.

Theorem 2.9

see [6]

Assume that M is a closed invariant set of descending flow and f satisfies the (PS) condition on M. If $\inf_{u\in M}f(u)>-\infty$, then $\inf_{u\in M}f(u)$ is a critical value of f and f has critical point in M corresponding to this value.

Let M and D be invariant sets of descending flow for f, $D\subset M$. Denote

$$C_{M}(D):= \bigl\{ u_{0}\in D, \text{ or } u_{0} \in M\setminus D \text{ and there is } t\in\bigl[0,\tau(u_{0})\bigr) \text{ such that }\varphi (t,u_{0})\in D \bigr\} . $$

If $D=C_{M}(D)$, then D is called a complete invariant set of descending flow relative to M. Clearly, $C_{M}(D)\supset D$ and $C_{M}(D)$ is the maximal subset of M which is retracted by D and $C_{M}(D)$ is the minimal one of all complete invariant sets of descending flow containing D and contained in M.

Theorem 2.10

see [6]

Assume that M is connected, M is an invariant set of descending flow, D is an open subsets of M, and D is also an invariant set of descending flow. Then

• (i) $C_{M}(D)$ is an open subset of X;

• (ii) if $C_{M}(D)\neq M$ and $\inf_{u\in\partial_{M} D}f(u)>-\infty$, then $\inf_{u\in\partial_{M} C_{M}(D)}f(u)\geq \inf_{u\in\partial_{M} D}f(u)$.

Theorem 2.11

see [6]

H denotes a Hilbert space and f a $C^{1}$ function defined on H. Assume that $f\in C^{1}(H,\mathbb{R})$ and $f'(u)=u-A(u)$ for $u\in H$, $D_{1}$ and $D_{2}$ are open convex subset of H with the properties that $D_{1}\cap D_{2}\neq\varnothing$, $A(\partial D_{1})\subset D_{1}$ and $A(\partial D_{2})\subset D_{2}$. Then there is a pseudogradient vector field W for f which enables $D_{1}$ and $D_{2}$ to be invariant sets of descending flow, $C_{H}(D_{1})\supset\partial D_{1}$, and $C_{H}(D_{2})\supset \partial D_{2}$.

Theorem 2.12

see [6]

Assume that M is connected and is an invariant set of descending flow, D is an open subset of M and is a complete invariant set of descending flow relative to M. If $D\neq M$, then $\partial_{M}D$, the boundary of D relative to M, is nonempty and is a complete invariant set relative to M.

3 Proof of Theorem 1.1

The case $\lambda\leq0$: mountain pass sign-changing solution for problem (1.1). We consider the convex cones

$$P= \bigl\{ u\in H_{0}^{s}(\Omega): u\geq0\text{ a.e. in } \Omega \bigr\} \quad \text{and}\quad {-}P= \bigl\{ u\in H_{0}^{s}(\Omega): u \leq0\text{ a.e. in } \Omega \bigr\} . $$

For $\epsilon>0$, we denote

$$D^{+}_{\epsilon}= \bigl\{ u\in H_{0}^{s}(\Omega): \operatorname{dist}(u,p)< \epsilon \bigr\} \quad\text{and}\quad D^{-}_{\epsilon}= \bigl\{ u\in H_{0}^{s}(\Omega): \operatorname{dist}(u,-p)< \epsilon \bigr\} . $$

Clearly, for any $\epsilon>0$, $D^{+}_{\epsilon}$ and $D^{-}_{\epsilon}$ are open convex subsets of $H_{0}^{s}(\Omega)$ and $O_{\epsilon}:= D^{+}_{\epsilon}\cap D^{-}_{\epsilon}\neq\varnothing$. Denote $D_{\epsilon}=\bar{D}^{+}_{\epsilon}\cup\bar{D}^{-}_{\epsilon }$. Then $D_{\epsilon}$ is a closed and symmetric set in $H_{0}^{s}(\Omega )$. Moreover, $H_{0}^{s}(\Omega)\setminus D_{\epsilon}$ contains only sign-changing functions. In the following, K denotes critical point set of I,

$$K= \bigl\{ u\in H_{0}^{s}(\Omega): I'(u)=0 \bigr\} . $$

Lemma 3.1

Assume that ($f_{1}$)-($f_{4}$) and $\lambda\leq0$ hold. Then there exists $\epsilon_{0}>0$ such that

$$ A \bigl(\partial D^{+}_{\epsilon} \bigr)\subset D^{+}_{\epsilon} \quad\textit{and}\quad A \bigl(\partial D^{-}_{\epsilon} \bigr)\subset D^{-}_{\epsilon}, \quad\forall\epsilon\in(0, \epsilon_{0}]. $$

(3.1)

Moreover, if $u\in D^{+}_{\epsilon}$ (or $D^{-}_{\epsilon}$) is a solution of problem (1.1), then $u\in P$ (or −P).

Proof

We just prove the case $A(\partial D^{-}_{\epsilon})\subset D^{-}_{\epsilon}$. The other case can be obtained similarly. We write $u\in H_{0}^{s}(\Omega)$ as $u=u^{+}+u^{-}$, where $u^{+}=\text{max}\{u,0\}$, $u^{-}=\text{min}\{u,0\}$. For $u\in\partial D^{+}_{\epsilon}$, denote $v=A(u)$. Then

$$ \operatorname{dist}(v,-P)=\inf_{\phi\in(-P)}\|v-\phi\| _{H_{0}^{s}(\Omega)}\leq\big\| v^{+} \big\| _{H_{0}^{s}(\Omega)}. $$

(3.2)

By Proposition 2.3(b), for any $r\in[2,2^{*}_{s})$, it follows that there exists $C_{r}>0$ such that

$$ \big\| u^{+}\big\| _{L^{r}(\Omega)}=\inf_{\phi\in(-P)}\|u-\phi\| _{L^{r}(\Omega)}\leq C_{r}\inf_{\phi\in(-P)}\|u-\phi\| _{H_{0}^{s}(\Omega)}. $$

(3.3)

By ($f_{1}$)-($f_{4}$), for any $\epsilon_{1}>0$, there exists $C_{\epsilon _{1}}>0$ such that

$$ \big|f(x,t)\big|\leq\epsilon_{1}|t|+C_{\epsilon_{1}}|t|^{p-1},\quad \forall x\in \Omega, \forall t\in\mathbb{R}. $$

(3.4)

Using Proposition 2.3, (3.2)-(3.4) and $\lambda \leq0$ it follows that

$$ \begin{aligned}[b] \operatorname{dist}(v,-P)\big\| v^{+} \big\| _{H_{0}^{s}(\Omega)}\leq{}&\big\| v^{+}\big\| ^{2}_{H_{0}^{s}(\Omega )}=\bigl(v,v^{+} \bigr)_{H_{0}^{s}(\Omega)}=\bigl(A(u),v^{+}\bigr)_{H_{0}^{s}(\Omega)} \\ ={}& \int_{\Omega}\bigl(\lambda u+f(x,u)\bigr)v^{+}\,dx \\ \leq{}& \int_{\Omega}\bigl(\epsilon_{1}\big|u^{+}\big|\big|v^{+}\big|+C_{\epsilon _{1}}\big|u^{+}\big|^{p-1}\big|v^{+}\big| \bigr)\,dx \\ \leq{}&\epsilon_{1}\big\| u^{+}\big\| _{L^{2}(\Omega)}\big\| v^{+}\big\| _{L^{2}(\Omega)}+C_{\epsilon_{1}} \big\| u^{+}\big\| ^{p-1}_{L^{p}(\Omega)}\big\| v^{+}\big\| _{L^{p}(\Omega)} \\ \leq{}& \Bigl[\epsilon_{1}C_{1}C_{2}\inf _{\phi\in(-P)}\|u-\phi\| _{H_{0}^{s}(\Omega)}\\ &+C_{3}C_{\epsilon_{1}}C_{p}^{p-1} \inf_{\phi\in(-P)}\|u-\phi\|_{H_{0}^{s}(\Omega)}^{p-1} \Bigr]\big\| v^{+} \big\| _{H_{0}^{s}(\Omega)}, \end{aligned} $$

(3.5)

where $C_{i}>0$, $i=1,2,3,p$ denote the embedding constant.

By (3.5), letting $\epsilon_{1}=\frac{1}{4C_{1}C_{2}}$, we have

$$\operatorname{dist}(v,-P)\leq\frac{1}{4}\operatorname{dist}(u,-P)+M\operatorname{dist}(u,-P)^{p-1}, $$

where $M=C_{3}C_{\epsilon_{1}}C_{p}^{p-1}$. Take $\epsilon_{0}\in (0,(4M)^{-\frac{1}{p-2}})$. If $\operatorname{dist}(u,-P)\leq\epsilon\leq\epsilon _{0}$, then

$$ \operatorname{dist}(v,-P)\leq\frac{1}{4}\operatorname{dist}(u,-P)+M\operatorname {dist}(u,-P)\frac{1}{4M}=\frac{1}{2}\operatorname{dist}(u,-P)< \epsilon, $$

(3.6)

which implies that $v=A(u)\in D^{-}_{\epsilon}$, i.e., $A(\partial D^{-}_{\epsilon})\subset D^{-}_{\epsilon}$.

Now if $u\in D^{-}_{\epsilon}$, $0<\epsilon\leq\epsilon_{0}$, is a nontrivial solution of problem (1.1), then $u=A(u)$ and hence by (3.6) if follows that dist$(u,-P)=0$, i.e., $u\in -P$. In a similar way we see that $u\in D^{+}_{\epsilon}\cap(K\setminus \{0\})$ implies $u\in P$. □

Using both Theorem 2.11 and Lemma 3.1, we have the following corollary.

Lemma 3.2

For any $\epsilon\in(0,\epsilon_{0}]$ and $u\in D^{\pm }_{\epsilon}\setminus K$, there is a pseudogradient vector field W for I which enables $D^{+}_{\epsilon}$ and $D^{-}_{\epsilon}$ to be invariant sets of descending flow. Then $O_{\epsilon}$ and $D_{\epsilon}$ are also invariant sets of descending flow. Here $\epsilon_{0}$ is taken from Lemma 3.1.

In order to apply Theorem 2.9, we have to check the validity of the Palais-Smale condition. Thanks to Lemma 2.5, we only need to verify Lemma 3.3.

Lemma 3.3

Assume that ($f_{1}$)-($f_{4}$) and $\lambda\leq0$ hold. Let $c\in \mathbb{R}$ and let $\{u_{j}\}$ be $(\mathit{PS})_{c}$ sequence in $H_{0}^{s}(\Omega )$. Then $\{u_{j}\}$ is bounded in $H_{0}^{s}(\Omega)$.

Proof

For any $j\in\mathbb{N}$ by (2.3) it easily follows that there exists $\kappa> 0$ such that

$$ \begin{aligned}[c] &\big|I(u_{j})\big|\leq\kappa, \\ &\bigg|\biggl\langle I'(u_{j}),\frac{u_{j}}{\|u_{j}\|_{H_{0}^{s}(\Omega)}}\biggr\rangle \bigg|\leq \kappa. \end{aligned} $$

(3.7)

By ($f_{2}$) and ($f_{4}$), we obtain

$$ \big|f(x,t)\big|\leq2\varepsilon|t|+c_{\varepsilon}p|t|^{p-1} \quad\text{and}\quad \big|F(x,t)\big|\leq\varepsilon|t|^{2}+c_{\varepsilon} |t|^{p}. $$

(3.8)

Let $\varepsilon= 1$, we have

$$ \begin{aligned}[b] &\bigg| \int_{\Omega\cap\{|u_{j}|\leq r\}}\biggl(F(x,u_{j})-\frac{1}{\mu }f(x,u_{j})u_{j} \biggr)\,dx\bigg| \\ &\quad\leq\biggl(r^{2}+c_{1}r^{p}+\frac{2}{\mu}r+ \frac{p}{\mu}c_{1}r^{p-1}\biggr)\Omega =:\bar{\kappa}. \end{aligned} $$

(3.9)

Also, by $\lambda\leq0$ and ($f_{3}$) and (3.9) we get

$$ \begin{aligned}[b] &I(u_{j})- \frac{1}{\mu}\bigl\langle I'(u_{j}),u_{j} \bigr\rangle \\ &\quad=\biggl(\frac{1}{2}-\frac{1}{\mu}\biggr) \bigl(\|u_{j} \|^{2}_{H_{0}^{s}(\Omega)}-\lambda\| u_{j}\|^{2}_{L^{2}(\Omega)} \bigr) - \int_{\Omega}\biggl(F(x,u_{j})-\frac{1}{\mu}f(x,u_{j})u_{j} \biggr)\,dx \\ &\quad\geq\biggl(\frac{1}{2}-\frac{1}{\mu}\biggr)\|u_{j} \|^{2}_{H_{0}^{s}(\Omega)}- \int_{\Omega\cap\{|u_{j}|\leq r\}} \biggl(F(x,u_{j})-\frac{1}{\mu }f(x,u_{j})u_{j} \biggr)\,dx \\ &\quad\geq\biggl(\frac{1}{2}-\frac{1}{\mu}\biggr)\|u_{j} \|^{2}_{H_{0}^{s}(\Omega)}-\bar {\kappa}. \end{aligned} $$

(3.10)

As a consequence of (3.7) and (3.10) we obtain

$$\kappa\bigl(1+\|u_{j}\|_{H_{0}^{s}(\Omega)}\bigr)\geq I(u_{j})- \frac{1}{\mu} \bigl\langle I'(u_{j}),u_{j} \bigr\rangle \geq \biggl(\frac{1}{2}-\frac{1}{\mu} \biggr) \|u_{j}\| ^{2}_{H_{0}^{s}(\Omega)}-\bar{\kappa}. $$

Hence, $\{u_{j}\}$ is bounded in $H_{0}^{s}(\Omega)$. □

Lemma 3.4

Assume that ($f_{1}$)-($f_{4}$) and $\lambda\leq 0$ hold. For any $c\in\mathbb{R}$, there exists constant $\rho _{1}=\rho_{1}(c)>0$ such that

$$\|u\|_{H^{s}_{0}(\Omega)}+\big\| A(u)\big\| _{H^{s}_{0}(\Omega)}\leq\rho_{1} \bigl(1+\big\| u-A(u)\big\| _{H^{s}_{0}(\Omega)} \bigr) $$

for every $u\in I^{c}:=\{u\in H^{s}_{0}(\Omega): I(u)\leq c\}$.

Proof

For $u\in H^{s}_{0}(\Omega)$, by ($f_{3}$) and $\lambda \leq0$ it follows that

$$ \begin{aligned}[b] &I(u)-\frac{1}{\mu}\bigl\langle I'(u),u\bigr\rangle \\ &\quad =\biggl(\frac{1}{2}-\frac{1}{\mu}\biggr) \int_{\Omega}\big|(-\Delta)^{\frac{s}{2}}u\big|^{2}\,dx -\biggl( \frac{1}{2}-\frac{1}{\mu}\biggr)\lambda \int_{\Omega}|u|^{2}\,dx\\ &\qquad- \int _{\Omega}\biggl(F(x,u)-\frac{1}{\mu}uf(x,u)\biggr)\,dx \\ &\quad\geq\biggl(\frac{1}{2}-\frac{1}{\mu}\biggr)\|u\|^{2}_{H^{s}_{0}(\Omega)}. \end{aligned} $$

If $u\in I^{c}$, then there exists $M_{1}>0$ such that

$$ \begin{aligned} \|u\|^{2}_{H^{s}_{0}(\Omega)}&\leq M_{1}\bigl(1+\big|\bigl\langle I'(u),u\bigr\rangle \big|\bigr) \\ &\leq M_{1}\bigl(1+\|u\|_{H^{s}_{0}(\Omega)}\big\| u-A(u)\big\| _{H^{s}_{0}(\Omega)} \bigr). \end{aligned} $$

Moreover, for any $\varepsilon> 0$ the Young inequality gives

$$ \|u\|^{2}_{H^{s}_{0}(\Omega)}\leq M_{1} \biggl(1+\frac{\varepsilon^{2}}{2}\|u\| ^{2}_{H^{s}_{0}(\Omega)}+\frac{1}{2\varepsilon^{2}} \big\| u-A(u)\big\| ^{2}_{H^{s}_{0}(\Omega)}\biggr). $$

Letting $\varepsilon<\text{min}\{\sqrt{\frac{1}{2}},\sqrt{\frac {2}{M_{1}}}\}$, then

$$ \begin{aligned} \biggl(1-M_{1}\frac{\varepsilon^{2}}{2}\biggr)\|u \|^{2}_{H^{s}_{0}(\Omega)}&\leq M_{1}\biggl(1+\frac{1}{2\varepsilon^{2}} \big\| u-A(u)\big\| ^{2}_{H^{s}_{0}(\Omega)}\biggr) \\ &\leq M_{1}\frac{1}{2\varepsilon^{2}}\bigl(1+\big\| u-A(u)\big\| ^{2}_{H^{s}_{0}(\Omega)} \bigr) \\ &\leq\frac{M_{1}}{2\varepsilon^{2}}\bigl(1+\big\| u-A(u)\big\| _{H^{s}_{0}(\Omega)}\bigr)^{2}. \end{aligned} $$

We get some $M_{2}>0$ such that

$$ \|u\|_{H^{s}_{0}(\Omega)}\leq M_{2} \bigl(1+\big\| u-A(u) \big\| _{H^{s}_{0}(\Omega)} \bigr), $$

which implies that

$$ \begin{aligned} \|u\|_{H^{s}_{0}(\Omega)}+\big\| A(u)\big\| _{H^{s}_{0}(\Omega)}&\leq2\|u \| _{H^{s}_{0}(\Omega)}+\big\| u-A(u)\big\| _{H^{s}_{0}(\Omega)} \\ &\leq(2M_{2}+1) \bigl(1+\big\| u-A(u)\big\| _{H^{s}_{0}(\Omega)}\bigr). \end{aligned} $$

Then Lemma 3.4 follows by taking $\rho_{1}=2M_{2}+1$. □

Lemma 3.5

Regularity of solutions

Under the assumptions of Theorem 1.1 or Theorem  1.2, if $u\in H^{s}_{0}(\Omega)$ is a weak solution of (1.1), then $u\in C^{\sigma}(\bar{\Omega })\cap C_{0}(\bar{\Omega})$ for some $\sigma\in(0,1)$. Moreover, if f is a $C^{1,\sigma}$ function, then $u\in C^{1,\sigma} (\bar{\Omega})\cap C_{0}(\bar{\Omega})$.

Proof

The proof is similar to Theorem 3.1 in [7] (see also [16]), we omit it. □

Proof of Theorem 1.1 (i)

In what follows, we use Theorem 2.9 to prove our problems.

(Trivial solution) By (3.8) and $\lambda\leq0$ it follows that

$$ \begin{aligned} I(u)&\geq\frac{1}{2}\|u\|_{H_{0}^{s}(\Omega)}^{2}- \frac{\lambda}{2} \int _{\Omega}|u|^{2}\,dx-\varepsilon \int_{\Omega}|u|^{2}\,dx -c_{\varepsilon} \int_{\Omega}|u|^{p}\,dx \\ &\geq\varepsilon\|u\|^{2}_{L^{2}(\Omega)}-c_{\varepsilon}\|u\| ^{p}_{L^{p}(\Omega)} \\ &\geq\varepsilon|\Omega|^{\frac{p-2}{p}}\|u\|^{2}_{L^{p}(\Omega )}-c_{\varepsilon} \|u\|^{p}_{L^{p}(\Omega)}, \end{aligned} $$

when ε is small enough.

By (3.3), we can see that, if $u\in\bar{D}^{+}_{\epsilon }\cap\bar{D}^{-}_{\epsilon}$ with $\epsilon\in [0,\epsilon_{0}]$, where $\epsilon_{0}$ is taken from Lemma 3.2, then there exists $C_{p}>0$ such that

$$\big\| u^{\mp}\big\| _{L^{p}(\Omega)}\leq C_{p}\inf_{\phi\in(\pm P)} \| u-\phi \|_{H_{0}^{s}(\Omega)}=C_{p}\operatorname{dist}(u,\pm P)\leq C_{p}\epsilon. $$

Therefore, there exists $\alpha_{\epsilon}>-\infty$ such that $\inf_{u\in\bar{D}^{+}_{\epsilon}\cap\bar{D}^{-}_{\epsilon }}I(u)=\alpha_{\epsilon}$. By Lemma 3.2, $\bar{D}^{+}_{\epsilon}\cap\bar{D}^{-}_{\epsilon}$ is a closed invariant set of descending flow. According to Theorem 2.9, I has a critical point $u_{0}\in\bar {D}^{+}_{\epsilon}\cap\bar{D}^{-}_{\epsilon}$ with critical value $\inf_{u\in\bar{D}^{+}_{\epsilon}\cap\bar{D}^{-}_{\epsilon}}I(u)$. It is easy to see that $D^{+}_{\epsilon}$ and $D^{-}_{\epsilon}$ are connected open invariant sets of descending flow for I, so $\partial D^{+}_{\epsilon}\cap K=\varnothing$ and $D^{-}_{\epsilon}\cap K=\varnothing$. Then we have $u_{0}\in D^{+}_{\epsilon}\cap D^{-}_{\epsilon}$. By Lemma 3.1, we get $u_{0}\in P\cap(-P)\equiv0$.

(Positive and negative solutions) By ($f_{3}$), there exist $c_{1}>0$ and $c_{2}>0$ such that

$$ F(x,t)\geq c_{1}|t|^{\mu}-c_{2},\quad \forall x\in\Omega, \forall t\in \mathbb{R}, $$

(3.11)

then, by $H_{0}^{s}(\Omega)\hookrightarrow L^{2}(\Omega)$,

$$ \begin{aligned} I(u)&\leq\frac{1}{2}\|u\|_{H_{0}^{s}(\Omega)}^{2}- \frac{\lambda}{2} \int _{\Omega}|u|^{2}\,dx-c_{1} \int_{\Omega}|u|^{\mu}\,dx+c_{2}|\Omega| \\ &\leq\frac{1}{2}(1-\tilde{c}\lambda)\|u\|_{H_{0}^{s}(\Omega)}^{2}-c_{1} \| u\|^{\mu}_{L^{\mu}(\Omega)}+c_{2}|\Omega|, \end{aligned} $$

where $\tilde{c}>0$ is embedding constant.

Let $\{e_{k}\}$ be the eigenfunctions of −Δ introduced in Proposition 2.2. Take $h_{R}(t)=Re_{1}\operatorname{cos}(\pi t)+Re_{2}\operatorname{sin}(\pi t)$ for $t\in[0,1]$, for some $R\in\mathbb{R}^{+}$. Then

$$ \begin{aligned} I\bigl(h_{R}(t)\bigr)\leq{}& \frac{1}{2}(1-\tilde{c}\lambda)\big\| h_{R}(t)\big\| _{H_{0}^{s}(\Omega)}^{2}-c_{1} \int_{\Omega}\big|h_{R}(t)\big|^{\mu}\,dx+c_{2}| \Omega| \\ ={}& \frac{1}{2}(1-\tilde{c}\lambda)R^{2}\big\| e_{1}\cos(\pi t)+e_{2}\sin(\pi t)\big\| _{H_{0}^{s}(\Omega)}^{2}\\ &-c_{1}R^{\mu} \big\| e_{1}\cos(\pi t)+e_{2}\sin(\pi t)\big\| ^{\mu }_{L^{\mu}(\Omega)}+c_{2}| \Omega|. \end{aligned} $$

Since $\mu>2$, there exists $R_{\epsilon}>0$ large enough such that $I(h_{R_{\epsilon}}(t))<\bar{\alpha}:=\inf_{\epsilon\in [0,\epsilon_{0}]}\alpha_{\epsilon}$, and then

$$I \bigl(h_{R_{\epsilon}}(0) \bigr)\leq\max_{t\in[0,1]}I \bigl(h_{R_{\epsilon }}(t) \bigr)< \inf_{u\in\bar{D}^{+}_{\epsilon}\cap\bar {D}^{-}_{\epsilon}}I(u). $$

By Proposition 2.2 ($e_{1}$ is positive), we have

$$h_{R_{\epsilon}}(0)=R_{\epsilon}e_{1}\in D^{+}_{\epsilon} \setminus\bar {D}^{-}_{\epsilon},\qquad h_{R_{\epsilon}}(1)=-R_{\epsilon}e_{1} \in D^{-}_{\epsilon}\setminus\bar{D}^{+}_{\epsilon}. $$

Note that $\bar{D}^{+}_{\epsilon}\cap D^{-}_{\epsilon}$ is an open subsets of $\bar{D}^{+}_{\epsilon}$ and an invariant set of descending flow. It is easy to see that $h_{R_{\epsilon}}(0)\in\bar{D}^{+}_{\epsilon}\setminus C_{\bar {D}^{+}_{\epsilon}}(\bar{D}^{+}_{\epsilon}\cap D^{-}_{\epsilon})$. Therefore, $C_{\bar{D}^{+}_{\epsilon}}(\bar{D}^{+}_{\epsilon}\cap D^{-}_{\epsilon})\neq\bar{D}^{+}_{\epsilon}$, then, by Theorem 2.12, $\partial_{\bar{D}^{+}_{\epsilon}}C_{\bar{D}^{+}_{\epsilon}}(\bar {D}^{+}_{\epsilon}\cap D^{-}_{\epsilon})$ is nonempty and is a complete set relative to $\bar{D}^{+}_{\epsilon}$. By Theorem 2.10, $C_{\bar{D}^{+}_{\epsilon}}(\bar{D}^{+}_{\epsilon}\cap D^{-}_{\epsilon})$ is an open subset of $\bar{D}^{+}_{\epsilon}$ and

$$\inf_{\partial_{\bar{D}^{+}_{\epsilon}}C_{\bar{D}^{+}_{\epsilon }}(\bar{D}^{+}_{\epsilon}\cap D^{-}_{\epsilon})}I(u)\geq \inf_{\partial_{\bar{D}^{+}_{\epsilon}}(\bar{D}^{+}_{\epsilon}\cap D^{-}_{\epsilon})}I(u)\geq \inf _{\bar{D}^{+}_{\epsilon}\cap D^{-}_{\epsilon}}I(u)>-\infty. $$

According to Theorem 2.9, I has a critical point $u_{+}$ in $\partial _{\bar{D}^{+}_{\epsilon}}C_{\bar{D}^{+}_{\epsilon}}(\bar {D}^{+}_{\epsilon}\cap D^{-}_{\epsilon})$. The facts that $\partial_{\bar{D}^{+}_{\epsilon}}C_{\bar{D}^{+}_{\epsilon}}(\bar {D}^{+}_{\epsilon}\cap D^{-}_{\epsilon})\subset\bar{D}^{+}_{\epsilon}$ and $\partial D^{+}_{\epsilon}\cap K=\varnothing$ imply $u_{+}\in D^{+}_{\epsilon}$, while $\partial_{\bar{D}^{+}_{\epsilon}}C_{\bar{D}^{+}_{\epsilon}}(\bar {D}^{+}_{\epsilon}\cap D^{-}_{\epsilon})\cap \bar{D}^{+}_{\epsilon}\cap D^{-}_{\epsilon}=\varnothing$ and $\partial D^{-}_{\epsilon}\cap K=\varnothing$ imply $u_{+}\notin\bar{D}^{-}_{\epsilon}$. Hence $u_{+}\in D^{+}_{\epsilon}\setminus\bar{D}^{-}_{\epsilon}$. Similarly, I has a critical point $u_{-}\in D^{-}_{\epsilon}\setminus \bar{D}^{+}_{\epsilon}$.

By Lemma 3.5, $u_{+}\in C^{\sigma}(\bar{\Omega})$, $u_{-}\in C^{\sigma }(\bar{\Omega})$. By Lemma 3.1 in follows that $u_{+}\geq0$ and $u_{-}\leq0$. Applying the strong maximum principle (see [17]), we get $u_{+}>0$ and $u_{-}<0$ in Ω. □

To estimate the number of nodal domains, we need the following results.

Proposition 3.6

see [7, 18]

Assume that $\Omega\subset\mathbb{R}^{n}$ ($n\geq2$) is a bounded open set and $u\in C(\bar{\Omega})\cap H^{s}_{0}(\Omega)$. Choose a subdomain $\Omega_{1}$ of Ω such that $u(x)=0$ for $x\in\partial\Omega_{1}\cap\Omega$. Then the restriction $u_{\Omega_{1}}$ of u to $\Omega_{1}$ belongs to $H_{0}^{s}(\Omega_{1})$. Furthermore, defining

$$ z(x)= \left \{ \textstyle\begin{array}{l@{\quad}l} u(x) &\textit{in } \Omega, \\ 0 &\textit{on } \Omega\setminus\Omega_{1}, \end{array}\displaystyle \right . $$

we have $z\in H^{s}_{0}(\Omega)$.

In what follows, we come to prove the second part of Theorem 1.1.

Proof of Theorem 1.1 (ii)

We introduce a set $\mathcal{N}$ associated to I as follows:

$$\mathcal{N}:= \bigl\{ u\in H^{s}_{0}(\Omega): u^{+}\neq0, u^{-} \neq0, I'(u)u^{+}=I'(u)u^{-}=0 \bigr\} . $$

Obviously the set $\mathcal{N}$ contains all sign-changing solutions of (1.1). Now set

$$\zeta=\inf_{u\in\mathcal{N}}I(u). $$

For $u\in\mathcal{N}$, we have from $\lambda\leq0$

$$ \|u\|^{2}_{H^{s}_{0}(\Omega)}-\lambda\|u\|^{2}_{L^{2}(\Omega)}= \int_{\Omega }f(x,u)u\,dx\geq0. $$

Then, by ($f_{3}$), we have

$$ I(u)= \int_{\Omega} \biggl[\frac{1}{2}f(x,u)u-F(x,u) \biggr]\,dx\geq \biggl(\frac{1}{2}-\frac{1}{\mu} \biggr) \int_{\Omega}f(x,u)u\,dx\geq0. $$

This proves $\zeta>-\infty$.

Write $u_{n}=u_{n}^{+}+u_{n}^{-}$, where $u_{n}^{+}=\max\{u_{n},0\}$ and $u_{n}^{-}=\{u_{n},0\}$. We claim that

$$ I \bigl(u_{n}^{\pm} \bigr)=\max _{t\in[0,+\infty)}I \bigl(tu_{n}^{\pm} \bigr). $$

(3.12)

For this purpose we set $h^{\pm}(t):= I(tu_{n}^{\pm})$ for $t\geq0$. Then $h^{\pm}(0)= 0$, and

$$\bigl(h^{\pm} \bigr)'(t)= \bigl\langle I' \bigl(tu_{n}^{\pm} \bigr),u_{n}^{\pm} \bigr\rangle =t \biggl(\big\| u_{n}^{\pm}\big\| _{H^{s}_{0}(\Omega)}^{2}- \lambda\big\| u_{n}^{\pm}\big\| _{L^{2}(\Omega)}^{2} - \int_{\Omega}\frac{f(x,tu_{n}^{\pm})}{tu_{n}^{\pm}} \bigl(u_{n}^{\pm } \bigr)^{2}\,dx \biggr). $$

Hence ($f_{5}$) implies that $t\mapsto\frac{(h^{\pm})'(t)}{t}$ is nonincreasing on $(0,\infty)$, and thus the set $\Theta:= \{t > 0: (h^{\pm})'(t)= 0\}$ is a closed subinterval of $(0,\infty)$ which contains $t = 1$ by definition of $\mathcal{N}$. Moreover, $h^{\pm}$ is increasing on $(0,\min\Theta)$ and decreasing on $(\max\Theta,\infty)$, hence

$$h^{\pm}(1)=\max_{t\in\Theta}h^{\pm}(t)=\max _{t\geq0}h^{\pm}(t). $$

This proves (3.12). Setting

$$\Pi_{n}:= \bigl\{ tu_{n}^{+}+su_{n}^{-}:t\geq0,s\geq0 \bigr\} , \quad n\in\mathbb{N}, $$

(3.12) implies that

$$\sup I(\Pi_{n})=\max_{t\geq0}I \bigl(tu_{n}^{+} \bigr)+\max_{t\geq0}I \bigl(tu_{n}^{-} \bigr)=I \bigl(u_{n}^{+} \bigr)+I \bigl(u_{n}^{-} \bigr). $$

Note that, fixing a n, $u_{n}^{+}$ and $u_{n}^{-}$ are linearly independent and $\Pi_{n}$ is a two-dimensional subspace of $H^{s}_{0}(\Omega)$. Since all norms in a finite dimensional space are equivalent, we have

$$ \begin{aligned}[b] I(u)&=\frac{1}{2}\|u\|^{2}_{H^{s}_{0}(\Omega)}- \frac{\lambda}{2}\|u\| ^{2}_{L^{2}(\Omega)}- \int_{\Omega}F(x,u)\,dx \\ &\leq \biggl(\frac{1}{2}-\frac{\lambda}{2}c_{0} \biggr)\|u\| ^{2}_{H^{s}_{0}(\Omega)}-c_{1}\|u\|^{\mu}_{H^{s}_{0}(\Omega)}+c_{2}| \Omega| \end{aligned} $$

(3.13)

for all $u\in\Pi_{n}$. Let

$$B_{R_{n}}:=\bigl\{ u\in\Pi_{n}: \|u\|_{H^{s}_{0}(\Omega)}< R_{n} \bigr\} . $$

Then by (3.13) and $\mu>2$, there exists $R_{n}>0$ such that

$$I(u)\leq-1 \quad\text{for all } u\in\Pi_{n}\setminus B_{R_{n}}. $$

Now we define the path

$$h_{n}: [0,1]\to H^{s}_{0}(\Omega),\quad h_{n}(t)=t \frac{R_{n}}{\|u_{n}^{+}\| _{H^{s}_{0}(\Omega)}}u_{n}^{+}+(1-t) \frac{R_{n}}{\|u_{n}^{-}\|_{H^{s}_{0}(\Omega)}}u_{n}^{-} $$

connecting $h_{n}(0)\in(\Pi_{n}\setminus B_{R_{n}}(0))\cap C_{X}(D^{-}_{\epsilon})$ and $h_{n}(1)\in(\Pi_{n}\setminus B_{R_{n}}(0))\cap C_{X}(D^{+}_{\epsilon})$ for all $\epsilon\in(0,\epsilon_{0}]$. Let $Q:=[0,1]\times[0,1]$, define the homotopy map $H_{n}: Q\to H^{s}_{0}(\Omega)$ by

$$H_{n}(t,s)=sh_{n}(t),\quad \forall(t,s)\in Q. $$

Note that $O_{\epsilon}$ is an open set in $H^{s}_{0}(\Omega)$, Theorem 2.10(i) implies that $C_{X}(O_{\epsilon})$ is an open set in $H^{s}_{0}(\Omega)$, then $\mathcal{O}_{n}:=H^{-1}_{n}(C_{X}(O_{\epsilon}))$ and $\mathcal{O}^{\pm}_{n}:=H^{-1}_{n}(C_{X}(D^{\pm}_{\epsilon}))$ are open subsets of Q. By $H_{n}(t,0)=0\in C_{X}(O_{\epsilon})$ for all $t\in[0,1]$ it follows that $\{(t,0):t\in[0,1]\}\subset\mathcal{O}_{n}$. In addition, by ($f_{3}$), we have

$$ \begin{aligned} I\bigl(H_{n}(t,1)\bigr)&=I \bigl(h_{n}(t)\bigr)=\frac{1}{2} \int_{\Omega}\big|(-\Delta)^{\frac {s}{2}}h_{n}(t)\big|^{2}\,dx- \frac{1}{2}\lambda \int_{\Omega} \big|h_{n}(t)\big|^{2}\,dx- \int_{\Omega}F\bigl(x,h_{n}(t)\bigr)\,dx \\ &\leq\frac{1}{2}\big\| h_{n}(t)\big\| ^{2}_{H^{s}_{0}(\Omega)}- \frac{1}{2}\lambda\big\| h_{n}(t)\big\| ^{2}_{L^{2}(\Omega)}-c_{1} \big\| h_{n}(t)\big\| ^{\mu}_{L^{\mu}(\Omega)} +c_{2}|\Omega| \\ &=\frac{1}{2}R_{n}^{2}\big\| \gamma_{n}(t) \big\| ^{2}_{H^{s}_{0}(\Omega)}-\frac {1}{2}\lambda R_{n}^{2} \big\| \gamma_{n}(t)\big\| ^{2}_{L^{2}(\Omega)}-c_{1} R^{\mu}_{n}\big\| \gamma_{n}(t)\big\| ^{\mu}_{L^{\mu}(\Omega)}+c_{2}| \Omega|, \end{aligned} $$

where $\gamma_{n}(t):=t\frac{u_{n}^{+}}{\|u_{n}^{+}\|_{H^{s}_{0}(\Omega )}}+(1-t)\frac{u_{n}^{-}}{\|u_{n}^{-}\|_{H^{s}_{0}(\Omega)}}$, it is easily to see that there exists $R_{0}>0$ such that $I(H_{n}(t,1))\leq-1$ for all $t\in[0,1]$ when $R_{n}\geq R_{0}$. Recall that $O_{\epsilon}$ is an invariant set of descending flow for I and $C_{X}(O_{\epsilon})\supset O_{\epsilon}$, we have

$$\inf_{u\in C_{X}(O_{\epsilon})}I(u)\geq\inf_{u\in O_{\epsilon}}I(u)\geq\inf _{u\in O_{\epsilon}} I \bigl(\varphi(t,u) \bigr)=0. $$

Hence,

$$\bigl\{ (t,1): t\in[0,1] \bigr\} \cap\mathcal{O}_{n}=\varnothing. $$

By Lemma 3.1 in [6], there exists a connected component $P_{n}$ of $\partial_{Q}\mathcal{O}_{n}$ intersecting the sets $\{(0,s):s\in [0,1]\}\subset\mathcal{O}_{n}^{-}$ and $\{(1,s):s\in[0,1]\}\subset \mathcal{O}_{n}^{+}$.

Furthermore, since

$$ \begin{gathered} H_{n}(P_{n})\subset H_{n}(\partial_{Q}\mathcal{O}_{n})\subset\partial H_{n}(\mathcal{O}_{n})=\partial C_{X}(O_{\epsilon}), \\ H_{n}\bigl(\mathcal{O}_{n}^{+}\cap\mathcal{O}_{n}^{-} \bigr)=H_{n}\bigl(\mathcal{O}_{n}^{+}\bigr)\cap H_{n} \bigl(\mathcal{O}_{n}^{-}\bigr)=C_{X}\bigl(D^{+}_{\epsilon} \bigr) \cap C_{X}\bigl(D^{-}_{\epsilon}\bigr)\subset C_{X}(O_{\epsilon}), \end{gathered} $$

it follows that

$$P_{n}\cap\mathcal{O}^{+}_{n}\cap\mathcal{O}^{-}_{n}= \varnothing. $$

By

$$\partial_{Q} \mathcal{O}_{n}\cap\mathcal{O}^{\pm}_{n} \supset P_{n}\cap \mathcal{O}^{\pm}_{n}\neq \varnothing, $$

we may take $(t_{n},s_{n})\in P_{n}\setminus(\mathcal{O}_{n}^{+}\cup\mathcal {O}_{n}^{-})$. Clearly,

$$v^{*}_{n}:=H_{n}(t_{n},s_{n})\in \mathcal{M}_{\epsilon}:=\partial C_{X}(O_{\epsilon})\setminus \bigl(C_{X} \bigl(D^{+}_{\epsilon} \bigr)\cup C_{X} \bigl(D^{-}_{\epsilon} \bigr) \bigr). $$

Note that $\partial C_{X}(O_{\epsilon})$ and $C_{X}(D_{\epsilon}^{\pm})$ are all invariant sets of $\varphi(t,v^{*}_{n})$, we get

$$ \bigl\{ \varphi \bigl(t,v^{*}_{n} \bigr): 0\leq t\leq\tau \bigl(v^{*}_{n} \bigr) \bigr\} \subset\mathcal {M}_{\epsilon}. $$

(3.14)

By (3.3), it follows that $\|u^{\pm}\|_{L^{r}(\Omega)}\leq C_{r}\epsilon_{0}$ for every $u\in\bar{D}^{+}_{\epsilon}\cap\bar {D}^{-}_{\epsilon}$, where $2\leq r<2^{*}_{s}$. Together with ($f_{5}$), Lemma 3.2 and the above arguments, we have

$$ \inf_{t\in[0,\tau(v^{*}_{n}))}I \bigl(\varphi \bigl(t,v^{*}_{n} \bigr) \bigr)\geq \inf_{u\in\partial C_{X}(O_{\epsilon})}I(u)\geq \inf_{u\in C_{X}(O_{\epsilon})}I(u)\geq0. $$

(3.15)

For $0< t_{1}< t_{2}<\tau(v^{*}_{n})$, we obtain

$$ \begin{aligned} \big\| \varphi\bigl(t_{2},v^{*}_{n} \bigr)-\varphi\bigl(t_{1},v^{*}_{n}\bigr)\big\| _{H^{s}_{0}(\Omega)}&\leq \int _{t_{1}}^{t_{2}}\big\| \varphi\bigl(s,v^{*}_{n} \bigr)-B\bigl(\varphi\bigl(s,v^{*}_{n}\bigr)\bigr)\big\| _{H^{s}_{0}(\Omega )}\,ds \\ &\leq2 \int_{t_{1}}^{t_{2}}\big\| \varphi\bigl(s,v^{*}_{n} \bigr)-A\bigl(\varphi\bigl(s,v^{*}_{n}\bigr)\bigr)\big\| _{H^{s}_{0}(\Omega)}\,ds \\ &\leq2 \biggl( \int_{t_{1}}^{t_{2}}\big\| \varphi\bigl(s,v^{*}_{n} \bigr)-A\bigl(\varphi \bigl(s,v^{*}_{n}\bigr)\bigr)\big\| ^{2}_{H^{s}_{0}(\Omega)}\,ds \biggr)^{\frac{1}{2}} (t_{2}-t_{1})^{\frac{1}{2}} \\ &\leq2 \bigl( I\bigl(\varphi\bigl(t_{1},v^{*}_{n}\bigr) \bigr)-I\bigl(\varphi\bigl(t_{2},v^{*}_{n}\bigr)\bigr) \bigr)^{\frac{1}{2}}(t_{2}-t_{1})^{\frac{1}{2}} \\ &\leq2\sqrt{I\bigl(v^{*}_{n}\bigr)}\sqrt{(t_{2}-t_{1})}. \end{aligned} $$

Now two cases may occur. If $\tau(v^{*}_{n})<\infty$, then the previous inequality implies that

$$ \lim_{t\to\tau(v^{*}_{n})^{-}}\big\| \varphi \bigl(t,v^{*}_{n} \bigr)-u_{n}^{*} \big\| _{H^{s}_{0}(\Omega)}=0 $$

(3.16)

for some $u_{n}^{*}\in H^{s}_{0}(\Omega)$.

If $\tau(v_{n}^{*})=+\infty$, by (3.15) ($I(\varphi (t,v^{*}_{n}))$ ($0\leq t<\tau(v_{n}^{*})$) is bounded), there is an increasing sequence $\{t_{n}\}$ with $t_{n}\to+\infty$ as $n\in+\infty$ such that

$$ \begin{aligned}[b] 0&=\lim_{n\to\infty} \frac{d}{dt}I\bigl(\varphi \bigl(t,v^{*}_{n}\bigr) \bigr)\bigg|_{t=t_{n}} \\ &=\lim_{n\to\infty}\bigl\langle I'\bigl(\varphi \bigl(t_{n},v^{*}_{n}\bigr)\bigr), -\varphi \bigl(t_{n},v^{*}_{n}\bigr)+B\bigl(\varphi\bigl(t_{n},v^{*}_{n} \bigr)\bigr)\bigr\rangle \\ &=-C_{1} \lim_{n\to\infty}\big\| \varphi\bigl(t_{n},v^{*}_{n} \bigr)-A\bigl(\varphi \bigl(t_{n},v^{*}_{n}\bigr)\bigr) \big\| ^{2}_{H^{s}_{0}(\Omega)}, \end{aligned} $$

(3.17)

which together with Lemma 3.4 implies that

$$ \begin{gathered} \lim_{n\to\infty}\bigl(\big\| \varphi \bigl(t_{n},v^{*}_{n}\bigr)\big\| _{H^{s}_{0}(\Omega )}+\big\| A\bigl(\varphi \bigl(t_{n},v^{*}_{n}\bigr)\bigr)\big\| _{H^{s}_{0}(\Omega)}\bigr) \\ \quad\leq C\lim_{n\to\infty}\bigl(1+\big\| \varphi\bigl(t_{n},v^{*}_{n} \bigr)-A\bigl(\varphi \bigl(t_{n},v^{*}_{n}\bigr)\bigr) \big\| _{H^{s}_{0}(\Omega)}\bigr) \\ \quad=C, \end{gathered} $$

then $\varphi(t_{n},v^{*}_{n})$ is bounded. Since A is compact, $A(\varphi (t_{n},v^{*}_{n}))$ is convergent, which implies that $\varphi(t_{n},v^{*}_{n})$ is convergent thanks to (3.17). So we find $u^{*}_{n}\in H^{s}_{0}(\Omega)$ such that (3.16) holds. In view of (3.14) we see that $u_{n}^{*}\in\mathcal{M}_{\epsilon }\subset\mathcal{N}$ is a sign-changing critical point of I. By (3.12), we have

$$ \begin{aligned} I\bigl(u_{n}^{*}\bigr)&\leq I \bigl(v^{*}_{n}\bigr) \\ &\leq\sup_{0\leq t\leq1,s\geq0}I \biggl(s \biggl(t\frac {u^{+}_{n}}{\|u_{n}^{+}\|_{H^{s}_{0}(\Omega)}}+(1-t) \frac{u_{n}^{-}}{ \|u_{n}^{-}\|_{H^{s}_{0}(\Omega)}} \biggr) \biggr) \\ &\leq\sup I(\Pi_{n})=I\bigl(u_{n}^{+}\bigr)+I \bigl(u_{n}^{-}\bigr) \\ &=I(u_{n})\to\zeta=\inf_{u\in\mathcal{N}}I(u). \end{aligned} $$

Thus, $\{u_{n}^{*}\}$ is a (PS) sequence in $\mathcal{N}$. Together with Lemma 2.5 and Lemma 3.3, I satisfies the (PS) condition, so we can find a critical point ū of I such that $\bar{u}\in\mathcal {M}_{\epsilon}$, and ū is a least energy sign-changing solution of (1.1).

Now we assume by contradiction that ū has three nodal domains $\Omega_{1}$, $\Omega_{2}$, $\Omega_{3}$ such that $\bar{u}>0$ in $\Omega_{1}$, $\bar{u}<0$ in $\Omega_{2}$, $\bar{u}>0$ in $\Omega_{3}$. By Proposition 3.6, we have $\bar{u}_{\chi\Omega_{1}\cup\Omega_{2}}\in\mathcal{N}$ and $I(\bar {u}_{\chi\Omega_{1}\cup\Omega_{2}})\geq\zeta$. $\Omega_{3}$ is a nodal domain of ū, hence

$$\bigl\langle I'(\bar{u}_{\chi\Omega_{3}}),\bar{u}_{\chi\Omega_{3}} \bigr\rangle = \bigl\langle I'(\bar{u}) ,\bar{u}_{\chi\Omega_{3}} \bigr\rangle =0, $$

which implies that

$$ \begin{aligned} I(\bar{u}_{\chi\Omega_{3}})={}&I(\bar{u}_{\chi\Omega_{3}})- \frac {1}{\mu}\bigl\langle I'(\bar{u}_{\chi\Omega_{3}}), \bar{u}_{\chi\Omega _{3}}\bigr\rangle \\ ={}& \biggl(\frac{1}{2}-\frac{1}{\mu}\biggr)\|\bar{u}_{\chi\Omega_{3}} \| ^{2}_{H^{s}_{0}(\Omega)}-\biggl(\frac{1}{2}-\frac{1}{\mu} \biggr)\lambda \|\bar{u}_{\chi\Omega_{3}}\|^{2}_{L^{2}(\Omega)}\\ &- \int_{\Omega}\biggl(F(x,\bar {u}_{\chi\Omega_{3}})- \frac{1}{\mu}\bar{u}_{\chi\Omega_{3}} f(x,\bar{u}_{\chi\Omega_{3}})\biggr)\,dx \\ \geq{}&\biggl(\frac{1}{2}-\frac{1}{\mu}\biggr)\|\bar{u}_{\chi\Omega_{3}} \| ^{2}_{H^{s}_{0}(\Omega)}>0. \end{aligned} $$

However, we have

$$\zeta\leq I(\bar{u}_{\chi\Omega_{1}\cup\Omega_{2}})< I(\bar{u}_{\chi \Omega_{1}\cup\Omega_{2}})+I( \bar{u}_{\chi\Omega_{3}}) \leq I(\bar{u})=\zeta. $$

This is a contradiction. □

4 Proof of Theorem 1.2

The case $\lambda\geq\lambda_{1}^{s}$: Linking type sign-changing solution for problem (1.1). Since $\lambda\geq\lambda^{s}_{1}$, we can suppose that

$$\lambda\in\bigl[\lambda^{s}_{k},\lambda^{s}_{k+1}\bigr)\quad \text{for some }k\in \mathbb{N}, $$

where $\lambda^{s}_{k}$ is the kth eigenvalue of the operator $(-\Delta )^{s}$, as defined in Section 2.

Let $N_{k}$ denote the eigenspace of $\lambda^{s}_{k}$. By Proposition 2.2, $\operatorname{dim}N_{k} < \infty$. We fix k and let

$$ \begin{gathered} Y_{k}\doteq N_{1} \oplus\cdots \oplus N_{k}, \\ Z_{k+1}\doteq N_{k+1} \oplus N_{k+2}\oplus\cdots, \end{gathered} $$

then $H_{0}^{s}(\Omega)=Y_{k}\oplus Z_{k+1}$.

Now we define a class of contractions of the Hilbert space X as follows.

Definition 4.1

Contractions; see [19]

Let X be a Hilbert space, if the map $\gamma(\cdot,\cdot)\in C([0,1]\times X,X)$ satisfies:

• (i) $\gamma(0,\cdot)=\mathit{id}$;

• (ii) $\forall t\in[0,1)$, $\gamma(t,\cdot)$ is a homeomorphism of X onto itself;

• (iii) $\gamma^{-1}(\cdot,\cdot)$ is continuous on $[0,1)\times X$;

• (iv) there exists a $x_{0}\in X$ such that $\gamma(1,x)=x_{0}$, $\forall x\in X$ and that $\gamma(t,x)\to x_{0}$ as $t\to1$ uniformly on bounded subsets of X,

then the set $\Gamma:= \{\gamma(\cdot,\cdot)\}$ is a class of contractions of the Hilbert space X.

Obviously, $\gamma(t, u) = (1-t)u\in\Gamma$.

The following concept of linking was introduced by Schechter and Tintarev [20].

Definition 4.2

Links; see [20]

Let X be a Hilbert space, a subset E of X links a closed subset F of X if $E \cap F= \varnothing$ and, for every $\gamma\in\Gamma$, there is a $t\in[0, 1]$ such that $\gamma (t, E)\cap F \neq\varnothing$.

In this section, our main tool is the following theorem.

Theorem 4.3

see Theorem B in [21]

Let X be a Hilbert space, assume that a compact subset E of X links a closed subset F which includes only sign-changing elements of X, $I' = \mathit{Id} -A$, where $A : X \to X$ is a compact operator, and I satisfies $(C)_{c}$ condition. If

$$a_{0}:=\sup_{E} I\leq b_{0}:=\inf _{F} I, $$

then there is a sign-changing critical point of I with critical value in $[b_{0}-\varepsilon, \sup_{(t,u)\in[0,1]\times E}I((1-t)u)+\varepsilon]$ for all ε small.

Lemma 4.4

Assume that ($f_{1}$)-($f_{4}$) and $\lambda_{k}^{s}\leq\lambda<\lambda _{k+1}^{s}$ hold. Then I satisfies the $(\mathit{PS})_{c}$ condition.

Proof

Let $c\in\mathbb{R}$ and let $\{u_{j}\}$ be a sequence in $H_{0}^{s}(\Omega)$ such that

$$ \begin{aligned}[c] & I(u_{j})\to c , \\ &\sup_{\varphi\in H_{0}^{s}(\Omega),\|\varphi\|_{H_{0}^{s}(\Omega )}=1}\bigl\{ \big|\bigl\langle I'(u_{j}), \varphi\bigr\rangle \big|\bigr\} \to0, \end{aligned} $$

(4.1)

as $j\to+\infty$. For any $j\in\mathbb{N}$ by (4.1) it easily follows that there exists $\kappa> 0$ such that

$$ \begin{aligned}[c] &\big|I(u_{j})\big|\leq\kappa, \\ &\bigg|\biggl\langle I'(u_{j}),\frac{u_{j}}{\|u_{j}\|_{H_{0}}}\biggr\rangle \bigg|\leq\kappa. \end{aligned} $$

(4.2)

Let us fix $\sigma\in(2, \mu)$, where $\mu> 2$ is given in assumption ($f_{3}$). By ($f_{3}$), there exist $r>0$ and $C>0$ such that

$$ F(x,t)\geq C|t|^{\mu},\quad \forall|t|\geq r. $$

(4.3)

By ($f_{2}$) and ($f_{4}$), we obtain

$$\big|f(x,t)\big|\leq2\varepsilon|t|+c_{\varepsilon}p|t|^{p-1}\quad \text{and}\quad \big|F(x,t)\big|\leq\varepsilon|t|^{2}+c_{\varepsilon} |t|^{p}. $$

Let $\varepsilon= 1$, we have

$$ \begin{aligned}[b] &\bigg| \int_{\Omega\cap\{|u_{j}|\leq r\}}\biggl(F(x,u_{j})-\frac{1}{\sigma }f(x,u_{j})u_{j} \biggr)\,dx\bigg| \\ &\quad\leq\biggl(r^{2}+c_{1}r^{p}+\frac{2}{\sigma}r+ \frac{p}{\sigma }c_{1}r^{p-1}\biggr)\Omega=:\bar{\kappa}, \end{aligned} $$

(4.4)

so that, using also (4.3) and (4.4),

$$ \begin{aligned}[b] &I(u_{j})- \frac{1}{\sigma}\bigl\langle I'(u_{j}),u_{j} \bigr\rangle \\ &\quad=\biggl(\frac{1}{2}-\frac{1}{\sigma}\biggr) \bigl(\|u_{j} \|^{2}_{H_{0}^{s}(\Omega)}-\lambda \|u_{j}\|^{2}_{L^{2}(\Omega)} \bigr) - \int_{\Omega}\biggl(F(x,u_{j})-\frac{1}{\sigma}f(x,u_{j})u_{j} \biggr)\,dx \\ &\quad\geq\biggl(\frac{1}{2}-\frac{1}{\sigma}\biggr) \bigl( \|u_{j}\|^{2}_{H_{0}^{s}(\Omega )}-\lambda\|u_{j} \|^{2}_{L^{2}(\Omega)}\bigr)+\biggl(\frac{\mu}{\sigma}-1\biggr) \int_{\Omega\cap\{|u_{j}|\geq r\}}F(x,u_{j})\,dx \\ & \qquad{}- \int_{\Omega\cap\{|u_{j}|\leq r\}} \biggl(F(x,u_{j})-\frac{1}{\sigma }f(x,u_{j})u_{j} \biggr)\,dx \\ &\quad\geq\biggl(\frac{1}{2}-\frac{1}{\sigma}\biggr) \bigl( \|u_{j}\|^{2}_{H_{0}^{s}(\Omega )}-\lambda\|u_{j} \|^{2}_{L^{2}(\Omega)}\bigr)+\biggl(\frac{\mu}{\sigma}-1\biggr) \int_{\Omega\cap\{|u_{j}|\geq r\}}C|u_{j}|^{\mu}\,dx-\bar{\kappa} \\ &\quad\geq\biggl(\frac{1}{2}-\frac{1}{\sigma}\biggr) \bigl( \|u_{j}\|^{2}_{H_{0}^{s}(\Omega )}-\lambda\|u_{j} \|^{2}_{L^{2}(\Omega)}\bigr)+\biggl(\frac{\mu}{\sigma}-1\biggr) C_{1}\|u_{j}\|_{L^{\mu}(\Omega)}^{\mu}-\bar{\kappa}. \end{aligned} $$

(4.5)

Moreover, for any $\varepsilon> 0$ the Young inequality (with conjugate exponents $\frac{\mu}{2} > 1$ and $\frac{\mu}{\mu-2}$) gives

$$ \|u_{j}\|^{2}_{L^{2}(\Omega)}\leq \frac{2\varepsilon}{\mu}\|u_{j}\|^{\mu }_{L^{\mu}(\Omega)}+ \frac{\mu-2}{\mu}\varepsilon^{-\frac{2}{\mu -2}}|\Omega|. $$

(4.6)

Hence, by (4.5) and (4.6) we deduce that

$$ \begin{aligned}[b] &I(u_{j})- \frac{1}{\sigma}\bigl\langle I'(u_{j}),u_{j} \bigr\rangle \\ &\quad\geq\biggl(\frac{1}{2}-\frac{1}{\sigma}\biggr)\|u_{j} \|^{2}_{H_{0}^{s}(\Omega )}+\biggl[\biggl(\frac{\mu}{\sigma}-1 \biggr)C_{1} -\lambda\biggl(\frac{1}{2}-\frac{1}{\sigma}\biggr) \frac{2\varepsilon}{\mu}\biggr]\| u_{j}\|_{L^{\mu}(\Omega)}^{\mu}-C_{\varepsilon}, \end{aligned} $$

(4.7)

where $C_{\varepsilon}=\lambda(\frac{1}{2}-\frac{1}{\sigma})\frac {\mu-2}{\mu}\varepsilon^{-\frac{2}{\mu-2}}|\Omega| +\bar{\kappa}$ is a constant such that $C_{\varepsilon}\to+\infty$ as $\varepsilon\to0$, where $\mu>\sigma>2$.

Now, choosing ε so small that

$$\biggl[ \biggl(\frac{\mu}{\sigma}-1 \biggr)C_{1} -\lambda \biggl( \frac{1}{2}-\frac{1}{\sigma} \biggr)\frac{2\varepsilon}{\mu} \biggr]>0, $$

by (4.7) we get

$$ I(u_{j})-\frac{1}{\sigma} \bigl\langle I'(u_{j}),u_{j} \bigr\rangle \geq \biggl( \frac {1}{2}-\frac{1}{\sigma} \biggr)\|u_{j} \|^{2}_{H_{0}^{s}(\Omega)}-C_{\varepsilon}. $$

(4.8)

As a consequence of (4.1), (4.2) and (4.8) we obtain

$$\kappa\bigl(1+\|u_{j}\|_{H_{0}^{s}(\Omega)}\bigr)\geq I(u_{j})- \frac{1}{\sigma } \bigl\langle I'(u_{j}),u_{j} \bigr\rangle \geq \biggl(\frac{1}{2}-\frac{1}{\sigma} \biggr)\| u_{j}\|^{2}_{H_{0}^{s}(\Omega)}-C_{\varepsilon}. $$

Hence, $\{u_{j}\}$ is bounded in $H_{0}^{s}(\Omega)$. By Lemma 2.5, then there exists $u_{0}\in H_{0}^{s}(\Omega)$ such that, up to a subsequence, $\| u_{j}-u_{0}\|_{H_{0}^{s}(\Omega)}\to0$ as $j\to+\infty$. So the assertion of Lemma 4.4 is proved. □

Lemma 4.5

Assume that ($f_{1}$)-($f_{4}$) and $\lambda_{k}^{s}\leq\lambda<\lambda _{k+1}^{s}$ hold. Then $I(u)\leq0$, $\forall u\in Y_{k}$.

Proof

Let $u\in \operatorname{span}\{e_{1}, \ldots, e_{k}\}$, then $u(x)=\sum_{i=1}^{k}a_{i}e_{i}(x)$, with some $a_{i}\in\mathbb {R}$, $i=1,\ldots,k$. Since $\{e_{1}, \ldots, e_{k}, \ldots\}$ is an orthonormal basis of $L^{2}(\Omega)$ and an orthogonal one of $H_{0}^{s}(\Omega)$ by Proposition 2.2(b), we get

$$ \int_{\Omega}\big|u(x)\big|^{2}\,dx=\sum_{i=1}^{k}a_{i}^{2} $$

(4.9)

and

$$ \begin{aligned}[b] \int_{\Omega\times\Omega}\frac{|u(x)-u(y)|^{2}}{|x-y|^{n+2s}}\,dx\,dy&= \sum_{i=1}^{k} \int_{\Omega\times\Omega}\frac {|a_{i}e_{i}(x)-a_{i}e_{i}(y)|^{2}}{|x-y|^{n+2s}}\,dx\,dy\\&=\sum_{i=1}^{k}a_{i}^{2} \|e_{i}\| ^{2}_{H_{0}^{s}(\Omega)} . \end{aligned}$$

(4.10)

Moreover, we obtain from ($f_{3}$)

$$ F(x,t)\geq0,\quad \forall x\in\Omega, t\in\mathbb{R}. $$

(4.11)

By Definition 2.1 and $e_{i}$ being the eigenfunctions of $(-\Delta)$, we have

$$\|e_{i}\|^{2}_{H_{0}^{s}(\Omega)}=(e_{i},e_{i})_{H_{0}^{s}(\Omega)}= \int_{\Omega }e_{i}(x) (-\Delta)^{s}e_{i}(x)\,dx = \int_{\Omega}\lambda^{s}_{i}e_{i}^{2}(x)\,dx= \lambda^{s}_{i}. $$

Then, by (4.9)-(4.11), we get

$$ \begin{aligned} I(u)&=\frac{1}{2}\sum_{i=1}^{k}a_{i}^{2} \bigl(\|e_{i}\|^{2}_{H_{0}^{s}(\Omega )}-\lambda \bigr)- \int_{\Omega}F \bigl(x,u(x) \bigr)\,dx \\ &\leq\frac{1}{2}\sum_{i=1}^{k}a_{i}^{2} \bigl(\lambda^{s}_{i}-\lambda \bigr)\leq0, \end{aligned} $$

thanks to the fact that $\lambda^{s}_{i}\leq\lambda^{s}_{k}\leq\lambda$ for any $i = 1,\ldots, k$. □

Lemma 4.6

Assume that ($f_{1}$)-($f_{4}$) and $\lambda_{k}^{s}\leq\lambda <\lambda^{s}_{k+1}$ hold. Then there exist $\rho> 0$, $\beta> 0$ such that

$$I(u)\geq\beta \quad\textit{for } \|u\|_{H_{0}^{s}(\Omega)}=\rho, u\in H_{0}^{s}( \Omega). $$

Proof

By $u\in Z_{k+1}$, we have $u(x)=\sum _{i=k+1}^{+\infty}a_{i}e_{i}(x)$, with some $a_{i}\in\mathbb{R}$, $i=k+1, k+2,\ldots$ . By $\lambda^{s}_{k+1}\leq\lambda^{s}_{k+2}\leq \cdots$, we have

$$ \begin{aligned} \lambda^{s}_{k+1}\|u\|^{2}_{L^{2}(\Omega)}&= \lambda^{s}_{k+1}\sum _{i=k+1}^{+\infty}a_{i}^{2} \\ &\leq\sum_{i=k+1}^{+\infty}\lambda_{i}^{s}a_{i}^{2}= \sum _{i=k+1}^{+\infty}a_{i}^{2} \|e_{i}\|^{2}_{H^{s}_{0}(\Omega)} \\ &=\|u\|^{2}_{H^{s}_{0}(\Omega)}. \end{aligned} $$

Hence, by $\lambda^{s}_{k}\leq\lambda<\lambda^{s}_{k+1}$, we obtain

$$ \int_{\Omega\times\Omega}\frac {|u(x)-u(y)|^{2}}{|x-y|^{n+2s}}\,dx\,dy-\lambda \int_{\Omega}\big|u(x)\big|^{2}\,dx\geq \biggl(1-\frac{\lambda}{\lambda_{k+1}} \biggr)\|u\|^{2}_{H_{0}^{s}(\Omega)} . $$

(4.12)

By ($f_{2}$), ($f_{4}$), for any $\varepsilon>0$, there exists a $C_{\varepsilon}$ such that

$$\big|F(x,t)\big|\leq\varepsilon|t|^{2}+C_{\varepsilon}|t|^{p}, $$

and then

$$ \begin{aligned}[b] I(u)\geq{}&\frac{1}{2} \int_{\Omega\times\Omega}\frac {|u(x)-u(y)|^{2}}{|x-y|^{n+2s}}\,dx\,dy -\frac{\lambda}{2} \int_{\Omega}\big|u(x)\big|^{2}\,dx\\ &-\varepsilon \int_{\Omega }\big|u(x)\big|^{2}\,dx-C_{\varepsilon} \int_{\Omega}\big|u(x)\big|^{p}\,dx \\ \geq{}&\frac {1}{2} \biggl(1-\frac{\lambda}{\lambda_{k+1}} \biggr) \int_{\Omega\times\Omega }\frac{|u(x)-u(y)|^{2}}{|x-y|^{n+2s}}\,dx\,dy\\& -\varepsilon| \Omega|^{(2^{*}_{s}-2)/2^{*}_{s}}\|u\|^{2}_{L^{2^{*}_{s}}(\Omega )}-|\Omega|^{(2^{*}_{s}-p)/2^{*}_{s}}C_{\varepsilon} \|u\|^{p}_{L^{2^{*}_{s}}(\Omega)}, \end{aligned} $$

(4.13)

thanks to the fact that $L^{2^{*}_{s}}(\Omega)\hookrightarrow L^{2}(\Omega )$ and $L^{2^{*}_{s}}(\Omega)\hookrightarrow L^{p}(\Omega)$ continuously (Ω being bounded and max$\{2, p\} = p <2^{*}_{s}$). By Proposition 2.3, there exists a C such that

$$\|u\|^{2}_{H_{0}^{s}(\Omega)}\geq C^{-1}\|u\|^{2}_{L^{2^{*}_{s}}(\Omega)} $$

and then

$$ \begin{aligned}[b] I(u)\geq{}&\frac{1}{2} \biggl(1- \frac{\lambda}{\lambda_{k+1}} \biggr)\|u\| ^{2}_{H_{0}^{s}(\Omega)}\\ & -\varepsilon| \Omega|^{(2^{*}_{s}-2)/2^{*}_{s}}C\|u \|^{2}_{H_{0}^{s}(\Omega )}-|\Omega|^{(2^{*}_{s}-p)/2^{*}_{s}}C_{\varepsilon} C^{p/2}\|u\|^{p}_{H_{0}^{s}(\Omega)}. \end{aligned} $$

(4.14)

Choosing $\varepsilon> 0$ such that $\frac{1}{2}(1-\frac{\lambda }{\lambda_{k+1}})-\varepsilon|\Omega|^{(2^{*}_{s}-2)/2^{*}_{s}}C>0$, by (4.14), we can choose $\rho> 0$ sufficiently small and $\beta> 0$ such that $I(u)\geq\beta$ for $\|u\|_{H_{0}^{s}(\Omega)}=\rho$, $u\in Z_{k+1}$. □

Lemma 4.7

Fix a $z\in Z_{k+1}$ be such that $\|z\|=\rho$, the space $Y_{k}\oplus\mathbb{R}z$ is a finite dimensional subspace of $H_{0}^{s}(\Omega)$, then there exists $R > \rho$ such that $I(u)\leq0$ for any $u\in Y_{k}\oplus\mathbb{R}z$ with $\|u\|_{H_{0}^{s}(\Omega)} \geq R$, where ρ is given in Lemma 4.6.

Proof

By ($f_{3}$), there exist $C>0$ and $C_{1}>0$ such that

$$ F(x,t)\geq Ct^{\mu}-C_{1}, \quad\forall t\in \mathbb{R}. $$

(4.15)

Let $u\in Y_{k}\oplus\mathbb{R}z$. Then the non-negativity of λ and (4.15) give

$$ \begin{aligned}[b] I(u)&\leq\frac{1}{2}\|u \|^{2}_{H_{0}^{s}(\Omega)}- \frac{\lambda}{2}\|u\| ^{2}_{L^{2}(\Omega)}- \int_{\Omega} \bigl(C\big|u(x)\big|^{\mu}-C_{1} \bigr)\,dx \\ &\leq\frac{1}{2}\|u\|^{2}_{H_{0}^{s}(\Omega)}-C_{2}\|u \|^{\mu }_{H_{0}^{s}(\Omega)}+C_{1}|\Omega| \end{aligned} $$

(4.16)

for some positive constant $C_{2}$, thanks to the fact that in any finite dimensional space all the norms are equivalent.

Hence, if $\|u\|_{H_{0}^{s}(\Omega)}\to+\infty$, then $I(u)\to-\infty$, since $\mu> 2$ by assumption, and so the assertion of Lemma 4.7 follows. □

Proof of Theorem 1.2

By Lemma 4.7, $z\in Z_{k+1}\setminus\{0\}$, $\|z\|_{H_{0}^{s}(\Omega )}=\rho$ and $0 <\rho<R$, define the sets E and F as follows:

$$\begin{gathered} E:=\bigl\{ u = v + sz: v\in Y_{k}, s\geq0, \|u\|_{H_{0}^{s}(\Omega)}= R\bigr\} \cup (Y_{k}\cap\bar{B}_{R}),\\ F:= \bigl\{ u\in Z_{k+1}:\|u \|_{H_{0}^{s}(\Omega)}=\rho\bigr\} , \end{gathered}$$

then E and F link each other (see [22, 23]) in the sense of Definition 4.2. It is easy to see that $E \subset H_{0}^{s}(\Omega)$ is a compact set, thanks to the fact that $Y_{k}\oplus\mathbb{R}z$ is a finite dimensional space. By Proposition 2.2(c), the closed set F includes only sign-changing elements of $H_{0}^{s}(\Omega)$.

Combining with Lemma 4.5-4.7, we have

$$\inf_{F} I(u)\geq\beta>0\geq\max_{E} I(u), $$

then there is a sign-changing critical point of I by using Theorem 4.3. □

5 Conclusions

Sign-changing solutions of nonlinear elliptic equations have attracted much attention in recent years. One reason is that sign-changing solutions arise naturally from mathematical models in biology and physics. Another reason is that there are richer structures of sign-changing solutions than that of negative and positive solutions for generic nonlinear and linear elliptic equations.

In this paper, we study the sign-changing solution for the nonlinear equation involving the fractional Laplacian. This type of operators arises in several areas such as anomalous diffusion, the thin obstacle problem, optimization, finance, phase transitions, stratified materials, crystal dislocation, soft thin films, semipermeable membranes, flame propagation, conservation laws, quasi-geostrophic flows, multiple scattering and materials science. Our problem (1.1) has different variational structures when λ takes different values. When $\lambda\leq0$, a positive, a negative and a sign-changing solution have been found by constructing different invariant sets on which the functional is bounded below. When $\lambda >\lambda_{1}^{s}$, by using a variation of the linking theorem, we also obtain a sign-changing solution. Our results are new, and the work established in this article is of quite a general nature and covers a variety of special cases associated with particular values of the parameters involved in the problem.