Existence of ground state for fractional Kirchhoff equation with $L^{2}$ critical exponents

Abstract

In this paper, we consider a class of fractional Kirchhoff equations with $L^{2}$ critical exponents. By using the scaling technique and concentration-compactness principle we obtain the existence and nonexistence of ground state for fractional Kirchhoff equation with $L^{2}$ critical exponent.

1 Introduction

In this paper, we consider the existence of ground state for the following fractional Kirchhoff equation:

$$ - \biggl(a+b \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr) (- \Delta )^{s}u+V(x) \vert u \vert ^{\gamma }u= \vert u \vert ^{\frac{8s}{N}}u+\mu u\quad \text{in } \mathbb{R}^{N}, $$

(1)

where $a,b>0$, $N>2s>\frac{N}{2}$ with $s\in (0,1)$, $0\leq \gamma \leq \frac{8s}{N}$, $2^{*}(s)=\frac{2N}{N-2s}$, and $V(x)$ is a bounded function in $\mathbb{R}^{N}$.

If $s=1$, then equation (1) is related to the stationary solutions of

$$ u_{tt}- \biggl(a+b \int _{\mathbb{R}^{N}} \vert \nabla u \vert ^{2}\,dx \biggr) \Delta u=f(x,u), $$

(2)

where $f(x,u)$ is a general nonlinear function. Equation (2) comes from free vibrations of elastic strings by taking into account the changes in length of the string produced by transverse vibrations [13]. After the pioneering works [17] and [15], equation (1) has attracted considerable attention. The existence and asymptotic behavior of nodal solutions of equation (1) were considered by Deng, Peng, and Shuai [5]. The existence and concentration behavior of positive solutions were studied in [8, 9]. The uniqueness and nondegeneracy of positive solutions were obtained by Li et al. [14] and the references therein. The existence of multipeak solutions was considered in [23].

Equation (1) can be viewed as an eigenvalue problem by taking μ as an unknown Lagrange multiplier. Hence some mathematicians considered equation (1) by studying some constrained variational problems and obtained the existence of ground state of equation (1). This technique was generally used for other types of equations, for example, semilinear Schrödinger equation [11, 24], Schrödinger–Poisson equation [4, 12], quasilinear Schrödinger equation [29, 30]; see also [1, 2, 18, 21, 22]. For $s=1$, as far as we know, the first work comes from Ye [25], who considered the following minimization problem:

$$ I_{c^{2}}:=\inf_{u\in S_{c}}I(u), $$

(3)

where

$$ I(u)=\frac{a}{2} \int _{\mathbb{R}^{N}} \vert \nabla u \vert ^{2}\,dx+ \frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \vert \nabla u \vert ^{2}\,dx \biggr)^{2}-\frac{1}{p} \int _{ \mathbb{R}^{N}} \vert u \vert ^{p}\,dx $$

and

$$ S_{c}:= \biggl\{ u\in H^{1}\bigl(\mathbb{R}^{N} \bigr): \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx=c^{2} \biggr\} . $$

Using the scaling technique and concentration-compactness principle, Ye obtained the sharp existence of global constraint minimizers of problem (3). Then Zeng and Zhang [28] improved the results of [25] and obtained the sharp existence and uniqueness of global constraint minimizers of problem (3). From [25, 28] we know that there is an $L^{2}$ critical exponent $p^{*}=2+\frac{8}{N}$ such that problem (3) has global constraint minimizers for $p< p^{*}$ and no global constraint minimizers for $p\geq p^{*}$. Then, for the $L^{2}$ critical exponent, Ye [26] and Zeng and Chen [31] added a perturbation function and obtained the existence of minimizers on $S_{c}$. Moreover, for the $L^{2}$ critical exponent, Ye [27] gave some mass concentration behavior. Recently, Guo, Zhang, and Zhou [7] considered the following minimization problem:

$$ d_{\beta }(p):=\inf_{u\in S_{1}}E_{p}^{\beta }(u), $$

(4)

where

$$ E_{p}^{\beta }=\frac{a}{2} \int _{\mathbb{R}^{N}} \vert \nabla u \vert ^{2}\,dx+ \frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \vert \nabla u \vert ^{2}\,dx \biggr)^{2}+ \frac{1}{2} \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{2}\,dx- \frac{\beta }{p} \int _{ \mathbb{R}^{N}} \vert u \vert ^{p}\,dx, $$

and $S_{1}:= \{u\in H(\mathbb{R}^{N}): \int _{\mathbb{R}^{N}} |u|^{2}\,dx=1 \}$ with $H=\{u\in H^{1}(\mathbb{R}^{N}): \int _{\mathbb{R}^{N}}V(x)|u|^{2}\,dx< \infty \}$. They first proved the sharp existence and nonexistence of global minimizer of problem (4) with $V(x)=0$. Then, for the trapping potential $V(x)$, they considered the existence of minimizers for problem (4). Especially, for the $L^{2}$ critical exponent, they proved that there is $\beta _{p^{*}}>0$ such that problem (4) has at least one minimizer for $\beta \leq \beta _{p^{*}}$ and has no minimizers for $\beta >\beta _{p^{*}}$. Furthermore, for minimizers of problem (4) with $p< p^{*}$ and $\beta >\beta _{p^{*}}$, they obtained the blowup behavior of minimizers as p tends to $p^{*}$.

For $s\in (0,1)$, Autuori, Fiscella, and Pucci [3] obtained the existence of solutions for equation (1) with critical nonlinearity. The existence of solutions of (1) with critical exponents was also considered in [19]. The multiplicity of solutions was obtained by Pucci, Xiang, and Zhang [20] and so on. Recently, Huang and Zhang [10] considered the existence and uniqueness of minimizers for the following problem:

$$ e(c):=\inf_{u\in S_{c}}E_{p}(u), $$

(5)

where

$$ E_{p}(u)=\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx+ \frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}-\frac{1}{p+2} \int _{\mathbb{R}^{N}} \vert u \vert ^{p+2}\,dx, $$

and $S_{c}:= \{u\in H^{s}(\mathbb{R}^{N}): \int _{\mathbb{R}^{N}} |u|^{2}\,dx=c^{2} \}$. Using the scaling technique and some energy estimates, they obtained the existence and uniqueness of minimizers for problem (5) if $p<\frac{8s}{N}$ and proved that there are no minimizers for problem (5) when $p\geq \frac{8s}{N}$.

For the existence of ground state of equation (1), we consider the following minimization problem:

$$ e(c):=\inf_{u\in S_{c}}I_{p}(u), $$

(6)

where

$$ \begin{aligned} I_{p}(u)&=\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx+ \frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}\\ &\quad {}+\frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{ \gamma +2}\,dx- \frac{1}{p+2} \int _{\mathbb{R}^{N}} \vert u \vert ^{p+2}\,dx, \end{aligned} $$

and

$$ S_{c}:= \biggl\{ u\in H^{s}\bigl(\mathbb{R}^{N} \bigr): \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx=c^{2} \biggr\} . $$

Here $H^{s}(\mathbb{R}^{N})$ is the Besov space defined by

$$ H^{s}= \biggl\{ u\in L^{2}\bigl(\mathbb{R}^{N} \bigr): \frac{u(x)-u(y)}{ \vert x-y \vert ^{\frac{N+2s}{2}}}\in L^{2}\bigl(\mathbb{R}^{N} \times \mathbb{R}^{N}\bigr) \biggr\} $$

with the norm

$$ \|u\|_{H^{s}(\mathbb{R}^{N})}= \biggl( \int _{\mathbb{R}^{N}} \bigl( \bigl\vert (- \Delta )^{\frac{s}{2}}u \bigr\vert ^{2}+ \vert u \vert ^{2} \bigr)\,dx \biggr)^{\frac{1}{2}}, $$

where

$$ \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx= \int \int _{ \mathbb{R}^{2N}}\frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy. $$

It is easy to see that there are no minimizers for problem (6) if $p>\frac{8s}{N}$. Indeed, for any $u\in S_{c}$ and constant $\lambda >0$, let $u_{\lambda }(x)=\lambda ^{\frac{N}{2}}u(\lambda x)$. Then

$$\begin{aligned}& \int _{\mathbb{R}^{N}}u_{\lambda }^{2}(x)\,dx= \int _{\mathbb{R}^{N}}u^{2}(x)\,dx=c^{2}, \\& \begin{aligned} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert u_{\lambda }^{2}(x)-u_{\lambda }^{2}(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy&= \lambda ^{2s} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert u^{2}(x)-u^{2}(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy\\ &=\lambda ^{2s} \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx, \end{aligned} \\& \int _{\mathbb{R}^{N}} \bigl\vert u_{\lambda }(x) \bigr\vert ^{2+p}\,dx=\lambda ^{\frac{Np}{2}} \int _{\mathbb{R}^{N}} \bigl\vert u(x) \bigr\vert ^{2+p}\,dx, \\& \int _{\mathbb{R}^{N}}V(x) \bigl\vert u_{\lambda }(x) \bigr\vert ^{2+\gamma }\,dx=\lambda ^{ \frac{N\gamma }{2}} \int _{\mathbb{R}^{N}}V \biggl(\frac{x}{\lambda } \biggr) \bigl\vert u(x) \bigr\vert ^{2+\gamma }\,dx. \end{aligned}$$

Hence we can deduce that

$$\begin{aligned} I_{p}(u_{\lambda })&=\frac{a}{2}\lambda ^{2s} \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx+\frac{b}{4}\lambda ^{4s} \biggl( \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad{}+\frac{1}{2+\gamma }\lambda ^{\frac{N\gamma }{2}} \int _{\mathbb{R}^{N}}V \biggl(\frac{x}{\lambda } \biggr) \bigl\vert u(x) \bigr\vert ^{2+\gamma }\,dx- \frac{1}{p+2} \lambda ^{\frac{Np}{2}} \int _{\mathbb{R}^{N}} \bigl\vert u(x) \bigr\vert ^{2+p}\,dx. \end{aligned}$$

(7)

Since $\gamma <\frac{8s}{N}$, it is easy to see that $\frac{N\gamma }{2}<4s$. If $p>\frac{8s}{N}$, then for λ large enough, the dominant term in (7) is $\frac{1}{p+2}\lambda ^{\frac{Np}{2}}\int _{\mathbb{R}^{N}}|u(x)|^{2+p}\,dx$. Then $I_{p}(u_{\lambda })\rightarrow -\infty$ as $\lambda \rightarrow \infty$. This means that there are no minimizers for problem (6) if $p>\frac{8s}{N}$. Therefore it seems that $p=\frac{8s}{N}$ is the $L^{2}$ critical exponent for problem (6). Moreover, from (7) with $V(x)=0$ we have $I_{p}(u_{\lambda })\rightarrow 0$ as $\lambda \rightarrow 0$. Hence $e(c)\leq 0$ for any $c>0$, and $0< p<2^{*}(s)-2$. For $p=\frac{8s}{N}$, similarly to the proof of [10, 28], using the Gagliardo–Nirenberg inequality (12), we have

$$\begin{aligned} I_{p}(u)&=\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx+ \frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}-\frac{N}{2N+8s} \int _{\mathbb{R}^{N}} \vert u \vert ^{2+\frac{8s}{N}}\,dx \\ &\geq \frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx+ \frac{b}{4} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{\frac{8s-2N}{N}} \biggr) \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}, \end{aligned}$$

(8)

where the definition of $c^{*}$ is given further. If $c\leq c^{*}$, then (8) means that $e(c)>0$, a contradiction to $e(c)\leq 0$, which indicates that for $p= \frac{8s}{N}$ and $c\leq c^{*}$, problem (6) with $V(x)=0$ has no minimizers. If $c>c^{*}$, then in view of Lemma 2.3, let $u_{\lambda }(x)=\frac{c\lambda ^{\frac{N}{2}}}{|U|_{2}}U(\lambda x)$. Then we have $e(c)\leq -\infty$, which means that for $p= \frac{8s}{N}$ and $c> c^{*}$, there are no minimizers for problem (6) with $V(x)=0$. In other words, for $V(x)=0$, there is no minimizer for problem (6) with $p=\frac{8s}{N}$. Hence, in this paper, when the potential function $V(x)$ satisfies some conditions, we consider the existence and nonexistence of minimizers for problem (6) with $p=\frac{8s}{N}$. In addition, we consider the existence and nonexistence of ground states for equation (1) under some conditions on the function $V(x)$. Moreover, in this paper, the energy estimate method used in [10, 28] is invalid because of the existence of a potential function $V(x)$. Hence we use the concentration-compactness principle to overcome the compactness of a minimizing sequence. Using this technique, it is natural that $\gamma \geq 2$ is necessary by Lemma 2.6.

In this paper, we assume that

$$ V(x)\in L^{\infty }\bigl(\mathbb{R}^{N}\bigr). $$

(9)

Let

$$ c^{*}= \bigl(b \bigl\vert U(x) \bigr\vert _{2}^{\frac{8s}{N}} \bigr)^{\frac{N}{8s-2N}}, $$

where the function $U(x)$ is defined in Sect. 2. We first give a nonexistence result.

Theorem 1.1

Let$p=\frac{8s}{N}$, and let$V(x)$satisfy (9). Then problem (6) has no minimizers if one of the following conditions holds:

• (1) $c>c^{*}$for any$\gamma \in [0, \frac{8s}{N})$.

• (2) $V(x)\geq 0$for any$c\in (0,c^{*})$and$\gamma \in [0, \frac{8s}{N})$.

• (3) For$\gamma \in (\frac{4s}{N},\frac{8s}{N})$and$|V|_{\infty }c^{\frac{8s-\gamma N+\gamma (4s-N)}{4s}}$small enough, we have

  $$ \begin{aligned} &\frac{ \vert V \vert _{\infty }}{\gamma +2} \biggl( \frac{N+4s}{2N \vert U \vert _{2}^{\frac{8s}{N}}} \biggr)^{\frac{\gamma N}{8s}}c^{ \frac{8s-\gamma N+\gamma (4s-N)}{4s}} \\ &\quad \leq \biggl( \frac{2as}{8s-\gamma N} \biggr)^{\frac{8s-\gamma N}{4s}} \biggl( \frac{bs}{\gamma N-4s} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr) \biggr)^{\frac{\gamma N-4s}{4s}}. \end{aligned} $$

From (2) of Theorem 1.1 we know that problem (6) has minimizers if and only if the function $V(x)$ has a negative part. Hence, in this paper, we first give a certain condition for $V(x)$ at infinity and get the following existence result.

Theorem 1.2

Let$p=\frac{8s}{N}$, $c\in (0,c^{*})$, $\gamma \in [2,\frac{8s}{N})$, $\frac{N\gamma }{2}+\alpha <4s$for some$\alpha >0$, and letabe small enough. Suppose that the function$V(x)$satisfies (9) and

$$ V(x)\sim - \vert x \vert ^{-\beta }\quad \textit{as } \vert x \vert \rightarrow \infty . $$

(10)

Then problem (6) has at least a minimizer.

According Theorem 1.2, we get the existence of minimizers of problem (6) for $V(x)$ tending to 0 at infinity with some rates as $|x|\rightarrow \infty$. Next, if we assume a general condition for $V(x)$ at infinity, then we have the following:

Theorem 1.3

Let$p=\frac{8s}{N}$, $c\in (0,c^{*})$, and$\gamma \in [2,\frac{8s}{N})$, and suppose that the function$V(x)$satisfies (9) and

$$ \lim_{|x|\rightarrow \infty }V(x)=0. $$

(11)

Then if$e(c)<0$, the problem (6) has at least one minimizer.

Throughout the paper, C denotes some constant, and $|u|_{p}$ denotes the $L^{p}$-norm of a function u.

2 Preliminary results

Since we want to consider the existence of minimizers for problem (6) with $p=\frac{8s}{N}$, we first introduce the following Gagliardo–Nirenber inequality [6]:

$$ \int _{\mathbb{R}^{N}} \vert u \vert ^{2+\frac{8s}{N}}\,dx\leq \frac{N+4s}{2N \vert U(x) \vert _{2}^{\frac{8s}{N}}} \biggl( \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx \biggr)^{\frac{4s-N}{N}} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}. $$

(12)

Here the function $U(x)$ is the unique ground state of the equation

$$ (-\Delta )^{s}u+\frac{4s-N}{2N}u= \vert u \vert ^{\frac{8s}{N}}u,\quad x\in \mathbb{R}^{N}. $$

(13)

Using the Pohozaev identity and equation (13) [6, 10], we can get that

$$ \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx= \int _{ \mathbb{R}^{N}} \vert u \vert ^{2}\,dx= \frac{2N}{N+4s} \int _{\mathbb{R}^{N}} \vert u \vert ^{2+ \frac{8s}{N}}\,dx. $$

(14)

Lemma 2.1

Assume that$V(x)\geq 0$. Then, for any$c\in (0,c^{*})$, we have$e(c)\geq 0$.

Proof

For any $u\in S_{c}$, using the Gagliardo–Nirenberg inequality (12), we get that

$$\begin{aligned} I_{p}(u)&=\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx+ \frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}+\frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{ \gamma +2}\,dx \\ &\quad {}-\frac{N}{2N+8s} \int _{\mathbb{R}^{N}} \vert u \vert ^{2+\frac{8s}{N}}\,dx \\ &\geq \frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}- \frac{c^{\frac{8s-2N}{N}}}{4 \vert U(x) \vert _{2}^{\frac{8s}{N}}} \biggl( \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad {}+ \frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{\gamma +2}\,dx \\ &\geq \frac{b}{4} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr) \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}+\frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{ \gamma +2}\,dx, \end{aligned}$$

(15)

which, together with $V(x)\geq 0$, implies that $I_{p}(u)>0$. Hence we have

$$ e(c)\geq 0. $$

 □

Lemma 2.2

Let$\gamma \in (\frac{4s}{N},\frac{8s}{N})$, and let$|V|_{\infty }c^{\frac{8s-\gamma N+\gamma (4s-N)}{4s}}$be small enough such that

$$ \begin{aligned} &\frac{ \vert V \vert _{\infty }}{\gamma +2} \biggl( \frac{N+4s}{2N \vert U \vert _{2}^{\frac{8s}{N}}} \biggr)^{\frac{\gamma N}{8s}}c^{ \frac{8s-\gamma N+\gamma (4s-N)}{4s}} \\ &\quad \leq \biggl( \frac{2as}{8s-\gamma N} \biggr)^{\frac{8s-\gamma N}{4s}} \biggl( \frac{bs}{\gamma N-4s} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr) \biggr)^{\frac{\gamma N-4s}{4s}}. \end{aligned} $$

Then$e(c)\geq 0$.

Proof

For any $u\in S_{c}$, using the Hölder and Gagliardo–Nirenberg inequalities, we have

$$\begin{aligned} \int _{\mathbb{R}^{N}} \vert u \vert ^{\gamma +2}\,dx&\leq \biggl( \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx \biggr)^{\frac{8s-\gamma N}{8s}} \biggl( \int _{\mathbb{R}^{N}} \vert u \vert ^{2+ \frac{8s}{N}}\,dx \biggr)^{\frac{\gamma N}{8s}} \\ &\leq \biggl(\frac{N+4s}{2N \vert U \vert _{2}^{\frac{8s}{N}}} \biggr)^{ \frac{\gamma N}{8s}}c^{\frac{8s-\gamma N+\gamma (4s-N)}{4s}} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{ \frac{\gamma N}{4s}}, \end{aligned}$$

which, combined with (15), indicates that

$$\begin{aligned} I_{p}(u)&=\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx+ \frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}+\frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{ \gamma +2}\,dx \\ &\quad {}-\frac{N}{2N+8s} \int _{\mathbb{R}^{N}} \vert u \vert ^{2+\frac{8s}{N}}\,dx \\ &\geq \frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx+ \frac{b}{4} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{\frac{8s-2N}{N}} \biggr) \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad {}-\frac{ \vert V \vert _{\infty }}{\gamma +2} \biggl( \frac{N+4s}{2N \vert U \vert _{2}^{\frac{8s}{N}}} \biggr)^{\frac{\gamma N}{8s}}c^{ \frac{8s-\gamma N+\gamma (4s-N)}{4s}} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{\frac{\gamma N}{4s}}. \end{aligned}$$

(16)

Let $\delta =\frac{8s-\gamma N}{4s}$ and $\beta =1-\delta =\frac{\gamma N-4s}{4s}$. Using the Young inequality, we have

$$\begin{aligned} &\frac{a}{2}t+\frac{b}{4} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr)t^{2}\\ &\quad \geq \biggl(\frac{a}{2\delta } \biggr)^{\delta } \biggl( \frac{b (1- (\frac{c}{c^{*}} )^{\frac{8s-2N}{N}} )}{4\beta } \biggr)^{\beta }t^{\delta +2\beta } \\ &\quad = \biggl(\frac{2as}{8s-\gamma N} \biggr)^{\frac{8s-\gamma N}{4s}} \biggl(\frac{bs}{\gamma N-4s} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr) \biggr)^{\frac{\gamma N-4s}{4s}} t^{ \frac{\gamma N}{4s}}. \end{aligned}$$

Thus from (16) it follows that

$$\begin{aligned} I_{p}(u)&\geq \biggl[ \biggl(\frac{2as}{8s-\gamma N} \biggr)^{ \frac{8s-\gamma N}{4s}} \biggl(\frac{bs}{\gamma N-4s} \biggl(1- \biggl( \frac{c}{c^{*}} \biggr)^{\frac{8s-2N}{N}} \biggr) \biggr)^{ \frac{\gamma N-4s}{4s}} \\ &\quad{}-\frac{ \vert V \vert _{\infty }}{\gamma +2} \biggl( \frac{N+4s}{2N \vert U \vert _{2}^{\frac{8s}{N}}} \biggr)^{\frac{\gamma N}{8s}}c^{ \frac{8s-\gamma N+\gamma (4s-N)}{4s}} \biggr] \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{\frac{\gamma N}{4s}}. \end{aligned}$$

(17)

If we choose $|V|_{\infty }c^{\frac{8s-\gamma N+\gamma (4s-N)}{4s}}$ small enough such that

$$ \begin{aligned} &\frac{ \vert V \vert _{\infty }}{\gamma +2} \biggl( \frac{N+4s}{2N \vert U \vert _{2}^{\frac{8s}{N}}} \biggr)^{\frac{\gamma N}{8s}}c^{ \frac{8s-\gamma N+\gamma (4s-N)}{4s}} \\ &\quad \leq \biggl( \frac{2as}{8s-\gamma N} \biggr)^{\frac{8s-\gamma N}{4s}} \biggl( \frac{bs}{\gamma N-4s} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr) \biggr)^{\frac{\gamma N-4s}{4s}}, \end{aligned} $$

then (17) indicates that

$$ e(c)\geq 0. $$

 □

Lemma 2.3

If$c>c^{*}$, then$e(c)<-\infty$.

Proof

Set

$$ u_{\lambda }(x)=\frac{c\lambda ^{\frac{N}{2}}}{ \vert U \vert _{2}}U(\lambda x). $$

Then using (14), we have

$$\begin{aligned}& \int _{\mathbb{R}^{N}}u_{\lambda }^{2}(x) \,dx=c^{2}, \\& \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert u_{\lambda }^{2}(x)-u_{\lambda }^{2}(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy= \frac{c^{2}\lambda ^{2s}}{ \vert U \vert _{2}^{2}} \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} U \bigr\vert ^{2}\,dx=c^{2}\lambda ^{2s}, \\& \int _{\mathbb{R}^{N}} \bigl\vert u_{\lambda }(x) \bigr\vert ^{2+\frac{8s}{N}}\,dx= \frac{(N+4s)c^{2+\frac{8s}{N}}\lambda ^{4s}}{2N \vert U \vert _{2}^{\frac{8s}{N}}}, \\& \int _{\mathbb{R}^{N}}V(x) \bigl\vert u_{\lambda }(x) \bigr\vert ^{2+\gamma }\,dx= \frac{c^{2+\gamma }\lambda ^{\frac{N\gamma }{2}}}{ \vert U \vert _{2}^{2+\gamma }} \int _{\mathbb{R}^{N}}V \biggl(\frac{x}{\lambda } \biggr) \bigl\vert U(x) \bigr\vert ^{2+ \gamma }\,dx. \end{aligned}$$

Hence we can deduce that $u_{\lambda }\in S_{c}$ and

$$\begin{aligned} I_{p}(u_{\lambda })&=\frac{a}{2}c^{2}\lambda ^{2s}+\frac{b}{4}c^{4} \lambda ^{4s}+ \frac{c^{\gamma +2}}{(2+\gamma ) \vert U \vert _{2}^{2+\gamma }} \lambda ^{\frac{N\gamma }{2}} \int _{\mathbb{R}^{N}}V \biggl( \frac{x}{\lambda } \biggr) \vert U \vert ^{2+\gamma }\,dx- \frac{c^{2+\frac{8s}{N}}}{4 \vert U \vert _{2}^{\frac{8s}{N}}}\lambda ^{4s} \\ &=\frac{a}{2}c^{2}\lambda ^{2s}+\frac{b}{4}c^{4} \lambda ^{4s} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{\frac{8s-2N}{N}} \biggr) \\ &\quad {}+ \frac{c^{\gamma +2}}{(2+\gamma ) \vert U \vert _{2}^{2+\gamma }}\lambda ^{ \frac{N\gamma }{2}} \int _{\mathbb{R}^{N}}V \biggl(\frac{x}{\lambda } \biggr) \vert U \vert ^{2+\gamma }\,dx. \end{aligned}$$

(18)

From $\gamma <\frac{8s}{N}$ we get that $\frac{N\gamma }{2}<4s$. Then (18) indicates that $I_{p}(u_{\lambda })\rightarrow -\infty$ as $\lambda \rightarrow \infty$, and the lemma is proved. □

Lemma 2.4

For any$c>0$, we have$e(c)\leq 0$.

Proof

For any $u\in S_{c}$ and constant $\lambda >0$, let $u_{\lambda }(x)=\lambda ^{\frac{N}{2}}u(\lambda x)$. Then $u_{\lambda }\in S_{c}$, and from (7) we have

$$\begin{aligned} I_{p}(u_{\lambda })&=\frac{a}{2}\lambda ^{2s} \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx+\frac{b}{4}\lambda ^{4s} \biggl( \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad{}+\frac{1}{2+\gamma }\lambda ^{\frac{N\gamma }{2}} \int _{\mathbb{R}^{N}}V \biggl(\frac{x}{\lambda } \biggr) \bigl\vert u(x) \bigr\vert ^{2+\gamma }\,dx- \frac{1}{p+2} \lambda ^{4s} \int _{\mathbb{R}^{N}} \bigl\vert u(x) \bigr\vert ^{2+\frac{8s}{N}}\,dx. \end{aligned}$$

(19)

Hence $I_{p}(u_{\lambda })\rightarrow 0$ as $\lambda \rightarrow 0$, which indicates that $e(c)\leq 0$. □

Lemma 2.5

Assume that the function$V(x)$satisfies condition (10), $\frac{N\gamma }{2}+\alpha <4s$, andais small enough. Then$e(c)<0$.

Proof

For fixed $|x_{0}|=2$, assume that $\varphi (x)\in C_{c}^{\infty }(\mathbb{R}^{N})$ is such that $\operatorname{supp} \varphi \in B_{1}(x_{0})$ and $\int _{\mathbb{R}^{N}}\varphi ^{2}(x)\,dx=c^{2}$. For constant $\lambda >0$, take

$$ \varphi _{\lambda }(x)=\lambda ^{\frac{N}{2}}\varphi (\lambda x). $$

Then

$$\begin{aligned}& \int _{\mathbb{R}^{N}}\varphi _{\lambda }^{2}(x)\,dx= \int _{\mathbb{R}^{N}} \varphi ^{2}(x)\,dx=c^{2}, \end{aligned}$$

(20)

$$\begin{aligned}& \begin{aligned}[b] \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert \varphi _{\lambda }^{2}(x)-\varphi _{\lambda }^{2}(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy&= \lambda ^{2s} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert \varphi ^{2}(x)-\varphi ^{2}(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy\\ &= \lambda ^{2s} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} \varphi \bigr\vert ^{2}\,dx, \end{aligned} \end{aligned}$$

(21)

$$\begin{aligned}& \int _{\mathbb{R}^{N}} \bigl\vert \varphi _{\lambda }(x) \bigr\vert ^{2+\frac{8s}{N}}\,dx= \lambda ^{4s} \int _{\mathbb{R}^{N}} \bigl\vert \varphi (x) \bigr\vert ^{2+\frac{8s}{N}}\,dx, \end{aligned}$$

(22)

and

$$ \int _{\mathbb{R}^{N}}V(x) \bigl\vert \varphi _{\lambda }(x) \bigr\vert ^{2+\gamma }\,dx= \lambda ^{\frac{N\gamma }{2}} \int _{\mathbb{R}^{N}}V\biggl(\frac{x}{\lambda }\biggr) \bigl\vert \varphi (x) \bigr\vert ^{2+\gamma }\,dx\leq -C\lambda ^{\frac{N\gamma }{2}+\alpha } $$

(23)

as $\lambda \rightarrow 0$.

From (20) we know that $\varphi _{\lambda }\in S_{c}$. Then (21)–(23) indicate that

$$\begin{aligned} \begin{aligned}[b] I_{p}(\varphi _{\lambda })&\leq \frac{a\lambda ^{2s}}{2} \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} \varphi \bigr\vert ^{2}\,dx+ \frac{b\lambda ^{4s}}{4} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} \varphi \bigr\vert ^{2}\,dx\\ &\quad {}-C\lambda ^{\frac{N\gamma }{2}+\alpha }- \frac{N\lambda ^{4s}}{2N+8s} \int _{\mathbb{R}^{N}} \bigl\vert \varphi (x) \bigr\vert ^{2+ \frac{8s}{N}}\,dx. \end{aligned} \end{aligned}$$

(24)

For $2\leq \gamma <\frac{8s}{N}$, we have $2s< N\leq \frac{N\gamma }{2}<4s$. If $\frac{N\gamma }{2}+\alpha <4s$, then there is a small $\lambda _{0}>0$ such that

$$ \frac{b\lambda _{0}^{4s}}{4} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} \varphi \bigr\vert ^{2}\,dx-C\lambda _{0}^{\frac{N\gamma }{2}+\alpha }- \frac{N\lambda _{0}^{4s}}{2N+8s} \int _{\mathbb{R}^{N}} \bigl\vert \varphi (x) \bigr\vert ^{2+ \frac{8s}{N}}\,dx< 0. $$

Moreover, if

$$ a< \frac{-\frac{b\lambda _{0}^{2s}}{2}\int _{\mathbb{R}^{N}} \vert (-\Delta )^{\frac{s}{2}} \varphi \vert ^{2}\,dx+2C\lambda _{0}^{\frac{N\gamma }{2}+\alpha -2s}+\frac{N\lambda _{0}^{2s}}{N+4s}\int _{\mathbb{R}^{N}} \vert \varphi (x) \vert ^{2+\frac{8s}{N}}\,dx}{\int _{\mathbb{R}^{N}} \vert (-\Delta )^{\frac{s}{2}} \varphi \vert ^{2}\,dx}, $$

then from (24) we can deduce that

$$ e(c)\leq \inf I_{p}(\varphi _{\lambda })< 0. $$

 □

Lemma 2.6

For any$c\in (0,c^{*})$and any$d\in (0,c)$, if$e(c)<0$, then

$$ e(c)< e(d)+e\bigl(\sqrt{c^{2}-d^{2}}\bigr). $$

Proof

Let $\{u_{n}\}$ be any minimizing sequence. Then

$$\begin{aligned} \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{\gamma +2} \,dx&= \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{( \gamma +2)\theta } \vert u_{n} \vert ^{(\gamma +2)(1-\theta )}\,dx \\ &\leq \biggl( \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2}\,dx \biggr)^{ \frac{(2+\gamma )\theta }{2}} \biggl( \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2^{*}(s)}\,dx \biggr)^{\frac{(2+\gamma )(1-\theta )}{2^{*}(s)}} \\ &\leq C \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{\frac{\gamma N}{4s}}, \end{aligned}$$

(25)

where $\theta =\frac{2s(2+\gamma )-\gamma N}{2(2+\gamma )s}$.

Using (12) and (25), we have

$$\begin{aligned} I_{p}(u_{n})&=\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx+\frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad {}+ \frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{\gamma +2} \,dx-\frac{N}{2N+8s} \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+\frac{8s}{N}}\,dx \\ &\geq \frac{b}{4} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{ \frac{8s-2N}{N}} \biggr) \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad {}-C \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{\frac{\gamma N}{4s}}, \end{aligned}$$

(26)

where $c^{*}= (b|U(x)|_{2}^{\frac{8s}{N}} )^{\frac{N}{8s-2N}}$. Since $\gamma <\frac{8s}{N}$, we have that $\frac{\gamma N}{4s}<2$. Since $\{u_{n}\}$ is a minimizing sequence and $c< c^{*}$, we have $e(c)=\lim_{n\rightarrow \infty }I_{p}(u_{n})$, and the sequence $\{u_{n}\}$ is bounded in the space $H^{s}(\mathbb{R}^{N})$. Moreover, from (26) we can deduce that $0>e(c)>-\infty$ and

$$\begin{aligned} &\lim_{n\rightarrow \infty }\frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{ \gamma +2} \,dx \\ &\quad \leq e(c)-\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx- \frac{b}{4} \biggl(1- \biggl(\frac{c}{c^{*}} \biggr)^{\frac{8s-2N}{N}} \biggr) \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad < 0. \end{aligned}$$

(27)

For $\lambda >1$, defining $\bar{u}_{n}=\lambda u_{n}$, we have

$$\begin{aligned}& \int _{\mathbb{R}^{N}}\bar{u}_{n}^{2}\,dx=\lambda ^{2} \int _{\mathbb{R}^{N}}u_{n}^{2}\,dx= \lambda ^{2}c^{2},\qquad \int _{\mathbb{R}^{N}}V(x)\bar{u}_{n}^{ \gamma +2}\,dx=\lambda ^{\gamma +2} \int _{\mathbb{R}^{N}}V(x)u_{n}^{ \gamma +2}\,dx, \\& \begin{aligned} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert \bar{u}_{n}^{2}(x)-\bar{u}_{n}^{2}(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy&= \lambda ^{2} \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}} \frac{ \vert u_{n}^{2}(x)-u_{n}^{2}(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx \,dy\\ &=\lambda ^{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx, \end{aligned} \\& \int _{\mathbb{R}^{N}} \vert \bar{u}_{n} \vert ^{2+\frac{8s}{N}}\,dx=\lambda ^{2+ \frac{8s}{N}} \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+\frac{8s}{N}} \,dx. \end{aligned}$$

Then

$$\begin{aligned} I_{p}(\bar{u}_{n})&=\frac{a\lambda ^{2}}{2} \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx+\frac{b\lambda ^{4}}{4} \biggl( \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad {}+ \frac{\lambda ^{2}}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{ \gamma +2} \,dx-\frac{N\lambda ^{2+\frac{8s}{N}}}{2N+8s} \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+ \frac{8s}{N}}\,dx \\ &\geq \lambda ^{4}I_{p}(u_{n})+\bigl(\lambda ^{2}-\lambda ^{4}\bigr) \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \\ &\quad {}+ \bigl(\lambda ^{4}- \lambda ^{2+\frac{8s}{N}} \bigr)\frac{N}{2N+8s} \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+ \frac{8s}{N}}\,dx \\ &\quad {}+ \bigl(\lambda ^{\gamma +2}-\lambda ^{4} \bigr) \frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{\gamma +2} \,dx, \end{aligned}$$

(28)

which, together with $\lambda >1$, $\gamma \geq 2$, and (27), indicates that

$$ e(\lambda c)\leq \lim_{n\rightarrow \infty } I_{p}( \bar{u}_{n})\leq \lambda ^{4} \lim_{n\rightarrow \infty } I_{p}(u_{n})=\lambda ^{4}e(c). $$

(29)

Since $e(c)<0$, this means that

$$ e(\lambda c)< \lambda e(c). $$

Then for any $d\in [0,c)$, we have

$$ e(c)< e(d)+e\bigl(\sqrt{c^{2}-d^{2}}\bigr). $$

 □

3 The proof of theorems

Proof of Theorem 1.1

(1) From Lemma 2.3 we know that $e(c)<-\infty$. Hence it is natural that for any $c>c^{*}$, there are no minimizers for problem (6).

(2) From Lemma 2.1 we know that since $V(x)\geq 0$, $e(c)\geq 0$. This, together with Lemma 2.4, indicates that $e(c)=0$. Assume that there is $u_{0}\in S_{c}$ such that

$$ I_{p}(u_{0})=e(c)=0, $$

which contradicts with (15) since $I_{p}(u_{0})>0$ for any $V(x)\geq 0$. Thus there are no minimizers for problem (6).

(3) From Lemma 2.2 we have that $e(c)\geq 0$. This, together with Lemma 2.4, indicates that $e(c)=0$. Similarly to the proof of (2), we can deduce that there are no minimizers for problem (6). □

Proof of Theorem 1.2

Let $\{u_{n}\}$ be a minimizing sequence of $e(c)$. From (26) we get that $\int _{\mathbb{R}^{N}} |(-\Delta )^{\frac{s}{2}} u_{n}|^{2}\,dx$ is bounded above, which, combined with $\int _{\mathbb{R}^{N}} |u_{n}|^{2}\,dx=c^{2}$, implies that $\{u_{n}\}$ is bounded in the space $H^{s}(\mathbb{R}^{N})$. Hence there is $u\in H^{s}(\mathbb{R}^{N})$ such that there is a subsequence of $\{u_{n}\}$, denoted still by $\{u_{n}\}$, such that $u_{n}\rightharpoonup u$ in $H^{s}(\mathbb{R}^{N})$. Then by the concentration-compactness principle [16] the sequence $\{u_{n}\}$ is compact. Hence the key point is excluding the case of vanishing (i.e., $u=0$ in $H^{s}(\mathbb{R}^{N})$) and dichotomy (i.e.m $u\neq 0$ in $H^{s}(\mathbb{R}^{N})$ but $0<|u|_{2}<c$).

For any $0< R<\infty$, set

$$ \delta =\limsup_{n\rightarrow \infty , y\in \mathbb{R}^{N}} \int _{B_{R}(y)} \vert u_{n} \vert ^{2}\,dx. $$

If $\delta =0$, then using the vanishing lemma (Lemma I.1 in [16]), we have

$$ u_{n}\rightarrow 0,\quad \text{in } L^{q}\bigl( \mathbb{R}^{N}\bigr), q\in \bigl(2, 2^{*}(s)\bigr). $$

This indicates that

$$\begin{aligned}& \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+ \frac{8s}{N}} \,dx=0, \end{aligned}$$

(30)

$$\begin{aligned}& \lim_{n\rightarrow \infty } \biggl\vert \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{2+ \gamma } \,dx \biggr\vert \leq \vert V_{\infty } \vert \lim_{n\rightarrow \infty } \int _{ \mathbb{R}^{N}} \vert u_{n} \vert ^{2+\gamma } \,dx=0. \end{aligned}$$

(31)

Using (30) and (31), we can deduce that

$$\begin{aligned} e(c)&=\lim_{n\rightarrow \infty } I_{p}(u_{n}) \\ &=\lim _{n\rightarrow \infty } \biggl[\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx+ \frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad {}+\frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{\gamma +2} \,dx- \frac{N}{2N+8s} \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+\frac{8s}{N}}\,dx \biggr] \\ &=\lim_{n\rightarrow \infty } \biggl(\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx+\frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2} \biggr) \geq 0, \end{aligned}$$

(32)

a contradiction to Lemma 2.5. Hence vanishing is impossible.

Now we assume that dichotomy occurs. Then there are $d\in (0,c)$ and bounded sequences $\{u_{n}^{1}\}$, $\{u_{n}^{2}\}$ in $H^{s}(\mathbb{R}^{N})$ such that for any $q\in [2, 2^{*}(s))$, we have

$$\begin{aligned}& \bigl\vert u_{n}-u_{n}^{2}-u_{n}^{2} \bigr\vert _{q}\leq \sigma _{q}(\varepsilon ), \end{aligned}$$

(33)

$$\begin{aligned}& \biggl\vert \int _{\mathbb{R}^{N}} \bigl\vert u_{n}^{1} \bigr\vert ^{2}\,dx-d^{2} \biggr\vert \leq \varepsilon ,\qquad \biggl\vert \int _{\mathbb{R}^{N}} \bigl\vert u_{n}^{2} \bigr\vert ^{2}\,dx-\bigl(c^{2}-d^{2}\bigr) \biggr\vert \leq \varepsilon , \end{aligned}$$

(34)

$$\begin{aligned}& \operatorname{dist}\bigl(\operatorname{supp} u_{n}^{1}, \operatorname{supp} u_{n}^{2}\bigr)\rightarrow \infty\quad \text{as } n\rightarrow \infty , \end{aligned}$$

(35)

and

$$ \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}} \bigl[ \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}- \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n}^{1} \bigr\vert ^{2}- \bigl\vert (- \Delta )^{\frac{s}{2}} u_{n}^{2} \bigr\vert ^{2} \bigr]\,dx. $$

(36)

Using (33)–(36), we can deduce that

$$\begin{aligned} e(c)&=\lim_{n\rightarrow \infty }I_{p}(u_{n})\geq \lim _{n \rightarrow \infty } \bigl[I_{p}\bigl(u_{n}^{1} \bigr)+I_{p}\bigl(u_{n}^{2}\bigr) \bigr]+ \sigma ( \varepsilon ) \\ &\geq e(d)+e\bigl(\sqrt{c^{2}-d^{2}}\bigr)+\sigma ( \varepsilon ), \end{aligned}$$

(37)

where $\sigma (\varepsilon )\rightarrow 0$ as $\varepsilon \rightarrow 0$. Let $\varepsilon \rightarrow 0$. Then (37) contradicts to Lemma 2.6. Hence dichotomy cannot occur, and for any $\varepsilon >0$, there exist $R_{\varepsilon }>0$ and $\{y_{n}\}\subset \mathbb{R}^{N}$ such that

$$ \int _{B_{R_{\varepsilon }}(y_{n})} \vert u_{n} \vert ^{2}\,dx \geq c-\varepsilon . $$

(38)

Next, we discuss this problem for two cases: $\{y_{n}\}$ is bounded and $y_{n}\rightarrow \infty$ as $n\rightarrow \infty$.

(1) If $\{y_{n}\}$ is bounded from above, then (38) indicates that

$$ u_{n}\rightarrow u\quad \text{in } L^{2}\bigl( \mathbb{R}^{N}\bigr). $$

Since $\{u_{n}\}$ is bounded in the space $H^{s}(\mathbb{R}^{N})$, the Gagliardo–Nirenberg inequality gives that

$$ \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+\frac{8s}{N}}\,dx \leq C \biggl( \int _{ \mathbb{R}^{N}} \vert u_{n} \vert ^{2}\,dx \biggr)^{\frac{4s-N}{2}}. $$

By Lebesgue’s dominate convergence theorem we get that

$$ \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+ \frac{8s}{N}} \,dx= \int _{\mathbb{R}^{N}} \vert u \vert ^{2+\frac{8s}{N}}\,dx. $$

(39)

Similarly to the proof of (39), we obtain that

$$ \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{2+ \gamma } \,dx= \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{2+\gamma }\,dx. $$

(40)

From [6] we know that the norm $\int _{\mathbb{R}^{N}} |(-\Delta )^{\frac{s}{2}} u_{n}|^{2}\,dx$ satisfies weak lower semi-continuity, that is,

$$ \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx\leq \liminf_{n \rightarrow \infty } \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx. $$

Then

$$ \begin{aligned} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2} &\leq \biggl( \liminf _{n\rightarrow \infty } \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2}\\ & \leq \liminf_{n \rightarrow \infty } \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2}, \end{aligned} $$

which, together with (39) and (40), implies that

$$ e(c)\leq I_{p}(u)\leq \liminf_{n\rightarrow \infty }I_{p}(u_{n})=e(c). $$

This implies that

$$\begin{aligned}& \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx= \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx,\\& \lim_{n\rightarrow \infty } \biggl( \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2}= \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u \bigr\vert ^{2}\,dx \biggr)^{2}. \end{aligned}$$

Then the sequence $\{u_{n}\}$ has a strongly convergent subsequence, which means that u is a minimizer of $e(c)$.

(2) If $y_{n}\rightarrow \infty$ as $n\rightarrow \infty$, then from the definition of $V(x)$ we know that

$$ \lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \int _{B_{R_{\varepsilon }}(y_{n})}V(x) \vert u_{n} \vert ^{\gamma +2} \,dx=0. $$

(41)

From (25) we have

$$ \begin{aligned} \lim_{n\rightarrow \infty } \biggl\vert \int _{\mathbb{R}^{N}\setminus B_{R_{\varepsilon }}(y_{n})}V(x) \vert u_{n} \vert ^{\gamma +2} \,dx \biggr\vert &\leq C\lim_{n \rightarrow \infty } \biggl( \int _{\mathbb{R}^{N}\setminus B_{R_{\varepsilon }}(y_{n})} \vert u_{n} \vert ^{2}\,dx \biggr)^{ \frac{2s(\gamma +2)-\gamma N}{4s}}\\ &\leq C \varepsilon ^{ \frac{2s(\gamma +2)-\gamma N}{4s}}, \end{aligned} $$

from which by letting $\varepsilon \rightarrow 0$ we have

$$ \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}\setminus B_{R_{\varepsilon }}(y_{n})}V(x) \vert u_{n} \vert ^{\gamma +2} \,dx=0. $$

This, together with (41), indicates that

$$ \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{\gamma +2} \,dx=0. $$

(42)

Using (42) and the Gagliardo–Nirenberg inequality (12), we deduce that

$$\begin{aligned} e(c)&=\lim_{n\rightarrow \infty } \biggl[\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx+\frac{b}{4} \biggl( \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\quad {}+\frac{1}{\gamma +2} \int _{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{\gamma +2} \,dx- \frac{N}{2N+8s} \int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{2+\frac{8s}{N}}\,dx \biggr] \\ &=\lim_{n\rightarrow \infty } \biggl[\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx+\frac{b}{4} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2}\\ &\quad {}- \frac{N}{2N+8s} \int _{ \mathbb{R}^{N}} \vert u_{n} \vert ^{2+\frac{8s}{N}}\,dx \biggr] \\ &\geq \lim_{n\rightarrow \infty } \biggl[\frac{a}{2} \int _{\mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx+\frac{b}{4} \biggl(1- \biggl( \frac{c}{c^{*}} \biggr)^{\frac{8s-2N}{N}} \biggr) \biggl( \int _{ \mathbb{R}^{N}} \bigl\vert (-\Delta )^{\frac{s}{2}} u_{n} \bigr\vert ^{2}\,dx \biggr)^{2} \biggr] \\ &>0, \end{aligned}$$

which contradicts to Lemma 2.5. Hence $y_{n}\rightarrow \infty$ as $n\rightarrow \infty$ cannot occur. □

Proof of Theorem 1.3.

The proof is similar to that of Theorem 1.2. We omit it. □