Boundary Value Problems

, 2019:25

# Least energy sign-changing solutions for the fractional Schrödinger–Poisson systems in $$\mathbb{R}^{3}$$

• Da-Bin Wang
• Yu-Mei Ma
• Wen Guan
Open Access
Research

## Abstract

In this paper, we study the following nonlinear fractional Schrödinger–Poisson system
$$\textstyle\begin{cases} (-\Delta )^{s}u+V(x)u+\phi u=K(x)f(u),&x\in \mathbb{R}^{3}, \\ (-\Delta )^{t} \phi =u^{2},&x\in {\mathbb{R}}^{3}. \end{cases}$$
(0.1)
where $$s\in (\frac{3}{4},1)$$, $$t\in (0,1)$$, $$V,K : \mathbb{R}^{3}\rightarrow \mathbb{R}$$ are continuous functions verifying some conditions about zero mass. By using the constraint variational method and the quantitative deformation lemma, we obtain the existence of least energy sign-changing solution to this system.

## Keywords

Fractional Schrödinger–Poisson systems Constraint variational methods Quantitative deformation lemma Sign-changing solution Zero mass

## 1 Introduction and main results

In this article, we are interested in the existence of the sign-changing solutions for the following fractional Schrödinger–Poisson system
$$\textstyle\begin{cases} (-\Delta )^{s}u+V(x)u+\phi u=K(x)f(u),&x\in \mathbb{R}^{3}, \\ (-\Delta )^{t} \phi =u^{2},&x\in {\mathbb{R}}^{3}, \end{cases}$$
(1.1)
where $$s\in (\frac{3}{4},1)$$, $$t\in (0,1)$$. The fractional Laplacian operator $$(-\Delta )^{s}$$ is defined by
\begin{aligned} (-\Delta )^{s}u(x)=C_{N,s}P.V. \int _{\mathbb{R}^{N}} \frac{u(x)-u(y)}{ \vert x-y \vert ^{N+2s}}\,\mathrm{d}y,\quad u\in \mathbb{S} \bigl(\mathbb{R} ^{N}\bigr), \end{aligned}
where $$P.V$$. stands for the Cauchy principal value, $$C_{N,s}$$ is a normalized constant, $$\mathbb{S}(\mathbb{R}^{N})$$ is the Schwartz space of rapidly decaying functions.
Throughout this paper, as in [3], we say that $$(V,K)\in \mathcal{K}$$ if continuous functions $$V, K:\mathbb{R}^{3}\rightarrow \mathbb{R}$$ satisfy the following conditions:
$$(H_{0})$$

$$V(x),K(x)>0$$ for all $$x\in \mathbb{R}^{3}$$ and $$K\in L^{\infty }(\mathbb{R}^{3})$$;

$$(H_{1})$$
If $$\{A_{n}\}\subset \mathbb{R}^{3}$$ is a sequence of Borel sets such that the Lebesgue measure $$|A_{n}|\leq R$$, for all n and some $$R>0$$, then
$$\lim_{r\rightarrow +\infty } \int _{A_{n}\bigcap B^{c}_{r}(0)}K(x) \,\mathrm{d}x=0, \quad \text{uniformly in }n=1,2,\ldots ;$$
One of the following conditions occurs:
$$(H_{2})$$

$$K/V\in L^{\infty }(\mathbb{R}^{3})$$; or

$$(H_{3})$$
There exists $$p\in (2,2^{*}_{s})$$ such that
$$\frac{K(x)}{V(x)^{\frac{2^{*}_{s}-p}{2^{*}_{s}-2}}}\rightarrow 0 \quad \text{as } \vert x \vert \rightarrow + \infty ,$$
where $$2^{*}_{s}=\frac{6}{3-2s}$$ is the fractional critical exponent.
As for the function f, we assume $$f\in C^{1}(\mathbb{R},\mathbb{R})$$ and:
$$(f_{1})$$

$$\lim_{t\rightarrow 0}f(t)/|t|=0$$, if $$(H_{2})$$ holds;

$$(f_{2})$$

$$\lim_{t\rightarrow 0}f(t)/|t|^{p-1}=A\in \mathbb{R}$$, if $$(H_{3})$$ holds;

$$(f_{3})$$

f has a “quasicritical growth”, namely, $$\lim_{|t|\rightarrow \infty }f(t)/|t|^{2^{*}_{s}-1}=0$$;

$$(f_{4})$$

$$\lim_{|t|\rightarrow \infty }F(t)/t^{4}=\infty$$, where $$F(t)=\int ^{t}_{0}f(s)\,\mathrm{d}s$$;

$$(f_{5})$$

The map $$t\mapsto f(t)/|t|^{3}$$ is nondecreasing on $$(- \infty ,0)$$ and $$(0,\infty )$$.

### Remark 1.1

Similar conditions as hypotheses $$(H_{0})$$$$(H_{3})$$ on functions V and K were firstly introduced in [3] and characterize a class of Schrödinger–Poisson problems as zero-mass problem.

When $$s=t=1$$, $$K(x)\equiv 1$$, system (1.1) reduces to the following Schrödinger–Poisson system
$$\textstyle\begin{cases} -\Delta u+V(x)+\lambda \phi (x)u=f(u),&x\in \mathbb{R}^{3}, \\ -\Delta \phi =u^{2},&x\in \mathbb{R}^{3}. \end{cases}$$
(1.2)
System (1.2) comes from time-dependent Schrödinger–Poisson equation, which describes quantum (nonrelativistic) particles interacting with the electromagnetic field generated by the motion. For more details on the mathematical and physical background of system (1.2), we refer the readers to the papers [10, 11] and the references therein. Since the so-called nonlocal term $$\lambda \phi _{u}(x) u$$ is involved, system (1.2) is called a nonlocal problem. The appearance of the nonlocal term in the equations not only makes it important in many physical applications but also causes some difficulties and challenges from a mathematical point of view. Therefore, in the past several decades, there has been an increasing attention toward systems (1.2) or similar problems, and the existence of positive, multiple, bound state, multi-bump, as well as semiclassical state solutions has been investigated; see, for example, [4, 7, 9, 10, 13, 16, 21, 24, 29, 37, 38, 43, 47, 48, 49, 60]. Besides, He and Zou [23] considered multiplicity of concentrating positive solutions for a class of double parameter perturbed Schrödinger–Poisson equation with critical growth.

For sign-changing solutions, Alves and Souto [2] proved that system (1.2) possesses a least-energy sign-changing solution, in which $$\mathbb{R}^{3}$$ is replaced by bounded domains with smooth boundary. Via a constraint variational method combined with the Brouwer degree theory, Wang and Zhou [50] investigated the existence of least-energy sign-changing solutions for the system (1.2) when $$f(u)=|u|^{p-1}u$$, $$p\in (3,5)$$. By using the constraint variational methods and the quantitative deformation lemma, Shuai and Wang [42] studied the existence and the asymptotic behavior of least energy sign-changing solution for system (1.2). Latter, under some more weak assumptions on f, Chen and Tang [15] improved and generalized some results obtained in [42]. For other work about sign-changing solution of system (1.2) or similar problems, we refer the reader to [9, 25, 26, 30, 32, 61] and the reference therein.

The nonlinear fractional Schrödinger–Poisson systems (1.1) also come from the following fractional Schrödinger equation
\begin{aligned} (-\Delta )^{s}u+V(x)u=f(x,u) \quad\text{in } \mathbb{R}^{N}. \end{aligned}
(1.3)
Equation (1.3) has been first proposed by Laskin [27, 28] as a result of expanding the Feynman path integral from the Brownian-like to the Lévy-like quantum mechanical paths. There are many interesting papers which considered the existence, multiplicity, uniqueness, regularity and asymptotic decay properties of the solutions to fractional Schrödinger equation (1.3), see [1, 5, 12, 18, 22, 35, 39, 40, 46, 59] and references therein. Besides, some more complicated fractional equations and systems were also studied, and indeed some interesting results were obtained, see [19, 45, 53, 54, 55] and references therein. Furthermore, there is a very interesting book [36], in which nonlocal fractional problems are systematic investigated. For sign-changing solutions, since the fractional Laplacian operator is nonlocal, there are important structural differences between the classical and the fractional Laplacian. In fact, for $$u\in H^{1}(\mathbb{R}^{N})$$, we have
$$\int _{\mathbb{R}^{N}} \vert \nabla u \vert ^{2}\, \mathrm{d}x= \int _{\mathbb{R}^{N}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,\mathrm{d}x+ \int _{\mathbb{R}^{N}} \bigl\vert \nabla u^{-} \bigr\vert ^{2} \,\mathrm{d}x.$$
However, for $$u\in H^{s}(\mathbb{R}^{N})$$, we have that
\begin{aligned} \int _{\mathbb{R}^{N}} \bigl\vert (-\triangle )^{\frac{s}{2}} u \bigr\vert ^{2}\,\mathrm{d}x&= \int _{\mathbb{R}^{N}} \bigl\vert (-\triangle )^{\frac{s}{2}} u^{+} \bigr\vert ^{2} \,\mathrm{d}x+ \int _{\mathbb{R}^{N}} \bigl\vert (-\triangle )^{\frac{s}{2}} u^{-} \bigr\vert ^{2} \,\mathrm{d}x\\ &\quad {}+2 \int _{\mathbb{R}^{N}}(-\triangle )^{\frac{s}{2}}u^{+}(- \triangle )^{\frac{s}{2}} u^{-}\,\mathrm{d}x. \end{aligned}

This fact makes the study of sign-changing solutions to fractional Schrödinger equation (1.3) particularly interesting, and indeed some interesting results were obtained; see, for example, [6, 14, 17, 51] and the references therein.

Before presenting our main result, let us first recall some Sobolev space as follows. We denote $$D^{s,2}(\mathbb{R}^{3})$$ by the closure of function space $$C^{\infty }_{c}(\mathbb{R}^{3})$$ with respect to the so-called Gagliardo seminorm
$$[u]^{2}:= \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y.$$
Since (1.1) is a zero-mass problem, it seems that the appropriate working space should be
$$X=\biggl\{ u\in D^{s,2}\bigl(\mathbb{R}^{3}\bigr): \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{2}(x)u^{2}(y)}{ \vert x-y \vert ^{3-2t}}\,\mathrm{d}x \,\mathrm{d}y< \infty \biggr\}$$
with the norm
$$\Vert u \Vert ^{2}_{X}= \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ C_{t} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u^{2}(x)u^{2}(y)}{ \vert x-y \vert ^{3-2t}}\,\mathrm{d}x\,\mathrm{d}y.$$

Then, we know that $$(X,\|\cdot \|_{X})$$ is a Banach space.

By the Lax–Milgram Theorem, for any $$u\in X$$, there exists a unique $$\phi _{u}^{t}\in D^{t,2}(\mathbb{R}^{3})$$ such that
$$\int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{(\phi ^{t}_{u}(x)-\phi ^{t}_{u}(y))(v(x)-v(y))}{ \vert x-y \vert ^{3+2t}}\,\mathrm{d}x\,\mathrm{d}y= \int _{\mathbb{R}^{3}}u^{2}v\,\mathrm{d}x$$
for any $$v\in D^{t,2}(\mathbb{R}^{3})$$, that is, $$\phi _{u}^{t}$$ is a weak solution of
$$(-\Delta )^{t} \phi =u^{2},\quad x\in {\mathbb{R}}^{3}.$$
In fact, we have that
$$\phi ^{t}_{u}(x)=C_{(t)} \int _{\mathbb{R}^{3}} \frac{u^{2}(y)}{ \vert x-y \vert ^{3-2t}}\,\mathrm{d}y,\quad x \in \mathbb{R}^{3}$$
(1.4)
where $$C_{(t)}=\pi ^{-\frac{3}{2}}2^{-2t}\frac{\varGamma (\frac{3-2t}{2})}{ \varGamma (t)}$$.
Using the expression of (1.4), we obtain that system (1.1) is merely a single equation for u:
\begin{aligned} (-\Delta )^{s}u+V(x)u+\phi _{u}^{t}u=K(x)f(u) \quad\text{in } \mathbb{R}^{3}. \end{aligned}
(1.5)
The condition $$(V,K)\in \mathcal{K}$$ is fascinating. It can be used to certify that the space E given by
$$E=\biggl\{ u\in D^{s,2}\bigl(\mathbb{R}^{3}\bigr): \int _{\mathbb{R}^{3}}V(x) \vert u \vert ^{2} \,\mathrm{d}x< + \infty \biggr\}$$
endowed with the norm
$$\Vert u \Vert ^{2}_{E}= \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ \int _{\mathbb{R}^{3}}V(x) \vert u \vert ^{2}\,\mathrm{d}x$$
is compactly embedded into the weighted Lebesgue space
$$L^{q}_{K}\bigl(\mathbb{R}^{3}\bigr)=\biggl\{ u:u \text{ is measurable on }\mathbb{R}^{3}\text{ and } \int _{\mathbb{R}^{3}}K \vert u \vert ^{q}< \infty \biggr\}$$
for some $$q\in (2,2^{*}_{s})$$, see Proposition 2.2 below.
However, because of the zero-mass situation, we need to consider a new space
$$H=\biggl\{ u\in D^{s,2}\bigl(\mathbb{R}^{3}\bigr): \int _{\mathbb{R}^{3}}V(x) \vert u \vert ^{2} \,\mathrm{d}x< \infty , \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u ^{2}(x)u^{2}(y)}{ \vert x-y \vert ^{3-2t}}\,\mathrm{d}x\,\mathrm{d}y< \infty \biggr\}$$
with the norm
$$\Vert u \Vert ^{2}= \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ \int _{\mathbb{R}^{3}}V(x) \vert u \vert ^{2}\,\mathrm{d}x+ C_{t} \int _{\mathbb{R} ^{3}} \int _{\mathbb{R}^{3}}\frac{u^{2}(x)u^{2}(y)}{ \vert x-y \vert ^{3-2t}} \,\mathrm{d}x\,\mathrm{d}y.$$
Since $$(X,\|\cdot \|_{X})$$ and $$(E,\|\cdot \|_{E})$$ are both Banach, it follows from $$(H_{0})$$ that $$(H,\|\cdot \|)$$ is also a Banach space. Denote the usual norm of $$L^{p}(\mathbb{R}^{3})$$ by $$|\cdot |_{p}$$. By Sobolev embedding theorem, the embedding $$E\hookrightarrow D^{s,2} \hookrightarrow L^{2^{*}_{s}}(\mathbb{R}^{3})$$ is continuous. Let $$S'>0$$ be the embedding constant, i.e.,
$$\vert u \vert _{2_{s}^{*}}^{2}\leq S^{\prime -1} \Vert u \Vert _{E}^{2},\quad u\in E.$$
(1.6)
The energy functional associated with system (1.1) is defined by
\begin{aligned}[b] J(u)&=\frac{1}{2} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \frac{1}{2} \int _{\mathbb{R}^{3}}V(x)u^{2}\,\mathrm{d}x+\frac{1}{4} \int _{\mathbb{R}^{3}}\phi ^{t}_{u}u^{2}\, \mathrm{d}x \\ &\quad{}- \int _{\mathbb{R} ^{3}}K(x)F(u)\,\mathrm{d}x,\quad u\in H, \end{aligned}
where $$F(u)=\int ^{u}_{0}f(t)\,dt$$.
Moreover, under our conditions, J belongs to $$C^{1}(H,\mathbb{R})$$, and the Fréchet derivative of J is
\begin{aligned}[b] \bigl\langle J'(u),v\bigr\rangle &= \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{(u(x)-u(y))(v(x)-v(y))}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \int _{\mathbb{R}^{3}}V(x)uv\,\mathrm{d}x \\ &\quad{}+ \int _{\mathbb{R}^{3}}\phi ^{t}_{u}uv\,\mathrm{d}x- \int _{\mathbb{R}^{3}}K(x)f(u)v\,\mathrm{d}x,\quad u, v \in H. \end{aligned}
As is well known, a critical point of J is a weak solution of system (1.1). Furthermore, if $$u\in H$$ is a weak solution of system (1.1) and $$u^{\pm }\neq 0$$, we say that u is a sign-changing solution of system (1.1), where
$$u^{+}(x)=\max \bigl\{ u(x),0\bigr\} ,\qquad u^{-}(x)=\min\bigl\{ u(x),0\bigr\} .$$

Since fractional Schrodinger equation is coupled with a fractional Poisson term $$\phi (x)u$$, the existence of multiple nonlocal terms causes some mathematical difficulties and makes the study of system (1.1) very interesting. In recent years, several scholars paid their attention to the existence of positive, ground state, semiclassical and other solutions to fractional Schrödinger–Poisson system (1.1) or similar problems; see [33, 41, 44, 56, 57, 58] and references therein. Besides, in [34], Luo and Tang considered a class of doubly singularly perturbed fractional Schrödinger–Poisson system with critical Sobolev exponent, and proved the existence of ground state and multiple solutions for this system. However, to the best of our knowledge, few papers considered sign-changing solutions to fractional Schrödinger–Poisson system (1.1) or similar problems. Via the quantitative deformation lemma and degree theory, Guo [20] studied the existence and asymptotic behavior of sign-changing solutions for system (1.1).

Our goal in this paper is to seek the least energy sign-changing solutions to system (1.1). As in [3, 15, 42, 50], to overcome the difficulties and challenges stemming from the nonlocal term, we borrow some ideas from [8]. Specifically, we first try to seek a minimizer of the energy functional J under the following constraint:
$$\mathcal{M}=\bigl\{ u\in H:u^{\pm }\neq 0, \bigl\langle J'(u),u^{+}\bigr\rangle = \bigl\langle J'(u),u^{-}\bigr\rangle =0\bigr\} ,$$
and then we will prove that the minimizer is a sign-changing solution of system (1.1).

The main result can be stated as follows.

### Theorem 1.1

Suppose that$$(V,K)\in \mathcal{K}$$andfsatisfies$$(f_{1})$$$$(f _{5})$$. Then system (1.1) possesses at least one least-energy sign-changing solution.

## 2 Preliminary results

In this section we give some propositions and lemmas for convenience.

### Proposition 2.1

([6])

Assume$$(V,K)\in \mathcal{K}$$. If$$(H_{2})$$holds, thenEis continuously embedded in$$L^{q}_{K}(\mathbb{R}^{3})$$for every$$q\in [2,2^{*}_{s}]$$; if$$(H_{3})$$holds, thenEis continuously embedded in$$L^{p}_{K}(\mathbb{R}^{3})$$.

### Proposition 2.2

([6])

Assume$$(V,K)\in \mathcal{K}$$. If$$(H_{2})$$holds, thenEis compactly embedded in$$L^{q}_{K}(\mathbb{R}^{3})$$for every$$q\in (2,2^{*}_{s})$$; if$$(H_{3})$$holds, thenEis compactly embedded in$$L^{p}_{K}( \mathbb{R}^{3})$$.

### Proposition 2.3

([6])

Suppose thatfsatisfies$$(f_{1})$$$$(f_{3})$$and$$(V,K)\in \mathcal{K}$$. Let$$\{v_{n}\}$$be such that$$v_{n}\rightharpoonup v$$inE. Then
$$\int _{\mathbb{R}^{3}}K(x)F(v_{n})\,\mathrm{d}x\rightarrow \int _{\mathbb{R}^{3}}K(x)F(v)\,\mathrm{d}x,\qquad \int _{\mathbb{R}^{3}}K(x)f(v _{n})v_{n}\,\mathrm{d}x \rightarrow \int _{\mathbb{R}^{3}}K(x)f(v)v \,\mathrm{d}x.$$

Similarly as in [30], we have following lemmas.

### Lemma 2.1

Assume that$$(V,K)\in \mathcal{K}$$andfsatisfies$$(f_{1})$$$$(f _{5})$$. Then, for any$$u\in E\setminus \{0\}$$,
$$\lim_{|t|\rightarrow \infty } \int _{\mathbb{R}^{3}}\frac{Kf(tu)u}{t ^{3}}=\infty .$$

### Lemma 2.2

Assume that$$(V,K)\in \mathcal{K}$$andfsatisfies$$(f_{1})$$$$(f _{5})$$. Then, for any$$u\in E\setminus \{0\}$$,
$$\lim_{|t|\rightarrow \infty } \int _{\mathbb{R}^{3}} \frac{KF(tu)}{t^{4}}=\infty .$$

### Lemma 2.3

Assume that$$(V,K)\in \mathcal{K}$$andfsatisfies$$(f_{1})$$$$(f _{5})$$. Then, for any$$u\in E\setminus \{0\}$$,
$$\lim_{t\rightarrow 0} \int _{\mathbb{R}^{3}}\frac{Kf(tu)u}{t}=0.$$

## 3 Technical lemmas

In this section, we prove some technical lemmas related to the existence of sign-changing solutions of system (1.1).

For $$u\in H$$ with $$u^{\pm }\neq 0$$, we define $$G_{u}: \mathbb{R}^{2} _{+}\rightarrow \mathbb{R}$$ by $$G_{u}(\alpha ,\beta )=J(\alpha u^{+}+ \beta u^{-})$$.

### Lemma 3.1

Assume that$$(V,K)\in \mathcal{K}$$and$$(f_{1})$$$$(f_{5})$$hold. Then,
1. (i)

the pair$$(\alpha ,\beta )$$is a critical point of$$G_{u}$$with$$\alpha ,\beta >0$$if and only if$$\alpha u^{+}+\beta u ^{-}\in \mathcal{M}$$;

2. (ii)

the map$$G_{u}$$has a unique critical point$$(\alpha _{+},\beta _{-})$$, with$$\alpha _{+}=\alpha _{+}(u)>0$$and$$\beta _{-}= \beta _{-}(u)>0$$, which is the unique maximum point of$$G_{u}$$.

### Proof

By definition of $$G_{u}$$, we have that
\begin{aligned} \nabla G_{u}(\alpha ,\beta )&=\bigl(\bigl\langle J' \bigl(\alpha u^{+}+\beta u^{-}\bigr),u ^{+}\bigr\rangle ,\bigl\langle J'\bigl(\alpha u^{+}+\beta u^{-}\bigr),u^{-}\bigr\rangle \bigr) \\ &=\biggl(\frac{1}{\alpha }\bigl\langle J'\bigl(\alpha u^{+}+\beta u^{-}\bigr),\alpha u^{+} \bigr\rangle ,\frac{1}{\beta }\bigl\langle J'\bigl(\alpha u^{+}+\beta u^{-}\bigr),\beta u ^{-}\bigr\rangle \biggr) \\ &=\biggl(\frac{1}{\alpha }g_{u}(\alpha ,\beta ), \frac{1}{\beta }h_{u}( \alpha ,\beta )\biggr), \end{aligned}
where
\begin{aligned}& \begin{aligned}[b] g_{u}(\alpha ,\beta )&= \alpha ^{2} \bigl\Vert u^{+} \bigr\Vert _{E}^{2}-\alpha \beta \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad{}+\alpha ^{4} \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x+\alpha ^{2} \beta ^{2} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x \\ &\quad{}- \int _{\mathbb{R}^{3}}K(x)f\bigl(\alpha u^{+}\bigr)\alpha u^{+} \,\mathrm{d}x, \end{aligned} \end{aligned}
(3.1)
\begin{aligned}& \begin{aligned}[b] h_{u}(\alpha ,\beta )&= \beta ^{2} \bigl\Vert u^{-} \bigr\Vert _{E}^{2}-\alpha \beta \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad{}+\beta ^{4} \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x+\alpha ^{2} \beta ^{2} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2} \,\mathrm{d}x \\ &\quad{}- \int _{\mathbb{R}^{3}}K(x)f\bigl(\beta u^{-}\bigr)\beta u^{-} \,\mathrm{d}x. \end{aligned} \end{aligned}
(3.2)

From above facts, item (i) is obvious.

In the following, we prove (ii).

Firstly, we assert that $$\mathcal{M}\neq \emptyset$$ .

In fact, to this end, we just prove the existence of a critical point of $$G_{u}$$. Letting $$u\in H$$ with $$u^{\pm }\neq 0$$ and $$\beta _{0} \geq 0$$ fixed, from (3.1), we obtain
\begin{aligned}& \begin{aligned}[b] g_{u}(\alpha ,\beta _{0})&= \alpha ^{2}\biggl( \bigl\Vert u^{+} \bigr\Vert _{E}^{2}-\frac{\beta _{0}}{\alpha } \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u ^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad{}+ \alpha ^{2} \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x+ \beta _{0}^{2} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x \\ &\quad{}- \int _{\mathbb{R}^{3}}\frac{K(x)f(\alpha u^{+})u^{+}}{ \alpha }\,\mathrm{d}x\biggr), \end{aligned} \\& \begin{aligned}[b] g_{u}(\alpha ,\beta _{0})&= \alpha ^{4}\biggl(\frac{1}{\alpha ^{2}} \bigl\Vert u^{+} \bigr\Vert _{E}^{2}-\frac{\beta _{0}}{\alpha ^{3}} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y \\ &\quad{}+ \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x+\frac{\beta _{0}^{2}}{\alpha ^{2}} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad{}- \int _{\mathbb{R}^{3}}\frac{K(x)f( \alpha u^{+})u^{+}}{\alpha ^{3}}\,\mathrm{d}x\biggr). \end{aligned} \end{aligned}
Then, according to Lemmas 2.1 and 2.3, we have that
\begin{aligned} g_{u}(\alpha ,\beta _{0})>0 \quad \text{for }\alpha \text{ small enough};\qquad g_{u}(\alpha ,\beta _{0})< 0 \quad \text{for } \alpha \text{ large enough}. \end{aligned}
Since $$g_{u}(\alpha ,\beta _{0})$$ is continuous, there exists $$\alpha _{0}>0$$ such that $$g_{u}(\alpha _{0},\beta _{0})=0$$. We assert $$\alpha _{0}$$ is unique. In fact, supposing by contradiction, that there exist $$0<\alpha _{1}<\alpha _{2}$$ such that $$g_{u}(\alpha _{1},\beta _{0})=g _{u}(\alpha _{2},\beta _{0})$$, we then have
\begin{aligned}[b]&\biggl(\frac{1}{\alpha _{1}^{2}}-\frac{1}{\alpha _{2}^{2}} \biggr) \biggl( \bigl\Vert u^{+} \bigr\Vert _{E} ^{2}+\beta _{0}^{2} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x\biggr) \\ &\qquad {}+\biggl(\frac{\beta _{0}}{\alpha _{2}^{3}}-\frac{\beta _{0}}{ \alpha _{1}^{3}}\biggr) \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u ^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad = \int _{\mathbb{R}^{3}}K(x)\biggl[\frac{f(\alpha _{1} u^{+})}{(\alpha _{1} u ^{+})^{3}}-\frac{f(\alpha _{2} u^{+})}{(\alpha _{2} u^{+})^{3}} \biggr]\bigl(u^{+}\bigr)^{4} \,\mathrm{d}x. \end{aligned}

Therefore, in view of $$(f_{5})$$ and $$0<\alpha _{1}<\alpha _{2}$$, we obtain a contradiction. That is, there exists a unique $$\alpha _{0}>0$$ such that $$g_{u}(\alpha _{0},\beta _{0})=0$$.

Thus, we can define a function $$\varphi _{1}:\mathbb{R}_{+}\rightarrow (0,\infty )$$ by
$$\varphi _{1}(\beta )=\alpha (\beta ),$$
where $$\alpha (\beta )$$ satisfies $$g_{u}(\alpha (\beta ),\beta )=0$$.

By the same arguments as above, we can define functions $$\varphi _{2}: \mathbb{R}_{+}\rightarrow (0,\infty )$$ by $$\varphi _{2}(\alpha )= \beta (\alpha )$$ which satisfies $$h_{u}(\alpha ,\beta (\alpha ))=0$$.

Furthermore, the functions $$\varphi _{i}$$, $$i=1,2$$, have the following three good properties:
1. (a)

$$\varphi _{i}$$ are continuous on $$\mathbb{R}_{+}$$.

2. (b)

$$\varphi _{1}(\beta )>0$$, $$\varphi _{2}(\alpha )>0$$ for any $$\alpha ,\beta \in \mathbb{R}_{+}$$.

3. (c)

$$\varphi _{1}(\beta )<\beta$$ and $$\varphi _{2}(\alpha )<\alpha$$ for α, β large.

By (c), there exists $$C_{1}>0$$ such that $$\varphi _{1}(\beta )\leq \beta$$ and $$\varphi _{2}(\alpha )\leq \alpha$$, respectively, when $$\alpha ,\beta >C_{1}$$. Let $$C_{2}=\max \{ \max_{\beta \in [0,C_{1}]}\varphi _{1}(\beta ),\max_{\alpha \in [0,C _{1}]}\varphi _{2}(\alpha )\}$$, $$C=\max \{C_{1},C_{2}\}$$, and define $$T:[0,C]\times [0,C]\rightarrow \mathbb{R}^{2}_{+}$$ by
$$T(\alpha ,\beta )=\bigl(\varphi _{1}(\beta ),\varphi _{2}( \alpha )\bigr).$$
It is easy to see that $$T(\alpha ,\beta )\in [0,C]\times [0,C]$$ for all $$(\alpha ,\beta )\in [0,C]\times [0,C]$$. In fact,
$$\textstyle\begin{cases}\varphi _{2}(\alpha )\leq \alpha \leq C_{1},&\alpha >C_{1},\\ \varphi _{2}(\alpha )\leq \max_{\alpha \in [0,C_{1}]}\varphi _{2}( \alpha )\leq C_{2},&\alpha \leq C_{1}, \end{cases}$$
that is to say, $$\varphi _{2}(\alpha )\leq C$$. Similarly, we have $$\varphi _{1}(\beta )\leq C$$. Since T is continuous, using Brouwer fixed point theorem, there exists $$(\alpha _{+}\beta _{-})\in [0,C] \times [0,C]$$ such that
\begin{aligned}[b] \bigl(\varphi _{1}(\beta _{-}),\varphi _{2}(\alpha _{+})\bigr)=(\alpha _{+},\beta _{-}). \end{aligned}
(3.3)
It follows from $$\varphi _{i}>0$$, (3.3) that $$\alpha _{+},\beta _{-}>0$$. According to the definition, we have
$$\frac{\partial G_{u}}{\partial \alpha }(\alpha _{+},\beta _{-})= \frac{ \partial G_{u}}{\partial \beta }(\alpha _{+},\beta _{-})=0.$$

We next prove the uniqueness of$$(\alpha _{+},\beta _{-})$$.

Case 1.$$u\in \mathcal{M}$$.

Supposing that $$u\in \mathcal{M}$$, one has
\begin{aligned}[b] \nabla G_{u}(1,1)&=\biggl( \frac{\partial G_{u}}{\partial \alpha }(1,1),\frac{ \partial G_{u}}{\partial \beta }(1,1)\biggr) \\ &=\bigl(\bigl\langle J'\bigl(u^{+}+u^{-} \bigr),u ^{+}\bigr\rangle ,\bigl\langle J' \bigl(u^{+}+u^{-}\bigr),u^{-}\bigr\rangle \bigr)=(0,0), \end{aligned}
which shows that $$(1,1)$$ is a critical point of $$G_{u}$$. Now, we need to prove that $$(1,1)$$ is the unique critical point of $$G_{u}$$ with positive coordinates. Let $$(\alpha _{0},\beta _{0})$$ be a critical point of $$G_{u}$$ such that $$0<\alpha _{0}\leq \beta _{0}$$. So, one has that
\begin{aligned}[b] &\alpha _{0}^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}_{E}-\alpha _{0} \beta _{0} \int _{\mathbb{R} ^{3}} \int _{\mathbb{R}^{3}} \frac{u^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y \\ &\qquad {}+\alpha _{0}^{4} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{+}} \bigl\vert u ^{+} \bigr\vert ^{2}\,\mathrm{d}x+\alpha _{0}^{2}\beta _{0}^{2} \int _{\mathbb{R}^{3}} \phi ^{t}_{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad = \int _{\mathbb{R}^{3}}K(x)f\bigl( \alpha _{0}u^{+} \bigr) \bigl(\alpha _{0}u^{+}\bigr)\,\mathrm{d}x \end{aligned}
(3.4)
and
\begin{aligned}[b] &\beta _{0}^{2} \bigl\Vert u^{-} \bigr\Vert ^{2}_{E}-\alpha _{0}\beta _{0} \int _{\mathbb{R} ^{3}} \int _{\mathbb{R}^{3}} \frac{u^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y \\ &\qquad {}+\beta _{0}^{4} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{-}} \bigl\vert u ^{-} \bigr\vert ^{2}\,\mathrm{d}x+\alpha _{0}^{2}\beta _{0}^{2} \int _{\mathbb{R}^{3}} \phi ^{t}_{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad = \int _{\mathbb{R}^{3}}K(x)f\bigl( \beta _{0}u^{-} \bigr) \bigl(\beta _{0}u^{-}\bigr)\,\mathrm{d}x. \end{aligned}
(3.5)
Thanks to $$0<\alpha _{0}\leq \beta _{0}$$ and (3.5), we have that
\begin{aligned}[b] &\frac{ \Vert u^{-} \Vert ^{2}_{E}}{\beta _{0}^{2}}- \frac{1}{\beta ^{2}_{0}} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \int _{\mathbb{R}^{3}}\phi ^{t}_{u} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad \geq \int _{\mathbb{R}^{3}}K(x) \frac{f(\beta _{0}u^{-})}{(\beta _{0}u^{-})^{3}}\bigl(u^{-} \bigr)^{4}\,\mathrm{d}x. \end{aligned}
(3.6)
On the other hand, for $$u\in \mathcal{M}$$, we have
\begin{aligned}[b] & \bigl\Vert u^{-} \bigr\Vert ^{2}_{E}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u ^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \int _{\mathbb{R}^{3}}\phi ^{t}_{u} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad = \int _{\mathbb{R}^{3}}K(x)\frac{f(u^{-})}{(u^{-})^{3}}\bigl(u^{-} \bigr)^{4} \,\mathrm{d}x. \end{aligned}
(3.7)
Combining (3.6) with (3.7), one has that
\begin{aligned}[b] &\biggl(\frac{1}{\beta _{0}^{2}}-1\biggr) \bigl\Vert u^{-} \bigr\Vert ^{2}_{E}+\biggl(1- \frac{1}{\beta _{0} ^{2}}\biggr) \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad \geq \int _{\mathbb{R}^{3}}K(x)\biggl[ \frac{f(\beta _{0}u^{-})}{(\beta _{0}u^{-})^{3}}- \frac{f(u^{-})}{(u^{-})^{3}} \biggr]\bigl(u^{-}\bigr)^{4}\,\mathrm{d}x. \end{aligned}

By the definitions of $$u^{\pm }$$ and $$u^{\pm }\neq 0$$, we know that $$\int _{\mathbb{R}^{3}}\int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{|x-y|^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y<0$$. So, if $$\beta _{0}>1$$, the left-hand side of the above inequality is negative, which is absurd because the right-hand side is nonnegative by condition $$(f_{5})$$. Therefore, we obtain that $$0<\alpha _{0}\leq \beta _{0} \leq 1$$.

Similarly, by (3.4) and $$0<\alpha _{0}\leq \beta _{0}$$, one has that
\begin{aligned}[b] &\biggl(\frac{1}{\alpha _{0}^{2}}-1\biggr) \bigl\Vert u^{+} \bigr\Vert ^{2}_{E}+\biggl(1- \frac{1}{\alpha _{0} ^{2}}\biggr) \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad \leq \int _{\mathbb{R}^{3}}K(x)\biggl[ \frac{f(\alpha _{0}u^{+})}{(\alpha _{0}u^{+})^{3}}-\frac{f(u^{+})}{(u ^{+})^{3}} \biggr]\bigl(u^{+}\bigr)^{4}\,\mathrm{d}x. \end{aligned}

Therefore, by condition $$(f_{5})$$, we must have $$\alpha _{0}\geq 1$$. Consequently, $$\alpha _{0}=\beta _{0}=1$$, which indicates that $$(1,1)$$ is the unique critical point of $$G_{u}$$ with positive coordinates.

Case 2.$$u\notin \mathcal{M}$$.

Let $$u\in H$$, $$u^{\pm }\neq 0$$ and $$(\alpha _{1},\beta _{1})$$, $$(\alpha _{2}, \beta _{2})$$ be the critical points of $$G_{u}$$ with positive coordinates. In view of (i), one has that
$$u_{1}=\alpha _{1}u^{+}+\beta _{1}u^{-}\in \mathcal{M},\qquad u_{2}=\alpha _{2}u ^{+}+\beta _{2}u^{-}\in \mathcal{M}.$$
So,
$$u_{2}=\biggl(\frac{\alpha _{2}}{\alpha _{1}}\biggr)\alpha _{1}u^{+}+ \biggl(\frac{\beta _{2}}{ \beta _{1}}\biggr)\beta _{1}u^{-}=\biggl( \frac{\alpha _{2}}{\alpha _{1}}\biggr)u_{1}^{+}+\biggl(\frac{ \beta _{2}}{\beta _{1}} \biggr)u_{1}^{-}\in \mathcal{M}.$$
It follows from $$u_{1}\in H$$ with $$u_{1}^{\pm }\neq 0$$ that $$(\frac{\alpha _{2}}{\alpha _{1}},\frac{\beta _{2}}{\beta _{1}})$$ is a critical point of the map $$G_{u_{1}}$$ with positive coordinates. Thanks to $$u_{1}\in \mathcal{M}$$, one has that
$$\frac{\alpha _{2}}{\alpha _{1}}=\frac{\beta _{2}}{\beta _{1}}=1,$$

Hence, $$\alpha _{1}=\alpha _{2}$$, $$\beta _{1}=\beta _{2}$$.

Finally, we prove that the unique critical point is the unique maximum point of$$G_{u}$$.

In fact, using Lemma (2.2), we have that
$$G_{u}(\alpha ,\beta )\rightarrow -\infty \quad \text{as } \bigl\vert (\alpha , \beta ) \bigr\vert \rightarrow \infty .$$
Hence, the maximum point of $$G_{u}(\alpha ,\beta )$$ cannot be achieved on the boundary of $$\mathbb{R}^{2}_{+}$$. Without loss of generality, we may assume that $$(0,\overline{\beta })$$ is a maximum point of $$G_{u}$$. But, according to Lemma (2.3), it is obvious that
\begin{aligned}[b] &G_{u}(\alpha ,\overline{\beta })\\ &\quad = \frac{\alpha ^{2}}{2} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert u^{+}(x)-u^{+}(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ \frac{\overline{ \beta }^{2}}{2} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{ \vert u^{-}(x)-u ^{-}(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\qquad{}-\alpha \overline{ \beta } \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{+}(y)u^{-}(x)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ \frac{\alpha ^{2}}{2} \int _{\mathbb{R}^{3}}V(x) \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &\qquad{}+\frac{\overline{\beta }^{2}}{2} \int _{\mathbb{R}^{3}}V(x) \bigl\vert u^{-} \bigr\vert ^{2} \,\mathrm{d}x+\frac{\alpha ^{2}}{4} \int _{\mathbb{R}^{3}}\phi ^{t}_{u^{+}} \bigl\vert u ^{+} \bigr\vert ^{2}\,\mathrm{d}x+\frac{\overline{\beta }^{2}}{4} \int _{\mathbb{R} ^{3}}\phi ^{t}_{u^{-}} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x \\ &\qquad{}+\frac{\alpha \overline{ \beta }}{4} \int _{\mathbb{R}^{3}}\bigl(\phi ^{t}_{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2}+\phi ^{t} _{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\bigr)\,\mathrm{d}x- \int _{\mathbb{R}^{3}}K(x)F\bigl(\alpha u ^{+}\bigr)\, \mathrm{d}x- \int _{\mathbb{R}^{3}}K(x)F\bigl(\overline{\beta } u^{-}\bigr) \,\mathrm{d}x, \end{aligned}
which is an increasing function with respect to α, if α is small enough. Hence, the pair $$(0,\overline{\beta })$$ is not a maximum point of $$G_{u}$$ in $$\mathbb{R}_{+}^{2}$$. □

### Lemma 3.2

Suppose that$$(V,K)\in \mathcal{K}$$and$$(f_{1})$$$$(f_{5})$$hold. If$$u\in H$$with$$u^{\pm }\neq 0$$is such that$$g_{u}(1,1)\leq 0$$and$$h_{u}(1,1)\leq 0$$, where$$g_{u}(\alpha ,\beta )$$, $$h_{u}(\alpha ,\beta )$$are given by (3.1) and (3.2), then the unique pair$$(\alpha _{+},\beta _{-})$$obtained in Lemma (2.1) satisfies$$0<\alpha _{+}$$, $$\beta _{-}\leq 1$$.

### Proof

Suppose $$\alpha _{+}\geq \beta _{-}>0$$. Since $$\alpha_{+} u^{+}+\beta _{-} u^{-} \in \mathcal{M}$$, we have
\begin{aligned}[b] &\alpha _{+}^{2} \bigl\Vert u^{+} \bigr\Vert _{E}^{2}+ \alpha _{+}^{4} \int _{\mathbb{R}^{3}} \phi _{u^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x+\alpha _{+}^{4} \int _{\mathbb{R} ^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &\qquad{}-\alpha _{+}^{2} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{+}(y)u^{-}(x)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad \geq\alpha _{+}^{2} \bigl\Vert u^{+} \bigr\Vert _{E}^{2}+\alpha _{+}^{4} \int _{\mathbb{R}^{3}}\phi _{u ^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x+\alpha _{+}^{2}\beta _{-}^{2} \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &\qquad{}-\alpha _{+}\beta _{-} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u ^{-}(y)+u^{+}(y)u^{-}(x)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad = \int _{\mathbb{R}^{3}}K(x)f\bigl(\alpha _{+}u^{+} \bigr) \bigl(\alpha _{+}u^{+}\bigr) \,\mathrm{d}x. \end{aligned}
(3.8)
Thanks to $$g_{u}(1,1)\leq 0$$, we have that
\begin{aligned}[b] &\bigl\Vert u^{+} \bigr\Vert _{E}^{2}+ \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x+ \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x \\ &\qquad{}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u ^{-}(y)+u^{+}(y)u^{-}(x)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ & \quad \leq \int _{\mathbb{R}^{3}}K(x)f\bigl(u^{+}\bigr) \bigl(u^{+}\bigr)\,\mathrm{d}x. \end{aligned}
(3.9)
From (3.8) and (3.9), we have that
\begin{aligned}[b] &\biggl(\frac{1}{\alpha _{+}^{2}}-1\biggr) \bigl\Vert u^{+} \bigr\Vert ^{2}_{E}+\biggl(1- \frac{1}{\alpha _{+} ^{2}}\biggr) \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u^{-}(y)+u ^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad \geq \int _{\mathbb{R}^{3}}K(x)\biggl[ \frac{f(\alpha _{+}u^{+})}{(\alpha _{+}u^{+})^{3}}-\frac{f(u^{+})}{(u ^{+})^{3}} \biggr]\bigl(u^{+}\bigr)^{4}. \end{aligned}

By $$(f_{5})$$, we must have $$\alpha _{+}\leq 1$$. Then the proof is completed. □

Next, we consider the following minimization problem
$$m=\inf \bigl\{ J(u):u\in \mathcal{M}\bigr\} .$$
(3.10)

### Lemma 3.3

Suppose that$$(V,K)\in \mathcal{K}$$andfsatisfies$$(f_{1})$$$$(f _{5})$$. Then$$m>0$$can be achieved.

### Proof

Firstly, we prove $$m>0$$.

For every $$u\in \mathcal{M}$$, we have $$\langle J'(u),u\rangle =0$$.

First of all, suppose that $$(H_{2})$$ is true. It follows from $$(f_{1})$$ and $$(f_{3})$$ that for any given $$\varepsilon >0$$, there exists $$C_{\varepsilon }>0$$ such that
$$\bigl\vert f(t) \bigr\vert \leq \varepsilon \vert t \vert +C_{\varepsilon } \vert t \vert ^{2^{*}_{s}-1},\quad t\in \mathbb{R}.$$
So, we have that
\begin{aligned}[b] \Vert u \Vert ^{2}_{E}&\leq \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u(x)-u(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ \int _{\mathbb{R} ^{3}}V(x)u^{2}\,\mathrm{d}x+ \int _{\mathbb{R}^{3}}\phi ^{t}_{u}u^{2} \, \mathrm{d}x \\ &= \int _{\mathbb{R}^{3}}K(x)f(u)u\,\mathrm{d}x\leq \varepsilon \vert K/V \vert _{\infty } \Vert u \Vert ^{2}_{E}+C_{\varepsilon } \vert K \vert _{\infty }(S^{\prime})^{-\frac{2^{*}_{s}}{2}} \Vert u \Vert ^{2^{*}_{s}}_{E}. \end{aligned}
(3.11)

Choosing $$\varepsilon <1/|K/V|_{\infty }$$, there exists a constant $$\theta _{1}>0$$ such that $$\|u\|^{2}_{E}\geq \theta _{1}$$.

Next, suppose that $$(H_{3})$$ holds. By the discussion of [6], there is a constant $$C_{p}>0$$, for every given $$\varepsilon \in (0,C _{p})$$, there exists $$R>0$$ large enough leading to
\begin{aligned} \int _{ \vert x \vert \geq R}K(x) \vert u \vert ^{p}\,dx\leq \varepsilon \int _{ \vert x \vert \geq R}\bigl(Vu^{2}+u ^{2^{*}_{s}}\bigr)\,dx,\quad u \in E. \end{aligned}
(3.12)
From $$(f_{2})$$ and $$(f_{3})$$, there are $$C_{1},C_{2}>0$$ such that
$$\bigl\vert f(t) \bigr\vert \leq C_{1} \vert t \vert ^{p-1}+C_{2} \vert t \vert ^{{2^{*}_{s}-1}},\quad t\in \mathbb{R}.$$
So, by Hölder’s inequality and Soblev inequalities, we have that
\begin{aligned}[b] \Vert u \Vert ^{2}_{E}&\leq \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u(x)-u(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y+ \int _{\mathbb{R} ^{3}}V(x)u^{2}\,\mathrm{d}x+ \int _{\mathbb{R}^{3}}\phi ^{t}_{u}u^{2} \, \mathrm{d}x \\ &= \int _{\mathbb{R}^{3}}K(x)f(u)u\,\mathrm{d}x\leq C_{1} \varepsilon \Vert u \Vert ^{2}_{E}+C_{1} \bigl(S'\bigr)^{\frac{-2}{2^{*}_{s}}}\bigl(\varepsilon +C_{2} \vert K \vert _{\infty }\bigr) \Vert u \Vert ^{2^{*}_{s}}_{E} \\ &\quad{}+C_{2} \vert K \vert _{L^{2^{*}_{s}/(2^{*} _{s}-p)}B_{R}(0)}\bigl(S' \bigr)^{\frac{-p}{2}} \Vert u \Vert ^{p}_{E}. \end{aligned}
(3.13)

Choosing $$\varepsilon <1/C_{1}$$, there exists a constant $$\theta _{2}>0$$ such that $$\|u\|^{2}_{E}\geq \theta _{2}$$.

Consequently, we conclude that $$\|u\|^{2}_{E}\geq \theta$$ for any $$u\in \mathcal{M}$$, where $$\theta =\max \{\theta _{1},\theta _{2}\}>0$$.

On the other hand, by condition $$(f_{5})$$, we have
$$H(t):=f(t)t-4F(t)\geq 0,\quad t\in \mathbb{R},$$
(3.14)
and $$H(t)$$ is increasing when $$t>0$$ and decreasing when $$t<0$$. Hence,
\begin{aligned}[b] J(u)&=J(u)-\frac{1}{4}\bigl\langle J'(u),u\bigr\rangle \\ &=\frac{1}{4} \Vert u \Vert _{E}^{2}+ \frac{1}{4} \int _{\mathbb{R}^{3}}K(x) \bigl(f(u)u-4F(u)\bigr)\,\mathrm{d}x \\ &\geq \frac{1}{4} \Vert u \Vert _{E}^{2} \\ &\geq \frac{1}{4}\theta . \end{aligned}

This implies that $$m\geq \frac{\theta }{4}>0$$.

In the following, we prove that m is achieved.

Let $$\{u_{n}\}\subset \mathcal{M}$$ such that $$J(u_{n})\rightarrow m$$. Then $$\|u_{n}\|_{E}\leq C$$. Hence, we may assume that there exists $$u\in E$$ such that $$u_{n}\rightharpoonup u$$, $$u_{n}^{\pm }\rightharpoonup u^{\pm }$$ in E.

By Proposition (2.3), we know that
$$\int _{\mathbb{R}^{3}}K(x)F(u_{n})\,dx\rightarrow \int _{\mathbb{R}^{3}}K(x)F(u)\,dx.$$

Hence $$\{\int _{\mathbb{R}^{3}}K(x)F(u_{n})\,dx\}$$ is bounded.

By definition of J, we get
$$\frac{1}{4} \int _{\mathbb{R}^{3}}\phi _{u_{n}}^{t}u_{n}^{2} \,\mathrm{d}x+ \frac{1}{2} \Vert u_{n} \Vert _{E}^{2}=J(u_{n})+ \int _{\mathbb{R}^{3}}KF(u_{n}) \,\mathrm{d}x,$$
which implies that $$\{u_{n}\}$$ is bounded in H.
Hence, by the uniqueness of the convergence, we get $$u_{n}\rightharpoonup u$$ and $$u_{n}^{\pm }\rightharpoonup u^{\pm }$$ in H. Thanks to $$u_{n}\in \mathcal{M}$$, we have that
\begin{aligned}[b] \bigl\langle J'(u_{n}),u_{n}^{\pm } \bigr\rangle &= \bigl\Vert u_{n}^{\pm } \bigr\Vert _{E}^{2}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u_{n}^{+}(x)u_{n}^{-}(y)+u _{n}^{-}(x)u_{n}^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\quad {}+ \int _{\mathbb{R}^{3}}\phi ^{t}_{u_{n}} \bigl\vert u_{n}^{\pm } \bigr\vert ^{2}\,\mathrm{d}x- \int _{\mathbb{R}^{3}}Kf\bigl(u_{n}^{\pm } \bigr)u_{n}^{\pm }\,\mathrm{d}x\\ &=0. \end{aligned}
(3.15)
Together (3.15) with Proposition 2.3, we get
\begin{aligned}[b] 0&< \theta\leq \bigl\Vert u_{n}^{\pm } \bigr\Vert _{E}^{2}\\ &\leq \bigl\Vert u_{n}^{\pm } \bigr\Vert _{E}^{2}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u_{n}^{+}(x)u_{n}^{-}(y)+u _{n}^{-}(x)u_{n}^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \int _{\mathbb{R}^{3}}\phi ^{t}_{u_{n}} \bigl\vert u_{n}^{\pm } \bigr\vert ^{2}\,\mathrm{d}x \\ &= \int _{\mathbb{R}^{3}}K(x)f\bigl(u_{n}^{\pm } \bigr)u_{n}^{\pm }\,\mathrm{d}x= \int _{\mathbb{R}^{3}}K(x)f\bigl(u^{\pm }\bigr)u^{\pm }\, \mathrm{d}x+o(1). \end{aligned}
(3.16)

Thus $$u^{\pm }\neq 0$$.

On the other hand, combining (1.4) with the Hardy–Littlewood–Sobolev inequality [31], we have that
$$\lim_{n\rightarrow \infty }\inf \int _{\mathbb{R}^{3}}\phi ^{t}_{u_{n}} \bigl\vert u _{n}^{\pm } \bigr\vert ^{2}\,\mathrm{d}x= \int _{\mathbb{R}^{3}}\phi ^{t}_{u} \bigl\vert u^{ \pm } \bigr\vert ^{2}\,\mathrm{d}x.$$
Then, by the weak lower semicontinuity of norm and Fatou’s lemma, we have
\begin{aligned}[b] & \bigl\Vert u^{\pm } \bigr\Vert _{E}^{2}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u ^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \int _{\mathbb{R}^{3}}\phi ^{t}_{u} \bigl\vert u^{\pm } \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad \leq \lim_{n\rightarrow \infty }\inf \biggl[ \bigl\Vert u_{n}^{\pm } \bigr\Vert _{E}^{2}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u_{n}^{+}(x)u_{n}^{-}(y)+u _{n}^{-}(x)u_{n}^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y \\ &\qquad{}+ \int _{\mathbb{R}^{3}}\phi ^{t}_{u_{n}} \bigl\vert u_{n}^{\pm } \bigr\vert ^{2}\,\mathrm{d}x\biggr]. \end{aligned}
(3.17)
Then, by (3.16) and (3.17), we have that
\begin{aligned}[b] & \bigl\Vert u^{\pm } \bigr\Vert _{E}^{2}- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u ^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y + \int _{\mathbb{R}^{3}}\phi ^{t}_{u} \bigl\vert u^{\pm } \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad \leq \int _{\mathbb{R}^{3}}K(x)f\bigl(u^{\pm }\bigr)u^{\pm }\, \mathrm{d}x. \end{aligned}
(3.18)
According to Lemmas 3.1 and 3.2, there exists $$(\overline{\alpha },\overline{\beta })\in (0,1]\times (0,1]$$ such that
$$\overline{u}:=\overline{\alpha }u^{+}+\overline{\beta }u^{-}\in \mathcal{M}.$$
Thanks to (3.14), we have that
\begin{aligned}[b] m&\leq J(\overline{u})-\frac{1}{4}\bigl\langle J'(\overline{u}), \overline{u}\bigr\rangle \\ &=\frac{1}{4} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{ \vert \overline{u}(x)-\overline{u}(y) \vert ^{2}}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y+ \frac{1}{4} \int _{\mathbb{R}^{3}}V(x) \vert \overline{u} \vert ^{2} \, \mathrm{d}x+\frac{1}{4} \int _{\mathbb{R}^{3}}K(x) \bigl(f(\overline{u}) \overline{u}-4F( \overline{u})\bigr)\,\mathrm{d}x \\ &=\frac{\overline{\alpha }^{2}}{4} \bigl\Vert u^{+} \bigr\Vert _{E}^{2}- \frac{\overline{\alpha }\overline{\beta }}{2} \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y+ \frac{\overline{\beta }^{2}}{4} \bigl\Vert u^{-} \bigr\Vert _{E}^{2} \\ &\quad{}+ \frac{1}{4} \int _{\mathbb{R}^{3}}K(x) \bigl(f\bigl(\overline{\alpha }u^{+} \bigr)\overline{ \alpha }u^{+}-4F\bigl(\overline{\alpha }u^{+}\bigr)\bigr)\,\mathrm{d}x +\frac{1}{4} \int _{\mathbb{R}^{3}}K(x) \bigl(f\bigl(\overline{\beta }u^{-} \bigr)\overline{\beta }u ^{-}-4F\bigl(\overline{\beta }u^{-} \bigr)\bigr)\,\mathrm{d}x \\ &\leq \frac{1}{4} \Vert u \Vert ^{2}_{E}+ \frac{1}{4} \int _{\mathbb{R}^{3}}K(x) \bigl(f(u)u-4F(u)\bigr)\,\mathrm{d}x \\ &\leq \lim_{n\rightarrow \infty }\biggl[J(u_{n})-\frac{1}{4} \bigl\langle J'(u _{n}),u_{n}\bigr\rangle \biggr]=m. \end{aligned}

Consequently, $$\overline{\alpha }=\overline{\beta }=1$$. Thus $$\overline{u}=u$$ and $$J(u)=m$$. □

## 4 Proof of the main result

In this section, we will prove Theorem 1.1. In fact, we just prove that the minimizer u for (3.10) is indeed a sign-changing solution of system (1.1).

### Proof

Since $$u\in \mathcal{M}$$, we have $$\langle J'(u),u^{+}\rangle =0= \langle J'(u),u^{-}\rangle$$. By Lemma 3.1, for $$(\alpha , \beta )\in \mathbb{R}^{2}_{+}$$ and $$(\alpha ,\beta )\neq (1,1)$$, we have
$$J\bigl(\alpha u^{+}+\beta u^{-}\bigr)< J \bigl(u^{+}+u^{-}\bigr)=m.$$
(4.1)

Set $$\xi _{1}=|u^{+}|_{2_{s}^{*}}$$, $$\xi _{2}=|u^{-}|_{2_{s}^{*}}$$ and $$\xi =\min \{\xi _{1},\xi _{2}\}$$. We denote the imbedding constant of $$H\hookrightarrow L^{2^{*}_{s}}(\mathbb{R}^{3})$$, that is, $$|u|_{2_{s}^{*}}\leq \widetilde{S}\|u\|$$, $$u\in H$$.

If $$J'(u)\neq 0$$, then there exist $$r, \mu >0$$ such that
$$\bigl\Vert J'(v) \bigr\Vert \geq \mu ,\qquad \Vert v-u \Vert \leq r.$$
(4.2)
Choose $$\delta \in (0,\min \{\xi /(2\widetilde{S}),r/3\})$$ and $$\sigma \in (0,\min \{1/2,\delta /(\sqrt{2}\|u\|)\})$$. Let $$D=(1-\sigma , 1+\sigma )\times (1-\sigma , 1+\sigma )$$ and $$\psi (\alpha ,\beta ):=\alpha u^{+}+\beta u^{-}$$, $$(\alpha ,\beta ) \in D$$. In view of (4.1), it is easy to see that
$$\overline{m}:=\max_{\partial D}J\circ \psi < m.$$
(4.3)
Let $$0<\varepsilon <\min \{(m-\overline{m})/2,\mu \delta /8\}$$ and $$S_{\delta }:=\{v\in H,\|v-u\|<\delta \}$$, according to quantitative deformation lemma [52], there exists a deformation $$\eta \in ([0,1] \times H, H)$$ satisfying
1. (a)

$$\eta (1,u)=u$$ if $$u\notin J^{-1}([m-2\varepsilon , m+2\varepsilon ])\cap S_{2\delta }$$;

2. (b)

$$\eta (1,J^{m+\varepsilon }\cap S_{\delta })\subset J^{m-\varepsilon }$$;

3. (c)

$$\|\eta (1,u)-u\|\leq \delta$$ for all $$u\in H$$, where $$J^{m+ \varepsilon }:=\{x|J(x)\leq m+\varepsilon \}$$.

Firstly, we need to prove that
$$\max_{(\alpha ,\beta )\in \overline{D}}J\bigl(\eta \bigl(1,\psi (\alpha , \beta )\bigr)\bigr)< m.$$
(4.4)

In fact, it is follows from Lemma 3.1 that $$J(\psi (\alpha , \beta ))\leq m< m+\varepsilon$$. That is, $$\psi (\alpha ,\beta )\in J ^{m+\varepsilon }$$.

On the other hand, we have
\begin{aligned}[b] \bigl\Vert \psi (\alpha ,\beta )-u \bigr\Vert ^{2}&\leq 2\bigl((\alpha -1)^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}+( \beta -1)^{2} \bigl\Vert u^{-} \bigr\Vert ^{2}\bigr) \\ &\leq 2\sigma \Vert u \Vert ^{2} \\ &< \delta ^{2}, \end{aligned}
which shows that $$\psi (\alpha ,\beta )\in S_{\delta }$$ for all $$(\alpha ,\beta )\in \overline{D}$$.

Therefore, according to (b), we have $$J(\eta (1,\psi (\alpha ,\beta )))< m- \varepsilon$$. Hence (4.4) holds.

In the following, we prove that
$$\eta \bigl(1,\psi (D)\bigr)\cap \mathcal{M}\neq \emptyset ,$$
(4.5)
which is a contradiction to the definition of m.
Let $$\gamma (\alpha ,\beta ):=\eta (1,\psi (\alpha ,\beta ))$$ and
\begin{aligned}& \begin{aligned}[b] \varPhi _{0}(\alpha ,\beta )&:=\bigl(\bigl\langle J'\bigl(\psi (\alpha ,\beta )\bigr),u^{+} \bigr\rangle ,\bigl\langle J'\bigl(\psi (\alpha ,\beta ) \bigr),u^{-}\bigr\rangle \bigr) \\ &=\bigl(\bigl\langle J'\bigl(\alpha u^{+}+\beta u^{-}\bigr),u^{+}\bigr\rangle ,\bigl\langle J' \bigl(\alpha u^{+}+ \beta u^{-}\bigr),u^{+}\bigr\rangle \bigr), \end{aligned} \\& \begin{aligned}[b] \varPhi _{1}(\alpha ,\beta )&:=\biggl( \frac{1}{\alpha }\bigl\langle J'\bigl(\gamma ( \alpha ,\beta ) \bigr),\gamma ^{+}(\alpha ,\beta )\bigr\rangle ,\frac{1}{\beta } \bigl\langle J'\bigl(\gamma (\alpha ,\beta )\bigr),\gamma ^{-}(\alpha ,\beta )\bigr\rangle \biggr). \end{aligned} \end{aligned}
Thanks to $$(f_{5})$$, we have that $$f'(u)u^{2}\geq 3f(u)u$$. Then, by direct calculation, we get
\begin{aligned}& \begin{aligned}[b] A_{11}&:=\frac{\partial \langle J'(\alpha u^{+}+\beta u^{-}),u^{+} \rangle }{\partial \alpha }\bigg|_{(1,1)} \\ &= \bigl\Vert u^{+} \bigr\Vert ^{2}_{E}+3 \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x+ \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x- \int _{\mathbb{R}^{3}}K(x)f'\bigl(u^{+}\bigr) \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &=-2 \bigl\Vert u ^{+} \bigr\Vert ^{2}_{E}+3 \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u ^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y-2 \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad{}- \int _{\mathbb{R}^{3}}K(x) \bigl(f'\bigl(u^{+} \bigr) \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x-3f \bigl(u^{+}\bigr)u ^{+}\bigr)\,\mathrm{d}x \\ &< \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u ^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y-2 \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2}\,\mathrm{d}x< 0, \end{aligned} \\& \begin{aligned}[b] A_{12}&:=\frac{\partial \langle J'(\alpha u^{+}+\beta u^{-}),u^{+} \rangle }{\partial \beta }\bigg|_{(1,1)} \\ &=- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y+2 \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{+} \bigr\vert ^{2} \,\mathrm{d}x, \end{aligned} \\& \begin{aligned}[b] A_{21}&:=\frac{\partial \langle J'(\alpha u^{+}+\beta u^{-}),u^{-} \rangle }{\partial \alpha }\bigg|_{(1,1)} \\ &=- \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{u^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x \,\mathrm{d}y+2 \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{-} \bigr\vert ^{2} \,\mathrm{d}x, \end{aligned} \end{aligned}
and
\begin{aligned}[b] A_{22}&:=\frac{\partial \langle J'(\alpha u^{+}+\beta u^{-}),u^{-} \rangle }{\partial \beta }\bigg|_{(1,1)} \\ &= \bigl\Vert u^{-} \bigr\Vert ^{2}_{E}+3 \int _{\mathbb{R}^{3}}\phi _{u^{-}}^{t} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x+ \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x- \int _{\mathbb{R}^{3}}K(x)f'\bigl(u^{-}\bigr) \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x \\ &=-2 \bigl\Vert u ^{-} \bigr\Vert ^{2}_{E}+3 \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u^{+}(x)u ^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y-2 \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad{}- \int _{\mathbb{R}^{3}}K(x) \bigl(f'\bigl(u^{-} \bigr) \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x-3f \bigl(u^{-}\bigr)u ^{-}\bigr)\,\mathrm{d}x \\ &< \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{u ^{+}(x)u^{-}(y)+u^{-}(x)u^{+}(y)}{ \vert x-y \vert ^{3+2s}}\,\mathrm{d}x\,\mathrm{d}y-2 \int _{\mathbb{R}^{3}}\phi _{u^{+}}^{t} \bigl\vert u^{-} \bigr\vert ^{2}\,\mathrm{d}x< 0. \end{aligned}
Thus
which implies that $$\operatorname{deg}(\varPhi _{0},D,0)=1$$.

So, combining (4.3) with (a), we obtain $$\psi =\gamma$$ on ∂D. Consequently, we obtain $$\operatorname{deg}(\varPhi _{1},D,0)= \operatorname{deg}(\varPhi _{0},D,0)=1$$. Therefore, $$\varPhi _{1}(\alpha _{0},\beta _{0})=0$$ for some $$(\alpha _{0},\beta _{0})\in D$$, so that $$\eta (1, \psi (\alpha _{0},\beta _{0}))=\gamma (\alpha _{0},\beta _{0})\in \mathcal{M}$$, which contradicts (4.4).

From the above discussion, we conclude that u is a sign-changing solution for system (1.1). □

## Notes

### Acknowledgements

The authors are thankful to the honorable reviewers and editors for their valuable reviewing of the manuscript.

### Availability of data and materials

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

### Authors’ contributions

All the authors have the same contribution. All authors read and approved the final manuscript.

### Funding

The paper is supported by the Natural Science Foundation of China (Grant no. 11561043).

### Competing interests

The authors declare that they have no competing interests.

## References

1. 1.
Alves, C.O., Miyagaki, O.H.: Existence and concentration of solutions for a class of fractional elliptic equation in $$\mathbb{R} ^{N}$$ via penalization method. Calc. Var. Partial Differ. Equ. 55, Article 47, 19 pp. (2016)
2. 2.
Alves, C.O., Souto, M.A.: Existence of least energy nodal solution for a Schrödinger–Poisson system in bounded domains. Z. Angew. Math. Phys. 65, 1153–1166 (2014)
3. 3.
Alves, C.O., Souto, M.A.S.: Existence of solutions for a class of nonlinear Schrödinger equations with potential vanishing at infinity. J. Differ. Equ. 254, 1977–1991 (2013)
4. 4.
Ambrosetti, A.: On Schrödinger–Poisson system. Milan J. Math. 76, 257–274 (2008)
5. 5.
Ambrosio, V.: Ground states for superlinear fractional Schrödinger equations in $$\mathbb{R}^{N}$$. Ann. Acad. Sci. Fenn., Math. 41, 745–756 (2016)
6. 6.
Ambrosio, V., Isernia, T.: Sign-changing solutions for a class of Schrödinger equations with vanishing potentials. arXiv:1609.09003v3
7. 7.
Azzollini, A., d’Avenia, P., Pomponio, A.: On the Schrödinger–Maxwell equations under the effect of a general nonlinear term. Ann. Inst. Henri Poincaré, Anal. Non Linéaire 27, 779–791 (2010)
8. 8.
Bartsch, T., Weth, T.: Three nodal solutions of singularly perturbed elliptic equations on domains without topology. Ann. Inst. Henri Poincaré, Anal. Non Linéaire 22, 259–281 (2005)
9. 9.
Batista, A.M., Furtado, M.F.: Positive and nodal solutions for a nonlinear Schrödinger–Poisson system with sign-changing potentials. Nonlinear Anal., Real World Appl. 39, 142–156 (2018)
10. 10.
Benci, V., Fortunato, D.: An eigenvalue problem for the Schrödinger–Maxwell equations. Topol. Methods Nonlinear Anal. 11, 283–293 (1998)
11. 11.
Benci, V., Fortunato, D.: Solitary waves of nonlinear Klein–Gordon equation coupled with Maxwell equations. Rev. Math. Phys. 14, 409–420 (2002)
12. 12.
Caffarelli, L., Silvestre, L.: An extension problem related to the fractional Laplacian. Commun. Partial Differ. Equ. 32, 1245–1260 (2007)
13. 13.
Cerami, G., Vaira, G.: Positive solutions for some non-autonomous Schrödinger–Poisson systems. J. Differ. Equ. 248, 521–543 (2010)
14. 14.
Chang, X., Wang, Z.: Nodal and multiple solutions of nonlinear problems involving the fractional Laplacian. J. Differ. Equ. 256(8), 2965–2992 (2014)
15. 15.
Chen, S., Tang, X.: Ground state sign-changing solutions for a class of Schrödinger–Poisson type problems in $$\mathbb{R}^{3}$$. Z. Angew. Math. Phys. 67, Article 102, 18 pp. (2016)
16. 16.
D’Aprile, T., Wei, J.: Standing waves in the Maxwell–Schrödinger equation and an optimal configuration problem. Calc. Var. Partial Differ. Equ. 25, 105–137 (2006)
17. 17.
Deng, Y., Shuai, W.: Sign-changing solutions for non-local elliptic equations involving the fractional Laplacian. Adv. Differ. Equ. 1–2, 109–134 (2018)
18. 18.
Figueiredo, G.M., Siciliano, G.: A multiplicity result via Ljusternick–Schnirelmann category and Morse theory for a fractional Schrödinger equation in $$\mathbb{R}^{N}$$. Nonlinear Differ. Equ. Appl. 23, 1–22 (2016)
19. 19.
Giacomoni, J., Mukherjee, T., Sreenadh, K.: Positive solutions of fractional elliptic equation with critical and singular nonlinearity. Adv. Nonlinear Anal. 6, 327–354 (2017)
20. 20.
Guo, L.: Sign-changing solutions for fractional Schrödinger–Poisson system in $$\mathbb{R}^{3}$$. Appl. Anal. (2018).
21. 21.
He, X., Zou, W.: Existence and concentration of ground states for Schrödinger–Poisson equations with critical growth. J. Math. Phys. 53, 023702 (2012) 19 pp.
22. 22.
He, X., Zou, W.: Existence and concentration result for the fractional Schrödinger equations with critical nonlinearities. Calc. Var. Partial Differ. Equ. 55, Article 91, 39 pp. (2016)
23. 23.
He, X., Zou, W.: Multiplicity of concentrating positive solutions for Schrödinger–Poisson equations with critical growth. Nonlinear Anal. 170, 142–170 (2018)
24. 24.
He, Y., Li, G.: Standing waves for a class of Schrödinger–Poisson equations in $$\mathbb{R}^{3}$$ involving critical Sobolev exponents. Ann. Acad. Sci. Fenn., Math. 40, 729–766 (2015)
25. 25.
Ianni, I.: Sign-changing radial solutions for the Schrödinger–Poisson–Slater problem. Topol. Methods Nonlinear Anal. 41, 365–385 (2013)
26. 26.
Kim, S., Seok, J.: On nodal solutions of the nonlinear Schrödinger–Poisson equations. Commun. Contemp. Math. 14, 1250041, 16 pp. (2012)
27. 27.
Laskin, N.: Fractional quantum mechanics and Lévy path integrals. Phys. Lett. A 268, 298–305 (2000)
28. 28.
Laskin, N.: Fractional Schrödinger equations. Phys. Rev. 66, 56–108 (2002)
29. 29.
Li, G.B., Peng, S.J., Yan, S.S.: Infinitely many positive solutions for the nonlinear Schrödinger–Poisson system. Commun. Contemp. Math. 12, 1069–1092 (2010)
30. 30.
Liang, Z., Xu, J., Zhu, X.: Revisit to sign-changing solutions for the nonlinear Schrödinger–Poisson system in $$\mathbb{R}^{3}$$. J. Math. Anal. Appl. 435, 783–799 (2016)
31. 31.
Lieb, E.H., Loss, M.: Analysis, 2nd edn. Graduate Studies in Mathematics. Am. Math. Soc., Providence (2001)
32. 32.
Liu, Z., Wang, Z., Zhang, J.: Infinitely many sign-changing solutions for the nonlinear Schrödinger–Poisson system. Ann. Mat. Pura Appl. 4, 775–794 (2016)
33. 33.
Liu, Z., Zhang, J.: Multiplicity and concentration of positive solutions for the fractional Schrödinger–Poisson systems with critical growth. ESAIM Control Optim. Calc. Var. 23, 1515–1542 (2017)
34. 34.
Luo, H., Tang, X.: Ground state and multiple solutions for the fractional Schrödinger–Poisson system with critical Sobolev exponent. Nonlinear Anal., Real World Appl. 42, 24–52 (2018)
35. 35.
Molica Bisci, G., Rădulescu, V.D.: Ground state solutions of scalar field fractional for Schrödinger equations. Calc. Var. Partial Differ. Equ. 54, 2985–3008 (2015)
36. 36.
Molica Bisci, G., Rădulescu, V.D., Servadei, R.: Variational Methods for Nonlocal Fractional Problems. Encyclopedia of Mathematics and Its Applications, vol. 162. Cambridge University Press, Cambridge (2016)
37. 37.
Ruiz, D.: The Schrödinger–Poisson equation under the effect of a nonlinear local term. J. Funct. Anal. 237, 655–674 (2006)
38. 38.
Ruiz, D.: On the Schrödinger–Poisson–Slater system: behavior of minimizers, radial and nonradial cases. Arch. Ration. Mech. Anal. 198, 349–368 (2010)
39. 39.
Secchi, S.: Ground state solutions for nonlinear fractional Schrödinger equations in $$\mathbb{R}^{N}$$. J. Math. Phys. 54, 031501 (2013)
40. 40.
Shang, X., Zhang, J.: Concentrating solutions of nonlinear fractional Schrödinger equation with potentials. J. Differ. Equ. 258, 1106–1128 (2015)
41. 41.
Shen, L., Yao, X.: Least energy solutions for a class of fractional Schrödinger equation systems. J. Math. Phys. 59, 081501 (2018)
42. 42.
Shuai, W., Wang, Q.: Existence and asymptotic behavior of sign-changing solutions for the nonlinear Schrödinger–Poisson system in $$\mathbb{R}^{3}$$. Z. Angew. Math. Phys. 66, 3267–3282 (2015)
43. 43.
Sun, J., Ma, S.: Ground state solutions for some Schrödinger–Poisson systems with periodic potentials. J. Differ. Equ. 260, 2119–2149 (2016)
44. 44.
Teng, K.M.: Existence of ground state solutions for the nonlinear fractional Schrödinger–Poisson systems with critical Sobolev exponent. J. Differ. Equ. 261, 3061–3106 (2016); Corrigendum, J. Differ. Equ. 262:3132–3138, 2017
45. 45.
Torres Ledesma, C.E.: Multiplicity result for non-homogeneous fractional Schrödinger–Kirchhoff-type equations in $$\mathbb{R} ^{n}$$. Adv. Nonlinear Anal. 7, 247–257 (2018)
46. 46.
Wang, D.B., Guo, M., Guan, W.: Existence of solutions for fractional Schrödinger equation with asymptotically periodic terms. J. Nonlinear Sci. Appl. 10, 625–636 (2017)
47. 47.
Wang, D.B., Xie, H.F., Guan, W.: Existence of positive ground state solutions for a class of asymptotically periodic Schrödinger–Poisson systems. Electron. J. Differ. Equ. 2017, 231, 1–13 (2017)
48. 48.
Wang, D.B., Xie, H.F., Guan, W.: Ground states for a class of asymptotically periodic Schrödinger–Poisson systems with critical growth. Electron. J. Qual. Theory Differ. Equ. 2017, 97, 1–18 (2017)
49. 49.
Wang, J., Tian, L., Xu, J., Zhang, F.: Existence and concentration of positive solutions for semilinear Schrödinger–Poisson systems in $$\mathbb{R}^{3}$$. Calc. Var. Partial Differ. Equ. 48, 243–273 (2013)
50. 50.
Wang, Z., Zhou, H.: Sign-changing solutions for the nonlinear Schrödinger–Poisson system in $$\mathbb{R}^{3}$$. Calc. Var. Partial Differ. Equ. 52, 927–943 (2015)
51. 51.
Wang, Z., Zhou, H.: Radial sign-changing solution for fractional Schrödinger equation. Discrete Contin. Dyn. Syst. 36, 499–508 (2016)
52. 52.
Willem, M.: Minimax Theorems. Birkhäuser, Boston (1996)
53. 53.
Xiang, M., Wang, F.: Fractional Schrödinger–Poisson–Kirchhoff type systems involving critical nonlinearities. Nonlinear Anal. 164, 1–26 (2017)
54. 54.
Xiang, M., Zhang, B., Rădulescu, V.D.: Multiplicity of solutions for a class of quasilinear Kirchhoff system involving the fractional p-Laplacian. Nonlinearity 29, 3186–3205 (2016)
55. 55.
Xiang, M., Zhang, B., Rădulescu, V.D.: Existence of solutions for perturbed fractional p-Laplacian equations. J. Differ. Equ. 260, 1392–1413 (2016)
56. 56.
Yang, Z., Yu, Y., Zhao, F.: Concentration behavior of ground state solutions for a fractional Schrödinger–Poisson system involving critical exponent. Commun. Contemp. Math. 1850027, 46 pp. (2018, in press).
57. 57.
Yu, Y., Zhao, F., Zhao, L.: The concentration behavior of ground state solutions for a fractional Schrödinger–Poisson system. Calc. Var. Partial Differ. Equ. 56, Article 116, 25 pp. (2017)
58. 58.
Zhang, J., do Ó, J.M., Squassina, M.: Fractional Schrödinger–Poisson systems with a general subcritical or critical nonlinearity. Adv. Nonlinear Stud. 16, 15–30 (2016)
59. 59.
Zhang, X., Zhang, B.L., Repovš, D.: Existence and symmetry of solutions for critical fractional Schrödinger equations with bounded potentials. Nonlinear Anal. 142, 48–68 (2016)
60. 60.
Zhao, L., Liu, H., Zhao, F.: Existence and concentration of solutions for the Schrödinger–Poisson equations with steep well potential. J. Differ. Equ. 255, 1–23 (2013)
61. 61.
Zhong, X., Tang, C.: Ground state sign-changing solutions for a Schrödinger–Poisson system with a critical nonlinearity in $$\mathbb{R}^{3}$$. Nonlinear Anal., Real World Appl. 39, 166–184 (2018)