1 Introduction and main results

Our goal of this paper is to consider the existence of nodal solution and ground state solution of the following Kirchhoff–Schrödinger–Poisson system:

$$ \textstyle\begin{cases} -(a+ b\int _{\mathbb{R}^{3}} \vert \nabla u \vert ^{2}\,dx)\Delta u +V(x)u+\lambda \phi u=f(x,u),&x\in \mathbb{R}^{3}, \\ -\Delta \phi =u^{2},&x\in \mathbb{R}^{3}, \end{cases} $$

where \(V(x)\) is a smooth function and \(b>0, \lambda >0\). When \(a=1\), \(b=0\), Kirchhoff–Schrödinger–Poisson equation reduces to the undermentioned Schrödinger–Poisson system

$$ \textstyle\begin{cases} -\Delta u+V(x)u+\lambda \phi u=f(x,u),&x\in \mathbb{R}^{3}, \\ -\Delta \phi =u^{2},&x\in \mathbb{R}^{3}. \end{cases} $$
(1.1)

System (1.1) is derived from the time-varying Schrödinger equation, which describes the interaction of quantum (non-relativistic) particles with the electromagnetic field generated by motion. On the other hand, recently a great attention has been given to the so-called Kirchhoff equations

$$ -\biggl(a+b \int _{\varOmega } \vert \nabla u \vert ^{2}\,dx\biggr) \Delta u=f(x,u), $$
(1.2)

where \(\varOmega \subset \mathbb{R}^{N}\) is a bounded domain or \(\varOmega =\mathbb{R}^{N}\), \(a>0\), \(b>0\) and u satisfies some boundary conditions. Problem (1.2) is related to the stationary analogue of the Kirchhoff–Schrödinger type equation

$$ u_{tt}-\biggl(a+b \int _{\varOmega } \vert \nabla u \vert ^{2}\,dx\biggr) \Delta u=f(x,u), $$
(1.3)

which was introduced by Kirchhoff [6] as a generalization of the well-known D’Alembert wave equation

$$ \rho \frac{\partial ^{2}u}{\partial t^{2}}- \biggl(\frac{p_{0}}{h}+ \frac{E}{2L} \int ^{L}_{0} \biggl\vert \frac{\partial u}{\partial x} \biggr\vert ^{2}\,dx \biggr)=f(x,u), $$
(1.4)

for free vibration of elastic strings. The Kirchhoff’s model takes into account the length variation of the string produced by the transverse vibration, so the nonlocal term appears. For more mathematical and physical background on Schrödinger–Poisson systems or Kirchhoff-type problems, we refer the readers to [1, 2, 13] and the references therein.

The appearance of nonlocal term not only makes it playing an important role in many physical applications, but also brings some difficulties and challenges in mathematical analysis. This fact makes the study of Kirchhoff–Schrödinger–Poisson system or similar problems particularly interesting. A lot of interesting results on the existence of nonlocal problems were obtained recently in, for example, [4, 5, 79, 11, 1317, 21, 25, 2729] and the cited references. We especially refer to the paper [10] for the existence of ground state positive solutions of Kirchhoff–Schrödinger-type equations with singular exponential nonlinearities in \(\mathbb{R}^{N}\).

In the past few years, many researchers began to search for nodal solutions to Kirchhoff–Schrödinger-type equations or similar problems and got some interesting results. Zhong and Tang [28] considered the following subcritical Schrödinger–Poisson system:

$$ \textstyle\begin{cases} -\Delta u+V(x)u+ k\phi u= \vert u \vert ^{2}u+\lambda f(u),&x\in \mathbb{R}^{3}, \\ -\Delta \phi =u^{2},&x\in \mathbb{R}^{3}, \end{cases} $$
(1.5)

where the nonlinearity \(f(u)\) satisfies 3-linear growth condition at infinity and linear growth at zero. With the help of the nodal Nehari manifold, they studied the existence and asymptotic behavior of least energy nodal solution to system (1.5).

Wang [18] studied the existence of a least energy sign-changing solution for the following Kirchhoff-type equation:

$$ \textstyle\begin{cases} -(a+ b\int _{\varOmega } \vert \nabla u \vert ^{2}\,dx)\Delta u= \vert u \vert ^{4}u+\lambda f(x,u), & x\in \varOmega , \\ u=0, & x\in \partial \varOmega , \end{cases} $$
(1.6)

where \(\varOmega \subset \mathbb{R}^{3}\) is a bounded domain, \(\lambda , a, b>0\) are fixed parameters. \(f(x,\cdot )\) is continuously differentiable for a.e. \(x\in \varOmega \). By using the constraint variational method and the degree theory, he got the existence of a least energy nodal solution to the Kirchhoff-type equation.

Wang, Zhang, and Guan [20] studied the following Schrödinger–Poisson system with critical growth:

$$ \textstyle\begin{cases} -\Delta u+V(x)u+\lambda \phi u= \vert u \vert ^{4}u+\mu f(u),&x\in \mathbb{R}^{3}, \\ -\Delta \phi =u^{2},&x\in \mathbb{R}^{3}, \end{cases} $$

where \(\mu , \lambda >0\), \(f\in C^{1}(\mathbb{R},\mathbb{R})\). They got the existence and asymptotic behavior of a least energy sign-changing solution to the above system.

Motivated by the above references, in this paper, we study the existence of both ground state and least energy nodal solution for the following critical Kirchhoff–Schrödinger–Poisson system with asymptotically 3-linear growth nonlinearity:

$$ \textstyle\begin{cases} -(a+ b\int _{\mathbb{R}^{3}} \vert \nabla u \vert ^{2}\,dx)\Delta u+V(x)u+\lambda \phi u= \vert u \vert ^{4}u+ k f(u),&x\in \mathbb{R}^{3}, \\ -\Delta \phi =u^{2},&x\in \mathbb{R}^{3}, \end{cases} $$
(1.7)

where a, b, k, λ are positive real numbers. Similar to [22], we suppose that \(V\in C(\mathbb{R}^{3},\mathbb{R}^{+})\) and satisfies that \(E\hookrightarrow \hookrightarrow L^{p}(\mathbb{R}^{3})\) (compact embedding) for \(2< p<6\), and \(E\hookrightarrow L^{6}(\mathbb{R}^{3})\) is continuous, where E is a Hilbert space defined by

$$\begin{aligned} E= \textstyle\begin{cases} H^{1}_{r}(\mathbb{R}^{3})=\{u\in H^{1}(\mathbb{R}^{3}):u(x)=u( \vert x \vert )\},& \text{if }V(x) \text{ is a constant}, \\ \{u\in D^{1,2}(\mathbb{R}^{3}):\int _{\mathbb{R}^{3}}V(x)u^{2} \,\mathrm{d}x< \infty \},&\text{if }V(x)\text{ is not a constant} \end{cases}\displaystyle \end{aligned}$$

with the inner product defined by

$$ \langle u,v\rangle = \int _{\mathbb{R}^{3}}\bigl(a \nabla u\cdot \nabla v+ V(x)uv\bigr)\,dx,\quad \forall u, v\in E $$

and the norm \(\|\cdot \|\):

$$ \Vert u \Vert ^{2}= \int _{\mathbb{R}^{3}}\bigl(a \vert \nabla u \vert ^{2}+ V(x)u^{2}\bigr)\,dx. $$

As for the function f, we assume \(f\in C(\mathbb{R},\mathbb{R})\) and satisfies the following hypotheses:

\((f_{1})\):

\(f(t)\cdot t>0\) for \(t\ne 0\);

\((f_{2})\):

\(\lim_{t\rightarrow \infty }\frac{f(t)}{t^{3}}=1\) and \(\frac{f(t)}{t^{3}}<1\) for all \(t\in \mathbb{R}\setminus \{0\}\);

\((f_{3})\):

\(\frac{f(t)}{|t|^{3}}\) is an increasing function in \((-\infty ,0)\) and \((0,+\infty )\).

Remark 1.1

We note that under conditions (\(f_{1}\))–(\(f_{3}\)), it is easy to see

$$ \lim_{t\rightarrow 0}\frac{f(t)}{t}=0. $$
(1.8)

The function \(f(t)=\frac{t^{5}}{1+t^{2}}\) is an example satisfying all conditions (\(f_{1}\))–(\(f_{3}\)).

It is well known that the equation \(-\Delta \phi =u^{2}\) can be solved as

$$\begin{aligned} \phi _{u}(x)=\frac{1}{4\pi } \int _{\mathbb{R}^{3}} \frac{u^{2}(y)}{ \vert x-y \vert }\,dy. \end{aligned}$$
(1.9)

So system (1.7) is merely a single equation on u:

$$\begin{aligned} -\biggl(a+ b \int _{\mathbb{R}^{3}} \vert \nabla u \vert ^{2}\,dx\biggr) \Delta u+V(x)u+\lambda \phi _{u} u= \vert u \vert ^{4}u+ k f(u), \quad x\in \mathbb{R}^{3}. \end{aligned}$$
(1.10)

Based on the results above, the energy functional associated with system (1.7) and so with (1.10) is defined by

$$\begin{aligned} J^{b}_{ k}(u)&=\frac{1}{2} \int _{\mathbb{R}^{3}}\bigl( a \vert \nabla u \vert ^{2}+V(x)u^{2}\bigr)\,dx+ \frac{b}{4}\biggl( \int _{\mathbb{R}^{3}} \vert \nabla u \vert ^{2}\,dx \biggr)^{2} \\ &\quad {}+\frac{\lambda }{4} \int _{\mathbb{R}^{3}}\phi _{u} u^{2}\,dx- k \int _{ \mathbb{R}^{3}}F(u)\,dx-\frac{1}{6} \int _{\mathbb{R}^{3}} \vert u \vert ^{6}\,dx \end{aligned}$$

for any \(u\in E\). Moreover, under our conditions, \(J^{b }_{ k}(u)\) belongs to \(C^{1}(E,\mathbb{R})\), and the Fréchet derivative of \(J^{b}_{ k}\) is

$$\begin{aligned} \bigl\langle \bigl(J^{b}_{ k}\bigr)'(u),v \bigr\rangle & = \int _{\mathbb{R}^{3}}\bigl({ a} \nabla u\cdot \nabla v+V(x)uv\bigr)\,dx+b \biggl( \int _{\mathbb{R}^{3}} \vert \nabla u \vert ^{2}\,dx\biggr) \biggl( \int _{\mathbb{R}^{N}}\nabla u\cdot \nabla v\,dx\biggr) \\ &\quad {}+\lambda \int _{\mathbb{R}^{3}}\phi _{u}uv\,dx - k \int _{\mathbb{R}^{3}}f(u)v\,dx- \int _{\mathbb{R}^{3}} \vert u \vert ^{4}uv\,dx \end{aligned}$$

for any \(u,v\in E\).

As it is well known, if \(u\in E\) is a solution of system (1.7) and \(u^{\pm }\neq 0\), then u is a nodal solution of system (1.7), where

$$ u^{+}=\max \bigl\{ u(x),0\bigr\} ,\qquad u^{-}=\min \bigl\{ u(x),0\bigr\} . $$

Note that, since system (1.7) involved pure critical nonlinearity \(|u|^{4}u\), it will prevent us from using the standard arguments as in [3, 12, 19, 22]. Hence, we need to show some techniques to overcome the lack of compactness in \(E\hookrightarrow L^{6}(\mathbb{R}^{3})\).

The main results can be stated as follows.

Theorem 1.1

Suppose that\((f_{1})\)\((f_{3})\)are satisfied. Then there exists\(k^{\star }>0\)such that, for all\(k\geq k^{\star }\), system (1.7) has a least energy nodal solution\(u_{b}\), which has precisely two nodal domains.

Remark 1.2

The least energy nodal solution \(u_{b}\) is a solution of (1.7) satisfying

$$ J^{b}_{ k}(u_{b })=\inf_{u\in \mathcal{M}_{k}^{b }}J^{ b }_{ k}(u), $$

where \(\mathcal{M}_{k}^{b }\) is defined by (2.1) in the next section. We recall that the nodal of a continuous function \(u:\mathbb{R}^{3}\to \mathbb{R}\) is the surface \(u^{-1}(0)\). Every connected component of \(\mathbb{R}^{3}\setminus u^{-1}(0)\) is called a nodal domain.

Theorem 1.2

Suppose that\((f_{1})\)\((f_{3})\)are satisfied. Then there exists\(k^{\star \star }>0\)such that, for all\(k\geq k^{\star \star }\), the\(c^{\ast }>0\)is achieved and

$$ J^{b }_{ k}({ u_{b }})>2c^{\ast }, $$

where\(c^{\ast }=\inf_{u\in \mathcal{N}_{ k}^{b }}J^{b }_{ k}(u)\), \(\mathcal{N}_{ k}^{b}=\{u\in H\backslash \{0\}|\langle (J^{ b }_{ k})'(u),u \rangle =0 \}\), and\(u_{b }\)is the least energy nodal solution obtained in Theorem 1.1. In particular, \(c^{\ast }>0\)is achieved either by a positive or a negative function\(v_{b }\)which is a ground state solution of system (1.7).

Theorem 1.3

Suppose that\((f_{1})\)\((f_{3})\)are satisfied. Then there exists\(k^{\star \star \star }>0\)such that, for all\(k\geq k^{\star \star \star }\), for any least energy nodal solution sequence\(\{u_{b _{n}}\}\)with\(b _{n}\rightarrow 0\)as\(n\rightarrow \infty \), there exists a subsequence, still denoted by\({ \{u_{b _{n}}\}}\), such that\(u_{b _{n}}\)converges to\(u_{0}\)weakly inEas\(n\rightarrow \infty \), where\(u_{0}\)is a least energy nodal solution of the following problem:

$$\begin{aligned} \textstyle\begin{cases} -a\Delta u+V(x)u+\lambda\phi u=|u|^{4}u+ k f(u), ~\ &x\in\mathbb{R}^{3},\\ -\Delta\phi=u^{2},&x\in\mathbb{R}^{3}. \end{cases}\displaystyle \end{aligned}$$
(1.11)

Comparing with the literature works, the above three results can be regarded as a generalization of those in [12, 19, 20]. As for Kirchhoff–Schrödinger–Poisson equation, to the best of our knowledge, few results involved the existence and asymptotic behavior of ground state nodal solutions in case of critical growth. It is worth noting that the Brower degree method used in [20, 23] is strictly dependent on the nonlinearity \(f\in C^{1}(\mathbb{R},\mathbb{R})\), so we have to find new ways to solve our model where we only allow \(f\in C(\mathbb{R},\mathbb{R})\). On the other hand, in our modeling, both of the nonlocal terms \(\int _{\mathbb{R}^{3}}|\nabla u|^{2}\,dx\) and ϕu appear, we need to overcome the difficulties caused by the nonlocal terms under a uniform variational framework. It is also due to the lack of compactness embedded in full space that we cannot use the method in [18]. Thankfully, after appropriate modifications, the deformation lemma used in [12] can be applied to get the existence of a least energy nodal solution of the Kirchhoff–Schrödinger–Poisson system.

2 Some technical lemmas

To fix some notations, the letter C, \(C_{i}\) will be repeatedly used to denote various positive constants whose exact values are irrelevant. \(|\cdot|_{p}\) denote the norm in \(L^{p}(\mathbb{R}^{3})\) for \(p>1\).

We first list some properties of \(\phi _{u}\) for our use, one can find the details in [14, 26].

Proposition 2.1

For any\(u\in E\), we have

  1. (i)

    there exists\(C>0\)such that

    $$ \int _{\mathbb{R}^{3}}\phi _{u}u^{2}\,dx\leq C \Vert u \Vert ^{4}\quad \forall u \in E; $$
  2. (ii)

    \(\phi _{u}\geq 0\), \(\forall u\in E\);

  3. (iii)

    \(\phi _{tu}=t^{2}\phi _{u}\), \(\forall t>0\)and\(u\in E\);

  4. (iv)

    if\(u_{n}\rightharpoonup u\)inE, then\(\phi _{u_{n}}\rightharpoonup \phi _{u}\)in\(D^{1,2}(\mathbb{R}^{3})\)and

    $$\begin{aligned} \int _{\mathbb{R}^{3}}\phi _{u_{n}}u_{n}^{2} \,dx\rightarrow \int _{ \mathbb{R}^{3}}\phi _{u}u^{2}\,dx. \end{aligned}$$

For fixed \(u\in E\) with \(u^{\pm }\neq 0\), the function \(\psi _{u}:[0,\infty )\times [0,\infty )\rightarrow \mathbb{R}\) and the mapping \(W_{u}:[0,\infty )\times [0,\infty )\rightarrow \mathbb{R}^{2}\) are well defined by

$$\begin{aligned}& \psi _{u}(s,t)=J^{b }_{ k}\bigl(su^{+}+tu^{-} \bigr), \\& W_{u}(s,t)=\bigl(\bigl\langle \bigl(J^{b }_{ k} \bigr)'\bigl(su^{+}+tu^{-}\bigr),su^{+} \bigr\rangle , \bigl\langle \bigl(J^{b }_{ k} \bigr)'\bigl(su^{+}+tu^{-}\bigr),tu^{-} \bigr\rangle \bigr), \end{aligned}$$

and

$$\begin{aligned} \mathcal{M}_{k}^{b }=\bigl\{ u\in E,u^{\pm }\neq 0 \text{ and } \bigl\langle \bigl(J^{b }_{ k} \bigr)'(u),u^{+}\bigr\rangle =\bigl\langle \bigl(J^{b }_{ k}\bigr)'(u),u^{-} \bigr\rangle =0\bigr\} . \end{aligned}$$
(2.1)

Lemma 2.1

Assume that\((f_{1})\)\((f_{3})\)are satisfied, if\(u\in E\)with\(u^{\pm }\neq 0\), then\(\psi _{u}\)has the following properties:

  1. (i)

    The pair\((s,t)\)is a critical point of\(\psi _{u}\)with\(s,t>0 \Leftrightarrow s u^{+}+t u^{-}\in \mathcal{M}_{k}^{b }\);

  2. (ii)

    The function\(\psi _{u}\)has a unique critical point\((s_{u}, t_{u})\)on\((0,\infty )\times (0,\infty )\), which is also the unique maximum point of\(\psi _{u}\)on\([0,\infty )\times [0,\infty )\); Furthermore, if\(\langle (J^{b }_{ k})'(u),u^{\pm }\rangle \leq 0\), then\(0< s_{u}\), \(t_{u}\leq 1\).

Proof

(i) By the definition of \(\psi _{u}\), we have that

$$\begin{aligned} \nabla \psi _{u}(s,t)&=\biggl(\frac{\partial \psi _{u}}{\partial s}, \frac{\partial \psi _{u}}{\partial t} \biggr) \\ &=\biggl(\frac{1}{s}\bigl\langle \bigl(J^{b }_{ k} \bigr)'\bigl(s u^{+}+t u^{-}\bigr),s u^{+} \bigr\rangle ,\frac{1}{t}\bigl\langle \bigl(J^{b }_{ k}\bigr)'\bigl(s u^{+}+t u^{-}\bigr),{ t u^{-}} \bigr\rangle \biggr). \end{aligned}$$

From the definition, item (i) is obvious.

(ii) It is easy to see

$$\begin{aligned} &\bigl\langle \bigl(J^{b }_{ k} \bigr)'\bigl(su^{+}+tu^{-}\bigr),su^{+} \bigr\rangle \\ &\quad =s^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}+b s^{4}\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\biggr)^{2}+bs^{2}t^{2}\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\biggr) \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx\biggr) \\ &\qquad{}+s^{4}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx+s^{2}t^{2} \lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx -s^{6} \int _{ \mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx \\ &\qquad{}- k \int _{\mathbb{R}^{3}}f\bigl(su^{+}\bigr)su^{+}\,dx \end{aligned}$$
(2.2)

and

$$\begin{aligned} &\bigl\langle \bigl(J^{b }_{ k} \bigr)'\bigl(su^{+}+tu^{-}\bigr),tu^{-} \bigr\rangle \\ &\quad =t^{2} \bigl\Vert u^{-} \bigr\Vert ^{2}+bt^{4} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx \biggr)^{2} \\ &\qquad{}+bs^{2}t^{2} \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx \int _{ \mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx +t^{4}\lambda \int _{\mathbb{R}^{3}} \phi _{u^{-}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx \\ &\qquad{}+s^{2}t^{2}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx -t^{6} \int _{\mathbb{R}^{3}} \bigl\vert u^{-} \bigr\vert ^{6}\,dx - k \int _{\mathbb{R}^{3}}f\bigl(tu^{-}\bigr)tu^{-}\,dx. \end{aligned}$$
(2.3)

From \((f_{1})\) and \((f_{2})\), for any \(\varepsilon >0\), there is \(C_{\varepsilon }>0\) satisfying

$$\begin{aligned} \bigl\vert f(t) \bigr\vert \leq \varepsilon \vert t \vert +C_{\varepsilon } \vert t \vert ^{4} \end{aligned}$$
(2.4)

for all \(t\in \mathbb{R}\). From the Sobolev embedding theorem it follows that

$$\begin{aligned} &\bigl\langle \bigl(J^{b }_{ k}\bigr)' \bigl(su^{+}+tu^{-}\bigr),su^{+}\bigr\rangle \\ &\quad \geq s^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}-s^{6} \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx- k \varepsilon s^{2} \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx- k C_{ \varepsilon } s^{q} \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{5}\,dx \\ &\quad \geq s^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}-C_{1}s^{6} \bigl\Vert u^{+} \bigr\Vert ^{6}- k\varepsilon C_{2}s^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}- k C_{\varepsilon } C_{3}s^{q} \bigl\Vert u^{+} \bigr\Vert ^{5} \\ &\quad \geq (1- k\varepsilon C_{4})s^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}-C_{4}s^{6} \bigl\Vert u^{+} \bigr\Vert ^{6}- k C_{4}s^{5} \bigl\Vert u^{+} \bigr\Vert ^{5}. \end{aligned}$$

By choosing \(\varepsilon >0\) such that \((1- k\varepsilon C_{4})>0\), we can infer that

$$\begin{aligned} \bigl\langle \bigl(J^{b }_{ k}\bigr)' \bigl(su^{+}+tu^{-}\bigr),su^{+}\bigr\rangle >0 \end{aligned}$$

for \(0< s\ll 1\) and all \(t\geq 0\). Similarly, there holds

$$\begin{aligned} \bigl\langle \bigl(J^{b}_{ k}\bigr)'\bigl(su^{+}+tu^{-}\bigr),tu^{-}\bigr\rangle >0 \end{aligned}$$

for \(0< t\ll 1\) and all \(s\geq 0\). Hence, there exists \(\delta _{1}>0\) such that

$$\begin{aligned} \bigl\langle \bigl(J^{b }_{ k} \bigr)'\bigl(\delta _{1}u^{+}+tu^{-} \bigr), \delta _{1}u^{+} \bigr\rangle >0,\qquad \bigl\langle \bigl(J^{b }_{ k}\bigr)'\bigl(su^{+}+ \delta _{1}u^{-}\bigr), \delta _{1}u^{-} \bigr\rangle >0 \end{aligned}$$
(2.5)

for all \(s\geq 0\), \(t\geq 0\). It is worth noting that assumption \((f_{1})\) implies

$$\begin{aligned} F(t)\geq 0, \quad t\in \mathbb{R}. \end{aligned}$$
(2.6)

Thus, choosing \(s=\delta _{2}'>\delta _{1}\), it follows that, for \(t\in [\delta _{1},\delta _{2}']\) and \(\delta _{2}'\gg 1\),

$$\begin{aligned} &\bigl\langle \bigl(J^{b }_{ k}\bigr)'\bigl( \delta _{2}'u^{+}+tu^{-}\bigr),\delta _{2}'u^{+} \bigr\rangle \\ &\quad \leq \bigl({\delta _{2}'}\bigr)^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}+b\bigl({\delta _{2}'} \bigr)^{4} \bigl\Vert u^{+} \bigr\Vert ^{4}+b \bigl({ \delta _{2}'}\bigr)^{4} \bigl\Vert u^{+} \bigr\Vert ^{2} \bigl\Vert u^{-} \bigr\Vert ^{2} \\ &\qquad{}+\bigl({\delta _{2}'}\bigr)^{4}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx+ \bigl({\delta _{2}'} \bigr)^{4}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx-\bigl({ \delta _{2}'} \bigr)^{6} \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx\\ &\quad \leq 0. \end{aligned}$$

Analogously, one can show that

$$\begin{aligned} &\bigl\langle \bigl(J^{b }_{ k}\bigr)' \bigl(su^{+}+tu^{-}\bigr),tu^{-}\bigr\rangle \\ &\quad \leq t^{2} \bigl\Vert u^{-} \bigr\Vert ^{2}+bt^{4} \bigl\Vert u^{-} \bigr\Vert ^{4}+bs^{2}t^{2} \bigl\Vert u^{+} \bigr\Vert ^{2} \bigl\Vert u^{-} \bigr\Vert ^{2} \\ &\qquad{}+t^{4}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx+s^{2}t^{2} \lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx-t^{6} \int _{ \mathbb{R}^{3}} \bigl\vert u^{-} \bigr\vert ^{6}\,dx. \end{aligned}$$

Choosing \(\delta _{2}>\delta _{2}'\gg 1\), we deduce

$$\begin{aligned} \bigl\langle \bigl(J^{b }_{ k} \bigr)'\bigl(\delta _{2}u^{+}+tu^{-} \bigr),\delta _{2}u^{+} \bigr\rangle < 0,\qquad \bigl\langle \bigl(J^{b }_{ k}\bigr)'\bigl(su^{+}+ \delta _{2}u^{-}\bigr), \delta _{2}u^{-} \bigr\rangle < 0 \end{aligned}$$
(2.7)

for all \(s,t\in [\delta _{1},\delta _{2}]\).

From (2.5) and (2.7), the assumptions of Miranda’s theorem (see Lemma 2.4 in [7]) are satisfied. Thus there is \((s_{u}, t_{u})\in (0,\infty )\times (0,\infty )\) satisfying \(W_{u}(s_{u},t_{u})=(0,0)\). So \(s_{u}u^{+}+t_{u}u^{-}\in \mathcal{M}_{k}^{b }\).

Now we turn to proving that the pair \((s_{u}, t_{u})\) is unique. We first suppose that \(u\in \mathcal{M}_{k}^{b }\), thus

$$\begin{aligned} & \bigl\Vert u^{+} \bigr\Vert ^{2}+b\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\biggr)^{2}+b \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx +\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx \\ &\quad{}+\lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx = \int _{ \mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}}f\bigl(u^{+}\bigr)u^{+}\,dx \end{aligned}$$
(2.8)

and

$$\begin{aligned} & \bigl\Vert u^{-} \bigr\Vert ^{2}+b\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx\biggr)^{2} +b \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx +\lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx \\ &\quad{}+\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx= \int _{ \mathbb{R}^{3}} \bigl\vert u^{-} \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}}f\bigl(u^{-}\bigr)u^{-}\,dx. \end{aligned}$$
(2.9)

We will show that the pair \((s_{u},t_{u})=(1,1)\) is the unique one such that \(s_{u}u^{+}+t_{u}u^{-}\in \mathcal{M}_{k}^{b }\). Let \((s_{0},t_{0})\) be a pair of numbers such that \(s_{0}u^{+}+t_{0}u^{-}\in \mathcal{M}_{k}^{b }\) with \(0< s_{0}\leq t_{0}\). We have

$$\begin{aligned} & s_{0}^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}+bs_{0}^{4}\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\biggr)^{2}+bs_{0}^{2}t_{0}^{2} \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx \\ &\qquad{}+s_{0}^{4}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx+s_{0}^{2}t_{0}^{2} \lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx \\ &\quad =s_{0}^{6} \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}}f\bigl(s_{0}u^{+} \bigr)s_{0}u^{+}\,dx \end{aligned}$$
(2.10)

and

$$\begin{aligned} & t_{0}^{2} \bigl\Vert u^{-} \bigr\Vert ^{2} +bt_{0}^{4}\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx\biggr)^{2}+bs_{0}^{2}t_{0}^{2} \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx \\ &\qquad{}+t_{0}^{4}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx+s_{0}^{2}t_{0}^{2} \lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx \\ &\quad =t_{0}^{6} \int _{\mathbb{R}^{3}} \bigl\vert u^{-} \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}}f\bigl(t_{0}u^{-} \bigr)t_{0}u^{-}\,dx. \end{aligned}$$
(2.11)

By comparing (2.9) and (2.11), we deduce

$$\begin{aligned} &\frac{ \Vert u^{-} \Vert ^{2}}{t_{0}^{2}}+b \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx \biggr)^{2}+b \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx \cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx \\ &\qquad{}+\lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx+\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx \\ &\quad \geq t_{0}^{2} \int _{\mathbb{R}^{3}} \bigl\vert u^{-} \bigr\vert ^{6}\,dx+ k \int _{ \mathbb{R}^{3}} \biggl[\frac{f(t_{0}u^{-})}{(t_{0}u^{-})^{3}} \biggr] \bigl(u^{-}\bigr)^{4}\,dx. \end{aligned}$$
(2.12)

Combining (2.9) with (2.12), one has that

$$ \biggl(\frac{1}{t_{0}^{2}}-1 \biggr) \bigl\Vert u^{-} \bigr\Vert ^{2} \geq \bigl(t_{0}^{2}-1\bigr) \int _{\mathbb{R}^{3}} \bigl\vert u^{-} \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}} \biggl[ \frac{f(t_{0}u^{-})}{(t_{0}u^{-})^{3}} -\frac{f(u^{-})}{(u^{-})^{3}} \biggr]\bigl(u^{-}\bigr)^{4}\,dx. $$

By using assumption \((f_{3})\), we get \(t_{0}\le 1\). Analogously, from (2.8), (2.10), and \(0< s_{0}\leq t_{0}\),

$$ \biggl(\frac{1}{s_{0}^{2}}-1 \biggr) \bigl\Vert u^{+} \bigr\Vert ^{2} \leq \bigl(s_{0}^{2}-1\bigr) \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}} \biggl[ \frac{f(s_{0}u^{+})}{(s_{0}u^{+})^{3}} -\frac{f(u^{+})}{(u^{+})^{3}} \biggr]\bigl(u^{+}\bigr)^{4}\,dx. $$

By using assumption \((f_{3})\), we get \(s_{0}\geq 1\). Consequently, \(s_{0}=t_{0}=1\).

In the case \(u\notin \mathcal{M}_{k}^{b }\), we suppose that there are \((s_{1},t_{1})\), \((s_{2},t_{2})\) such that

$$ u_{1}=s_{1}u^{+}+t_{1}u^{-} \in \mathcal{M}_{k}^{b },\qquad u_{2}=s_{2}u^{+}+t_{2}u^{-} \in \mathcal{M}_{k}^{b }. $$

Thus,

$$ u_{2}=\biggl(\frac{s_{2}}{s_{1}}\biggr)s_{1}u^{+}+ \biggl(\frac{t_{2}}{t_{1}}\biggr)t_{1}u^{-} =\biggl( \frac{s_{2}}{s_{1}}\biggr)u_{1}^{+}+\biggl(\frac{t_{2}}{t_{1}} \biggr)u_{1}^{-}\in \mathcal{M}_{k}^{b }. $$

According to \(u_{1}\in \mathcal{M}_{k}^{b }\) and the fact of the previous case, one has that

$$ \frac{s_{2}}{s_{1}}=\frac{t_{2}}{t_{1}}=1. $$

Thus \(s_{1}=s_{2}\), \(t_{1}=t_{2}\). Therefore \((s_{u}, t_{u})\) is the unique critical point of \(\psi _{u}\) in \((0,\infty )\times (0,\infty )\).

In the following, we show that the critical point \((s_{u},t_{u})\) of \(\psi _{u}\) is its unique maximum point on \([0,+\infty )\times [0,+\infty )\). By definition

$$\begin{aligned} \psi _{u}(s,t)&=\frac{s^{2}}{2} \bigl\Vert u^{+} \bigr\Vert ^{2}+\frac{bs^{4}}{4} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx \biggr)^{2}+ \frac{s^{4}}{4}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx- \frac{s^{6}}{6} \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx \\ &\quad{}- \int _{\mathbb{R}^{3}}F\bigl(su^{+}\bigr)\,dx+\frac{t^{2}}{2} \bigl\Vert u^{-} \bigr\Vert + \frac{bt^{4}}{4}\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx\biggr)^{2}+ \frac{t^{4}}{4}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx \\ &\quad{}-\frac{t^{6}}{6} \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx- \int _{ \mathbb{R}^{3}}F\bigl(tu^{-}\bigr)\,dx+\frac{s^{2}t^{2}}{4} \lambda \int _{ \mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx \\ &\quad{}+\frac{s^{2}t^{2}}{4}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx+ \frac{bs^{2}t^{2}}{2} \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx. \end{aligned}$$

Now (2.6) implies that

$$ \lim_{|(s,t)|\rightarrow \infty }\psi _{u}(s,t)=-\infty . $$

By contradiction, we suppose that the boundary point \((0, t_{0})\) is a maximum point of \(\psi _{u}\) with \(t_{0}\geq 0\). By direct computation, it follows that

$$\begin{aligned} (\psi _{u})'_{s}(s,t_{0})&=s \bigl\Vert u^{+} \bigr\Vert ^{2}+bs^{3}\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\biggr)^{2}+ast_{0}^{2}\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\biggr) \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx\biggr) \\ &\quad{}+s^{3}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx+ \frac{st_{0}^{2}}{2}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{-} \bigr\vert ^{2}\,dx \\ &\quad{}+\frac{st_{0}^{2}}{2}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx-s^{5} \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx- \int _{\mathbb{R}^{3}}f\bigl(su^{+}\bigr)u^{+}\,dx\\ & >0 \end{aligned}$$

when \(s\ll 1\). It follows that \(\psi _{u}\) is an increasing function with respect to s when \(s\ll 1\), which is a contradiction. Analogously, \(\psi _{u}\) cannot achieve its global maximum on the boundary point \((s,0)\) with \(s\geq 0\).

In the remainder of our proof, we will prove that \(0< s_{u}\), \(t_{u}\leq 1\) when \(\langle (J^{b }_{ k})'(u),u^{\pm }\rangle \leq 0\). Suppose \(s_{u}\geq t_{u}>0\). One has

$$\begin{aligned} &s_{u}^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}+bs_{u}^{4} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx \biggr)^{2}+bs_{u}^{4} \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx \\ &\qquad{}+s_{u}^{4}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx+ s_{u}^{4} \lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx \\ &\quad \geq s_{u}^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}+bs_{u}^{4} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx \biggr)^{2}+bs_{u}^{2}t_{u}^{2} \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx \\ &\qquad{}+s_{u}^{4}\lambda \int _{\mathbb{R}^{3}}\phi _{u^{+}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx+ s_{u}^{2}t_{u}^{2} \lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx \\ &\quad =s_{u}^{6} \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}}f\bigl(s_{u}u^{+} \bigr)s_{u}u^{+}\,dx. \end{aligned}$$
(2.13)

In view of \(\langle (J^{b }_{ k})'(u),u^{+}\rangle \leq 0\), one has that

$$\begin{aligned} & \bigl\Vert u^{+} \bigr\Vert ^{2}+b \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx \biggr)^{2}+b \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\cdot \int _{ \mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx+\lambda \int _{\mathbb{R}^{3}} \phi _{u^{+}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx \\ &\quad{}+\lambda \int _{\mathbb{R}^{3}}\phi _{u^{-}} \bigl\vert u^{+} \bigr\vert ^{2}\,dx\leq \int _{ \mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}}f\bigl(u^{+}\bigr)u^{+}\,dx. \end{aligned}$$
(2.14)

By comparing (2.13) and (2.14), it follows that

$$ \biggl(\frac{1}{s_{u}^{2}}-1\biggr) \bigl\Vert u^{+} \bigr\Vert ^{2}\geq \bigl(s_{u}^{2}-1\bigr) \int _{ \mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}} \biggl[ \frac{f(s_{u}u^{+})}{(s_{u}u^{+})^{3}} -\frac{f(u^{+})}{(u^{+})^{3}} \biggr]\bigl(u^{+}\bigr)^{4}\,dx. $$

It implies \(s_{u}\leq 1\). Therefore \(0< s_{u}\), \(t_{u}\leq 1\). □

Lemma 2.2

If\(u\in \mathcal{M}^{b }_{k}\), then\(tu\notin \mathcal{M}^{b }_{k}\)for every\(t>0\), \(t\neq 1\). More precisely,

$$\begin{aligned} &\bigl\langle \bigl(J^{b }_{ k}\bigr)'(tu),tu^{\pm } \bigr\rangle >0\quad \textit{for } t\in (0,1), \\ &\bigl\langle \bigl(J^{b }_{ k}\bigr)'(tu),tu^{\pm } \bigr\rangle < 0\quad \textit{for } t>1. \end{aligned}$$

Proof

From (2.2) and \(u\in \mathcal{M}^{b }_{k}\), we have that

$$\begin{aligned} \bigl\langle \bigl(J^{b }_{ k}\bigr)'(tu),tu^{+} \bigr\rangle &=t^{2}\bigl(1-t^{2}\bigr) \bigl\Vert u^{+} \bigr\Vert ^{2}+t^{4}\bigl(1-t^{2} \bigr) \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx \\ &\quad{}+ k t^{4} \int _{\mathbb{R}^{3}} \biggl(f\bigl(u^{+}\bigr)- \frac{f(tu^{+})}{t^{3}} \biggr) u^{+}\,dx. \end{aligned}$$

According to \((f_{3})\), when \(0< t<1\),

$$\begin{aligned} \bigl\langle \bigl(J^{b }_{ k}\bigr)'(tu),tu^{+} \bigr\rangle >0, \end{aligned}$$

while in the case \(t>1\),

$$\begin{aligned} \bigl\langle \bigl(J^{b }_{ k}\bigr)'(tu),tu^{+} \bigr\rangle < 0. \end{aligned}$$

Similarly, it is easy to get

$$\begin{aligned} &\bigl\langle \bigl(J^{b }_{ k}\bigr)'(tu),tu^{-} \bigr\rangle >0\quad \text{for } t \in (0,1),\qquad \bigl\langle \bigl(J^{b }_{ k}\bigr)'(tu),tu^{-} \bigr\rangle < 0\quad \text{for } t>1. \end{aligned}$$

The proof is complete. □

Lemma 2.3

Let\(c_{b }^{ k}=\inf_{u\in \mathcal{M}_{k}^{b }}J^{ b }_{ k}(u)\), then we have that

$$\begin{aligned} \lim_{ k\rightarrow \infty } c_{b }^{ k}=0. \end{aligned}$$

Proof

For any \(u\in \mathcal{M}_{k}^{b }\), we can deduce

$$\begin{aligned} & \bigl\Vert u^{\pm } \bigr\Vert ^{2}+b \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{\pm } \bigr\vert ^{2}\,dx \biggr)^{2}+b \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+} \bigr\vert ^{2}\,dx\cdot \int _{ \mathbb{R}^{3}} \bigl\vert \nabla u^{-} \bigr\vert ^{2}\,dx +\lambda \int _{\mathbb{R}^{3}} \phi _{u^{\pm }} \bigl\vert u^{\pm } \bigr\vert ^{2}\,dx \\ &\quad{}+\lambda \int _{\mathbb{R}^{3}}\phi _{u^{\mp }} \bigl\vert u^{\pm } \bigr\vert ^{2}\,dx= k \int _{\mathbb{R}^{3}}f\bigl(u^{\pm }\bigr)u^{\pm }\,dx+ \int _{\mathbb{R}^{3}} \bigl\vert u^{ \pm } \bigr\vert ^{6}\,dx. \end{aligned}$$

Hence, in view of (2.4), it follows that

$$\begin{aligned} \bigl\Vert u^{\pm } \bigr\Vert ^{2}&\leq k \int _{\mathbb{R}^{3}}f\bigl(u^{\pm }\bigr)u^{\pm }\,dx+ \int _{ \mathbb{R}^{3}} \bigl\vert u^{\pm } \bigr\vert ^{6}\,dx \\ &\leq k\varepsilon C_{1} \bigl\Vert u^{\pm } \bigr\Vert ^{2}+ k C_{2} \bigl\Vert u^{\pm } \bigr\Vert ^{5}+C_{3} \bigl\Vert u^{\pm } \bigr\Vert ^{6}. \end{aligned}$$

Therefore, we have that

$$\begin{aligned} (1- k\varepsilon C_{1}) \bigl\Vert u^{\pm } \bigr\Vert ^{2}\leq k C_{2} \bigl\Vert u^{\pm } \bigr\Vert ^{5}+C_{3} \bigl\Vert u^{\pm } \bigr\Vert ^{6}. \end{aligned}$$

We now choose ε small enough such that \((1- k\varepsilon C_{1})>0\), so there is \(\rho >0\) such that

$$ \bigl\Vert u^{\pm } \bigr\Vert \geq \rho $$
(2.15)

for all \(u\in \mathcal{M}_{k}^{b }\). For any \(u\in \mathcal{M}_{k}^{b }\), in view of the definition of \(\mathcal{M}_{k}^{b }\), \(\langle (J^{b }_{ k})'(u),u\rangle =0\). From assumption \((f_{3})\), we have

$$\begin{aligned} f(t)t-4F(t)\geq 0, \end{aligned}$$
(2.16)

and \(f(t)t-4F(t)\) is increasing in \((0,+\infty )\) and decreasing in \((-\infty ,0)\). Hence, one gets

$$\begin{aligned} J^{b }_{ k}(u)&=J^{b }_{ k}(u)- \frac{1}{4}\bigl\langle \bigl(J^{ b }_{ k} \bigr)'(u),u\bigr\rangle \\ &=\frac{1}{4} \Vert u \Vert ^{2}+\frac{1}{12} \int _{\mathbb{R}^{3}} \vert u \vert ^{6}\,dx+ \frac{ k}{4} \int _{\mathbb{R}^{3}}\bigl[f(u)u-4F(u)\bigr]\,dx \\ &\geq \frac{1}{4} \Vert u \Vert ^{2} \end{aligned}$$

for any \(u\in \mathcal{M}_{k}^{b }\).

From the above discussion, we can see that \(c_{b }^{ k}=\inf_{u\in \mathcal{M}_{k}^{b }}J^{b }_{ k}(u)\) is well defined.

Let \(u\in E\) with \(u^{\pm }\neq 0\) be fixed. According to Lemma 2.1, for each \(k>0\), there exist \(s_{ k}, t_{ k}>0\) such that \(s_{ k}u^{+}+t_{ k}u^{-}\in \mathcal{M}_{k}^{b }\). Hence, by (2.6), the Sobolev embedding theorem and Proposition 2.1, we have

$$\begin{aligned} 0&\leq c^{ k}=\inf_{u\in \mathcal{M}_{k}^{b }}J^{b }_{ k}(u) \leq J^{b }_{ k}\bigl(s_{ k}u^{+}+t_{ k}u^{-} \bigr) \\ &\leq \frac{1}{2} \bigl\Vert s_{ k}u^{+}+t_{ k}u^{-} \bigr\Vert ^{2}+\frac{b}{4}\biggl( \int _{ \mathbb{R}^{3}} \bigl\vert \nabla \bigl(s_{ k}u^{+}+t_{ k}u^{-} \bigr) \bigr\vert ^{2}\,dx\biggr)^{2}\\ &\quad {}+ \frac{\lambda }{4} \int _{\mathbb{R}^{3}}\phi _{s_{ k}u^{+}+t_{ k}u^{-}} \bigl\vert s_{ k}u^{+}+t_{ k}u^{-} \bigr\vert ^{2}\,dx \\ &\leq { \frac{s_{ k}^{2}}{2} \bigl\Vert u^{+} \bigr\Vert ^{2}+\frac{t_{ k}^{2}}{2} \bigl\Vert u^{-} \bigr\Vert ^{2}+Cs_{ k}^{4} \bigl\Vert u^{+} \bigr\Vert ^{4}+Ct_{ k}^{4} \bigl\Vert u^{-} \bigr\Vert ^{4} } \end{aligned}$$

for some constants \(C>0\). We now define

$$\begin{aligned} \varPhi _{u}=\bigl\{ (s_{ k},t_{ k})\in [0,\infty ) \times [0,\infty ):W_{u}(s_{ k},t_{ k})=(0,0), k>0 \bigr\} . \end{aligned}$$

Hence we have that

$$\begin{aligned} s_{ k}^{6}& \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx+t_{ k}^{6} \int _{ \mathbb{R}^{3}} \bigl\vert u^{-} \bigr\vert ^{6}\,dx \\ &\leq s_{ k}^{6} \int _{\mathbb{R}^{3}} \bigl\vert u^{+} \bigr\vert ^{6}\,dx+t_{ k}^{6} \int _{ \mathbb{R}^{3}} \bigl\vert u^{-} \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}}f\bigl(s_{ k}u^{+} \bigr)s_{ k}u^{+}\,dx+ k \int _{\mathbb{R}^{3}}f\bigl(t_{ k}u^{-} \bigr)t_{ k}u^{-}\,dx \\ &= \bigl\Vert s_{ k}u^{+}+t_{ k}u^{-} \bigr\Vert ^{2}+b\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla \bigl(s_{ k}u^{+}+t_{ k}u^{-} \bigr) \bigr\vert ^{2}\,dx\biggr)^{2}+\lambda \int _{\mathbb{R}^{3}}\phi _{s_{ k}u^{+}+t_{ k}u^{-}} \bigl\vert s_{ k}u^{+}+t_{ k}u^{-} \bigr\vert ^{2}\,dx \\ &\leq s_{ k}^{2} \bigl\Vert u^{+} \bigr\Vert ^{2}+t_{ k}^{2} \bigl\Vert u^{-} \bigr\Vert ^{2}+Cs_{ k}^{4} \bigl\Vert u^{+} \bigr\Vert ^{4}+Ct_{ k}^{4} \bigl\Vert u^{-} \bigr\Vert ^{4} . \end{aligned}$$

It follows that \(\varPhi _{u}\) is a bounded set. We suppose that \(k_{n}\rightarrow \infty \) as \(n\rightarrow \infty \). For \((s_{ k_{n}}, t_{ k_{n}})\in \varPhi _{u}\), there exist \(s_{0}\) and \(t_{0}\) such that

$$ (s_{ k_{n}}, t_{ k_{n}})\rightarrow (s_{0}, t_{0}) $$

as \(n\rightarrow \infty \) (in the subsequence sense). We suppose that \(s_{0} > 0\) or \(t_{0}> 0\). Thanks to \(s_{ k_{n}}u^{+}+t_{ k_{n}}u^{-}\in \mathcal{M}_{b }^{ k_{n}}\), we get

$$\begin{aligned} & \bigl\Vert s_{ k_{n}}u^{+}+t_{ k_{n}}u^{-} \bigr\Vert ^{2}+b\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla \bigl(s_{ k_{n}}u^{+}+t_{ k_{n}}u^{-} \bigr) \bigr\vert ^{2}\,dx\biggr)^{2} \\ &\qquad{}+\lambda \int _{\mathbb{R}^{3}}\phi _{s_{ k_{n}}u^{+}+t_{ k_{n}}u^{-}} \bigl\vert s_{ k_{n}}u^{+}+t_{ k_{n}}u^{-} \bigr\vert ^{2}\,dx \\ &\quad = \int _{\mathbb{R}^{3}} \bigl\vert s_{ k_{n}}u^{+}+t_{ k_{n}}u^{-} \bigr\vert ^{6}\,dx+ k_{n} \int _{\mathbb{R}^{3}}f\bigl(s_{ k_{n}}u^{+} +t_{ k_{n}}u^{-}\bigr) \bigl(s_{ k_{n}}u^{+}+t_{ k_{n}}u^{-} \bigr)\,dx. \end{aligned}$$
(2.17)

According to \(s_{ k_{n}}u^{+}\rightarrow s_{0}u^{+}\) and \(t_{ k_{n}}u^{-}\rightarrow t_{0}u^{-}\) in E, \(\int _{\mathbb{R}^{3}}|\nabla (s_{ k_{n}}u^{+}+t_{ k_{n}}u^{-})|^{2}\,dx \leq \|s_{ k_{n}}u^{+}+t_{ k_{n}}u^{-}\|^{2}\), (2.4) and (2.6), so as \(n\rightarrow \infty \), there holds

$$\begin{aligned} \int _{\mathbb{R}^{3}}f\bigl(s_{ k_{n}}u^{+}+t_{ k_{n}}u^{-} \bigr) \bigl(s_{ k_{n}}u^{+}+t_{ k_{n}}u^{-} \bigr)\,dx \rightarrow \int _{\mathbb{R}^{3}}f\bigl(s_{0}u^{+}+t_{0}u^{-} \bigr) \bigl(s_{0}u^{+}+t_{0}u^{-} \bigr)\,dx>0. \end{aligned}$$

Because \(k_{n}\rightarrow \infty \) as \(n\rightarrow \infty \) and \(\{s_{ k_{n}}u^{+}+t_{ k_{n}}u^{-}\}\) is bounded in E, following the Sobolev embedding theorem, we have a contradiction with equality (2.17). Thus, \(s_{0}=t_{0}=0\), and so \(\lim_{ k\rightarrow \infty } c_{b }^{ k}=0\). □

Lemma 2.4

There exists\(k^{\star }>0\)such that, for all\(k\geq k^{\star }\), the infimum\(c_{b }^{ k}\)is achieved.

Proof

In view of the definition of \(c_{b }^{ k}\), we deduce that there exists a sequence \(\{u_{n}\}\subset \mathcal{M}^{ k}_{b }\) satisfying

$$ \lim_{n\rightarrow \infty }J^{b }_{ k}(u_{n})=c_{b }^{ k}. $$

Following from (2.8) and (2.9), \(\{u_{n}\}\) is bounded in E. So in the subsequence sense, there exists \(u_{b }=u^{+}_{b }+u^{-}_{b }\in E\) such that \(u_{n}\rightharpoonup u_{b }\). Since the embedding \(E\hookrightarrow L^{p}(\mathbb{R}^{3})\) is compact for \(p\in (2, 6)\), we deduce

$$\begin{aligned} &u_{n}\rightarrow u_{b } \quad \text{in } L^{p} \bigl(\mathbb{R}^{3}\bigr), \forall p\in (2, 6), \\ &u_{n}(x)\rightarrow u_{b }(x) \quad \text{a.e. } x\in \mathbb{R}^{3}. \end{aligned}$$

Then we have

$$\begin{aligned} &u^{\pm }_{n}\rightharpoonup u_{b}^{\pm }\quad \text{in } E, \\ &u^{\pm }_{n}\rightarrow u_{b}^{\pm } \quad \text{in } L^{p}\bigl(\mathbb{R}^{3}\bigr), \\ &u^{\pm }_{n}(x)\rightarrow u_{b}^{\pm }(x) \quad \text{a.e. } x\in \mathbb{R}^{3}. \end{aligned}$$

Denote \(\beta :=\frac{(S)^{\frac{3}{2}}}{3}\), where

$$ S:=\inf_{u\in E\backslash \{0\}} \frac{ \Vert u \Vert ^{2}}{(\int _{\mathbb{R}^{3}} \vert u \vert ^{6}\,dx)^{\frac{1}{3}}}. $$

The Sobolev embedding theorem insures that \(\beta >0\). Lemma 2.3 implies that there exists \(k^{\star }>0\) such that \(c_{b }^{ k}<\beta \) for all \(k\geq k^{\star }\). Fix \(k\geq k^{\star }\), in view of Lemma 2.1, we have

$$\begin{aligned} J^{b }_{ k}\bigl(su^{+}_{n}+tu ^{-}_{n}\bigr)\leq J^{b }_{ k}(u_{n}) \end{aligned}$$

for all \(s, t\in [0,+\infty )\). Because \(u^{\pm }_{n}\rightharpoonup u_{b}^{\pm } \) in E, E is a Hilbert space, we can deduce

$$\begin{aligned} \bigl\Vert u^{\pm }_{n} \bigr\Vert ^{2}- \bigl\Vert u^{\pm }_{n}-u_{b}^{\pm } \bigr\Vert ^{2}=2\bigl(u^{\pm }_{n},u_{b}^{\pm }\bigr)- \bigl\Vert u_{b}^{\pm } \bigr\Vert ^{2}, \end{aligned}$$

where we can assume that the sequence \(\{\|u^{\pm }_{n}\|\}\) is convergent, so we have

$$ \lim_{n\to \infty } \bigl\Vert u^{\pm }_{n} \bigr\Vert ^{2}=\lim_{n\to \infty } \bigl\Vert u^{\pm }_{n}-u_{b}^{\pm } \bigr\Vert ^{2}+ \bigl\Vert u_{b}^{\pm } \bigr\Vert ^{2}. $$

Obviously, we can let \(n\rightarrow \infty \) in both sides of the above equation. On the other hand, by (2.4) we have

$$\begin{aligned} \int _{\mathbb{R}^{3}}F\bigl(su_{n}^{\pm }\bigr)\,dx \rightarrow \int _{\mathbb{R}^{3}}F\bigl(su_{b}^{ \pm }\bigr)\,dx. \end{aligned}$$

Thus, we get

$$\begin{aligned} &\liminf_{n\rightarrow \infty }J^{b }_{ k} \bigl(su^{+}_{n}+tu^{-}_{n}\bigr)\\ &\quad \geq \frac{s^{2}}{2}\lim_{n\rightarrow \infty }\bigl( \bigl\Vert u^{+}_{n}-u_{b}^{+} \bigr\Vert ^{2}+ \bigl\Vert u_{b}^{+} \bigr\Vert ^{2}\bigr) +\frac{t^{2}}{2}\lim _{n\rightarrow \infty }\bigl( \bigl\Vert u^{-}_{n}-u_{b}^{-} \bigr\Vert ^{2}+ \bigl\Vert u_{b}^{-} \bigr\Vert ^{2}\bigr) \\ &\qquad{}+\frac{bs^{2}}{4} \biggl[\liminf_{n\rightarrow \infty }\biggl( \int _{ \mathbb{R}^{3}} \bigl\vert \nabla u^{+}_{n} \bigr\vert ^{2}\,dx\biggr) \biggr]^{2}+\frac{bt^{2}}{4} \biggl[\liminf_{n\rightarrow \infty }\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-}_{n} \bigr\vert ^{2}\,dx\biggr) \biggr]^{2} \\ &\qquad{}+\frac{bs^{2}t^{2}}{4} \liminf_{n\rightarrow \infty }\biggl( \int _{ \mathbb{R}^{3}} \bigl\vert \nabla u^{+}_{n} \bigr\vert ^{2}\,dx\biggr)\cdot \liminf_{n\rightarrow \infty }\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{-}_{n} \bigr\vert ^{2}\,dx\biggr) \\ &\qquad{}+\frac{\lambda s^{4}}{4}\liminf_{n\rightarrow \infty } \int _{ \mathbb{R}^{3}}\phi _{u_{n}^{+}} \bigl\vert u_{n}^{+} \bigr\vert ^{2}\,dx + \frac{\lambda t^{4}}{4}\liminf_{n\rightarrow \infty } \int _{ \mathbb{R}^{3}}\phi _{u_{n}^{-}} \bigl\vert u_{n}^{-} \bigr\vert ^{2}\,dx \\ &\qquad{}-\frac{s^{6}}{6}\lim_{n\rightarrow \infty }\bigl( \bigl\vert u^{+}_{n}-u_{b}^{+} \bigr\vert _{6}^{6}+ \bigl\vert u_{b}^{+} \bigr\vert _{6}^{6}\bigr)- \frac{t^{6}}{6}\lim_{n\rightarrow \infty }\bigl( \bigl\vert u^{-}_{n}-u_{b}^{-} \bigr\vert _{6}^{6}+ \bigl\vert u_{b}^{-} \bigr\vert _{6}^{6}\bigr) \\ &\qquad{}- k \int _{\mathbb{R}^{3}}F\bigl(su_{b}^{+}\bigr)\,dx- k \int _{\mathbb{R}^{3}}F\bigl(tu_{b}^{-}\bigr)\,dx \\ &\qquad{}+\frac{\lambda s^{2}t^{2}}{4}\liminf_{n\rightarrow \infty } \int _{ \mathbb{R}^{3}}\phi _{u_{n}^{+}} \bigl\vert u_{n}^{-} \bigr\vert ^{2}\,dx + \frac{\lambda s^{2}t^{2}}{4}\liminf_{n\rightarrow \infty } \int _{ \mathbb{R}^{3}}\phi _{u_{n}^{-}} \bigl\vert u_{n}^{+} \bigr\vert ^{2}\,dx. \end{aligned}$$

By using Fatou’s lemma, there holds

$$\begin{aligned} \liminf_{n\rightarrow \infty }J^{b }_{ k} \bigl(su^{+}_{n}+tu^{-}_{n}\bigr)& \geq J^{b }_{ k}\bigl(su_{b}^{+}+tu_{b}^{-}\bigr)+ \frac{s^{2}}{2}\lim_{n \rightarrow \infty } \bigl\Vert u^{+}_{n}-u_{b}^{+} \bigr\Vert ^{2} +\frac{t^{2}}{2}\lim_{n \rightarrow \infty } \bigl\Vert u^{-}_{n}-u_{b}^{-} \bigr\Vert ^{2} \\ &\quad{}-\frac{s^{6}}{6}\lim_{n\rightarrow \infty } \bigl\vert u^{+}_{n}-u_{b}^{+} \bigr\vert _{6}^{6}- \frac{t^{6}}{6}\lim_{n\rightarrow \infty } \bigl\vert u^{-}_{n}-u_{b}^{-} \bigr\vert _{6}^{6} \\ &=J^{b }_{ k}\bigl(su_{b}^{+}+tu_{b}^{-}\bigr)+ \frac{s^{2}}{2}A_{1} - \frac{s^{6}}{6}B_{1}+ \frac{t^{2}}{2}A_{2} -\frac{t^{6}}{6}B_{2}, \end{aligned}$$

where

$$\begin{aligned} &A_{1}=\lim_{n\rightarrow \infty } \bigl\Vert u^{+}_{n}-u_{b}^{+} \bigr\Vert ^{2},\qquad A_{2}= \lim_{n\rightarrow \infty } \bigl\Vert u^{-}_{n}-u_{b}^{-} \bigr\Vert ^{2}, \\ &B_{1}=\lim_{n\rightarrow \infty } \bigl\vert u^{+}_{n}-u_{b}^{+} \bigr\vert _{6}^{6},\qquad B_{2}= \lim _{n\rightarrow \infty } \bigl\vert u^{-}_{n}-u_{b}^{-} \bigr\vert _{6}^{6}. \end{aligned}$$

From the above fact, one has that

$$\begin{aligned} &J^{b }_{ k}\bigl(su_{b}^{+}+tu_{b}^{-} \bigr)+\frac{s^{2}}{2}A_{1} - \frac{s^{6}}{6}B_{1}+ \frac{t^{2}}{2}A_{2} -\frac{t^{6}}{6}B_{2}\leq c_{ b }^{ k} \end{aligned}$$
(2.18)

for all \(s \geq 0\), \(t \geq 0\).

Claim 1. \(u_{b}^{\pm }\neq 0\). In fact, by contradiction, if \(u_{b}^{+}=0\), we divide it into two cases.

Case 1:\(B_{1}=0\). In this case, if \(A_{1}=0\), in view of the fact (2.15), we obtain \(\|u_{b}^{+}\|>0\), which is absurd. If \(A_{1}>0\), we let \(t=0\) in (2.18) that \(\frac{s^{2}}{2}A_{1}\leq c_{b }^{ k}\) for all \(s\geq 0\), which is false.

Case 2:\(B_{1}>0\). In this case, by the definition of S, we deduce

$$\begin{aligned} \beta =\frac{(S)^{\frac{3}{2}}}{3}\leq \frac{1}{3} \biggl( \frac{A_{1}}{(B_{1})^{\frac{1}{3}}} \biggr)^{\frac{3}{2}}. \end{aligned}$$

On the other hand,

$$\begin{aligned} \frac{1}{3} \biggl(\frac{A_{1}}{(B_{1})^{\frac{1}{3}}} \biggr)^{ \frac{3}{2}}=\max _{s\geq 0} \biggl\{ \frac{s^{2}}{2}A_{1}- \frac{s^{6}}{6}B_{1} \biggr\} . \end{aligned}$$

Thanks to \(c_{b }^{ k}<\beta \), by substituting \(t=0\) into (2.18), we have that

$$\begin{aligned} \beta \leq \max_{s\geq 0}\biggl\{ \frac{s^{2}}{2}A_{1}- \frac{s^{6}}{6}B_{1} \biggr\} \le c_{b }^{ k}< \beta , \end{aligned}$$

which is a contradiction. Thus \(u_{b}^{+}\neq 0\). Similarly, we also get \(u_{b}^{-}\neq 0\). Therefore \(u_{b}^{\pm }\neq 0\) as claimed.

Claim 2. \(B_{1}=B_{2}=0\). We only prove \(B_{1}=0\). By contradiction, we suppose that \(B_{1}>0\). We have two cases.

Case 1:\(B_{2}>0\). Let \(s_{a}\) and \(t_{b}\) be the numbers such that

$$\begin{aligned}& \frac{s_{a}^{2}}{2}A_{1}-\frac{s_{a}^{6}}{6}B_{1} =\max _{s\geq 0}\biggl\{ \frac{s^{2}}{2}A_{1}- \frac{s^{6}}{6}B_{1}\biggr\} , \\& \frac{t_{b}^{2}}{2}A_{2}-\frac{t_{b}^{6}}{6}B_{2} =\max _{t\geq 0}\biggl\{ \frac{t^{2}}{2}A_{2}- \frac{t^{6}}{6}B_{2}\biggr\} . \end{aligned}$$

Since \(\psi _{u}\) is continuous, we have \((s_{u},t_{u})\in [0, s_{a}]\times [0, t_{b}]\) satisfying

$$\begin{aligned} \psi _{u}(s_{u},t_{u})=\max _{(s,t)\in [0, s_{a}]\times [0, t_{b}]} \psi _{u}(s,t). \end{aligned}$$

Note that if \(0< t\ll 1\), we deduce

$$\begin{aligned} \psi _{u}(s,0)=J^{b}_{ k}\bigl(su_{b}^{+} \bigr)< J^{b }_{ k}\bigl(su_{b}^{+}\bigr)+J^{ b }_{ k} \bigl(tu_{b}^{-}\bigr)\leq J^{b }_{ k} \bigl(su_{b}^{+}+tu_{b}^{-}\bigr)=\psi _{u}(s,t) \end{aligned}$$

for all \(s\in [0, s_{a}]\). Thus there is \(t_{0}\in [0, t_{b}]\) such that

$$ \psi _{u}(s,0)\leq \psi _{u}(s,t_{0}) $$

for all \(s\in [0,s_{a}]\). It follows that any point of the form \((s, 0)\) with \(0\leq s\leq s_{a}\) is not the maximizer of \(\psi _{u}\). Thus, \((s_{u},t_{u})\notin [0, s_{a}]\times \{0\}\). Similarly, it shows that \((s_{u},t_{u})\notin \{0\}\times [0,t_{b}]\). By direct computation, we get

$$\begin{aligned}& \frac{s^{2}}{2}A_{1} -\frac{s^{6}}{6}B_{1}>0, \end{aligned}$$
(2.19)
$$\begin{aligned}& \frac{t^{2}}{2}A_{2} -\frac{t^{6}}{6}B_{2}>0 \end{aligned}$$
(2.20)

for all \(s\in (0,s_{a}]\), \(t\in (0,t_{b}]\). Hence there hold

$$\begin{aligned}& \beta \leq \frac{s_{a}^{2}}{2}A_{1}-\frac{s_{a}^{6}}{6}B_{1} + \frac{t^{2}}{2}A_{2}-\frac{t^{6}}{6}B_{2}, \\& \beta \leq \frac{t_{b}^{2}}{2}A_{2}-\frac{t_{b}^{6}}{6}B_{2} + \frac{s^{2}}{2}A_{1}-\frac{s^{6}}{6}B_{1} \end{aligned}$$

for all \(s\in [0,s_{a}]\), \(t\in [0,t_{b}]\). In view of (2.18), it follows that

$$\begin{aligned} \psi _{u}(s,t_{b})\leq 0,\qquad \psi _{u}(s_{a},t) \leq 0 \end{aligned}$$

for all \(s\in [0,s_{a}]\), \(t\in [0,t_{b}]\). That is, \((s_{u},t_{u})\notin \{s_{a}\}\times [0,t_{b}]\) and \((s_{u},t_{u})\notin \times [0,s_{a}]\times \{t_{b}\}\). Hence, we can deduce that \((s_{u},t_{u})\in (0, s_{a})\times (0, t_{b})\). By Lemma 2.1, it follows that \((s_{u},t_{u})\) is a critical point of \(\psi _{u}\). Thus, \(s_{u}u^{+}+t_{u}u^{-}\in \mathcal{M}_{k}^{b }\). By (2.18), (2.19), and (2.20), we deduce

$$\begin{aligned} c_{b }^{ k}&\geq J^{b }_{ k} \bigl(s_{u}u_{b}^{+}+t_{u}u_{b}^{-}\bigr)+ \frac{s_{u}^{2}}{2}A_{1}-\frac{s_{u}^{6}}{6}B_{1} + \frac{t_{u}^{2}}{2}A_{2}-\frac{t^{6}_{u}}{6}B_{2} \\ &>J^{b }_{ k}\bigl(s_{u}u_{b}^{+}+t_{u}u_{b}^{-} \bigr) \\ &\geq c_{b }^{ k}. \end{aligned}$$

It is impossible. The proof of Case 1 is completed.

Case 2:\(B_{2}= 0\). From the definition of \(J^{b }_{ k}\), it is easy to show that there exists \(t_{0}\in [0,\infty )\) such that \(J^{b }_{ k}(su_{b}^{+}+tu_{b}^{-})\leq 0\) for all \((s,t)\in [0,s_{a}]\times [t_{0},\infty )\). Thus, there is \((s_{u},t_{u})\in [0,s_{a}]\times [0,\infty )\) satisfying

$$\begin{aligned} \psi _{u}(s_{u},t_{u})=\max _{(s,t)\in [0,s_{a}]\times [0,\infty )} \psi _{u}(s,t). \end{aligned}$$

We need to prove that \((s_{u},t_{u})\in (0,s_{a})\times (0,\infty )\). Similarly, it is noticed that \(\psi _{u}(s,0)<\psi _{u}(s,t)\) for \(s\in [0,s_{a}]\) and \(0< t\ll 1\), that is, \((s_{u},t_{u})\notin [0,s_{a}]\times \{0\}\). Also, for s small enough, we get \(\psi _{u}(0,t)<\psi _{u}(s,t)\) for \(t\in [0,\infty )\), that is, \((s_{u},t_{u})\notin \{0\}\times [0,\infty )\). We note that

$$\begin{aligned} \beta &\leq \frac{s_{a}^{2}}{2}A_{1}-\frac{s_{a}^{6}}{6}B_{1}+ \frac{t^{2}}{2}A_{2} \end{aligned}$$

for all \(t\in [0,\infty )\). Thus also from (2.20) and \(B_{2}=0\), we have \(\psi _{u}(s_{a},t)\leq 0\) for all \(t\in [0,\infty )\). Hence, \((s_{u},t_{u})\notin \{s_{a}\}\times [0,\infty )\). That is, \((s_{u},t_{u})\) is an inner maximizer of \(\psi _{u}\) in \([0,s_{a})\times [0,\infty )\). So \(s_{u}u^{+}+t_{u}u^{-}\in \mathcal{M}_{k}^{b }\). Hence, by using (2.19), we obtain

$$\begin{aligned} c_{b }^{ k}&\geq J^{b }_{ k} \bigl(s_{u}u_{b}^{+}+t_{u}u_{b}^{-}\bigr)+ \frac{s_{u}^{2}}{2}A_{1}-\frac{s_{u}^{6}}{6}B_{1} + \frac{t_{u}^{2}}{2}A_{2}-\frac{t^{6}_{u}}{6}B_{2} \\ &>J^{b }_{ k}\bigl(s_{u}u_{b}^{+}+t_{u}u_{b}^{-} \bigr) \\ &\geq c_{b }^{ k}, \end{aligned}$$

which is a contradiction. It is similar for \(B_{2}=0\). From the above discussion, we know that Claim 2 is true.

Claim 3.\(c^{ k}_{b }\) is achieved. Since \(u_{b}^{\pm }\neq 0\), by Lemma 2.1, there are \(s_{u}, t_{u}>0\) such that \(\widetilde{u}:=s_{u}u_{b}^{+}+t_{u}u_{b}^{-}\in \mathcal{M}^{ k}_{b }\). On the other hand, \(u_{n}\rightharpoonup u_{b}\) in E, then \(\int _{\mathbb{R}^{3}}(V(x)u_{n}^{2})\,dx\rightarrow \int _{\mathbb{R}^{3}}(V(x)u_{b}^{2})\,dx\) and \(\liminf_{n\rightarrow \infty }\|u_{n}\|\ge \|u_{b}\|\), so we get

$$\begin{aligned} \liminf_{n\rightarrow \infty } \int _{\mathbb{R}^{3}} \vert \nabla u_{n} \vert ^{2}\,dx \geq \int _{\mathbb{R}^{3}} \vert \nabla u_{b} \vert ^{2}\,dx. \end{aligned}$$

On the other hand, by (2.4), we deduce

$$\begin{aligned} \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{3}}f\bigl(u_{n}^{\pm } \bigr)u_{n}^{ \pm }\,dx= \int _{\mathbb{R}^{3}}f(u_{b})^{\pm }u_{b}^{\pm }\,dx. \end{aligned}$$

Thanks to Proposition 2.1, we get

$$\begin{aligned} \bigl\langle \bigl(J^{b }_{ k}\bigr)'(u_{b}),u_{b}^{\pm } \bigr\rangle & \leq \liminf_{n\rightarrow \infty } \bigl\Vert u^{\pm }_{n} \bigr\Vert ^{2}+b \biggl[ \liminf_{n\rightarrow \infty }\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{\pm }_{n} \bigr\vert ^{2}\,dx\biggr) \biggr]^{2}\\ &\quad {}+b\liminf _{n\rightarrow \infty }\biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u^{+}_{n} \bigr\vert ^{2}\,dx\biggr)\cdot \liminf_{n\rightarrow \infty }\biggl( \int _{ \mathbb{R}^{3}} \bigl\vert \nabla u^{-}_{n} \bigr\vert ^{2}\,dx\biggr) \\ &\quad{}+\liminf_{n\rightarrow \infty } \int _{\mathbb{R}^{3}}\phi _{u_{n}} \bigl\vert u_{n}^{ \pm } \bigr\vert ^{2}\,dx -\lim _{n\rightarrow \infty } \int _{\mathbb{R}^{3}}f\bigl(u_{n}^{ \pm } \bigr)u_{n}^{\pm }\,dx -\lim_{n\rightarrow \infty } \int _{\mathbb{R}^{3}} \bigl\vert u_{n}^{ \pm } \bigr\vert ^{6} \\ & \leq \lim_{n\rightarrow \infty }\bigl\langle \bigl(J^{b }_{ k} \bigr)'(u_{n}),u_{n}^{ \pm }\bigr\rangle =0. \end{aligned}$$

Therefore from Lemma 2.2 we have that \(0< s_{u}\), \(t_{u}\leq 1\). Since \(u_{n}\in \mathcal{M}^{ k}_{b }\), \(B_{1}=B_{2}=0\) and \(\|u\|\) is lower semicontinuous, it follows that

$$\begin{aligned} c_{b }^{ k}&\leq J^{b }_{ k}( \widetilde{u}))-\frac{1}{4} \bigl\langle \bigl(J^{b }_{ k} \bigr)'(\widetilde{u}),\widetilde{u}\bigr\rangle \\ &=\frac{1}{4} \Vert \widetilde{u} \Vert ^{2}+ \frac{1}{12} \vert \widetilde{u} \vert _{6}^{6}+ \frac{ k}{4} \int _{\mathbb{R}^{3}}\bigl[f(\widetilde{u})\widetilde{u}-4F( \widetilde{u})\bigr]\,dx \\ &=\frac{1}{4}\bigl( \bigl\Vert s_{u_{b}}u_{b}^{+} \bigr\Vert ^{2}+ \bigl\Vert t_{u_{b}}u_{b}^{-} \bigr\Vert ^{2}\bigr)+\frac{1}{12}\bigl( \bigl\vert s_{u_{b}}u_{b}^{+} \bigr\vert _{6}^{6}+ \bigl\vert t_{u_{b}}u_{b}^{-} \bigr\vert _{6}^{6}\bigr) \\ &\quad{}+\frac{ k}{4} \int _{\mathbb{R}^{3}}\bigl[f\bigl(s_{u_{b}}u_{b}^{+}\bigr) \bigl(s_{u_{b}}u_{b}^{+}\bigr)-4F\bigl(s_{u_{b}}u_{b}^{+} \bigr)\bigr]\,dx\\ &\quad {}+ \frac{ k}{4} \int _{\mathbb{R}^{3}}\bigl[f\bigl(x,t_{u_{b}}u_{b}^{-}\bigr) \bigl(t_{u_{b}}u_{b}^{-}\bigr)-4F\bigl(x,t_{u_{b}}u_{b}^{-} \bigr)\bigr]\,dx. \end{aligned}$$

By using \(0< s_{u_{b}}\), \(t_{u_{b}}\leq 1\), \(f(t)t-4F(t)\) is increasing in \((0,+\infty )\) and decreasing in \((-\infty ,0)\), we have

$$\begin{aligned} c_{b }^{ k}&\leq \frac{1}{4} \Vert u_{b } \Vert ^{2}+\frac{1}{12} \vert u_{b } \vert _{6}^{6}+\frac{ k}{4} \int _{\mathbb{R}^{3}}\bigl[f(u_{b })u_{b }-4F(u_{b }) \bigr]\,dx \\ &\leq \liminf_{n\rightarrow \infty }\biggl[J^{b }_{ k}(u_{n})- \frac{1}{4}\bigl\langle \bigl(J^{b }_{ k} \bigr)'(u_{n}),u_{n}\bigr\rangle \biggr] \\ &= \liminf_{n\rightarrow \infty }J^{b }_{ k}(u_{n}) \\ &= c_{b }^{ k}. \end{aligned}$$

Therefore the infimum \(c^{ k}_{b }\) is achieved by \(u_{b }=u_{b}^{+}+u_{b}^{-}\in \mathcal{M}^{ k}_{b }\). □

3 The proof of the main results

In this section, we prove the main results. Firstly, we prove Theorem 1.1. In fact, thanks to Lemma 2.4, we should prove that the minimizer \(u_{b }\) for \(c^{ k}_{b }\) is indeed a nodal solution of system (1.7), but \(\mathcal{M}_{k}^{b }\) is not a smooth manifold, we will apply a new method to complete our certification.

3.1 The proof of Theorem 1.1

Proof

Since \(u_{b }\in \mathcal{M}^{ k}_{b }\) and \(J^{b }_{ k}(u_{b }^{+}+u_{b }^{-}) =c^{ k}_{b }\), we have \(\langle (J^{b }_{ k})'(u_{b }),u_{b }^{+}\rangle = \langle (J^{b }_{ k})'(u_{b }),u_{b }^{-}\rangle =0\). By Lemma 2.1, for \((s,t)\in (\mathbb{R}_{+}\times \mathbb{R}_{+})\backslash (1,1)\), we have

$$\begin{aligned} J^{b }_{ k}\bigl(su_{b }^{+}+tu_{b }^{-} \bigr)< J^{b }_{ k}\bigl(u_{ b }^{+}+u_{b }^{-} \bigr) =c^{ k}_{b }. \end{aligned}$$
(3.1)

If \((J^{b }_{ k})'(u_{b })\neq 0\), then there exist \(\delta >0\) and \(\theta >0\) such that

$$ \bigl\Vert \bigl(J^{b }_{ k}\bigr)'(v) \bigr\Vert \geq \theta \quad \text{for all } \Vert v-u_{ b } \Vert \leq 3\delta . $$

We know by result (2.15), if \(u\in \mathcal{M}_{k}^{b }\), there exists \(L>0\) such that \(\|u_{b}^{\pm }\|>L\), and we can assume \(6\delta < L\). Let \(Q:=(\frac{1}{2},\frac{3}{2})\times (\frac{1}{2},\frac{3}{2})\) and \(g(s,t)=su_{b }^{+}+tu_{b }^{-}\), \((s,t)\in Q\). In view of (3.1), it is easy to see that

$$\begin{aligned} \overline{c}^{ k}_{b }:=\max _{\partial Q}I\circ g< c^{ k}_{ b }. \end{aligned}$$
(3.2)

Let \(\varepsilon :=\min \{(c^{ k}_{b }-\overline{c}^{ k}_{b })/4, \theta \delta /8\}\) and \(S_{\delta }:=B(u_{b },\delta )\), according to Lemma 2.3 of [24], there exists a deformation \(\eta \in C([0,1]\times E,E)\) satisfying

  1. (a)

    \(\eta (t,v)=v\) if \(t=0\), or \(v\notin (J^{b }_{ k})^{-1}([c^{ k}_{b }-2\varepsilon ,c^{ k}_{ b }+2\varepsilon ])\cap S_{2\delta }\);

  2. (b)

    \(\eta (1,(J^{b }_{ k})^{c^{ k}_{b }+\varepsilon }\cap S_{ \delta })\subset (J^{b }_{ k})^{c^{ k}_{b }-\varepsilon }\);

  3. (c)

    \(J^{b }_{ k}(\eta (1,v))\leq J^{b }_{ k}(v)\) for all \(v\in E\);

  4. (d)

    \(J^{b }_{ k}(\eta (\cdot ,v))\) is nonincreasing for every \(v\in E\).

We remind that, for a functional \(\varPhi :E\to \mathbb{R}\), the level set \(\varPhi ^{\mu}\) is defined by \(\varPhi ^{\mu}=\{u\in E:\varPhi (u)\le \mu\}\). Firstly, we need to prove that

$$\begin{aligned} \max_{(s,t)\in \bar{Q}}J^{b }_{ k}\bigl( \eta \bigl(1,g(s,t)\bigr)\bigr)< c^{ k}_{ b }. \end{aligned}$$
(3.3)

In fact, it follows from Lemma 2.1 that \(J^{b }_{ k}(g(s,t))\leq c^{ k}_{b }< c^{ k}_{b }+ \varepsilon \). That is,

$$ g(s,t)\in \bigl(J^{b }_{ k}\bigr)^{c^{ k}_{b }+\varepsilon }. $$

On the other hand, from (a) and (d), we get

$$\begin{aligned} J^{b }_{ k}\bigl(\eta (1,v)\bigr)\leq J^{b }_{ k}\bigl(\eta (0,v)\bigr)=J^{ b }_{ k}(v), \quad \forall v\in E. \end{aligned}$$
(3.4)

For \((s,t)\in Q\), when \(s\neq 1\) or \(t\neq 1\), according to (3.1) and (3.4),

$$\begin{aligned} J^{b }_{ k}\bigl(\eta \bigl(1,g(s,t)\bigr)\bigr)\leq J^{b}_{ k}\bigl(g(s,t)\bigr)< c^{ k}_{ b }. \end{aligned}$$

If \(s= 1\) and \(t= 1\), that is, \(g(1,1)=u_{b }\), so that it holds \(g(1,1)\in (J^{b }_{ k})^{c^{ k}_{b }+\varepsilon }\cap S_{ \delta }\), then by (b)

$$\begin{aligned} J^{b }_{ k}\bigl(\eta \bigl(1,g(1,1)\bigr)\bigr)\leq c^{ k}_{b }-\varepsilon < c^{ k}_{b }. \end{aligned}$$

Thus (3.3) holds. In the following, we prove that \(\eta (1,g(Q))\cap \mathcal{M}^{ k}_{b }\neq \varnothing \), which contradicts the definition of \(c^{ k}_{b }\). Let \(\varphi (s,t):=\eta (1,g(s,t))\) and

$$\begin{aligned} \varPsi (s,t):=\biggl(\frac{1}{s}\bigl\langle \bigl(J^{b }_{ k} \bigr)'\bigl(\varphi (s,t)\bigr),\bigl( \varphi (s,t) \bigr)^{{+}}\bigr\rangle , \frac{1}{t}\bigl\langle \bigl(J^{b }_{ k}\bigr)'\bigl( \varphi (s,t) \bigr),\bigl(\varphi (s,t)\bigr)^{{-}}\bigr\rangle \biggr). \end{aligned}$$

The claim holds if there exists \((s_{0},t_{0})\in Q\) such that \(\varPsi (s_{0},t_{0})=(0,0)\). Since

$$\begin{aligned} \bigl\Vert g(s,t)-u_{b } \bigr\Vert ^{2} &= \bigl\Vert (s-1)u_{b }^{+}+(t-1)u_{b }^{-} \bigr\Vert ^{2} \\ &\geq \vert s-1 \vert ^{2} \bigl\Vert u_{b }^{+} \bigr\Vert ^{2} \\ &> \vert s-1 \vert ^{2}(6\delta )^{2}, \end{aligned}$$

and \(|s-1|^{2}(6\delta )^{2}>4\delta ^{2}\Leftrightarrow s<2/3\) or \(s>4/3\), using the item (a) above and the range of s, for \(s=\frac{1}{2}\) and for every \(t\in [\frac{1}{2},\frac{3}{2}]\), we have \(g(\frac{1}{2},t)\notin S_{2\delta }\). So from (a) we have \(\varphi (\frac{1}{2},t)=g(\frac{1}{2},t)\). Thus

$$\begin{aligned} \varPsi \biggl(\frac{1}{2},t\biggr)=\large \biggl(2\biggl\langle \bigl(J^{b }_{ k}\bigr)'\biggl( \frac{1}{2}u_{b }^{+}+t u_{b }^{-} \biggr),\frac{1}{2}u_{b }^{+}\biggr\rangle , \frac{1}{t}\biggl\langle \bigl( J^{b }_{ k} \bigr)'\biggl(\frac{1}{2}u_{b }^{+}+t u_{b }^{-}\biggr),tu^{-}\biggr\rangle \large \biggr). \end{aligned}$$

By Lemma 2.2, we know that

$$\begin{aligned} &\biggl\langle \bigl(J^{b }_{ k}\bigr)'\biggl( \frac{1}{2}u_{b }^{+}+t u_{b }^{-} \biggr), \frac{1}{2}u_{b }^{+}\biggr\rangle \\ &\quad =\biggl\langle \bigl(J^{b }_{ k}\bigr)'\biggl( \frac{1}{2}u_{b }^{+}\biggr),\frac{1}{2}u_{b }^{+} \biggr\rangle + \frac{t^{2}b}{4} \int _{\mathbb{R}^{3}} \bigl\vert \nabla u_{b }^{-} \bigr\vert ^{2}\,dx \cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla u_{b }^{+} \bigr\vert ^{2}\,dx +\frac{t^{2}b }{4} \int _{\mathbb{R}^{3}}\phi _{u_{b }^{-}} \bigl\vert u_{b }^{+} \bigr\vert ^{2}\,dx \\ &\quad \geq \biggl\langle \bigl(J^{b }_{ k}\bigr)'\biggl( \frac{1}{2}u_{b }\biggr),\frac{1}{2}u_{b }^{+} \biggr\rangle >0, \end{aligned}$$

from which we obtain

$$\begin{aligned} \biggl\langle \bigl(J^{b }_{ k} \bigr)'\biggl(\frac{1}{2}u_{b}^{+}+t u_{b}^{-} \biggr),\frac{1}{2}u_{b}^{+} \biggr\rangle >0\quad \text{for every } t\in \biggl[\frac{1}{2},\frac{3}{2}\biggr]. \end{aligned}$$
(3.5)

Similarly, for \(s=\frac{3}{2}\) and for every \(t\in [\frac{1}{2},\frac{3}{2}]\), we have \(\varphi (\frac{3}{2},t)=g(\frac{3}{2},t)\), so that

$$\begin{aligned} &\biggl\langle \bigl(J^{b }_{ k}\bigr)'\biggl( \frac{3}{2}u_{b}^{+}+t u_{b}^{-}\biggr),\frac{3}{2}u_{b}^{+} \biggr\rangle \\ &\quad =\biggl\langle \bigl(J^{b }_{ k} \bigr)'\biggl(\frac{3}{2}u_{b}^{+}\biggr), \frac{3}{2}u_{b}^{+} \biggr\rangle +\frac{9t^{2}}{4}b \int _{\mathbb{R}^{3}} \bigl\vert \nabla u_{b}^{-} \bigr\vert ^{2}\,dx \cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla u_{b}^{+} \bigr\vert ^{2}\,dx +\frac{9t^{2}}{4}b \int _{\mathbb{R}^{3}}\phi _{u_{b}^{-}} \bigl\vert u_{b}^{+} \bigr\vert ^{2}\,dx \\ &\quad \leq \biggl\langle \bigl(J^{b }_{ k} \bigr)'\biggl(\frac{3}{2}u_{b}\biggr),\frac{3}{2}u_{b}^{+} \biggr\rangle < 0, \end{aligned}$$

so that

$$\begin{aligned} \biggl\langle \bigl(J^{b }_{ k} \bigr)'\biggl(\frac{3}{2}u_{b}^{+}+t u_{b}^{-} \biggr),\frac{3}{2}u_{b}^{+} \biggr\rangle < 0\quad \text{for every } t\in \biggl[\frac{1}{2},\frac{3}{2}\biggr]. \end{aligned}$$
(3.6)

Similarly, we have

$$\begin{aligned} \biggl\langle \bigl(J^{b }_{ k} \bigr)'\biggl(s u_{b}^{+}+\frac{1}{2} u_{b}^{-} \biggr),\frac{1}{2}u_{b}^{-} \biggr\rangle >0\quad \text{for every } s\in \biggl[\frac{1}{2},\frac{3}{2}\biggr], \end{aligned}$$
(3.7)
$$\begin{aligned} \biggl\langle \bigl(J^{b }_{ k}\bigr)'\biggl(s u_{b}^{+}+\frac{3}{2} u_{b}^{-}\biggr),\frac{3}{2}u_{b}^{-} \biggr\rangle < 0\quad \text{for every } s\in \biggl[\frac{1}{2}, \frac{3}{2}\biggr]. \end{aligned}$$
(3.8)

Since Ψ is continuous on Q, according to (3.5)–(3.7), by Miranda’s theorem (Lemma 2.4 [7]), we have \(\varPsi (s_{0},t_{0})=0\) for some \((s_{0},t_{0})\in Q\), so \(\eta (1,g(s_{0},t_{0}))=\varphi (s_{0},t_{0})\in \mathcal{M}^{ k}_{ b }\). By (3.3), we have a contradiction. From the above discussion, we conclude that \(u_{b }\) is a nodal solution for system (1.7).

Finally, we prove that \(u_{b }\) has exactly two nodal domains. To this end, we first write \(u_{b }\) as

$$ u_{b }=u_{1}+u_{2}+u_{3} $$

with \(u_{1}\geq 0\), \(u_{2}\leq 0\). Set \(\varOmega _{i}=\{x\in \mathbb{R}^{3}:u_{i}(x)\ne 0\}\). We further assume \(\varOmega _{i}\cap \varOmega _{j}=\emptyset \) for \(i\neq j\), \(i,j=1,2,3\). Since \(u_{b }\) is a nodal solution, we suppose the nodal domains \(\varOmega _{1}\ne \emptyset \), \(\varOmega _{2}\ne \emptyset \). By contradiction, we suppose \(u_{b }\) possesses more than two nodal domains, then we have \(u_{3}\ne 0\) and so \(\varOmega _{3}\ne \emptyset \). Setting \(v:=u_{1}+u_{2}\), we easily see that \(v^{\pm }\neq 0\). So, there exists a positive pair \((s_{v},t_{v})\) such that

$$ s_{v}u_{1}+t_{v}u_{2}\in \mathcal{M}^{ k}_{b }. $$

Thus,

$$ J^{b }_{ k}(s_{v}u_{1}+t_{v}u_{2}) \geq c^{ k}_{b }. $$

Moreover, using the fact that \(\langle (J^{b }_{ k})'(u_{b }),u_{i} \rangle =0\), from the definition, we get \(\langle (J^{b }_{ k})'(v),v^{\pm } \rangle \le 0\). So, thanks to Lemma 2.1, we have that

$$ (s_{v},t_{v})\in (0,1]\times (0,1]. $$

By direct calculation,

$$\begin{aligned}& 0=\bigl\langle \bigl(J^{b }_{ k} \bigr)'(u_{b }),u_{3} \bigr\rangle \\& \hphantom{0}= \Vert u_{3} \Vert ^{2} +b \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}\,dx\cdot \int _{\mathbb{R}^{N}} \vert \nabla u_{3} \vert ^{2}\,dx \\& \hphantom{0}\quad{} +b \int _{\mathbb{R}^{3}} \vert \nabla u_{2} \vert ^{2}\,dx\cdot \int _{ \mathbb{R}^{N}} \vert \nabla u_{3} \vert ^{2}\,dx +b \biggl( \int _{\mathbb{R}^{N}} \vert \nabla u_{3} \vert ^{2}\,dx \biggr)^{2} \\& \hphantom{0}\quad{}+\lambda \int _{\mathbb{R}^{3}}\phi _{u_{1}} \vert u_{3} \vert ^{2}\,dx+ \frac{\lambda }{4} \int _{\mathbb{R}^{3}}\phi _{u_{2}} \vert u_{3} \vert ^{2}\,dx + \lambda \int _{\mathbb{R}^{3}}\phi _{u_{3}} \vert u_{3} \vert ^{2}\,dx \\& \hphantom{0}\quad{}- \lambda \int _{\mathbb{R}^{3}} \vert u_{3} \vert ^{6}\,dx- \frac{ k}{4} \int _{ \mathbb{R}^{3}}f(u_{3})u_{3}\,dx \\& \hphantom{0}=\bigl\langle \bigl(J^{b }_{ k}\bigr)'(u_{3}),u_{3} \bigr\rangle +b \int _{ \mathbb{R}^{3}}\bigl( \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}\bigr)\,dx \cdot \int _{ \mathbb{R}^{N}} \vert \nabla u_{3} \vert ^{2}\,dx \\& \hphantom{0}\quad{}+ \lambda \int _{\mathbb{R}^{3}}\phi _{u_{1}} \vert u_{3} \vert ^{2}\,dx+ \lambda \int _{\mathbb{R}^{3}}\phi _{u_{2}} \vert u_{3} \vert ^{2}\,dx \\& \hphantom{0}< 4J^{b }_{ k}(u_{3})+b \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}\,dx \cdot \int _{\mathbb{R}^{N}} \vert \nabla u_{3} \vert ^{2}\,dx+b \int _{\mathbb{R}^{3}} \vert \nabla u_{2} \vert ^{2}\,dx\cdot \int _{\mathbb{R}^{N}} \vert \nabla u_{3} \vert ^{2}\,dx \\& \hphantom{0}\quad{}+ \lambda \int _{\mathbb{R}^{3}}\phi _{u_{1}} \vert u_{3} \vert ^{2}\,dx+ \lambda \int _{\mathbb{R}^{3}}\phi _{u_{2}} \vert u_{3} \vert ^{2}\,dx, \end{aligned}$$
(3.9)
$$\begin{aligned}& \begin{aligned}[b] -\bigl\langle \bigl(J^{b }_{ k} \bigr)'(u_{3}),u_{3}\bigr\rangle &=b \int _{\mathbb{R}^{3}}\bigl( \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}\bigr)\,dx \cdot \int _{\mathbb{R}^{N}} \vert \nabla u_{3} \vert ^{2}\,dx\\ &\quad {}+ \lambda \int _{\mathbb{R}^{3}}\phi _{u_{1}} \vert u_{3} \vert ^{2}\,dx+ \lambda \int _{\mathbb{R}^{3}}\phi _{u_{2}} \vert u_{3} \vert ^{2}\,dx. \end{aligned} \end{aligned}$$
(3.10)

Then, by using (2.16), we get

$$\begin{aligned} c^{ k}_{b } &\leq J^{b }_{ k}(s_{v}u_{1}+t_{v}u_{2})= J^{b }_{ k}(s_{v}u_{1}+t_{v}u_{2}) -\frac{1}{4}\bigl\langle \bigl(J^{b }_{ k} \bigr)'(s_{v}u_{1}+t_{v}u_{2}), s_{v}u_{1}+t_{v}u_{2}\bigr\rangle \\ &=\frac{1}{4}\bigl( \Vert s_{v}u_{1} \Vert ^{2}+ \Vert t_{v}u_{2} \Vert ^{2} \bigr) +\frac{ k}{4} \int _{\mathbb{R}^{3}}\bigl[f(s_{v}u_{1}) (s_{v}u_{1}) -4F(s_{v}u_{1})\bigr]\,dx \\ &\quad{}+\frac{ k}{4} \int _{\mathbb{R}^{3}}\bigl[f(t_{v}u_{2}) (t_{v}u_{2}) -4F(t_{v}u_{2})\bigr]\,dx+ \frac{s^{6}_{v}}{12} \int _{\mathbb{R}^{3}} \vert u_{1} \vert ^{6}\,dx+ \frac{t^{6}_{v}}{12} \int _{\mathbb{R}^{3}} \vert u_{2} \vert ^{6}\,dx \\ &\leq \frac{1}{4}\bigl( \Vert u_{1} \Vert ^{2}+ \Vert u_{2} \Vert ^{2}\bigr) +\frac{ k}{4} \int _{ \mathbb{R}^{3}}\bigl[f(u_{1})u_{1} -4F(u_{1})\bigr]\,dx \\ &\quad{}+\frac{ k}{4} \int _{\mathbb{R}^{3}}\bigl[f(u_{2})u_{2} -4F(u_{2})\bigr]\,dx + \frac{1}{12} \int _{\mathbb{R}^{3}} \vert u_{1} \vert ^{6}\,dx+ \frac{1}{12} \int _{ \mathbb{R}^{3}} \vert u_{2} \vert ^{6}\,dx \\ &=J^{b }_{ k}(u_{1}+u_{2}) - \frac{1}{4}\bigl\langle \bigl(J^{b }_{ k} \bigr)'(u_{1}+u_{2}),(u_{1}+u_{2}) \bigr\rangle . \end{aligned}$$

Similar to the computation of (3.10), from \(\langle (J^{b}_{k })'(u_{b }),u_{b }\rangle =0\), there holds

$$\begin{aligned} &-\bigl\langle \bigl(J^{b}_{k }\bigr)' (u_{1}+u_{2}),u_{1}+u_{2}\bigr\rangle \\ &\quad = \bigl\langle \bigl(J^{b}_{k }\bigr)'(u_{3}),u_{3} \bigr\rangle +2b \int _{\mathbb{R}^{3}}\bigl( \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}\bigr)\,dx \cdot \int _{\mathbb{R}^{3}} \vert \nabla u_{3} \vert ^{2}\,dx \\ &\qquad{}+\lambda \int _{\mathbb{R}^{3}}(\phi _{u_{1}}+\phi _{u_{2}}) \vert u_{3} \vert ^{2}\,dx+ \lambda \int _{\mathbb{R}^{3}}\phi _{u_{3}}\bigl( \vert u_{1} \vert ^{2}+ \vert u_{2} \vert ^{2}\bigr)\,dx. \end{aligned}$$
(3.11)

By using (3.9), (3.10), and (3.11), we get

$$\begin{aligned} c^{ k}_{b }&\le J^{b }_{ k}(u_{1}+u_{2}) -\frac{1}{4} \bigl\langle \bigl(J^{b }_{ k} \bigr)'(u_{1}+u_{2}),(u_{1}+u_{2}) \bigr\rangle \\ &=J^{b }_{ k}(u_{1}+u_{2}) + \frac{1}{4}\bigl\langle \bigl(J^{b }_{ k} \bigr)'(u_{ b }),u_{3}\bigr\rangle + \frac{b}{4} \int _{\mathbb{R}^{3}} \vert \nabla u_{1} \vert ^{2}\,dx \cdot \int _{\mathbb{R}^{N}} \vert \nabla u_{3} \vert ^{2}\,dx \\ &\quad{}+\frac{b}{4} \int _{\mathbb{R}^{3}} \vert \nabla u_{2} \vert ^{2}\,dx\cdot \int _{ \mathbb{R}^{N}} \vert \nabla u_{3} \vert ^{2}\,dx +\frac{\lambda }{4} \int _{ \mathbb{R}^{3}}\phi _{u_{3}} \vert u_{1} \vert ^{2}\,dx+\frac{\lambda }{4} \int _{ \mathbb{R}^{3}}\phi _{u_{3}} \vert u_{2} \vert ^{2}\,dx \\ &< J^{b }_{ k}(u_{1})+J^{b }_{ k}(u_{2})+J^{b }_{ k}(u_{3}) +\frac{b}{4} \int _{\mathbb{R}^{N}}\bigl( \vert \nabla u_{2} \vert ^{2}+ \vert \nabla u_{3} \vert ^{2}\bigr)\,dx \cdot \int _{\mathbb{R}^{N}} \vert \nabla u_{1} \vert ^{2}\,dx \\ &\quad{}+\frac{b}{4} \int _{\mathbb{R}^{N}}\bigl( \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{3} \vert ^{2}\bigr)\,dx \cdot \int _{\mathbb{R}^{N}} \vert \nabla u_{2} \vert ^{2}\,dx \\ &\quad {}+\frac{b}{4} \int _{ \mathbb{R}^{N}} \bigl( \vert \nabla u_{1} \vert ^{2}+ \vert \nabla u_{2} \vert ^{2}\bigr)\,dx \cdot \int _{ \mathbb{R}^{N}} \vert \nabla u_{3} \vert ^{2}\,dx \\ &\quad{}+\frac{\lambda }{4} \int _{\mathbb{R}^{3}}\phi _{u_{3}}\bigl( \vert u_{1} \vert ^{2}+ \vert u_{2} \vert ^{2}\bigr)\,dx+ \frac{\lambda }{4} \int _{\mathbb{R}^{3}}\phi _{u_{1}+u_{2}} \vert u_{3} \vert ^{2}\,dx \\ &=J^{b }_{ k}(u_{b })=c^{ k}_{b }. \end{aligned}$$

So we get \(u_{3}=0\) and \(u_{b }\) has exactly two nodal domains. □

3.2 The proof of Theorem 1.2

To prove Theorem 1.2, we should first prove that there exists a ground state solution of (1.7) for k large enough, and then to prove that the energy of sign-changing solution \(u_{b }\) is strictly larger than twice of that of the ground state solution.

Proof

Similar to the proof of Lemma 2.4, we claim that there exists \(k_{1}^{\star }>0\) such that, for all \(k\geq k_{1}^{\star }\), and \(\forall b >0\), there exists \(v_{b }\in \mathcal{N}^{ k}_{b }\) such that \(J^{b }_{ k}(v_{b }) =c^{\ast }>0\). We give a brief proof of this claim.

We first list some results for the Nehari manifold \(\mathcal{N}^{ k}_{b }\). One can prove them by following the ideas as those in Lemma 2.4.

  1. (i)

    If \(v\in \mathcal{N}^{ k}_{b }\), then \(J^{b }_{ k}(tv)\le J^{b }_{ k}(v)\) for all \(t\ge 0\);

  2. (ii)

    There exists \(\rho >0\) such that \(\|v\|\ge \rho \) for all \(v \in \mathcal{N}^{ k}_{b }\);

  3. (iii)

    There exists \(M>0\) such that \(\|v\|\le M\) for all \(v \in \mathcal{N}^{ k}_{b }\).

According to the definition of \(c^{\ast }\), there is a sequence \(\{v_{n}\}\subset \mathcal{N}^{ k}_{b }\) such that \(\lim_{n\rightarrow \infty }J^{b }_{ k}(v_{n})=c^{\ast }\). By property (iii), \(\{v_{n}\}\) is bounded in E. In the subsequence sense, there exists \(v_{b }\in E\) such that \(v_{n}\rightharpoonup v_{b }\).

Denote \(\beta :=\frac{(S)^{\frac{3}{2}}}{3}\), where \(S:=\inf_{u\in E\backslash \{0\}} \frac{\|u\|^{2}}{(\int _{\mathbb{R}^{3}}|u|^{6}\,dx)^{\frac{1}{3}}}\). Similar to the proof of Lemma 2.3, there is \(k^{\star }>0\) such that \(c^{\ast }<\beta \) for all \(k\geq k^{\star }\). Therefore, \(\liminf_{n\rightarrow \infty }J^{b }_{ k}(tv_{n})\geq J^{ b }_{ k}(tv_{b })+\frac{t^{2}}{2}A-\frac{t^{6}}{6}B\), where \(A=\lim_{n\rightarrow \infty }\|v_{n}-v_{b }\|^{2}\), \(B=\lim_{n \rightarrow \infty }|v_{n}-v_{b }|_{6}^{6}\). From the above fact and property (i), we have

$$\begin{aligned} J^{b }_{ k}(tv_{b })+\frac{t^{2}}{2}A- \frac{t^{6}}{6}B\le c^{*} \end{aligned}$$
(3.12)

for all \(t \geq 0\).

Firstly, we prove that \(v_{b }\neq 0 \). By contradiction, we suppose \(v_{b }=0\).

Case 1:\(B=0\). If \(A=0\), that is, \(v_{n}\rightarrow v_{b } \) in E, then \(v_{b }\in \mathcal{N}^{ k}_{b }\), and so we have \(\|v_{b }\|>\rho \) by property (ii), which contradicts our supposition. If \(A>0\), \(\frac{t^{2}}{2}A\leq c^{\ast }\) for all \(t\geq 0\), which is a contradiction.

Case 2:\(B>0\). According to the definition of S, we have that \(\beta =\frac{(S)^{\frac{3}{2}}}{3}\leq \frac{1}{3}( \frac{A}{(B)^{\frac{1}{3}}})^{\frac{3}{2}}\). It is easy to see that

$$ \frac{1}{3}\biggl(\frac{A}{(B)^{\frac{1}{3}}}\biggr)^{\frac{3}{2}}= \frac{\tilde{t}^{2}}{2}A-\frac{\tilde{t}^{6}}{6}B:=\max_{t\geq 0}\biggl\{ \frac{t^{2}}{2}A-\frac{t^{6}}{6}B\biggr\} , $$

so we have that

$$ \beta \leq \max_{t\geq 0}\biggl\{ \frac{t^{2}}{2}A- \frac{t^{6}}{6}B\biggr\} \le c_{ b }^{ k}< \beta , $$

which is a contradiction.

Secondly, we claim that \(B=0\). By contradiction, we suppose that \(B>0\). Firstly, we can maximize \(\psi _{v_{b }}(t)=J^{b }_{ k}(tv_{b })\) in \([0,\infty )\). Indeed, there exists \(t_{0}\in [0,\infty )\) such that \(J^{b }_{ k}(tv_{b })\leq 0\) for all \(t\in [t_{0},\infty )\). Let \(t_{v}\) be an inner maximizer of \(\psi _{v}\) in \([0,\infty )\). \(J^{b }_{ k}(\tilde{t}v_{b })+\beta \le J^{b }_{ k}( \tilde{t}v_{b })+\frac{\tilde{t}^{2}}{2}A- \frac{\tilde{t}^{6}}{6}B\le c^{*}<\beta \) implies that \(J^{b }_{ k}(\tilde{t}v_{b })<0\). So \(t_{v}\le \tilde{t}\) and \(\frac{t_{v}^{2}}{2}A-\frac{t_{v}^{6}}{6}B>0\). Thus from \(t_{v}v_{b }\in \mathcal{N}_{b }^{ k}\) we get a contradiction by

$$ c^{*}\le J^{b }_{ k}(t_{v}v_{b })< J^{b }_{ k}(t_{v}v_{ b})+ \frac{t_{v}^{2}}{2}A-\frac{t_{v}^{6}}{6}B\le c^{*}. $$

Lastly, we prove that \(c^{\ast }\) is achieved by \(v_{b}\). From the above arguments, we have \(v_{b}\neq 0\) and \(\widetilde{v}:=t_{v}v_{b }\in \mathcal{ N}^{ k}_{b }\). Furthermore, because \(v_{n}\rightharpoonup v_{b }\) in E and \(v_{n}\in \mathcal{N}^{ k}_{b }\), we have that \(\langle (J^{b }_{ k})'(v_{b }),v_{b }\rangle \le 0\). Similar to Lemma 2.1, we have \(0< t_{v}\le 1\). Also as in the proof of Lemma 2.4, we have

$$\begin{aligned}& \begin{aligned} c^{*}&\le J^{b }_{ k}(\tilde{v})- \frac{1}{4} \bigl\langle \bigl(J^{b }_{ k} \bigr)'( \tilde{v}),\tilde{v}\bigr\rangle \\ &=\frac{1}{4} \Vert t_{v}v_{b } \Vert ^{2}+\frac{1}{12} \vert t_{v}v_{b } \vert _{6}^{6}+ \frac{k}{4} \int _{\mathbb{R}^{3}}\bigl[f(t_{v}v_{b })t_{v}v_{b }-4F(t_{v}v_{b }) \bigr]\,dx \\ & \le \frac{1}{4} \Vert v_{b } \Vert ^{2}+ \frac{1}{12} \vert v_{b } \vert _{6}^{6}+ \frac{k}{4} \int _{\mathbb{R}^{3}}\bigl[f(v_{b })v_{b }-4F(v_{b }) \bigr]\,dx, \end{aligned} \\& \lim \inf_{n\to \infty }\biggl[J^{b }_{ k}(v_{n})- \frac{1}{4} \bigl\langle \bigl(J^{ b }_{ k} \bigr)'(v_{n}),v_{n}\bigr\rangle \biggr]=c^{*}. \end{aligned}$$

Therefore, \(t_{v}=1\), and \(c^{\ast }\) is achieved by \(v_{b }\in \mathcal{N}^{ k}_{b }\).

By standard arguments, the critical points of the functional \(J^{b }_{ k}\) on \(\mathcal{N}^{ k}_{b }\) are critical points of \(J^{b }_{ k}\) in E, and we obtain \((J^{b }_{ k})'(v_{b }) =0\), so \(v_{b }\) is a positive or negative solution. That is, \(v_{b }\) is a ground state solution of system (1.7). For all \(k\geq k^{\star }\), and \(\forall b >0\), problem (1.7) has a least energy nodal solution \(u_{b }\). Let

$$ k^{\star \star }=\max \bigl\{ k^{\star }, k^{\star }_{1} \bigr\} . $$

Suppose that \(u_{b }=u^{+}+u^{-}\). As in the proof of Lemma 2.1, there exist \(s_{u^{+}},t_{u^{-}}\in (0,1)\) such that

$$ s_{u^{+}}u^{+}\in \mathcal{N}^{ k}_{b }, \quad t_{u^{-}}u^{-} \in \mathcal{N}^{ k}_{b }. $$

Hence, by Lemma 2.1, we deduce

$$\begin{aligned} 2c^{\ast }\leq J^{b }_{ k}\bigl(s_{u^{+}}u^{+} \bigr)+J^{b }_{ k}\bigl(t_{u^{-}}u^{-}\bigr) \leq J^{b }_{ k}\bigl(s_{u^{+}}u^{+}+t_{u^{-}}u^{-} \bigr)< J^{b }_{ k}\bigl(u^{+}+u^{-} \bigr)=c^{ k}_{b }. \end{aligned}$$

 □

3.3 Proof of Theorem 1.3

At the end of the section, we give an analysis for the behavior of \(u_{b }\) as \(b \rightarrow 0\). We regard \(b >0\) as a parameter in equation (1.7).

Proof

For any \(b >0\), let \(u_{b }\in E\) be the least energy nodal solution of system (1.7) obtained in Theorem 1.1. We will complete our proof with the following three assertions. We recall that \(u_{b _{n}}\) is a least energy nodal solution of system (1.7) with \(b =b _{n}\to 0\) as \(n\to \infty\).

Claim (a). As n is large enough, \(\{u_{b _{n}}\}\) is bounded in E.

Choose a test function \(\phi \in C^{\infty }_{c}(\mathbb{R}^{3})\) with \(\phi ^{\pm }\neq 0\). From (2.7), for any \(b \in [0,1]\), there exists a pair of positive numbers \(( k_{1}, k_{2})\) such that

$$\begin{aligned} \bigl\langle \bigl(J^{b }_{ k}\bigr)'\bigl( k_{1}\phi ^{+}+ k_{2}\phi ^{-}\bigr), k_{1} \phi ^{+}\bigr\rangle < 0, \end{aligned}$$

and

$$\begin{aligned} \bigl\langle \bigl(J^{b }_{ k}\bigr)'\bigl( k_{1}\phi ^{+}+ k_{2}\phi ^{-}\bigr), k_{2} \phi ^{-}\bigr\rangle < 0. \end{aligned}$$

Thus, according to Lemma 2.1(ii), for any \(b \in [0,1]\), there is a unique pair \(s_{\phi }(b ), t_{\phi }(b )\in (0,1]\times (0,1]\) such that

$$\begin{aligned} \overline{\phi }:=s_{\phi }(b ) k_{1}\phi ^{+}+t_{\phi }(b ) k_{2} \phi ^{-}\in \mathcal{M}^{ k}_{b }. \end{aligned}$$
(3.13)

Hence, for any \(b \in [0,1]\), by using (2.4), we get

$$\begin{aligned} J^{b }_{ k}(u_{b })&\leq J^{b }_{ k}( \overline{\phi })=J^{ b }_{ k}(\overline{\phi })- \frac{1}{4}\bigl\langle \bigl(J^{b }_{ k} \bigr)'( \overline{\phi }),\overline{\phi }\bigr\rangle \\ &=\frac{1}{4} \Vert \overline{\phi } \Vert ^{2}+ \frac{ k}{4} \int _{\mathbb{R}^{3}}\bigl[f( \overline{\phi })\overline{\phi }-4F( \overline{\phi })\bigr]\,dx+ \frac{1}{12} \int _{\mathbb{R}^{3}} \vert \overline{\phi } \vert ^{6}\,dx \\ &\leq \frac{1}{4} \Vert \overline{\phi } \Vert ^{2}+ \frac{ k}{4} \int _{ \mathbb{R}^{3}}\bigl(C_{1}\overline{\phi }^{2}+C_{2}\overline{\phi }^{q}\bigr)\,dx+ \frac{1}{12} \int _{\mathbb{R}^{3}} \vert \overline{\phi } \vert ^{6}\,dx \\ &\leq \frac{1}{4}\bigl( k_{1}^{2} \bigl\Vert \phi ^{+} \bigr\Vert ^{2}+ k_{2}^{2} \bigl\Vert \phi ^{-} \bigr\Vert ^{2}\bigr)+\frac{ k}{4} \int _{\mathbb{R}^{3}}\bigl(C_{1}{ k_{1}}^{2} \bigl\vert \phi ^{+} \bigr\vert ^{2}+C_{1}{ k_{2}}^{2} \bigl\vert \phi ^{-} \bigr\vert ^{2}\bigr)\,dx \\ &\quad{}+ \frac{ k}{4} \int _{\mathbb{R}^{3}}\bigl(C_{2}{ k_{1}}^{5} \bigl\vert \phi ^{+} \bigr\vert ^{5}+C_{2}{ k_{2}}^{5} \bigl\vert \phi ^{-} \bigr\vert ^{5}\bigr)\,dx +\frac{{ k_{1}}^{6}}{12} \int _{ \mathbb{R}^{3}} \bigl\vert \phi ^{+} \bigr\vert ^{6}\,dx+\frac{{ k_{2}}^{6}}{12} \int _{ \mathbb{R}^{3}} \bigl\vert \phi ^{-} \bigr\vert ^{6}\,dx \\ &:=C^{\ast }, \end{aligned}$$

where \(C^{\ast }>0\) is a constant independent of b. So, as n is large enough, it follows that

$$\begin{aligned} C^{\ast }+1 \geq J^{ k}_{b _{n}}(u_{b _{n}})=J^{ k}_{ b _{n}}(u_{b _{n}})- \frac{1}{4}\bigl\langle \bigl(J^{ k}_{b _{n}} \bigr)'(u_{ b _{n}}),u_{b _{n}}\bigr\rangle \geq \frac{1}{4} \Vert u_{b _{n}} \Vert ^{2}. \end{aligned}$$

Therefore, we can deduce Claim (a) from the above inequality.

Claim (b). System (1.11) possesses a nodal solution \(u_{0}\).

Since \(\{u_{b _{n}}\}\) is bounded in E, in the subsequence sense, there exists \(u_{0}\in E\) such that

$$\begin{aligned} &u_{b _{n}}\rightharpoonup u_{0} \quad \text{in } E, \\ &u_{b _{n}}\rightarrow u_{0}\quad \text{in } L^{p} \bigl(\mathbb{R}^{3}\bigr) \quad \text{for } p\in (2,6), \\ &u_{b _{n}}\rightarrow u_{0}\quad \text{a.e. in } \mathbb{R}^{3}. \end{aligned}$$
(3.14)

Thanks to \(\{u_{b _{n}}\}\) being a least energy nodal solution of system (1.7) with \(b =b_{n}\), we have that

$$\begin{aligned} & \int _{\mathbb{R}^{3}}\bigl(a\nabla u_{b _{n}}\cdot \nabla v +V(x)u_{ b _{n}}v\bigr)\,dx+b_{n}\biggl( \int _{\mathbb{R}^{3}} \vert \nabla u_{b _{n}} \vert ^{2}\,dx\biggr) \biggl( \int _{\mathbb{R}^{3}}\nabla u_{b _{n}}\cdot \nabla v\,dx\biggr) \\ &\quad{}+\lambda \int _{\mathbb{R}^{3}}\phi _{u_{b_{n}}}u_{ b_{n}}v\,dx - k \int _{\mathbb{R}^{3}}f(u_{b _{n}})v \,dx- \int _{\mathbb{R}^{3}} \vert u_{b _{n}} \vert ^{4}u_{b _{n}}v \,dx=0 \end{aligned}$$
(3.15)

for any \(v\in C^{\infty }_{c}(\mathbb{R}^{3})\). Combining (3.14), (3.15) with Claim (a), we have that

$$\begin{aligned} & \int _{\mathbb{R}^{3}}\bigl(a\nabla u_{0}\cdot \nabla v +V(x)u_{0}v\bigr)\,dx+\lambda \int _{\mathbb{R}^{3}} \phi_{u_{0}}u_{0}v \,dx \\ &\quad{}- k \int _{\mathbb{R}^{3}}f(u_{0})v \,dx- \int _{\mathbb{R}^{3}} \vert u_{0} \vert ^{4}u_{0}v \,dx=0 \end{aligned}$$

for any \(v\in C^{\infty }_{c}(\mathbb{R}^{3})\). It implies that \(u_{0}\) is a weak solution of the Kirchhoff equation (1.11). We next deduce that \(u_{0}^{\pm }\neq 0\). Since \(u_{b _{n}}\in \mathcal{M}^{ k}_{b _{n}}\), we have

$$\begin{aligned} & \bigl\Vert u_{b _{n}}^{\pm } \bigr\Vert ^{2}+b_{n} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla u_{ b _{n}}^{\pm } \bigr\vert ^{2}\,dx\biggr)^{2}+b_{n} \int _{\mathbb{R}^{3}}\nabla u_{ b_{n}}^{\pm } \vert ^{2}\,dx\cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla u_{ b _{n}}^{\mp } \bigr\vert ^{2}\,dx) \\ &\qquad{}+\lambda \int _{\mathbb{R}^{3}}\phi _{u_{b _{n}}^{\pm }} \bigl\vert u_{ b _{n}}^{\pm } \bigr\vert ^{2}\,dx+\lambda \int _{\mathbb{R}^{3}}\phi _{u_{ b_{n}}^{\mp }} \bigl\vert u_{b _{n}}^{\pm } \bigr\vert ^{2}\,dx\\ &\quad = \int _{\mathbb{R}^{3}} \bigl\vert u_{ b _{n}}^{\pm } \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}}f\bigl(u_{b _{n}}^{ \pm } \bigr)u_{b _{n}}^{\pm }\,dx. \end{aligned}$$

Hence, by using Claim (a) and the continuous embedding \(E\hookrightarrow L^{6}(\mathbb{R}^{3})\), we have \(u_{b _{n}}\) is bounded in \(L^{6}(\mathbb{R}^{3})\), thus there exists \(k^{\star }_{2}>0\) such that, for all \(k\geq k^{\star }_{2}\), we have that

$$\begin{aligned} \rho \leq \bigl\Vert u_{b _{n}}^{\pm } \bigr\Vert ^{2}\leq \int _{\mathbb{R}^{3}} \bigl\vert u_{ b _{n}}^{\pm } \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}}f\bigl(u_{b _{n}}^{ \pm } \bigr)u_{b _{n}}^{\pm }\,dx\leq 2 k \int _{\mathbb{R}^{3}}f\bigl(u_{ b _{n}}^{\pm } \bigr)u_{b _{n}}^{\pm }\,dx. \end{aligned}$$

By using (2.4), we have that

$$\begin{aligned} 0< \int _{\mathbb{R}^{3}}f\bigl(u_{0}^{\pm } \bigr)u_{0}^{\pm }\,dx. \end{aligned}$$

Since \(u_{0}\) is a solution of system (1.11), we have that

$$\begin{aligned} \bigl\Vert u_{0}^{\pm } \bigr\Vert ^{2}\geq k \int _{\mathbb{R}^{3}}f\bigl(x,u_{0}^{\pm } \bigr)u_{0}^{ \pm }\,dx+ \int _{\mathbb{R}^{3}} \bigl\vert u_{0}^{\pm } \bigr\vert ^{6}\,dx\geq k \int _{ \mathbb{R}^{3}}f\bigl(u_{0}^{\pm } \bigr)u_{0}^{\pm }\,dx>0. \end{aligned}$$

It implies \(u_{0}^{\pm }\neq 0\).

Claim (c). Problem (1.11) possesses a least energy nodal solution \(v_{0}\).

Similar to the proof of Theorem 1.1, there is \(k^{\star }_{3}>0\) such that, for all \(k\geq k^{\star }_{3}\), problem (1.11) possesses a least energy nodal solution \(v_{0}\), where \(J^{0}_{k}(v_{0})=c^{0}_{k}\) and \((J^{0}_{k})'(v_{0})=0\). Let

$$ k^{\star \star \star }=\max \bigl\{ k^{\star }, k^{\star }_{2}, k^{\star }_{3} \bigr\} . $$

According to Lemma 2.1, there exists a positive pair \((s_{b _{n}}, t_{b _{n}})\in (0,\infty )\times (0,\infty )\) such that \(s_{b_{n}}v^{+}_{0}+t_{b _{n}}v^{-}_{0}\in \mathcal{M}^{ k}_{ b _{n}}\). That is,

$$\begin{aligned} &s_{b_{n}}^{2} \bigl\Vert v_{0}^{+} \bigr\Vert ^{2}+\lambda s_{b_{n}}^{4} \int _{\mathbb{R}^{3}}\phi _{v_{0}^{+}} \bigl\vert v_{0}^{+} \bigr\vert ^{2}\,dx+\lambda s_{ b _{n}}^{2}t_{b _{n}}^{2} \int _{\mathbb{R}^{3}}\phi _{v_{0}^{-}} \bigl\vert v_{0}^{+} \bigr\vert ^{2}\,dx +b_{n}s_{b _{n}}^{4} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla v_{0}^{+} \bigr\vert ^{2}\,dx\biggr)^{2} \\ &\qquad{}+b_{n}s_{b _{n}}^{2}t_{b _{n}}^{2} \int _{\mathbb{R}^{3}} \bigl\vert \nabla v_{0}^{+} \bigr\vert ^{2}\,dx\cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla v_{0}^{-} \bigr\vert ^{2}\,dx \\ &\quad =s_{ b _{n}}^{6} \int _{\mathbb{R}^{3}} \bigl\vert v_{0}^{+} \bigr\vert ^{6}\,dx+ k \int _{ \mathbb{R}^{3}}f\bigl(s_{b _{n}}v_{0}^{+} \bigr)s_{b _{n}}v_{0}^{+}\,dx \end{aligned}$$
(3.16)

and

$$\begin{aligned} &t_{b _{n}}^{2} \bigl\Vert v_{0}^{-} \bigr\Vert ^{2}+\lambda t_{b _{n}}^{4} \int _{\mathbb{R}^{3}}\phi _{v_{0}^{-}} \bigl\vert v_{0}^{-} \bigr\vert ^{2}\,dx+\lambda s_{ b _{n}}^{2}t_{b _{n}}^{2} \int _{\mathbb{R}^{3}}\phi _{v_{0}^{+}} \bigl\vert v_{0}^{-} \bigr\vert ^{2} \,dx+b_{n}t_{b _{n}}^{4} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \nabla v_{0}^{-} \bigr\vert ^{2}\,dx\biggr)^{2} \\ &\qquad{}+b_{n}s_{b _{n}}^{2}t_{b _{n}}^{2} \int _{\mathbb{R}^{3}} \bigl\vert \nabla v_{0}^{+} \bigr\vert ^{2}\,dx\cdot \int _{\mathbb{R}^{3}} \bigl\vert \nabla v_{0}^{-} \bigr\vert ^{2}\,dx \\ &\quad =t_{ b _{n}}^{6} \int _{\mathbb{R}^{3}} \bigl\vert v_{0}^{-} \bigr\vert ^{6}\,dx+ k \int _{ \mathbb{R}^{3}}f\bigl(t_{b _{n}}v_{0}^{-} \bigr)t_{b _{n}}v_{0}^{-}\,dx. \end{aligned}$$
(3.17)

By recalling Claim (a), up to a subsequence, we can deduce \(s_{b _{n}}\rightarrow s_{0}\) and \(t_{b _{n}}\rightarrow t_{0}\), then it follows from (3.16) and (3.17) that

$$\begin{aligned} &s_{0}^{2}\|v_{0}^{+}\|^{2}+\lambda s_{0}^{4}\int_{\mathbb{R}^{3}}\phi_{v_{0}^{+}}|v_{0}^{+}|^{2}dx+\lambda s_{0}^{2}t_{0}^{2}\int_{\mathbb{R}^{3}}\phi_{v_{0}^{-}}|v_{0}^{+}|^{2}\\ &=s_{0}^{6}\int_{\mathbb{R}^{3}}|v_{0}^{+}|^{6}dx+ k\int_{\mathbb{R}^{3}}f(s_{0}v_{0}^{+})s_{0}v_{0}^{+}dx \end{aligned}$$
(3.18)

and

$$\begin{aligned} &t_{0}^{2}\|v_{0}^{-}\|^{2}+\lambda t_{0}^{4}\int_{\mathbb{R}^{3}}\phi_{v_{0}^{-}}|v_{0}^{-}|^{2}+\lambda s_{0}^{2}t_{0}^{2}\int_{\mathbb{R}^{3}}\phi_{v_{0}^{+}}|v_{0}^{-}|^{2}\\ &=t_{0}^{6}\int_{\mathbb{R}^{3}}|v_{0}^{-}|^{6}dx+ k\int_{\mathbb{R}^{3}}f(t_{0}v_{0}^{-})t_{0}v_{0}^{-}dx. \end{aligned}$$
(3.19)

Thanks to \(v_{0}\) being a weak solution of problem (1.11), we get

$$\begin{aligned} \begin{aligned}[b] &\bigl\Vert v_{0}^{+} \bigr\Vert ^{2}+\lambda \int _{\mathbb{R}^{3}}\phi_{v_{0}^{+}}|v_{0}^{+}|^{2}\,dx +\lambda \int_{\mathbb{R}^{3}}\phi_{v_{0}^{-}}|v_{0}^{+}|^{2}\,dx \\ &\quad = \int _{\mathbb{R}^{3}} \bigl\vert v_{0}^{+} \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}}f\bigl(v_{0}^{+} \bigr)v_{0}^{+}\,dx \end{aligned} \end{aligned}$$
(3.20)

and

$$\begin{aligned} \begin{aligned}[b] &\bigl\Vert v_{0}^{-} \bigr\Vert ^{2}+\lambda \int _{\mathbb{R}^{3}}\phi_{v_{0}^{-}}|v_{0}^{-}|^{2}\,dx +\lambda \int_{\mathbb{R}^{3}}\phi_{v_{0}^{+}}|v_{0}^{-}|^{2}\,dx \\ &\quad = \int _{\mathbb{R}^{3}} \bigl\vert v_{0}^{-} \bigr\vert ^{6}\,dx+ k \int _{\mathbb{R}^{3}}f\bigl(v_{0}^{-} \bigr)v_{0}^{-}\,dx. \end{aligned} \end{aligned}$$
(3.21)

By comparing formulas (3.18)–(3.21), it is obvious that \((s_{0}, t_{0}) = (1, 1)\). Similar to the proof of Lemma 2.1, we have

$$\begin{aligned} J^{0}_{k}(v_{0})\leq J^{0}_{k}(u_{0})= \lim_{n\rightarrow \infty }J^{ b _{n}}_{k}(u_{b _{n}}) \leq \lim_{n\rightarrow \infty }J^{ b _{n}}_{k}\bigl(s_{b _{n}}v^{+}_{0}+t_{b _{n}}v^{-}_{0} \bigr) =J^{0}_{k}\bigl(v^{+}_{0}+v^{-}_{0} \bigr)=J^{0}_{k}(v_{0}). \end{aligned}$$

The above inequality implies that \(u_{0}\) is a least energy nodal solution of problem (1.11). So far, we have proved Theorem 1.3. □