Sign-changing solutions for coupled Schrödinger system

Abstract

In this paper we study the following nonlinear Schrödinger system:

$$ \textstyle\begin{cases} -\Delta u+\alpha u = \vert u \vert ^{p-1}u+\frac{2}{q+1} \lambda \vert u \vert ^{ \frac{p-3}{2}}u \vert v \vert ^{\frac{q+1}{2}},\quad x \in \mathbb{R}^{3}, \\ -\Delta v+\beta v = \vert v \vert ^{q-1}v+\frac{2}{p+1} \lambda \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}v ,\quad x \in \mathbb{R}^{3}, \\ u(x)\rightarrow 0,\qquad v(x)\rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty , \end{cases} $$

where $3\leq p, q<5$, α, β are positive parameters. We show that there exists $\lambda _{k}>0$ such that the equation has at least k radially symmetric sign-changing solutions and at least k seminodal solutions for each $k\in \mathbb{N}$ and $\lambda \in (0, \lambda _{k})$. Moreover, we show the existence of a least energy radially symmetric sign-changing solution for each $\lambda \in (0, \lambda _{0})$ where $\lambda _{0}\in (0, \lambda _{1}]$.

1 Background and main results

Consider the following nonlinear coupled Schrödinger system:

$$ \textstyle\begin{cases} -\Delta u+\alpha u = \vert u \vert ^{p-1}u+\frac{2}{q+1} \lambda \vert u \vert ^{ \frac{p-3}{2}}u \vert v \vert ^{\frac{q+1}{2}} ,\quad x \in \Omega , \\ -\Delta v+\beta v = \vert v \vert ^{q-1}v+\frac{2}{p+1} \lambda \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}v ,\quad x \in \Omega , \\ u=v=0\quad \text{on } \partial \Omega . \end{cases} $$

(1.1)

Here $\Omega =\mathbb{R}^{N}$ or Ω is a smooth bounded domain in $\mathbb{R}^{N}$, α, β are positive parameters and $\lambda \neq 0$ is a coupling constant.

In the case $p=q=3$, system (1.1) becomes the cubic system:

$$ \textstyle\begin{cases} -\Delta u +\alpha u= u^{3}+\lambda u v^{2}, \quad x\in \Omega , \\ -\Delta v +\beta v = v^{3}+\lambda u^{2} v, \quad x\in \Omega , \\ u=v=0 \quad \text{on }\partial \Omega ,\end{cases} $$

(1.2)

which arises in the study of many physical phenomena like nonlinear optics and Bose–Einstein condensation (cf. [15, 17]). Therefore, in the last decades, system (1.2) has received great interest from mathematicians. When Ω is the entire space $\mathbb{R}^{N}$, the existence of least energy and other finite energy solutions of (1.2) was studied in [2, 11, 12, 18, 21, 22, 27] and the references therein. In particular, when $\lambda >0$ is sufficiently large, infinitely many radially symmetric sign-changing solutions of (1.2) were obtained in [23]. Liu and Wang [20] studied a general m-coupled system ($m\ge 2$) and proved that system (1.2) has infinitely many nontrivial solutions, but whether solutions obtained in [20] are positive or sign-changing cannot be determined there (see also [21]). When $\Omega \subset \mathbb{R}^{N}$ ($N=2, 3$) is a smooth bounded domain, there are also many papers studying (1.2). Lin and Wei [18] proved that a least energy solution of (1.2) exists within an appropriate range of λ. Dancer, Wei, and Weth [14] and Noris and Ramos [24] proved the existence of infinitely many positive solutions of (1.2). When Ω is a ball, a multiplicity result on positive radially symmetric solutions was given in [29]. Later, by using a global bifurcation approach, the result of [29] was reproved by [4] without requiring the symmetric condition. Under some more general assumptions, Sato and Wang [26] proved that system (1.2) has infinitely many semipositive solutions (i.e., at least one component is positive). In [14], the authors proved the existence of unbounded sequence solutions for $N\leq 3$ and $\lambda \leq -1$. As pointed out above, for $\lambda \leq -1$, Wei and Weth [29] proved that (1.2) has a radially symmetric solution, which turns out to be a positive solution.

We remark that the existence of infinitely many sign-changing solutions or seminodal solutions to (1.2) was solved by Chen, Lin, and Zou [10] and Liu, Liu, and Wang [19] independently, where $N\leq 3$ and $\lambda <0$.

To the best of our knowledge, the existence of sign-changing solutions to (1.1) has not ever been studied in the literature when $\Omega =\mathbb{R}^{3}$ and $3\leq p, q<5$. The main goal of this paper is to study the existence of sign-changing solutions, seminodal solutions, and least energy sign-changing solutions to problem (1.1) when $\lambda >0$ is small. This will complement the study made in [14, 19, 21, 22, 29].

Definition 1.1

A solution $(u, v)$ is called nontrivial if $u\not \equiv 0$ and $v\not \equiv 0$, a solution $(u, v)$ is semitrivial if $(u, v)$ is type of $(u, 0)$ or $(0, v)$. We call a solution $(u, v)$ positive if $u>0$ and $v>0$ in $\mathbb{R}^{N}$, a solution $(u, v)$ sign-changing if both u and v change sign, a solution $(u, v)$ seminodal if one changes sign and the other one is positive.

The first main result of the current paper is as follows.

Theorem 1.1

Assume $\alpha , \beta >0$. Then for any $k\in \mathbb{N}$ there exists $\lambda _{k}>0$ such that system (1.1) possesses at least k radially symmetric sign-changing solutions for each fixed $\lambda \in (0, \lambda _{k})$.

We can also study some further properties of the sign-changing solutions obtained in Theorem 1.1. It is well known that a nontrivial solution $(u, v)\in H^{1}(\mathbb{R}^{N})\times H^{1}(\mathbb{R}^{N})$ is called a least energy solution if its energy is minimal among the energy of all nontrivial solutions. A sign-changing solution is called a least energy sign-changing solution if it has the least energy among all sign-changing solutions. Precisely, we have the following theorem.

Theorem 1.2

Assume $\alpha , \beta >0$. Then there exists $\lambda _{0}\in (0, \lambda _{1}]$ such that system (1.1) possesses a least energy radially symmetric sign-changing solution for each fixed $\lambda \in (0, \lambda _{0})$.

Theorem 1.3

Assume $\alpha , \beta >0$. Then for any $k\in \mathbb{N}$ there exists $\lambda _{k}>0$ such that system (1.1) possesses at least k seminodal solutions for each fixed $\lambda \in (0, \lambda _{k})$.

Remark 1.1

We can prove that system (1.1) possesses at least k seminodal solutions with the first component positive and the second component radially symmetric sign-changing or the first component radially symmetric sign-changing and the second component positive.

The structure of this paper is as follows. In Sect. 2 we prove the existence of at least k radially symmetric sign-changing solutions. The main tool will be the use of a new notion of vector genus by [28] and a new constrained problem by [10], which will be used to construct minimax values. Remark that the ideas in [10, 28] cannot be used directly, and here we will give some new ideas. The crucial idea in this paper is turning to study a new problem with two constraints to obtain sign-changing solutions of (1.1). This idea has never been used for (1.1) in the literature up to our knowledge. We will give all the necessary details of the proof. Section 3 is then dedicated to the proof of Theorem 1.2 by using a minimizing argument. Finally in Sect. 4 we will present the proof of Theorem 1.3 applying the arguments in Sect. 2 and Sect. 3.

We give some notations here. Throughout this paper, we denote the norm of $L^{p}(\mathbb{R}^{N})$ by $|u|_{p}= (\int _{\mathbb{R}^{N}} |u|^{p} \,dx )^{\frac{1}{p}}$, the norm of $H^{1}(\mathbb{R}^{N})$ by $\|u\|^{2}=\int _{\mathbb{R}^{N}}(|\nabla u|^{2}+ |u|^{2}) \,dx$, and positive constants (possibly different in different places) by C. Define $H_{r}:= H_{r}^{1}(\mathbb{R}^{N})\times H_{r}^{1}(\mathbb{R}^{N})$ as a subspace of $H:= H^{1}(\mathbb{R}^{N})\times H^{1}(\mathbb{R}^{N})$ with norm $\|(u, v)\|_{H_{r}}^{2}:=\|u\|_{\alpha}^{2}+\|v\|_{\beta}^{2}$ where

$$ \begin{aligned}&H_{r}^{1}\bigl(\mathbb{R}^{N}\bigr):=\bigl\{ u\in H^{1}\bigl(\mathbb{R}^{N}\bigr): u \text{ is radially symmetric}\bigr\} , \\ & \Vert u \Vert _{\alpha}^{2}:= \int _{\mathbb{R}^{N}}\bigl( \vert \nabla u \vert ^{2}+ \alpha \vert u \vert ^{2}\bigr) \,dx .\end{aligned} $$

2 Proof of Theorem 1.1

In this section, we assume that $N=3$, $3\leq p, q<2^{*}-1=5$ and $\alpha , \beta >0$. Without loss of generality, we assume $p\leq q$. Let $\lambda \in (0, 1)$. For any $k\in \mathbb{N}$, let $X_{k+1}\subset H_{r}^{1}(\mathbb{R}^{3})$, $\operatorname{dim}X_{k+1}=k+1$, and there exists $u_{0}\in X_{k+1}$ and $u_{0}>0$. Then there exists $m>0$ such that for any $(u, v)\in X_{k+1}\times X_{k+1}$ satisfying $|u|_{p+1}^{p+1}, |v|_{q+1}^{q+1}<2$, we have

$$ \Vert u \Vert _{\alpha}^{2}< m,\qquad \Vert v \Vert _{\beta}^{2} < m. $$

(2.1)

Without loss of generality, we can assume $m>1$. Obviously, the sign-changing solutions of system (1.1) are the critical points of the $C^{2}$ functional $\Phi _{\lambda}: H_{r}\rightarrow \mathbb{R}$ given by

$$ \begin{aligned}\Phi _{\lambda}(u, v) &:= \frac{1}{2} \bigl( \Vert u \Vert _{\alpha}^{2}+ \Vert v \Vert _{\beta}^{2}\bigr)-\frac{1}{p+1} \vert u \vert _{p+1}^{p+1}-\frac{1}{q+1} \vert v \vert _{q+1}^{q+1} \\ &\quad{}-\frac{4\lambda}{(p+1)(q+1)} \int _{ \mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q+1}{2}} \,dx . \end{aligned} $$

(2.2)

We will look for solutions of Eq. (1.1) as critical points of the functional $\Phi _{\lambda}$ restricted to the sphere

$$ \mathcal{A}:=\bigl\{ (u, v)\in H_{r}: \vert u \vert _{p+1}=1, \vert v \vert _{q+1}=1 \bigr\} . $$

To obtain at least k sign-changing critical points, we need to define several minimax energy levels using a new definition of vector genus introduced by [28]. As in [28], we recall vector genus and take the transformations

$$ \sigma _{i}: \mathcal{A}\rightarrow \mathcal{A},\qquad \sigma _{1}(u, v)=(-u, v),\qquad \sigma _{2}(u, v)=(u, -v), \quad i=1, 2. $$

Consider the class of sets

$$ \mathcal{F}=\bigl\{ A\subset \mathcal{A}: A \text{ is a closed set and } \sigma _{i}(u, v)\in A, \forall (u, v)\in A, i=1, 2\bigr\} $$

and for each $A\in \mathcal{F}$ and $k_{1}, k_{2}\in \mathbb{N}$, the class of functions

$$ \begin{aligned}F_{(k_{1}, k_{2})}(A)={}&\Biggl\{ f=(f_{1}, f_{2}): A\rightarrow \prod_{i=1}^{2} \mathbb{R}^{k_{i}-1}: f_{i}: A\rightarrow \mathbb{R}^{k_{i}-1}\text{ continuous}, \\ &{}f_{i}\bigl(\sigma _{i}(u, v)\bigr)=-f_{i}(u, v) \text{ for each } i, f_{i}\bigl( \sigma _{j}(u, v) \bigr)=f_{i}(u, v) \text{ for } i\neq j \Biggr\} . \end{aligned} $$

where $\mathbb{R}^{0}:=\{0\}$.

Definition 2.1

(Vector genus, see [28]) For every nonempty and closed set $A\subset H_{0}^{1}(\Omega )$ such that $-A=A$, we define

$$ \gamma (A):=\inf \bigl\{ k: \text{there exists } h:A \rightarrow \mathbb{R}^{k} \backslash \{0\} \text{ continuous and odd}\bigr\} $$

and $\gamma (A):=\infty$ if no such k exists.

Let $A\in \mathcal{F}$ and take any $k_{1}, k_{2}\in \mathbb{N}$. We say that $\gamma (A)\geq (k_{1}, k_{2})$ if for every $f\in F_{(k_{1}, k_{2})}(A)$ there exists $(u, v)\in A$ such that $f(u, v)=(f_{1}(u, v), f_{2}(u, v))=(0, 0)$. We denote

$$ \Gamma ^{(k_{1}, k_{2})}:=\bigl\{ A\in \mathcal{F}: \gamma (A)\geq (k_{1}, k_{2})\bigr\} . $$

Remark 2.1

Note that Definition 2.1 does not actually define the quantity $\gamma (A)$ but gives the meaning of $\gamma (A)\geq (k_{1}, k_{2})$ only. A different notation of genus was introduced by Chang, Wang, and Zhang in [8].

Lemma 2.1

(see [28])

Let $f=(f_{1}, f_{2}): \prod_{i=1}^{2} S^{k_{i}}\rightarrow \prod_{i=1}^{2} \mathbb{R}^{k_{i}}$ be a continuous function such that $f_{i}(\sigma _{i}(u, v))=-f_{i}(u, v)$, $f_{i}(\sigma _{j}(u, v))=f_{i}(u, v)$ for any $i, j=1, 2$, $i\neq j$, then there exists $(u_{0}, v_{0})\in \prod_{i=1}^{2} S^{k_{i}}$ such that $f(u_{0}, v_{0})=(0, \ldots ,0)$.

Lemma 2.2

(see [28])

The following properties hold.

• (1) Take $A_{1}\times A_{2}\subset \mathcal{A}$ and let $\eta _{i}: S^{k_{i}-1}\rightarrow A_{i}$ be a homeomorphism such that $\eta _{i}(-x)=-\eta _{i}(x)$ for every $x\in S^{k_{i}-1}$, $i=1, 2$. Then $A_{1}\times A_{2} \in \Gamma ^{(k_{1}, k_{2})}$, where $S^{k_{i}-1}=\{x\in \mathbb{R}^{k_{i}}: |x|=1\}$.

• (2) We have $\overline{\eta (A)}\in \Gamma ^{(k_{1}, k_{2})}$ whenever $A \in \Gamma ^{(k_{1}, k_{2})}$ and a continuous map $\eta : A\rightarrow \mathcal{A}$ is such that $\eta \circ \sigma _{i}=\sigma _{i}\circ \eta$, $\forall i=1, 2$.

Together with the notation of vector genus, to obtain sign-changing solutions, we will use cones of positive or negative functions based on the works such as [5, 13, 30]. We define the cone

$$ \mathcal{P}_{1}:=\bigl\{ (u, v)\in H_{r}: u\geq 0\bigr\} ,\qquad \mathcal{P}_{2}:= \bigl\{ (u, v)\in H_{r}: v\geq 0\bigr\} , $$

and take $\mathcal{P}:=\bigcup_{i=1}^{2} (\mathcal{P}_{i}\cup - \mathcal{P}_{i} )$. Moreover, for any $\delta >0$, we define

$$ \mathcal{P}_{\delta}:=\bigl\{ (u, v)\in H_{r}: \operatorname{dist}\bigl((u, v), \mathcal{P}\bigr)< \delta \bigr\} , $$

where

$$\begin{aligned}& \begin{aligned} \operatorname{dist} \bigl((u, v), \mathcal{P}\bigr):={}& \min \bigl\{ \operatorname{dist}_{p+1} (u, \mathcal{P}_{1}) , \operatorname{dist}_{p+1} (u, -\mathcal{P}_{1}), \\ &\operatorname{dist}_{q+1} (v, \mathcal{P}_{2}) , \operatorname{dist}_{q+1} (v, -\mathcal{P}_{2})\bigr\} , \end{aligned}\\& \operatorname{dist}_{p+1} (u, \pm \mathcal{P}_{1}) :=\inf _{\omega \in \pm \mathcal{P}_{1}} \vert u-\omega \vert _{p+1}= \bigl\vert u^{\mp} \bigr\vert _{p+1},\\& \operatorname{dist}_{q+1} (v, \pm \mathcal{P}_{2}) :=\inf _{\omega \in \pm \mathcal{P}_{2}} \vert v-\omega \vert _{q+1}= \bigl\vert v^{\mp} \bigr\vert _{q+1}, \end{aligned}$$

where $u^{\pm}:=\max \{0, \pm u\}$.

Lemma 2.3

For any $0<\delta <2^{-\frac{1}{p+1}}$, there holds $A\backslash \mathcal{P}_{\delta }\neq \emptyset$ whenever $A \in \Gamma ^{(k_{1}, k_{2})}$ with $k_{1}, k_{2}\geq 2$.

Proof

For any $A \in \Gamma ^{(k_{1}, k_{2})}$, define $f=(f_{1}, f_{2})$ by

$$\begin{aligned}& f_{1}(u, v)= \biggl( \int _{ \mathbb{R}^{3}} \vert u \vert ^{p}u \,dx , 0, \ldots , 0 \biggr),\\& f_{2}(u, v)= \biggl( \int _{ \mathbb{R}^{3}} \vert v \vert ^{q}v \,dx , 0, \ldots , 0 \biggr), \end{aligned}$$

then $f\in F_{(k_{1}, k_{2})}(A)$, so by Definition 2.1, there exists $(u_{0}, v_{0})\in A$ such that $f(u_{0}, v_{0})=(0, \ldots ,0)$. By $A\in \mathcal{A}$, we deduce that

$$\begin{aligned}& \int _{ \mathbb{R}^{3}} \bigl(u_{0}^{+} \bigr)^{p+1} \,dx = \int _{ \mathbb{R}^{3}} \bigl(u_{0}^{-} \bigr)^{p+1} \,dx =\frac{1}{2},\\& \int _{ \mathbb{R}^{3}} \bigl(v_{0}^{+} \bigr)^{q+1} \,dx = \int _{ \mathbb{R}^{3}} \bigl(v_{0}^{-} \bigr)^{q+1} \,dx =\frac{1}{2}, \end{aligned}$$

therefore, $\operatorname{dist} ((u_{0}, v_{0}), \mathcal{P})=2^{-\frac{1}{p+1}}$, and so $(u_{0}, v_{0})\in A \backslash \mathcal{P}_{\delta}$ for any $0<\delta <2^{-\frac{1}{p+1}}$. □

For technical reasons, we will work on the neighborhood of $\mathcal{A}$ in $H_{r}^{1}(\mathbb{R}^{3})$,

$$ \mathcal{A}^{*}:=\biggl\{ (u, v)\in H_{r}: \frac{1}{2}< \vert u \vert _{p+1}^{p+1}< 2, \frac{1}{2}< \vert v \vert _{q+1}^{q+1}< 2 \biggr\} , $$

(2.3)

when $u\in \mathcal{A}^{*}$, $(u, v)\not \equiv (0, 0)$. Define

$$\begin{aligned}& \mathcal{B}_{m}^{*}:=\bigl\{ (u, v)\in \mathcal{A}^{*} : \Vert u \Vert _{\alpha}^{2}< m, \Vert v \Vert _{\beta}^{2}< m \bigr\} , \end{aligned}$$

(2.4)

$$\begin{aligned}& \mathcal{B}_{m}:=\bigl\{ (u, v)\in \mathcal{A} : \Vert u \Vert _{\alpha}^{2}< m, \Vert v \Vert _{\beta}^{2}< m \bigr\} , \end{aligned}$$

(2.5)

$$\begin{aligned}& \mathcal{C}_{m}:= \bigl\{ (u, v)\in \mathcal{A} : \Vert u \Vert _{\alpha}^{2}=m, \Vert v \Vert _{\beta}^{2}=m \bigr\} . \end{aligned}$$

(2.6)

Let $S_{p}$ and $S_{q}$ be the sharp constants of the Sobolev embedding $H_{r}^{1}(\mathbb{R}^{3})\hookrightarrow L^{p+1}(\mathbb{R}^{3})$ and $H_{r}^{1}(\mathbb{R}^{3})\hookrightarrow L^{q+1}(\mathbb{R}^{3})$, respectively,

$$ \Vert u \Vert _{\alpha}^{2}\geq S_{p} \vert u \vert _{p+1}^{2},\qquad \Vert v \Vert _{\beta}^{2} \geq S_{q} \vert v \vert _{q+1}^{2},\quad \forall u, v\in H_{r}^{1}\bigl( \mathbb{R}^{3}\bigr). $$

(2.7)

For any $(u, v)\in H_{r}\backslash \{(0, 0)\}$, we have

$$ \sup_{t, s\geq 0} \Phi _{\lambda}(tu, sv)= \Phi _{\lambda}(t_{u,v, \lambda}u, s_{u,v,\lambda}v)=:\Psi _{\lambda}(u, v), $$

(2.8)

where $t_{u,v,\lambda}, s_{u,v,\lambda}\geq 0$ satisfy

$$ \frac{\partial}{\partial t}\Phi _{\lambda}(tu, sv)|_{(t_{u,v,\lambda}, s_{u,v,\lambda})}= \frac{\partial}{\partial s}\Phi _{\lambda}(tu, sv)|_{(t_{u,v, \lambda}, s_{u,v,\lambda})}=0. $$

Note that for $t, s\geq 0$,

$$ \begin{aligned}\Phi _{\lambda}(tu, sv) :={}& \frac{1}{2} \bigl(t^{2} \Vert u \Vert _{ \alpha}^{2}+s^{2} \Vert v \Vert _{\beta}^{2}\bigr)-\frac{t^{p+1}}{p+1} \vert u \vert _{p+1}^{p+1}- \frac{s^{q+1}}{q+1} \vert v \vert _{q+1}^{q+1} \\ &{}-\frac{4\lambda}{(p+1)(q+1)} t^{\frac{p+1}{2}} s^{ \frac{q+1}{2}} \int _{ \mathbb{R}^{3}} \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{ \frac{q+1}{2}} \,dx . \end{aligned} $$

(2.9)

Define

$$ \begin{aligned}F(u, v, \lambda ; t, s)&:= t \Vert u \Vert _{\alpha}^{2}-t^{p} \vert u \vert _{p+1}^{p+1}- \frac{2}{q+1} t^{\frac{p-1}{2}} s^{\frac{q+1}{2}} \lambda \int _{ \mathbb{R}^{3}} \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{\frac{q+1}{2}} \,dx \\ &:= t F_{1}(u, v, \lambda ; t, s) \end{aligned} $$

and

$$ \begin{aligned}G(u, v, \lambda ; t, s)&:= s \Vert v \Vert _{\beta}^{2}-s^{q} \vert v \vert _{q+1}^{q+1}- \frac{2}{p+1} t^{\frac{p+1}{2}} s^{\frac{q-1}{2}} \lambda \int _{ \mathbb{R}^{3}} \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{\frac{q+1}{2}} \,dx \\ &:=s G_{1}(u, v, \lambda ; t, s), \end{aligned} $$

which implies

$$ F_{1}(u, v, \lambda ; t_{u,v,\lambda}, s_{u,v,\lambda})=G_{1}(u, v, \lambda ; t_{u,v,\lambda}, s_{u,v,\lambda})=0. $$

(2.10)

Since $F_{1}(u, v, \lambda ; t, s)$ and $G_{1}(u, v, \lambda ; t, s)$ are decreasing with respect to $t>0$ and $s>0$, respectively, $F_{1}(u, v, \lambda ; 0, 0)>0$, $G_{1}(u, v, \lambda ; 0, 0)>0$, so $t_{u,v,\lambda}$, $s_{u,v,\lambda}$ are unique. Note that for $t, s\geq 0$, $3\leq p, q<5$, by (2.9), we can choose some positive constant T such that $\Phi _{\lambda}(tu, sv)<0$ for any $t, s>T$, therefore, $t_{u,v,\lambda}, s_{u,v,\lambda}\in [0, T]$.

Define

$$ \widetilde{m}> \biggl[(q+1) S_{p}\biggl( \frac{1}{2}\biggr)^{\frac{2}{p+1}} \biggr]^{ \frac{2}{p+q-2}} + \frac{4(p+1)(q+1)}{(p-1)(\frac{S_{p}}{8})^{\frac{2}{p-1}}}m^{ \frac{p+1}{p-1}}+m. $$

(2.11)

Then $B_{m}\subset B_{\widetilde{m}}$, $B_{m}^{*}\subset B_{\widetilde{m}}^{*}$.

Lemma 2.4

For any $k\in \mathbb{N}$, there exist $\widetilde{\lambda}\in (0, 1)$ and $T_{1}>T_{2}>0$ such that for any $\lambda \in (0, \widetilde{\lambda})$ and $(u, v)\in B_{\widetilde{m}}^{*}$, we have

$$ T_{2}\leq t_{u,v,\lambda}, s_{u,v,\lambda} \leq T_{1}. $$

(2.12)

Furthermore, there exist $\lambda _{k}\in (0, \widetilde{\lambda}]$ and $c_{k}>0$ such that for any $\lambda \in (0, \lambda _{k})$, we have

$$ \sup_{(u,v)\in \mathcal{B}_{m}} \sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)< c_{k}\leq \inf_{(u,v)\in \mathcal{C}_{\widetilde{m }}} \sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv) . $$

(2.13)

Proof

We see from (2.9) and (2.10) that

$$ \begin{aligned}\sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)&= \Phi _{\lambda}(t_{u,v, \lambda}u, s_{u,v,\lambda}v) \\ &=\biggl(\frac{1}{2}-\frac{1}{p+1}\biggr)t_{u,v,\lambda}^{2} \Vert u \Vert _{\alpha}^{2}+\biggl( \frac{1}{2}-\frac{1}{q+1}\biggr)s_{u,v,\lambda}^{q+1} \vert v \vert _{q+1}^{q+1} \\ &\quad{}+\frac{(q-1)}{(p+1)(q+1)} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{ \mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q+1}{2}} \,dx . \end{aligned} $$

(2.14)

Firstly, we claim that there exist $\widetilde{\lambda}\in (0, 1)$ and $T_{1}>T_{2}>0$ such that for any $\lambda \in (0, \widetilde{\lambda})$ and $(u, v)\in B_{\widetilde{m}}^{*}$, we have

$$ T_{2}\leq t_{u,v,\lambda}, s_{u,v,\lambda}\leq T_{1}. $$

By (2.10),

$$\begin{aligned}& t_{u,v,\lambda}\leq \biggl(\frac{ \Vert u \Vert _{\alpha}^{2}}{ \vert u \vert _{p+1}^{p+1}} \biggr)^{\frac{1}{p-1}}< (2 \widetilde{m})^{\frac{1}{p-1}}< 2\widetilde{m},\\& s_{u,v,\lambda}\leq \biggl(\frac{ \Vert v \Vert _{\beta}^{2}}{ \vert v \vert _{q+1}^{q+1}} \biggr)^{\frac{1}{q-1}}< (2 \widetilde{m})^{\frac{1}{q-1}}< 2\widetilde{m}. \end{aligned}$$

Thus, we obtain that

$$ t_{u,v,\lambda}, s_{u,v,\lambda}< 2\widetilde{m}=:T_{1}. $$

Define

$$ \widetilde{\lambda}= \frac{(q+1) S_{p} (\frac{1}{2})^{\frac{2}{p+1}}}{8 (2\widetilde{m} )^{\frac{p+q-2}{2}}}. $$

We see from (2.11) that $\widetilde{\lambda}\in (0, 1)$. Moreover, by (2.7) and (2.10), for any $\lambda \in (0, \widetilde{\lambda})$, we have

$$\begin{aligned} t_{u,v,\lambda}^{p-1} \vert u \vert _{p+1}^{p+1} &= \Vert u \Vert _{\alpha}^{2}- \frac{2}{q+1} t_{u,v,\lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{ \frac{q+1}{2}} \lambda \int _{ \mathbb{R}^{3}} \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{ \frac{q+1}{2}} \,dx \\ &> S_{p} \biggl(\frac{1}{2}\biggr)^{\frac{2}{p+1}} - \frac{2}{q+1} (2 \widetilde{m})^{\frac{p+q-2}{2}}\lambda \vert u \vert _{p+1}^{\frac{p+1}{2}} \vert v \vert _{q+1}^{\frac{q+1}{2}} \\ &> S_{p} \biggl(\frac{1}{2}\biggr)^{\frac{2}{p+1}} - \frac{4}{q+1} (2 \widetilde{m})^{\frac{p+q-2}{2}}\lambda \\ &> \frac{1}{2} S_{p} \biggl(\frac{1}{2} \biggr)^{\frac{2}{p+1}}> \frac{S_{p}}{4}. \end{aligned}$$

Then we get $t_{u,v,\lambda}> (\frac{S_{p}}{8} )^{\frac{1}{p-1}}$. Similarly, we have $s_{u,v,\lambda}> (\frac{S_{q}}{8} )^{\frac{1}{q-1}}$. Thus, we get

$$ t_{u,v,\lambda}, s_{u,v,\lambda}>\min \biggl\{ \biggl( \frac{S_{p}}{8} \biggr)^{ \frac{1}{p-1}}, \biggl(\frac{S_{q}}{8} \biggr)^{\frac{1}{q-1}} \biggr\} =:T_{2}. $$

This completes $T_{2}\leq t_{u,v,\lambda}\leq T_{1}$.

Now we prove the existence of $\lambda _{k}$ and $c_{k}$. For any $(u, v)\in \overline{B}_{\widetilde{m}}$ and $\lambda \in (0, \widetilde{\lambda}]$, by (2.14), there holds

$$ \begin{aligned}& \biggl\vert \sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)-\biggl( \frac{1}{2}-\frac{1}{p+1} \biggr)t_{u,v,\lambda}^{2} \Vert u \Vert _{\alpha}^{2}- \biggl( \frac{1}{2}-\frac{1}{q+1}\biggr)s_{u,v,\lambda}^{q+1} |v|_{q+1}^{q+1} \biggr\vert \\ &\quad = \biggl\vert \frac{(q-1)}{(p+1)(q+1)} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{ \mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q+1}{2}} \,dx \biggr\vert \leq C\lambda . \end{aligned} $$

Hence,

$$ \begin{aligned} & \sup_{(u,v)\in B_{m}} \sup _{t,s\geq 0} \Phi _{ \lambda}(tu, sv) \\ &\quad \leq \sup_{(u,v)\in B_{m}} \biggl[ \biggl(\frac{1}{2}- \frac{1}{p+1}\biggr)t_{u,v, \lambda}^{2} \Vert u \Vert _{\alpha}^{2}+\biggl(\frac{1}{2}- \frac{1}{q+1}\biggr)s_{u,v, \lambda}^{q+1} \vert v \vert _{q+1}^{q+1} \biggr]+C\lambda \\ &\quad \leq \sup_{(u,v)\in B_{m}} \biggl[ \biggl(\frac{1}{2}- \frac{1}{p+1}\biggr) \biggl( \frac{ \Vert u \Vert _{\alpha}^{2}}{ \vert u \vert _{p+1}^{p+1} } \biggr)^{\frac{2}{p-1}} \Vert u \Vert _{\alpha}^{2}+\biggl(\frac{1}{2}- \frac{1}{q+1}\biggr) \biggl( \frac{ \Vert v \Vert _{\beta}^{2} }{ \vert v \vert _{q+1}^{q+1}} \biggr)^{\frac{q+1}{q-1}} \biggr]+C\lambda \\ &\quad \leq \biggl(\frac{1}{2}-\frac{1}{p+1}\biggr) m^{\frac{p+1}{p-1}} +\biggl(\frac{1}{2}- \frac{1}{q+1} \biggr)m^{\frac{q+1}{q-1}}+C\lambda \\ &\quad \leq 2 \biggl(\frac{1}{2}-\frac{1}{q+1}\biggr)m^{\frac{p+1}{p-1}}+C \lambda < (q+1) m^{\frac{p+1}{p-1}}+C\lambda , \end{aligned} $$

and

$$ \begin{aligned} &\inf_{(u,v)\in \mathcal{C}_{\widetilde{m }}} \sup _{t,s\geq 0} \Phi _{\lambda}(tu, sv) \\ &\quad \geq \inf_{(u,v)\in \mathcal{C}_{\widetilde{m }}} \biggl[ \biggl(\frac{1}{2}- \frac{1}{p+1}\biggr)t_{u,v,\lambda}^{2} \Vert u \Vert _{ \alpha}^{2}+\biggl(\frac{1}{2}- \frac{1}{q+1}\biggr)s_{u,v,\lambda}^{q+1} \vert v \vert _{q+1}^{q+1} \biggr]-C\lambda \\ &\quad >\inf_{(u,v)\in \mathcal{C}_{\widetilde{m }}} \biggl(\frac{1}{2}- \frac{1}{p+1}\biggr)t_{u,v,\lambda}^{2} \Vert u \Vert _{\alpha}^{2}-C \lambda \\ &\quad \geq \biggl(\frac{1}{2}-\frac{1}{p+1}\biggr) \biggl( \frac{S_{p}}{8} \biggr)^{ \frac{2}{p-1}}\widetilde{m}-C\lambda , \end{aligned} $$

then by (2.11), we can choose

$$\begin{aligned}& \lambda _{k}=\min \biggl\{ \frac{q+1}{2C} m^{\frac{p+1}{p-1}} , \widetilde{\lambda} \biggr\} ,\\& c_{k}=\biggl(\frac{1}{2}-\frac{1}{p+1}\biggr) \biggl(\frac{S_{p}}{8} \biggr)^{ \frac{2}{p-1}}\widetilde{m}-C\lambda _{k} \end{aligned}$$

such that $c_{k}>0$ for any $0<\lambda <\lambda _{k}$ the conclusion holds. □

For any $(u, v)\in B_{\widetilde{m}}^{*}$, the following linear problem

$$ \textstyle\begin{cases} -\Delta \varphi +\alpha \varphi -\frac{2}{q+1} t_{u,v,\lambda}^{ \frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{ \frac{p-3}{2}}\varphi \vert v \vert ^{\frac{q+1}{2}}=t_{u,v,\lambda}^{p-1} \vert u \vert ^{p-1}u, \\ - \Delta \psi +\beta \psi -\frac{2}{p+1} t_{u,v,\lambda}^{ \frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}\psi =s_{u,v,\lambda}^{q-1} \vert v \vert ^{q-1}v, \\ \varphi (x)\rightarrow 0,\qquad \psi (x)\rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty , \end{cases} $$

(2.15)

has a unique solution $(\varphi , \psi )\in H_{r}\setminus \{ (0, 0)\}$. Then we can choose $\lambda _{k}$ small enough such that for any $\varphi , \psi \in H_{r}^{1}(\mathbb{R}^{3})$,

$$ \begin{aligned} \int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \varphi \,dx &= \frac{ \Vert \varphi \Vert _{\alpha}^{2}- \frac{2}{q+1} t_{u,v,\lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{\frac{p-3}{2}}\varphi ^{2} \vert v \vert ^{\frac{q+1}{2}} \,dx }{t_{u,v,\lambda}^{p-1}} \\ &\geq \frac{\frac{1}{2} \Vert \varphi \Vert _{\alpha}^{2}}{t_{u,v,\lambda}^{p-1}}>0 \end{aligned} $$

and

$$ \begin{aligned} \int _{\mathbb{R}^{3}} \vert v \vert ^{q-1}v \psi \,dx &= \frac{ \Vert \psi \Vert _{\beta}^{2}- \frac{2}{p+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}\psi ^{2} \,dx }{s_{u,v,\lambda}^{q-1}} \\ &\geq \frac{\frac{1}{2} \Vert \psi \Vert _{\beta}^{2}}{s_{u,v,\lambda}^{q-1}}>0. \end{aligned} $$

Define

$$ \mu :=\frac{1}{\int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \varphi \,dx },\qquad \nu :=\frac{1}{\int _{\mathbb{R}^{3}} \vert v \vert ^{q-1}v \psi \,dx }, $$

then $\mu >0$, $\nu >0$ and $(\widetilde{\varphi}, \widetilde{\psi}):=(\mu \varphi , \nu \psi )$ is the unique solution of

$$ \textstyle\begin{cases} -\Delta \widetilde{\varphi}+\alpha \widetilde{\varphi}-\frac{2}{q+1} t_{u,v,\lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{\frac{p-3}{2}}\widetilde{\varphi} \vert v \vert ^{ \frac{q+1}{2}}=\mu t_{u,v,\lambda}^{p-1} \vert u \vert ^{p-1}u, \\ - \Delta \widetilde{\psi}+\beta \widetilde{ \psi}-\frac{2}{p+1} t_{u,v, \lambda}^{\frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}\widetilde{\psi } =\nu s_{u,v, \lambda}^{q-1} \vert v \vert ^{q-1}v, \\ \int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u\widetilde{\varphi} \,dx =\int _{ \mathbb{R}^{3}} \vert v \vert ^{q-1}v\widetilde{\psi} \,dx =1, \\ \widetilde{\varphi}(x)\rightarrow 0,\qquad \widetilde{\psi}(x) \rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty . \end{cases} $$

(2.16)

Fixed any $k\in \mathbb{N}$, we define

$$ A_{1}:=\bigl\{ u\in X_{k+1}: \vert u \vert _{p+1}=1 \bigr\} ,\qquad A_{2}:=\bigl\{ v\in X_{k+1}: \vert v \vert _{q+1}=1\bigr\} . $$

There is an odd homeomorphism from $S^{k}$ to $A_{1}$ and $A_{2}$. By Lemma $2.2 (1)$, $A : =A_{1}\times A_{2}\in \Gamma ^{(k+1, k+1)}$. Observe that from (2.1) we deduce that $A\subset B_{m}$, and so by (2.13),

$$ \sup_{(u,v)\in A} \sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)< c_{k}. $$

Define

$$ \Gamma _{\lambda}^{(k_{1}, k_{2})}:=\Bigl\{ A\in \Gamma ^{(k_{1}, k_{2})} : A\subset B_{\widetilde{m}}, \sup_{(u,v)\in A} \sup _{t,s\geq 0} \Phi _{\lambda}(tu, sv)< c_{k} \Bigr\} . $$

Observe that $\Gamma _{\lambda}^{(k_{1}, k_{2})}\neq \emptyset$, $\Gamma _{\lambda}^{(k_{1}, k_{2})}\subset \Gamma _{\lambda}^{(k_{1}', k_{2}')}$ when $k_{1}\geq k_{1}'$ and $k_{2}\geq k_{2}'$. We are now ready to define a sequence of minimax energy levels which will turn out to be critical levels for $\Phi _{\lambda}$ over $\mathcal{A}$. For every $k_{1}, k_{2}\in [2, k+1]$ and $0<\delta <2^{-\frac{1}{p+1}}$, define

$$ d_{\lambda ,\delta}^{k_{1},k_{2}}:=\inf_{A\in \Gamma _{\lambda}^{(k_{1},k_{2})}} \sup_{A\backslash \mathcal{P}_{\delta}}\sup_{t,s\geq 0} \Phi _{ \lambda}(tu, sv). $$

(2.17)

It is easy to see that

$$ d_{\lambda ,\delta}^{k_{1},k_{2}}< c_{k}\quad \text{for any } 0< \delta < 2^{-\frac{1}{p+1}}, 2\leq k_{1}, k_{2}\leq k+1. $$

(2.18)

As a step towards to the proof of Theorem 1.1, we will prove that $d_{\lambda ,\delta}^{k_{1},k_{2}}$ is indeed a critical level of $\Phi _{\lambda}$ for δ sufficiently small. To prove Theorem 1.1, it is necessary to find a pseudogradient for $\Phi _{\lambda}$ over $\mathcal{A}$ for which $\mathcal{P}_{\delta}$ is positively invariant for the associated flow. We can now define the operator

$$ K: B_{\widetilde{m}}^{*}\rightarrow H_{r};\quad (u, v)\mapsto ( \widetilde{\varphi}, \widetilde{\psi}), $$

that is, for any $(u, v)\in B_{\widetilde{m}}^{*}$, $K(u, v)=(\widetilde{\varphi}, \widetilde{\psi})$ is the unique solution of (2.16). It is easy to prove that $K(\sigma _{i}(u, v))=\sigma _{i}(K(u, v))$, $i=1, 2$.

Now, we give some property of the operator K. We can now prove that K is a compact $C^{1}$ operator.

Lemma 2.5

The operator K is of class $C^{1}$.

Proof

Define $C^{1}$ maps $J_{i}: B_{\widetilde{m}}^{*}\times H_{r}^{1}(\mathbb{R}^{3})\times \mathbb{R}\rightarrow H_{r}^{1}(\mathbb{R}^{3})\times \mathbb{R}$, $i=1, 2$, by

$$ \begin{aligned} &J_{1} \bigl((u, v), \omega , \gamma \bigr) \\ &= \biggl(\omega -(-\Delta +\alpha )^{-1} \biggl( \frac{2}{q+1} t_{u,v, \lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{\frac{p-3}{2}}\omega \vert v \vert ^{\frac{q+1}{2}} +\gamma t_{u,v, \lambda}^{p-1} \vert u \vert ^{p-1}u\biggr), \\ &\qquad \int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \omega \,dx -1 \biggr) \end{aligned} $$

and

$$ \begin{aligned} &J_{2} \bigl((u, v), \omega , \gamma \bigr) \\ &\quad = \biggl(\omega -(-\Delta +\beta )^{-1}\biggl( \frac{2}{p+1} t_{u,v,\lambda}^{ \frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}} \omega +\gamma s_{u,v,\lambda}^{q-1} \vert v \vert ^{q-1}v\biggr), \\ &\quad \quad \int _{\mathbb{R}^{3}} \vert v \vert ^{q -1}v \omega \,dx -1 \biggr) \end{aligned} $$

then by (2.16), $J_{1} ((u, v), \widetilde{\varphi}, \mu )=J_{2} ((u, v), \widetilde{\psi}, \nu )=0$. Moreover, the derivatives of $J_{1}$ and $J_{2}$ with respect to $(\omega , \gamma )$ at the point $((u, v), \widetilde{\varphi}, \mu )$ and $((u, v), \widetilde{\psi}, \nu )$ in the direction $(\omega _{0}, \gamma _{0})$, respectively, are

$$ \begin{aligned}& D_{\omega ,\gamma}J_{1} \bigl((u, v), \widetilde{\varphi}, \mu \bigr) (\omega _{0}, \gamma _{0}) \\ &\quad = \biggl(\omega _{0}-(-\Delta +\alpha )^{-1} \biggl( \frac{2}{q+1} t_{u,v, \lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{\frac{p-3}{2}}\omega _{0} \vert v \vert ^{\frac{q+1}{2}} +\gamma _{0} t_{u,v, \lambda}^{p-1} \vert u \vert ^{p-1}u\biggr), \\ &\qquad \int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \omega _{0} \,dx \biggr) \end{aligned} $$

and

$$ \begin{aligned} &D_{\omega ,\gamma}J_{2} \bigl((u, v), \widetilde{\psi}, \nu \bigr) (\omega _{0}, \gamma _{0}) \\ &\quad = \biggl(\omega _{0}-(-\Delta +\beta )^{-1}\biggl( \frac{2}{p+1} t_{u,v, \lambda}^{\frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}} \omega _{0} +\gamma _{0} s_{u,v,\lambda}^{q-1} \vert v \vert ^{q-1}v\biggr), \\ &\qquad \int _{\mathbb{R}^{3}} \vert v \vert ^{q -1}v \omega _{0} \,dx \biggr). \end{aligned} $$

We claim that $D_{\omega ,\gamma}J_{1} ((u, v), \widetilde{\varphi}, \mu )$ and $D_{\omega ,\gamma}J_{2} ((u, v), \widetilde{\psi}, \nu )$ are bijective maps. In fact, for any $(\omega , \gamma )\in H_{r}^{1}(\mathbb{R}^{3})\times \mathbb{R}$, the following linear problems

$$\begin{aligned}& -\Delta \omega _{1}+\alpha \omega _{1}- \frac{2}{q+1} t_{u,v,\lambda}^{ \frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{ \frac{p-3}{2}}\omega _{1} \vert v \vert ^{\frac{q+1}{2}}=-\Delta \omega +\alpha \omega ,\\& -\Delta \omega _{2}+\alpha \omega _{2}- \frac{2}{q+1} t_{u,v,\lambda}^{ \frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{ \frac{p-3}{2}}\omega _{2} \vert v \vert ^{\frac{q+1}{2}}=t_{u,v,\lambda}^{p-1} \vert u \vert ^{p-1}u, \end{aligned}$$

have unique solutions $\omega _{1}, \omega _{2}\in H_{r}^{1}(\mathbb{R}^{3})$, $\omega _{2}\neq 0$ by $u\in B_{\widetilde{m}}^{*}$ and (2.12), then we define

$$ \gamma _{0}= \frac{\gamma -\int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \omega _{1} \,dx }{\int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \omega _{2} \,dx }, $$

we have

$$ D_{\omega ,\gamma}J_{1} \bigl((u, v), \widetilde{\varphi}, \mu \bigr) ( \omega _{1}+\gamma _{0} \omega _{2}, \gamma _{0})=(\omega , \gamma ), $$

that is, $D_{\omega ,\gamma}J_{1} ((u, v), \widetilde{\varphi}, \mu )$ is surjective. Similarly, $D_{\omega ,\gamma}J_{2} ((u, v), \widetilde{\psi}, \nu )$ is surjective.

If $D_{\omega ,\gamma}J_{1} ((u, v), \widetilde{\varphi}, \mu )(\omega _{0}, \gamma _{0})=(0, 0)$, then

$$ \textstyle\begin{cases} -\Delta \omega _{0}+\alpha \omega _{0} = \frac{2}{q+1} t_{u,v, \lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{\frac{p-3}{2}}\omega _{0} \vert v \vert ^{\frac{q+1}{2}} +\gamma _{0} t_{u,v, \lambda}^{p-1} \vert u \vert ^{p-1}u, \\ \int _{\mathbb{R}^{3}} \vert u \vert ^{p-1}u \omega _{0} \,dx =0, \end{cases} $$

so $\omega _{0}\equiv 0$, $\gamma _{0} t_{u,v,\lambda}^{p-1}|u|^{p-1}u\equiv 0$, by $t_{u,v,\lambda}>0$, $u\in B_{\widetilde{m}}^{*}$, we have $\gamma _{0}=0$, this implies $D_{\omega ,\gamma}J_{1} ((u, v), \widetilde{\varphi}, \mu )$ is injective. Therefore, $D_{\omega ,\gamma}J_{1} ((u, v), \widetilde{\varphi}, \mu )$ is bijective. Similarly, $D_{\omega ,\gamma}J_{2} ((u, v), \widetilde{\psi}, \nu )$ is a bijective map. Then we can apply the implicit theorem to the $C^{1}$ maps $D_{\omega ,\gamma}J_{1} ((u, v), \widetilde{\varphi}, \mu )$ and $D_{\omega ,\gamma}J_{2} ((u, v), \widetilde{\psi}, \nu )$, we have the conclusions. □

Lemma 2.6

Let $\{(u_{n}, v_{n})\}_{n\geq 1}\subset B_{\widetilde{m}}$. For any $0<\lambda <\lambda _{k}$, there exists $(\widetilde{\varphi}_{0}, \widetilde{\psi}_{0})\in H_{r}$ such that, up to a subsequence,

$$ K(u_{n}, v_{n})\rightarrow (\widetilde{ \varphi}_{0}, \widetilde{\psi}_{0}),\quad \textit{strongly in } H_{r}. $$

Proof

Since $\{(u_{n}, v_{n})\}_{n\geq 1}\subset B_{\widetilde{m}}$, we have

$$\begin{aligned}& (u_{n}, v_{n})\rightharpoonup (u_{0}, v_{0})\quad \text{weakly in } H_{r},\\& u_{n}\rightarrow u_{0},\quad \text{strongly in } L^{p+1}\bigl( \mathbb{R}^{3}\bigr),\\& v_{n}\rightarrow v_{0},\quad \text{strongly in } L^{q+1}\bigl( \mathbb{R}^{3}\bigr), \end{aligned}$$

and $|u_{0}|_{p+1}=|v_{0}|_{q+1}=1$. By (2.12), we also have

$$ t_{u_{n},v_{n},\lambda}\rightarrow t_{u_{0},v_{0},\lambda}>0,\qquad s_{u_{n},v_{n}, \lambda} \rightarrow s_{u_{0},v_{0},\lambda}>0. $$

Then by (2.3), (2.7), (2.12), and (2.15),

$$ \begin{aligned}\frac{1}{2} \Vert \varphi _{n} \Vert ^{2}_{\alpha }&\leq \Vert \varphi _{n} \Vert ^{2}_{\alpha}- \frac{2}{q+1} t_{u_{n},v_{n},\lambda}^{ \frac{p-3}{2}} s_{u_{n},v_{n},\lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{\frac{p-3}{2}}\varphi _{n}^{2} \vert v_{n} \vert ^{ \frac{q+1}{2}} \,dx \\ & =t_{u_{n},v_{n},\lambda}^{p-1} \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{p-1}u_{n} \varphi _{n} \,dx \\ &\leq C \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{p} \vert \varphi _{n} \vert \,dx \\ &\leq C \vert u_{n} \vert _{p+1}^{p} \vert \varphi _{n} \vert _{p+1} \leq C \Vert \varphi _{n} \Vert _{\alpha}. \end{aligned} $$

Similar estimates hold for $\psi _{n}$, we get $\|\psi _{n}\|^{2}_{\beta}\leq C \| \psi _{n}\|_{\beta}$, so $\{(\varphi _{n}, \psi _{n})\}_{n\geq 1}\subset H_{r}$ are bounded. Thus

$$\begin{aligned}& (\varphi _{n}, \psi _{n})\rightharpoonup (\varphi _{0}, \psi _{0}) \quad \text{weakly in } H_{r},\\& \varphi _{n}\rightarrow \varphi _{0},\quad \text{strongly in } L^{p+1}\bigl( \mathbb{R}^{3} \bigr),\\& \psi _{n}\rightarrow \psi _{0},\quad \text{strongly in } L^{q+1}\bigl( \mathbb{R}^{3} \bigr). \end{aligned}$$

Then by (2.15) and Hölder’s inequality,

$$ \begin{aligned}& \int _{\mathbb{R}^{3}} \bigl(\nabla \varphi _{n} \nabla ( \varphi _{n}-\varphi _{0})+\alpha \varphi _{n} (\varphi _{n}- \varphi _{0}) \bigr) \,dx \\ &\quad = \frac{2}{q+1} t_{u_{n},v_{n},\lambda}^{\frac{p-3}{2}} s_{u_{n},v_{n}, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{ \frac{p-3}{2}}\varphi _{n}(\varphi _{n}-\varphi _{0}) \vert v_{n} \vert ^{ \frac{q+1}{2}} \,dx \\ &\quad \quad{}+ t_{u_{n},v_{n},\lambda}^{p-1} \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{p-1}u_{n} (\varphi _{n}-\varphi _{0}) \,dx \\ &\quad \rightarrow 0,\quad \text{as } n\rightarrow \infty . \end{aligned} $$

Hence,

$$ \Vert \varphi _{n} \Vert ^{2}_{\alpha }= \int _{\mathbb{R}^{3}} (\nabla \varphi _{n} \nabla \varphi _{0}+\alpha \varphi _{n} \varphi _{0} ) \,dx +o(1)= \Vert \varphi _{0} \Vert ^{2}_{\alpha}+o(1). $$

Similarly, we have $\|\psi _{n}\|^{2}_{\beta}=\|\psi _{0}\|^{2}_{\beta}+o(1)$. Therefore, we have $(\varphi _{n}, \psi _{n})\rightarrow (\varphi _{0}, \psi _{0})$ strongly in $H_{r}$ and $(\varphi _{0}, \psi _{0})$ satisfies

$$ \textstyle\begin{cases} -\Delta \varphi _{0}+\alpha \varphi _{0}-\frac{2}{q+1} t_{u_{0},v_{0}, \lambda}^{\frac{p-3}{2}} s_{u_{0},v_{0},\lambda}^{\frac{q+1}{2}} \lambda \vert u_{0} \vert ^{\frac{p-3}{2}}\varphi _{0} \vert v_{0} \vert ^{\frac{q+1}{2}}=t_{u_{0},v_{0}, \lambda}^{p-1} \vert u_{0} \vert ^{p-1}u_{0}, \\ -\Delta \psi _{0}+\beta \psi _{0}-\frac{2}{p+1} t_{u_{0},v_{0}, \lambda}^{\frac{p+1}{2}} s_{u_{0},v_{0},\lambda}^{\frac{q-3}{2}} \lambda \vert u_{0} \vert ^{\frac{p+1}{2}} \vert v_{0} \vert ^{\frac{q-3}{2}}\psi _{0} =s_{u_{0},v_{0}, \lambda}^{q-1} \vert v_{0} \vert ^{q-1}v_{0}, \\ \varphi _{0}(x)\rightarrow 0,\qquad \psi _{0}(x)\rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty , \end{cases} $$

since $|u_{0}|_{p+1}=|v_{0}|_{q+1}=1$, so $\varphi _{0}\neq 0$, $\psi _{0}\neq 0$ and

$$\begin{aligned}& \mu _{n}:= \frac{1}{\int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{p-1}u_{n}\varphi _{n} \,dx } \rightarrow \frac{1}{\int _{\mathbb{R}^{3}} \vert u_{0} \vert ^{p-1}u_{0}\varphi _{0} \,dx }=: \mu _{0},\\& \nu _{n}:= \frac{1}{\int _{\mathbb{R}^{3}} \vert v_{n} \vert ^{q-1}v_{n}\psi _{n} \,dx } \rightarrow \frac{1}{\int _{\mathbb{R}^{3}} \vert v_{0} \vert ^{q-1}v_{0}\psi _{0} \,dx }=: \nu _{0}. \end{aligned}$$

We see that

$$ (\widetilde{\varphi}_{n}, \widetilde{\psi}_{n})=(\mu _{n} \varphi _{n}, \nu _{n}\psi _{n})\rightarrow (\mu _{0} \varphi _{0}, \nu _{0} \psi _{0})=:(\widetilde{\varphi}_{0}, \widetilde{\psi}_{0}),\quad \text{strongly in } H_{r}. $$

This completes the proof. □

Define

$$ B_{\widetilde{m}, \lambda}:=\Bigl\{ (u, v)\in B_{\widetilde{m}}: \sup _{t,s \geq 0} \Phi _{\lambda}(tu, sv)< c_{k}\Bigr\} , $$

then by (2.13) we obtain $B_{m}\subset B_{\widetilde{m}, \lambda}$.

Lemma 2.7

For any $0<\delta <2^{-\frac{1}{p+1}}$ sufficiently small, we have that

$$ \operatorname{dist} \bigl(K(u, v), \mathcal{P} \bigr)< \frac{\delta}{2},\quad \forall (u, v)\in B_{\widetilde{m}, \lambda},\qquad \operatorname{dist} \bigl((u, v), \mathcal{P} \bigr)< \delta . $$

Proof

Suppose by contradiction that there exist $\delta _{n}\rightarrow 0$ and $(u_{n}, v_{n})\in B_{\widetilde{m}, \lambda}$ satisfying $\operatorname{dist} ((u_{n}, v_{n}), \mathcal{P} )<\delta _{n}$ and $\operatorname{dist} (K(u_{n}, v_{n}), \mathcal{P} )\geq \frac{\delta _{n}}{2}$. We suppose that $\operatorname{dist} ((u_{n}, v_{n}), \mathcal{P} )=|u_{n}^{-}|_{p+1}$ without loss of generality. Let $(\widetilde{\varphi}_{n}, \widetilde{\psi}_{n})=K(u_{n}, v_{n})$ and $\widetilde{\varphi}_{n}=\mu _{n}\varphi _{n}$, $\widetilde{\psi}_{n}=\nu _{n} \psi _{n}$. By a similar proof as in Lemma 2.6, we have that $\mu _{n}$ and $\nu _{n}$ are uniformly bounded. By (2.12), we can take $\lambda _{k}$ smaller if necessary such that for any $\lambda \in (0, \lambda _{k})$ and $(u, v)\in B_{\widetilde{m}}^{*}$, we get

$$ \frac{1}{2} \bigl\Vert \widetilde{\varphi}_{n}^{-} \bigr\Vert _{\alpha}^{2}\leq \bigl\Vert \widetilde{ \varphi}_{n}^{-} \bigr\Vert _{\alpha}^{2}- \frac{2}{q+1} t_{u_{n},v_{n}, \lambda}^{\frac{p-3}{2}} s_{u_{n},v_{n},\lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{\frac{p-3}{2}}\bigl({ \widetilde{\varphi}_{n}^{-}} \bigr)^{2} \vert v_{n} \vert ^{\frac{q+1}{2}} \,dx . $$

This together with (2.7) and (2.16) allows us to get

$$\begin{aligned}& \bigl\vert \widetilde{\varphi}_{n}^{-} \bigr\vert ^{2}_{p+1}\leq \frac{1}{S_{p}} \bigl\Vert \widetilde{\varphi}_{n}^{-} \bigr\Vert _{\alpha}^{2} \\& \quad \leq C \biggl( \bigl\Vert \widetilde{\varphi}_{n}^{-} \bigr\Vert _{\alpha}^{2}- \frac{2}{q+1} t_{u_{n},v_{n},\lambda}^{\frac{p-3}{2}} s_{u_{n},v_{n}, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{ \frac{p-3}{2}}\bigl({\widetilde{\varphi}_{n}^{-}} \bigr)^{2} \vert v_{n} \vert ^{ \frac{q+1}{2}} \,dx \biggr) \\& \quad = -C\mu _{n} t_{u_{n},v_{n},\lambda}^{p-1} \int _{\mathbb{R}^{3}} \vert u_{n} \vert ^{p-1}u_{n} \widetilde{\varphi}_{n}^{-} \,dx \\& \quad \leq C \int _{\mathbb{R}^{3}}\bigl(u_{n} ^{-} \bigr)^{p} \widetilde{\varphi}_{n}^{-} \,dx \leq C \bigl\vert u_{n} ^{-} \bigr\vert ^{p}_{p+1} \bigl\vert \widetilde{\varphi}_{n}^{-} \bigr\vert _{p+1} \leq C\delta _{n}^{p} \bigl\vert \widetilde{\varphi}_{n}^{-} \bigr\vert _{p+1}, \end{aligned}$$

and hence $\operatorname{dist} (K(u_{n}, v_{n}), \mathcal{P} )\leq | \widetilde{\varphi}_{n}^{-}|_{p+1}\leq C\delta _{n}^{p} < \frac{\delta _{n}}{2}$ for n sufficiently large, which is a contradiction. This completes the proof. □

Now define a map

$$ V: B_{\widetilde{m}}^{*}\rightarrow H_{r};\qquad (u, v)\mapsto (u, v)-K(u, v). $$

It is easy to prove that $V(\sigma _{i}(u, v))=\sigma _{i}(V(u, v))$, $i=1, 2$. We will prove that if $(u, v)\in B_{\widetilde{m}}\backslash \mathcal{P}$, $V(u, v)=0$, then $(t_{u,v,\lambda}u, s_{u,v,\lambda}v)$ is a sign-changing solution of Eq. (1.1). Firstly, we prove that V satisfies the Palais–Smale type condition and V is a pseudogradient for $\sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)$ over $B_{\widetilde{m}}$. Denote $\Psi _{\lambda}(u, v):=\sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)$.

Lemma 2.8

(Palais–Smale type condition) Let $(u_{n}, v_{n})\in B_{\widetilde{m}}$ be such that

$$ \Psi _{\lambda}(u_{n}, v_{n})\rightarrow c< c_{k}\quad \textit{and} \quad V(u_{n}, v_{n})\rightarrow 0\quad \textit{strongly in } H_{r}. $$

Then there exists $(u_{0}, v_{0})\in B_{\widetilde{m}}$ such that $(u_{n}, v_{n})\rightarrow (u_{0}, v_{0})$ strongly in $H_{r}$, up to a subsequence, and $V(u_{0}, v_{0})=0$. We also have

$$ \textit{For any } (u, v)\in B_{\widetilde{m}},\quad \bigl\langle \nabla \Psi _{\lambda}(u, v), V(u, v)\bigr\rangle _{H_{r}}\geq \frac{T_{2}^{2}}{2} \bigl\Vert V(u, v) \bigr\Vert _{H_{r}}^{2}. $$

Proof

Similar as Lemma 2.6, we have, up to a subsequence,

$$\begin{aligned}& (u_{n}, v_{n})\rightharpoonup (u_{0}, v_{0})\quad \text{weakly in } H_{r},\\& K(u_{n}, v_{n})\rightarrow (\widetilde{ \varphi}_{0}, \widetilde{\psi}_{0})\quad \text{strongly in } H_{r}. \end{aligned}$$

Then we have, as $n\rightarrow \infty$,

$$ \begin{aligned}o(1)&=\bigl\langle V(u_{n}, v_{n}), (u_{n}-u_{0}, v_{n}-v_{0}) \bigr\rangle _{H_{r}} \\ &= \langle u_{n}-\widetilde{\varphi}_{n}, u_{n}-u_{0} \rangle _{H_{r}}+ \langle v_{n}-\widetilde{\psi}_{n}, v_{n}-v_{0} \rangle _{H_{r}} \\ &= \langle u_{n}, u_{n}-u_{0} \rangle _{H_{r}}-\langle \widetilde{\varphi}_{n}, u_{n}-u_{0} \rangle _{H_{r}}+\langle v_{n}, v_{n}-v_{0}\rangle _{H_{r}}-\langle \widetilde{\psi}_{n}, v_{n}-v_{0} \rangle _{H_{r}} \end{aligned} $$

whence

$$ \langle u_{n}, u_{n}-u_{0} \rangle _{H_{r}}+\langle v_{n}, v_{n}-v_{0} \rangle _{H_{r}}=o(1). $$

Then $(u_{n}, v_{n})\rightarrow (u_{0}, v_{0})$ strongly in $H_{r}$ and $(u_{0}, v_{0})\in \overline{B}_{\widetilde{m}}$,

$$ \Phi _{\lambda}(t_{u_{0},v_{0},\lambda}u_{0}, s_{u_{0},v_{0},\lambda}v_{0})= \lim_{n\rightarrow \infty}\Phi _{\lambda}(t_{u_{n},v_{n},\lambda}u_{n}, s_{u_{n},v_{n},\lambda}v_{n})=c< c_{k}, $$

then by (2.13), we have $(u_{0}, v_{0})\in B_{\widetilde{m}}$, $V(u_{0}, v_{0})=\lim_{n\rightarrow \infty}V(u_{n}, v_{n})=0$.

Finally, we prove that V is a pseudogradient for $\Psi _{\lambda}(u, v)$ over $B_{\widetilde{m}}$. By (2.9) and (2.10) we can prove that

$$\begin{aligned}& \begin{aligned}\bigl\langle \nabla \Psi _{\lambda}(u, v), (\omega , 0) \bigr\rangle _{H_{r}}&=t_{u,v,\lambda}^{2} \int _{\mathbb{R}^{3}} ( \nabla u \nabla \omega +\alpha u \omega ) \,dx \\ &\quad{}-\frac{2\lambda}{q+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \int _{\mathbb{R}^{3}} \vert u \vert ^{\frac{p-3}{2}}u \omega \vert v \vert ^{\frac{q+1}{2}} \,dx, \end{aligned} \end{aligned}$$

(2.19)

$$\begin{aligned}& \begin{aligned}\bigl\langle \nabla \Psi _{\lambda}(u, v), (0, \omega ) \bigr\rangle _{H_{r}}&=s_{u,v,\lambda}^{2} \int _{\mathbb{R}^{3}} ( \nabla v \nabla \omega +\beta v \omega ) \,dx \\ &\quad{}-\frac{2\lambda}{p+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \int _{\mathbb{R}^{3}} \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{ \frac{q-3}{2}}v\omega \,dx \end{aligned} \end{aligned}$$

(2.20)

hold for any $(u, v)\in \mathcal{B}_{\widetilde{m}}$ and $\omega \in H_{r}^{1}(\mathbb{R}^{3})$. We can take $\lambda _{k}$ smaller if necessary such that for any $\lambda \in (0, \lambda _{k})$ by (2.19), (2.20), (2.12), and (2.16)

$$ \begin{aligned} &\bigl\langle \nabla \Psi _{\lambda}(u, v), V(u, v) \bigr\rangle _{H_{r}} \\ &\quad = t_{u,v,\lambda}^{2} \int _{\mathbb{R}^{3}} \bigl( \nabla u \nabla (u- \widetilde{\varphi} )+ \alpha u (u-\widetilde{\varphi} ) \bigr) \,dx \\ &\quad \quad{}+s_{u,v,\lambda}^{2} \int _{\mathbb{R}^{3}} \bigl( \nabla v \nabla (v-\widetilde{\psi} )+ \beta v (v-\widetilde{\psi} ) \bigr) \,dx \\ &\quad \quad{}-\frac{2}{q+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{ \frac{p-3}{2}}u(u-\widetilde{ \varphi} ) \vert v \vert ^{\frac{q+1}{2}} \,dx \\ &\quad \quad{}-\frac{2}{p+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}v(v-\widetilde{\psi}) \,dx \\ &\quad =t_{u,v,\lambda}^{2} \Vert u-\widetilde{\varphi} \Vert _{\alpha}^{2}+ s_{u,v, \lambda}^{2} \Vert v-\widetilde{\psi} \Vert _{\beta}^{2} \\ &\quad \quad{}+t_{u,v,\lambda}^{2} \int _{\mathbb{R}^{3}} \bigl( \nabla \widetilde{\varphi} \nabla (u- \widetilde{\varphi} )+\alpha \widetilde{\varphi}(u-\widetilde{\varphi} ) \bigr) \,dx \\ &\quad \quad{}+s_{u,v,\lambda}^{2} \int _{\mathbb{R}^{3}} \bigl( \nabla \widetilde{ \psi} \nabla (u- \widetilde{\psi} )+\alpha \widetilde{\psi}(v-\widetilde{\psi} ) \bigr) \,dx \\ &\quad \quad{}-\frac{2}{q+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{ \frac{p-3}{2}}u(u-\widetilde{ \varphi} ) \vert v \vert ^{\frac{q+1}{2}} \,dx \\ &\quad \quad{}-\frac{2}{p+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}v(v-\widetilde{\psi}) \,dx \\ &\quad =t_{u,v,\lambda}^{2} \Vert u-\widetilde{\varphi} \Vert _{\alpha}^{2}+ s_{u,v, \lambda}^{2} \Vert v-\widetilde{\psi} \Vert _{\beta}^{2} \\ &\quad \quad{}-\frac{2}{q+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{ \frac{p-3}{2}}(u-\widetilde{ \varphi} )^{2} \vert v \vert ^{\frac{q+1}{2}} \,dx \\ &\quad \quad{}-\frac{2}{p+1} t_{u,v,\lambda}^{\frac{p+1}{2}} s_{u,v, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}(v-\widetilde{\psi})^{2} \,dx \\ &\quad \geq \frac{t_{u,v,\lambda}^{2}}{2} \Vert u-\widetilde{\varphi} \Vert _{ \alpha}^{2}+ \frac{s_{u,v,\lambda}^{2}}{2} \Vert v-\widetilde{ \psi} \Vert _{ \beta}^{2} \\ &\quad \geq \frac{T_{2}^{2}}{2} \bigl( \Vert u-\widetilde{\varphi} \Vert _{\alpha}^{2}+ \Vert v-\widetilde{\psi} \Vert _{\beta}^{2} \bigr)=\frac{T_{2}^{2}}{2} \bigl\Vert V(u, v) \bigr\Vert ^{2}_{H_{r}}. \end{aligned} $$

This completes the proof. □

Lemma 2.9

There exists a unique global solution $\eta =(\eta _{1}, \eta _{2}): \mathbb{R}^{+}\times B_{\widetilde{m}, \lambda}\rightarrow H_{r}$ for the initial value problem

$$ \textstyle\begin{cases} \frac{d}{dt}\eta (t, (u, v))=-V (\eta (t, (u, v)) ), \\ \eta (0, (u, v))=(u, v)\in B_{\widetilde{m},\lambda}. \end{cases} $$

(2.21)

Moreover,

• (1) For any $t>0$ and $(u, v)\in B_{\widetilde{m},\lambda}$, there holds $\eta (t, (u, v))\in B_{\widetilde{m},\lambda}$;

• (2) For any $t>0$, $(u, v)\in B_{\widetilde{m},\lambda}$, there holds $\eta (t, \sigma _{i}(u, v))=\sigma _{i} ( \eta (t, (u, v)) )$, $i=1, 2$;

• (3) For any $(u, v)\in B_{\widetilde{m},\lambda}$, $\Psi _{\lambda} ( \eta (t, (u, v)) )$ is nonincreasing in t;

• (4) There exists $\delta _{0}\in (0, 2^{-\frac{1}{p+1}})$ such that, for any $0<\delta <\delta _{0}$, $(u, v)\in B_{\widetilde{m},\lambda}\cap \mathcal{P}_{\delta}$ and $t>0$, there holds $\eta (t, (u, v))\in \mathcal{P}_{\delta}$.

Proof

It follows from Lemma 2.5 that $V\in C^{1}(B_{\widetilde{m}}^{*}, H_{r})$. As $B_{\widetilde{m},\lambda}\subset B_{\widetilde{m}}\subset B_{ \widetilde{m}}^{*}$, we get that $V\in C^{1}(B_{\widetilde{m},\lambda}, H_{r})$. Then there exists a solution $\eta : [0, T_{\max})\times B_{\widetilde{m},\lambda}\rightarrow H_{r}$, where $T_{\max}$ is the maximal time such that (2.21) has a solution $\eta \in B_{\widetilde{m}}^{*}$.

For any $(u, v)\in B_{\widetilde{m},\lambda}$ and $t\in (0, T_{\max})$, there holds

$$ \begin{aligned}&\frac{d}{dt} \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p+1} \,dx \\ &\quad = -(p+1) \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p-1}\eta _{1}\bigl(t, (u, v) \bigr)V_{1} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr) \,dx \\ &\quad = -(p+1) \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p-1}\eta _{1}\bigl(t, (u, v)\bigr) \bigl[\eta _{1}\bigl(t, (u, v)\bigr)-K_{1} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)\bigr] \,dx \\ &\quad = (p+1)-(p+1) \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p+1} \,dx , \end{aligned} $$

so we have

$$ \frac{d}{dt}\biggl[e^{(p+1)t} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p+1} \,dx -1 \biggr)\biggr] =0. $$

Then

$$ \begin{aligned}e^{(p+1)t} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p+1} \,dx -1 \biggr) &= \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(0, (u, v)\bigr) \bigr\vert ^{p+1} \,dx -1 \\ &= \int _{\mathbb{R}^{3}} \vert u \vert ^{p+1} \,dx -1\equiv 0. \end{aligned} $$

Similarly, there holds

$$ \begin{aligned}e^{(q+1)t} \biggl( \int _{\mathbb{R}^{3}} \bigl\vert \eta _{2}\bigl(t, (u, v)\bigr) \bigr\vert ^{q+1} \,dx -1 \biggr) &= \int _{\mathbb{R}^{3}} \bigl\vert \eta _{2}\bigl(0, (u, v)\bigr) \bigr\vert ^{q+1} \,dx -1 \\ &= \int _{\mathbb{R}^{3}} \vert v \vert ^{q+1} \,dx -1\equiv 0, \end{aligned} $$

we deduce that for any $(u, v)\in B_{\widetilde{m},\lambda}$ and $t\in [0, T_{\max})$,

$$ \int _{\mathbb{R}^{3}} \bigl\vert \eta _{1}\bigl(t, (u, v)\bigr) \bigr\vert ^{p+1} \,dx \equiv \int _{ \mathbb{R}^{3}} \bigl\vert \eta _{2}\bigl(t, (u, v) \bigr) \bigr\vert ^{q+1} \,dx \equiv 1. $$

Thus, for any $t\in [0, T_{\max})$, $(u, v)\in B_{\widetilde{m}}$, we have $\eta (t, (u, v))\in B_{\widetilde{m}}^{*}\cap \mathcal{A}=B_{ \widetilde{m}}$. If $T_{\max}<+\infty$, then $\eta (T_{\max}, (u, v))\in \mathcal{C}_{\widetilde{m}}$. There holds $\Psi _{\lambda} (\eta (T_{\max}, (u, v)) )\geq c_{k}$ by (2.13). Moreover,

$$ \begin{aligned}\frac{d}{dt}\Psi _{\lambda} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)&= \biggl\langle \nabla \Psi _{\lambda}\bigl(\eta \bigl(t, (u, v)\bigr) \bigr), \frac{d}{dt} \eta \bigl(t, (u, v)\bigr)\biggr\rangle _{H_{r}} \\ &= -\bigl\langle \nabla \Psi _{\lambda}\bigl(\eta \bigl(t, (u, v)\bigr) \bigr), V \bigl( \eta \bigl(t, (u, v)\bigr) \bigr)\bigr\rangle _{H_{r}} \\ &\leq -\frac{T_{2}^{2}}{2} \bigl\Vert V \bigl(\eta \bigl(t, (u, v)\bigr) \bigr) \bigr\Vert ^{2}_{H_{r}} \leq 0. \end{aligned} $$

(2.22)

On the other hand, we see from $(u, v)\in B_{\widetilde{m},\lambda}$ and (2.22),

$$ \Psi _{\lambda} \bigl(\eta \bigl(T_{\max}, (u, v)\bigr) \bigr) \leq \Psi _{\lambda} \bigl(\eta \bigl(0, (u, v)\bigr) \bigr)=\Psi _{\lambda}(u, v)< c_{k}, $$

it yields a contradiction, so $T_{\max}=+\infty$, $\eta (t, (u, v))\in B_{\widetilde{m},\lambda}$ and $(1)(3)$ hold.

Since $V(\sigma _{i}(u, v))=\sigma _{i}(V(u, v))$, $i=1, 2$, then $(2)$ holds.

Take $\delta _{0}>0$ as in Lemma 2.7, note that as $t\rightarrow 0$,

$$ \begin{aligned}\eta \bigl(t, (u, v)\bigr)&= (u, v)+t \frac{d}{dt}\eta \bigl(t, (u, v)\bigr)|_{t=0}+o(t) \\ &= (u, v)-t V(u, v)+o(t)=(1-t) (u, v)+t K(u, v)+o(t), \end{aligned} $$

hence for any $0<\delta <\delta _{0}$, $(u, v)\in B_{\widetilde{m},\lambda}\cap \mathcal{P}_{\delta}$, we have

$$ \begin{aligned}\operatorname{dist} \bigl(\eta \bigl(t, (u, v)\bigr), \mathcal{P} \bigr)&= \operatorname{dist} \bigl((1-t) (u, v)+t K(u, v)+o(t), \mathcal{P} \bigr) \\ &\leq (1-t)\operatorname{dist} \bigl((u, v), \mathcal{P} \bigr)+t \operatorname{dist} \bigl(K(u, v), \mathcal{P} \bigr)+o(t) \\ &< (1-t)\delta +\frac{t\delta}{2}+o(t)< \delta , \end{aligned} $$

for sufficiently small $t>0$, and $(4)$ holds. This completes the proof. □

To prove Theorem 1.1, we will give that $d_{\lambda ,\delta}^{k_{1},k_{2}}$ is indeed critical energy level for $\delta >0$ sufficiently small.

Lemma 2.10

For any $k\in \mathbb{N}$, $k_{1}, k_{2}\in [2, k+1]$, $0<\delta <\delta _{0}$, and $0<\lambda <\lambda _{k}$, there exists $(\widetilde{u}_{0}, \widetilde{v}_{0})\in H_{r}$ such that $(\widetilde{u}_{0}, \widetilde{v}_{0})$ is a sign-changing solution of Eq. (1.1) and $\Phi _{\lambda}(\widetilde{u}_{0}, \widetilde{v}_{0})=d_{\lambda , \delta}^{k_{1},k_{2}}$.

Proof

By (2.18) we see that $d_{\lambda ,\delta}^{k_{1},k_{2}}< c_{k}$. Assume that there is small $0<\varepsilon <1$ such that for any $(u, v)\in B_{\widetilde{m},\lambda}$, $|\Psi _{\lambda}(u, v)-d_{\lambda ,\delta}^{k_{1},k_{2}}|\leq 2 \varepsilon$, $\operatorname{dist} ((u, v), \mathcal{P} )\geq \delta$, there holds $\|V(u, v)\|_{H_{r}}^{2}\geq \varepsilon$. By (2.17), there exists $A\in \Gamma _{\lambda}^{(k_{1},k_{2})}$ such that

$$ \sup_{A\backslash{\mathcal{P}_{\delta}}} \Psi _{\lambda}(u, v) < d_{ \lambda ,\delta}^{k_{1},k_{2}}+\varepsilon , $$

(2.23)

then $\sup_{A} \Psi _{\lambda}(u, v)< c_{k}$, $A\subset B_{\widetilde{m},\lambda}$. Thus we consider the set $A_{0}=\eta (\frac{4}{T_{2}^{2}}, A)$, $A_{0}\in B_{\widetilde{m},\lambda}$ by Lemma 2.9(1). From Lemma 2.2(2), Lemma 2.3, and Lemma 2.9(3), we get

$$ \sup_{A_{0}} \Psi _{\lambda}(u, v)\leq \sup _{A} \Psi _{\lambda}(u, v)< c_{k}, $$

so $A_{0}\in \Gamma _{\lambda}^{(k_{1},k_{2})}$ and $A_{0}\backslash{\mathcal{P}_{\delta}}\neq \emptyset$. Then, by (2.15), (2.19), and Lemma 2.9(3), for the $\varepsilon >0$, $t\in [0, \frac{4}{T_{2}^{2}}]$, there exists $(u, v)\in A$ such that $\eta (\frac{4}{T_{2}^{2}}, (u, v))\in A_{0}\backslash {\mathcal{P}_{ \delta}}$ satisfying

$$ \begin{aligned}d_{\lambda ,\delta}^{k_{1},k_{2}}&\leq \sup _{A_{0} \backslash {\mathcal{P}_{\delta}}} \Psi _{\lambda}(u, v)< \Psi _{ \lambda} \biggl(\eta \biggl(\frac{4}{T_{2}^{2}}, (u, v)\biggr) \biggr)+\varepsilon \\ &\leq \Psi _{\lambda} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)+\varepsilon \leq \Psi _{\lambda} (u, v )+\varepsilon < d_{\lambda ,\delta}^{k_{1},k_{2}}+2 \varepsilon . \end{aligned} $$

(2.24)

We conclude that $\|V (\eta (t, (u, v)) )\|^{2}_{H_{r}}\geq \varepsilon$ for any $t\in [0, \frac{4}{T_{2}^{2}}]$ and

$$ \begin{aligned}\frac{d}{dt}\Psi _{\lambda} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)&=- \bigl\langle \nabla \Psi _{\lambda}\bigl( \eta \bigl(t, (u, v)\bigr) \bigr), V \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)\bigr\rangle _{H_{r}} \\ &\leq -\frac{T_{2}^{2}}{2} \bigl\Vert V \bigl(\eta \bigl(t, (u, v)\bigr) \bigr) \bigr\Vert ^{2}_{H_{r}} \leq -\frac{T_{2}^{2}}{2} \varepsilon . \end{aligned} $$

Therefore, by integrating over 0 to $\frac{4}{T_{2}^{2}}$ and (2.24), we have

$$ \begin{aligned}\bigl(d_{\lambda ,\delta}^{k_{1},k_{2}}-\varepsilon \bigr)-\bigl(d_{ \lambda ,\delta}^{k_{1},k_{2}}+\varepsilon \bigr)&< \Psi _{\lambda} \biggl( \eta \biggl(\frac{4}{T_{2}^{2}}, (u, v)\biggr) \biggr)-\Psi _{\lambda}(u, v) \\ &\leq -\frac{T_{2}^{2}}{2}\varepsilon \int _{0}^{\frac{4}{T_{2}^{2}}} \,dt =-2\varepsilon , \end{aligned} $$

it yields a contradiction, and therefore, for any $\varepsilon =\frac{1}{n}>0$, there exists $(u_{n}, v_{n})\in B_{\widetilde{m},\lambda}$ such that

$$ \bigl\vert \Psi _{\lambda}(u_{n}, v_{n})-d_{\lambda ,\delta}^{k_{1},k_{2}} \bigr\vert \leq 2\varepsilon ,\qquad \bigl\Vert V(u_{n}, v_{n}) \bigr\Vert ^{2}_{H_{r}}\leq \varepsilon \quad \text{and}\quad \operatorname{dist} \bigl((u_{n}, v_{n}), \mathcal{P} \bigr)\geq \delta . $$

By Lemma 2.8, there exists $(u_{0}, v_{0})\in B_{\widetilde{m},\lambda}$ such that $(u_{n}, v_{n})\rightarrow (u_{0}, v_{0})$ strongly in $H_{r}$, up to a subsequence. Hence, we have

$$ \Psi _{\lambda}(u_{0}, v_{0})=d_{\lambda ,\delta}^{k_{1},k_{2}}, \qquad V(u_{0}, v_{0})=0\quad \text{and}\quad \operatorname{dist} \bigl((u_{0}, v_{0}), \mathcal{P} \bigr)\geq \delta . $$

We conclude that $(u_{0}, v_{0})$ is sign-changing and $(u_{0}, v_{0})=K(u_{0}, v_{0})=(\widetilde{\varphi}_{0}, \widetilde{\psi}_{0})$. It follows from (2.16) that $(u_{0}, v_{0})$ satisfies

$$ \textstyle\begin{cases} -\Delta u_{0}+\alpha u_{0}=\mu t_{u_{0},v_{0},\lambda}^{p-1} \vert u_{0} \vert ^{p-1}u_{0} +\frac{2}{q+1} t_{u_{0},v_{0},\lambda}^{\frac{p-3}{2}} s_{u_{0},v_{0}, \lambda}^{\frac{q+1}{2}} \lambda \vert u_{0} \vert ^{\frac{p-3}{2}}u_{0} \vert v_{0} \vert ^{ \frac{q+1}{2}}, \\ -\Delta v_{0}+\beta v_{0}=\nu s_{u_{0},v_{0},\lambda}^{q-1} \vert v_{0} \vert ^{q-1}v_{0} +\frac{2}{p+1} t_{u_{0},v_{0},\lambda}^{\frac{p+1}{2}} s_{u_{0},v_{0}, \lambda}^{\frac{q-3}{2}} \lambda \vert u_{0} \vert ^{\frac{p+1}{2}} \vert v_{0} \vert ^{ \frac{q-3}{2}}v_{0}, \\ u_{0}(x)\rightarrow 0,\qquad v_{0}(x)\rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty . \end{cases} $$

(2.25)

On the other hand, $t_{u_{0},v_{0},\lambda}$ and $s_{u_{0},v_{0},\lambda}$ satisfy

$$\begin{aligned}& \Vert u_{0} \Vert _{\alpha}^{2}=t_{u_{0},v_{0},\lambda}^{p-1} \vert u_{0} \vert _{p+1}^{p+1}+ \frac{2}{q+1} t_{u_{0},v_{0},\lambda}^{\frac{p-3}{2}} s_{u_{0},v_{0}, \lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{0} \vert ^{ \frac{p+1}{2}} \vert v_{0} \vert ^{\frac{q+1}{2}} \,dx ,\\& \Vert v_{0} \Vert _{\beta}^{2}=s_{u_{0},v_{0},\lambda}^{q-1} \vert v_{0} \vert _{q+1}^{q+1}+ \frac{2}{p+1} t_{u_{0},v_{0},\lambda}^{\frac{p+1}{2}} s_{u_{0},v_{0}, \lambda}^{\frac{q-3}{2}} \lambda \int _{ \mathbb{R}^{3}} \vert u_{0} \vert ^{ \frac{p+1}{2}} \vert v_{0} \vert ^{\frac{q+1}{2}} \,dx , \end{aligned}$$

then we have $\mu =\nu =1$. Hence, we have that $(t_{u_{0},v_{0},\lambda} u_{0}, s_{u_{0},v_{0},\lambda} v_{0})$ is a sign-changing solution of Eq. (1.1) by problem (2.25) and

$$ \Phi _{\lambda}(\widetilde{u}_{0}, \widetilde{v}_{0}):= \Phi _{ \lambda}(t_{u_{0},v_{0},\lambda} u_{0}, s_{u_{0},v_{0},\lambda} v_{0})= \Psi _{\lambda}(u_{0}, v_{0})=d_{\lambda ,\delta}^{k_{1},k_{2}}. $$

This completes the proof. □

Proof of Theorem 1.1

Observe that from Lemma 2.10 we know that for any $k\in \mathbb{N}$, $k_{1}, k_{2}\in [2, k+1]$, $0<\delta <\delta _{0}$, and $0<\lambda <\lambda _{k}$, there exists a sign-changing solution $(\widetilde{u}_{0}, \widetilde{v}_{0})$ with $\Phi _{\lambda}(\widetilde{u}_{0}, \widetilde{v}_{0})=d_{\lambda , \delta}^{k_{1},k_{2}}$. For any fixed $k_{1}\in [2, k+1]$, we have

$$ d_{\lambda ,\delta}^{k_{1},2}\leq d_{\lambda ,\delta}^{k_{1},3} \leq \cdots \leq d_{\lambda ,\delta}^{k_{1},k}\leq d_{\lambda ,\delta}^{k_{1},k+1}< c_{k}. $$

Suppose that problem (1.1) has at most $k-1$ sign-changing solutions by contradiction, then there exists $k_{2}\in [2, k]$ satisfying

$$ d:=d_{\lambda ,\delta}^{k_{1},k_{2}}=d_{\lambda ,\delta}^{k_{1},k_{2}+1}< c_{k}. $$

Now define

$$ \mathcal{M}:=\bigl\{ (u, v)\in B_{\widetilde{m}}: (u, v) \text{ sign-changing}, \Psi _{\lambda}(u, v)=d, V(u, v)=0\bigr\} , $$

then $\mathcal{M}\subset \mathcal{F}$ is finite. So there exist $N\in [1, k-1]$ and $\{(u_{n}, v_{n})\}_{1\leq n\leq N}\subset \mathcal{M}$ such that

$$ \mathcal{M}=\bigl\{ \bigl\{ (u_{n}, v_{n})\bigr\} \cup \bigl\{ (-u_{n}, v_{n})\bigr\} \cup \bigl\{ (u_{n}, -v_{n})\bigr\} \cup \bigl\{ (-u_{n}, -v_{n})\bigr\} \bigr\} _{1\leq n\leq N}. $$

For any $1\leq n\leq N$, there exist open neighborhoods $\Omega _{n}^{1}$, $\Omega _{n}^{2}$, $\Omega _{n}^{3}$, $\Omega _{n}^{4}$ of $\{(u_{n}, v_{n})\}$, $\{(-u_{n}, v_{n})\}$, $\{(u_{n}, -v_{n})\}$, $\{(-u_{n}, -v_{n})\}$, respectively, such that

$$\begin{aligned}& \Omega _{n}^{1}\cap \Omega _{n}^{2} \cap \Omega _{n}^{3}\cap \Omega _{n}^{4}= \emptyset ,\\& \mathcal{M}\subset \bigcup_{n=1}^{3} \bigl( \Omega _{n}^{1}\cup \Omega _{n}^{2} \cup \Omega _{n}^{3}\cup \Omega _{n}^{4} \bigr)=: \Omega . \end{aligned}$$

Define

$$ \mathcal{M}_{\rho}:=\bigl\{ (u, v)\in B_{\widetilde{m}}: \operatorname{dist}_{H_{r}} \bigl((u, v), \mathcal{M} \bigr)< \rho \bigr\} , $$

we can choose $\rho >0$ small enough such that $\mathcal{M}_{2\rho}\subset \Omega$. Since $\mathcal{M}$ is finite, then there is $\varepsilon _{0}\in (0, \frac{c_{k}-d}{2})$ such that for any $(u, v)\in B_{\widetilde{m}}\backslash (\mathcal{P}_{\delta }\cup \mathcal{M}_{\rho})$, $|\Psi _{\lambda}(u, v)-d|\leq 2 \varepsilon _{0}$, we have

$$ \bigl\Vert V(u, v) \bigr\Vert _{H_{r}}^{2} \geq \varepsilon _{0}. $$

(2.26)

In fact, if for any $\varepsilon =\frac{1}{n}>0$ there exists $(u_{n}, v_{n})\in B_{\widetilde{m}}\backslash (\mathcal{P}_{\delta } \cup \mathcal{M}_{\rho})$ satisfying $|\Psi _{\lambda}(u_{n}, v_{n})-d|\leq 2 \varepsilon$, then there holds $\|V(u_{n}, v_{n})\|_{H_{r}}^{2}\leq \varepsilon$. Then, by Lemma 2.8, there exists $(u_{0}, v_{0})\in B_{\widetilde{m}}\backslash (\mathcal{P}_{\delta } \cup \mathcal{M}_{\rho})$ such that $(u_{n}, v_{n})\rightarrow (u_{0}, v_{0})$ strongly in $H_{r}$, up to a subsequence, $\Psi _{\lambda}(u_{0}, v_{0})=d$ and $V(u_{0}, v_{0})=0$. Therefore, $(u_{0}, v_{0})\in \mathcal{M}_{\rho}$. It yields a contradiction.

Moreover, for $(u, v)\in \mathcal{M}$, $V(u, v)=0$, then for $\rho >0$ small enough, there exists $T_{0}>0$ such that for any $(u, v)\in \overline{\mathcal{M}}_{2\rho}$,

$$ \bigl\Vert V(u, v) \bigr\Vert _{H_{r}}\leq T_{0}. $$

(2.27)

Let

$$ T:=\frac{1}{2}\min \biggl\{ 1, \frac{\rho T_{2}^{2}}{4T_{0}} \biggr\} . $$

(2.28)

By (2.17), for $\varepsilon _{0}>0$, there exists $A\in \Gamma _{\lambda}^{(k_{1},k_{2}+1)}$ such that

$$ \sup_{A\backslash {\mathcal{P}_{\delta}}} \Psi _{\lambda}(u, v)< d_{ \lambda ,\delta}^{k_{1},k_{2}+1} +\frac{T\varepsilon _{0}}{2}=d+ \frac{T\varepsilon _{0}}{2}. $$

(2.29)

Let $B:=A\backslash {\mathcal{M}_{2\rho}}$, then $B\subset \mathcal{F}$.

We claim that $\gamma (B)\geq (k_{1}, k_{2})$. In view of a contradiction, suppose that $\gamma (B)<(k_{1}, k_{2})$. From Definition 2.1, we know that there exists $f\in F_{(k_{1},k_{2})}(B)$ such that $f(u, v)= (f_{1}(u, v), f_{2}(u, v) )\neq (0, 0)$ for any $(u, v)\in B$. Take $\widetilde{f}=(\widetilde{f}_{1}, \widetilde{f}_{2})\in C (H_{r}, \mathbb{R}^{k_{1}-1}\times \mathbb{R}^{k_{2}-1} )$ such that $\widetilde{f}|_{B}=f$ by Tietze’s extension theorem. Define

$$\begin{aligned}& F_{1}(u, v):=\widetilde{f}_{1}(u, v)+ \widetilde{f}_{1}\bigl(\sigma _{2}(u, v)\bigr) - \widetilde{f}_{1}\bigl(\sigma _{1}(u, v)\bigr)- \widetilde{f}_{1}(-u, -v),\\& F_{2}(u, v):=\widetilde{f}_{2}(u, v)+ \widetilde{f}_{2}\bigl(\sigma _{1}(u, v)\bigr) - \widetilde{f}_{2}\bigl(\sigma _{2}(u, v)\bigr)- \widetilde{f}_{2}(-u, -v), \end{aligned}$$

then $F:=(F_{1}, F_{2})\in C (H_{r}, \mathbb{R}^{k_{1}-1}\times \mathbb{R}^{k_{2}-1} )$, $F|_{B}=4\widetilde{f}$, $F_{i}(\sigma _{i}(u, v))=-4\widetilde{f}_{i}(u, v)=-F_{i}(u, v)$ and $F_{i}(\sigma _{j}(u, v))=4\widetilde{f}_{i}(u, v)=F_{i}(u, v)$, $i\neq j$, $i, j=1, 2$.

Define the continuous function

$$ g(u, v):= \textstyle\begin{cases} 1,& (u, v)\in \bigcup_{n=1}^{3} ( \overline{\Omega _{n}^{1}}\cup \overline{\Omega _{n}^{2}} ), \\ -1,& (u, v)\in \bigcup_{n=1}^{3} ( \overline{\Omega _{n}^{3}}\cup \overline{\Omega _{n}^{4}} ) \end{cases} $$

and $g(\sigma _{1}(u, v))=g(u, v)$, $g(\sigma _{2}(u, v))=-g(u, v)$. Take $\widetilde{g}\in C(H_{r}, \mathbb{R})$ such that $\widetilde{g}|_{\Omega}=g$ by Tietze’s extension theorem. Define

$$ G(u, v):=\widetilde{g}(u, v)+\widetilde{g}\bigl(\sigma _{1}(u, v) \bigr) - \widetilde{g}\bigl(\sigma _{2}(u, v)\bigr)-\widetilde{g}(-u, -v), $$

then $G\in C(H_{r}, \mathbb{R})$, $G|_{\Omega}=4\widetilde{g}$, $G(\sigma _{1}(u, v))=G(u, v)$, and $G(\sigma _{2}(u, v))=-G(u, v)$. Therefore, we can define

$$\begin{aligned}& H_{1}(u, v):=F_{1}(u, v)\in \mathbb{R}^{k_{1}-1},\\& H_{2}(u, v):= \bigl(F_{2}(u, v), G(u, v) \bigr)\in \mathbb{R}^{k_{2}}, \end{aligned}$$

then $H:=(H_{1}, H_{2})\in C (A, \mathbb{R}^{k_{1}-1}\times \mathbb{R}^{k_{2}} )$ and $H\in F_{(k_{1},k_{2}+1)}(A)$. Since $A\in \Gamma _{\lambda}^{(k_{1},k_{2}+1)}$, $\gamma (A)\geq (k_{1}, k_{2}+1)$, so there exists $(u, v)\in A$ such that $H(u, v)=(0, 0)$. If $(u, v)\in B=A\backslash \mathcal{M}_{2\rho}$, then

$$ F(u, v)=4\widetilde{f}(u, v)=4f(u, v)\neq (0, 0), $$

a contradiction. Thus $(u, v)\in \mathcal{M}_{2\rho}$, then

$$ G(u, v)=4\widetilde{g}(u, v)=4g(u, v)\neq (0, 0), $$

a contradiction. Therefore, $\gamma (B)\geq (k_{1}, k_{2})$.

Since $B\subset A\subset B_{\widetilde{m}}$, $\sup_{B} \Psi _{\lambda}(u, v)\leq \sup_{A} \Psi _{ \lambda}(u, v)< c_{k}$, then we have $B\subset B_{\widetilde{m},\lambda}$ and $B\in \Gamma _{\lambda}^{(k_{1},k_{2})}$. Define $B_{0}:=\eta (\frac{\rho}{2T_{0}}, B)$, then $B_{0}\subset B_{\widetilde{m},\lambda}$, $B_{0}\in \Gamma ^{(k_{1},k_{2})}$, $B_{0}\backslash P_{\delta}\neq \emptyset$, and $\sup_{B_{0}} \Psi _{\lambda}(u, v)\leq \sup_{B} \Psi _{\lambda}(u, v)< c_{k}$ by Lemma 2.2(2) and Lemma 2.3, so $B_{0}\in \Gamma _{\lambda}^{(k_{1},k_{2})}$. Thus $\sup_{B_{0}\backslash \mathcal{P}_{\delta}} \Psi _{\lambda}(u, v)\geq d_{\lambda ,\delta}^{k_{1},k_{2}}$ by (2.17).

We claim that $\eta (t, (u, v))\notin \mathcal{M}_{\rho}$ for any $t\in (0, \frac{\rho}{2T_{0}})$, $(u, v)\in B$. In view of a contradiction, if there exists $t_{0}\in (0, \frac{\rho}{2T_{0}})$ such that $\eta (t_{0}, (u, v))\in \mathcal{M}_{\rho}$, for $(u, v)\in B=A\backslash \mathcal{M}_{2\rho}$, by the continuity of η, there exists $0\leq t_{1}< t_{2}\leq t_{0}$ satisfying $\eta (t_{1}, (u, v))\in \partial \mathcal{M}_{2\rho}$, $\eta (t_{2}, (u, v))\in \partial \mathcal{M}_{\rho}$, and $\eta (t, (u, v))\in \mathcal{M}_{2\rho}\backslash \mathcal{M}_{ \rho}$ for any $t\in (t_{1}, t_{2})$. Then by (2.27) we have

$$ \rho \leq \bigl\Vert \eta \bigl(t_{1}, (u, v)\bigr)-\eta \bigl(t_{2}, (u, v)\bigr) \bigr\Vert _{H_{r}}= \biggl\Vert \int _{t_{1}}^{t_{2}} V \bigl( \eta \bigl(t, (u, v) \bigr) \bigr) \biggr\Vert _{H_{r}}\leq 2T_{0}(t_{2}-t_{1}), $$

so $\frac{\rho}{2T_{0}}\leq t_{2}-t_{1}\leq t_{0}-0<\frac{\rho}{2T_{0}}$, this yields a contradiction.

For $\varepsilon _{0}>0$, there exists $(u, v)\in B$ such that $\eta (\frac{\rho}{2T_{0}}, (u, v))\in B_{0}\backslash \mathcal{P}_{ \delta}$ satisfies

$$ d_{\lambda ,\delta}^{k_{1},k_{2}}\leq \sup_{B_{0}\backslash { \mathcal{P}_{\delta}}}\Psi _{\lambda}(u, v)< \Psi _{\lambda} \biggl( \eta \biggl( \frac{\rho}{2T_{0}}, (u, v)\biggr) \biggr)+\frac{T\varepsilon _{0}}{2}. $$

Moreover, $\eta (t, (u, v))\in B_{\widetilde{m},\lambda}$ for any $t\geq 0$, then by Lemma 2.9(4), $\eta (t, (u, v))\notin \mathcal{P}_{\delta}$ for any $t\in [0, \frac{\rho}{2T_{0}}]$. Therefore,

$$ \eta \bigl(t, (u, v)\bigr)\in B_{\widetilde{m}}\backslash ( \mathcal{P}_{ \delta}\cup \mathcal{M}_{\rho } ). $$

(2.30)

In particular, $(u, v)\notin P_{\delta}$. Moreover, by (2.29) and Lemma 2.9$(3)$, we get

$$ \begin{aligned}d_{\lambda ,\delta}^{k_{1},k_{2}}&\leq \sup _{B_{0} \backslash {\mathcal{P}_{\delta}}} \Psi _{\lambda}(u, v)< \Psi _{ \lambda} \biggl(\eta \biggl(\frac{\rho}{2T_{0}}, (u, v)\biggr) \biggr)+ \frac{T\varepsilon _{0}}{2} \\ &\leq \Psi _{\lambda} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)+ \frac{T\varepsilon _{0}}{2} \\ &\leq \Psi _{\lambda} (u, v )+\frac{T\varepsilon _{0}}{2} < d_{ \lambda ,\delta}^{k_{1},k_{2}+1}+ \frac{T\varepsilon _{0}}{2}+ \frac{T\varepsilon _{0}}{2}, \end{aligned} $$

(2.31)

that is,

$$ \bigl\vert \Psi _{\lambda}(u, v)-d \bigr\vert \leq \frac{T\varepsilon _{0}}{2}< 2 \varepsilon _{0}. $$

So we see from (2.26) and Lemma 2.8 that

$$ \begin{aligned}\frac{d}{dt}\Psi _{\lambda} \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)&=- \bigl\langle \nabla \Psi _{\lambda}\bigl(\eta \bigl(t, (u, v)\bigr) \bigr), V \bigl(\eta \bigl(t, (u, v)\bigr) \bigr)\bigr\rangle _{H_{r}} \\ &\leq -\frac{T_{2}^{2}}{2} \bigl\Vert V \bigl(\eta \bigl(t, (u, v)\bigr) \bigr) \bigr\Vert ^{2}_{H_{r}} \leq -\frac{T_{2}^{2}}{2} \varepsilon _{0}. \end{aligned} $$

(2.32)

Finally, we deduce from (2.28), (2.31), and (2.32) that

$$\begin{aligned} d_{\lambda ,\delta}^{k_{1},k_{2}}& < \Psi _{\lambda} \biggl(\eta \biggl( \frac{\rho}{2T_{0}}, (u, v)\biggr) \biggr)+\frac{T\varepsilon _{0}}{2} \\ & \leq \Psi _{\lambda} (u, v )+\frac{T\varepsilon _{0}}{2}- \int _{0}^{\frac{\rho}{2T_{0}}}\frac{T_{2}^{2}}{2}\varepsilon _{0} \,dt \\ & < d_{\lambda ,\delta}^{k_{1},k_{2}}+\frac{T\varepsilon _{0}}{2}+ \frac{T\varepsilon _{0}}{2}-\frac{T_{2}^{2}}{2}\varepsilon _{0} \frac{\rho}{2T_{0}} \\ & = d_{\lambda ,\delta}^{k_{1},k_{2}}+\frac{\varepsilon _{0}}{2}\biggl(2T- \frac{T_{2}^{2}\rho}{2T_{0}}\biggr)\leq d_{\lambda ,\delta}^{k_{1},k_{2}}, \end{aligned}$$

this yields a contradiction. This completes the proof. □

3 Proof of Theorem 1.2

Using Theorem 1.1, for $k=1$, there exists $\lambda _{1}>0$ such that system (1.1) has a radially symmetric sign-changing solution $(u_{1}, v_{1})$ for any $\lambda \in (0, \lambda _{1})$ and for $k_{1}=k_{2}=2$,

$$ \Phi _{\lambda}(u_{1}, v_{1})=d_{\lambda ,\delta}^{2,2}< c_{1}. $$

Let

$$ U_{\lambda}:=\bigl\{ (u, v)\in H_{r}: (u, v) \text{ is a sign-changing solution of } (1.1)\bigr\} , $$

then $U_{\lambda}\neq \emptyset$ by Theorem 1.1, we can define

$$ d_{\lambda}:=\inf_{(u,v)\in U_{\lambda}} \Phi _{\lambda}(u, v) $$

and $d_{\lambda}< c_{1}$. Let $(u_{n}, v_{n})\in U_{\lambda}$ be a minimizing sequence of $d_{\lambda}$ with $\Phi _{\lambda}(u_{n}, v_{n})\rightarrow d_{\lambda}$, $\Phi _{\lambda}(u_{n}, v_{n})< c_{1}$ and $\Phi _{\lambda}'(u_{n}, v_{n})=0$. Then

$$ \begin{aligned} & \biggl(\frac{1}{2}- \frac{1}{p+1}\biggr) \bigl( \Vert u_{n} \Vert _{\alpha}^{2}+ \Vert v_{n} \Vert ^{2}_{\beta}\bigr) \\ &\quad \leq \biggl(\frac{1}{2}-\frac{1}{p+1}\biggr) \bigl( \Vert u_{n} \Vert _{\alpha}^{2}+ \Vert v_{n} \Vert ^{2}_{ \beta}\bigr) +\biggl( \frac{1}{p+1}-\frac{1}{q+1}\biggr) \vert v_{n} \vert _{q+1}^{q+1} \\ &\quad \quad{}+\frac{2}{p+1}\biggl(\frac{1}{p+1}-\frac{1}{q+1} \biggr) \lambda \int _{ \mathbb{R}^{3}} \vert u_{n} \vert ^{\frac{p+1}{2}} \vert v_{n} \vert ^{\frac{q+1}{2}} \,dx \\ &\quad =\Phi _{\lambda}(u_{n}, v_{n})- \frac{1}{p+1}\Phi _{\lambda}'(u_{n}, v_{n}) (u_{n}, v_{n})< c_{1}. \end{aligned} $$

(3.1)

Observe that $\{(u_{n}, v_{n})\}_{n\geq 1}$ is bounded in $H_{r}$, we may assume that, up to a subsequence,

$$\begin{aligned}& (u_{n}, v_{n})\rightharpoonup (u_{0}, v_{0})\quad \text{weakly in } H_{r},\\& u_{n}\rightarrow u_{0},\quad \text{strongly in } L^{p+1}\bigl( \mathbb{R}^{3}\bigr),\\& v_{n}\rightarrow v_{0},\quad \text{strongly in } L^{q+1}\bigl( \mathbb{R}^{3}\bigr). \end{aligned}$$

Since $\Phi _{\lambda}'(u_{n}, v_{n})=0$, it is standard to prove that

$$ (u_{n}, v_{n})\rightarrow (u_{0}, v_{0})\quad \text{strongly in } H_{r}, $$

and $\Phi _{\lambda}'(u_{0}, v_{0})=0$, $\Phi _{\lambda}(u_{0}, v_{0})=d_{\lambda}$.

Moreover, $\Phi _{\lambda}'(u_{n}, v_{n})(u_{n}^{\pm}, 0)=0$ and $\Phi _{\lambda}'(u_{n}, v_{n})(0, v_{n}^{\pm})=0$, we deduce from (2.7) and (3.1) that

$$ \begin{aligned}S_{p} \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{2}&\leq \bigl\Vert u_{n}^{\pm} \bigr\Vert _{\alpha}^{2}= \bigl\vert u_{n}^{ \pm} \bigr\vert _{p+1}^{p+1}+ \frac{2}{q+1} \lambda \int _{ \mathbb{R}^{3}} \bigl\vert u_{n}^{ \pm} \bigr\vert ^{\frac{p+1}{2}} \vert v_{n} \vert ^{\frac{q+1}{2}} \,dx \\ &\leq \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{p+1}+\frac{2}{q+1} \lambda \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{ \frac{p+1}{2}} \vert v_{n} \vert _{q+1}^{\frac{q+1}{2}} \\ &< \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{p+1}+\frac{2}{q+1} \biggl[ \frac{c_{1}}{(\frac{1}{2}-\frac{1}{p+1})S_{q}} \biggr]^{\frac{q+1}{4}} \lambda \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{\frac{p+1}{2}}. \end{aligned} $$

We can choose $0<\lambda _{0}<\lambda _{1}$ small enough such that for any $\lambda \in (0, \lambda _{0})$ we have

$$ S_{p} \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{2}< 2 \bigl\vert u_{n}^{\pm} \bigr\vert _{p+1}^{p+1}, $$

which implies $|u_{n}^{\pm}|_{p+1}\geq \xi _{1}>0$ for any $n\geq 1$. Similarly, $|v_{n}^{\pm}|_{q+1}\geq \xi _{2}>0$ for any $n\geq 1$. Therefore, $|u_{0}^{\pm}|_{p+1}\geq \xi _{1}>0$, $|v_{0}^{\pm}|_{q+1}\geq \xi _{2}>0$, and so Eq. (1.1) has a least energy sign-changing solution $(u_{0}, v_{0})$. This completes the proof.  □

4 The proof of Theorem 1.3

In this section, we obtain seminodal solutions $(u, v)$ such that u is positive, v is sign-changing and use the same notations as in Sect. 2 for convenience. Define the $C^{1}$ functional

$$ \begin{aligned}\Phi _{\lambda}(u, v) &:= \frac{1}{2} \bigl( \Vert u \Vert _{\alpha}^{2}+ \Vert v \Vert _{\beta}^{2}\bigr)-\frac{1}{p+1} \bigl\vert u^{+} \bigr\vert _{p+1}^{p+1}- \frac{1}{q+1} \vert v \vert _{q+1}^{q+1} \\ &\quad{}-\frac{4\lambda}{(p+1)(q+1)} \int _{ \mathbb{R}^{3}} \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q+1}{2}} \,dx , \end{aligned} $$

where $(u, v)\in \widetilde{H}_{r}:=\{(u, v)\in H_{r}: u^{+}\neq 0, v \neq 0\}$,

$$\begin{aligned}& \mathcal{A}:=\bigl\{ (u, v)\in H_{r}: \bigl\vert u^{+} \bigr\vert _{p+1}=1, \vert v \vert _{q+1}=1 \bigr\} ,\\& \mathcal{A}^{*}:=\biggl\{ (u, v)\in H_{r}: \frac{1}{2}< \bigl\vert u^{+} \bigr\vert _{p+1}^{p+1}< 2, \frac{1}{2}< \vert v \vert _{q+1}^{q+1}< 2 \biggr\} ,\\& \mathcal{B}_{m}^{*}:=\bigl\{ (u, v)\in \mathcal{A}^{*} : \Vert u \Vert _{\alpha}^{2}< m, \Vert v \Vert _{\beta}^{2}< m \bigr\} ,\qquad \mathcal{B}_{m}:= \mathcal{B}_{m}^{*} \cap \mathcal{A}. \end{aligned}$$

As in Sect. 2, for any $(u, v)\in \mathcal{A}$, we define

$$ \sup_{t,s\geq 0} \Phi _{\lambda}(tu, sv)= \Phi _{\lambda}(t_{u,v, \lambda}u, s_{u,v,\lambda}v)=:\Psi _{\lambda}(u, v). $$

(4.1)

It is easy to prove that Lemma 2.4 also holds in this section by trivial modifications. Then define

$$ B_{\widetilde{m}, \lambda}:=\Bigl\{ (u, v)\in B_{\widetilde{m}}: \sup _{t,s \geq 0} \Phi _{\lambda}(tu, sv) < c_{k} \Bigr\} . $$

For any $(u, v)\in \mathcal{B}_{\widetilde{m}}^{*}$, $\lambda \in (0, \lambda _{k})$, we consider the following linear problem:

$$ \textstyle\begin{cases} -\Delta \varphi +\alpha \varphi -\frac{2}{q+1} t_{u,v,\lambda}^{ \frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{ \frac{p-3}{2}}\varphi \vert v \vert ^{\frac{q+1}{2}}=t_{u,v,\lambda}^{p-1} (u^{+})^{p}, \\ -\Delta \psi +\beta \psi -\frac{2}{p+1} t_{u,v,\lambda}^{ \frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \vert u \vert ^{ \frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}\psi =s_{u,v,\lambda}^{q-1} \vert v \vert ^{q-1}v, \\ \varphi (x)\rightarrow 0,\qquad \psi (x)\rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty , \end{cases} $$

(4.2)

then (4.2) has a unique solution $(\varphi , \psi )\in H_{r}\backslash \{(0, 0)\}$. Define

$$ \mu :=\frac{1}{\int _{\mathbb{R}^{3}}(u^{+})^{p} \varphi \,dx }>0, \qquad \nu :=\frac{1}{\int _{\mathbb{R}^{3}} \vert v \vert ^{q-1}v \psi \,dx }>0. $$

Then $(\widetilde{\varphi}, \widetilde{\psi}):=(\mu \varphi , \nu \psi )$ is the unique solution of the following problem:

$$ \textstyle\begin{cases} -\Delta \widetilde{\varphi}+\alpha \widetilde{\varphi}-\frac{2}{q+1} t_{u,v,\lambda}^{\frac{p-3}{2}} s_{u,v,\lambda}^{\frac{q+1}{2}} \lambda \vert u \vert ^{\frac{p-3}{2}}\widetilde{\varphi} \vert v \vert ^{ \frac{q+1}{2}}=\mu t_{u,v,\lambda}^{p-1} (u^{+})^{p}, \\ -\Delta \widetilde{\psi}+\beta \widetilde{ \psi}-\frac{2}{p+1} t_{u,v, \lambda}^{\frac{p+1}{2}} s_{u,v,\lambda}^{\frac{q-3}{2}} \lambda \vert u \vert ^{\frac{p+1}{2}} \vert v \vert ^{\frac{q-3}{2}}\widetilde{\psi } =\nu s_{u,v, \lambda}^{q-1} \vert v \vert ^{q-1}v, \\ \int _{\mathbb{R}^{3}}(u^{+})^{p} \widetilde{\varphi} \,dx =\int _{ \mathbb{R}^{3}} \vert v \vert ^{q-1}v\widetilde{\psi} \,dx =1, \\ \widetilde{\varphi}(x)\rightarrow 0,\qquad \widetilde{\psi}(x) \rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty . \end{cases} $$

(4.3)

We can now also define the operator

$$\begin{aligned}& K: B_{\widetilde{m}}^{*}\rightarrow H_{r};\qquad (u, v)\mapsto ( \widetilde{\varphi}, \widetilde{\psi}), \\& K\bigl(\sigma _{2}(u, v)\bigr)=\sigma _{2}\bigl(K(u, v)\bigr). \end{aligned}$$

(4.4)

Then, by similar proofs as in Lemma 2.5 and Lemma 2.6, we have that $K\in C^{1}(B_{\widetilde{m}}^{*}, H_{r})$ and K satisfies the Palais–Smale type condition. Define the map

$$ V: B_{\widetilde{m}}^{*}\rightarrow H_{r};\qquad (u, v)\mapsto (u, v)-K(u, v). $$

Consider the class of sets

$$ \mathcal{F}=\bigl\{ A\in \mathcal{A}: A \text{ is a closed set and } \sigma _{2}(u, v)\in A, \forall (u, v)\in A\bigr\} $$

(4.5)

for each $A\in \mathcal{F}$ and $k_{2}\geq 2$, the class of functions

$$ F_{(1, k_{2})}(A)=\bigl\{ f: A\rightarrow \mathbb{R}^{k_{2}-1}: f \text{ continuous and } f\bigl(\sigma _{2}(u, v)\bigr)=-f(u, v) \bigr\} . $$

(4.6)

To obtain seminodal solutions, we should also define a cone of positive functions, that is,

$$\begin{aligned}& \mathcal{P}_{2}:=\bigl\{ (u, v)\in H_{r}: v\geq 0\bigr\} ,\qquad \mathcal{P}= \mathcal{P}_{2}\cup -\mathcal{P}_{2}, \\& \operatorname{dist}_{q+1} \bigl((u, v), \mathcal{P} \bigr):=\min \bigl\{ \operatorname{dist}_{q+1} (v, \mathcal{P}_{2}) , \operatorname{dist}_{q+1} (v, -\mathcal{P}_{2}) \bigr\} , \end{aligned}$$

(4.7)

thus, v is sign-changing if $\operatorname{dist}_{q+1} ((u, v), \mathcal{P})>0$.

Under the new definitions (4.4)–(4.6), we define vector genus, slightly different from Definition 2.1.

Definition 4.1

Let $A\in \mathcal{F}$ and take any $k_{2}\in \mathbb{N}$ with $k_{2}\geq 2$. We say that $\gamma (A)\geq (1, k_{2})$ if for every $f\in F_{(1, k_{2})}(A)$ there exists $(u, v)\in A$ such that $f(u, v)=0$. We denote

$$ \Gamma ^{(1, k_{2})}:=\bigl\{ A\in \mathcal{F}: \gamma (A)\geq (1, k_{2}) \bigr\} . $$

Lemma 4.1

• (1) Take $A:=A_{1}\times A_{2}\subset \mathcal{A}$ and let $\eta : S^{k_{2}-1}\rightarrow A_{2}$ be a homeomorphism such that $\eta (-x)=-\eta (x)$ for every $x\in S^{k_{2}-1}$. Then $A\in \Gamma ^{(1, k_{2})}$;

• (2) We have $\overline{\eta (A)}\in \Gamma ^{(1, k_{2})}$ whenever $A \in \Gamma ^{(1, k_{2})}$ and a continuous map $\eta : A\rightarrow \mathcal{A}$ is such that $\eta \circ \sigma _{2}=\sigma _{2}\circ \eta$.

Proof

$(1)$ For every $f\in F_{(1, k_{2})}(A)$ and $u\in A_{1}$, we define a map

$$ h: S^{k_{2}-1}\rightarrow \mathbb{R}^{k_{2}-1};\qquad h(x):=f \bigl(u, \eta (x)\bigr), $$

then by (4.6) it is easy to see that h is continuous and

$$ h(-x)=f\bigl(u, \eta (-x)\bigr)=f\bigl(u, -\eta (x)\bigr)=-f\bigl(u, \eta (x) \bigr)=-h(x). $$

Then Borsuk–Ulam theorem yields $x_{0}\in S^{k_{2}-1}$ such that $h(x_{0})=f(u, \eta (x_{0}))=0$. By Definition 4.1, we have $A\in \Gamma ^{(1, k_{2})}$.

$(2)$ Fix any $f\in F_{(1, k_{2})}(\overline{\eta (A)})$, then by (4.6) we have $f\circ \eta \in F_{(1, k_{2})}(A)$. Since $A\in \Gamma ^{(1, k_{2})}$, there exists $(u_{0}, v_{0})\in A$ such that $f\circ \eta (u_{0}, v_{0})=0$. Then by $\eta (u_{0}, v_{0})\in \overline{\eta (A)}$ we have $\gamma (\overline{\eta (A)})\geq (1, k_{2})$, that is, $\overline{\eta (A)}\in \Gamma ^{(1, k_{2})}$. This completes the proof. □

Lemma 4.2

Assume $k_{2}\geq 2$. Then, for any $0<\delta <2^{-\frac{1}{q+1}}$ and $A\in \Gamma ^{(1, k_{2})}$, we have $A\backslash \mathcal{P}_{\delta }\neq \emptyset$.

Proof

For any $A\in \Gamma ^{(1, k_{2})}$, define f by

$$ f(u, v)= \biggl( \int _{\mathbb{R}^{3}} \vert v \vert ^{q}v \,dx , 0, \ldots , 0\biggr), $$

then $f\in F_{(1, k_{2})}(A)$, so by Definition 4.1, there exists $(u_{0}, v_{0})\in A$ such that $f(u_{0}, v_{0})=0$. We deduce from $A\in \mathcal{A}$ that

$$ \int _{\mathbb{R}^{3}}\bigl(v_{0}^{+} \bigr)^{q+1} \,dx = \int _{\mathbb{R}^{3}}\bigl(v_{0}^{-} \bigr)^{q+1} \,dx =\frac{1}{2}. $$

Therefore, $\operatorname{dist}_{q+1} ((u_{0}, v_{0}), \mathcal{P})=2^{-\frac{1}{q+1}}$, and so $(u_{0}, v_{0})\in A\backslash \mathcal{P}_{\delta}$ for any $0<\delta <2^{-\frac{1}{q+1}}$. This completes the proof. □

Fixed any $k\in \mathbb{N}$, we define

$$ A_{1}:=\biggl\{ cu_{0}: c=\frac{1}{ \vert u_{0} \vert _{p+1}}, u_{0}>0\biggr\} ,\qquad A_{2}:= \bigl\{ v\in X_{k+1}: \vert v \vert _{q+1}=1\bigr\} . $$

By Lemma 4.1(1), $A:=A_{1}\times A_{2}\in \Gamma ^{(1, k+1)}$, $A\subset B_{\widetilde{m}}$, and $\sup_{A} \Psi _{\lambda}(u, v)< c_{k}$. Then we can define

$$ \Gamma _{\lambda}^{(1, k_{2})}:=\Bigl\{ A\in \Gamma ^{(1, k_{2})}: A \subset B_{\widetilde{m}}, \sup_{A} \Psi _{\lambda}(u, v)< c_{k} \Bigr\} . $$

For any $k_{2}\in [2, k+1]$ and $0<\delta <2^{-\frac{1}{q+1}}$, we define a sequence of minimax energy level:

$$ d_{\lambda ,\delta}^{1,k_{2}}:=\inf_{A\in \Gamma _{\lambda}^{(1,k_{2})}} \sup _{A\backslash \mathcal{P}_{\delta}}\sup_{t,s\geq 0} \Phi _{ \lambda}(tu, sv). $$

It is easy to see that

$$ d_{\lambda ,\delta}^{1,k_{2}}< c_{k}\quad \text{for any } 0< \delta < 2^{-\frac{1}{q+1}} \text{ and } 2\leq k_{2}\leq k+1. $$

Lemma 2.7 and Lemma 2.8 also hold in Sect. 4.

Lemma 4.3

There exists a unique global solution $\eta : \mathbb{R}^{+}\times B_{\widetilde{m},\lambda}\rightarrow H_{r}$ for the initial value problem

$$ \textstyle\begin{cases} \frac{d}{dt}\eta (t, (u, v))=-V (\eta (t, (u, v)) ), \\ \eta (0, (u, v))=(u, v)\in B_{\widetilde{m},\lambda}. \end{cases} $$

(4.8)

Moreover, (1), (3), (4) of Lemma 2.9hold and

• (2) For any $t>0$, $(u, v)\in B_{\widetilde{m},\lambda}$, $\eta (t, \sigma _{2}(u, v))=\sigma _{2} ( \eta (t, (u, v)) )$.

Proof

From the above discussion, we see that $V\in C^{1}(B_{\widetilde{m}}^{*}, H_{r})$. As $B_{\widetilde{m},\lambda}\subset B_{\widetilde{m}}\subset B_{ \widetilde{m}}^{*}$, we get that $V\in C^{1}(B_{\widetilde{m},\lambda}, H_{r})$, then there exists a solution $\eta : [0, T_{\max})\times B_{\widetilde{m},\lambda}\rightarrow H_{r}$, where $T_{\max}$ is the maximal time such that (4.8) has s solution $\eta \in B_{\widetilde{m}}^{*}$.

For any $(u, v)\in B_{\widetilde{m},\lambda}$ and $t\in (0, T_{\max})$, there holds

$$ \begin{aligned}& \frac{d}{dt} \int _{\mathbb{R}^{3}}\bigl(\eta _{1}^{+} \bigl(t, (u, v)\bigr)\bigr)^{p+1} \,dx \\ &\quad = -(p+1) \int _{\mathbb{R}^{3}}\bigl(\eta _{1}^{+} \bigl(t, (u, v)\bigr)\bigr)^{p}V \bigl( \eta _{1}^{+} \bigl(t, (u, v)\bigr) \bigr) \,dx \\ &\quad = -(p+1) \int _{\mathbb{R}^{3}}\bigl(\eta _{1}^{+} \bigl(t, (u, v)\bigr)\bigr)^{p}\bigl[\eta _{1}^{+} \bigl(t, (u, v)\bigr)-K_{1} \bigl(\eta ^{+}\bigl(t, (u, v)\bigr) \bigr)\bigr] \,dx \\ &\quad = (p+1)-(p+1) \int _{\mathbb{R}^{3}}\bigl(\eta _{1}^{+} \bigl(t, (u, v)\bigr)\bigr)^{p+1} \,dx , \end{aligned} $$

so we have

$$ \frac{d}{dt}\biggl[e^{(p+1)t} \biggl( \int _{\mathbb{R}^{3}}\bigl(\eta _{1}^{+} \bigl(t, (u, v)\bigr)\bigr)^{p+1} \,dx -1 \biggr)\biggr] =0. $$

Since $\int _{\mathbb{R}^{3}}(\eta _{1}^{+}(0, (u, v)))^{p+1} \,dx =\int _{ \mathbb{R}^{3}}(u^{+})^{p+1} \,dx =1$, then for any $t\in [0, T_{\max})$,

$$ \int _{\mathbb{R}^{3}}\bigl(\eta _{1}^{+} \bigl(t, (u, v)\bigr)\bigr)^{p+1} \,dx \equiv 1. $$

The rest of the proof is the same as Lemma 2.9. This completes the proof. □

Proof of Theorem 1.2

Observe that from Lemma 2.10, for any $k_{2}\in [2, k+1]$, $0<\delta <\delta _{0}$ small, there exists $(u_{0}, v_{0})\in B_{\widetilde{m}}$ such that

$$ \Psi _{\lambda}(u_{0}, v_{0})=d_{\lambda ,\delta}^{1,k_{2}}, \qquad V(u_{0}, v_{0})=0\quad \text{and}\quad \operatorname{dist}_{q+1} \bigl((u_{0}, v_{0}), \mathcal{P} \bigr)\geq \delta . $$

We conclude that $v_{0}$ is sign-changing and $(u_{0}, v_{0})=K(u_{0}, v_{0})=(\widetilde{\varphi}_{0}, \widetilde{\psi}_{0})$. It follows from (4.3) that $(u_{0}, v_{0})$ satisfies

$$ \textstyle\begin{cases} -\Delta u_{0}+\alpha u_{0}=\mu t_{u,v,\lambda}^{p-1} (u_{0}^{+})^{p} +\frac{2}{q+1} t_{u_{0},v_{0},\lambda}^{\frac{p-3}{2}} s_{u_{0},v_{0}, \lambda}^{\frac{q+1}{2}} \lambda \vert u_{0} \vert ^{\frac{p-3}{2}}u_{0} \vert v_{0} \vert ^{ \frac{q+1}{2}}, \\ -\Delta v_{0}+\beta v_{0}=\nu s_{u_{0},v_{0},\lambda}^{q-1} \vert v_{0} \vert ^{q-1}v_{0} +\frac{2}{p+1} t_{u_{0},v_{0},\lambda}^{\frac{p+1}{2}} s_{u_{0},v_{0}, \lambda}^{\frac{q-3}{2}} \lambda \vert u_{0} \vert ^{\frac{p+1}{2}} \vert v_{0} \vert ^{ \frac{q-3}{2}}v_{0}, \\ u_{0}(x)\rightarrow 0,\qquad v_{0}(x)\rightarrow 0,\quad \text{as } \vert x \vert \rightarrow \infty , \end{cases} $$

(4.9)

and $|u_{0}^{+}|_{p+1}=|v_{0}|_{q+1}=1$, then by (4.1) we have $\mu =\nu =1$. Moreover, (4.9) yields

$$ \bigl\Vert u_{0}^{-} \bigr\Vert _{\alpha}^{2}=\frac{2}{q+1} t_{u_{0},v_{0},\lambda}^{ \frac{p-3}{2}} s_{u_{0},v_{0},\lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{0} \vert ^{\frac{p-3}{2}}\bigl(u^{-}_{0}\bigr)^{2} \vert v_{0} \vert ^{ \frac{q+1}{2}}. $$

We can take $\lambda _{k}$ small enough if necessary such that for any $\lambda \in (0, \lambda _{k})$ and $(u_{0}, v_{0})\in \mathcal{B}_{\widetilde{m}}^{*}$,

$$ \bigl\Vert u_{0}^{-} \bigr\Vert _{\alpha}^{2}-\frac{2}{q+1} t_{u_{0},v_{0},\lambda}^{ \frac{p-3}{2}} s_{u_{0},v_{0},\lambda}^{\frac{q+1}{2}} \lambda \int _{\mathbb{R}^{3}} \vert u_{0} \vert ^{\frac{p-3}{2}}\bigl(u^{-}_{0}\bigr)^{2} \vert v_{0} \vert ^{ \frac{q+1}{2}} \geq \frac{1}{2} \bigl\Vert u_{0}^{-} \bigr\Vert _{\alpha}^{2}, $$

then $\|u_{0}^{-}\|_{\alpha}^{2}=0$, so $u_{0}\geq 0$. By the strong maximum principle, $u_{0}>0$. Hence we have that $(t_{u_{0},v_{0},\lambda}u_{0}, s_{u_{0},v_{0},\lambda}v_{0})$ is a seminodal solution of (1.1) with $t_{u_{0},v_{0},\lambda}u_{0}$ positive and $s_{u_{0},v_{0},\lambda}v_{0}$ sign-changing,

$$ \Phi _{\lambda}(t_{u_{0},v_{0},\lambda}u_{0}, s_{u_{0},v_{0},\lambda}v_{0}) =\Psi _{\lambda}(u_{0}, v_{0})=d_{\lambda ,\delta}^{1,k_{2}}. $$

By similar proof as Theorem 1.1, we complete the proof. □