Positive solutions for coupled Schrödinger system with critical exponent in $\mathbb{R}^{N}$ ($N\geq5$)

Abstract

In this paper, we study the following coupled Schrödinger system:

$$ \textstyle\begin{cases} -\Delta u+u=u^{2^{*}-1}+\beta u^{\frac{2^{*}}{2}-1}v^{\frac {2^{*}}{2}}+f(u), &x\in\mathbb{R}^{N}, \\ -\Delta v+v=v^{2^{*}-1}+\beta u^{\frac{2^{*}}{2}}v^{\frac {2^{*}}{2}-1}+g(v), &x\in\mathbb{R}^{N}, \\ u,v>0, &x\in\mathbb{R}^{N}, \end{cases} $$

where $N\geq5$ and $2^{*}=\frac{2N}{N-2}$. Note that the nonlinearity and the coupling terms are both of critical growth. Using the mountain pass theorem, Ekeland’s variational principle and the concentration-compactness principle, we show that this system has at least one positive least energy solution for each $\beta\in(-1,0)\cup (0,+\infty)$.

1 Introduction

In this paper, we consider the following coupled nonlinear Schrödinger system:

$$ \textstyle\begin{cases} -\Delta u+u=u^{2^{*}-1}+\beta u^{\frac{2^{*}}{2}-1}v^{\frac {2^{*}}{2}}+f(u), &x\in\mathbb{R}^{N}, \\ -\Delta v+v=v^{2^{*}-1}+\beta u^{\frac{2^{*}}{2}}v^{\frac {2^{*}}{2}-1}+g(v), &x\in\mathbb{R}^{N}, \\ u,v > 0, &x\in\mathbb{R}^{N}, \end{cases} $$

(1.1)

where $N\geq3$, $2^{*}=\frac{2N}{N-2}$ and $\beta\in(-1,+\infty )\setminus \{0\}$. The functions $f,g$ satisfy the following conditions:

(F₁):

$f,g\in C^{2}(\mathbb{R})$, $\lim_{t\rightarrow 0^{+}}\frac{f(t)}{t}=0$, $\lim_{t\rightarrow\infty}\frac {f(t)}{t^{2^{*}-1}}=0$ for $t\geq0$,

$$\lim_{t\rightarrow0^{+}}\frac{g(t)}{t}=0,\qquad \lim_{t\rightarrow\infty} \frac{g(t)}{t^{2^{*}-1}}=0\quad \text{for }t\geq0; $$

(F₂):

There exist $\theta_{1},\theta_{2}>0$ small enough such that

$$tf'(t)\geq(1+\theta_{1})f(t)>0, \qquad tg'(t)\geq(1+\theta _{2})g(t)>0\quad \text{for } t>0; $$

(F₃):

$f(t)$, $g(t)$ are odd.

In recent years, there have been a lot of studies on the following coupled system of nonlinear Schrödinger equations:

$$ \textstyle\begin{cases} -\Delta u+\lambda_{1}u=\mu_{1}u^{2p-1}+\beta u^{p-1}v^{p}, &x\in\Omega, \\ -\Delta v+\lambda_{2}v=\mu_{2}v^{2p-1}+\beta u^{p}v^{p-1}, &x\in \Omega, \\ u,v > 0, x\in\Omega, u=v=0, &x\in\partial\Omega, \end{cases} $$

(1.2)

where $\Omega=\mathbb{R}^{N}$ ($N\geq3$) or Ω is a smooth bounded domain in $\mathbb{R}^{N}$, $1< p\leq\frac{2^{*}}{2}$, $\mu_{1},\mu_{2}>0$ and $\beta\neq0$. For the case $p=2$, system (1.2) arises in the Hartree-Fock theory for a binary mixture of Bose-Einstein condensates in two different hyperfine states, see more details in [1–3]. The sign of β determines whether the interactions of the two states are attractive if $\beta>0$ or repulsive if $\beta<0$.

For problem (1.2), we are interested in the existence of a nontrivial solution $(u,v)$, i.e. $u\not\equiv0$ and $v\not\equiv 0$. However, one easily sees that (1.2) may admit a semitrivial solution of the form $(u,0)$ or $(0,v)$, which may cause some difficulties. When $2<2p<2^{*}$, system (1.2) is a problem of subcritical growth. The existence and multiplicity of nontrivial solutions have been extensively studied, see [4–13] and the references therein.

For the critical case $2p=2^{*}$, when Ω is a smooth bounded domain, there exist papers [14–17] studying this case. In [14], Chen and Zou studied problem (1.2) with $N=4$. In [15], Chen and Zou studied problem (1.2) with $N\geq5$ and they showed that if $-\lambda_{1}(\Omega)<\lambda_{1}\leq\lambda _{2}<0$, then (1.2) has a positive least energy solution for any $\beta\neq0$, where $\lambda_{1}(\Omega)$ is the first eigenvalue of −Δ with the Dirichlet boundary condition. In [17], by a minimization method, Ye and Peng showed the existence of positive least energy solution for the special case $\lambda_{1}=\lambda_{2}$. When $2p=2^{*}$ and $\Omega=\mathbb{R}^{N}$, by the Pohozaev identity, we see that problem (1.2) has only a trivial solution if $\lambda_{1}\lambda _{2}>0$. To get nontrivial solutions, one usually adds lower order perturbation terms to the right-hand side of system (1.2), i.e. considering problem (1.1). Problem (1.1) can be seen as a counterpart of the following single equation:

$$ -\Delta u+u= \vert u \vert ^{2^{*}-2}u+f(u), \quad x\in \mathbb{R}^{N} $$

(1.3)

or

$$ -\Delta v+v= \vert v \vert ^{2^{*}-2}v+g(v), \quad x\in \mathbb{R}^{N}. $$

(1.4)

Deng in [18] proved that if $N\geq4$ and (F₁)-(F₃) hold, then (1.3) (or (1.4)) has at least one positive least energy radial solution, denoted by $u_{1}$ (or $v_{1}$) and the corresponding energy denoted by $B_{1}$ (or $B_{2}$). Hence we deduce that $(u_{1},0)$ and $(0,v_{1})$ are semitrivial solutions to problem (1.1), which may be an interference in the process of searching for nontrivial solutions. Recently, the author proves the special case $N=4$ in [19]. In this paper, we consider (1.1) with higher dimensions $N\geq5$. To state our main results, we denote $H:= H^{1}(\mathbb{R} ^{N})\times H^{1}(\mathbb{R}^{N})$ with the norm defined as $\Vert (u,v) \Vert _{H}= [\int_{\mathbb{R}^{N}}( \vert \nabla u \vert ^{2}+ \vert u \vert ^{2})+\int _{\mathbb{R} ^{N}}( \vert \nabla v \vert ^{2}+ \vert v \vert ^{2}) ]^{\frac{1}{2}}$, $\forall (u,v)\in H$. It is well known that weak solutions of (1.1) correspond to critical points of the functional $I:H\rightarrow\mathbb{R}$ defined as follows:

$$I(u,v)=\frac{1}{2} \bigl\Vert (u,v) \bigr\Vert _{H}^{2}- \frac{1}{2^{*}} \int _{\mathbb{R} ^{N}}\bigl( \vert u \vert ^{2^{*}}+ \vert v \vert ^{2^{*}}+2\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}\bigr)- \int_{\mathbb{R}^{N}}F(u)- \int_{\mathbb{R}^{N}}G(v), $$

for any $(u,v)\in H$, where $F(s)=\int_{0}^{s}f(t)\,dt$, $G(s)=\int _{0}^{s}g(t)\,dt$. We say $(u,v)\in H$ is a positive least energy solution of (1.1) if $(u,v)$ is a nontrivial solution of (1.1) with $u>0$, $v>0$ and

$$I(u,v)=\inf\bigl\{ I(\varphi,\psi)\vert (\varphi,\psi) \text{ is a nontrivial solution of (1.1)}\bigr\} . $$

Our main result is as follows.

Theorem 1.1

Suppose that (F₁)-(F₃) hold and $N\geq5$.

• (1) For any $\beta>0$, problem (1.1) has at least one positive least energy solution.

• (2) For any $\beta\in(-1,0)$, problem (1.1) has at least one radial and positive least energy solution.

Remark 1.2

• (1) For $\beta<0$, we do not know whether the solution obtained in Theorem 1.1 is a least energy solution of (1.1) in H or not.

• (2) When $N=4$, it is proved in [19] that (1.1) has a radially positive solution for any $\beta>0$ and $\beta\neq1$. Comparing this with Theorem 1.1(1), we see that the case $N\geq5$ is completely different from the case $N=4$. In the proof of Theorem 1.1(1), we should point out that $2^{*}<4$ is an essential condition, which makes the method not applicable to the case $N=4$.

• (3) The method to prove Theorem 1.1(2) can be similarly used to show that when $N=4$ and $-1<\beta<0$, (1.1) has at least one radially positive least energy solution.

By (F₁), (F₃), for any $\varepsilon>0$, there exists $C_{\varepsilon }>0$ such that

$$\begin{aligned}& f(t)t\leq\varepsilon \vert t \vert ^{2}+C_{\varepsilon} \vert t \vert ^{2^{*}}, \qquad g(t)t\leq \varepsilon \vert t \vert ^{2}+C_{\varepsilon} \vert t \vert ^{2^{*}}, \quad \forall t\in\mathbb{R}, \end{aligned}$$

(1.5)

$$\begin{aligned}& F(t)\leq\varepsilon\frac{ \vert t \vert ^{2}}{2}+C_{\varepsilon} \frac { \vert t \vert ^{2^{*}}}{2^{*}}, \qquad G(t)\leq\varepsilon\frac { \vert t \vert ^{2}}{2}+C_{\varepsilon} \frac{ \vert t \vert ^{2^{*}}}{2^{*}},\quad \forall t\in \mathbb{R}. \end{aligned}$$

(1.6)

By (F₂), (F₃), we have

$$ 0< (2+\theta_{1})F(t)\leq tf(t), \qquad 0< (2+ \theta_{2})G(t)\leq tg(t),\quad \forall t\in\mathbb{R} $$

(1.7)

(see Remark 1.3 in [18]), then $\frac{F(t)}{t^{2}}$ and $\frac{f(t)}{ \vert t \vert }$ is nondecreasing on $t\in \mathbb{R}\setminus \{0\}$.

Since the nonlinearity and the coupling terms in problem (1.1) are both critical, the existence of nontrivial solutions to (1.1) depends heavily on the least energy solutions of the corresponding limit problem

$$ \textstyle\begin{cases} -\Delta u= \vert u \vert ^{2^{*}-2}u+ \beta \vert u \vert ^{\frac{2^{*}}{2}-2}u \vert v \vert ^{\frac{2^{*}}{2}}, & x\in\mathbb{R}^{N}, \\ -\Delta v= \vert v \vert ^{2^{*}-2}v+ \beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}-2}v, & x\in\mathbb{R}^{N}, \\ u,v\in D^{1,2}(\mathbb{R}^{N}),& \end{cases} $$

(1.8)

where $D^{1,2}(\mathbb{R}^{N})=\{u\in L^{2}(\mathbb{R}^{N}) | | \nabla u| \in L^{2}(\mathbb{R}^{N})\}$.

Recall that S is the best constant of $D^{1,2}(\mathbb{R}^{N})\hookrightarrow L^{2^{*}}(\mathbb{R}^{N})$, i.e.

$$S=\inf_{u\in D^{1,2}(\mathbb{R}^{N})\setminus \{0\}} \frac{\int_{\mathbb{R}^{N}}|\nabla u|^{2}}{(\int_{\mathbb{R}^{N}}|u|^{2^{*}})^{\frac{2}{2^{*}}}}. $$

For any $\varepsilon>0$ and $y\in\mathbb{R}^{N}$, S is achieved by the Aubin-Talenti instanton (see [20, 21])

$$ U_{\varepsilon,y}(x)=\frac{ [N(N-2)\varepsilon ^{2}]^{\frac{N-2}{4}}}{(\varepsilon^{2}+ \vert x-y \vert ^{2})^{\frac{N-2}{2}}} $$

(1.9)

and

$$ \int_{\mathbb{R}^{N}} \vert \nabla U_{\varepsilon,y} \vert ^{2}= \int _{\mathbb{R} ^{N}} \vert U_{\varepsilon,y} \vert ^{2^{*}}=S^{\frac{N}{2}}. $$

(1.10)

As showed in [15], the following manifold

$$ \begin{aligned}[b] P&= \left\{(u,v)\in D^{1,2}\bigl(\mathbb{R}^{N} \bigr)\times D^{1,2}\bigl(\mathbb {R}^{N}\bigr)\left|\vphantom{\textstyle\begin{cases} \int_{\mathbb{R}^{N}} \vert \nabla u \vert ^{2}=\int_{\mathbb{R}^{N}}( \vert u \vert ^{2^{*}}+\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}), \\ \int_{\mathbb{R}^{N}} \vert \nabla v \vert ^{2}=\int_{\mathbb{R}^{N}}( \vert v \vert ^{2^{*}}+\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}) \end{cases}\displaystyle }\right.\right.\\ &\left.\quad {} u,v\not\equiv 0, \textstyle\begin{cases} \int_{\mathbb{R}^{N}} \vert \nabla u \vert ^{2}=\int_{\mathbb{R}^{N}}( \vert u \vert ^{2^{*}}+\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}), \\ \int_{\mathbb{R}^{N}} \vert \nabla v \vert ^{2}=\int_{\mathbb{R}^{N}}( \vert v \vert ^{2^{*}}+\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}) \end{cases}\displaystyle \right\} \end{aligned} $$

(1.11)

contains all nontrivial solutions to problem (1.8). Set $A:=\inf_{(u,v)\in P}J(u,v)$, where $J(u,v)$ is the corresponding energy functional. It is proved in [15] that when $N\geq5$, A is attained and $A<\frac{1}{N}S^{\frac{N}{2}}$ for each $\beta>0$; and while $\beta<0$, $A=\frac{2}{N}S^{\frac{N}{2}}$ is not attained. This fact brings about the difference of the existence result in Theorem 1.1 between $\beta>0$ and $\beta<0$. To prove Theorem 1.1, we easily see that the functional I possesses a mountain pass geometry and then a $(PS)$ sequence exists. For $\beta >0$, we could pull the mountain pass energy down below $\min\{A,B_{1},B_{2}\}$, then the $(PS)$ condition holds for I. However, the above energy estimate cannot be directly applied to the case $\beta<0$ since A is not attained when $\beta<0$. We overcome this difficulty by working in the radially symmetric Sobolev subspace $H_{r}=H^{1}_{r}(\mathbb{R} ^{N})\times H^{1}_{r}(\mathbb{R}^{N})$, where $H^{1}_{r}(\mathbb{R}^{N})=\{u\in H^{1}(\mathbb{R}^{N})\vert u(x)=u( \vert x \vert )\}$ and using the constrained minimization on the following manifold defined similarly to (1.11):

$$\begin{aligned} M&= \left\{(u,v)\in H_{r}\left\vert\vphantom{\textstyle\begin{cases} \int_{\mathbb{R}^{N}}( \vert \nabla u \vert ^{2}+ \vert u \vert ^{2})=\int_{\mathbb {R}^{N}}( \vert u \vert ^{2^{*}}+\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}+f(u)u), \\ \int_{\mathbb{R}^{N}}( \vert \nabla v \vert ^{2}+ \vert v \vert ^{2})=\int_{\mathbb {R}^{N}}( \vert v \vert ^{2^{*}}+\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}+g(v)v) \end{cases}\displaystyle }\right.\right.\\ &\quad {}\left. u,v\not\equiv0,\textstyle\begin{cases} \int_{\mathbb{R}^{N}}( \vert \nabla u \vert ^{2}+ \vert u \vert ^{2})=\int_{\mathbb {R}^{N}}( \vert u \vert ^{2^{*}}+\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}+f(u)u), \\ \int_{\mathbb{R}^{N}}( \vert \nabla v \vert ^{2}+ \vert v \vert ^{2})=\int_{\mathbb {R}^{N}}( \vert v \vert ^{2^{*}}+\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}+g(v)v) \end{cases}\displaystyle \right\}. \end{aligned} $$

Then Theorem 1.1 is proved. It is necessary to point out that due to the existence of the perturbation terms in I, we need the assumption $\beta>-1$ to show that the manifold M is a suitable one for our problem, i.e. a minimizer of I constrained on M is a nontrivial solution of (1.1).

Throughout this paper, we use standard notations. For simplicity, we write $\int_{\Omega} h$ to mean the Lebesgue integral of $h(x)$ over a domain $\Omega\subset\mathbb{R}^{N}$. $L^{p}:= L^{p}(\mathbb {R}^{N})$ ($1\leq p<+\infty$) is the usual Lebesgue space with the standard norm $\vert \cdot \vert _{p}$. We use ‘→’ and ‘⇀’ to denote the strong and weak convergence in the related function space, respectively. C, $\{C_{i}\}_{i=1}^{+\infty}$ will denote a positive constant unless specified. We use ‘:=’ to denote definitions. $B_{r}(x):=\{y\in\mathbb{R}^{N} || y-x| < r\}$. We use ‘$X^{-1}$’ to denote the dual space of X. We denote a subsequence of a sequence $\{u_{n}\}$ as $\{u_{n}\}$ to simplify the notation unless specified.

The paper is organized as follows. In Section 2, we prove Theorem 1.1 with $\beta>0$; in Section 3, we give the proof of Theorem 1.1 with $\beta\in(-1,0)$.

2 Proof of Theorem 1.1 with $\beta>0$

In this section, we consider the case $\beta>0$. We first give some preliminary results.

Lemma 2.1

Suppose that (F₁)-(F₃) hold and $\beta>0$, then I possesses a mountain pass geometry around $(0,0)$:

• (1) There exist $\rho,\sigma>0$ such that $\inf_{ \Vert (u,v) \Vert _{H}=\rho}I(u,v)\geq\sigma$;

• (2) There exists $(u_{0},v_{0})\in H$ such that $\Vert (u_{0},v_{0}) \Vert _{H}>\rho$ and $I(u_{0},v_{0})<0$.

Proof

For any $(u,v)\in H\setminus \{(0,0)\}$, by (1.6) and the Sobolev embedding inequality, there is a constant $C>0$ such that

$$I(u,v)\geq\frac{1}{4} \bigl\Vert (u,v) \bigr\Vert _{H}^{2}-\frac{C(1+\beta)}{2^{*}} \bigl\Vert (u,v) \bigr\Vert _{H}^{2^{*}}, $$

then there exist $\sigma,\rho>0$ such that $I(u,v)\geq\sigma$ for all $\Vert (u,v) \Vert _{H}=\rho$.

By (1.7), there exists $C>0$ such that

$$ F(s)\geq C \vert s \vert ^{2+\theta_{1}},\qquad G(s)\geq C \vert s \vert ^{2+\theta_{2}},\quad \forall s\in\mathbb{R}. $$

(2.1)

For any $t\geq0$, we see that $I(tu,tv)\rightarrow-\infty$ as $t\rightarrow+\infty$. Then there exists $t_{0}>0$ such that $I(t_{0}u,t_{0}v)<0$ and $\Vert (t_{0}u,t_{0}v)\Vert _{H}>\rho$. □

By the mountain pass theorem (see, e.g., Theorem 2.10 in [22]), there exists a $(PS)_{\mathcal{B}}$ sequence $\{(u_{n},v_{n})\}\subset H$ such that

$$I(u_{n},v_{n})\rightarrow\mathcal{B}, \qquad I'(u_{n},v_{n})\rightarrow0 \quad \text{in } H^{-1}, $$

where

$$\mathcal{B}= \inf_{\gamma\in\Gamma}\max_{t\in [0,1]}I\bigl( \gamma(t)\bigr)>0, $$

and $\Gamma=\{\gamma\in C([0,1],H)\vert \gamma(0)=0,I(\gamma(t))<0\}$.

Lemma 2.2

Suppose that (F₁)-(F₃) hold and $\beta>0$, then for any $(u,v)\in H\setminus \{(0,0)\}$, there exists a unique $\tilde {t}=\tilde{t}_{(u,v)}>0$ such that $\Psi(\tilde{t}u,\tilde{t}v)=0$ and $I(\tilde{t}u,\tilde{t}v)=\max_{t\geq0}I(tu,tv)$, where

$$\Psi(u,v)= \bigl\Vert (u,v) \bigr\Vert _{H}^{2} - \int_{\mathbb{R} ^{N}}\bigl( \vert u \vert ^{2^{*}}+ \vert v \vert ^{2^{*}}+2\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac {2^{*}}{2}}\bigr)- \int_{\mathbb{R}^{N}}f(u)u- \int_{\mathbb{R}^{N}}g(v)v. $$

Proof

For any $(u,v)\in H\setminus \{(0,0)\}$ and any $t\geq0$, by (F₁)-(F₃), we see that

$$h(t)=\frac{t^{2}}{2} \bigl\Vert (u,v) \bigr\Vert _{H}^{2}- \frac {t^{2^{*}}}{2^{*}} \int_{\mathbb {R}^{N}}\bigl( \vert u \vert ^{2^{*}}+ \vert v \vert ^{2^{*}}+2\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}\bigr)- \int_{\mathbb {R}^{N}}F(tu)- \int_{\mathbb{R}^{N}}G(tv) $$

has a unique critical point $\tilde{t}>0$ corresponding to its maximum. Then $h(\tilde{t})=\max_{t\geq0}h(t)$ and $h'(\tilde{t})=0$. So $\Psi(\tilde{t}u,\tilde{t}v)=0$. □

Set

$$\mathcal{M}=\bigl\{ (u,v)\in H\setminus\bigl\{ (0,0)\bigr\} \vert \Psi (u,v)=0\bigr\} . $$

By Lemma 2.2, $\mathcal{M}\neq\varnothing$. Indeed, $\mathcal{M}$ contains all nontrivial and semitrivial solutions of (1.1). For any $(u,v)\in\mathcal{M}$, by $\beta>0$ and (1.7), we have $I(u,v)=I(u,v)-\frac{1}{2} \Psi(u,v)\geq0$, i.e. $I(u,v)$ is bounded from below on $\mathcal{M}$. Moreover, it is easy to check that

$$\mathcal{B}=\inf_{\substack{(u,v)\neq(0,0)\\ (u,v)\in H}}\max_{t\geq0}I(tu,tv)= \inf_{(u,v)\in\mathcal{M}}I(u,v). $$

For each $\beta>0$, as showed in [17], we set

$$S_{\beta}:=\inf_{u,v\in D^{1,2}(\mathbb{R}^{N})\setminus \{0\} }\frac {\int_{\mathbb{R}^{N}}( \vert \nabla u \vert ^{2}+ \vert \nabla v \vert ^{2})}{[\int_{\mathbb{R} ^{N}}( \vert u \vert ^{2^{*}}+ \vert v \vert ^{2^{*}}+2\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac {2^{*}}{2}})]^{\frac{2}{2^{*}}}}. $$

Then $S_{\beta}\geq0$ is well defined.

Lemma 2.3

Lemma 2.1, Lemma 2.2, [17]

For each $\beta>0$,

$$S_{\beta}=\frac{1+\tau_{0}^{2}}{(1+\tau_{0}^{\frac{2N}{N-2}}+2\beta\tau _{0}^{\frac{N}{N-2}})^{\frac{N-2}{N}}}S $$

and $S_{\beta}$ is attained by $(\tau_{0} U_{\varepsilon,y},U_{\varepsilon,y})$, where $\tau_{0}$ is the unique positive zero point of $\varphi (\tau)=1+\beta\tau^{\frac{N}{N-2}}-\beta\tau^{\frac {4-N}{N-2}}-\tau^{\frac{4}{N-2}}$.

For $\rho>0$, let $\psi\in C_{0}^{\infty}(B_{2\rho}(0))$ be a cut-off function with $0\leq\psi\leq1$ and $\psi\equiv1$ for $\vert x \vert \leq \rho$. For $\varepsilon>0$, denote

$$ w_{\varepsilon}:=\psi U_{\varepsilon,0}. $$

(2.2)

Then, by [23], we have

$$ \begin{gathered} \int_{\mathbb{R}^{N}} \vert \nabla w_{\varepsilon} \vert ^{2}=S^{\frac{N}{2}}+O\bigl(\varepsilon ^{N-2}\bigr), \qquad \int_{\mathbb{R}^{N}} \vert w_{\varepsilon} \vert ^{2^{*}}=S^{\frac{N}{2}}+O \bigl(\varepsilon^{N}\bigr), \\ \int_{\mathbb{R}^{N}}\vert w_{\varepsilon }\vert ^{2}=C_{1} \varepsilon^{2}+O\bigl(\varepsilon^{N-2}\bigr), \end{gathered} $$

(2.3)

where $C_{1}>0$ is a constant independent of ε.

Lemma 2.4

Lemma 2.3, [18]

Suppose that (F ₂) holds. For any sequences $\{t_{\varepsilon}\}$, $\{s_{\varepsilon}\}$ satisfying that there exist two constants $0< C_{2}< C_{3}<+\infty$ independent of ε such that $C_{2}\leq t_{\varepsilon},s_{\varepsilon}\leq C_{3}$ for ε small enough, then

$$\lim_{\varepsilon\rightarrow0^{+}}\frac{\int_{\mathbb{R} ^{N}}F(t_{\varepsilon}w_{\varepsilon})}{ \varepsilon^{2}}=+\infty, \qquad \lim _{\varepsilon\rightarrow0^{+}}\frac{\int_{\mathbb{R} ^{N}}G(s_{\varepsilon}w_{\varepsilon})}{ \varepsilon^{2}}=+\infty. $$

As mentioned in Section 1, $u_{1}$, $v_{1}\in H^{1}(\mathbb{R}^{N})$ are respectively radially positive least energy solutions of (1.3) and (1.4) with the corresponding energy denoted by $B_{1}$, $B_{2}$, i.e.

$$ I(u_{1},0)=B_{1}< \frac{1}{N} S^{\frac{N}{2}}, \qquad I(0,v_{1})=B_{2}< \frac{1}{N} S^{\frac{N}{2}}. $$

(2.4)

By standard regularity arguments, $u_{1},v_{1}\in C(\mathbb{R}^{N})$ and $u_{1}(0)=\max_{x\in\mathbb{R}^{N}}u_{1}(x)$, $v_{1}(0)=\max_{x\in\mathbb{R} ^{N}}v_{1}(x)$.

Lemma 2.5

Suppose that (F₁)-(F₃) hold and $\beta>0$, then

$$\mathcal{B}< \min\biggl\{ B_{1},B_{2},\frac{1}{N}S_{\beta}^{\frac{N}{2}} \biggr\} . $$

Proof

The proof consists of two steps.

Step 1: We prove that $\mathcal{B}<\frac{1}{N}S_{\beta }^{\frac{N}{2}}$.

For $\varepsilon>0$, denote $(u_{\varepsilon},v_{\varepsilon}):= (\tau_{0}w_{\varepsilon}, w_{\varepsilon})$, where $\tau_{0}$ is given in Lemma 2.3. By Lemma 2.2, there exists a unique $t_{\varepsilon}>0$ such that

$$ (t_{\varepsilon}u_{\varepsilon},t_{\varepsilon}v_{\varepsilon}) \in \mathcal{M} $$

(2.5)

and $I(t_{\varepsilon}u_{\varepsilon},t_{\varepsilon}v_{\varepsilon})=\max_{t\geq0}I(tu_{\varepsilon},tv_{\varepsilon})$.

We claim that $\{t_{\varepsilon}\}_{\varepsilon>0}$ is bounded from below by a positive constant. Otherwise, there exists a sequence $\{\varepsilon_{n}\}\subset\mathbb{R}_{+}$ satisfying $\lim_{n\rightarrow+\infty}t_{\varepsilon_{n}}=0$ and $I(t_{\varepsilon_{n}}u_{\varepsilon_{n}},t_{\varepsilon _{n}}v_{\varepsilon_{n}})=\max_{t\geq0} I(tu_{\varepsilon_{n}},tv_{\varepsilon_{n}})$, then by (2.2)-(2.5) and (F₁), (F₃), we have $0< \mathcal{B}\leq\lim_{n\rightarrow+\infty}I(t_{\varepsilon _{n}}u_{\varepsilon_{n}}, t_{\varepsilon_{n}}v_{\varepsilon_{n}})=0$, which is impossible. So there exist $0< C_{4}< C_{5}$ independent of ε satisfying that

$$ C_{4}\leq t_{\varepsilon}\leq C_{5}\quad \text{for all }\varepsilon>0. $$

(2.6)

Then, by (2.2)-(2.6) and Lemmas 2.3 and 2.4, we see that

$$\begin{aligned} \mathcal{B} \leq& I(t_{\varepsilon}u_{\varepsilon},t_{\varepsilon}v_{\varepsilon}) \\ \leq&\max_{t>0} \biggl\{ \frac{t^{2}}{2}\bigl\Vert (u_{\varepsilon },v_{\varepsilon})\bigr\Vert _{H}^{2} - \frac{t^{2^{*}}}{2^{*}} \int_{\mathbb{R}^{N}}\bigl(\vert u_{\varepsilon } \vert ^{2^{*}}+ \vert v_{\varepsilon} \vert ^{2^{*}}+2 \beta \vert u_{\varepsilon}\vert ^{\frac{2^{*}}{2}} \vert v_{\varepsilon} \vert ^{\frac{2^{*}}{2}}\bigr) \biggr\} \\ &{} - \int_{\mathbb{R} ^{N}}\bigl[F(t_{\varepsilon}u_{\varepsilon})+G(t_{\varepsilon }v_{\varepsilon}) \bigr] \\ \leq&\frac{ \Vert (u_{\varepsilon},v_{\varepsilon}) \Vert _{H}^{2}}{N} \biggl[\frac{ \Vert (u_{\varepsilon},v_{\varepsilon}) \Vert _{H}^{2}}{\int _{\mathbb{R} ^{N}}( \vert u_{\varepsilon} \vert ^{2^{*}}+ \vert v_{\varepsilon} \vert ^{2^{*}}+2\beta \vert u_{\varepsilon} \vert ^{\frac{2^{*}}{2}} \vert v_{\varepsilon} \vert ^{\frac{2^{*}}{2}})} \biggr]^{\frac{2}{2^{*}-2}}\\ &{}- \int _{\mathbb{R}^{N}}F(t_{\varepsilon}u_{\varepsilon})- \int_{\mathbb{R} ^{N}}G(t_{\varepsilon}v_{\varepsilon}) \\ \leq&\frac{1}{N}\frac{[(1+\tau_{0}^{2})S^{\frac{N}{2}}+C_{1}(1+\tau _{0}^{2})\varepsilon^{2}+O(\varepsilon^{N-2})]^{\frac{N}{2}}}{[(1+2\beta\tau _{0}^{\frac{N}{N-2}}+\tau_{0}^{\frac{2N}{N-2}})S^{\frac {N}{2}}+O(\varepsilon^{N})]^{\frac{N-2}{2}}} - \int_{\mathbb{R}^{N}}F(t_{\varepsilon}u_{\varepsilon})- \int _{\mathbb{R} ^{N}}G(t_{\varepsilon}v_{\varepsilon}) \\ \leq&\frac{1}{N}\frac{ (1+\tau_{0}^{2} )^{\frac{N}{2}}S^{\frac{N}{2}}}{(1+2\beta\tau_{0}^{\frac{N}{N-2}}+\tau_{0}^{\frac {2N}{N-2}})^{\frac{N-2}{2}}}+C_{6}\varepsilon^{2}+O \bigl(\varepsilon^{N-2}\bigr) - \int_{\mathbb{R}^{N}}F(t_{\varepsilon}u_{\varepsilon})- \int _{\mathbb{R} ^{N}}G(t_{\varepsilon}v_{\varepsilon}) \\ < & \frac{1}{N}S_{\beta}^{\frac{N}{2}}\quad \text{for } \varepsilon>0\text{ small enough}. \end{aligned}$$

Step 2: We prove that $\mathcal{B}< B_{1}$ and $\mathcal{B}< B_{2}$.

The idea of this proof follows from Lemma 2.7 in [15].

We define a function $H:\mathbb{R}^{2}\rightarrow\mathbb{R}$ by

$$H(t,s)=\Psi(tu_{1},tsv_{1}). $$

It is easy to check that $H(1,0)=0$ and $H_{t}(1,0)\neq0$. Then, by the implicit function theorem, there exist $\delta>0$ and a function $t(s)\in C^{1}(-\delta,\delta)$ such that

$$t(0)=1,\qquad t^{\prime}(s)=-\frac{H_{s}(t,s)}{H_{t}(t,s)}\quad \text{and}\quad H \bigl(t(s),s\bigr)=0,\quad \forall s\in(-\delta,\delta), $$

which implies that

$$ \bigl(t(s)u_{1},t(s)sv_{1}\bigr)\in \mathcal{M},\quad \forall s\in (-\delta,\delta). $$

(2.7)

Since $\frac{2^{*}}{2}<2<2^{*}$ and $\beta>0$, by direct calculation and (F₁), (F₂), we have

$$\lim_{s\rightarrow0}\frac{t'(s)}{ \vert s \vert ^{\frac {2^{*}}{2}-2}s}=\frac{-2^{*}\beta\int_{\mathbb {R}^{N}} \vert u_{1} \vert ^{\frac {2^{*}}{2}} \vert v_{1} \vert ^{\frac{2^{*}}{2}}}{(2^{*}-2)\int_{\mathbb{R}^{N}} \vert u_{1} \vert ^{2^{*}} +\int_{\mathbb{R}^{N}}[f'(u_{1})u_{1}^{2}-f(u_{1})u_{1}]}< 0, $$

i.e.

$$t'(s)=\frac{-2^{*}\beta\int_{\mathbb{R} ^{N}} \vert u_{1} \vert ^{\frac{2^{*}}{2}} \vert v_{1} \vert ^{\frac{2^{*}}{2}}}{(2^{*}-2) \int_{\mathbb{R} ^{N}} \vert u_{1} \vert ^{2^{*}} +\int_{\mathbb{R}^{N}}[f'(u_{1})u_{1}^{2}-f(u_{1})u_{1}]} \vert s \vert ^{\frac {2^{*}}{2}-2}s \bigl(1+o(1)\bigr)\quad \text{as } s\rightarrow0. $$

So

$$ t(s)=1-\frac{2\beta\int_{\mathbb{R} ^{N}} \vert u_{1} \vert ^{\frac{2^{*}}{2}} \vert v_{1} \vert ^{\frac{2^{*}}{2}}}{(2^{*}-2) \int_{\mathbb{R} ^{N}} \vert u_{1} \vert ^{2^{*}} +\int_{\mathbb{R}^{N}}[f'(u_{1})u_{1}^{2}-f(u_{1})u_{1}]} \vert s \vert ^{\frac {2^{*}}{2}} \bigl(1+o(1)\bigr)\quad \text{as }s\rightarrow0. $$

(2.8)

Then

$$ t^{2^{*}}(s)=1-\frac{22^{*}\beta\int _{\mathbb{R} ^{N}} \vert u_{1} \vert ^{\frac{2^{*}}{2}} \vert v_{1} \vert ^{\frac{2^{*}}{2}}}{(2^{*}-2) \int_{\mathbb{R} ^{N}} \vert u_{1} \vert ^{2^{*}} +\int_{\mathbb{R}^{N}}[f'(u_{1})u_{1}^{2}-f(u_{1})u_{1}]} \vert s \vert ^{\frac {2^{*}}{2}} \bigl(1+o(1)\bigr) \quad \text{as }s\rightarrow0. $$

(2.9)

By (F₂), we see that the function $\widetilde{F}(t):=\frac{1}{2}f(t)t-F(t)$ is nondecreasing on $(0,+\infty)$. By (2.8), we may assume that $0< t(s)\leq1$ for $\vert s \vert$ small enough. So

$$ \widetilde{F}\bigl(t(s)u_{1}\bigr)\leq\widetilde{ F}(u_{1})\quad \text{for } \vert s \vert \text{ small enough}. $$

(2.10)

By (F₁), (F₃), we have $\lim_{s\rightarrow0}\frac {g(t(s)sv_{1})t(s)sv_{1}}{ \vert s \vert ^{\frac{2^{*}}{2}}}=\lim_{s\rightarrow 0}\frac{G(t(s)sv_{1})}{ \vert s \vert ^{\frac{2^{*}}{2}}}=0$. Thus, by (2.4), (2.7)-(2.10) and $\beta>0$, we see that for $\forall s\in(-\delta,\delta)$,

$$\begin{aligned} \mathcal{B} \leq& I\bigl(t(s)u_{1},t(s)s v_{1}\bigr)- \frac{1}{2} \Psi\bigl(t(s)u_{1},t(s)s v_{1}\bigr) \\ =& \frac{1}{N}t^{2^{*}}(s) \int_{\mathbb {R}^{N}}\bigl( \vert u_{1} \vert ^{2^{*}}+ \vert s \vert ^{2^{*}} \vert v_{1} \vert ^{2^{*}}+2\beta \vert s \vert ^{\frac{2^{*}}{2}} \vert u_{1} \vert ^{\frac{2^{*}}{2}} \vert v_{1} \vert ^{\frac{2^{*}}{2}}\bigr) \\ &{}+ \int_{\mathbb{R}^{N}} \biggl[\frac{1}{2} f\bigl(t(s)u_{1} \bigr)t(s)u_{1}-F\bigl(t(s)u_{1}\bigr) \biggr]+ \int_{\mathbb{R}^{N}} \biggl[\frac{1}{2} g\bigl(t(s)sv_{1} \bigr)t(s)sv_{1}-G\bigl(t(s)sv_{1}\bigr) \biggr] \\ \leq&\frac{1}{N} \int_{\mathbb{R}^{N}} \vert u_{1} \vert ^{2^{*}}+ \int _{\mathbb{R} ^{N}} \biggl[\frac{1}{2} f(u_{1})u_{1}-F(u_{1}) \biggr] - \vert s \vert ^{\frac{2^{*}}{2}}\frac{2\beta}{2^{*}} \int_{\mathbb {R}^{N}} \vert u_{1} \vert ^{\frac {2^{*}}{2}} \vert v_{1} \vert ^{\frac{2^{*}}{2}}+o\bigl( \vert s \vert ^{\frac{2^{*}}{2}}\bigr) \\ < &\frac{1}{N} \int_{\mathbb{R}^{N}} \vert u_{1} \vert ^{2^{*}}+ \int _{\mathbb{R}^{N}} \biggl[\frac{1}{2} f(u_{1})u_{1}-F(u_{1}) \biggr]=B_{1}\quad \text{as } \vert s \vert >0 \text{ small enough}. \end{aligned}$$

Hence $\mathcal{B}< B_{1}$. Similarly, we have $\mathcal{B}< B_{2}$. Therefore the proof of the lemma is completed. □

Lemma 2.6

[22], Vanishing lemma

Let $r>0$ and $2\leq q<2^{*}$. If $\{u_{n}\}$ is bounded in $H^{1}(\mathbb{R}^{N})$ and

$$\sup_{y\in\mathbb{R}^{N}} \int _{B_{r}(y)} \vert u_{n} \vert ^{q} \rightarrow0,\quad n\rightarrow+\infty, $$

then $u_{n}\rightarrow0$ in $L^{p}(\mathbb{R}^{N})$ for $2< p<2^{*}$.

Proof of Theorem 1.1 with $\beta>0$

By Lemma 2.1, there exists a sequence $\{(u_{n},v_{n})\}\subset H$ such that

$$ \lim_{n\rightarrow+\infty}I(u_{n},v_{n})= \mathcal{B},\qquad \lim_{n\rightarrow+\infty}I'(u_{n},v_{n})=0. $$

(2.11)

By (1.7), we easily see that $\{(u_{n},v_{n})\}$ is uniformly bounded in H.

Let

$$ \delta_{1}:=\lim_{n\rightarrow+\infty}\sup _{y\in \mathbb{R} ^{N}} \int_{B_{1}(y)} \vert u_{n} \vert ^{2}, \qquad \delta_{2}:=\lim_{n\rightarrow +\infty}\sup _{z\in\mathbb{R}^{N}} \int_{B_{1}(z)} \vert v_{n} \vert ^{2}, $$

(2.12)

then $\delta_{1},\delta_{2}\in[0,+\infty)$. If $\delta_{1}=\delta_{2}=0$, then by the vanishing lemma 2.6, we have $u_{n}\rightarrow0$ in $L^{p}(\mathbb{R}^{N})$ and $v_{n}\rightarrow0$ in $L^{p}(\mathbb {R}^{N})$, $\forall 2< p<2^{*}$. By (F₁), (F₃), for any $\varepsilon>0$, there exists $C_{\varepsilon}>0$ such that

$$ \bigl\vert F(t) \bigr\vert \leq\varepsilon\bigl( \vert t \vert ^{2}+ \vert t \vert ^{2^{*}}\bigr)+C_{\varepsilon} \vert t \vert ^{p},\qquad \bigl\vert G(t) \bigr\vert \leq \varepsilon\bigl( \vert t \vert ^{2}+ \vert t \vert ^{2^{*}}\bigr)+C_{\varepsilon} \vert t \vert ^{p}. $$

(2.13)

Then

$$\limsup_{n\rightarrow+\infty} \int_{\mathbb{R}^{N}} \bigl\vert F(u_{n}) \bigr\vert \leq \limsup_{n\rightarrow+\infty} \biggl[\varepsilon \int_{\mathbb{R} ^{N}}\bigl( \vert u_{n} \vert ^{2}+ \vert u_{n} \vert ^{2^{*}}\bigr)+C_{\varepsilon} \int_{\mathbb {R}^{N}} \vert u_{n} \vert ^{p} \biggr] \leq C\varepsilon, $$

which shows that $\lim_{n\rightarrow+\infty}\int_{\mathbb{R} ^{N}}F(u_{n})=0$ since $\varepsilon>0$ is arbitrary. Similarly,

$$\lim_{n\rightarrow+\infty}\int_{\mathbb{R}^{N}}G(v_{n})=0. $$

By (2.11) and the boundedness of $\{(u_{n},v_{n})\}$, we see that

$$ \lim_{n\rightarrow+\infty} \bigl\Vert (u_{n},v_{n}) \bigr\Vert _{H}^{2}=\lim_{n\rightarrow+\infty} \int_{\mathbb {R}^{N}}\bigl( \vert u_{n} \vert ^{2^{*}}+ \vert v_{n} \vert ^{2^{*}}+2\beta \vert u_{n} \vert ^{\frac{2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}}\bigr)=N\mathcal{B}. $$

(2.14)

For n large, we may assume that $u_{n},v_{n}\in D^{1,2}(\mathbb {R}^{N})\setminus \{0\}$. Hence, by (2.14), we see that

$$S_{\beta}\leq\frac{ \Vert (u_{n},v_{n}) \Vert _{H}^{2}}{[\int_{\mathbb{R} ^{N}}( \vert u_{n} \vert ^{2^{*}}+ \vert v_{n} \vert ^{2^{*}}+2\beta \vert u_{n} \vert ^{\frac{2^{*}}{2}} \vert v_{n} \vert ^{\frac {2^{*}}{2}})]^{\frac{2}{2^{*}}}}\rightarrow(N\mathcal{B})^{\frac{2}{N}}, $$

i.e. $\mathcal{B}\geq\frac{1}{N}S_{\beta}^{\frac{N}{2}}$, which contradicts Lemma 2.5.

So we deduce that at least one of the following two inequalities $\delta_{1}>0$ and $\delta_{2}>0$ holds. Without loss of generality, we may assume that $\delta_{1}>0$. There exists a sequence $\{y_{n}\}\subset\mathbb{R}^{N}$ such that

$$ \int_{B_{1}(y_{n})} \vert u_{n} \vert ^{2}\geq \frac{\delta_{1}}{2}>0. $$

(2.15)

Set

$$\tilde{u}_{n}(x):=u_{n}(x+y_{n}),\qquad \tilde{v}_{n}(x):=v_{n}(x+y_{n}). $$

Then $\{(\tilde{u}_{n},\tilde{v}_{n})\}$ is also a bounded $(PS)_{\mathcal{B}}$ sequence for I. Up to a subsequence, we may assume that there exists $(\tilde{u},\tilde{v})\in H$ such that $(\tilde{u}_{n},\tilde{v}_{n})\rightharpoonup(\tilde{u},\tilde{v})$ in H, then $I'(\tilde{u},\tilde{v})=0$. Moreover, by the Sobolev embedding theorem, we have

$$\textstyle\begin{cases} (\tilde{u}_{n},\tilde{v}_{n})\rightarrow(\tilde{u},\tilde {v})& \text{in } L_{\mathrm{loc}}^{p}(\mathbb{R}^{N})\times L_{\mathrm{loc}}^{p}(\mathbb{R}^{N}), \forall 2\leq p< 2^{*}, \\ \tilde{u}_{n}(x)\rightarrow\tilde{ u}(x), \tilde {v}_{n}(x)\rightarrow\tilde{ v}(x)& \text{a.e. in } \mathbb{R}^{N}, \end{cases} $$

which and (2.15) imply that $\tilde{u}\not\equiv0$.

If $\tilde{v}\equiv0$, then ũ is a nontrivial solution of $-\Delta u+u= \vert u \vert ^{2^{*}-2}u+f(u)$ in $\mathbb{R}^{N}$. Then $I(\tilde {u},0)\geq B_{1}$. Hence, by $\beta>0$, (1.7) and Fatou’s lemma, we have

$$\begin{aligned} \mathcal{B} &=\lim_{n\rightarrow+\infty} \biggl(I( \tilde {u}_{n},\tilde{v}_{n})-\frac{1}{2}\bigl\langle I'(\tilde{u}_{n},\tilde {v}_{n}),( \tilde{u}_{n},\tilde{v}_{n})\bigr\rangle \biggr) \\ &=\lim_{n\rightarrow+\infty} \biggl[\frac {1}{N} \int_{\mathbb{R}^{N}}\bigl( \vert \tilde{u}_{n} \vert ^{2^{*}}+\vert \tilde {v}_{n}\vert ^{2^{*}}+2\beta \vert \tilde{u}_{n} \vert ^{\frac{2^{*}}{2}} \vert \tilde{v}_{n} \vert ^{\frac{2^{*}}{2}}\bigr) + \int_{\mathbb{R}^{N}} \biggl(\frac{1}{2}f(\tilde{u}_{n}) \tilde {u}_{n}-F(\tilde {u}_{n}) \biggr) \\ &\quad {} + \int _{\mathbb{R} ^{N}} \biggl(\frac{1}{2}g(\tilde{v}_{n}) \tilde{v}_{n}-G(\tilde{v}_{n}) \biggr) \biggr] \\ &\geq\liminf_{n\rightarrow+\infty} \biggl[\frac{1}{N} \int_{\mathbb{R}^{N}} \vert \tilde{u}_{n} \vert ^{2^{*}}+ \int_{\mathbb {R}^{N}} \biggl(\frac{1}{2}f(\tilde{u}_{n}) \tilde{u}_{n}-F(\tilde{u}_{n}) \biggr) \biggr] \\ &\geq\frac{1}{N} \int_{\mathbb{R}^{N}}\vert \tilde {u}\vert ^{2^{*}}+ \int_{\mathbb{R} ^{N}} \biggl(\frac{1}{2}f(\tilde{u})\tilde{u}-F( \tilde{u}) \biggr) \\ &= I(\tilde{u},0)\geq B_{1}, \end{aligned} $$

which contradicts Lemma 2.5. So $\tilde{v}\not\equiv0$. Then $(\tilde{u},\tilde{v})$ is a nontrivial solution to (1.1). Thus $(\tilde{u},\tilde{v})\in\mathcal{M}$ and

$$\begin{aligned} \mathcal{B} \leq& I(\tilde{u},\tilde{v})-\frac{1}{2}\bigl\langle I'(\tilde{u},\tilde{v}),(\tilde{u},\tilde{v})\bigr\rangle \\ =&\frac{1}{N} \int_{\mathbb{R}^{N}}\bigl(\vert \tilde {u} \vert ^{2^{*}}+ \vert \tilde {v} \vert ^{2^{*}}+2\beta \vert \tilde{u} \vert ^{\frac{2^{*}}{2}} \vert \tilde{v}\vert ^{\frac{2^{*}}{2}}\bigr)\\ &{} + \int_{\mathbb{R}^{N}} \biggl(\frac{1}{2}f(\tilde{u})\tilde{u}-F( \tilde {u}) \biggr)+ \int_{\mathbb{R}^{N}} \biggl(\frac{1}{2}g(\tilde{v})\tilde {v}-G( \tilde{v}) \biggr) \\ \leq&\liminf_{n\rightarrow+\infty} \biggl\{ \frac{1}{N} \int_{\mathbb{R}^{N}}\bigl( \vert \tilde{u}_{n} \vert ^{2^{*}}+\vert \tilde {v}_{n}\vert ^{2^{*}}+2\beta \vert \tilde{u}_{n} \vert ^{\frac{2^{*}}{2}} \vert \tilde{v}_{n} \vert ^{\frac{2^{*}}{2}}\bigr) + \int_{\mathbb{R}^{N}} \biggl(\frac{1}{2}f(\tilde{u}_{n}) \tilde {u}_{n}-F(\tilde {u}_{n}) \biggr) \\ &{} + \int _{\mathbb{R}^{N}} \biggl(\frac{1}{2}g(\tilde{v}_{n}) \tilde{v}_{n}-G(\tilde {v}_{n}) \biggr) \biggr\} \\ =&\liminf_{n\rightarrow+\infty} \biggl(I(\tilde{u}_{n}, \tilde{v}_{n})-\frac{1}{2}\bigl\langle I'( \tilde{u}_{n},\tilde {v}_{n}),(\tilde{u}_{n}, \tilde{v}_{n})\bigr\rangle \biggr)=\mathcal{B}, \end{aligned}$$

which shows that $I(\tilde{u},\tilde{v})=\mathcal{B}$. Moreover,

$$I(\tilde{u},\tilde{v})=m:=\inf\bigl\{ I(u,v)\vert (u,v) \text{ is a nontrivial solution of } (1.1)\bigr\} . $$

Indeed, since $(\tilde{u},\tilde{v})$ is a nontrivial solution to (1.1), $I(\tilde{u},\tilde{v})\geq m$. On the other hand, for any nontrivial solution $(u,v)$ to (1.1), then $(u,v)\in\mathcal{M}$, which shows that $I(u,v)\geq\mathcal {B}$. Hence $m\geq\mathcal{B}=I(\tilde{u},\tilde{v})$ since $(u,v)$ is arbitrary. So $I(\tilde{u},\tilde{v})=m$.

Since the functional I and the manifold $\mathcal{M}$ are symmetric, we see that $( \vert \tilde{u} \vert , \vert \tilde{v} \vert )$ is also a nontrivial solution to (1.1) and $I( \vert \tilde{u} \vert , \vert \tilde{v} \vert )=\mathcal{B}$. By regularity and the maximum principle, we obtain that $\vert \tilde {u} \vert , \vert \tilde{v}\vert >0$ in $\mathbb{R}^{N}$. □

3 Proof of Theorem 1.1 with $\beta\in(-1,0)$

In this section, we study the existence of radially positive least energy solutions to (1.1) when $-1<\beta<0$. Denote $H_{r}:=H^{1}_{r}(\mathbb{R}^{N})\times H^{1}_{r}(\mathbb{R}^{N})$. To prove the theorem, we set

$$\begin{aligned} M&= \biggl\{ (u,v)\in H_{r}\bigg\vert u\not \equiv0, v\not\equiv 0, \int_{\mathbb{R}^{N}}\bigl( \vert \nabla u \vert ^{2}+ \vert u \vert ^{2}\bigr)= \int_{\mathbb {R}^{N}}\bigl( \vert u \vert ^{2^{*}}+\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}+f(u)u\bigr), \\ &\quad \int_{\mathbb{R}^{N}}\bigl( \vert \nabla v \vert ^{2}+ \vert v \vert ^{2}\bigr)= \int _{\mathbb{R} ^{N}}\bigl( \vert v \vert ^{2^{*}}+\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}+g(v)v\bigr) \biggr\} . \end{aligned} $$

Then $M\neq\varnothing$. In fact, taking $u,v\in C_{0}^{\infty }(\mathbb{R} ^{N})$, $u,v\in H^{1}_{r}(\mathbb{R}^{N})$ with $u,v\not\equiv0$ and $\operatorname{supp}(u)\cap \operatorname{supp}(v)=\varnothing$, then by (F₁)-(F₃), there exist $t_{1},t_{2}>0$ such that $(t_{1}u,t_{2}v)\in M$. For any $(u,v)\in H_{r}$, by $\beta\in (-1,0)$, the Hölder inequality and the Cauchy inequality, we have

$$ 2 \vert \beta \vert \int_{\mathbb{R}^{N}} \vert u \vert ^{\frac {2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}} \leq2 \vert \beta \vert \biggl( \int_{\mathbb{R}^{N}} \vert u \vert ^{2^{*}} \biggr)^{\frac{1}{2}} \biggl( \int_{\mathbb{R}^{N}} \vert v \vert ^{2^{*}} \biggr)^{\frac{1}{2}}< \int _{\mathbb{R} ^{N}}\bigl( \vert u \vert ^{2^{*}}+ \vert v \vert ^{2^{*}}\bigr). $$

(3.1)

Then the minimization problem

$$B:=\inf_{(u,v)\in M}I(u,v) $$

is well defined and $B\geq0$.

Lemma 3.1

Suppose that (F₁)-(F₃) hold and $\beta\in(-1,0)$, then $B>0$ and I is coercive on M. Moreover, there exists $C_{0}>0$ such that $\int_{\mathbb{R}^{N}} \vert u \vert ^{2^{*}}, \int_{\mathbb{R}^{N}} \vert v \vert ^{2^{*}}\geq C_{0}$ for any $(u,v)\in M$.

Proof

Since $\beta\in(-1,0)$, for each $(u,v)\in M$, by (3.1) and (1.7), we have

$$I(u,v)=\frac{1}{N} \int_{\mathbb {R}^{N}}\bigl( \vert u \vert ^{2^{*}}+ \vert v \vert ^{2^{*}}+2\beta \vert u \vert ^{\frac{2^{*}}{2}} \vert v \vert ^{\frac{2^{*}}{2}}\bigr) + \int_{\mathbb{R}^{N}} \biggl(\frac{1}{2}f(u)u-F(u)+ \frac{1}{2}g(v)v-G(v) \biggr)>0, $$

which implies that $B>0$ and I is coercive on M.

For any $(u,v)\in M$, by $\beta<0$, (1.5) and the Sobolev embedding inequality, we have

$$\Vert u \Vert ^{2}\leq \int_{\mathbb {R}^{N}} \vert u \vert ^{2^{*}}+ \int _{\mathbb{R}^{N}}f(u)u\leq\frac{1}{2} \Vert u \Vert ^{2}+C \int_{\mathbb {R}^{N}} \vert u \vert ^{2^{*}} \leq \frac{1}{2} \Vert u \Vert ^{2}+C S^{-\frac{2^{*}}{2}} \Vert u \Vert ^{2^{*}}, $$

which implies that $\Vert u \Vert ^{2}\geq C_{1}$ for some $C_{1}>0$ and then $\int _{\mathbb{R}^{N}} \vert u \vert ^{2^{*}}\geq\frac{1}{2C} \Vert u \Vert ^{2}\geq C_{2}$ for some $C_{2}>0$. Similarly, $\int_{\mathbb{R}^{N}} \vert v \vert ^{2^{*}}\geq C_{3}$ for some $C_{3}>0$. Set $C_{0}=\min\{C_{2},C_{3}\}$, then the lemma is proved. □

Lemma 3.2

Suppose that (F₁)-(F₃) hold and $\beta\in(-1,0)$, then $B<\min \{B_{1}+\frac{1}{N} S^{\frac{N}{2}},B_{2}+\frac{1}{N} S^{\frac{N}{2}} \}$.

Proof

We first prove that there exist $(t_{\varepsilon }u_{1},sw_{\varepsilon})\in M$, where $u_{1}$, $w_{\varepsilon}$ are defined in (2.4) and (2.2). It is enough to prove that there exist $t_{\varepsilon},s_{\varepsilon}>0$ solving the following system:

$$ \textstyle\begin{cases} t^{2}\int_{\mathbb{R}^{N}}( \vert \nabla u_{1} \vert ^{2}+u_{1}^{2})\\ \quad =t^{2^{*}}\int_{\mathbb{R} ^{N}} \vert u_{1} \vert ^{2^{*}}+t^{\frac{2^{*}}{2}}s^{\frac{2^{*}}{2}}\beta\int_{\mathbb{R} ^{N}} \vert u_{1} \vert ^{\frac{2^{*}}{2}} \vert w_{\varepsilon} \vert ^{\frac{2^{*}}{2}}+\int _{\mathbb{R} ^{N}}f(tu_{1})tu_{1}, \\ s^{2}\int_{\mathbb{R}^{N}}( \vert \nabla w_{\varepsilon} \vert ^{2}+w_{\varepsilon }^{2})\\ \quad =s^{2^{*}}\int_{\mathbb{R}^{N}} \vert w_{\varepsilon} \vert ^{2^{*}}+t^{\frac {2^{*}}{2}}s^{\frac{2^{*}}{2}}\beta\int_{\mathbb{R}^{N}} \vert u_{1} \vert ^{\frac {2^{*}}{2}} \vert w_{\varepsilon} \vert ^{\frac{2^{*}}{2}}+\int_{\mathbb{R} ^{N}}g(sw_{\varepsilon})sw_{\varepsilon}, \\ t,s>0. \end{cases} $$

(3.2)

Since $\int_{\mathbb{R}^{N}}( \vert \nabla u_{1} \vert ^{2}+u_{1}^{2})=\int_{\mathbb{R} ^{N}}( \vert u_{1} \vert ^{2^{*}}+f(u_{1})u_{1})$ and $\frac{2^{*}}{2}<2$, by (F₂) and the second equation of (3.2), we have $0< s^{\frac{2^{*}}{2}}=h(t)$, $t>1$, where

$$h(t):=\frac{(t^{2-\frac{2^{*}}{2}}-t^{\frac{2^{*}}{2}})\int_{\mathbb {R}^{N}} \vert u_{1} \vert ^{2^{*}} +t^{2-\frac{2^{*}}{2}}\int_{\mathbb{R}^{N}} (\frac {f(u_{1})}{u_{1}}-\frac {f(tu_{1})}{tu_{1}} )u_{1}^{2}}{\beta\int_{\mathbb{R}^{N}} \vert u_{1} \vert ^{\frac {2^{*}}{2}} \vert w_{\varepsilon} \vert ^{\frac{2^{*}}{2}}}. $$

Moreover, $h(1)=0$ and $\lim_{t\rightarrow+\infty }h(t)=+\infty$. Then (3.2) is equivalent to

$$\begin{aligned} \tilde{h}_{\varepsilon}(t) &:= \int_{\mathbb{R}^{N}}\bigl(\vert \nabla w_{\varepsilon }\vert ^{2}+w_{\varepsilon}^{2}\bigr)-\bigl[h(t) \bigr]^{\frac{2(2^{*}-2)}{2^{*}}} \int_{\mathbb{R} ^{N}} \vert w_{\varepsilon} \vert ^{2^{*}} - \beta\bigl[h(t)\bigr]^{\frac{2^{*}-4}{2^{*}}}t^{\frac{2^{*}}{2}} \int_{\mathbb{R} ^{N}} \vert u_{1} \vert ^{\frac{2^{*}}{2}} \vert w_{\varepsilon} \vert ^{\frac{2^{*}}{2}} \\ &\quad {} - \int_{\mathbb{R}^{N}}\frac {g([h(t)]^{\frac {2}{2^{*}}}w_{\varepsilon})}{[h(t)]^{\frac{2}{2^{*}}}w_{\varepsilon }}w_{\varepsilon}^{2}=0, \quad t>1. \end{aligned} $$

We see that $\lim_{t\rightarrow1^{+}}\tilde{h}_{\varepsilon}(t)=+\infty>0$ and $\lim_{t\rightarrow+\infty}\tilde {h}_{\varepsilon}(t)=-\infty$, so there exists $t_{\varepsilon}>1$ such that $\tilde{h}_{\varepsilon}(t_{\varepsilon})=0$. Set $s_{\varepsilon }=[h(t_{\varepsilon})]^{\frac{2}{2^{*}}}>0$. Then (3.2) has a solution $(t_{\varepsilon},s_{\varepsilon})$.

If $\lim_{\varepsilon\rightarrow0^{+}}s_{\varepsilon}=0$, then by (2.3) and (1.5), we have

$$0\leq \int_{\mathbb{R}^{N}}\frac{g(s_{\varepsilon}w_{\varepsilon })}{s_{\varepsilon}w_{\varepsilon}}w_{\varepsilon}^{2}\leq \int _{\mathbb{R} ^{N}}w_{\varepsilon}^{2}+s_{\varepsilon}^{2^{*}-2}C \int_{\mathbb{R} ^{N}} \vert w_{\varepsilon} \vert ^{2^{*}} \rightarrow0\quad \text{as } \varepsilon \rightarrow0^{+}. $$

We deduce from the second equation of system (3.2) that

$$ S^{\frac{N}{2}}=\lim_{\varepsilon\rightarrow 0^{+}} \int_{\mathbb{R}^{N}}\bigl( \vert \nabla w_{\varepsilon} \vert ^{2}+w_{\varepsilon}^{2}\bigr) \leq\lim_{\varepsilon\rightarrow0^{+}} \biggl(s_{\varepsilon}^{2^{*}-2} \int_{\mathbb{R}^{N}}\vert w_{\varepsilon }\vert ^{2^{*}}+ \int_{\mathbb{R} ^{N}}\frac{g(s_{\varepsilon}w_{\varepsilon})}{s_{\varepsilon}w_{\varepsilon}}w_{\varepsilon}^{2} \biggr) =0, $$

(3.3)

which is impossible. So there exists $s_{0}>0$ independent of ε such that $s_{\varepsilon}\geq s_{0}$ for ε small. If $\lim_{\varepsilon\rightarrow0^{+}}t_{\varepsilon }=+\infty$, then $\lim_{\varepsilon\rightarrow 0^{+}}s_{\varepsilon}=\lim_{\varepsilon\rightarrow 0^{+}}[h(t_{\varepsilon})]^{\frac{2}{2^{*}}}=+\infty$. Note that

$$\int_{\mathbb{R}^{N}} \vert u_{1} \vert ^{\frac{2^{*}}{2}} \vert w_{\varepsilon} \vert ^{\frac {2^{*}}{2}}\leq\max_{x\in\mathbb{R}^{N}}u_{1}(x) \int_{\mathbb {R}^{N}}\vert w_{\varepsilon}\vert ^{\frac{2^{*}}{2}}\leq o \bigl(\varepsilon^{2}\bigr)\rightarrow0\quad \text{as } \varepsilon \rightarrow0^{+}, $$

where we have used the fact that $\int_{\mathbb{R}^{N}}\vert w_{\varepsilon}\vert ^{\frac {2^{*}}{2}}\leq o(\varepsilon^{2})$, which is given in Lemma 3.1 of [15]. Then, by the second equation of (3.2) and (F₁), we have

$$\begin{aligned} 0&=\lim_{\varepsilon\rightarrow 0^{+}}s_{\varepsilon}^{2-2^{*}} \int_{\mathbb{R}^{N}}\bigl(\vert \nabla w_{\varepsilon }\vert ^{2}+w_{\varepsilon}^{2}\bigr) \\ &\geq\lim_{\varepsilon\rightarrow 0^{+}} \biggl( \int_{\mathbb{R}^{N}}\vert w_{\varepsilon }\vert ^{2^{*}}+\biggl( \frac {t_{\varepsilon}}{s_{\varepsilon}}\biggr)^{\frac{2^{*}}{2}}\beta \int _{\mathbb{R} ^{N}} \vert u_{1} \vert ^{\frac{2^{*}}{2}} \vert w_{\varepsilon} \vert ^{\frac{2^{*}}{2}} \biggr) \\ &\geq\lim_{\varepsilon\rightarrow0^{+}} \biggl( \int_{\mathbb{R}^{N}} \vert w_{\varepsilon} \vert ^{2^{*}}+ \frac{(\beta\int _{\mathbb{R} ^{N}} \vert u_{1} \vert ^{\frac{2^{*}}{2}} \vert w_{\varepsilon} \vert ^{\frac{2^{*}}{2}})^{2}}{ (t_{\varepsilon}^{2-2^{*}}-1)\int_{\mathbb{R}^{N}} \vert u_{1} \vert ^{2^{*}} +t_{\varepsilon}^{2-2^{*}}\int_{\mathbb{R}^{N}}f(u_{1})u_{1}-\int_{\mathbb {R}^{N}}\frac {f(t_{\varepsilon}u_{1})}{(t_{\varepsilon}u_{1})^{2^{*}-1}}u_{1}^{2^{*}}} \biggr)\\ &=S^{\frac{N}{2}}, \end{aligned} $$

which is a contradiction. So there exist $t_{1}$, $s_{1}$ independent of ε such that $1\leq t_{\varepsilon}\leq t_{1}$ and $s_{0}\leq s_{\varepsilon}\leq s_{1}$ for ε small. Then we have

$$\vert \beta \vert t_{\varepsilon}^{\frac{2^{*}}{2}}s_{\varepsilon}^{\frac{2^{*}}{2}} \int_{\mathbb{R}^{N}} \vert u_{1} \vert ^{\frac {2^{*}}{2}}\vert w_{\varepsilon}\vert ^{\frac{2^{*}}{2}} \leq \vert \beta \vert s_{0}^{\frac{2^{*}}{2}-2}\Bigl(t_{1}\max_{x\in\mathbb{R} ^{N}}u_{1}(x) \Bigr)^{\frac{2^{*}}{2}}s_{\varepsilon}^{2} \int_{\mathbb {R}^{N}}\vert w_{\varepsilon}\vert ^{\frac{2^{*}}{2}}\leq s_{\varepsilon}^{2}o\bigl(\varepsilon^{2}\bigr). $$

Therefore,

$$ \begin{aligned}[b] B&\leq I(t_{\varepsilon}u_{1},s_{\varepsilon}v_{\varepsilon}) \\ &\leq\max_{t>0} \biggl(\frac{t_{\varepsilon}^{2}}{2} \int_{\mathbb{R}^{N}}\bigl( \vert \nabla u_{1} \vert ^{2}+ \vert u_{1} \vert ^{2}\bigr)- \frac {t_{\varepsilon}^{2^{*}}}{2^{*}} \int_{\mathbb{R}^{N}} \vert u_{1} \vert ^{2^{*}}- \int_{\mathbb {R}^{N}}F(t_{\varepsilon}u_{1}) \biggr) \\ &\quad {} +\max_{s>0} \biggl[\frac{s^{2}}{2} \biggl( \int_{\mathbb{R}^{N}}\bigl( \vert \nabla w_{\varepsilon} \vert ^{2}+\vert w_{\varepsilon}\vert ^{2}\bigr)+o\bigl( \varepsilon^{2}\bigr) \biggr)-\frac{s^{2^{*}}}{2^{*}} \int_{\mathbb{R} ^{N}} \vert w_{\varepsilon} \vert ^{2^{*}} \biggr]- \int_{\mathbb{R}^{N}}G(s_{\varepsilon}w_{\varepsilon}) \\ &\leq I(u_{1},0)+\frac{1}{N}S^{\frac{N}{2}}+O\bigl(\varepsilon ^{2}\bigr)+O\bigl(\varepsilon^{N-2}\bigr)- \int_{\mathbb{R}^{N}}G(s_{\varepsilon}w_{\varepsilon}) \\ &< B_{1}+\frac{1}{N}S^{\frac{N}{2}}\quad \text{for } \varepsilon >0 \text{ small}. \end{aligned} $$

(3.4)

Similarly, we can also prove that $B< B_{2}+\frac{1}{N} S^{\frac{N}{2}}$. □

Lemma 3.3

Suppose that (F₁)-(F₃) hold and $\beta\in(-1,0)$, then there exists a bounded $(PS)_{B}$ sequence $\{(u_{n},v_{n})\}\subset M$ for I.

Proof

By Lemma 3.1 and Ekeland’s variational principle (see [22]), there exists a minimizing sequence $\{(u_{n},v_{n})\}\subset M$ satisfying that

$$\begin{aligned}& I(u_{n},v_{n})\leq B+\frac{1}{n}, \end{aligned}$$

(3.5)

$$\begin{aligned}& I(u,v)\geq I(u_{n},v_{n})- \frac{1}{n} \bigl\Vert (u_{n},v_{n})-(u,v) \bigr\Vert _{H},\quad \forall(u,v)\in M. \end{aligned}$$

(3.6)

Lemma 3.1 shows that $\{(u_{n},v_{n})\}$ is uniformly bounded in $H_{r}$. For any $(\varphi,\phi)\in H_{r}$ with $\Vert \varphi \Vert$, $\Vert \phi \Vert \leq1$ and each $n\in\mathbb{N}$, define $h_{n}$, $j_{n}:\mathbb{R}^{3}\rightarrow \mathbb{R}$ by

$$\begin{aligned}& \begin{aligned} h_{n}(t,s,l)&= \int_{\mathbb{R}^{N}}\bigl(\bigl\vert \nabla (u_{n}+t \varphi+su_{n}) \bigr\vert ^{2}+ \vert u_{n}+t\varphi+su_{n}\vert ^{2}\bigr)- \int_{\mathbb{R} ^{N}} \vert u_{n}+t\varphi+su_{n} \vert ^{2^{*}} \\ &\quad {}-\beta \int_{\mathbb{R}^{N}} \vert u_{n}+t\varphi+su_{n} \vert ^{\frac {2^{*}}{2}} \vert v_{n}+t\phi+lv_{n} \vert ^{\frac{2^{*}}{2}}\\ &\quad {}- \int _{\mathbb{R} ^{N}}f(u_{n}+t\varphi+su_{n}) (u_{n}+t\varphi+su_{n}), \end{aligned} \\& \begin{aligned} j_{n}(t,s,l)&= \int_{\mathbb{R}^{N}}\bigl(\bigl\vert \nabla (v_{n}+t \phi+lv_{n}) \bigr\vert ^{2}+ \vert v_{n}+t \phi+lv_{n}\vert ^{2}\bigr)- \int_{\mathbb {R}^{N}}\vert v_{n}+t\phi +lv_{n}\vert ^{2^{*}} \\ &\quad {}-\beta \int_{\mathbb{R}^{N}} \vert u_{n}+t\varphi+su_{n} \vert ^{\frac {2^{*}}{2}} \vert v_{n}+t\phi+lv_{n} \vert ^{\frac{2^{*}}{2}}\\ &\quad {}- \int _{\mathbb{R} ^{N}}g(v_{n}+t\phi+lv_{n}) (v_{n}+t\phi+lv_{n}). \end{aligned} \end{aligned}$$

Let $\mathbf{0}=(0,0,0)$. Then $h_{n},j_{n}\in C^{1}(\mathbb{R}^{3},\mathbb {R})$ satisfy that $h_{n}(\mathbf{0})=j_{n}(\mathbf{0})=0$ and

$$\begin{gathered} \frac{\partial h_{n}}{\partial s}(\mathbf{0})=\bigl(2-2^{*}\bigr) \int_{\mathbb{R} ^{N}} \vert u_{n} \vert ^{2^{*}}+ \frac{4-2^{*}}{2}\beta \int_{\mathbb{R}^{N}} \vert u_{n} \vert ^{\frac {2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}}+ \int_{\mathbb{R}^{N}}\bigl[f(u_{n})u_{n}-f'(u_{n})u_{n}^{2} \bigr], \\ \frac{\partial h_{n}}{\partial l}(\mathbf{0})=\frac{\partial j_{n}}{\partial s}(\mathbf{0})=- \frac{2^{*}}{2}\beta \int_{\mathbb{R} ^{N}} \vert u_{n} \vert ^{\frac{2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}}, \\ \frac{\partial j_{n}}{\partial l}(\mathbf{0})=\bigl(2-2^{*}\bigr) \int_{\mathbb{R}^{N}}\vert v_{n} \vert ^{2^{*}}+ \frac{4-2^{*}}{2}\beta \int_{\mathbb{R}^{N}} \vert u_{n}\vert ^{\frac {2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}}+ \int_{\mathbb{R}^{N}}\bigl[g(v_{n})v_{n}-g'(v_{n})v_{n}^{2} \bigr]. \end{gathered} $$

Define the matrix

$$A_{n}:= \left ( \begin{matrix} {\frac{\partial h_{n}}{\partial s}(\mathbf{0})} & {\frac{\partial h_{n}}{\partial l}(\mathbf{0})}\\ {\frac{\partial j_{n}}{\partial s}(\mathbf{0})}& {\frac {\partial j_{n}}{\partial l}(\mathbf{0})} \end{matrix} \right ). $$

We see that (F₂), (F₃) show that $f(u_{n})u_{n}-f'(u_{n})u_{n}^{2}<0$ and $g(v_{n})v_{n}-g'(v_{n})v_{n}^{2}<0$. Then, by $-1<\beta<0$, (3.1) and Lemma 3.2, we have

$$\begin{aligned} \operatorname{det}A_{n}&\geq\bigl(2^{*}-2 \bigr)^{2} \int_{\mathbb{R} ^{N}} \vert u_{n} \vert ^{2^{*}} \int_{\mathbb{R}^{N}} \vert v_{n} \vert ^{2^{*}}\\ &\quad {}+ \biggl[\biggl(\frac {4-2^{*}}{2}\biggr)^{2}-\biggl(\frac{2^{*}}{2} \biggr)^{2} \biggr]\beta^{2} \biggl( \int_{\mathbb{R} ^{N}} \vert u_{n} \vert ^{\frac{2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}} \biggr)^{2} \\ &\quad {} +\bigl(2-2^{*}\bigr)\frac{4-2^{*}}{2}\beta \int_{\mathbb{R} ^{N}} \vert u_{n} \vert ^{\frac{2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}} \biggl( \int_{\mathbb{R}^{N}} \vert u_{n} \vert ^{2^{*}} + \int_{\mathbb{R}^{N}} \vert v_{n} \vert ^{2^{*}} \biggr) \\ &\geq\bigl(2^{*}-2\bigr)^{2} \int_{\mathbb {R}^{N}} \vert u_{n} \vert ^{2^{*}} \int _{\mathbb{R}^{N}} \vert v_{n} \vert ^{2^{*}}- \bigl(2^{*}-2\bigr)^{2}\beta^{2} \biggl( \int_{\mathbb {R}^{N}} \vert u_{n} \vert ^{\frac {2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}} \biggr)^{2} \\ &\geq\bigl(2^{*}-2\bigr)^{2}\bigl(1-\beta^{2}\bigr) \int_{\mathbb {R}^{N}} \vert u_{n} \vert ^{2^{*}} \int _{\mathbb{R}^{N}} \vert v_{n} \vert ^{2^{*}}\geq \bigl(2^{*}-2\bigr)^{2}\bigl(1-\beta^{2}\bigr)C_{0}^{2}>0. \end{aligned}$$

By the implicit function theorem, there exist $\delta_{n}>0$ and functions $s_{n}(t)$, $l_{n}(t)\in C^{1}(-\delta_{n},\delta_{n})$ such that $s_{n}(0)=l_{n}(0)=0$,

$$h_{n}\bigl(t,s_{n}(t),l_{n}(t)\bigr)=0, \qquad j_{n}\bigl(t,s_{n}(t),l_{n}(t)\bigr)=0,\quad \forall t\in(-\delta_{n},\delta_{n}) $$

and

$$ \textstyle\begin{cases} s_{n}'(0)=\frac{1}{\operatorname{det}A_{n}} (\frac{\partial j_{n}}{\partial t}(\mathbf{0})\frac{\partial h_{n}}{\partial l}(\mathbf {0})-\frac{\partial j_{n}}{\partial l}(\mathbf{0})\frac{\partial h_{n}}{\partial t}(\mathbf{0}) )\\ l_{n}'(0)=\frac{1}{\operatorname{det}A_{n}} (\frac{\partial j_{n}}{\partial s}(\mathbf{0})\frac{\partial h_{n}}{\partial t}(\mathbf {0})-\frac{\partial j_{n}}{\partial t}(\mathbf{0})\frac{\partial h_{n}}{\partial s}(\mathbf{0}) ). \end{cases} $$

Since $\{(u_{n},v_{n})\}$ is uniformly bounded in H, we see that

$$ \bigl\vert s_{n}'(0) \bigr\vert , \bigl\vert l_{n}'(0) \bigr\vert \leq C, $$

(3.7)

where $C>0$ is independent of n. Denote

$$\varphi_{n,t}:= u_{n}+t\varphi+s_{n}(t)u_{n}, \qquad \phi_{n,t}:= v_{n}+t\phi +l_{n}(t)v_{n}, $$

then $(\varphi_{n,t},\phi_{n,t})\in M$ for $\forall t\in(-\delta _{n},\delta_{n})$. It follows from (3.6) that

$$ I(\varphi_{n,t},\phi_{n,t})-I(u_{n},v_{n}) \geq-\frac{1}{n}\bigl\Vert \bigl(t\varphi +s_{n}(t)u_{n},t \phi+l_{n}(t)v_{n}\bigr)\bigr\Vert _{H}. $$

(3.8)

By $(u_{n},v_{n})\in M$ and the Taylor expansion, we have

$$ \begin{aligned}[b] I(\varphi_{n,t}, \phi_{n,t})-I(u_{n},v_{n})&=\bigl\langle I'(u_{n},v_{n}),\bigl(t\varphi +s_{n}(t)u_{n},t\phi+l_{n}(t)v_{n}\bigr) \bigr\rangle +r(n,t) \\ &=t\bigl\langle I'(u_{n},v_{n}),(\varphi,\phi) \bigr\rangle +r(n,t), \end{aligned} $$

(3.9)

where $r(n,t)=o( \Vert (t\varphi+s_{n}(t)u_{n},t\phi+l_{n}(t)v_{n}) \Vert _{H})$ as $t\rightarrow0$. By (3.7), we see that

$$ \limsup_{t\rightarrow0}\biggl\Vert \biggl(\varphi+ \frac {s_{n}(t)}{t}u_{n}, \phi+\frac{l_{n}(t)}{t}v_{n} \biggr) \biggr\Vert _{H}\leq C, $$

(3.10)

where C is independent of n. Hence $r(n,t)=o(t)$. By (3.8)-(3.10) and letting $t\rightarrow0$, we have

$$\bigl\vert \bigl\langle I'(u_{n},v_{n}),( \varphi,\phi)\bigr\rangle \bigr\vert \leq\frac{C}{n}, $$

where C is independent of n. Hence $I'(u_{n},v_{n})\rightarrow0$, i.e. $\{(u_{n},v_{n})\}$ is a bounded $(PS)_{B}$ sequence for I. □

Proof of Theorem 1.1 with $\beta\in(-1,0)$

By Lemmas 3.3 and 3.1, there exists a bounded $(PS)_{B}$ sequence $\{(u_{n},v_{n})\}\subset M$ satisfying that

$$ \int_{\mathbb{R}^{N}} \vert u_{n} \vert ^{2^{*}}, \int_{\mathbb{R}^{N}} \vert v_{n} \vert ^{2^{*}}\geq C_{0}, $$

(3.11)

where $C_{0}>0$ is given in Lemma 3.1. Up to a subsequence, there exists $(u,v)\in H_{r}$ such that $(u_{n},v_{n})\rightharpoonup(u,v)$ in $H_{r}$. Then $I'(u,v)=0$. Moreover, by (F₁), (F₃) we see that

$$ \textstyle\begin{cases} \int_{\mathbb{R}^{N}}F(u_{n})\rightarrow\int_{\mathbb{R}^{N}}F(u), \int_{\mathbb{R} ^{N}}f(u_{n})u_{n}\rightarrow\int_{\mathbb{R}^{N}}f(u)u,\\ \int_{\mathbb{R}^{N}}G(v_{n})\rightarrow\int_{\mathbb{R}^{N}}G(v), \int_{\mathbb{R} ^{N}}g(v_{n})v_{n}\rightarrow\int_{\mathbb{R}^{N}}g(v)v. \end{cases} $$

(3.12)

If $u\equiv0$ and $v\equiv0$, then

$$ \textstyle\begin{cases} \int_{\mathbb{R}^{N}} \vert \nabla u_{n} \vert ^{2}\leq\int_{\mathbb {R}^{N}} \vert u_{n} \vert ^{2^{*}}+\beta\int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{\frac{2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}}+o_{n}(1),\\ \int_{\mathbb{R}^{N}} \vert \nabla v_{n} \vert ^{2}\leq\int_{\mathbb {R}^{N}} \vert v_{n} \vert ^{2^{*}}+\beta\int _{\mathbb{R}^{N}} \vert u_{n} \vert ^{\frac{2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}}+o_{n}(1), \end{cases} $$

(3.13)

where $o_{n}(1)\rightarrow0$ as $n\rightarrow+\infty$. Similarly to the proof of (3.2) in Lemma 3.2, there exist $t_{n},s_{n}>0$ such that $(t_{n}u_{n},s_{n}v_{n})\in P$, i.e.

$$ \textstyle\begin{cases} t_{n}^{2}\int_{\mathbb{R}^{N}} \vert \nabla u_{n} \vert ^{2}=t_{n}^{2^{*}}\int_{\mathbb{R} ^{N}} \vert u_{n} \vert ^{2^{*}}+t_{n}^{\frac{2^{*}}{2}}s_{n}^{\frac{2^{*}}{2}}\beta\int_{\mathbb{R} ^{N}} \vert u_{n} \vert ^{\frac{2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}},\\ s_{n}^{2}\int_{\mathbb{R}^{N}} \vert \nabla v_{n} \vert ^{2}=s_{n}^{2^{*}}\int_{\mathbb{R} ^{N}} \vert v_{n} \vert ^{2^{*}}+t_{n}^{\frac{2^{*}}{2}}s_{n}^{\frac{2^{*}}{2}}\beta\int_{\mathbb{R} ^{N}} \vert u_{n} \vert ^{\frac{2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}}. \end{cases} $$

(3.14)

So $J(t_{n}u_{n},s_{n}v_{n})=\frac{1}{N}\int_{\mathbb{R}^{N}}(\vert \nabla (t_{n}u_{n}) \vert ^{2}+ \vert \nabla(s_{n}v_{n})\vert ^{2})\geq A$. Set

$$\begin{gathered} c_{1}:=\lim_{n\rightarrow+\infty} \int_{\mathbb{R} ^{N}} \vert \nabla u_{n} \vert ^{2},\qquad c_{2}:=\lim_{n\rightarrow+\infty } \int_{\mathbb{R}^{N}} \vert \nabla v_{n} \vert ^{2}, \\ d_{1}:=\lim_{n\rightarrow+\infty} \int_{\mathbb{R} ^{N}} \vert u_{n} \vert ^{2^{*}}, \qquad d_{2}:=\lim_{n\rightarrow+\infty } \int_{\mathbb{R}^{N}} \vert v_{n} \vert ^{2^{*}}, \qquad e:=\lim_{n\rightarrow+\infty} \vert \beta \vert \int_{\mathbb {R}^{N}} \vert u_{n} \vert ^{\frac {2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}}. \end{gathered} $$

By (3.13), we see that $c_{1}+e\leq d_{1}$ and $c_{2}+e\leq d_{2}$. By (3.1) we have $e^{2}< d_{1}d_{2}$. If $t_{n}\rightarrow+\infty$ as $n\rightarrow+\infty$, then the first equation of (3.14) implies that $s_{n}\rightarrow+\infty$. Hence, by the second equation of (3.14), we show that

$$ \begin{aligned}[b] 0&=\lim_{n\rightarrow+\infty}\frac{\int_{\mathbb {R}^{N}}\vert \nabla v_{n}\vert ^{2}}{ s_{n}^{2^{*}-2}}=\lim _{n\rightarrow+\infty} \biggl( \int _{\mathbb{R}^{N}} \vert v_{n} \vert ^{2^{*}}- \frac{t_{n}^{2^{*}}(\beta\int_{\mathbb {R}^{N}} \vert u_{n} \vert ^{\frac {2^{*}}{2}} \vert v_{n} \vert ^{\frac{2^{*}}{2}})^{2}}{t_{n}^{2^{*}}\int_{\mathbb{R} ^{N}} \vert u_{n} \vert ^{2^{*}}-t_{n}^{2}\int_{\mathbb{R}^{N}} \vert \nabla u_{n} \vert ^{2}} \biggr)\\ &=\frac {d_{1}d_{2}-e^{2}}{d_{1}}>0, \end{aligned} $$

(3.15)

which is a contradiction. So we may assume that $t_{n}\rightarrow t_{\infty}\geq0$ and $s_{n}\rightarrow s_{\infty}\geq0$.

If $e=0$, then (3.13) and (3.14) imply that $t_{\infty}, s_{\infty}\leq1$. If $e>0$, we assume that $t_{\infty}>1$. Then, by the first equation of (3.14), we have $s_{\infty}>1$. Similarly to the proof of (3.15), we see that

$$c_{2}>s_{\infty}^{2-2^{*}}c_{2}\geq d_{2}-\frac{e^{2}}{d_{1}-c_{1}}\geq d_{2}-e=c_{2}, $$

which is a contradiction. Therefore, $t_{\infty}\leq1$. Similarly, $s_{\infty}\leq1$. So we have

$$B=\lim_{n\rightarrow+\infty}\frac{1}{N} \bigl\Vert (u_{n},v_{n}) \bigr\Vert _{H}^{2} \geq \frac{1}{N}\lim_{n\rightarrow+\infty} \int_{\mathbb{R} ^{N}}\bigl( \bigl\vert \nabla(t_{n}u_{n}) \bigr\vert ^{2}+ \bigl\vert \nabla(s_{n}v_{n}) \bigr\vert ^{2}\bigr)\geq A. $$

By Lemma 3.2 and Section 1, we have $\frac{2}{N}S^{\frac{N}{2}}=A\leq B< B_{1}+\frac{1}{N}S^{\frac{N}{2}}$, which implies that $\frac{1}{N}S^{\frac{N}{2}}\leq B_{1}$. It contradicts (2.4). Therefore the case $u,v\equiv0$ does not occur.

If $u\not\equiv0$ and $v\equiv0$, then u is a nontrivial solution of $-\Delta u+u= \vert u \vert ^{2^{*}-2}u+f(u)$ in $\mathbb{R}^{N}$. Then $I(u,0)\geq B_{1}$. By $(u_{n},v_{n})\in M$ and $\beta<0$, we have

$$\begin{aligned} \Vert v_{n} \Vert ^{2}&= \int_{\mathbb {R}^{N}} \vert v_{n} \vert ^{2^{*}}+\beta \int_{\mathbb{R}^{N}} \vert u_{n} \vert ^{\frac{2^{*}}{2}} \vert v_{n} \vert ^{\frac {2^{*}}{2}}+o_{n}(1)\leq S^{-\frac{2^{*}}{2}} \Vert v_{n} \Vert ^{2^{*}}+o_{n}(1), \end{aligned} $$

where $o_{n}(1)\rightarrow0$ as $n\rightarrow+\infty$. By (3.11) we have $\Vert v_{n} \Vert ^{2}\geq S^{\frac{N}{2}}$. So by (3.12) we see that

$$\begin{aligned} B&=\lim_{n\rightarrow+\infty} \biggl(I(u_{n},v_{n})- \frac{1}{2^{*}} \bigl\langle I'(u_{n},v_{n}),(u_{n},v_{n}) \bigr\rangle \biggr) \\ &=\lim_{n\rightarrow+\infty} \biggl\{ \frac{1}{N}\Vert v_{n} \Vert ^{2}+ \int_{\mathbb{R}^{N}} \biggl(\frac {1}{2^{*}}g(v_{n})v_{n} \biggr)-G(v_{n}) \biggr)\\ &\quad {}+\frac{1}{N} \Vert u_{n} \Vert ^{2}+ \int_{\mathbb{R}^{N}} \biggl(\frac{1}{2^{*}}f(u_{n})u_{n}-F(u_{n}) \biggr) \biggr\} \\ &\geq\frac{1}{N}S^{\frac{N}{2}}+\frac{1}{N} \Vert u \Vert ^{2}+ \int_{\mathbb{R}^{N}} \biggl(\frac{1}{2^{*}}f(u)u-F(u) \biggr) \\ &=\frac{1}{N}S^{\frac{N}{2}}+I(u,0)\geq\frac{1}{N}S^{\frac{N}{2}}+B_{1}, \end{aligned} $$

which is a contradiction to Lemma 3.2. So $u\not\equiv0$ and $v\equiv0$ do not occur. Similarly, $u\equiv0$ and $v\not\equiv0$ do not occur. Therefore, $u\not\equiv0$ and $v\not\equiv0$, i.e. $(u,v)$ is a nontrivial solution to (1.1). Similarly to the proof in Section 2, we get that $(u,v)$ is a radial and positive least energy solution of (1.1) with $I(u,v)=B$. □