1 Introduction

In this paper, we consider the following coupled nonlinear equations of Choquard type:

$$ \textstyle\begin{cases} -\Delta u+\lambda _{1}(x)u=\mu _{1}(I_{\alpha }* \vert u \vert ^{ \frac{N+\alpha }{N}}) \vert u \vert ^{\frac{\alpha }{N}-1}u+\beta (I_{\alpha }* \vert v \vert ^{ \frac{N+\alpha }{N}}) \vert u \vert ^{\frac{\alpha }{N}-1}u,\quad x\in {\mathbb{R}}^{N}, \\ -\Delta v+\lambda _{2}(x)v=\mu _{2}(I_{\alpha }* \vert v \vert ^{ \frac{N+\alpha }{N}}) \vert v \vert ^{\frac{\alpha }{N}-1}v+\beta (I_{\alpha }* \vert u \vert ^{ \frac{N+\alpha }{N}}) \vert v \vert ^{\frac{\alpha }{N}-1}v,\quad x\in {\mathbb{R}}^{N}, \\ u, v\in H^{1}({\mathbb{R}}^{N}), \end{cases} $$
(1.1)

where \(N\ge 3\), \(\alpha \in (0,N)\), \(\mu _{1}, \mu _{2}, \beta >0\), \(\frac{N+\alpha }{N}\) is the lower critical exponent due to the Hardy–Littlewood–Sobolev inequality (see [9, Theorem 3.1]), \(I_{\alpha }:{\mathbb{R}}^{N}\setminus \{0\}\mapsto \mathbb{R}\) defined by

$$ I_{\alpha }= \frac{\Gamma (\frac{N-\alpha }{2})}{2^{\alpha }\pi ^{\frac{N}{2}}\Gamma (\frac{\alpha }{2}) \vert x \vert ^{N-\alpha }} $$

is the Riesz potential, and \(\lambda _{1}(x)\) and \(\lambda _{2}(x)\) are nonnegative functions. Elliptic equations of this type have wide application in physical problems, such as in Hartree–Fock theory [8, 10, 12] and in nonlinear optics [13, 14]. The readers can refer to [2, 18, 19] for more physical backgrounds.

Mathematically, Choquard type equations have received considerable attention in the past few years, see [1, 3–5, 7, 8, 11, 15–17] and the reference therein for scale equations. There are also some results concerned with solutions of a nonlinearly coupled Choquard system. In [21], Wang and Shi proved the existence of positive solutions of

$$ \textstyle\begin{cases} -\Delta u+\lambda _{1}u=\mu _{1}(I_{\alpha }* \vert u \vert ^{2})u+\beta (I_{ \alpha }* \vert v \vert ^{2})u,\quad x\in {\mathbb{R}}^{N}, \\ -\Delta v+\lambda _{2}v=\mu _{2}(I_{\alpha }* \vert v \vert ^{2})v+\beta (I_{ \alpha }* \vert u \vert ^{2})v,\quad x\in {\mathbb{R}}^{N}, \\ u, v\in H^{1}({\mathbb{R}}^{N}), \end{cases} $$
(1.2)

for \(\lambda _{1}, \lambda _{2}>0\) and \(\beta \in (-\infty ,\chi _{0})\cup (\min \{\lambda ^{2}\mu ,\lambda ^{ \frac{1}{2}}\nu \},+\infty )\), where \(\lambda =\lambda _{2}/\lambda _{1}\) and \(\chi _{0}>0\) depends on \(\mu _{1}\), \(\mu _{2}\), λ. Particularly, when \(\lambda _{1}=\lambda _{2}>0\), they showed that system (1.2) has a positive ground state \((\sqrt{k}_{0}w_{0},\sqrt{l}_{0} w_{0})\), where \((k_{0},l_{0})\) is the solution of

$$ \textstyle\begin{cases} \mu _{1} k+\beta l=1, \\ \mu _{2} l+\beta k=1,\end{cases} $$
(1.3)

and \(w_{0}\) is a positive ground state of

$$ -\Delta u+\lambda _{1}u=\bigl(I_{\alpha }* \vert u \vert ^{2}\bigr)u,\quad x\in {\mathbb{R}}^{N}, u\in H^{1}\bigl({\mathbb{R}}^{N}\bigr). $$
(1.4)

In [22], Wang and Yang established the existence and nonexistence of normalized solutions of system (1.2) with trapping potentials. In [20], Wang obtained the multiplicity of nontrivial solutions of a nonlinearly coupled Choquard system with general subcritical exponents and perturbations.

For a Choquard system with upper critical exponents, You, Wang, and Zhao [25, 26] derived the existence of a positive ground state of the following system:

$$ \textstyle\begin{cases} -\Delta u+\lambda _{1}u=\mu _{1}(I_{\alpha }* \vert u \vert ^{{ \frac{N+\alpha }{N-2}}})u^{\frac{\alpha +2}{N-2}}+\beta (I_{\alpha }* \vert v \vert ^{ \frac{N+\alpha }{N-2}})u^{\frac{\alpha +2}{N-2}},\quad x\in \Omega , \\ -\Delta v+\lambda _{2}v=\mu _{2}(I_{\alpha }* \vert v \vert ^{{ \frac{N+\alpha }{N-2}}})v^{\frac{\alpha +2}{N-2}}+\beta (I_{\alpha }* \vert u \vert ^{ \frac{N+\alpha }{N-2}})v^{\frac{\alpha +2}{N-2}},\quad x\in \Omega , \\ u, v\in H_{0}^{1}(\Omega ), \end{cases} $$
(1.5)

where \(N\ge 5\), Ω is a bounded smooth domain in \({\mathbb{R}}^{N}\), \(-\lambda _{1}(\Omega )<\lambda _{1}\), \(\lambda _{2}<0\), and \(\lambda _{1}(\Omega )\) represents the first eigenvalue of −Δ on Ω with the Dirichlet boundary condition. More precisely, they obtained that system (1.5) has a positive ground state if

$$ \textstyle\begin{cases} \beta \in (-\bar{\beta },0)\cup (0,\min \{\mu _{1},\mu _{2}\})\cup ( \max \{\mu _{1},\mu _{2}\},+\infty )& \text{for } \alpha =N-4, \\ \beta \in (-\infty ,0)\cup (\frac{\alpha +2}{N-2}\max \{\mu _{1}, \mu _{2}\},+\infty )& \text{for } \alpha \in (0,N-4). \end{cases} $$

For the special case \(-\lambda _{1}(\Omega )<\lambda _{1}=\lambda _{2}<0\), they proved that system (1.5) has a positive ground state \((\sqrt{\bar{k}}w^{*},\sqrt{ \bar{l}} w^{*})\) if

$$ \textstyle\begin{cases} \beta \in (0,\min \{\mu _{1},\mu _{2}\})\cup (\max \{\mu _{1},\mu _{2} \},+\infty )& \text{for }\alpha =N-4, \\ \beta \in (\frac{\alpha +2}{N-2}\max \{\mu _{1},\mu _{2}\},+ \infty )& \text{for }\alpha \in (0,N-4), \end{cases} $$

where \(w^{*}\) is a positive ground state of

$$ -\Delta u+\lambda _{1}u=\bigl(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N-2}}\bigr) u ^{ \frac{\alpha +2}{N-2}},\quad u\in H_{0}^{1}(\Omega ), $$
(1.6)

and k̄, l̄ is a solution of

$$ \textstyle\begin{cases} \mu _{1} k^{\frac{\alpha +2}{N-2}}+\beta k^{\frac{\alpha +4-N}{2(N-2)}}l^{ \frac{N+\alpha }{2(N-2)}}=1, \\ \mu _{2} l^{\frac{\alpha +2}{N-2}}+\beta k^{\frac{N+\alpha }{2(N-2)}}l^{ \frac{\alpha +4-N}{2(N-2)}}=1, \\ k, l>0,\end{cases} $$
(1.7)

satisfying

$$ \bar{k}=\min \bigl\{ k | (k,l) \text{ solves (1.7)}\bigr\} . $$

In the current paper, we study the nonlinearly coupled system (1.1) with lower critical exponents. Since system (1.1) with positive constant potentials has no nontrivial solution in \(H:=H^{1}({\mathbb{R}}^{N})\times H ^{1}({\mathbb{R}}^{N})\) by the Pohozaev identity, we assume that \(\lambda _{1}\), \(\lambda _{2}\) are functions dependent on \(x\in {\mathbb{R}}^{N}\). We aim to prove the existence of positive ground states of system (1.1). Furthermore, for the case \(\lambda _{1}(x)=\lambda _{2}(x):=\lambda (x)\), we will introduce an approach which is different with [21, 25, 26] to prove that system (1.1) has a positive ground state of the form \((kw,lw)\), where w is a positive ground state of

$$ -\Delta u+\lambda (x)u=\bigl(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u \vert ^{ \frac{\alpha }{N}-1}u,\quad x\in { \mathbb{R}}^{N}, u\in H^{1}\bigl({ \mathbb{R}}^{N} \bigr). $$
(1.8)

For this purpose, we assume that

\((C1)\):

\(\lambda _{i}(x)\ge 0\) for all \(x\in { \mathbb{R}}^{N}\), \(\lambda _{i}(x)\in L^{\infty }({\mathbb{R}}^{N})\) and \(\lim_{|x|\to \infty }\lambda _{i}(x)=1\), \(i=1,2\);

\((C2)\):

\(\liminf_{|x|\to \infty }(1-\lambda _{i}(x))|x|^{2} \ge \frac{N^{2}(N-2)}{4(N+1)}\), \(i=1,2\).

Note that under assumptions \((C1)\) and \((C2)\), the scale equation

$$ -\Delta u+\lambda _{i}(x)u=\mu _{i} \bigl(I_{\alpha }* \vert u \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert u \vert ^{\frac{\alpha }{N}-1}u,\quad x\in {\mathbb{R}}^{N}, u\in H^{1} \bigl({\mathbb{R}}^{N}\bigr), i=1,2, $$
(1.9)

has a ground state \(w_{i}\), \(i=1,2\) (see [16, Theorem 3,Theorem 6]). Moreover, we may assume that \(w_{i}\) is positive since \(|w_{i}|\) is also a ground state of (1.9). Clearly, system (1.1) has a trivial solution \((0,0)\) and two semi-trivial solutions \((w_{1},0)\) and \((0,w_{2})\) for all \(\beta \in \mathbb{R}\). Here we deal with the nontrivial solution, that is, a solution \((u,v)\) of (1.1) with \(u\not \equiv 0\) and \(v\not \equiv 0\). Denote \(\int _{{\mathbb{R}}^{N}} \cdot dx\) by ∫⋅ for simplicity, and define the functional \(I: H\mapsto {\mathbb{R}}\) corresponding to system (1.1) by

$$ \begin{aligned} I(u,v)&= \frac{1}{2} \int \vert \nabla u \vert ^{2}+\lambda _{1}(x)u^{2}+ \vert \nabla v \vert ^{2}+\lambda _{2}(x)v^{2} \\ &\quad{} - \frac{N}{2(N+\alpha )} \int \biggl(\mu _{1}\bigl(I_{ \alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u \vert ^{\frac{N+\alpha }{N}}+\mu _{2} \bigl(I_{ \alpha }* \vert v \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v \vert ^{\frac{N+\alpha }{N}}\\ &\quad {}+2\beta \bigl(I_{ \alpha }* \vert u \vert ^{\frac{N+\alpha }{N}} \bigr) \vert v \vert ^{\frac{N+\alpha }{N}}\biggr).\end{aligned} $$

Set

$$ \begin{aligned} \mathcal{M}&= \biggl\{ (u,v)\in H, u, v\not \equiv 0, \\ &\quad \int \vert \nabla u \vert ^{2}+\lambda _{1}(x)u^{2}= \int \mu _{1}\bigl(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u \vert ^{ \frac{N+\alpha }{N}}+\beta \bigl(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v \vert ^{ \frac{N+\alpha }{N}}, \\ &\quad \int \vert \nabla v \vert ^{2}+\lambda _{2}(x)v^{2}= \int \mu _{2}\bigl(I_{\alpha }* \vert v \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v \vert ^{ \frac{N+\alpha }{N}}+\beta \bigl(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v \vert ^{ \frac{N+\alpha }{N}} \biggr\} . \end{aligned} $$

It is obvious that if \((u,v)\) is a solution of system (1.1), then \((u,v)\in \mathcal{M}\). Define

$$ {\mathcal{B}}=\inf_{\mathcal{M}}I(u,v). $$

A solution \((u,v)\) of system (1.1) is called a positive solution if \(u>0\), \(v>0\) and a ground state if \(I(u,v)=\mathcal{B}\). We first show that \(\mathcal{B}\) is attained by some positive ground state of system (1.1) in the case when \(\lambda _{1}(x)=\lambda _{2}(x):=\lambda (x)\).

Theorem 1.1

Assume that \((C1)\) and \((C2)\) hold. If \(\lambda _{1}(x)=\lambda _{2}(x):=\lambda (x)\), then \((t_{m} s_{m}w, t_{m} w)\) is a positive ground state of system (1.1) for all \(\beta >0\), where w is a positive ground state of (1.8), \(t_{m}= (\mu _{2}+\beta s_{m}^{\frac{N+\alpha }{N}} )^{- \frac{N}{2\alpha }}\), and \(s_{m}>0\) is a minimum point of a function \(g(s):{\mathbb{R}}^{+}\mapsto {\mathbb{R}}\) defined by

$$ g(s)= \frac{1+s^{2}}{ (\mu _{2}+\mu _{1}s^{\frac{2(N+\alpha )}{N}}+2\beta s^{\frac{N+\alpha }{N}} )^{\frac{N}{N+\alpha }}}. $$
(1.10)

Remark 1.2

If we apply a method as in the proof of [25, Theorem 1.3] and [26, Theorem 1.3] to our case, we can prove that system (1.1) has a ground state of the form \((kw,lw)\) only if \(\beta \ge \frac{\alpha }{N}\max \{\mu _{1},\mu _{2}\}\). In the current paper, we use an alternative approach inspired by [24], which is based on studying the minimum point of \(g(s)\), and we show that system (1.1) possesses a ground state of this form for all \(\beta >0\).

Remark 1.3

The method we adopted in the proof of Theorem 1.1 is also valid for the upper critical system (1.5). As we mentioned previously, system (1.5) has a ground state of the form \((k w^{*},l w^{*})\) if \(N\ge 5\), \(-\lambda _{1}(\Omega )<\lambda _{1}=\lambda _{2}<0\), and

$$ \textstyle\begin{cases} \beta \in (\frac{\alpha +2}{N-2}\max \{\mu _{1},\mu _{2}\},+ \infty )& \text{for }\alpha \in (0,N-4), \\ \beta \in (0,\min \{\mu _{1},\mu _{2}\})\cup (\max \{\mu _{1}, \mu _{2}\},+\infty )& \text{for } \alpha =N-4, \end{cases} $$

(see [25, Theorem 1.3] and [26, Theorem 1.3]). However, we can prove that under the same assumptions on \(\lambda _{1}\), \(\lambda _{2}\), N, system (1.5) has a ground state in the same form if

$$ \textstyle\begin{cases} \beta \in (0,+\infty )& \text{for }\alpha \in (0,N-4), \\ \beta \in (\max \{\mu _{1},\mu _{2}\},+\infty )& \text{for } \alpha =N-4 \end{cases} $$

(see Theorem A.1 in Appendix). Although our approach can only deal with the case \(\beta > \max \{\mu _{1},\mu _{2}\}\) for \(\alpha =N-4\), in the case \(\alpha \in (0,N-4)\), the existence of a ground state of \(( k w^{*}, l w^{*})\) type is obtained for all \(\beta >0\).

Next, for any \(\lambda _{1}(x)\), \(\lambda _{2}(x)\) satisfying \((C1)\) and \((C2)\), we have the following result.

Theorem 1.4

Assume that \((C1)\) and \((C2)\) hold. Then system (1.1) has a positive ground state for all \(\beta >0\).

In the proof of Theorem 1.4, we need to give an accurate estimate of the least energy so as to overcome the lack of compactness and show that both components of the solution we obtained are nontrivial. For this purpose, some results of equation (1.9) will be used. Denote the functional associated with (1.9) by

$$ I_{i}(u)=\frac{1}{2} \int \vert \nabla u \vert ^{2}+\lambda _{i}(x)u^{2}- \frac{N}{2(N+\alpha )}\mu _{i} \int \bigl(I_{\alpha }* \vert u \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert u \vert ^{\frac{N+\alpha }{N}}, $$

and set

$$ \mathcal{N}_{i}=\bigl\{ u\in H^{1}\bigl({ \mathbb{R}}^{N}\bigr)\setminus \{0\}| \bigl\langle I_{i}'(u),u \bigr\rangle =0\bigr\} ,\qquad B_{i}=\inf_{{\mathcal{N}}_{i}}I_{i}(u), \quad i=1,2. $$

Then, from [16, Theorem 3,Theorem 6] and some calculation, we see that \(B_{i}\) is attained and

$$ B_{i}\le \frac{\alpha }{2(N+\alpha )}\mu _{i}^{-\frac{N}{\alpha }}S_{1}^{ \frac{N+\alpha }{\alpha }}, $$
(1.11)

where

$$ S_{1}=\inf_{(u,v)\in L^{2}({\mathbb{R}}^{N})\setminus \{0\}} \frac{ \int u^{2}}{ ( \int (I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}) \vert u \vert ^{\frac{N+\alpha }{N}} )^{\frac{N}{N+\alpha }}}. $$
(1.12)

By [9, Theorem 3.1], \(S_{1}\) has a unique minimizer

$$ U_{*}(x):=C \biggl(\frac{a}{a^{2}+ \vert x-b \vert ^{2}} \biggr)^{\frac{N}{2}}. $$
(1.13)

We should also study the minimizing problem

$$ \begin{aligned} S_{0}&=\inf_{{(u,v)\in L \atop u\not \equiv 0,v\not \equiv 0}} \biggl( \biggl(\int (u^{2}+v^{2})\biggr)\\ &\quad {}\Big/\biggl( \biggl( \int \mu _{1}\bigl(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u \vert ^{\frac{N+\alpha }{N}} +\mu _{2}\bigl(I_{\alpha }* \vert v \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v \vert ^{\frac{N+\alpha }{N}}\\ &\quad {}+2\beta \bigl(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v \vert ^{\frac{N+\alpha }{N}} \biggr)^{\frac{N}{N+\alpha }}\biggr)\biggr), \end{aligned} $$
(1.14)

where \(L= L^{2}({\mathbb{R}}^{N})\times L^{2}({\mathbb{R}}^{N})\). Problem (1.14) can be seen as an extension of the classical problem (1.12). By a similar approach as in the proof of Theorem 1.1, we obtain the following result.

Theorem 1.5

If \(\beta >0\), then \(S_{0}=g(s_{m})S_{1}\), and \((s_{m}U_{*},U_{*})\) is a solution of (1.14), where \(g(s)\) is defined in (1.10) and \(s_{m}\) is a minimum point of \(g(s)\). If \(\beta <0\), then

$$ S_{0}= \bigl(\mu _{1}^{-\frac{N}{\alpha }}+\mu _{2}^{-\frac{N}{\alpha }}\bigr)^{ \frac{\alpha }{N+\alpha }} S_{1} $$

and \(S_{0}\) is not attained.

Theorem 1.5 not only plays an important role in the proof of Theorem 1.4, but also extends the classical results of [9, Theorem 3.1].

2 Proof of Theorem 1.1

In order to prove Theorem 1.1, we study the minimizing problem

$$ \begin{aligned} A&=\inf_{{(u,v)\in H \atop u\not \equiv 0,v\not \equiv 0}} \biggl(\biggl( \int \vert \nabla u \vert ^{2}+\lambda (x)u^{2}+ \vert \nabla v \vert ^{2}+\lambda (x)v^{2}\biggr)\\ &\quad {}\Big/\biggl( \biggl( \int \mu _{1}\bigl(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u \vert ^{\frac{N+\alpha }{N}}+\mu _{2}\bigl(I_{\alpha }* \vert v \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v \vert ^{\frac{N+\alpha }{N}}\\ &\quad {}+2\beta \bigl(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v \vert ^{\frac{N+\alpha }{N}} \biggr)^{\frac{N}{N+\alpha }}\biggr)\biggr). \end{aligned} $$

Up to multiplication by a scalar, we know that a minimizer of A is a ground state of system (1.1) for \(\lambda _{1}(x)=\lambda _{2}(x):=\lambda (x)\). Set

$$ A_{1}=\inf_{u\in H^{1}({\mathbb{R}}^{N})\setminus \{0\}} \frac{ \int \vert \nabla u \vert ^{2}+\lambda (x)u^{2}}{ ( \int (I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}) \vert u \vert ^{\frac{N+\alpha }{N}} )^{\frac{N}{N+\alpha }}}. $$
(2.1)

Letting w be a solution of (1.8), we know that \(A_{1}\) is attained by w. By studying a function \(g:{\mathbb{R}}^{+}\mapsto {\mathbb{R}}\) defined by

$$ g(s)= \frac{1+s^{2}}{(\mu _{2}+\mu _{1}s^{\frac{2(N+\alpha )}{N}}+2\beta s^{\frac{N+\alpha }{N}})^{\frac{N}{N+\alpha }}}, $$

we are able to obtain the relationship between A and \(A_{1}\) and show that A is attained.

Lemma 2.1

If \(\beta >0\), then there is \(s_{m}>0\) such that \(g(s_{m})=\min_{s\ge 0}g(s)\).

Proof

By simple calculation, we have

$$ g'(s)= \frac{2s (\mu _{2}-\mu _{1}s^{\frac{2\alpha }{N}}-\beta s^{\frac{\alpha -N}{N}}+\beta s^{\frac{N+\alpha }{N}} )}{ (\mu _{2}+\mu _{1}s^{\frac{2(N+\alpha )}{N}}+2\beta s^{\frac{N+\alpha }{N}} )^{\frac{2N+\alpha }{N+\alpha }}}. $$

Let \(h(s)= \mu _{2}-\mu _{1}s^{\frac{2\alpha }{N}}-\beta s^{ \frac{\alpha -N}{N}}+\beta s^{\frac{N+\alpha }{N}}\). If \(\beta >0\), then \(h(s)\to -\infty \) as \(s\to 0\), and \(h(s)\to +\infty \) as \(s\to +\infty \). Thus, there exists \(s_{m}>0\) such that \(h(s_{m})=0\) and \(g(s_{m})=\min_{s\ge 0}g(s)\). □

Lemma 2.2

Assume that \((C1)\) and \((C2)\) hold. If \(\beta >0\), then \(A=g(s_{m})A_{1}\).

Proof

We follow a similar approach as in [6, Theorem 1.1] and [24, Lemma 2.1] to prove this Lemma. For any \(z\in H^{1}({\mathbb{R}}^{N})\setminus \{0\}\), we set \((u,v):=(s_{m}z,z)\). Then it follows that

$$ A\le \frac{(1+ s_{m}^{2}) \int \vert \nabla z \vert ^{2}+\lambda (x)z^{2}}{ (( \mu _{2}+\mu _{1}s_{m}^{\frac{2(N+\alpha )}{N}}+2\beta s_{m}^{\frac{N+\alpha }{N}}) \int (I_{\alpha }* \vert z \vert ^{\frac{N+\alpha }{N}})z^{\frac{N+\alpha }{N}} )^{\frac{N}{N+\alpha }}}, $$
(2.2)

which indicates

$$ A\le g(s_{m})A_{1}. $$
(2.3)

Let \((u_{n},v_{n})\in H\) be a minimizing sequence of A, and set \(\xi _{n}=\tau _{n}u_{n}\), where

$$ \tau _{n}= \biggl( \frac{ \int (I_{\alpha }* \vert v_{n} \vert ^{\frac{N+\alpha }{N}}) \vert v_{n} \vert ^{\frac{N+\alpha }{N}}}{ \int (I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}) \vert u_{n} \vert ^{\frac{N+\alpha }{N}}} \biggr)^{\frac{N}{2(N+\alpha )}}. $$

Then we have

$$ \int \bigl(I_{\alpha }* \vert \xi _{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert \xi _{n} \vert ^{ \frac{N+\alpha }{N}}= \int \bigl(I_{\alpha }* \vert v_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{ \frac{N+\alpha }{N}}. $$
(2.4)

From (2.4) and the property of the Riesz potential that \(I_{\alpha }=I_{\frac{\alpha }{2}}*I_{\frac{\alpha }{2}}\), we obtain

$$ \begin{aligned} &\int \bigl(I_{\alpha }* \vert \xi _{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{ \frac{N+\alpha }{N}}\\ &\quad = \int \bigl(I_{\frac{\alpha }{2}}* \vert \xi _{n} \vert ^{\frac{N+\alpha }{N}} \bigr) \bigl(I_{\frac{\alpha }{2}}* \vert v_{n} \vert ^{ \frac{N+\alpha }{N}} \bigr) \\ &\quad \le \biggl( \int \bigl\vert I_{\frac{\alpha }{2}}* \vert \xi _{n} \vert ^{ \frac{N+\alpha }{N}} \bigr\vert ^{2} \biggr)^{\frac{1}{2}} \biggl( \int \bigl\vert I_{ \frac{\alpha }{2}}* \vert v_{n} \vert ^{\frac{N+\alpha }{N}} \bigr\vert ^{2} \biggr)^{ \frac{1}{2}} \\ &\quad = \biggl( \int \bigl(I_{\alpha }* \vert \xi _{n} \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert \xi _{n} \vert ^{\frac{N+\alpha }{N}} \biggr)^{ \frac{1}{2}} \biggl( \int \bigl(I_{\alpha }* \vert v_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{ \frac{N+\alpha }{N}} \biggr)^{\frac{1}{2}} \\ &\quad = \int \bigl(I_{\alpha }* \vert v_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{ \frac{N+\alpha }{N}}.\end{aligned} $$
(2.5)

By (2.4) and (2.5), we have

$$ \begin{gathered} A+o(1) \\ \quad = \biggl( \int \vert \nabla u_{n} \vert ^{2}+\lambda (x)u_{n}^{2}+ \vert \nabla v_{n} \vert ^{2}+\lambda (x)v_{n}^{2}\biggr)\\ \qquad {}\Big/\biggl( \biggl( \int \mu _{1}\bigl(I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u_{n} \vert ^{\frac{N+\alpha }{N}}+\mu _{2}\bigl(I_{\alpha }* \vert v_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{\frac{N+\alpha }{N}}\\ \qquad {}+2\beta \bigl(I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{\frac{N+\alpha }{N}} \biggr)^{\frac{N}{N+\alpha }}\biggr) \\ \quad \ge \frac{\tau _{n}^{-2} \int \vert \nabla \xi _{n} \vert ^{2}+\lambda (x)\xi _{n}^{2}+\int \vert \nabla v_{n} \vert ^{2}+\lambda (x)v_{n}^{2}}{ ((\mu _{2}+\mu _{1}\tau _{n}^{-\frac{2(N+\alpha )}{N}}+2\beta \tau _{n}^{-\frac{N+\alpha }{N}}) \int (I_{\alpha }* \vert \xi _{n} \vert ^{\frac{N+\alpha }{N}}) \vert \xi _{n} \vert ^{\frac{N+\alpha }{N}} )^{\frac{N}{N+\alpha }}} \\ \quad = g\bigl(\tau _{n}^{-1}\bigr)A_{1}\ge g(s_{m}) A_{1}. \end{gathered} $$
(2.6)

Combining (2.3) with (2.6), we conclude that \(A=g(s_{m})A_{1}\). □

Proof of Theorem 1.1

From the proof of Lemma 2.1, we see that there exists \(s_{m}>0\) such that \(h(s_{m})=0\), that is,

$$ \mu _{2}-\mu _{1}s_{m}^{\frac{2\alpha }{N}}- \beta s_{m}^{ \frac{\alpha -N}{N}}+\beta s_{m}^{\frac{N+\alpha }{N}}=0. $$
(2.7)

From (2.7), we get

$$ \mu _{1}s_{m}^{\frac{2(N+\alpha )}{N}}+\beta s_{m}^{ \frac{N+\alpha }{N}}=s^{2} \bigl(\mu _{1}s_{m}^{\frac{2\alpha }{N}}+\beta s_{m}^{ \frac{\alpha -N}{N}} \bigr)=s^{2}\bigl(\mu _{2}+\beta s_{m}^{\frac{N+\alpha }{N}} \bigr). $$

Then it follows

$$ \mu _{2}+\mu _{1}s_{m}^{\frac{2(N+\alpha )}{N}}+2\beta s_{m}^{ \frac{N+\alpha }{N}}=\mu _{2}+\beta s_{m}^{\frac{N+\alpha }{N}}+ \bigl(\mu _{1}s_{m}^{ \frac{2(N+\alpha )}{N}}+\beta s_{m}^{\frac{N+\alpha }{N}} \bigr)=\bigl(1+s^{2}\bigr) \bigl( \mu _{2}+\beta s_{m}^{\frac{N+\alpha }{N}}\bigr), $$

which yields

$$ g(s_{m})= \frac{1+s_{m}^{2}}{(\mu _{2}+\mu _{1}s_{m}^{\frac{2(N+\alpha )}{N}}+2\beta s_{m}^{\frac{N+\alpha }{N}})^{\frac{N}{N+\alpha }}}= \frac{(1+s_{m}^{2})^{\frac{\alpha }{N+\alpha }}}{(\mu _{2}+\beta s_{m}^{\frac{N+\alpha }{N}})^{\frac{N}{N+\alpha }}}. $$
(2.8)

Let \(t_{m}=(1+\beta s_{m}^{\frac{N+\alpha }{N}})^{-\frac{N}{2\alpha }}\), then \(t_{m}(s_{m}w,w)\) is a positive solution of system (1.1). Moreover, by (2.7), (2.8), and Lemma 2.2, we have

$$ \begin{aligned} {\mathcal{B}}&\le I\bigl(t_{m}(s_{m}w,w) \bigr) = \frac{\alpha }{2(N+\alpha )}t_{m}^{2}\bigl(1+s_{m}^{2} \bigr) \int \vert \nabla w \vert ^{2}+ \lambda (x)w^{2} \\ & = \frac{\alpha }{2(N+\alpha )}\bigl(1+s_{m}^{2}\bigr) \bigl( \mu _{2}+ \beta s_{m}^{\frac{N+\alpha }{N}}\bigr)^{-\frac{N}{\alpha }}A_{1}^{ \frac{N+\alpha }{\alpha }} \\ & = \frac{\alpha }{2(N+\alpha )}\bigl(g(s_{m})A_{1} \bigr)^{ \frac{N+\alpha }{\alpha }} \\ & = \frac{\alpha }{2(N+\alpha )}A^{ \frac{N+\alpha }{\alpha }}. \end{aligned} $$

On the other hand, \(\forall (u,v)\in \mathcal{M}\), we have

$$ I(u,v)=\frac{\alpha }{2(N+\alpha )} \int \vert \nabla u \vert ^{2}+\lambda (x)u^{2}+ \vert \nabla v \vert ^{2}+\lambda (x)v^{2}\ge \frac{\alpha }{2(N+\alpha )}A^{ \frac{N+\alpha }{\alpha }}, $$

which indicates that \(\mathcal{B}\ge \frac{\alpha }{2(N+\alpha )}A^{ \frac{N+\alpha }{\alpha }}\). Thus, \(\mathcal{B}=\frac{\alpha }{2(N+\alpha )}A^{ \frac{N+\alpha }{\alpha }}=I(t_{m}s_{m}w,t_{m}w)\), that is, \((t_{m}s_{m}w,t_{m}w)\) is a positive ground state of system (1.1). □

3 Proof of Theorem 1.5

In this section, we prove Theorem 1.5, which is essential in the proof of Theorem 1.4. Recalling the definition of \(U_{*}\), we have the following lemma.

Lemma 3.1

If \(\beta >0\), then \(S_{0}=g(s_{m})S_{1}\), and \(S_{0}\) is attained by \((s_{m}U_{*},U_{*})\).

Proof

By a similar approach as that in Lemma 2.2, we see that \(S_{0}=g(s_{m})S_{1}\). Then the conclusion follows from

$$ \frac{(1+ s_{m}^{2}) \int \vert U_{*} \vert ^{2}}{ (( \mu _{2}+\mu _{1}s_{m}^{\frac{2(N+\alpha )}{N}}+2\beta s_{m}^{\frac{N+\alpha }{N}}) \int (I_{\alpha }* \vert U_{*} \vert ^{\frac{N+\alpha }{N}}) \vert U_{*} \vert ^{\frac{N+\alpha }{N}} )^{\frac{N}{N+\alpha }}}=g(s_{m})S_{1}. $$

 □

Lemma 3.2

If \(\beta <0\), then \(S_{0}= (\mu _{1}^{-\frac{N}{\alpha }}+\mu _{2}^{-\frac{N}{\alpha }})^{ \frac{\alpha }{N+\alpha }} S_{1}\), and \(S_{0}\) is not attained.

Proof

Denote \((u_{0},v_{y}):=(\mu _{1}^{-\frac{N}{2\alpha }}U_{*}(x),\mu _{2}^{- \frac{N}{2\alpha }}U_{*}(x+e_{1}y))\), where \(e=(1,0,\ldots,0)\in {\mathbb{R}}^{N}\). Then \(v_{y}^{\frac{N+\alpha }{N}}\rightharpoonup 0 \) in \(L^{\frac{2N}{N+\alpha }}({\mathbb{R}}^{N})\) as \(y\to +\infty \). Taking account of the fact that \(I_{\alpha }*|u_{0}|^{\frac{N+\alpha }{N}}\in L^{\frac{2N}{N-\alpha }}({ \mathbb{R}}^{N})\), we have

$$ \lim_{y\to +\infty } \int \bigl(I_{\alpha }* \vert u_{0} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{y} \vert ^{ \frac{N+\alpha }{N}}=0. $$

Then, for \(|y|\) sufficiently large,

$$ \begin{aligned} S_{0}&\le \biggl( \int u_{0}^{2}+v_{y}^{2}\biggr)\\ &\quad {}\Big/\biggl( \biggl( \int \mu _{1}\bigl(I_{\alpha }* \vert u_{0} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u_{0} \vert ^{\frac{N+\alpha }{N}}+\mu _{2}\bigl(I_{\alpha }* \vert v_{y} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{y} \vert ^{\frac{N+\alpha }{N}}\\ &\quad {}+2\beta \bigl(I_{\alpha }* \vert u_{0} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{y} \vert ^{\frac{N+\alpha }{N}} \biggr)^{\frac{N}{N+\alpha }}\biggr) \\ &=\frac{ (\mu _{1}^{-\frac{N}{\alpha }}+\mu _{2}^{-\frac{N}{\alpha }})\int U_{*}^{2}}{ ( (\mu _{1}^{-\frac{N}{\alpha }}+\mu _{2}^{-\frac{N}{\alpha }})\int (I_{\alpha }* \vert U_{*} \vert ^{\frac{N+\alpha }{N}}) \vert U_{*} \vert ^{\frac{N+\alpha }{N}}+o(1) )^{\frac{N}{N+\alpha }}}. \end{aligned} $$

By letting \(y\to +\infty \), we get

$$ S_{0}\le \bigl(\mu _{1}^{-\frac{N}{\alpha }}+\mu _{2}^{-\frac{N}{\alpha }}\bigr)^{ \frac{\alpha }{N+\alpha }} S_{1}. $$

On the other hand, since \(\beta <0\), we know that

$$ \begin{aligned} S_{0}&\ge \inf_{{(u,v)\in L \atop u\ne 0,v \ne 0}} \frac{ \int u^{2}+v^{2}}{ ( \int \mu _{1}(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}) \vert u \vert ^{\frac{N+\alpha }{N}}+\mu _{2}(I_{\alpha }* \vert v \vert ^{\frac{N+\alpha }{N}}) \vert v \vert ^{\frac{N+\alpha }{N}} )^{\frac{N}{N+\alpha }}} \\ &\ge \frac{ (\mu _{1}^{-\frac{N}{\alpha }}+\mu _{2}^{-\frac{N}{\alpha }}){\int U_{*}}^{2}}{ ( (\mu _{1}^{-\frac{N}{\alpha }}+\mu _{2}^{-\frac{N}{\alpha }})\int (I_{\alpha }* \vert {U_{*}} \vert ^{\frac{N+\alpha }{N}}) \vert {U_{*}} \vert ^{\frac{N+\alpha }{N}} )^{\frac{N}{N+\alpha }}} \\ &= \bigl(\mu _{1}^{-\frac{N}{\alpha }}+\mu _{2}^{-\frac{N}{\alpha }} \bigr)^{ \frac{\alpha }{N+\alpha }} S_{1}. \end{aligned} $$

Therefore,

$$ S_{0}= \bigl(\mu _{1}^{-\frac{N}{\alpha }}+\mu _{2}^{-\frac{N}{\alpha }}\bigr)^{ \frac{\alpha }{N+\alpha }} S_{1}. $$
(3.1)

If \(S_{0}\) is attained by \((u,v)\) with \(u\not \equiv 0\), \(v\not \equiv 0\), then

$$ \begin{aligned} S_{0}&=\frac{ \int u^{2}+v^{2}}{ ( \int \mu _{1}(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}) \vert u \vert ^{\frac{N+\alpha }{N}}+\mu _{2}(I_{\alpha }* \vert v \vert ^{\frac{N+\alpha }{N}}) \vert v \vert ^{\frac{N+\alpha }{N}}+2\beta (I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}) \vert v \vert ^{\frac{N+\alpha }{N}} )^{\frac{N}{N+\alpha }}} \\ &>\frac{ \int u^{2}+v^{2}}{ ( \int \mu _{1}(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}) \vert u \vert ^{\frac{N+\alpha }{N}}+\mu _{2}(I_{\alpha }* \vert v \vert ^{\frac{N+\alpha }{N}}) \vert v \vert ^{\frac{N+\alpha }{N}} )^{\frac{N}{N+\alpha }}} \\ &\ge \bigl(\mu _{1}^{-\frac{N}{\alpha }}+\mu _{2}^{-\frac{N}{\alpha }} \bigr)^{ \frac{\alpha }{N+\alpha }} S_{1}, \end{aligned} $$

which contradicts (3.1). Thus, the conclusion holds. □

Proof of Theorem 1.5

By Lemmas 3.1 and 3.2, we see that Theorem 1.5 holds. □

4 Proof of Theorem 1.4

In this section, we define

$$ B=\inf_{\eta \in \Gamma }\max_{t\in [0,1]} I\bigl(\eta (t) \bigr), $$

where

$$ \Gamma =\bigl\{ \eta \in {\mathcal{C}}\bigl([0,1],H\bigr)| \eta (0)=(0,0), I \bigl( \eta (1)\bigr)< 0\bigr\} . $$

Set

$$ \mathcal{N}=\bigl\{ (u,v)\in H\setminus \bigl\{ (0,0)\bigr\} | \bigl\langle I'(u,v), (u,v) \bigr\rangle =0\bigr\} . $$

By simple calculation and analysis, we see that for any \((u,v)\ne (0,0)\), there exists \(t_{0}>0\) such that \(t_{0}(u,v)\in \mathcal{N}\) and \(I(t_{0}u,t_{0}v)=\max_{t\ge 0}I(tu,tv)\). Then, as in the proof of [23, Theorem 4.2], we know that

$$ B=\inf_{(u,v)\in H\setminus {(0,0)}}\max_{t\ge 0} I(tu,tv)=\inf _{ \mathcal{N}}I(u,v). $$

Moreover, since \({\mathcal{M}}\subset {\mathcal{N}}\), we have \(B\le \mathcal{B}\). We will show that B is attained by some positive solution \((u,v)\) of system (1.1). To begin with, we give an estimate of the upper bound of B, which is important in recovering the compactness of the Palais–Smale sequence.

Lemma 4.1

Assume that \((C1)\) and \((C2)\) hold. If \(\beta >0\), then

$$ B< \min \biggl\{ B_{1},B_{2},\frac{\alpha }{2(N+\alpha )}S_{0}^{ \frac{N+\alpha }{\alpha }} \biggr\} . $$

Proof

We first show that

$$ B< \frac{\alpha }{2(N+\alpha )}S_{0}^{\frac{N+\alpha }{\alpha }}. $$
(4.1)

Recall \((s_{m}U_{*}, U_{*})\) defined in Theorem 1.5, and let \(t>0\) be the constant such that \(t(s_{m}U_{*}, U_{*})\in \mathcal{N}\). Then, by Theorem 1.5 and direct calculation, we see that

$$\begin{aligned} B&\le I\bigl(t(s_{m}U_{*}, U_{*})\bigr) \\ & = \frac{1}{2}t^{2} \int \bigl(1+s_{m}^{2}\bigr) \vert \nabla U_{*} \vert ^{2}+\bigl(\lambda _{1}(x)s_{m}^{2}+ \lambda _{2}(x)\bigr)U_{*}^{2} \\ &\quad {}- \frac{N}{2(N+\alpha )}t^{ \frac{2(N+\alpha )}{N}} \bigl(\mu _{2}+\mu _{1}s_{m}^{ \frac{2(N+\alpha )}{N}}+2\beta s_{m}^{\frac{N+\alpha }{N}} \bigr) \int \bigl(I_{\alpha }* \vert U_{*} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert U_{*} \vert ^{ \frac{N+\alpha }{N}} \\ & = \frac{1}{2}t^{2} \int \bigl(1+s_{m}^{2}\bigr) U_{*}^{2} \\ &\quad {}- \frac{N}{2(N+\alpha )}t^{ \frac{2(N+\alpha )}{N}} \bigl(\mu _{2}+\mu _{1}s_{m}^{ \frac{2(N+\alpha )}{N}}+2\beta s_{m}^{\frac{N+\alpha }{N}} \bigr) \int \bigl(I_{\alpha }* \vert U_{*} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert U_{*} \vert ^{ \frac{N+\alpha }{N}} \\ & \quad{} + \frac{1}{2}t^{2} \int s_{m}^{2} \vert \nabla U_{*} \vert ^{2}+s_{m}^{2}\bigl(\lambda _{1}(x)-1\bigr)U_{*}^{2}+ \vert \nabla U_{*} \vert ^{2}+\bigl( \lambda _{2}(x)-1 \bigr)U_{*}^{2} \\ & \le \frac{\alpha }{2(N+\alpha )}\bigl(g(s_{m})S_{1} \bigr)^{ \frac{N+\alpha }{\alpha }}\\ &\quad {}+\frac{1}{2}t^{2} \int \bigl(1+s_{m}^{2}\bigr) \vert \nabla U_{*} \vert ^{2}+s_{m}^{2}\bigl( \lambda _{1}(x)-1\bigr)U_{*}^{2}+\bigl(\lambda _{2}(x)-1\bigr)U_{*}^{2} \\ &= \frac{\alpha }{2(N+\alpha )}S_{0}^{ \frac{N+\alpha }{\alpha }}+\frac{1}{2}t^{2} \int \bigl(1+s_{m}^{2}\bigr) \vert \nabla U_{*} \vert ^{2}+s_{m}^{2}\bigl( \lambda _{1}(x)-1\bigr)U_{*}^{2}+\bigl(\lambda _{2}(x)-1\bigr)U_{*}^{2}. \end{aligned}$$

Denote \(\phi _{i}(u)=\frac{1}{2}\int |\nabla u|^{2}+(\lambda _{i}(x)-1)u^{2}\), \(i=1,2 \). To get (4.1), it suffices to show

$$ \phi _{i}(U_{*})< 0, \quad i=1,2, $$
(4.2)

for some \(b\in {\mathbb{R}}^{N}\). By the fact that

$$ \int \frac{ \vert x \vert ^{2}}{(1+ \vert x \vert ^{2})^{N+2}}=\frac{N-2}{4(N+1)} \int \frac{1}{x^{2}(1+x^{2})^{N}}, $$

we obtain

$$ \int \vert \nabla U_{*} \vert ^{2}= \frac{N^{2}(N-2)}{4(N+1)} \int \frac{ \vert U_{*} \vert ^{2}}{ \vert x \vert ^{2}}. $$

After a transformation \(x=b+a y\), we have

$$ a^{2}\phi _{i}(U_{*})= \int \biggl(\frac{N^{2}(N-2)}{4(N+1) \vert y \vert ^{2}}-a^{2}\bigl(1- \lambda _{i}(b+a y)\bigr) \biggr)\frac{C^{2}}{(1+ \vert y \vert ^{2})^{N}}\,dy. $$

Then from \((C2)\) we see that (4.2) holds for \(b=0\), and (4.1) follows.

Next, we show \(B< B_{i}\), \(i=1,2\). Let \(w_{i}\) be a positive solution of (1.9) for \(i=1, 2\) and \(t(\tau )>0\) such that \((\sqrt{t(\tau )}w_{1},\sqrt{t(\tau )}\tau w_{1})\in \mathcal{N}\). Then

$$ t(\tau )^{\frac{\alpha }{N}}= \frac{ \int \vert \nabla w_{1} \vert ^{2}+\lambda _{1}(x)w_{1}^{2}+\tau ^{2}( \vert \nabla w_{1} \vert ^{2}+\lambda _{2}(x)w_{1}^{2})}{ (\mu _{1}+2\beta \tau ^{\frac{N+\alpha }{N}}+\mu _{2}\tau ^{\frac{2(N+\alpha )}{N}})\int (I_{\alpha }* \vert w_{1} \vert ^{\frac{N+\alpha }{N}}) \vert w_{1} \vert ^{\frac{N+\alpha }{N}}}. $$

By simple calculation, we get

$$ \lim_{\tau \to 0^{+}} \frac{t'(\tau )}{ \vert \tau \vert ^{\frac{\alpha }{N}-1}\tau }=- \frac{2(N+\alpha )}{\alpha \mu _{1}}\beta . $$

It follows that

$$ t(\tau )=1-\frac{2N}{\alpha \mu _{1}}\beta \tau ^{\frac{N+\alpha }{N}}\bigl(1+o(1)\bigr), \quad \text{as } \tau \to 0, $$

and

$$ t(\tau )^{\frac{N+\alpha }{N}}=1-\frac{2(N+\alpha )}{\alpha \mu _{1}} \beta \tau ^{\frac{N+\alpha }{N}} \bigl(1+o(1)\bigr),\quad \text{as } \tau \to 0. $$

Therefore,

$$ \begin{aligned} B&\le I\bigl(\sqrt{t(\tau )}w_{1},\sqrt{t( \tau )}\tau w_{1}\bigr) \\ & = \frac{\alpha }{2(N+\alpha )}t(\tau )^{ \frac{N+\alpha }{N}}\bigl(\mu _{1}+2\beta \tau ^{\frac{N+\alpha }{N}}+\mu _{2} \tau ^{\frac{2(N+\alpha )}{N}}\bigr) \int \bigl(I_{\alpha }* \vert w_{1} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert w_{1} \vert ^{\frac{N+\alpha }{N}} \\ & = \frac{\alpha }{2(N+\alpha )} \int \mu _{1}\bigl(I_{ \alpha }* \vert w_{1} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert w_{1} \vert ^{\frac{N+\alpha }{N}}\\ &\quad {}- \frac{N}{N+\alpha }\beta \tau ^{\frac{N+\alpha }{N}} \int \bigl(I_{\alpha }* \vert w_{1} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert w_{1} \vert ^{\frac{N+\alpha }{N}} +o\bigl( \tau ^{ \frac{N+\alpha }{N}}\bigr) \\ & < B_{1}\quad \text{for } \tau >0 \text{ small enough.} \end{aligned} $$

Similarly, we have \(B< B_{2}\). □

Next, we prove a Brezis–Lieb type lemma.

Lemma 4.2

Let \(\{(u_{n},v_{n})\}\) be a bounded sequence in H, and \((u_{n},v_{n})\to (u,v)\) a.e on \({\mathbb{R}}^{N}\) as \(n\to \infty \). Then

$$ \int \bigl(I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{ \frac{N+\alpha }{N}}- \int \bigl(I_{\alpha }* \vert u_{n}-u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n}-v \vert ^{ \frac{N+\alpha }{N}}\to \int \bigl(I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v \vert ^{ \frac{N+\alpha }{N}} $$

as \(n\to \infty \).

Proof

From the Brezis–Lieb lemma [23], we know that

$$\begin{aligned}& \vert u_{n} \vert ^{\frac{N+\alpha }{N}}- \vert u_{n}-u \vert ^{\frac{N+\alpha }{N}}\to \vert u \vert ^{ \frac{N+\alpha }{N}},\quad \text{in } L^{\frac{2N}{N+\alpha }}\bigl({ \mathbb{R}}^{N}\bigr), \\& \vert v_{n} \vert ^{\frac{N+\alpha }{N}}- \vert v_{n}-v \vert ^{\frac{N+\alpha }{N}}\to \vert v \vert ^{ \frac{N+\alpha }{N}},\quad \text{in } L^{\frac{2N}{N+\alpha }}\bigl({ \mathbb{R}}^{N}\bigr), \end{aligned}$$

as \(n\to \infty \). Then, according to the Hardy–Littlewood–Sobolev inequality, we have

$$\begin{aligned}& I_{\alpha }* \bigl( \vert u_{n} \vert ^{\frac{N+\alpha }{N}}- \vert u_{n}-u \vert ^{ \frac{N+\alpha }{N}} \bigr)\to I_{\alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\quad \text{in } L^{\frac{2N}{N-\alpha }}\bigl({ \mathbb{R}}^{N}\bigr), \\& I_{\alpha }* \bigl( \vert v_{n} \vert ^{\frac{N+\alpha }{N}}- \vert v_{n}-v \vert ^{ \frac{N+\alpha }{N}} \bigr)\to I_{\alpha }* \vert v \vert ^{\frac{N+\alpha }{N}}\quad \text{in } L^{\frac{2N}{N-\alpha }}\bigl({ \mathbb{R}}^{N}\bigr), \end{aligned}$$

as \(n\to \infty \). Observing that

$$ \begin{gathered} \int \bigl(I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{ \frac{N+\alpha }{N}}- \int \bigl(I_{\alpha }* \vert u_{n}-u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n}-v \vert ^{ \frac{N+\alpha }{N}} \\ \quad = \int I_{\alpha }*\bigl( \vert u_{n} \vert ^{\frac{N+\alpha }{N}}- \vert u_{n}-u \vert ^{ \frac{N+\alpha }{N}}\bigr) \bigl( \vert v_{n} \vert ^{\frac{N+\alpha }{N}}- \vert v_{n}-v \vert ^{ \frac{N+\alpha }{N}}\bigr) \\ \qquad{} + \int I_{\alpha }*\bigl( \vert v_{n} \vert ^{\frac{N+\alpha }{N}}- \vert v_{n}-v \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert u_{n}-u \vert ^{\frac{N+\alpha }{N}} \\ \qquad{} + \int I_{\alpha }*\bigl( \vert u_{n} \vert ^{\frac{N+\alpha }{N}}- \vert u_{n}-u \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert v_{n}-v \vert ^{\frac{N+\alpha }{N}},\end{gathered} $$
(4.3)

and

$$ \vert u_{n}-u \vert ^{\frac{N+\alpha }{N}}\rightharpoonup 0,\qquad \vert v_{n}-v \vert ^{ \frac{N+\alpha }{N}}\rightharpoonup 0 \quad \text{in } L^{ \frac{2N}{N+\alpha }}\bigl({\mathbb{R}}^{N}\bigr), $$

we see that the conclusion holds. □

Proof of Theorem 1.4

According to the mountain pass theorem [23], we obtain that there is \(\{(u_{n},v_{n})\}\subset \mathcal{N}\) satisfying

$$ I(u_{n},v_{n})\to B, I'(u_{n},v_{n}) \to 0\quad \text{in } H^{-1}. $$

It follows that

$$ \begin{aligned} B+o(1)&\ge I(u_{n},v_{n})- \frac{N}{2(N+\alpha )} \bigl\langle I'(u_{n},v_{n}),(u_{n},v_{n}) \bigr\rangle \\ & = \frac{\alpha }{2(N+\alpha )} \int \vert \nabla u_{n} \vert ^{2}+\lambda _{1}(x)u_{n}^{2}+ \vert \nabla v_{n} \vert ^{2}+ \lambda _{2}(x)v_{n}^{2} \end{aligned} $$

for n large enough, which combined with assumption \((C1)\) implies that \(\{(u_{n},v_{n})\}\) is bounded in H. Then we may assume that

$$ \begin{gathered} (u_{n},v_{n})\rightharpoonup (u,v)\quad \text{in } H, \\ (u_{n},v_{n})\to (u,v)\quad \text{in } L_{loc}^{2}\bigl({ \mathbb{R}}^{N}\bigr)\times L_{loc}^{2}\bigl({\mathbb{R}}^{N}\bigr), \\ (u_{n},v_{n})\to (u,v)\quad \text{a.e on } { \mathbb{R}}^{N}.\end{gathered} $$

Since \(|u_{n}|^{\frac{N+\alpha }{N}}\) and \(|v_{n}|^{\frac{N+\alpha }{N}}\) are bounded in \(L^{\frac{2N}{N+\alpha }}({\mathbb{R}}^{N})\), we have

$$ \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\rightharpoonup \vert u \vert ^{\frac{N+\alpha }{N}}, \qquad \vert v_{n} \vert ^{\frac{N+\alpha }{N}} \rightharpoonup \vert v \vert ^{ \frac{N+\alpha }{N}} \quad \text{in } L^{\frac{2N}{N+\alpha }} \bigl({ \mathbb{R}}^{N}\bigr). $$

Using the Hardy–Littlewood–Sobolev inequality, we obtain

$$ I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\rightharpoonup I_{\alpha }* \vert u \vert ^{ \frac{N+\alpha }{N}},\qquad I_{\alpha }* \vert v_{n} \vert ^{\frac{N+\alpha }{N}} \rightharpoonup I_{\alpha }* \vert v \vert ^{\frac{N+\alpha }{N}} \quad \text{in } L^{\frac{2N}{N-\alpha }}\bigl({ \mathbb{R}}^{N}\bigr). $$

Observing that

$$ \vert u_{n} \vert ^{\frac{\alpha }{N}-1}u_{n}\to \vert u \vert ^{\frac{\alpha }{N}-1}u,\qquad \vert v_{n} \vert ^{ \frac{\alpha }{N}-1}v_{n}\to \vert v \vert ^{\frac{\alpha }{N}-1}v \quad \text{in } L_{loc}^{\frac{2N}{\alpha }}\bigl({\mathbb{R}}^{N}\bigr), $$

we have, for any \(\phi \in C_{0}^{\infty }({\mathbb{R}}^{N})\),

$$ \begin{gathered} \int \bigl(I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u_{n} \vert ^{ \frac{\alpha }{N}-1}u_{n} \phi \to \int \bigl(I_{\alpha }* \vert u \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert u \vert ^{\frac{\alpha }{N}-1}u\phi , \\ \int \bigl(I_{\alpha }* \vert v_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{ \frac{\alpha }{N}-1}v_{n} \phi \to \int \bigl(I_{\alpha }* \vert v \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert v \vert ^{\frac{\alpha }{N}-1}v\phi , \\ \int \bigl(I_{\alpha }* \vert u_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v_{n} \vert ^{ \frac{\alpha }{N}-1}v_{n} \phi \to \int \bigl(I_{\alpha }* \vert u \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert v \vert ^{\frac{\alpha }{N}-1}v\phi , \\ \int \bigl(I_{\alpha }* \vert v_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u_{n} \vert ^{ \frac{\alpha }{N}-1}u_{n} \phi \to \int \bigl(I_{\alpha }* \vert v \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert u \vert ^{\frac{\alpha }{N}-1}u\phi ,\end{gathered} $$
(4.4)

as \(n\to \infty \). Taking account of \(I'(u_{n},v_{n})\to 0\), (4.4), and the fact that \(C_{0}^{\infty }({\mathbb{R}}^{N})\) is dense in \(H^{1}({\mathbb{R}}^{N})\), we have \(I'(u,v)=0\). Denote \(z_{n}=u_{n}-u\), \(\omega _{n}=v_{n}-v\), then \((z_{n},\omega _{n})\rightharpoonup (0,0)\) in H, \((z_{n},\omega _{n})\to (0,0)\) in \(L_{loc}^{2}({\mathbb{R}}^{N})\times L_{loc}^{2}({\mathbb{R}}^{N})\), and \((z_{n},\omega _{n})\to (0,0)\) a.e on \({\mathbb{R}}^{N}\). By \((C1)\), there exists \(R>0\) sufficiently large such that

$$ \begin{aligned} \int \lambda _{1}(x)z_{n}^{2}+\lambda _{2}(x)\omega _{n}^{2}&= \int _{{ \mathbb{R}}^{N}\setminus B(0,R)}z_{n}^{2}+\omega _{n}^{2}+ \int _{B(0,R)} \lambda _{1}(x)z_{n}^{2}+ \lambda _{2}(x)\omega _{n}^{2}+o(1)\\ &= \int z_{n}^{2}+ \omega _{n}^{2}+o(1). \end{aligned} $$
(4.5)

Denote

$$ \begin{aligned} J(u,v)&= \frac{1}{2} \int \vert \nabla u \vert ^{2}+u^{2}+ \vert \nabla v \vert ^{2}+v^{2} \\ &\quad{} - \frac{N}{2(N+\alpha )} \int \bigl(\mu _{1}\bigl(I_{ \alpha }* \vert u \vert ^{\frac{N+\alpha }{N}}\bigr) \vert u \vert ^{\frac{N+\alpha }{N}}+\mu _{2} \bigl(I_{ \alpha }* \vert v \vert ^{\frac{N+\alpha }{N}}\bigr) \vert v \vert ^{\frac{N+\alpha }{N}}\\ &\quad {}+2\beta \bigl(I_{ \alpha }* \vert u \vert ^{\frac{N+\alpha }{N}} \bigr) \vert v \vert ^{\frac{N+\alpha }{N}}\bigr).\end{aligned} $$

Combining (4.5) with Lemma 4.2, we have, for n large enough,

$$ \bigl\langle J'(z_{n},w_{n}),(z_{n}, \omega _{n})\bigr\rangle =\bigl\langle I'(u_{n},v_{n}),(u_{n},v_{n}) \bigr\rangle -\bigl\langle I'(u,v),(u,v)\bigr\rangle =o(1) $$
(4.6)

and

$$ B+o(1)=I(u_{n},v_{n})=I(u,v)+J(z_{n}, \omega _{n})+o(1). $$
(4.7)

Set

$$ C_{n}= \int \vert \nabla z_{n} \vert ^{2}+z_{n}^{2}, \qquad D_{n}= \int \vert \nabla \omega _{n} \vert ^{2}+\omega _{n}^{2}. $$

Then it follows

$$ B=I(u,v)+\frac{\alpha }{2(N+\alpha )}(C_{n}+D_{n})+o(1). $$
(4.8)

We will show that \(u\not \equiv 0\), \(v\not \equiv 0\) by excluding the following three cases:

(i) \((u,v)\equiv (0,0)\). By (4.8), we know that

$$ C_{n}+D_{n}>0. $$

Denote

$$ E_{n}= \int \mu _{1}\bigl(I_{\alpha }* \vert z_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert z_{n} \vert ^{ \frac{N+\alpha }{N}}, \qquad F_{n}= \int \mu _{2}\bigl(I_{\alpha }* \vert w_{n} \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert w_{n} \vert ^{\frac{N+\alpha }{N}}. $$

If \(E_{n}\to 0\), then \(\int (I_{\alpha }*|w_{n}|^{\frac{N+\alpha }{N}})|z_{n}|^{ \frac{N+\alpha }{N}}\to 0\). So we have

$$\begin{aligned} \int \vert \nabla z_{n} \vert ^{2}+z_{n}^{2}+ \vert \nabla w_{n} \vert ^{2}+w_{n}^{2}&= \int \mu _{1}\bigl(I_{\alpha }* \vert w_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert w_{n} \vert ^{ \frac{N+\alpha }{N}}+o(1). \\ &\le \mu _{1} S_{1}^{-\frac{N+\alpha }{N}} \biggl( \int \vert w_{n} \vert ^{2} \biggr)^{\frac{N+\alpha }{N}}. \\ &\le \mu _{1} S_{1}^{-\frac{N+\alpha }{N}} \biggl( \int \vert \nabla z_{n} \vert ^{2}+z_{n}^{2}+ \vert \nabla w_{n} \vert ^{2}+w_{n}^{2} \biggr)^{ \frac{N+\alpha }{N}}, \end{aligned}$$

which implies

$$ \int \vert \nabla z_{n} \vert ^{2}+z_{n}^{2}+ \vert \nabla w_{n} \vert ^{2}+w_{n}^{2} \ge \mu _{1}^{-\frac{N}{\alpha }} S_{1}^{\frac{N+\alpha }{\alpha }}. $$

Then, by (4.8) and (1.11), we obtain

$$ B=I(u,v)+\frac{\alpha }{2(N+\alpha )}(C_{n}+D_{n})+o(1)\ge \frac{\alpha }{2(N+\alpha )}\mu _{1}^{-\frac{N}{\alpha }}S_{1}^{ \frac{N+\alpha }{\alpha }}>B_{1}, $$

which contradicts Lemma 4.1. Similarly, \(F_{n}\to 0\) also leads to a contradiction. Thus, \(E_{n}\ge \delta \) and \(F_{n}\ge \delta \) for some \(\delta >0\) and n large enough. Then there exists \(t_{n}>0\) such that

$$ \bigl\langle J'(t_{n}z_{n},t_{n} \omega _{n}),(t_{n}z_{n},t_{n}\omega _{n}) \bigr\rangle =0 $$

and

$$ \begin{gathered} J(t_{n}z_{n},t_{n} \omega _{n}) \\ \quad = \max_{s_{n}\ge 0}J(s_{n}z_{n},s_{n} \omega _{n}) \\ \quad \ge \max_{s_{n}\ge 0}\frac{1}{2}s_{n}^{2} \int \vert z_{n} \vert ^{2}+ \omega _{n}^{2} \\ \qquad {} -\frac{N s_{n}^{\frac{2(N+\alpha )}{N}}}{2(N+\alpha )} \int \bigl(\mu _{1}\bigl(I_{\alpha }* \vert z_{n} \vert ^{\frac{N+\alpha }{N}}\bigr) \vert z_{n} \vert ^{ \frac{N+\alpha }{N}}+\mu _{2}\bigl(I_{\alpha }* \vert \omega _{n} \vert ^{ \frac{N+\alpha }{N}}\bigr) \vert \omega _{n} \vert ^{\frac{N+\alpha }{N}}\\ \qquad {}+2 \beta \bigl(I_{ \alpha }* \vert z_{n} \vert ^{\frac{N+\alpha }{N}} \bigr) \vert \omega _{n} \vert ^{ \frac{N+\alpha }{N}}\bigr) \\ \quad \ge \frac{\alpha }{2(N+\alpha )}S_{0}^{ \frac{N+\alpha }{\alpha }}, \end{gathered} $$
(4.9)

where the last inequality follows by Theorem 1.5. Moreover, by (4.6), we have \(t_{n}\to 1\). Then we have

$$ B=I(u,v)+J(z_{n},\omega _{n})=J(t_{n}z_{n},t_{n} \omega _{n})\ge \frac{\alpha }{2(N+\alpha )}S_{0}^{\frac{N+\alpha }{\alpha }}, $$

which also contradicts Lemma 4.1.

(ii) \(u\equiv 0\), \(v\not \equiv 0\). In this case, it is clear that v is a solution of (1.9) for \(i=2\). Then, by (4.7), we have \(B\ge I(0,v)\ge B_{2}\), which contradicts Lemma 4.1.

(iii) \(v\equiv 0\), \(u\not \equiv 0\). By similar arguments as in case (ii), we see that \(B\ge B_{1}\), which also contradicts Lemma 4.1.

Thus, we have proved that \(u\not \equiv 0\), \(v\not \equiv 0\), and \(I'(u,v)=0\). Then \(I(u,v)\ge B\), which combining with (4.7), (4.8) indicates \(I(u,v)=B\). Hence, \((u,v)\) is a ground state of system (1.1). Moreover, since \(I(|u|,|v|)=B\) and \((|u|,|v|)\in \mathcal{N}\), we know that \((|u|,|v|)\) is also a ground state of (1.1). By the strong maximum principle, we have \(|u|>0\), \(|v|>0\). Thus, system (1.1) has a positive ground state \((|u|,|v|)\). □

Remark 4.3

Let \((u,v)\) be a solution obtained in Theorem 1.4. Then it is obvious that \((u,v)\in \mathcal{M}\). Moreover, since \(\mathcal{M}\in \mathcal{N}\), we have

$$ B=I(u,v)\le \mathcal{B}\le I(u,v), $$

which implies that \(B=\mathcal{B}\).