Multiple positive solutions for nonlinear coupled fractional Laplacian system with critical exponent

Abstract

In this paper, we study the following critical system with fractional Laplacian:

$$\textstyle\begin{cases} (-\Delta)^{s}u+\lambda_{1}u=\mu_{1}|u|^{2^{\ast}-2}u+\frac{\alpha \gamma}{2^{\ast}}|u|^{\alpha-2}u|v|^{\beta} & \text{in } \Omega, \\ (-\Delta)^{s}v+\lambda_{2}v= \mu_{2}|v|^{2^{\ast}-2}v+\frac{\beta \gamma}{2^{\ast}}|u|^{\alpha}|v|^{\beta-2}v & \text{in } \Omega, \\ u=v=0 & \text{in } \mathbb{R}^{N}\setminus\Omega, \end{cases} $$

where $(-\Delta)^{s}$ is the fractional Laplacian, $0< s<1$, $\mu_{1},\mu_{2}>0$, $2^{\ast}=\frac{2N}{N-2s}$ is a fractional critical Sobolev exponent, $N>2s$, $1<\alpha$, $\beta<2$, $\alpha+\beta=2^{\ast}$, Ω is an open bounded set of $\mathbb{R}^{N}$ with Lipschitz boundary and $\lambda_{1},\lambda_{2}>-\lambda_{1,s}(\Omega)$, $\lambda_{1,s}(\Omega)$ is the first eigenvalue of the non-local operator $(-\Delta)^{s}$ with homogeneous Dirichlet boundary datum. By using the Nehari manifold, we prove the existence of a positive ground state solution of the system for all $\gamma>0$. Via a perturbation argument and using the topological degree and a pseudo-gradient vector field, we show that this system has a positive higher energy solution. Then the asymptotic behaviors of the positive ground state solutions are analyzed when $\gamma\rightarrow0$.

1 Introduction

The fractional Laplacian operator and fractional Sobolev space arise in a quite natural way in many different contexts, such as the thin obstacle problem, finance, phase transitions, anomalous diffusion, flame propagation and many others (see [1–4] and the references therein). In recent years, the corresponding non-local equation or systems involving fractional Laplacian with nonlinear terms have attracted the attention of many researchers, both for their interesting theoretical structure and their concrete applications (see [5–11] and the references therein).

There have been a lot of studies that consider a Laplacian equation or a Laplacian system (see [12–16] and the references therein). Compared to the Laplacian problem, the fractional Laplacian problem is non-local and more difficult to handle. For the following fractional Laplacian equation:

$$ \textstyle\begin{cases} (-\Delta)^{s}u= f(x,u) &\text{in } \Omega,\\ u=0& \text{on } \mathbb{R}^{N}\setminus \Omega. \end{cases} $$

(1)

Servadei and Valdinoci [17] showed that (1) has mountain pass type solution which is not identically zero. When $f(x,u)=\lambda u^{q}+u^{2^{\ast }-1}$, Barrios, Colorado, Servadei and Soria [18] obtained the existence and multiplicity solutions for system (1) under different conditions of λ.

For the following fractional Laplacian equation:

$$ \textstyle\begin{cases} (-\Delta)^{s}u= f(x,u) &\text{in } \Omega,\\ u=0& \text{on } \partial \Omega.\end{cases} $$

(2)

Caffarelli and Silvestre [19] studied an extension problem related to the fractional Laplacian in ${\mathbb{R}}^{n}$, which can transform the non-local problem into a local problem in ${\mathbb{R}} _{+}^{n+1}$. This method can be extended to bounded regions and is extensively used in recent articles. For example, when $f(x,u)=\lambda u^{q}+u^{\frac{n+s}{n-s}}$, Barrios, Colorado, de Pablo and Sánchez [5] proved the existence and multiplicity of solutions for equation (2) under suitable conditions of s and q. When $f(x,u)=|u|^{2^{\ast }-2}u+f(x)$ Colorado, de Pablo and Sánchez [6] proved the existence and the multiplicity of solutions for equation (2) under appropriate conditions on the size of f.

The following Brézis–Nirenberg problem for the fractional Laplacian:

$$\textstyle\begin{cases} (-\Delta)^{s}u+\lambda_{i} u=|u|^{2^{\ast }-2}u & \text{in } \Omega, \\ u=0 & \text{in } \mathbb{R}^{N}\setminus \Omega, \end{cases} $$

has been investigated by Servadei and Valdinoci [20, 21] and obtained a non-trivial solutions.

It is also natural to study the coupled system of equations. Li and Yang [22] considered the following subcritical case fractional Laplacian system:

$$\textstyle\begin{cases} (-\Delta)^{s} u= \lambda \vert u\vert ^{q-2}u+{\frac{2\alpha }{\alpha +\beta }}\vert u\vert ^{ \alpha -2}u\vert v\vert ^{\beta } & \text{in } \Omega,\\ (-\Delta)^{s} v = \mu \vert v\vert ^{q-2}v+{\frac{2 \beta }{\alpha +\beta }}\vert u\vert ^{\alpha }\vert v\vert ^{\beta -2}v& \text{in } \Omega,\\ u=0,\qquad v=0& \text{on } \mathbb{R}^{N}\setminus \Omega, \end{cases} $$

by using the Nehari manifold, fibering maps and the Lusternik–Schnirelmann category, they prove that the problem has at least $\operatorname{cat}(\Omega) +1$ distinct positive solutions, where $\operatorname{cat}( \Omega)$ denotes the Lusternik–Schnirelmann category of Ω in itself. When the boundary conditions are replaced by $u=0$, $v=0$ on ∂Ω, X. He, Squassina and Zou [23] using variational methods and a Nehari manifold decomposition proved that the system admits at least two positive solutions when the pair of parameters $(\lambda,\mu)$ belong to certain subset of $\mathbb{R} ^{2}$.

We address the following critical system involving a fractional Laplacian:

$$ \textstyle\begin{cases} (-\Delta)^{s}u+\lambda_{1}u=\mu_{1}|u|^{2^{\ast }-2}u+\frac{\alpha \gamma }{2^{\ast }}|u|^{\alpha -2}u|v|^{\beta } & \text{in } \Omega, \\ (-\Delta)^{s}v+\lambda_{2}v= \mu_{2}|v|^{2^{\ast }-2}v+\frac{\beta \gamma }{2^{\ast }}|u|^{\alpha }|v|^{\beta -2}v & \text{in } \Omega, \\ u=v=0 & \text{in } \mathbb{R}^{N}\setminus \Omega, \end{cases} $$

(3)

where $(-\Delta)^{s}$ is the fractional Laplacian, $0< s<1$, $\mu_{1}, \mu_{2}>0$, $2^{\ast }=\frac{2N}{N-2s}$ is a fractional critical Sobolev exponent, $N>2s$, $1<\alpha$, $\beta <2$, $\alpha +\beta =2^{ \ast }$, Ω is an open bounded set of $\mathbb{R}^{N}$ with Lipschitz boundary and $\lambda_{1},\lambda_{2}>-\lambda_{1,s}(\Omega)$, $\lambda_{1,s}(\Omega)$ is the first eigenvalue of the non-local operator $(-\Delta)^{s}$ with homogeneous Dirichlet boundary datum.

The fractional Laplacian $(-\Delta)^{s}$ is defined by

$$-(-\Delta)^{s}u(x)=\frac{C(N,s)}{2} \int_{R^{N}} \frac{u(x+y)+u(x-y)-2u(x)}{\vert y\vert ^{N+2s}}\,dy, \quad x\in \mathbb{R}^{N}, $$

with

$$C(N,s)= \biggl( \int_{R^{N}} \frac{1-\cos (\varsigma_{1})}{\vert \varsigma \vert ^{N+2s}}\,d\varsigma \biggr) ^{-1}=2^{2s}\pi^{-\frac{N}{2}} \frac{\Gamma (\frac{N+2s}{2})}{\Gamma (2-s)}s(1-s). $$

Guo, Luo and Zou [11], showed that when $\lambda_{1},\lambda _{2}\in (-\lambda_{1,s}(\Omega),0)$, (3) has a positive ground state solution for all $\gamma >0$. For more recent advances on this topic, see [24–26] and the references therein.

In [27], we have consider the following critical system:

$$\textstyle\begin{cases} (-\Delta)^{s}u= \mu_{1}|u|^{2^{\ast }-2}u+\frac{\alpha \gamma }{2^{ \ast }}|u|^{\alpha -2}u|v|^{\beta } & \text{in } \mathbb{R}^{n}, \\ (-\Delta)^{s}v= \mu_{2}|v|^{2^{\ast }-2}v+\frac{\beta \gamma }{2^{ \ast }}|u|^{\alpha }|v|^{\beta -2}v & \text{in } \mathbb{R}^{n}, \\ u,v\in D_{s}(\mathbb{R}^{n}). \end{cases} $$

By using the Nehari manifold, under proper conditions, we establish the existence and nonexistence of a positive least energy solution of the above system.

In this paper, we study system (3) from another aspect to obtain the ground state solutions, higher energy solution and an analysis the asymptotic behaviors of the positive ground state solutions.

Let ${D_{s}(\Omega)}$ be Hilbert space as the completion of $C^{\infty }_{c}(\Omega)$ equipped with the norm

$$\Vert u\Vert ^{2}_{D_{s}(\Omega)}=\frac{C(N,s)}{2} \int_{\mathbb{R}^{N}} \int_{\mathbb{R}^{N}}\frac{\vert u(x)-u(y)\vert ^{2}}{\vert y-x\vert ^{N+2s}}\,dx\,dy. $$

Let

$$ S_{s}=\inf_{u\in {D_{s}(\mathbb{R}^{N})\setminus \{0\}}}\frac{ \Vert u\Vert ^{2}_{D_{s}(\mathbb{R}^{N})}}{(\int_{R^{N}}\vert u\vert ^{2^{\ast }}\,dx)^{\frac{2}{2^{ \ast }}}} $$

(4)

be the sharp embedding constant of ${D_{s}(\mathbb{R}^{N})}\hookrightarrow L^{2^{\ast }}(\mathbb{R}^{N})$ and $S_{s}$ is attained (see [28]) in $\mathbb{R}^{N}$ by $\widetilde{u}_{\epsilon, y}= \kappa (\varepsilon^{2}+|x-y|)^{-\frac{N-2s}{2}}$, where $\kappa \neq 0 \in \mathbb{R}$, $\varepsilon >0$ and $y\in \mathbb{R}^{N}$. That is,

$$S_{s}=\frac{\Vert \widetilde{u}_{\epsilon,y} \Vert ^{2}_{D_{s}(\mathbb{R}^{N})}}{( \int_{\mathbb{R}^{N}}\vert \widetilde{u}_{\epsilon,y} \vert ^{2^{\ast }}\,dx)^{\frac{2}{2^{ \ast }}}}. $$

The energy functional associated with (3) is given by

$$\begin{aligned} E_{\gamma }(u,v) =&\frac{1}{2} \bigl\Vert (u,v) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}+ \frac{1}{2} \int_{\Omega } \bigl(\lambda_{1}u^{2}+ \lambda_{2}v^{2} \bigr)\,dx \\ &{}-\frac{1}{2^{ \ast }} \int_{\Omega } \bigl(\mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+ \gamma \vert u\vert ^{\alpha }\vert v\vert ^{\beta } \bigr)\,dx, \end{aligned}$$

where $\mathcal{D}_{s}(\Omega):=D_{s}(\Omega)\times D_{s}(\Omega)$ is endowed with norm $\Vert (u,v)\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}=\|u\| ^{2}_{D_{s}(\Omega)}+\|v\|^{2}_{D_{s}(\Omega)}$. Define the Nehari manifold

$$\begin{aligned} &\mathbb{M}= \biggl\{ (u,v)\in \mathcal{D}_{s}(\Omega)\backslash \bigl\{ (0,0) \bigr\} : \bigl\Vert (u,v) \bigr\Vert ^{2}_{\mathcal{D}_{s}( \Omega)}+ \int_{\Omega } \bigl(\lambda_{1}u^{2}+ \lambda_{2}v^{2} \bigr)\,dx \\ &\hphantom{\mathbb{M}=}= \int_{\Omega } \bigl(\mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+ \gamma \vert u\vert ^{\alpha }\vert v\vert ^{\beta } \bigr)\,dx \biggr\} , \\ &A_{\gamma }:=\inf_{(u,v)\in \mathbb{M}}E_{\gamma }(u,v) = \inf _{(u,v)\in \mathbb{M}}\frac{s}{N} \biggl( \bigl\Vert (u,v) \bigr\Vert ^{2}_{ \mathcal{D}_{s}(\Omega)}+ \int_{\Omega } \bigl(\lambda_{1}u^{2}+ \lambda_{2}v ^{2} \bigr)\,dx \biggr) \\ &\hphantom{A_{\gamma }}=\inf_{(u,v)\in \mathbb{M}}\frac{s}{N} \int_{\Omega } \bigl( \mu _{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+\gamma \vert u\vert ^{\alpha }\vert v\vert ^{ \beta } \bigr)\,dx. \end{aligned}$$

We say that $(u,v)$ is a non-trivial solution of (3) if $u\neq 0$, $v\neq 0$ and $(u,v)$ solves (3). Any non-trivial solution of (3) is in $\mathbb{M}$. Due to the fact that if we take $\varphi,\psi \in \mathcal{C}^{\infty }_{0}(\Omega)$ with $\varphi,\psi \not \equiv 0$ and $\operatorname{supp}(\varphi)\cap \operatorname{supp}(\psi)=\emptyset$, then there exist $t_{1},t_{2}>0$ such that $(t_{1}\varphi,t_{2}\psi) \in \mathbb{M}$, so $\mathbb{M}\neq \emptyset$.

Our main results are as follows.

Theorem 1.1

• (i) Assume $-\lambda _{1,s}(\Omega)<\min \{\lambda _{1},\lambda _{2}\}<0$ and $N>4s$. Then system (3) has a positive ground state solution $(u_{\gamma },v_{\gamma })\in \mathcal{D}_{s}(\Omega)$ with $E_{\gamma }(u_{\gamma },v_{\gamma })=A_{\gamma }$ for all $\gamma >0$.

• (ii) Assume $-\lambda _{1,s}(\Omega)<\min \{\lambda _{1},\lambda _{2}\}<0$, $N>4s$ and let $\gamma_{n}$ be a sequence with $\gamma_{n}\rightarrow 0$ as $n\rightarrow +\infty$. Then, passing to a subsequence, $({u}_{\gamma _{n}},{v}_{\gamma _{n}})\rightarrow (\overline{u},\overline{v})$ strongly in $D_{s}(\Omega)\times D_{s}(\Omega)$ as $n\rightarrow +\infty$, and one of the following conclusions holds:

  • (1) $(\overline{u},0)$ is a positive ground state solution of

    $$\textstyle\begin{cases} (-\Delta)^{s}u+\lambda_{1} u=\mu_{1}|u|^{2^{\ast }-2}u & \textit{in } \Omega; \\ u=0 & \textit{on } \mathbb{R}^{N}\setminus \Omega. \end{cases} $$

  • (2) $(0,\overline{v})$ is a positive ground state solution of

    $$\textstyle\begin{cases} (-\Delta)^{s}v+\lambda_{2} v=\mu_{2}|v|^{2^{\ast }-2}v & \textit{in } \Omega, \\ v=0 & \textit{on } \mathbb{R}^{N}\setminus \Omega. \end{cases} $$

  If

  $$\biggl(\frac{\mu_{1}}{\mu_{2}} \biggr)^{-\frac{N-2s}{2s}}< \frac{m_{\lambda_{2}}}{m _{\lambda_{1}}} \quad { \textit{implies that}}\quad m_{\lambda _{1},\mu_{1}}< m_{\lambda_{2},\mu_{2}}, $$

  then (1) holds.

  If

  $$\biggl(\frac{\mu_{1}}{\mu_{2}} \biggr)^{-\frac{N-2s}{2s}}>\frac{m_{\lambda_{2}}}{m _{\lambda_{1}}} \quad { \textit{implies that}}\quad m_{\lambda _{1},\mu_{1}}>m_{\lambda_{2},\mu_{2}}, $$

  then (2) holds, where $m_{\lambda_{i}}$ and $m_{\lambda_{i}, \mu_{i}}$ see Lemma 2.1 and Remark 2.1 in the next section.

Theorem 1.2

Assume $-\lambda _{1,s}(\Omega)<\min \{\lambda _{1},\lambda _{2}\}<0$ and $N>4s$, then there exists a $\gamma_{0}>0$ such that, for $|\gamma |<\gamma _{0}$, system (3) has a positive higher energy solution $(\widehat{u}_{\gamma },\widehat{v}_{\gamma })$ with $E_{\gamma }( \widehat{u}_{\gamma },\widehat{v}_{\gamma })>A_{\gamma }$.

Remark 1.1

Although the method in this paper to obtain the ground state solution is different from Z. Guo, S. Luo and W. Zou [11], we get similar result as Theorem 1.2 in [11].

Remark 1.2

In the proof of Theorem 1.1, we should point out that $1<\alpha, \beta <2$ is an essential condition.

Remark 1.3

In the proof of Theorem 1.1, we need $N>4s$, due to $1<\alpha, \beta <2$ and $2<\alpha +\beta =2^{\ast }<4$. For $2s< N<4s$, the method in this paper does not work and it should be interesting to get a ground state solution.

Remark 1.4

It is easy to see that, for $\gamma >0$ sufficiently small, the higher energy solutions in Theorem 1.2 are different from the ground state solutions in Theorem 1.1. That is system (3) has at least two positive solutions for $\lambda_{1},\lambda_{2}<0$ and $\gamma >0$ sufficiently small.

In order to prove Theorem 1.1, we use the classical mountain pass theorem, due to each equation in this system is critical exponent, so the embedding for $\mathcal{D}_{s}(\Omega)\hookrightarrow L^{2^{ \ast }}(\Omega)$ is not compact embedding. Thus, we need estimate $A_{\gamma }$ such that $A_{\gamma }$ is strict less than $\min \{ \mu^{-\frac{N-2s}{2s}}_{1}\frac{s}{N}S^{\frac{N}{2s}}_{s}, \mu^{-\frac{N-2s}{2s}}_{2}\frac{s}{N}S^{\frac{N}{2s}}_{s}\}$ (see Lemma 2.4). The main idea to prove Theorem 1.2 is to regard system (3) as a perturbation of system (24) by $\frac{\alpha \gamma }{2^{\ast }}|u|^{\alpha -2}u|v|^{\beta }$ and $\frac{\beta \gamma }{2^{\ast }}|u|^{\alpha }|v|^{\beta -2}v$, then use the topological degree and the pseudo-gradient vector field to show some lemmas that will be used to get another positive solution. The idea is originally from [29].

The paper is organized as follows. In Sect. 2, we introduce some preliminaries that will be used to prove theorems. In Sect. 3, we prove Theorem 1.1 and Theorem 1.2 will be proved in Sect. 4.

2 Some preliminaries

For the following fractional Brézis–Nirenberg problem:

$$ \textstyle\begin{cases} (-\Delta)^{s}u+\lambda_{i} u=\mu_{i}|u|^{2^{\ast }-2}u & \text{in } \Omega, \\ u=0 & \text{on } \mathbb{R}^{N}\setminus \Omega, \end{cases} $$

(5)

we define

$$J_{\lambda_{i},\mu_{i}}(u)=\frac{1}{2}\Vert u\Vert ^{2}_{D_{s}(\Omega)}+ \frac{1}{2} \int_{\Omega }\lambda_{i}u^{2}\,dx- \frac{1}{2^{\ast }} \int_{\Omega }\mu_{i}\vert u\vert ^{2^{\ast }}\,dx $$

and

$$m_{\lambda_{i},\mu_{i}}=\inf_{u\in \mathbb{M}_{i}}J_{\lambda _{i},\mu_{i}}(u), $$

where

$$\mathbb{M}_{i}= \biggl\{ u\in D_{s}(\Omega)\backslash \{0\}: \Vert u\Vert ^{2} _{D_{s}(\Omega)}+ \int_{\Omega }\lambda_{i}u^{2}\,dx = \int_{\Omega }\mu _{i}\vert u\vert ^{2^{\ast }} \,dx \biggr\} . $$

Lemma 2.1

(See [20])

When $\mu_{i}=1$ and assume $-\lambda_{1,s}(\Omega)<\min \{\lambda_{1},\lambda_{2}\}<0$ and $N>4s$, then (5) has a non-trivial ground state solution such that

$$\begin{aligned} J_{i}(u_{\lambda_{i}})=m_{\lambda_{i}}< \frac{s}{N}S^{\frac{N}{2s}} _{s}, \quad i=1,2. \end{aligned}$$

(6)

Remark 2.1

By Lemma 2.1, it is easy to see, when $\mu_{i}=1$, if $u_{\lambda_{i}}$ is a non-trivial ground state solution of (5), then $u_{\lambda _{i},\mu _{i}}=\mu ^{-\frac{1}{2^{\ast }-2}}_{i}u_{\lambda _{i}}$ is a non-trivial ground state solution of (5) for $0<\mu_{i} \neq 1$ and the energy of (5) satisfies

$$\begin{aligned} J_{\lambda_{i},\mu_{i}}(u_{\lambda_{i},\mu_{i}})=m_{\lambda_{i},\mu _{i}}< \mu^{-\frac{N-2s}{2s}}_{i}\frac{s}{N}S^{\frac{N}{2s}}_{s}. \end{aligned}$$

(7)

In order to prove Theorem 1.1, we give the following lemmas.

Lemma 2.2

Define $\widehat{A_{\gamma }}:=\inf_{\sigma \in \Gamma } \max_{t\in [0,1]}E_{\gamma }(\sigma (t))$, then there exist a sequence $\{(u_{n},v_{n})\}\subset \mathcal{D}_{s}(\Omega)$ such that

$$\begin{aligned} E_{\gamma }(u_{n},v_{n})\rightarrow \widehat{A_{\gamma }}\quad \textit{and}\quad E'_{\gamma }(u_{n},v_{n}) \rightarrow 0 \quad \textit{as } n\rightarrow + \infty, \end{aligned}$$

(8)

where

$$\Gamma = \bigl\{ \sigma \in \mathcal{C} \bigl([0,1],\mathcal{D}_{s}( \Omega) \bigr): \sigma (0)=(0,0),\sigma (1)=(u_{0},v_{0}) \bigr\} . $$

Proof

We first claim that $E_{\gamma }$ possesses a mountain pass geometry around $(0,0)$;

• (1) there exist $\alpha,\rho >0$, such that $E_{\gamma }(u,v)>\alpha$ for all $\Vert (u,v)\Vert _{\mathcal{D}_{s}(\Omega)}=\rho$;

• (2) there exist $(u_{0},v_{0})\in \mathcal{D}_{s}(\Omega)$ such that $\Vert (u_{0},v_{0})\Vert _{\mathcal{D}_{s}(\Omega)}>\rho$ and $E_{\gamma }(u _{0},v_{0})<0$.

Since $\lambda_{1},\lambda_{2}>-\lambda_{1,s}(\Omega)$ and the Sobolev embedding theorem $D_{s}(\Omega)\hookrightarrow L^{2}(\Omega)$, it is easy to see $\Vert \cdot \Vert _{\lambda_{i}}$, $i=1,2$, are equivalent to $\Vert \cdot \Vert _{D_{s}(\Omega)}$, where $\Vert u\Vert _{\lambda_{i}}= ( \Vert u\Vert ^{2} _{D_{s}(\Omega)}+\int_{\Omega }\lambda_{i}u^{2}\,dx ) ^{ \frac{1}{2}}$. On the one hand, by the Hölder inequality and the Young inequality, we have

$$\int_{\Omega }\vert u\vert ^{\alpha }\vert v\vert ^{\beta }\,dx\leq \frac{\alpha }{2^{ \ast }} \int_{\Omega }\vert u\vert ^{2^{\ast }}\,dx+ \frac{\beta }{2^{\ast }} \int_{\Omega }\vert v\vert ^{2^{\ast }}\,dx. $$

Hence

$$\begin{aligned} E_{\gamma }(u,v) =&\frac{1}{2} \bigl\Vert (u,v) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}+ \frac{1}{2} \int_{\Omega } \bigl(\lambda_{1}u^{2}+ \lambda_{2}v^{2} \bigr)\,dx \\ &{}-\frac{1}{2^{ \ast }} \int_{\Omega } \bigl(\mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+ \gamma \vert u\vert ^{\alpha }\vert v\vert ^{\beta } \bigr)\,dx \\ \geq& \frac{1}{2} \bigl\Vert (u,v) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}+ \frac{1}{2} \int_{\Omega } \bigl(\lambda_{1}u^{2}+ \lambda_{2}v^{2} \bigr)\,dx \\ &{}-\frac{1}{2^{ \ast }} \int_{\Omega } \bigl((\mu_{1}+\alpha \gamma)\vert u \vert ^{2^{\ast }}+(\mu _{2}+\beta \gamma)\vert v\vert ^{2^{\ast }} \bigr)\,dx \\ \geq& C_{1} \bigl\Vert (u,v) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}-C_{2} \bigl\Vert (u,v) \bigr\Vert ^{2^{ \ast }}_{\mathcal{D}_{s}(\Omega)}. \end{aligned}$$

Choose $\rho >0$ sufficiently small, if $\Vert (u,v)\Vert ^{2}_{\mathcal{D} _{s}(\Omega)}=\rho$, then

$$E_{\gamma }(u,v)\geq C_{1} \bigl\Vert (u,v) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}-C _{2} \bigl\Vert (u,v) \bigr\Vert ^{2^{\ast }}_{\mathcal{D}_{s}(\Omega)}>\frac{1}{4}C_{1} \rho^{2}>0. $$

On the other hand, we can choose $\varphi,\psi \in \mathcal{C}^{ \infty }_{0}(\Omega)$ with $\varphi,\psi \not \equiv 0$ and $\operatorname{suup}(\varphi)\cap \operatorname{suup}(\psi)=\emptyset$, then there exists $t_{0}>0$ such that $E_{\gamma }(t_{0}\varphi,t_{0}\psi)<0$ and $\Vert (t_{0}\varphi,t_{0}\psi)\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}>\rho$. Then we can take $(u_{0},v_{0})=(t_{0}\varphi,t_{0}\psi)$.

By the mountain pass theorem, for the constant $0< \widehat{A_{\gamma }}:=\inf_{\sigma \in \Gamma }\max_{t\in [0,1]}E_{\gamma }(\sigma (t))$, there exists a $(PS)_{\widehat{A_{\gamma }}}$ sequence $\{(u_{n},v_{n})\}\subset \mathcal{D}_{s}(\Omega)$, that is,

$$\begin{aligned} E_{\gamma }(u_{n},v_{n})\rightarrow \widehat{A_{\gamma }}\quad \text{and}\quad E'_{\gamma }(u_{n},v_{n}) \rightarrow 0 \quad \text{as } n\rightarrow + \infty, \end{aligned}$$

where

$$\Gamma = \bigl\{ \sigma \in \mathcal{C} \bigl([0,1],\mathcal{D}_{s}( \Omega) \bigr): \sigma (0)=(0,0),\sigma (1)=(u_{0},v_{0}) \bigr\} . $$

 □

Lemma 2.3

$\widehat{A_{\gamma }}=\inf_{\mathcal{D}_{s}(\Omega)\setminus {\{(0,0)\}}} \max_{t>0}E_{\gamma }(tu,tv)=A_{\gamma }$.

Proof

For any $(u,v)\in \mathcal{D}_{s}(\Omega)$ with $(u,v)\neq (0,0)$, there exists a unique $t_{\gamma,u,v}>0$ such that

$$\begin{aligned} \max_{t>0}E_{\gamma }(tu,tv) &=E_{\gamma }(t_{\gamma,u,v}u,t _{\gamma,u,v}v) \\ &=\frac{s}{N}t^{2}_{\gamma,u,v} \biggl( \bigl\Vert (u,v) \bigr\Vert ^{2}_{\mathcal{D}_{s}( \Omega)}+ \int_{\Omega } \bigl(\lambda_{1}u^{2}+ \lambda_{2}v^{2} \bigr)\,dx \biggr) \\ &=\frac{s}{N}t^{2^{\ast }}_{\gamma,u,v} \int_{\Omega } \bigl(\mu_{1}\vert u\vert ^{2^{ \ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+\gamma \vert u\vert ^{\alpha }\vert v\vert ^{\beta } \bigr)\,dx, \end{aligned}$$

where $t_{\gamma,u,v}>0$ satisfies

$$\begin{aligned} t^{2^{\ast }-2}_{\gamma,u,v}=\frac{\Vert (u,v)\Vert ^{2}_{\mathcal{D}_{s}( \Omega)}+\int_{\Omega }(\lambda_{1}u^{2}+\lambda_{2}v^{2})\,dx}{ \int_{\Omega }(\mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+\gamma \vert u\vert ^{ \alpha }\vert v\vert ^{\beta })\,dx}, \end{aligned}$$

(9)

which implies that $(t_{\gamma,u,v}u,t_{\gamma,u,v}v)\in \mathbb{M}$. Combining this with $\max_{t>0}E_{\gamma }(tu,tv)=E_{\gamma }(t _{\gamma,u,v}u, t_{\gamma,u,v}v)$, by the definition of $A_{\gamma }$ and $\widehat{A_{\gamma }}$, we can deduce that

$$\widehat{A_{\gamma }}=\inf_{\mathcal{D}_{s}(\Omega)\setminus {\{(0,0)\}}} \max _{t>0}E_{\gamma }(tu,tv)=A_{\gamma }. $$

 □

Lemma 2.4

$A_{\gamma }<\min \{m_{\lambda_{1},\mu_{1}}, m_{\lambda_{2}, \mu _{2}}\}<\min \{\mu^{-\frac{N-2s}{2s}}_{1}\frac{s}{N}S^{\frac{N}{2s}} _{s}, \mu^{-\frac{N-2s}{2s}}_{2}\frac{s}{N}S^{\frac{N}{2s}}_{s}\}$.

Proof

By Remark 2.1, we obtain $\min \{m_{\lambda_{1},\mu_{1}}, m _{\lambda_{2},\mu_{2}}\}<\min \{\mu^{-\frac{N-2s}{2s}}_{1}\frac{s}{N}S ^{\frac{N}{2s}}_{s}, \mu^{-\frac{N-2s}{2s}}_{2}\frac{s}{N}S^{ \frac{N}{2s}}_{s}\}$.

Next we prove that $A_{\gamma }< m_{\lambda_{1},\mu_{1}}$ and $A_{\gamma }< m_{\lambda_{2},\mu_{2}}$.

Define a function $H:\mathbb{R}^{2}\rightarrow \mathbb{R}$ by

$$H(t,\tau)=\Psi (tu_{\lambda_{1},\mu_{1}},t\tau v_{\lambda_{2},\mu _{2}}), $$

where

$$\Psi (u,v)= \bigl\Vert (u,v) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}+ \int_{\Omega } \bigl( \lambda_{1}u^{2}+ \lambda_{2}v^{2} \bigr)\,dx - \int_{\Omega } \bigl(\mu_{1}\vert u\vert ^{2^{ \ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+\gamma \vert u\vert ^{\alpha }\vert v\vert ^{\beta } \bigr)\,dx. $$

Since $H(1,0)=0$ and $H_{t}(1,0)\neq 0$, by the implicit function theorem, there exist $\delta >0$ and a function $t(\tau)\in C^{1}(- \delta,\delta)$ such that

$$t(0)=1,\qquad t'(\tau)=-\frac{H_{\tau }(t,\tau)}{H_{t}(t,\tau)} \quad \text{and}\quad H \bigl(t(\tau),\tau \bigr)=0,\quad \forall \tau \in (-\delta, \delta), $$

which implies that

$$\bigl( t(\tau)u_{\lambda_{1},\mu_{1}},t(\tau)\tau v_{\lambda_{2},\mu _{2}} \bigr) \in \mathbb{M}, \quad \forall \tau \in (-\delta,\delta). $$

Since $1<\beta <2$, by direct calculation, we have

$$\lim_{\tau \rightarrow 0}\frac{t'(\tau)}{\vert \tau \vert ^{\beta -2}\tau }=\frac{- \beta \gamma \int_{\Omega }u_{\lambda_{1},\mu_{1}}v_{\lambda_{2},\mu _{2}}\,dx}{(2^{\ast }-2)\int_{\Omega }\mu_{1}\vert u_{\lambda_{1},\mu_{1}}\vert ^{2^{ \ast }}\,dx}< 0. $$

That is,

$$t'(\tau)=\frac{-\beta \gamma \int_{\Omega }u_{\lambda_{1},\mu_{1}}v _{\lambda_{2},\mu_{2}}\,dx}{(2^{\ast }-2)\int_{\Omega }\vert u_{\lambda_{1},\mu_{1}}\vert ^{2^{\ast }}\,dx} \vert \tau \vert ^{\beta -2} \tau \bigl(1+o(1) \bigr) \quad \text{as } \tau \rightarrow 0. $$

So

$$t(\tau)=1-\frac{\gamma \int_{\Omega }u_{\lambda_{1},\mu_{1}}v_{ \lambda_{2},\mu_{2}}\,dx}{(2^{\ast }-2)\int_{\Omega }\mu_{1}\vert u_{\lambda_{1},\mu_{1}}\vert ^{2^{\ast }}\,dx}\vert \tau \vert ^{\beta } \bigl(1+o(1) \bigr)\quad \text{as } \tau \rightarrow 0. $$

Consequently, we have

$$t^{2^{\ast }}(\tau)=1-\frac{2^{\ast }\gamma \int_{\Omega }u_{\lambda _{1},\mu_{1}}v_{\lambda_{2},\mu_{2}}\,dx}{(2^{\ast }-2)\int_{\Omega }\mu _{1}\vert u_{\lambda_{1},\mu_{1}}\vert ^{2^{\ast }}\,dx}\vert \tau \vert ^{\beta } \bigl(1+o(1) \bigr) \quad \text{as } \tau \rightarrow 0. $$

Thus

$$\begin{aligned} A_{\gamma } &\leq E_{\gamma } \bigl(t(\tau)u_{\lambda_{1},\mu_{1}},t(\tau) \tau v_{\lambda_{2},\mu_{2}} \bigr)-\frac{1}{2}\Psi (tu_{\lambda_{1},\mu _{1}},t\tau v_{\lambda_{2},\mu_{2}}) \\ &\leq \frac{s}{N}t^{2^{\ast }}(\tau) \int_{\Omega } \bigl(\mu_{1}\vert u_{\lambda_{1},\mu_{1}} \vert ^{2^{\ast }}+\tau^{2^{\ast }}\mu_{2} \vert v_{\lambda_{2},\mu_{2}}\vert ^{2^{\ast }}+\gamma \tau^{\beta }\vert u_{\lambda_{1},\mu_{1}}\vert ^{\alpha } \vert v_{\lambda_{2},\mu_{2}}\vert ^{\beta } \bigr)\,dx \\ &\leq \frac{s}{N} \int_{\Omega }\mu_{1}\vert u_{\lambda_{1},\mu_{1}}\vert ^{2^{ \ast }}\,dx-\vert \tau \vert ^{\beta }\frac{\beta \gamma }{2^{\ast }} \int_{\Omega }\vert u_{\lambda_{1},\mu_{1}}\vert ^{\alpha } \vert v_{\lambda_{2},\mu_{2}}\vert ^{\beta }\,dx+o \bigl(\vert \tau \vert ^{\beta } \bigr) \\ &< \frac{s}{N} \int_{\Omega }\mu_{1}\vert u_{\lambda_{1},\mu_{1}}\vert ^{2^{ \ast }}\,dx=m_{\lambda_{1},\mu_{1}}, \quad\text{as } \vert \tau \vert >0 \text{ small enough}. \end{aligned}$$

Hence $A_{\gamma }< m_{\lambda_{1},\mu_{1}}$. Similarly, by the same arguments, we have $A_{\gamma }< m_{\lambda_{2},\mu_{2}}$.

This completes the proof of Lemma 2.4. □

3 Proof of Theorem 1.1

In this section, we prove the first part of Theorem 1.1 by two steps. We prove the existence of ground state solutions for system (3) in step one, then we claim there exist a positive ground state solutions. Finally, we prove the second part of Theorem 1.1.

Proof

Proof of the first part of Theorem  1.1 .

Step one. Prove the existence of ground state solutions for system (3).

By (8) and Lemma 2.3, there exists $\{(u_{n},v_{n})\}\subset \mathcal{D}_{s}(\Omega)$ such that

$$\lim_{n\rightarrow +\infty }E_{\gamma }(u_{n},v_{n})=A_{\gamma }, \qquad \lim_{n\rightarrow +\infty }E'_{\gamma }(u_{n},v_{n})=0. $$

Next, we claim $\{(u_{n},v_{n})\}$ is bounded in $\mathcal{D}_{s}( \Omega)$.

Let $z_{n}=\{(u_{n},v_{n})\}$, assuming by contradiction that $\|z_{n}\|:=\|z_{n}\|_{\mathcal{D}_{s}(\Omega)}\rightarrow +\infty$ as $n\rightarrow +\infty$. Put

$$\widetilde{z_{n}}=(\widetilde{u_{n}},\widetilde{v_{n}})= \frac{z_{n}}{ \Vert z_{n}\Vert }= \biggl(\frac{u_{n}}{\Vert z_{n}\Vert },\frac{v_{n}}{\Vert z_{n}\Vert } \biggr). $$

Since $\{z_{n}\}$ is a $(PS)_{A_{\gamma }}$ sequence for $E_{\gamma }$ and $\|z_{n}\|\rightarrow +\infty$ as $n\rightarrow +\infty$, we have

$$\begin{aligned} &\frac{\Vert z_{n}\Vert ^{2}}{2} \bigl\Vert (\widetilde{u_{n}}, \widetilde{v_{n}}) \bigr\Vert ^{2} _{\mathcal{D}_{s}(\Omega)}+ \frac{\Vert z_{n}\Vert ^{2}}{2} \int_{\Omega } \bigl( \lambda_{1}\vert \widetilde{u_{n}}\vert ^{2}+\lambda_{2}\vert \widetilde{v_{n}}\vert ^{2} \bigr)\,dx \\ &\quad {}-\frac{\Vert z_{n}\Vert ^{2^{\ast }}}{2^{\ast }} \int_{\Omega } \bigl(\mu_{1}\vert \widetilde{u_{n}} \vert ^{2^{\ast }}+\mu_{2}\vert \widetilde{v_{n}} \vert ^{2^{\ast }}+ \gamma \vert \widetilde{u_{n}}\vert ^{\alpha }\vert \widetilde{v_{n}}\vert ^{\beta } \bigr) \,dx=A _{\gamma }+o_{n}(1), \end{aligned}$$

(10)

$$\begin{aligned} &\Vert z_{n}\Vert ^{2} \bigl\Vert ( \widetilde{u_{n}},\widetilde{v_{n}}) \bigr\Vert ^{2}_{ \mathcal{D}_{s}(\Omega)}+\Vert z_{n}\Vert ^{2} \int_{\Omega } \bigl(\lambda_{1}\vert \widetilde{u_{n}}\vert ^{2}+\lambda_{2}\vert \widetilde{v_{n}}\vert ^{2} \bigr)\,dx \\ &\quad -\Vert z_{n}\Vert ^{2^{\ast }} \int_{\Omega } \bigl(\mu_{1}\vert \widetilde{u_{n}} \vert ^{2^{ \ast }}+\mu_{2}\vert \widetilde{v_{n}} \vert ^{2^{\ast }}+\gamma \vert \widetilde{u_{n}}\vert ^{\alpha }\vert \widetilde{v_{n}}\vert ^{\beta } \bigr) \,dx=o_{n}(1). \end{aligned}$$

(11)

Combining (10) with (11), we obtain

$$\frac{s}{N}\Vert z_{n}\Vert ^{2^{\ast }-2} \int_{\Omega } \bigl(\mu_{1}\vert \widetilde{u_{n}} \vert ^{2^{\ast }}+\mu_{2}\vert \widetilde{v_{n}} \vert ^{2^{\ast }}+ \gamma \vert \widetilde{u_{n}}\vert ^{\alpha }\vert \widetilde{v_{n}}\vert ^{\beta } \bigr) \,dx=o _{n}(1), $$

as $n\rightarrow +\infty$, we have a contradiction. Consequently, $\{(u_{n},v_{n})\}$ is bounded in $\mathcal{D}_{s}( \Omega)$. Thus, by the Sobolev embedding theorem, there exist $(u,v)\in \mathcal{D}_{s}(\Omega)$ such that

$$ \textstyle\begin{cases} (u_{n},v_{n})\rightharpoonup (u,v), & \text{weakly in } \mathcal{D}_{s}(\Omega), \\ (u_{n},v_{n})\rightarrow (u,v), & \text{strongly in } L^{p}(\Omega)\times L^{p}(\Omega), \text{for } 2\leq p< 2^{\ast }, \\ (u_{n},v_{n})\rightarrow (u,v), & \text{a.e. } \Omega. \end{cases} $$

(12)

Consequently, we have $E'_{\gamma }(u,v)=0$. Set $w_{n}=u_{n}-u$ and $\sigma_{n}=v_{n}-v$. Then, by the Brézis–Lieb lemma [30],

$$\begin{aligned} \begin{aligned} &\Vert u_{n}\Vert ^{2^{\ast }}_{2^{\ast }}= \Vert u\Vert ^{2^{\ast }}_{2^{\ast }}+\Vert w_{n}\Vert ^{2^{\ast }}_{2^{\ast }}+o_{n}(1), \\ & \Vert v_{n}\Vert ^{2^{\ast }}_{2^{ \ast }}=\Vert v\Vert ^{2^{\ast }}_{2^{\ast }}+\Vert \sigma_{n}\Vert ^{2^{\ast }}_{2^{ \ast }}+o_{n}(1). \end{aligned} \end{aligned}$$

(13)

By Lemma 2.1 in [31], we also have

$$\begin{aligned} \int_{\Omega }\vert w_{n}\vert ^{\alpha }\vert \sigma_{n}\vert ^{\beta }\,dx= \int_{\Omega }\vert u_{n}\vert ^{\alpha } \vert v_{n}\vert ^{\beta }\,dx- \int_{\Omega }\vert u\vert ^{\alpha }\vert v\vert ^{ \beta }\,dx+o_{n}(1). \end{aligned}$$

(14)

We have

$$\begin{aligned} \begin{aligned} &\Vert w_{n}\Vert ^{2}_{D_{s}(\Omega)}=\Vert u_{n}\Vert ^{2}_{D_{s}(\Omega)}-\Vert u\Vert ^{2}_{D_{s}(\Omega)}+o_{n}(1), \\ &\Vert \sigma_{n}\Vert ^{2}_{D_{s}(\Omega)}=\Vert v_{n}\Vert ^{2}_{D_{s}(\Omega)}-\Vert v\Vert ^{2}_{D_{s}(\Omega)}+o_{n}(1), \end{aligned} \end{aligned}$$

(15)

and $E'_{\gamma }(u_{n},v_{n})\rightarrow 0$ as $n\rightarrow +\infty$. Combining this with (13), (14) and (15), we obtain

$$\begin{aligned} \bigl\Vert (w_{n},\sigma_{n}) \bigr\Vert ^{2}_{D_{s}(\Omega)}= \int_{\Omega } \bigl(\mu_{1}\vert w_{n} \vert ^{2^{\ast }}+\mu_{2}\vert \sigma_{n}\vert ^{2^{\ast }}+\gamma \vert w_{n}\vert ^{ \alpha }\vert \sigma_{n}\vert ^{\beta } \bigr)\,dx+o_{n}(1) \end{aligned}$$

(16)

and

$$\begin{aligned} E_{\gamma }(u_{n},v_{n}) =&E_{\gamma }(u,v)+ \frac{1}{2} \bigl\Vert (w_{n},\sigma_{n}) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)} \\ &{}-\frac{1}{2^{\ast }} \int_{ \Omega } \bigl(\mu_{1}\vert w_{n} \vert ^{2^{\ast }}+\mu_{2}\vert \sigma_{n}\vert ^{2^{\ast }}+ \gamma \vert w_{n}\vert ^{\alpha }\vert \sigma_{n}\vert ^{\beta } \bigr)\,dx+o_{n}(1). \end{aligned}$$

(17)

By (16) and (17), we have

$$\begin{aligned} E_{\gamma }(u_{n},v_{n})=E_{\gamma }(u,v)+ \frac{s}{N} \bigl\Vert (w_{n},\sigma_{n}) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}+o_{n}(1). \end{aligned}$$

(18)

Next, we prove that $(u_{n},v_{n})\rightarrow (u,v)$ strongly in $\mathcal{D}_{s}(\Omega)$. Let

$$\lim_{n\rightarrow +\infty } \bigl\Vert (w_{n},0) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}=l_{1},\qquad \lim_{n\rightarrow +\infty } \bigl\Vert (0,\sigma_{n}) \bigr\Vert ^{2}_{ \mathcal{D}_{s}(\Omega)}=l_{2}, \qquad \lim_{n\rightarrow +\infty } \bigl\Vert (w_{n}, \sigma_{n}) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}=l, $$

if $l=0$, then we have proved $(u_{n},v_{n})\rightarrow (u,v)$ strongly in $\mathcal{D}_{s}(\Omega)$, if $l>0$ then

$$\lim_{n\rightarrow +\infty } \bigl\Vert (w_{n}, \sigma_{n}) \bigr\Vert ^{2}_{\mathcal{D} _{s}(\Omega)}=l_{1}+l_{2} \geq \max \{l_{1}, l_{2}\}. $$

Case one $l_{1}=0$ or $l_{2}=0$.

If $l_{2}=0$, then (16) turns to

$$\begin{aligned} \Vert w_{n}\Vert ^{2}_{D_{s}(\Omega)}= \int_{\Omega }\mu_{1}\vert w_{n}\vert ^{2^{ \ast }}\,dx+o_{n}(1). \end{aligned}$$

(19)

By the Sobolev embedding $D_{s}(\Omega)\hookrightarrow L^{2^{\ast }}( \Omega)$, we have $\|w_{n}\|^{2}_{D_{s}(\Omega)}\geq S_{s} ( \int _{\Omega }|w_{n}|^{2^{\ast }}\,dx ) ^{\frac{2}{2^{\ast }}}$, hence $\|w_{n}\|^{2}_{D_{s}(\Omega)}\geq \mu^{-\frac{2}{2^{\ast }}}_{1}S _{s} ( \int_{\Omega }\mu_{1}|w_{n}|^{2^{\ast }}\,dx ) ^{\frac{2}{2^{ \ast }}}$ combining this with (19), we can deduce that

$$l_{1}\geq \mu^{-\frac{N-2s}{2s}}_{1} S^{\frac{N}{2s}}_{s}. $$

Similarly, if $l_{1}=0$, we have

$$l_{2}\geq \mu^{-\frac{N-2s}{2s}}_{2} S^{\frac{N}{2s}}_{s}. $$

Since $E_{\gamma }(u,v)\geq 0$, let $n\rightarrow +\infty$ in (18), we obtain

$$A_{\gamma }\geq \max \biggl\{ \mu^{-\frac{N-2s}{2s}}_{1} \frac{s}{N}S^{ \frac{N}{2s}}_{s}, \mu^{-\frac{N-2s}{2s}}_{2} \frac{s}{N}S^{ \frac{N}{2s}}_{s} \biggr\} . $$

This contradicts Lemma 2.4. Thus $E_{\gamma }(u,v)=A_{\gamma }$ and $E'_{\gamma }(u,v)=0$. That is $(u,v)$ is a non-trivial solution of system (3).

Case two $l_{1}\neq 0$, $l_{2}\neq 0$ and $l>0$, we prove system (3) has a ground state solution.

For case two, in order to obtain a ground state solution for (3), we borrow some ideas from [11]. First we give the following lemma.

Lemma 3.1

(A result in [11])

Define

$$\begin{aligned}& \widetilde{S}_{s}=\inf_{(u,v)\in \mathcal{D}_{s}(\mathbb{R}^{N})\setminus \{(0,0)\}}\frac{\Vert (u,v)\Vert ^{2} _{\mathcal{D}_{s}(\Omega)}}{ ( \int_{\mathbb{R}^{N}}(\mu_{1}\vert u\vert ^{2^{ \ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+\gamma \vert u\vert ^{\alpha }\vert v\vert ^{\beta })\,dx ) ^{\frac{2}{2^{\ast }}}}, \\& \widetilde{S}_{s,\lambda_{1},\lambda_{2}}:=\inf_{(u,v)\in \mathcal{D}_{s}(\Omega)\setminus \{(0,0)\}}\frac{\Vert (u,v)\Vert ^{2} _{\mathcal{D}_{s}(\Omega)}+\int_{\Omega }(\lambda_{1}u^{2}+\lambda _{2}v^{2})\,dx}{ ( \int_{\Omega }(\mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{ \ast }}+\gamma \vert u\vert ^{\alpha }\vert v\vert ^{\beta })\,dx ) ^{\frac{2}{2^{ \ast }}}}. \end{aligned}$$

Then

$$\widetilde{S}_{s,\lambda_{1},\lambda_{2}}< \widetilde{S}_{s}. $$

Let $\{(u_{n},u_{n})\}$ be a minimizing sequence for $S_{s,\lambda _{1},\lambda_{2}}$ normalized by

$$\begin{aligned} \int_{\Omega } \bigl(\mu_{1}\vert u_{n} \vert ^{2^{\ast }}+\mu_{2}\vert v_{n}\vert ^{2^{\ast }}+ \gamma \vert u_{n}\vert ^{\alpha }\vert v_{n}\vert ^{\beta } \bigr)\,dx=1, \end{aligned}$$

(20)

that is,

$$\begin{aligned} \bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}+ \int_{\Omega } \bigl( \lambda_{1}u_{n}^{2}+ \lambda_{2}v_{n}^{2} \bigr)\,dx= \widetilde{S}_{s,\lambda _{1},\lambda_{2}}+o_{n}(1). \end{aligned}$$

(21)

Since $\{u_{n}\}$ and $\{v_{n}\}$ are bounded in $D_{s}(\Omega)$, (12) holds and

$$\int_{\Omega } \bigl(\mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+\gamma \vert u\vert ^{ \alpha }\vert v\vert ^{\beta } \bigr)\,dx\leq 1. $$

By (21), we have

$$\widetilde{S}_{s,\lambda_{1},\lambda_{2}}- \int_{\Omega } \bigl(\lambda_{1}u _{n}^{2}+ \lambda_{2}v_{n}^{2} \bigr)\,dx+o_{n}(1) \geq \bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{2} _{\mathcal{D}_{s}(\Omega)}\geq \widetilde{S}_{s}. $$

By (12) and Lemma 3.1, we have

$$- \int_{\Omega } \bigl(\lambda_{1}u^{2}+ \lambda_{2}v^{2} \bigr)\,dx\geq \widetilde{S}_{s}- \widetilde{S}_{s,\lambda_{1},\lambda_{2}}>0, $$

which implies that $(u,v)\not \equiv (0,0)$. By (15) and (21), we obtain

$$\begin{aligned} \widetilde{S}_{s,\lambda_{1},\lambda_{2}}= \bigl\Vert (w_{n}, \sigma_{n}) \bigr\Vert ^{2} _{D_{s}(\Omega)}+ \bigl\Vert (u,v) \bigr\Vert ^{2}_{D_{s}(\Omega)}+ \int_{\Omega } \bigl( \lambda_{1}u^{2}+ \lambda_{2}v^{2} \bigr)\,dx+o_{n}(1). \end{aligned}$$

(22)

Combining (13), (14) with (20), we have

$$\begin{aligned} 1 =& \int_{\Omega } \bigl(\mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+\gamma \vert u\vert ^{\alpha }\vert v\vert ^{\beta } \bigr)\,dx \\ &{}+ \int_{\Omega } \bigl(\mu_{1}\vert w_{n} \vert ^{2^{\ast }}+ \mu_{2}\vert \sigma_{n}\vert ^{2^{\ast }}+\gamma \vert w_{n}\vert ^{\alpha }\vert \sigma_{n}\vert ^{ \beta } \bigr)\,dx+o_{n}(1). \end{aligned}$$

Since

$$\int_{\Omega } \bigl(\mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+\gamma \vert u\vert ^{ \alpha }\vert v\vert ^{\beta } \bigr)\,dx\leq 1 $$

and

$$\int_{\Omega } \bigl(\mu_{1}\vert w_{n} \vert ^{2^{\ast }}+\mu_{2}\vert \sigma_{n}\vert ^{2^{ \ast }}+\gamma \vert w_{n}\vert ^{\alpha }\vert \sigma_{n}\vert ^{\beta } \bigr)\,dx\leq 1, $$

we have

$$\begin{aligned} 1 \leq & \biggl( \int_{\Omega } \bigl(\mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{ \ast }}+\gamma \vert u\vert ^{\alpha }\vert v\vert ^{\beta } \bigr)\,dx \biggr) ^{\frac{2}{2^{ \ast }}} \\ &{}+ \biggl( \int_{\Omega } \bigl(\mu_{1}\vert w_{n} \vert ^{2^{\ast }}+\mu_{2}\vert \sigma_{n}\vert ^{2^{ \ast }}+\gamma \vert w_{n}\vert ^{\alpha }\vert \sigma_{n}\vert ^{\beta } \bigr)\,dx \biggr) ^{\frac{2}{2^{ \ast }}} \\ \leq& \biggl( \int_{\Omega } \bigl(\mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{ \ast }}+\gamma \vert u\vert ^{\alpha }\vert v\vert ^{\beta } \bigr)\,dx \biggr) ^{\frac{2}{2^{ \ast }}} \\ &{}+\frac{1}{\widetilde{S}_{s}} \bigl\Vert (w_{n},\sigma_{n}) \bigr\Vert ^{2}_{\mathcal{D} _{s}(\Omega)}+o_{n}(1). \end{aligned}$$

(23)

Combining (23), (22), Lemma 3.1 with $\widetilde{S}_{s,\lambda_{1},\lambda_{2}}>0$, we have

$$\begin{aligned} & \bigl\Vert (u,v) \bigr\Vert ^{2}_{D_{s}(\Omega)}+ \int_{\Omega } \bigl(\lambda_{1}u^{2}+ \lambda_{2}v^{2} \bigr)\,dx \\ &\quad \leq \widetilde{S}_{s,\lambda_{1},\lambda_{2}} \biggl( \int_{\Omega } \bigl( \mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+\gamma \vert u\vert ^{\alpha }\vert v\vert ^{ \beta } \bigr)\,dx \biggr) ^{\frac{2}{2^{\ast }}} \\ &\qquad {}+ \biggl( \frac{\widetilde{S}_{s,\lambda_{1},\lambda_{2}}}{ \widetilde{S}_{s}}-1 \biggr) \bigl\Vert (w_{n}, \sigma_{n}) \bigr\Vert ^{2}_{\mathcal{D} _{s}(\Omega)}+o_{n}(1) \\ &\quad \leq \widetilde{S}_{s,\lambda_{1},\lambda_{2}} \biggl( \int_{\Omega } \bigl( \mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{\ast }}+\gamma \vert u\vert ^{\alpha }\vert v\vert ^{ \beta } \bigr)\,dx \biggr) ^{\frac{2}{2^{\ast }}}+o_{n}, \end{aligned}$$

which implies that

$$\frac{\Vert (u,v)\Vert ^{2}_{D_{s}(\Omega)}+\int_{\Omega }(\lambda_{1}u^{2}+ \lambda_{2}v^{2})\,dx}{ ( \int_{\Omega }(\mu_{1}\vert u\vert ^{2^{\ast }}+\mu _{2}\vert v\vert ^{2^{\ast }}+\gamma \vert u\vert ^{\alpha }\vert v\vert ^{\beta })\,dx ) ^{\frac{2}{2^{ \ast }}}}\leq \widetilde{S}_{s,\lambda_{1},\lambda_{2}}. $$

Therefore, $\widetilde{S}_{s,\lambda_{1},\lambda_{2}}$ is attained by $(u,v)$. Thus, system (3) has a ground state solution.

Combining case one with case two, we prove that system (3) has a ground state solution.

Step two. We claim that there exists a positive ground state solution. Since

$$\begin{aligned} & \int_{\mathbb{R}^{N}} \int_{\mathbb{R}^{N}} \frac{\vert u(x)-u(y)\vert ^{2}}{\vert x-y\vert ^{N+2s}}\,dx\,dy- \int_{\mathbb{R}^{N}} \int_{\mathbb{R}^{N}}\frac{\vert \vert u(x)\vert -\vert u(y)\vert \vert ^{2}}{\vert x-y\vert ^{N+2s}}\,dx\,dy \\ &\quad = {2 \int _{\mathbb{R}^{N}} \int _{\mathbb{R}^{N}}\frac{\vert u(x)\vert \vert u(y)\vert -u(x)u(y)}{\vert x-y\vert ^{N+2s}}\,dx\,dy\geq 0}, \end{aligned}$$

we have

$$|\!|\!|u|\!|\!|_{D_{s}(\Omega)}\leq \Vert u\Vert _{D_{s}(\Omega)}. $$

Then, for the minimizing sequence $(u_{n},v_{n})\in \mathbb{M}$, we have

$$\begin{aligned} &|\!|\!|u_{n}|\!|\!|^{2}_{D_{s}(\Omega)}+|\!|\!|v_{n}|\!|\!|^{2}_{D_{s}(\Omega)}+ \int_{\Omega } \bigl(\lambda_{1}\vert u_{n} \vert ^{2}+\lambda_{2}\vert v_{n}\vert ^{2} \bigr)\,dx \\ &\quad \leq \Vert u_{n}\Vert ^{2}_{D_{s}(\Omega)}+ \Vert v_{n}\Vert ^{2}_{D_{s}(\Omega)}+ \int_{\Omega } \bigl(\lambda_{1}u^{2}_{n}+ \lambda_{2}v^{2}_{n} \bigr)\,dx \\ &\quad = \int_{\Omega } \bigl(\mu_{1}\vert u_{n} \vert ^{2^{\ast }}+\mu_{2}\vert v_{n}\vert ^{2^{ \ast }}+\gamma \vert u_{n}\vert ^{\alpha }\vert v_{n}\vert ^{\beta } \bigr)\,dx, \end{aligned}$$

this implies that there exists $t_{n}\in (0,1]$ such that $(t_{n}|u _{n}|,t_{n}|v_{n}|)\in \mathbb{M}$. Hence, we can choose a minimizing sequence $(\overline{u}_{n},\overline{v}_{n})=(t_{n}|u_{n}|,t_{n}|v _{n}|)$ and the weak limit $(\overline{u},\overline{v})$ is nonnegative. By the strong maximum principle for the fractional Laplacian (see Proposition 2.17 in [4]), we have u̅ and v̅ are both positive.

Proof of the second part of Theorem 1.1

Let $\gamma_{n}$ be a sequence with $\gamma_{n}\rightarrow 0$ as $n\rightarrow +\infty$. $\{(u_{\gamma _{n}},v_{\gamma _{n}})\}$ is bounded in $D_{s}(\Omega)\times D_{s}(\Omega)$, then there exists a subsequence, still denoted by $\{(u_{\gamma _{n}},v_{\gamma _{n}})\}$, such that $(u_{\gamma _{n}},v_{\gamma _{n}})\rightharpoonup (\overline{u},\overline{v})$ weakly in $D_{s}(\Omega)\times D_{s}(\Omega)$. Then $(\overline{u},\overline{v})$ satisfies

$$ \textstyle\begin{cases} (-\Delta)^{s}\overline{u}+\lambda_{1}\overline{u}=\mu_{1}| \overline{u}|^{2^{\ast }-2}\overline{u} & \text{in } \Omega, \\ (-\Delta)^{s}\overline{v}+\lambda_{2}\overline{v}= \mu_{2}| \overline{v}|^{2^{\ast }-2}\overline{v} & \text{in } \Omega, \\ \overline{u}=\overline{v}=0 & \text{in } \mathbb{R}^{N}\setminus \Omega. \end{cases} $$

(24)

Since $E'_{0}(u_{\gamma_{n}},v_{\gamma_{n}})\rightarrow 0$ and $\lim_{n\rightarrow +\infty }E_{0}(u_{\gamma_{n}},v_{\gamma _{n}})=\lim_{n\rightarrow +\infty }E_{\gamma_{n}}(u_{\gamma _{n}},v_{\gamma_{n}})$, we have

$$E'_{0}(\overline{u},\overline{v})=0\quad \text{and} \quad \lim_{n\rightarrow +\infty }E_{0}(u_{\gamma_{n}},v_{\gamma_{n}})= \lim_{n\rightarrow +\infty }A_{\gamma_{n}}>0. $$

Next, we claim $A_{\gamma }$ is strictly decreasing for all $\gamma >0$.

Let $\gamma_{2}>\gamma_{1}>0$, then, by (9), we have

$$\begin{aligned} t^{2^{\ast }-2}_{\gamma_{2},u_{\gamma_{1}},v_{\gamma_{1}}} &=\frac{\Vert (u_{\gamma_{1}},v_{\gamma_{1}})\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}+ \int_{\Omega }(\lambda_{1}u_{\gamma_{1}}^{2}+\lambda_{2}v_{\gamma_{1}} ^{2})\,dx}{\int_{\Omega }(\mu_{1}\vert u_{\gamma_{1}}\vert ^{2^{\ast }}+\mu_{2}\vert v_{\gamma_{1}}\vert ^{2^{\ast }}+\gamma_{2} \vert u_{\gamma_{1}}\vert ^{\alpha }\vert v_{\gamma_{2}}\vert ^{\beta })\,dx} \\ & < \frac{\Vert (u_{\gamma_{1}},v_{\gamma_{1}})\Vert ^{2}_{ \mathcal{D}_{s}(\Omega)}+\int_{\Omega }(\lambda_{1}u_{\gamma_{1}} ^{2}+\lambda_{2}v_{\gamma_{1}}^{2})\,dx}{\int_{\Omega }(\mu_{1}\vert u_{\gamma_{1}}\vert ^{2^{\ast }}+\mu_{2} \vert v_{\gamma_{1}}\vert ^{2^{\ast }}+\gamma _{1}\vert u_{\gamma_{1}}\vert ^{\alpha }\vert v_{\gamma_{2}}\vert ^{\beta })\,dx}=1. \end{aligned}$$

Consequently,

$$\begin{aligned} A_{\gamma_{2}}&\leq \max_{t>0}E_{\gamma_{2}}(tu_{\gamma_{1}},tv_{\gamma _{1}}) \\ &=\frac{s}{N}t^{2^{\ast }-2}_{\gamma_{2},u_{\gamma_{1}},v_{ \gamma_{1}}} \biggl( \bigl\Vert (u_{\gamma_{1}},v_{\gamma_{1}}) \bigr\Vert ^{2}_{ \mathcal{D}_{s}(\Omega)}+ \int_{\Omega } \bigl(\lambda_{1}u_{\gamma_{1}} ^{2}+\lambda_{2}v_{\gamma_{1}}^{2} \bigr)\,dx \biggr) \\ &=t^{2^{\ast }-2}_{\gamma_{2},u_{\gamma_{1}},v_{\gamma_{1}}}A_{\gamma _{1}}< A_{\gamma_{1}}. \end{aligned}$$

Hence, $A_{\gamma }$ is strictly decreasing for $\gamma >0$. By Lemma 2.4 and the strictly decreasing for $A_{\gamma }$, we have

$$\begin{aligned} 0< \lim_{n\rightarrow +\infty }A_{\gamma_{n}}\leq A_{0}\leq \min \{m_{\lambda_{1},\mu_{1}},m_{\lambda_{2},\mu_{2}}\}< \min \biggl\{ \mu^{-\frac{N-2s}{2s}}_{1}\frac{s}{N}S^{\frac{N}{2s}}_{s}, \mu^{-\frac{N-2s}{2s}}_{2}\frac{s}{N}S^{\frac{N}{2s}}_{s} \biggr\} . \end{aligned}$$

(25)

By the same arguments as prove the first part of Theorem 1.1, we have

$${(u_{\gamma _{n}},v_{\gamma _{n}})\rightarrow (\overline{u},\overline{v})} \quad \text{strongly in } D_{s}(\Omega)\times D_{s}(\Omega). $$

Combining this with (25), one of the following conclusions holds:

• (1) $(\overline{u},0)$ is a positive ground state solution of

  $$\textstyle\begin{cases} (-\Delta)^{s}u+\lambda_{1} u=\mu_{1}|u|^{2^{\ast }-2}u & \text{in } \Omega; \\ u=0 & \text{on } \mathbb{R}^{N}\setminus \Omega. \end{cases} $$

• (2) $(0,\overline{v})$ is a positive ground state solution of

  $$\textstyle\begin{cases} (-\Delta)^{s}v+\lambda_{2} v=\mu_{2}|v|^{2^{\ast }-2}v & \text{in } \Omega, \\ v=0 & \text{on } \mathbb{R}^{N}\setminus \Omega. \end{cases} $$

Since

$$m_{\lambda _{1},\mu _{1}}=\mu ^{-\frac{N-2s}{2s}}_{1}m_{\lambda _{1}},\qquad m_{\lambda _{2},\mu _{2}}=\mu ^{-\frac{N-2s}{2s}}_{2}m_{\lambda _{2}} $$

and

$$\biggl(\frac{\mu _{1}}{\mu _{2}} \biggr)^{-\frac{N-2s}{2s}}< \frac{m_{\lambda _{2}}}{m_{\lambda _{1}}} \quad \text{implies that}\quad m_{\lambda _{1},\mu _{1}}< m_{\lambda _{2},\mu _{2}}, $$

by the definition of $A_{\gamma }$, we know that (1) holds.

Similarly, if

$$\biggl(\frac{\mu _{1}}{\mu _{2}} \biggr)^{-\frac{N-2s}{2s}}>\frac{m_{\lambda _{2}}}{m_{\lambda _{1}}}\quad \text{implies that}\quad m_{\lambda _{1},\mu _{1}}>m_{\lambda _{2},\mu _{2}}, $$

then (2) occurs. This completes the proof of Theorem 1.1.  □

4 Proof of Theorem 1.2

Define

$$X=S_{1}\times S_{2}, $$

where

$$\begin{aligned} {S_{i}:= \bigl\{ u\in D_{s}(\Omega): J'_{\lambda _{i},\mu _{i}}(u)=0, J_{\lambda _{i},\mu _{i}}(u)=m_{\lambda _{i},\mu _{i}} \bigr\} ,} \end{aligned}$$

(26)

for $i=1,2$. Then we have following lemma.

Lemma 4.1

X is compact in $\mathcal{D}_{s}(\Omega)$ and there exist constants $C_{2}>C_{1}>0$ such that

$$C_{1}\leq \Vert u\Vert ^{2}_{D_{s}(\Omega)}+ \int_{\Omega }\lambda_{1}u^{2}\,dx,\qquad \Vert v \Vert ^{2}_{D_{s}(\Omega)}+ \int_{\Omega }\lambda_{2}v^{2}\,dx\leq C _{2}, \quad \forall (u,v)\in X. $$

Proof

By Remark 2.1, we know $S_{i}$ is nonempty and $(u_{\lambda _{1},\mu_{1}},v_{\lambda_{2},\mu_{2}})\in X$. Next, we claim $S_{i}$ is compact in $\mathcal{D}_{s}(\Omega)$. Suppose there exists a sequence $\{u_{n}\}\subset S_{1}$, then $\{u_{n}\}$ is a bounded $(PS)_{m_{\lambda_{1},\mu _{1}}}$ sequence of $J_{\lambda_{1},\mu_{1}}$ and

$$\Vert u_{n}\Vert ^{2}_{D_{s}(\Omega)}+ \int_{\Omega }\lambda_{1}u_{n}^{2} \,dx= \int_{\Omega }\mu_{1}u_{n}^{2^{\ast }} \,dx+o_{n}(1). $$

Thus, there exists a subsequence $u_{\infty }$ such that $u_{n}\rightharpoonup u_{\infty }$ in $D_{s}(\Omega)$ and $J'_{\lambda_{1},\mu_{1}}(u_{ \infty })=0$.

Since $m_{\lambda_{1},\mu_{1}}\leq \mu^{-\frac{N-2s}{2s}}_{1} \frac{s}{N}S^{\frac{N}{2s}}_{s}$ and $J_{\lambda_{1},\mu_{1}}$ satisfies the $(PS)_{m_{\lambda_{1},\mu_{1}}}$ condition, by the same arguments as proving step one in Theorem 1.1, we can obtain $u_{n}\rightarrow u_{\infty }$ strongly in $\mathcal{D}_{s}(\Omega)$ and $u_{\infty } \in S_{1}$. This proves that $S_{1}$ is compact in $\mathcal{D}_{s}( \Omega)$. Similarly, $S_{2}$ is compact in $\mathcal{D}_{s}(\Omega)$.

Since $X=S_{1}\times S_{2}$ and $m_{\lambda_{1},\mu _{1}}>0$, $m_{\lambda_{2},\mu_{2}}>0$, it is easy to see that X is compact and Lemma 4.1 holds. □

By Lemma 2.3 and Remark 2.1, we have

$$\begin{aligned}& \begin{aligned} &J_{\lambda_{1},\mu_{1}}(u_{\lambda_{1},\mu_{1}})=\max_{t>0}J _{\lambda_{1},\mu_{1}}(tu_{\lambda_{1},\mu_{1}})=m_{\lambda_{1},\mu _{1}}, \\ & J_{\lambda_{2},\mu_{2}}(v_{\lambda_{2},\mu_{2}})= \max _{s>0}J _{\lambda_{2},\mu_{2}}(sv_{\lambda_{2},\mu_{2}})=m_{\lambda_{2},\mu _{2}}. \end{aligned} \end{aligned}$$

(27)

Thus, there exist $0< t_{0}<1<t_{1}$, $0< s_{0}<1<s_{1}$ such that

$$\begin{aligned} &J_{\lambda_{1},\mu_{1}}(tu_{\lambda_{1},\mu_{1}})\leq \frac{m_{ \lambda_{1},\mu_{1}}}{4} \quad \text{for } t\in (0,t_{0}]\cup [t_{1},+ \infty ), \end{aligned}$$

(28)

$$\begin{aligned} & J_{\lambda_{2},\mu_{2}}(sv_{\lambda_{2},\mu_{2}})\leq \frac{m_{ \lambda_{2},\mu_{2}}}{4} \quad \text{for } s \in (0,s_{0}]\cup [s_{1},+ \infty). \end{aligned}$$

(29)

Define

$$\begin{aligned} \widetilde{\sigma }_{1}(t):=tu_{\lambda_{1},\mu_{1}} \quad \text{for } 0 \leq t \leq t_{1},\qquad \widetilde{\sigma }_{2}(s):=sv_{\lambda_{2},\mu_{2}} \quad \text{for } 0 \leq s\leq s_{1}, \end{aligned}$$

and

$$\widetilde{\sigma }(t,s):= \bigl(\widetilde{\sigma }_{1}(t), \widetilde{\sigma }_{2}(s) \bigr). $$

Then, there exists a constant $C_{0}>0$ such that

$$\begin{aligned} \max_{(t,s)\in [0,t_{1}]\times [0,s_{1}]} \bigl\Vert \widetilde{\sigma }(t,s) \bigr\Vert _{\mathcal{D}_{s}(\Omega)}\leq C_{0}. \end{aligned}$$

(30)

For convenience we denote $Q=[0,t_{1}]\times [0,s_{1}]$. For $\gamma \geq 0$ and $C_{2}$ as appearing in Lemma 4.1, we define

$$\widehat{c_{\gamma }}:=\inf_{\sigma \in \widehat{\Gamma }} \max _{(t,s)\in Q}E_{\gamma } \bigl(\sigma (t,s) \bigr),\qquad d_{\gamma }= \max_{(t,s)\in Q}E_{\gamma } \bigl(\widetilde{ \sigma }(t,s) \bigr), $$

where

$$\begin{aligned} \widehat{\Gamma }:={}& \Bigl\{ \sigma \in \mathcal{C} \bigl(Q, \mathcal{D}_{s}( \Omega) \bigr):\max_{(t,s)\in Q} \bigl\Vert \sigma (t,s) \bigr\Vert _{\mathcal{D}_{s}( \Omega)}\leq C_{0} +2C_{2} \\ &{} \sigma (t,s)=\widetilde{\sigma }(t,s),\text{for } (t,s)\in Q \setminus \bigl\{ (t_{0},t_{1})\times (s_{0},s_{1}) \bigr\} \Bigr\} . \end{aligned}$$

(31)

Since $\widetilde{\sigma }(t,s)\in \widehat{\Gamma }$, Γ̂ is nonempty.

Lemma 4.2

$\lim_{\gamma \rightarrow 0}\widehat{c_{\gamma }}=\lim_{\gamma \rightarrow 0}d_{\gamma }=\widehat{c_{0}}=m_{\lambda _{1},\mu_{1}}+m_{\lambda_{2},\mu_{2}}$.

Proof

On the one hand, since $\gamma >0$, we have $E_{\gamma }( \widetilde{\sigma }(t,s))\leq E_{0}(\widetilde{\sigma }(t,s))$. Consequently

$$\begin{aligned} d_{\gamma } &\leq d_{0}=\max_{(t,s)\in Q}E_{0} \bigl( \widetilde{\sigma }(t,s) \bigr)=\max_{t\in [0,t_{1}]}J_{\lambda_{1}, \mu_{1}} \bigl(\widetilde{\sigma }_{1}(t) \bigr)+\max_{s\in [0,s_{1}]}J _{\lambda_{2},\mu_{2}} \bigl(\widetilde{\sigma }_{2}(s) \bigr) \\ &=J_{\lambda_{1},\mu_{1}} \bigl(\widetilde{\sigma }_{1}(1) \bigr)+J_{\lambda_{2}, \mu_{2}} \bigl(\widetilde{\sigma }_{2}(1) \bigr)=J_{\lambda_{1},\mu_{1}}(u_{ \lambda_{1},\mu_{1}})+J_{\lambda_{2},\mu_{2}}(v_{\lambda_{2},\mu_{2}})=m _{\lambda_{1},\mu_{1}}+m_{\lambda_{2},\mu_{2}}. \end{aligned}$$

Since $\widetilde{\sigma }\in \widehat{\Gamma }$, we obtain $\widehat{c_{\gamma }}\leq d_{\gamma }$, thus

$$\begin{aligned} \limsup_{\gamma \rightarrow 0}\widehat{c_{\gamma }}\leq \liminf_{\gamma \rightarrow 0}d_{\gamma }\leq \limsup_{\gamma \rightarrow 0}d_{\gamma } \leq d_{0},\quad \widehat{c_{0}}\leq d_{0}. \end{aligned}$$

(32)

On the other hand, for any $\sigma (t,s)=(\sigma_{1}(t,s),\sigma_{2}(t,s)) \in \widehat{\Gamma }$, we define $\Upsilon (\sigma):[t_{0},t_{1}] \times [s_{0},s_{1}]\rightarrow \mathbb{R}^{2}$ by

$$\Upsilon (\sigma):= \bigl(J_{5} \bigl(\sigma_{1}(t,s) \bigr)-J_{6} \bigl(\sigma_{2}(t,s) \bigr),J _{5} \bigl(\sigma_{1}(t,s) \bigr)+J_{6} \bigl( \sigma_{2}(t,s) \bigr)-2 \bigr), $$

where $J_{5},J_{6}:D_{s}(\Omega)\rightarrow \mathbb{R}$ are defined by

$$ J_{5}(u)=\textstyle\begin{cases} \frac{\int_{\Omega }\mu_{1}\vert u\vert ^{2^{\ast }}\,dx}{\Vert u\Vert ^{2}_{D_{s}(\Omega)}+\int_{\Omega }\lambda_{1}\vert u\vert ^{2}\,dx}, & \text{if } u\neq 0, \\ 0,& \text{if } u=0, \end{cases} $$

(33)

and

$$ J_{6}(u)=\textstyle\begin{cases} \frac{\int_{\Omega }\mu_{2}\vert u\vert ^{2^{\ast }}\,dx}{\Vert u\Vert ^{2}_{D_{s}(\Omega)}+\int_{\Omega }\lambda_{2}\vert u\vert ^{2}\,dx},& \text{if } u\neq 0, \\ 0,& \text{if } u=0. \end{cases} $$

(34)

By the Sobolev embedding theorem $D_{s}(\Omega)\hookrightarrow L^{2^{ \ast }}(\Omega)$, for any $u\in D_{s}(\Omega)$, we have

$$\int_{\Omega }\mu_{i}\vert u\vert ^{2^{\ast }}\,dx \leq C \biggl(\Vert u\Vert ^{2}_{D_{s}(\Omega)}+ \int_{\Omega }\lambda_{i}\vert u\vert ^{2} \,dx \biggr)^{\frac{2^{\ast }}{2}},\quad i=1,2. $$

Consequently, we can deduce $J_{5}$, $J_{6}$ are continuous and

$$\begin{aligned} \Upsilon (\widetilde{\sigma }) (t,s)={}& \biggl( \frac{t^{2^{\ast }-2} \int_{\Omega }\mu_{1}\vert u_{\lambda_{1},\mu_{1}}\vert ^{2^{\ast }}\,dx}{ \Vert u_{\lambda_{1},\mu_{1}}\Vert ^{2}_{D_{s}(\Omega)}+\int_{\Omega }\lambda_{1}\vert u_{\lambda_{1},\mu_{1}}\vert ^{2}\,dx}- \frac{s^{2^{\ast }-2}\int_{\Omega }\mu _{2}\vert v_{\lambda_{2},\mu_{2}}\vert ^{2^{\ast }}\,dx}{\Vert v_{\lambda_{2},\mu_{2}}\Vert ^{2}_{D_{s}(\Omega)}+\int_{\Omega }\lambda_{2} \vert v_{\lambda_{2},\mu_{2}}\vert ^{2}\,dx}, \\ &{} \frac{t^{2^{\ast }-2}\int_{\Omega }\mu_{1}\vert u_{\lambda_{1},\mu_{1}}\vert ^{2^{\ast }}\,dx}{\Vert u_{\lambda_{1},\mu_{1}}\Vert ^{2} _{D_{s}(\Omega)}+\int_{\Omega }\lambda_{1}\vert u_{\lambda_{1},\mu_{1}}\vert ^{2}\,dx}-\frac{s ^{2^{\ast }-2}\int_{\Omega }\mu_{2}\vert v_{\lambda_{2},\mu_{2}}\vert ^{2^{ \ast }}\,dx}{\Vert v_{\lambda_{2},\mu_{2}}\Vert ^{2}_{D_{s}(\Omega)}+\int_{ \Omega }\lambda_{2}\vert v_{\lambda_{2},\mu_{2}}\vert ^{2}\,dx}-2 \biggr). \end{aligned}$$

Since

$$\int_{\Omega }\mu_{1}\vert u_{\lambda_{1},\mu_{1}}\vert ^{2^{\ast }}\,dx=\Vert u_{\lambda_{1},\mu_{1}}\Vert ^{2}_{D_{s}(\Omega)}+ \int_{\Omega }\lambda_{1} \vert u_{\lambda_{1},\mu_{1}} \vert ^{2}\,dx $$

and

$$\int_{\Omega }\mu_{2}\vert v_{\lambda_{2},\mu_{2}}\vert ^{2^{\ast }}\,dx=\Vert v_{\lambda_{2},\mu_{2}}\Vert ^{2}_{D_{s}(\Omega)}+ \int_{\Omega }\lambda_{2}\vert v_{\lambda_{2},\mu_{2}}\vert ^{2}\,dx. $$

Thus, $\Upsilon (\widetilde{\sigma })(1,1)=(0,0)$. By direct calculation, we have

$$\deg \bigl(\Upsilon (\widetilde{\sigma }),[t_{0},t_{1}] \times [s_{0},s_{1}],(0,0) \bigr)=1. $$

By (31), we know that, for any $(t,s)\in \partial ([t_{0},t _{1}]\times [s_{0},s_{1}])$, $\Upsilon (\widetilde{\sigma })(t,s)= \Upsilon (\sigma)(t,s)\neq (0,0)$. Therefore

$$\deg \bigl(\Upsilon (\sigma),[t_{0},t_{1}]\times [s_{0},s_{1}],(0,0) \bigr)=\deg \bigl( \Upsilon ( \widetilde{ \sigma }),[t_{0},t_{1}]\times [s_{0},s_{1}],(0,0) \bigr)=1. $$

Then there exist $(t_{2},s_{2})\in [t_{0},t_{1}]\times [s_{0},s_{1}]$ such that $\Upsilon (\sigma)(t_{2},s_{2})=(0,0)$, thus

$$J_{5} \bigl(\sigma_{1}(t_{2},s_{2}) \bigr)=J_{6} \bigl(\sigma_{2}(t_{2},s_{2}) \bigr)=1. $$

This implies

$$\sigma _{i}(t_{2},s_{2})\in \mathbb{M}_{i}\quad \text{and} \quad \sigma _{i}(t_{2},s_{2}) \neq 0 \quad \text{for } i=1,2. $$

By (7) and $\sigma_{i}(t_{2},s_{2})\in \mathbb{M}_{i}$, we have

$$\begin{aligned} \max_{(t,s)\in Q}E_{0} \bigl(\sigma (t,s) \bigr)&\geq E_{0} \bigl(\sigma (t_{2},s _{2}) \bigr) \\ &=J_{\lambda_{1},\mu_{1}} \bigl(\sigma_{1}(t_{2},s_{2}) \bigr)+J_{\lambda _{2},\mu_{2}} \bigl(\gamma_{2}(t_{2},s_{2}) \bigr) \\ &\geq m_{\lambda_{1},\mu_{1}}+m _{\lambda_{2},\mu_{2}}=d_{0}. \end{aligned}$$

Therefore $\widehat{c_{0}}\geq d_{0}$, combining this with (32), we obtain $\widehat{c_{0}}= d_{0}$.

By the definition of $\widehat{c}_{\gamma }$ and $d_{\gamma }$, we have

$$\widehat{c}_{\gamma }\leq d_{\gamma }\leq d_{0}. $$

Next, we prove $\liminf_{\gamma \rightarrow 0}\widehat{c_{\gamma }}\geq d_{0}$. Assume by contradiction that $\liminf_{\gamma \rightarrow 0}\widehat{c_{\gamma }}< d_{0}$. Then there exist $\epsilon >0$, $\gamma_{n}\rightarrow 0$ and $\sigma_{n}=(\sigma_{n,1},\sigma_{n,2})\in \widehat{\Gamma }$ such that

$$\max_{(t,s)\in Q}E_{\gamma_{n}} \bigl(\sigma_{n}(t,s) \bigr)\leq d_{0}-2 \epsilon. $$

By the definition of Γ̂ in (31), there exists $n_{0}$ large enough such that

$$\max_{(t,s)\in Q}\frac{1}{2^{\ast }}\gamma_{n} \biggl\vert \int_{ \Omega } \bigl\vert \sigma_{n,1}(t,s) \bigr\vert ^{\alpha } \bigl\vert \sigma_{n,2}(t,s) \bigr\vert ^{\beta }\,dx \biggr\vert \leq C\gamma_{n}\leq \epsilon,\quad \forall n\geq n_{0}. $$

Thus, $\max_{(t,s)\in Q}E_{0}(\sigma_{n}(t,s))\leq \max_{(t,s)\in Q}E_{\gamma_{n}}(\sigma_{n}(t,s))+\epsilon \leq d _{0}-\epsilon$, $\forall n\geq n_{0}$. Since $\widehat{c}_{0} \leq d_{0}$, this is a contradiction. Therefore $\liminf_{\gamma \rightarrow 0}\widehat{c_{\gamma }}\geq d_{0}$. Combining this again with (32), we complete the proof. □

Define

$$X^{\delta }:= \bigl\{ (u,v)\in \mathcal{D}_{s}(\Omega):\operatorname{dist}\bigl((u,v),X \bigr)\leq \delta \bigr\} , \qquad E^{c}_{\gamma }:= \bigl\{ (u,v)\in \mathcal{D}_{s}(\Omega):E _{\gamma }(u,v)\leq c \bigr\} . $$

Lemma 4.3

Let $d>0$ be a fixed number and let $\{(u_{n},v_{n})\}\subset X^{d}$ be a sequence. Then up to a subsequence, $(u_{n},v_{n})\rightharpoonup (u _{0},v_{0})\in X^{2d}$.

Proof

By Lemma 4.1 and the definition of $X^{d}$, there exists a sequence $\{(\overline{u}_{n},\overline{v}_{n})\}\subset X$ such that

$$\operatorname{dist}\bigl( (u_{n},v_{n}),X \bigr) =\operatorname{dist}\bigl( (u_{n},v_{n}),( \overline{u}_{n}, \overline{v}_{n}) \bigr) \leq d. $$

By Lemma 4.1, we also know that there exist $(\overline{u}, \overline{v})\in X$ such that $(\overline{u}_{n},\overline{v}_{n}) \rightarrow (\overline{u},\overline{v})$ strongly in $\mathcal{D}_{s}( \Omega)$. Consequently, when n is sufficiently large, we have

$$\operatorname{dist}\bigl( (\overline{u}_{n},\overline{v}_{n}),( \overline{u}, \overline{v}) \bigr) \leq d. $$

Thus, $\{(u_{n},v_{n})\}$ is bounded and up to a subsequence, $(u_{n},v_{n})\rightharpoonup (u_{0},v_{0})$ in $\mathcal{D}_{s}( \Omega)$. Since $B_{2d}(\overline{u},\overline{v})$ is weakly closed in $\mathcal{D}_{s}(\Omega)$, we get $(u_{0},v_{0})\in B_{2d}( \overline{u},\overline{v})\subset X^{2d}$. □

Lemma 4.4

Let $d_{1}:=\frac{1}{2}(\frac{Nm_{\lambda_{1},\mu_{1}}}{s})^{ \frac{1}{2}}$ and $d\in (0,d_{1})$. Suppose that there exist sequences $\{\gamma _{j}\}$, with $\gamma _{j}>0$ and $\gamma _{j}\rightarrow 0$, and $\{(u_{j},v_{j})\}\subset X^{d}$ satisfying

$$\lim_{j\rightarrow +\infty }E_{\gamma_{j}}(u_{j},v_{j}) \leq \widehat{c}_{0},\qquad \lim_{j\rightarrow +\infty }E^{\prime}_{\gamma _{j}}(u_{j},v_{j})=0. $$

Then $(u_{j},v_{j})$ converges strongly to an element $(u,v)\in X$.

Proof

By the choice of $d_{1}$ and Lemma 4.3 $(u_{j},v_{j})\rightharpoonup (u,v)\in X^{2d}$, we can deduce that $u\not \equiv 0$ and $v\not \equiv 0$. Since $\{(u_{j},v_{j})\}$ is bounded and $\lim_{j\rightarrow +\infty }E'_{\gamma_{j}}(u_{j},v_{j})=0$, for all $(\varphi,\psi)\in \mathcal{D}_{s}(\Omega)$,

$$\begin{aligned} \bigl\langle E^{\prime}_{0}(u,v),(\varphi,\psi) \bigr\rangle = & \langle u,\varphi \rangle_{D_{s}(\Omega)}+\langle v,\psi \rangle_{D_{s}(\Omega)}+ \int_{\Omega }(\lambda_{1}u\varphi +\lambda_{2}v \psi)\,dx \\ &{}- \int_{\Omega } \bigl(\mu_{1}\vert u\vert ^{2^{\ast }-2}u\varphi +\mu_{2}\vert v\vert ^{2^{ \ast }-2}v\psi \bigr)\,dx \\ = &\lim_{j\rightarrow +\infty } \biggl[ \bigl\langle E^{\prime}_{\gamma_{j}}(u _{j},v_{j}),(\varphi,\psi) \bigr\rangle + \frac{\alpha \gamma_{j}}{2^{ \ast }} \int_{\Omega }\vert u_{j}\vert ^{\alpha -2}u_{j} \varphi \vert v_{j}\vert ^{\beta }\,dx \\ &{}+\frac{\beta \gamma_{j}}{2^{\ast }} \int_{\Omega }\vert u_{j}\vert ^{\alpha } \vert v_{j}\vert ^{\beta -2}v_{j}\psi \,dx \biggr] \\ = &0, \end{aligned}$$

where

$$\langle u,\varphi \rangle_{D_{s}(\Omega)}= \int_{\mathbb{R}^{N}} \int_{\mathbb{R}^{N}} \frac{(u(x)-u(y))(\varphi (x)-\varphi (y))}{\vert y-x\vert ^{N+2s}}\,dx\,dy. $$

Hence, $E'_{0}(u,v)=0$. Since $(u_{j},v_{j})\in X^{d}$ for all j, we have

$$\begin{aligned} & \bigl\langle E^{\prime}_{0}(u_{j},v_{j}),( \varphi,\psi) \bigr\rangle \\ &\quad = \bigl\langle E ^{\prime}_{\gamma_{j}}(u_{j},v_{j}),( \varphi,\psi) \bigr\rangle +\frac{\alpha \gamma_{j}}{2^{\ast }} \int_{\Omega }\vert u_{j}\vert ^{\alpha -2}u_{j} \varphi \vert v_{j}\vert ^{\beta }\,dx+\frac{\beta \gamma_{j}}{2^{\ast }} \int_{\Omega } \vert u_{j}\vert ^{\alpha } \vert v_{j}\vert ^{\beta -2}v_{j}\psi \,dx \\ &\quad =O(1) \bigl\Vert (\varphi,\psi) \bigr\Vert _{\mathcal{D}_{s}(\Omega)}. \end{aligned}$$

We have

$$\begin{aligned} \widehat{c}_{0} &\geq \lim_{j\rightarrow +\infty }E_{\gamma_{j}}(u _{j},v_{j}) \\ &=\lim_{j\rightarrow +\infty }E_{0}(u_{j},v_{j})- \lim_{j\rightarrow +\infty }\frac{\gamma_{j}}{2^{\ast }} \int_{\Omega }\vert u_{j}\vert ^{\alpha } \vert v_{j}\vert ^{\beta }\,dx \\ &=\lim_{j\rightarrow +\infty }E_{0}(u_{j},v_{j}):=m. \end{aligned}$$

(35)

So $\{(u_{j},v_{j})\}$ is a $(PS)_{m}$ sequence of $E_{0}$ with $m:=\lim_{j\rightarrow +\infty }E_{0}(u_{j},v_{j})$. Thus, we have

$$\begin{aligned} E_{0}(u,v) &=\frac{1}{2} \bigl\Vert (u,v) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}+ \frac{1}{2} \int_{\Omega } \bigl(\lambda_{1}u^{2}+ \lambda_{2}v^{2} \bigr)\,dx-\frac{1}{2^{ \ast }} \int_{\Omega } \bigl(\mu_{1}\vert u\vert ^{2^{\ast }}+\mu_{2}\vert v\vert ^{2^{\ast }} \bigr)\,dx \\ &=\frac{s}{N} \biggl[ \bigl\Vert (u,v) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}+ \int_{\Omega } \bigl(\lambda_{1}u^{2}+ \lambda_{2}v^{2} \bigr)\,dx \biggr] \\ &\leq \frac{s}{N}\liminf_{j\rightarrow +\infty } \biggl[ \bigl\Vert (u_{j},v_{j}) \bigr\Vert ^{2}_{\mathcal{D}_{s}(\Omega)}+ \int_{\Omega } \bigl(\lambda _{1}u_{j}^{2}+ \lambda_{2}v_{j}^{2} \bigr)\,dx \biggr] \\ &=\liminf_{j\rightarrow +\infty } \biggl[ E_{0}(u_{j},v_{j})- \frac{1}{2^{ \ast }} \bigl\langle E'_{0}(u_{j},v_{j}),(u_{j},v_{j}) \bigr\rangle \biggr] =m. \end{aligned}$$

Then, by Lemma 4.2, we have $m\geq E_{0}(u,v)\geq \widehat{c} _{0}$. Combining this with (35), we get $m=E_{0}(u,v)= \widehat{c}_{0}$. This implies $(u_{j},v_{j})\rightarrow (u,v)$ strongly in $\mathcal{D}_{s}(\Omega)$ and $(u,v)\in X$. □

Lemma 4.5

Let $d_{1}$ be as in Lemma 4.4. For a small $\delta \in (0, {d_{1}})$, there exist constants $0< \sigma <1$ and $\gamma_{1} >0$ such that $\|E'_{\gamma }(u,v)\|\geq \sigma$ for any $(u,v)\in E^{d_{\gamma }}_{\gamma }\cap (X^{\delta }\setminus X^{\frac{ \delta }{2}})$ and $\gamma \in (0,\gamma_{1})$.

Proof

Assume by contradiction. Suppose there exist a number $\delta_{0} \in (0,d_{1})$, a positive sequence $\{\gamma_{j}\}$ with $\lim_{j\rightarrow +\infty }\gamma_{j}=0$ and a sequence $\{(u_{j},v_{j})\}\in E^{d_{\gamma_{j}}}_{\gamma_{j}}\cap (X^{\delta _{0}}\setminus X^{\frac{\delta_{0}}{2}})$ such that $\lim_{j\rightarrow +\infty }E'_{\gamma_{j}}(u_{j},v_{j})=0$. Then, by Lemma 4.2, we have

$$\lim_{j\rightarrow +\infty }E_{\gamma_{j}}(u_{j},v_{j}) \leq \widehat{c}_{0},\quad \bigl\{ (u_{j},v_{j}) \bigr\} \subset X^{\delta_{0}}, \delta _{0}< d_{1}, $$

and

$$\lim_{j\rightarrow +\infty }E'_{\gamma_{j}}(u_{j},v_{j})=0. $$

Then, by Lemma 4.4, we know there exist $(u,v)\in X$ such that $(u_{j},v_{j})\rightarrow (u,v)$ strongly in $\mathcal{D}_{s}(\Omega)$. Hence, $\operatorname{dist}((u_{j},v_{j}),X)\rightarrow 0$ as $j\rightarrow + \infty$. This contradicts $(u_{j},v_{j})\notin X^{ \frac{\delta_{0}}{2}}$. □

In the next part of this paper, we let $0<\sigma <1$, $\gamma_{1}>0$ and $\delta \in (0,\frac{d_{1}}{2})$ such that the conclusions in Lemma 4.5 hold.

Lemma 4.6

There exist $\gamma _{2}\in (0,\gamma _{1})$ and $\varsigma >0$ such that, for any $\gamma \in (0,\gamma _{2})$,

$$\begin{aligned} E_{\gamma } \bigl( {\widetilde{\sigma }(t,s)} \bigr)\geq \widehat{c}_{\gamma }-\varsigma \quad \textit{implies that} \quad {\widetilde{ \sigma }(t,s)}\in X^{\frac{\delta }{2}}. \end{aligned}$$

(36)

Proof

Suppose by contradiction that there exist $\gamma_{n}\rightarrow 0$, $\varsigma_{n}\rightarrow 0$ and $(t_{n},s_{n})\in Q$ such that

$$\begin{aligned} E_{\gamma_{n}} \bigl( {\widetilde{\sigma }(t_{n},s_{n})} \bigr) \geq \widehat{c}_{\gamma_{n}}-\varsigma_{n} \quad \textit{and} \quad {\widetilde{\sigma }(t_{n},s_{n})}\notin X^{\frac{ \delta }{2}}. \end{aligned}$$

(37)

We assume $(t_{n},s_{n})\rightarrow (\overline{t},\overline{s})\in Q$. Since

$$\begin{aligned} E_{0} \bigl( {\widetilde{\sigma }(t_{n},s_{n})} \bigr)\geq E_{\gamma _{n}} \bigl( {\widetilde{\sigma }(t_{n},s_{n})} \bigr)\geq \widehat{c}_{\gamma_{n}}-\varsigma_{n}, \end{aligned}$$

(38)

we take the limit on both sides of (38), we have

$$\begin{aligned} E_{0} \bigl( {\widetilde{\sigma }(\overline{t},\overline{s})} \bigr) \geq \lim_{n\rightarrow +\infty }\widehat{c}_{\gamma_{n}}. \end{aligned}$$

By Lemma 4.2, we have

$$E_{0} \bigl( {\widetilde{\sigma }(\overline{t},\overline{s})} \bigr) \geq \lim_{n\rightarrow +\infty }\widehat{c}_{\gamma_{n}}=m _{\lambda_{1},\mu_{1}}+m_{\lambda_{2},\mu_{2}}. $$

Combining this with (27) and (32), we can deduce that $(\overline{t},\overline{s})=(1,1)$. Hence,

$$\lim_{n\rightarrow +\infty } \bigl\Vert {\widetilde{\sigma }(t_{n},s_{n})}- {\widetilde{\sigma }(1,1)} \bigr\Vert =0. $$

However, ${\widetilde{\sigma }(1,1)}=(u_{\lambda_{1}, \mu_{1}},v_{\lambda_{2},\mu_{2}})\in X$, which contradicts (37). □

Next, we set

$$\begin{aligned} \varsigma_{0}:=\min \biggl\{ \frac{\varsigma }{2}, \frac{m_{\lambda_{1},\mu _{1}}}{4},\frac{\delta \sigma^{2}}{8} \biggr\} , \end{aligned}$$

(39)

where δ, σ are given in Lemma 4.5, ς is from Lemma 4.6. By Lemma 4.2, we know that there exists $\gamma_{0}\in (0,\gamma_{2}]$ such that

$$\begin{aligned} \vert \widehat{c}_{\gamma }-d_{\gamma }\vert < \varsigma_{0},\qquad \bigl\vert \widehat{c}_{\gamma }-(m_{\lambda_{1},\mu_{1}}+m_{\lambda_{2},\mu_{2}}) \bigr\vert < \varsigma_{0},\quad \forall \gamma \in (0, \gamma_{0}). \end{aligned}$$

(40)

Lemma 4.7

For fixed $\gamma \in (0,\gamma_{0})$, there exist ${\{(u_{n},v_{n})\}^{\infty }_{n=1}}\subset X^{\delta } \cap E^{d_{\gamma }}_{\gamma }$ such that

$${E'_{\gamma }(u_{n},v_{n})\rightarrow 0}\quad \textit{in } \mathcal{D}_{s}(\Omega) \textit{ as } n\rightarrow + \infty. $$

Proof

Assume by contradiction, for fixed $\gamma \in (0,\gamma_{0})$, that there exists $0< l(\gamma)<1$ such that

$$\bigl\Vert E'_{\gamma }(u,v) \bigr\Vert \geq l(\gamma) \quad \text{on } X^{\delta }\cap E ^{d_{\gamma }}_{\gamma } . $$

Then there exists a pseudo-gradient vector field $T_{\gamma }$ in $\mathcal{D}_{s}(\Omega)$ which is defined on a neighborhood $Z_{\gamma }$ of $X^{\delta }\cap E^{d_{\gamma }}_{\gamma }$ such that, for any $(u,v)\in Z_{\gamma }$,

$$\begin{aligned} & \bigl\Vert T_{\gamma }(u,v) \bigr\Vert \leq 2 \min \bigl\{ 1, \bigl\Vert E'_{\gamma }(u,v) \bigr\Vert \bigr\} , \\ & \bigl\langle E'_{\gamma }(u,v), T_{\gamma }(u,v) \bigr\rangle \geq \min \bigl\{ 1, \bigl\Vert E'_{\gamma }(u,v) \bigr\Vert \bigr\} \bigl\Vert E'_{\gamma }(u,v) \bigr\Vert . \end{aligned}$$

Let $\eta_{\gamma }$ be a Lipschitz continuous function on $\mathcal{D}_{s}(\Omega)$ such that

$$0\leq \eta_{\gamma }\leq 1,\quad \eta_{\gamma }=1\quad \text{on } X^{\delta } \cap E^{d_{\gamma }}_{\gamma }\quad \text{and}\quad \eta_{\gamma }=0\quad \text{on } \mathcal{D}_{s}(\Omega)\setminus Z_{\gamma }. $$

Let $\xi_{\gamma }$ be a Lipschitz continuous function on $\mathbb{R}$ such that

$$0\leq \xi_{\gamma }\leq 1,\quad \xi_{\gamma }(l)\equiv 1\quad \text{if } \vert l-\widehat{c}_{\gamma }\vert \leq \frac{\varsigma }{2}\quad \text{and}\quad \xi_{ \gamma }(l)\equiv 0 \quad \text{if } \vert l- \widehat{c}_{\gamma } \vert \geq \varsigma. $$

Let

$$ e_{\gamma }(u,v):= \textstyle\begin{cases} -\eta_{\gamma }(u,v)\xi_{\gamma }(E_{\gamma }(u,v))T_{\gamma }(u,v),& \text{if } (u,v)\in Z_{\gamma }, \\ 0,& \text{if } (u,v)\in H\setminus Z_{\gamma }. \end{cases} $$

(41)

Then there exists a global solution $\psi_{\gamma }:\mathcal{D}_{s}( \Omega)\times [0,+\infty)\rightarrow \mathcal{D}_{s}(\Omega)$ for the initial value problem

$$ \textstyle\begin{cases} \frac{d}{d\theta }\psi _{\gamma }(u,v,\theta)=e_{\gamma }(\psi _{\gamma }(u,v,\theta)), \\ \psi_{\gamma }(u,v,0)=(u,v). \end{cases} $$

(42)

Then we can deduce that $\psi_{\gamma }$ has the following properties:

• (1) $\psi_{\gamma }(u,v,\theta)=(u,v)$ if $\theta =0$ or $(u,v) \in \mathcal{D}_{s}(\Omega)\setminus Z_{\gamma }$ or $\vert E_{\gamma }(u,v)-\widehat{c}_{\gamma } \vert \geq \varsigma$.

• (2) $\Vert \frac{d}{d\theta }\psi_{\gamma }(u,v,\theta) \Vert \leq 2$.

• (3) $\frac{d}{d\theta }E_{\gamma } (\psi_{\gamma }(u,v,\theta) )= \langle E'_{\gamma } (\psi_{\gamma }(u,v, \theta) ),e_{\gamma } ( \psi_{\gamma }(u,v,\theta) ) \rangle \leq 0$.

In order to prove Lemma 4.7, we use the above properties, Lemma 4.5 and Lemma 4.6, then divide two step to prove it.

Step one. We show that, for any $(t,s)\in Q$, there exists $\theta_{t,s}\in [0,+\infty)$ such that $\psi_{\gamma }( \widetilde{\sigma }(t,s), \theta_{t,s})\in E^{\widehat{c}_{\gamma }- \varsigma_{0}}_{\gamma }$, where $\varsigma_{0}$ is seen in (39).

Suppose by contradiction that there exists $(t,s)\in Q$ such that

$$E_{\gamma }(\psi_{\gamma } \bigl(\widetilde{\sigma }(t,s), \theta \bigr)> \widehat{c}_{\gamma }-\varsigma_{0}\quad \text{for any } \theta \geq 0. $$

Since $\varsigma_{0}<\varsigma$, by Lemma 4.6, we have $\widetilde{\sigma }(t,s)\in X^{\frac{\delta }{2}}$. By (40), we get

$$E_{\gamma } \bigl(\widetilde{\sigma }(t,s) \bigr)\leq d_{\gamma }< \widehat{c}_{ \gamma }+\varsigma_{0}. $$

By the property (3), we have

$$\widehat{c}_{\gamma }-\varsigma_{0}< E_{\gamma }( \psi_{\gamma } \bigl( \widetilde{\sigma }(t,s), \theta \bigr)\leq d_{\gamma }< \widehat{c}_{ \gamma }+\varsigma_{0},\quad \forall \theta \geq 0. $$

This implies $\xi_{\gamma }(E_{\gamma }(\psi_{\gamma }( \widetilde{\sigma }(t,s), \theta)))\equiv 1$. If $\psi_{\gamma }( \widetilde{\sigma }(t,s), \theta)\in X^{\delta }$ for all $\theta \geq 0$, then

$$\eta_{\gamma } \bigl(\psi_{\gamma } \bigl(\widetilde{\sigma }(t,s), \theta \bigr) \bigr) \equiv 1\quad \text{and}\quad \bigl\Vert E'_{\gamma } \bigl(\psi_{\gamma } \bigl(\widetilde{\sigma }(t,s), \theta \bigr) \bigr) \bigr\Vert \geq l(\gamma)\quad \text{for all } \theta >0. $$

Consequently,

$$E_{\gamma } \biggl(\psi _{\gamma } \biggl(\widetilde{\sigma }(t,s), \frac{\varsigma }{l^{2}(\gamma)} \biggr) \biggr)\leq \widehat{c}_{\gamma }+ \frac{\varsigma }{2}- \int ^{\frac{\varsigma }{l^{2}(\gamma)}}_{0}l^{2}(\gamma)\,dt= \widehat{c}_{\gamma }-\frac{\varsigma }{2}, $$

which is a contradiction. Thus, there exists $\theta_{t,s}>0$ such that $\psi_{\gamma }(\widetilde{\sigma }(t,s), \theta_{t,s})\notin X^{ \delta }$.

Since $\widetilde{\sigma }(t,s)\in X^{\frac{\delta }{2}}$, there exist $0<\theta^{1}_{t,s}<\theta^{2}_{t,s}\leq \theta_{t,s}$ such that

$$\psi_{\gamma } \bigl(\widetilde{\sigma }(t,s), \theta^{1}_{t,s} \bigr)\in \partial X^{\frac{\delta }{2}},\qquad \psi_{\gamma } \bigl(\widetilde{\sigma }(t,s), \theta^{2}_{t,s} \bigr)\in \partial X^{\delta } $$

and

$$\psi_{\gamma } \bigl(\widetilde{\sigma }(t,s), \theta \bigr)\in X^{\delta } \setminus X^{\frac{\delta }{2}}\quad \text{for all } \theta \in \bigl( \theta ^{1}_{t,s},\theta^{2}_{t,s} \bigr). $$

Then, by Lemma 4.5, we have $\|E'_{\gamma }(\psi_{\gamma }( \widetilde{\sigma }(t,s), \theta))\|\geq \sigma$ for all $\theta \in (\theta^{1}_{t,s},\theta^{2}_{t,s})$. Then, by the property (2), we have

$$\frac{\delta }{2}\leq \bigl\Vert \psi_{\gamma } \bigl(\widetilde{\sigma }(t,s),\theta^{2}_{t,s} \bigr)-\psi_{\gamma } \bigl( \widetilde{\sigma }(t,s), \theta^{1}_{t,s} \bigr) \bigr\Vert \leq 2 \bigl\vert \theta^{2}_{t,s}- \theta^{1}_{t,s} \bigr\vert , $$

thus, $|\theta^{2}_{t,s}-\theta^{1}_{t,s}|\geq \frac{\delta }{4}$. Consequently,

$$\begin{aligned} E_{\gamma } \bigl(\psi_{\gamma } \bigl(\widetilde{\sigma }(t,s), \theta^{2}_{t,s} \bigr) \bigr) & \leq E_{\gamma } \bigl( \psi_{\gamma } \bigl(\widetilde{\sigma }(t,s), \theta^{1} _{t,s} \bigr) \bigr)+ \int^{\theta^{2}_{t,s}}_{\theta^{1}_{t,s}}\frac{d}{d\theta } E_{\gamma } \bigl(\psi_{\gamma }(u,v,\theta) \bigr)\,d\theta \\ &\leq \widehat{c}_{\gamma }+\varsigma_{0}-\sigma^{2} \bigl(\theta^{2}_{t,s}- \theta^{1}_{t,s} \bigr)\leq \widehat{c}_{\gamma }+\varsigma_{0}-\frac{ \delta \sigma^{2}}{4} \\ &\leq \widehat{c}_{\gamma }-\varsigma_{0}, \end{aligned}$$

which is a contradiction.

By step one we can define $T(t,s):=\inf \{\theta \geq 0: E_{\gamma }( \psi_{\gamma }(\widetilde{\sigma }(t,s), \theta))\leq \widehat{c} _{\gamma }-\varsigma_{0}\}$ and let $\sigma (t,s):=\psi_{\gamma }( \widetilde{\sigma }(t,s), T(t,s))$. Then $E_{\gamma }(\sigma (t,s)) \leq \widehat{c}_{\gamma }-\varsigma_{0}$ for all $(t,s)\in Q$.

Step two. We claim $\sigma (t,s)\in \widehat{\Gamma }$.

By (27)–(28) and (39), (40), for any $(t,s)\in Q\setminus (t_{0},t_{1})\times (s_{0},s_{1})$, we have

$$\begin{aligned} E_{\gamma } \bigl( {\widetilde{\sigma }(t,s)} \bigr) &\leq E_{0} \bigl( {\widetilde{\sigma }(t,s)} \bigr)=J_{\lambda_{1},\mu_{1}} \bigl( {\widetilde{ \sigma }_{1}(t)} \bigr)+J_{\lambda_{2},\mu_{2}} \bigl( {\widetilde{\sigma }_{2}(s)} \bigr) \\ &\leq \frac{m_{\lambda_{1},\mu_{1}}}{4}+m_{\lambda_{2},\mu_{2}} \leq m_{\lambda_{1},\mu_{1}}+m_{\lambda_{2},\mu_{2}}-3 \varsigma_{0}< \widehat{c}_{\gamma }-\varsigma_{0}, \end{aligned}$$

which implies that $T(t,s)=0$ and so $\sigma (t,s)= \widetilde{\sigma }(t,s)$.

By the definition of Γ̂ in (31), we need to prove that ${\|\sigma (t,s)\|_{\mathcal{D}_{s}(\Omega)}}\leq 2C _{2}+C_{0}$ for all $(t,s)\in Q$ and $T(t,s)$ is continuous with respect to $(t,s)$.

For any $(t,s)\in Q$, if $E_{\gamma }(\widetilde{\sigma }(t,s))\leq \widehat{c}_{\gamma }-\varsigma_{0}$, we have $T(t,s)=0$ and so $\sigma (t,s)= {\widetilde{\sigma }(t,s)}$. By (30), we have $\|\sigma (t,s)\|_{\mathcal{D}_{s}(\Omega)}\leq C_{0}< 2C_{2}+C_{0}$.

If $E_{\gamma }(\widetilde{\sigma }(t,s))>\widehat{c}_{\gamma }- \varsigma_{0}$, then, by Lemma 4.6, we have $\widetilde{\sigma }(t,s)\in X^{\frac{\delta }{2}}$ and

$$\widehat{c}_{\gamma }-\varsigma_{0}< E_{\gamma } \bigl( \psi_{\gamma } \bigl( \widetilde{\sigma }(t,s), \theta \bigr)\leq d_{\gamma }< \widehat{c}_{ \gamma }+\varsigma_{0},\quad \forall \theta \in \big[0,T(t,s) \bigr). $$

This implies $\xi_{\gamma }(E_{\gamma }(\psi_{\gamma }( \widetilde{\sigma }(t,s), \theta)))\equiv 1$ for $\theta \in [0,T(t,s))$. If $\psi_{\gamma }(\widetilde{\sigma }(t,s), T(t,s))\notin X^{\delta }$, then there exist $0<\theta^{1}_{t,s}< \theta^{2}_{t,s}<T(t,s)$ as above. Then we can prove that

$$E_{\gamma }(\psi_{\gamma } \bigl(\widetilde{\sigma }(t,s), \theta^{2}_{t,s} \bigr) \leq \widehat{c}_{\gamma }- \varsigma_{0}, $$

which contradicts the definition of $T(t,s)$. Therefore,

$$\sigma (t,s)=\psi_{\gamma } \bigl(\widetilde{\sigma }(t,s), T(t,s) \bigr) \in X ^{\delta }. $$

Then there exist $(u,v)\in X$ such that $\|\sigma (t,s)-(u,v)\|_{ \mathcal{D}_{s}(\Omega)}\leq \delta \leq {\frac{C_{0}}{2}}$. By Lemma 4.1, we have

$$\bigl\Vert \sigma (t,s) \bigr\Vert _{\mathcal{D}_{s}(\Omega)}\leq \bigl\Vert (u,v) \bigr\Vert _{\mathcal{D}_{s}(\Omega)}+\frac{C_{0}}{2}\leq 2C_{2}+C_{0}. $$

In order to prove the continuity of $T(t,s)$, we fix any $( \widetilde{t},\widetilde{s})\in Q$. First, we assume that $E_{\gamma }(\sigma (\widetilde{t},\widetilde{s}))<\widehat{c}_{\gamma }- \varsigma_{0}$. Then, by the definition of $T(t,s)$, we have $T(\widetilde{t},\widetilde{s})=0$, that is,

$$E_{\gamma } \bigl(\widetilde{\sigma }(\widetilde{t},\widetilde{s}) \bigr)< \widehat{c}_{\gamma }-\varsigma _{0}. $$

By the continuity of σ̃, there exists $\tau >0$ such that, for any $(t,s)\in (\widetilde{t}-\tau,\widetilde{t}+\tau) \times (\widetilde{s}-\tau,\widetilde{s}+\tau)\cap Q$, we have $E_{\gamma }(\widetilde{\sigma }(t,s))<\widehat{c}_{ \gamma }-\varsigma _{0}$, that is, $T(t,s)=0$ and T is continuous at $(\widetilde{t},\widetilde{s})$. Now, we assume that $E_{\gamma }( \sigma (\widetilde{t},\widetilde{s}))=\widehat{c}_{\gamma }-\varsigma _{0}$. Then from the previous proof we have

$$\sigma (\widetilde{t},\widetilde{s})=\psi _{\gamma } \bigl(\widetilde{ \sigma }(\widetilde{t},\widetilde{s}), T(\widetilde{t},\widetilde{s}) \bigr)\in X^{\delta } , $$

and so

$$\bigl\Vert E'_{\gamma }(\psi_{\gamma } \bigl( \widetilde{\sigma }(\widetilde{t},\widetilde{s}),T(\widetilde{t},\widetilde{s}) \bigr) \bigr\Vert \geq l(\gamma)>0. $$

Then, for any $\omega >0$, we have

$$E_{\gamma }(\psi_{\gamma } \bigl(\widetilde{\sigma }(\widetilde{t}, \widetilde{s}),T(\widetilde{t},\widetilde{s})+\omega \bigr)< \widehat{c} _{\gamma }-\varsigma_{0}. $$

By the continuity of $\psi_{\gamma }$, there exists $\tau >0$ such that, for any $(t,s)\in (\widetilde{t}-\tau,\widetilde{t}+\tau)\times ( \widetilde{s}-\tau,\widetilde{s}+\tau)\cap Q$, we have $E_{\gamma }(\psi_{\gamma }(\widetilde{\sigma }(t,s)),T(\widetilde{t}, \widetilde{s})+\omega))<\widehat{c}_{\gamma }-\varsigma_{0}$, so $T(t,s)\leq T(\widetilde{t},\widetilde{s})+\omega$. It follows that

$$0< \limsup_{(t,s)\rightarrow (\widetilde{t},\widetilde{s})}T(t,s) \leq T(\widetilde{t},\widetilde{s}). $$

If $T(\widetilde{t},\widetilde{s})=0$, we have

$$\lim_{(t,s)\rightarrow (\widetilde{t},\widetilde{s})}T(t,s)= T( \widetilde{t},\widetilde{s}). $$

If $T(\widetilde{t},\widetilde{s})>0$, then, for any $0<\omega <T( \widetilde{t},\widetilde{s})$, by the same arguments, we have

$$E_{\gamma }(\psi_{\gamma } \bigl(\widetilde{\sigma }(\widetilde{t}, \widetilde{s}),T(\widetilde{t},\widetilde{s})-\omega \bigr)>\widehat{c} _{\gamma }-\varsigma_{0}. $$

By the continuity of $\psi_{\gamma }$ again, we have

$$\lim_{(t,s)\rightarrow (\widetilde{t},\widetilde{s})}T(t,s)= T( \widetilde{t},\widetilde{s}). $$

So T is continuous at $(\widetilde{t},\widetilde{s})$. This completes the proof of step two.

Now, we have proved that $\sigma (t,s)\in \widehat{\Gamma }$ and $\max_{(t,s)\in Q}E_{\gamma }(\sigma (t,s))\leq \widehat{c}_{\gamma }-\varsigma_{0}$, which contradicts the definition of $\widehat{c} _{\gamma }$. This completes the proof. □

Proof of Theorem 1.2

Let us fix $d_{1}:=\frac{1}{2}(\frac{Nm_{\lambda_{1},\mu_{1}}}{s})^{ \frac{1}{2}}$. By Lemma 4.7, there exists some $\gamma_{0}>0$ such that, for any fixed $\gamma \in (0,\gamma_{0})$, a Palais–Smale sequence $\{(u^{\gamma }_{n},v^{\gamma }_{n})\}$ with $(u^{\gamma } _{n},v^{\gamma }_{n})\in X^{\delta }$ exists. Since X is compact, we can deduce that $\{(u^{\gamma }_{n},v^{\gamma }_{n})\}$ is bounded in $\mathcal{D}_{s}(\Omega)$. By Lemma 4.3, there exist $(u_{\gamma },v_{\gamma })\in X^{d}$ such that $(u^{\gamma }_{n},v ^{\gamma }_{n})\rightharpoonup (u_{\gamma },v_{\gamma })$ weakly in $\mathcal{D}_{s}(\Omega)$. Therefore, $E'_{\gamma }(u_{\gamma },v _{\gamma })=0$. By the choice of d, we have $u_{\gamma }\neq 0$ and $v_{\gamma }\neq 0$. Hence, $(u_{\gamma },v_{\gamma })$ is the desired solution to (3). □