Multiplicity of positive solutions for fractional elliptic systems involving sign-changing weight

Abstract

In this paper, we study the multiplicity results of positive solutions for a fractional elliptic system involving both concave-convex and critical growth terms. With the help of Morse theory and the Ljusternik-Schnirelmann category, we investigate how the coefficient $h(x)$ of the critical nonlinearity affects the number of positive solutions of that problem and we get some results as regards the relationship between the number of positive solutions and the topology of the global maximum set of h.

1 Introduction and the main result

This paper is concerned with the number of positive solutions for the following fractional elliptic system:

$$ (E_{f,g}) \textstyle\begin{cases} (-\Delta)^{\frac{s}{2}} u=f(x) \vert u \vert ^{q-2}u+\frac{\alpha}{\alpha+\beta}h(x) \vert u \vert ^{\alpha -2}u \vert v \vert ^{\beta}, &\text{in } \Omega,\\ (-\Delta)^{\frac{s}{2}} v=g(x) \vert v \vert ^{q-2}v+\frac{\beta}{\alpha+\beta}h(x) \vert u \vert ^{\alpha} \vert v \vert ^{\beta-2}v, &\text{in } \Omega,\\ u=v=0,& \text{on } \partial\Omega, \end{cases} $$

where $\Omega\subset\mathbb{R}^{N}$ is a smooth bounded domain, $N>s$ with $s\in(0,2)$ fixed, $\alpha,\beta>1$ satisfy $2< q<\alpha+\beta =2_{s}^{*}=\frac{2N}{N-s}$, $2_{s}^{*}$ is the fractional Sobolev critical exponent, and $(-\Delta)^{\frac{s}{2}}$ is the fractional Laplacian. Moreover, f, g, h are continuous functions satisfying:

(H₁):

There exist a non-empty closed set $M=\{z\in \overline{\Omega};h(z)=\max_{x\in\overline{\Omega}}h(x)=1\}$ and a positive number $\rho\geq N$ such that $h(z)-h(x)=O( \vert x-z \vert ^{\rho})$ as $x\rightarrow z$ and uniformly in $z\in M$.

(H₂):

$f(z),g(z)>0$ for $z\in M$, and $h(x)\geq0$ for $x\in\overline{\Omega}$.

Remark 1.1

Let $M_{r}=\{z\in\mathbb{R}^{N};\operatorname{dist}(z,M)< r\}$ for $r>0$. Then by (H₁)-(H₂), there exist $C_{0},r_{0}>0$ such that

$$ f(z),g(z),h(z)>0\quad\text{for all } z\in M_{r_{0}}\subset\Omega $$

and

$$ h(z)-h(x)\leq C_{0} \vert x-z \vert ^{\rho}\quad\text{for all } x\in B_{r_{0}}(z)\subset\Omega $$

uniformly in $z\in M$, where $B_{r_{0}}(z)=\{x\in\mathbb{R}^{N}; \vert x-z \vert < r_{0}\}$.

In recent years, problems involving fractional operators have received special attention since they have important applications in many sciences. We limit here ourself to giving a non-exhaustive list of fields and papers in which these operators are used: obstacle problem [1, 2], optimization and finance [3, 4], phase transition [5, 6], material science [7], anomalous diffusion [8, 9], conformal geometry and minimal surfaces [10–12]. The list may continue with applications in crystal dislocation, soft thin films, multiple scattering, quasi-geostrophic flows, water waves, and so on. The interested reader may consult also the references in the cited papers. Setting $\alpha+\beta=p\leq2_{s}^{*}$, $f(x)\equiv g(x)$, $h(x)\equiv1$ and $u=v$, $(E_{f,g})$ reduces to the following fractional elliptic equation:

$$ (E_{\lambda}) \textstyle\begin{cases} (-\Delta)^{\frac{s}{2}} u=\lambda \vert u \vert ^{q-2}u+ \vert u \vert ^{p-2}u, &\text{in } \Omega,\\ u=0,& \text{on } \partial\Omega. \end{cases} $$

Goyal and Sreenadh [13] studied the existence and multiplicity of positive solutions to $(E_{\lambda})$. Moreover, by Nehari manifold and Fibering maps, Chen and Deng [14] obtained the existence of multiple solutions to $(E_{\lambda})$ for subcritical case and critical case. For the fractional Laplacian system with $1< q<2$, He, Squassina, and Zou [15] proved that $(E_{\lambda,\mu})$ ($(E_{f,g})$ with $f(x)\equiv\lambda$ and $g(x)\equiv\mu$) permits at least two positive solutions when λ and μ are small enough. Recently, Fan [16] established a relationship between the number of positive solutions of $(E_{f,g})$ and the topology of the global maximum set of h for $1< q<2$. Similar results were taken by Chen and Deng [17]. The tool they used is the decomposition of the Nehari manifold.

There are several existence results for the following problem:

$$ \varepsilon^{s}(-\Delta)^{\frac{s}{2}} u+V(x)u=f(u),\quad x\in \mathbb{R}^{N}, $$

(1.1)

where ε is a positive parameter, f has a subcritical growth, V possesses a local minimum. For $\varepsilon=1$, we would like to cite [18, 19] for the existence of one positive solution imposing a global condition on V. For ε is a small positive constant, several scholars established the existence and concentration of positive solutions for (1.1), by imposing different conditions on V and f (see [20–24]). In particular, with the help of the Nehari manifold and the Ljusternik-Schnirelmann category, Figueiredo and Siciliano [23] obtained a relationship between the number of positive solutions and the topology of the minimum set of V.

An interesting question now is how the weight potential $h(x)$ of a critical term affects the number of positive solutions of $(E_{f,g})$ involving a critical nonlinearity and sign-changing weight potentials. Furthermore, we wonder if there is a similar relationship between the number of positive solutions of $(E_{f,g})$ and the topology of the global maximum set of h as that in [23]. In this paper, with the help of the Ljusternik-Schnirelmann category and Morse theory, we shall give an answer to these questions in the following. The main results of our work are stated as follows: On the one hand, we arrive at the following result by means of the Ljusternik-Schnirelmann category.

Theorem 1.1

Assume (H₁)-(H₂) and $q>\frac{2s}{N-s}$ hold. Then, for each $\delta< r_{0}$, there exists $\Lambda_{\delta}>0$ such that if $\Vert f_{+} \Vert _{L^{q^{*}}(\Omega)}+ \Vert g_{+} \Vert _{L^{q^{*}}(\Omega)}<\Lambda_{\delta}$ ($q^{*}=\frac{2_{s}^{*}}{2_{s}^{*}-q}$), $(E_{f,g})$ has at least $\operatorname{cat}_{M_{\delta}}(M)$ distinct positive solutions, where $f_{+}=\max\{f,0\}$, $g_{+}=\max\{g,0\}$, and cat means Ljusternik-Schnirelmann category (see [25]).

On the other hand, with the use of the Morse theory we are able to deduce the next result.

Theorem 1.2

Assume (H₁)-(H₂) and $\alpha,\beta>2$ hold. Then $(E_{f,g})$ has at least $2P_{1}(M)-1$ positive solutions, if non-degenerate, possibly counted with their multiplicity.

Remark 1.2

We denote by $P_{t}(M_{\delta})$ the Poincaré polynomial of M. It is clear that in general, we get a better result using Morse theory; indeed, if for example M is obtained by a contractible domain cutting off k disjoint contractible sets, it is $\operatorname{cat}_{M_{\delta}}(M)=2$ and $P_{1}(M)=1+k$. However, by using the Ljusternik-Schnirelmann category no non-degeneracy condition is required.

Remark 1.3

Concerning regularity, one can get a priori estimate for the solutions to $(E_{f,g})$ and hence obtain, as in [26], Proposition 5.2, $u,v\in C^{\infty}(\overline{\Omega})$ for $s=1$, $u,v\in C^{0,s}(\overline {\Omega})$ if $0< s<1$ and $u,v\in C^{1,s-1}$ if $1< s<2$.

Discussion In recent years, there are many papers considering the relationship between the number of positive solutions of the elliptic equation and the topology of the global maximum set of its weight potentials. Our result Theorem 1.1 generalizes this result to fractional elliptic systems with more general weight potentials. Furthermore, by using Morse theory, we give a better result about the number of positive solutions in Theorem 1.2 than Theorem 1.1. However, the non-degeneracy condition is used in Theorem 1.2.

This paper is organized as follows: In Section 2, we introduce some notations and preliminaries. In Section 3, we give some technical results which are crucial to the proof of Theorems 1.1 and 1.2. In Section 4, we give the proof of Theorem 1.1. In Section 5, we prove Theorem 1.2.

2 Notations and preliminaries

In this section, we collect preliminary facts for future reference. First of all, let us write the standard notations which we will use in this paper. We denote the upper half-space in $\mathbb {R}^{N+1}_{+}$ by

$$ \mathbb{R}^{N+1}_{+}:=\bigl\{ (x,y);(x_{1},x_{2}, \ldots x_{N},y)\in\mathbb {R}^{N+1},y>0\bigr\} . $$

Denote the half cylinder with base Ω by $C_{\Omega}=\Omega \times(0,\infty)\subset\mathbb{R}^{N+1}_{+}$ and its lateral boundary by $\partial_{L}C_{\Omega}=\partial\Omega\times[0,\infty)$. We shall use C ($C_{i}$, $i=1,2,\ldots$) to denote any positive constant.

Let $\varphi_{j}$, $\lambda_{j}$ be the eigenfunctions and eigenvalues of −Δ in Ω with zero Dirichlet boundary data. The fractional Laplacian $(-\Delta)^{\frac{s}{2}}$ is defined in the space of functions

$$ H_{0}^{\frac{s}{2}}(\Omega):=\Biggl\{ u=\sum _{j=1}^{\infty}a_{j}\varphi_{j}\in L^{2}(\Omega); \Vert u \Vert _{ H_{0}^{\frac{s}{2}}(\Omega )}= \Biggl(\sum _{j=1}^{\infty}a^{2}_{j} \lambda^{\frac{s}{2}}_{j} \Biggr)^{\frac{1}{2}}< \infty \Biggr\} $$

and $\Vert u \Vert _{ H_{0}^{\frac{s}{2}}(\Omega)}= \Vert (-\Delta)^{\frac{s}{4}}u \Vert _{L^{2}(\Omega)}$. The dual space $H^{-\frac{s}{2}}(\Omega)$ is defined in the standard way as well as the inverse operator $(-\Delta)^{-\frac{s}{2}}$.

Definition 2.1

We say that $(u,v)\in H_{0}^{\frac{s}{2}}(\Omega)\times H_{0}^{\frac{s}{2}}(\Omega)$ is a solution of $(E_{f,g})$ if the identity

$$\begin{aligned}& \int_{\Omega}(-\Delta)^{\frac{s}{4}}u(- \Delta)^{\frac{s}{4}}\varphi _{1}+(-\Delta)^{\frac{s}{4}}v(-\Delta)^{\frac{s}{4}}\varphi_{2}\,dx \\& \quad = \int _{\Omega}\bigl(f(x) \vert u \vert ^{q-2}u \varphi_{1}+g(x) \vert v \vert ^{q-2}v\varphi_{2} \bigr)\,dx \\& \qquad{}+\frac{\alpha}{\alpha+\beta} \int_{\Omega}h(x) \vert u \vert ^{\alpha-2}u \vert v \vert ^{\beta}\varphi_{1}\,dx+\frac{\beta}{\alpha+\beta} \int_{\Omega}h(x) \vert u \vert ^{\alpha} \vert v \vert ^{\beta -2}v\varphi_{2}\,dx \end{aligned}$$

holds for all $(\varphi_{1},\varphi_{2})\in H_{0}^{\frac{s}{2}}(\Omega )\times H_{0}^{\frac{s}{2}}(\Omega)$.

Associated with $(E_{f,g})$ we consider the energy functional

$$ \begin{aligned} J_{f,g}(u,v)&:=\frac{1}{2} \int_{\Omega}\bigl( \bigl\vert (-\Delta)^{\frac{s}{4}}u \bigr\vert ^{2}+ \bigl\vert (-\Delta)^{\frac{s}{4}}v \bigr\vert ^{2}\bigr)\,dx\\ &\quad {}-\frac{1}{q} \int_{\Omega}\bigl(f(u_{+})^{q}+g(v_{+})^{q}\bigr) \,dx-\frac{1}{2_{s}^{*}} \int_{\Omega}h(u_{+})^{\alpha}(v_{+})^{\beta}\,dx, \end{aligned} $$

where $u_{+}=\max\{u,0\}$ and $v_{+}=\max\{v,0\}$. $J_{f,g}$ is well defined in $H_{0}^{\frac{s}{2}}(\Omega)\times H_{0}^{\frac{s}{2}}(\Omega)$, and moreover, the critical points of $J_{f,g}$ correspond to weak solutions of $(E_{f,g})$.

To treat the nonlocal problem $(E_{f,g})$, we will use an extension argument introduced by Caffarelli and Silvestre [27], which allows us to investigate $(E_{f,g})$ by studying a local problem via classical variational methods. We define the extension operator and fractional Laplacian for functions in $H_{0}^{\frac{s}{2}}(\Omega)$.

Definition 2.2

Given a function $u\in H_{0}^{\frac{s}{2}}(\Omega)$, we define its s-harmonic extension $\omega=E_{s}(u)$ to the cylinder $C_{\Omega}$ as a solution to the problem

$$ \textstyle\begin{cases} \operatorname{div}(y^{1-s}\nabla\omega)=0, &\text{in } C_{\Omega},\\ \omega=0,&\text{on } \partial_{L} C_{\Omega},\\ \omega=u,& \text{on } \Omega\times\{0\}, \end{cases} $$

and

$$ (-\Delta)^{\frac{s}{2}}u(x)=-K_{s}\lim_{y\rightarrow0^{+}}y^{1-s} \frac{\partial\omega}{\partial y}(x,y), $$

where $K_{s}$ is a normalization constant.

The extension function $\omega(x,y)$ belongs to the space $X_{0}^{s}(C_{\Omega})=\overline{C_{0}^{\infty}(C_{\Omega})}$ under the norm

$$ \Vert \omega \Vert _{X_{0}^{s}(C_{\Omega})}= \biggl(K_{s} \int _{C_{\Omega}} y^{1-s} \vert \nabla\omega \vert ^{2}\,dx\,dy \biggr)^{\frac{1}{2}}. $$

The extension operator is an isometry between $H_{0}^{\frac{s}{2}}(\Omega )$ and $X_{0}^{s}(C_{\Omega})$, namely

$$ \Vert \omega \Vert _{X_{0}^{s}(C_{\Omega})}= \Vert u \Vert _{H_{0}^{\frac{s}{2}}(\Omega)},\quad \forall u\in H_{0}^{\frac{s}{2}}(\Omega). $$

(2.1)

With this extension, we can transform $(E_{\lambda,\mu})$ into the following local problem:

$$ (\widehat{E}_{f,g}) \textstyle\begin{cases} -\operatorname{div}(y^{1-s}\nabla\omega_{1})=0,\qquad-\operatorname{div}(y^{1-s}\nabla\omega _{1})=0,&\text{in } C_{\Omega},\\ \omega_{1}=\omega_{2}=0,&\text{on } \partial_{L} C_{\Omega},\\ \frac{\partial\omega_{1}}{\partial\nu^{s}}=f(x) \vert \omega _{1} \vert ^{q-2}\omega_{1}+\frac{\alpha}{\alpha+\beta}h(x) \vert \omega_{1} \vert ^{\alpha-2}\omega_{1} \vert \omega _{2} \vert ^{\beta}, &\text{on } C_{\Omega}\times\{0\},\\ \frac{\partial\omega_{2}}{\partial\nu^{s}}=g(x) \vert \omega _{2} \vert ^{q-2}\omega_{1}+\frac{\beta}{\alpha+\beta}h(x) \vert \omega_{1} \vert ^{\alpha} \vert \omega_{2} \vert ^{\beta-2}\omega_{2}, &\text{on } C_{\Omega}\times\{0\},\\ \omega_{1}=u,\qquad\omega_{2}=v, &\text{on } C_{\Omega}\times\{0\}, \end{cases} $$

where

$$ \frac{\partial\omega_{i}}{\partial\nu^{s}}:=-K_{s}\lim_{y\rightarrow 0^{+}}y^{1-s} \frac{\partial\omega_{i}}{\partial y},\quad i=1,2. $$

In the following, we will study $(\widehat{E}_{f,g})$ in the framework of the Sobolev space $X=X_{0}^{s}(C_{\Omega})\times X_{0}^{s}(C_{\Omega})$ using the standard norm

$$ \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}= \biggl(K_{s} \int_{\Omega}y^{1-s}\bigl( \vert \nabla \omega_{1} \vert ^{2}+ \vert \nabla \omega_{2} \vert ^{2}\bigr)\,dx\,dy \biggr)^{\frac{1}{2}}. $$

An energy solution to $(\widehat{E}_{f,g})$ is a function $(\omega _{1},\omega_{2})\in X$ satisfying

$$\begin{aligned} & K_{s} \int_{C_{\Omega}} y^{1-s}\nabla\omega_{1}\nabla \varphi_{1} \,dx\,dy+ K_{s} \int_{C_{\Omega}} y^{1-s}\nabla\omega_{2}\nabla \varphi_{2} \,dx\,dy \\ &\quad= \int_{\Omega\times\{0\}}\bigl(f(x) \vert \omega_{1} \vert ^{q-2}\omega_{1}\varphi_{1}+g(x) \vert \omega_{2} \vert ^{q-2}\omega_{2} \varphi_{2}\bigr)\,dx \\ &\qquad {}+\frac{\alpha}{\alpha+\beta} \int_{\Omega\times\{0\}} h(x) \vert \omega_{1} \vert ^{\alpha-2}\omega_{1} \vert \omega_{2} \vert ^{\beta}\varphi_{1}\,dx+\frac{\beta}{\alpha+\beta } \int_{\Omega\times\{0\}} h(x) \vert \omega_{1} \vert ^{\alpha} \vert \omega_{2} \vert ^{\beta-2} \omega_{2}\varphi _{2}\,dx, \end{aligned}$$

for all $(\varphi_{1},\varphi_{2})\in X$. If $(\omega_{1},\omega_{2})$ satisfies $(\widehat{E}_{f,g})$, then the trace $(u,v)=(\omega_{1}(\cdot,0),\omega_{2}(\cdot,0))$ is a solution of $(E_{f,g})$. The converse is also true. Therefore, both formulations are equivalent. We define the associated energy functional to $(\widehat{E}_{f,g})$ by

$$ \begin{aligned} I_{f,g}(\omega_{1},\omega_{2})&=\frac{1}{2} \bigl\Vert (\omega_{1},\omega _{2}) \bigr\Vert _{X}^{2}-\frac{1}{q} \int_{\Omega\times\{0\} }\bigl(f(x) (\omega_{1})_{+}^{q}+g(x) (\omega_{2})_{+}^{q}\bigr)\,dx\\ &\quad {}-\frac{1}{2_{s}^{*}} \int _{\Omega\times\{0\}} h(x) (\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx, \end{aligned} $$

where $(\omega_{1})_{+}=\max\{\omega_{1}(x,0),0\}$ and $(\omega_{2})_{+}=\max \{\omega_{2}(x,0),0\}$. Clearly, critical points of $I_{f,g}$ in X correspond to critical points of $J_{f,g}$ in $H_{0}^{\frac{s}{2}}(\Omega)\times H_{0}^{\frac{s}{2}}(\Omega)$.

In the following lemmas, we will list some relevant inequalities from [14, 15].

Lemma 2.1

For every $1\leq r\leq2_{s}^{*}$, and every $\omega\in X_{0}^{s}(C_{\Omega})$, we have

$$ \biggl( \int_{\Omega\times\{0\}} \vert \omega \vert ^{r}\,dx \biggr)^{\frac{2}{r}}\leq C \int_{C_{\Omega}} y^{1-s} \vert \nabla\omega \vert ^{2}\,dx\,dy, $$

for some positive constant C. Furthermore, the space $X_{0}^{s}(C_{\Omega})$ is compactly embedded into $L^{r}(\Omega)$, for every $r<2_{s}^{*}$.

Remark 2.1

When $r=2_{s}^{*}$, the best constant is denoted by $S(s,N)$, that is,

$$ S(s,N):=\inf_{\omega\in X_{0}^{s}(C_{\Omega})\backslash\{0\}}\frac{\int _{C_{\Omega}} y^{1-s} \vert \nabla\omega \vert ^{2}\,dx\,dy}{ (\int_{\Omega\times\{0\}} \vert \omega \vert ^{2_{s}^{*}}\,dx )^{\frac{2}{2_{s}^{*}}}}. $$

(2.2)

It is not achieved in any bounded domain and, for all $\omega\in X^{s}(\mathbb{R}_{+}^{N+1})$,

$$ S(s,N) \biggl( \int_{\mathbb{R}^{N}\times\{0\}} \vert \omega \vert ^{2_{s}^{*}}\,dx \biggr)^{\frac{2}{2_{s}^{*}}}\leq \int_{\mathbb {R}_{+}^{N+1}}y^{1-s} \vert \nabla\omega \vert ^{2}\,dx\,dy $$

(2.3)

$S(s,N)$ is achieved for $\Omega=\mathbb{R}^{N}$ by functions $\omega _{\varepsilon}$, which are the s-harmonic extensions of

$$ u_{\varepsilon}(x):=\frac{\varepsilon^{\frac{(N-s)}{2}}}{(\varepsilon ^{2}+ \vert x \vert ^{2})^{\frac{(N-s)}{2}}},\quad\varepsilon >0, x\in \mathbb{R}^{N}. $$

(2.4)

The constant $S(s,N)$ given in (2.2) takes the exact value

$$ S(s,N)=\frac{2\pi^{\frac{s}{2}}\Gamma(\frac{2-s}{2})\Gamma(\frac{N+s}{2})(\Gamma(\frac{N}{2}))^{\frac{s}{N}}}{\Gamma(\frac{s}{2})\Gamma(\frac{N-s}{2})(\Gamma(N))^{\frac{s}{N}}}, $$

and it is achieved for $\Omega=\mathbb{R}^{N}$ by the functions $\omega_{\varepsilon}=E_{s}(u_{\varepsilon})$.

We consider the following minimization problem:

$$ S_{s,\alpha,\beta}:=\inf_{(\omega_{1},\omega_{2})\in X\backslash\{ (0,0)\}}\frac{\int_{C_{\Omega}} y^{1-s}( \vert \nabla\omega _{1} \vert ^{2}+ \vert \nabla\omega_{2} \vert ^{2})\,dx\,dy}{ (\int_{\Omega\times\{0\}} \vert \omega_{1} \vert ^{\alpha} \vert \omega_{2} \vert ^{\beta}\,dx )^{\frac{2}{2_{s}^{*}}}}. $$

(2.5)

From [15], we have a relationship between $S(s,N)$ and $S_{s,\alpha,\beta}$.

Lemma 2.2

For the constants $S(s,N)$ and $S_{s,\alpha,\beta}$ introduced in (2.2) and (2.5), we have

$$ S_{s,\alpha,\beta}= \biggl( \biggl(\frac{\alpha}{\beta} \biggr)^{\frac{\beta}{\alpha+\beta}}+ \biggl(\frac{\beta}{\alpha} \biggr)^{\frac{\alpha}{\alpha+\beta}} \biggr)S(s,N). $$

In particular,the constant $S_{s,\alpha,\beta}$ is achieved for $\Omega=\mathbb{R}^{N}$.

As $I_{f,g}$ is not bounded on X, we consider the behaviors of $I_{f,g}$ on the Nehari manifold setting

$$ N_{f,g}=\bigl\{ (\omega_{1},\omega_{2})\in X \backslash\bigl\{ (0,0)\bigr\} ;I'_{f,g}( \omega_{1},\omega_{2}) (\omega_{1}, \omega_{2})=0\bigr\} . $$

Clearly, $(\omega_{1},\omega_{2})\in N_{f,g}$ if and only if

$$ \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}^{2}= \int_{\Omega\times \{0\}}\bigl(f(x) (\omega_{1})_{+}^{q}+g(x) (\omega_{2})_{+}^{q}\bigr)\,dx+ \int_{\Omega\times \{0\}}h(x) (\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx. $$

For any $(\omega_{1},\omega_{2})\in N_{f,g}$, we have

$$ I_{f,g}(\omega_{1},\omega_{2})= \biggl( \frac{1}{2}-\frac{1}{q} \biggr) \bigl\Vert (\omega_{1}, \omega_{2}) \bigr\Vert _{X}^{2}+ \biggl( \frac{1}{q}-\frac{1}{2_{s}^{*}} \biggr) \int_{\Omega\times\{0\}}h(x) (\omega _{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx>0. $$

(2.6)

Thus $I_{f,g}$ is bounded from below on $N_{f,g}$. Let

$$ \begin{aligned}[b] &\psi_{f,g}(\omega_{1},\omega_{2}) \\ &\quad:=I'_{f,g}(\omega_{1},\omega_{2}) (\omega_{1},\omega_{2}) \\ &\quad= \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}^{2}- \int _{\Omega\times\{0\}}\bigl(f(x) (\omega_{1})_{+}^{q}+g(x) (\omega_{2})_{+}^{q}\bigr)\,dx- \int _{\Omega\times\{0\}}h(x) (\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx. \end{aligned} $$

(2.7)

Then, for $(\omega_{1},\omega_{2})\in N_{f,g}$,

$$\begin{aligned} & \psi'_{f,g}(\omega_{1},\omega_{2}) (\omega_{1},\omega_{2}) \\ &\quad=\bigl(2-2_{s}^{*}\bigr) \bigl\Vert (\omega_{1}, \omega_{2}) \bigr\Vert _{X}^{2}+ \bigl(2_{s}^{*}-q\bigr) \int_{\Omega\times\{0\}}\bigl(f(x) (\omega _{1})_{+}^{q}+g(x) (\omega_{2})_{+}^{q}\bigr)\,dx \end{aligned}$$

(2.8)

$$\begin{aligned} &\quad=(2-q) \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}^{2}-\bigl(2_{s}^{*}-q\bigr) \int_{\Omega\times\{0\}}h(x) (\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx< 0. \end{aligned}$$

(2.9)

Define

$$ \alpha_{f,g}=\inf_{(\omega_{1},\omega_{2})\in N_{f,g}}I_{f,g}(\omega _{1},\omega_{2}). $$

Then, analogous to [14, 15], we have the following results.

Lemma 2.3

Suppose that $(\omega^{0}_{1},\omega^{0}_{2})$ is a local minimizer for $I_{f,g}$ on $N_{f,g}$. Then $(\omega_{1},\omega_{2})$ is a critical point of $I_{f,g}$.

Proof

If $(\omega^{0}_{1},\omega^{0}_{2})\in N_{f,g}$ is a local minimizer of $I_{f,g}$, then $(\omega^{0}_{1},\omega^{0}_{2})$ is a nontrivial solution of the optimization problem

$$ \text{minimize }I_{f,g}(\omega_{1}, \omega_{2})\quad \text{subject to }\bigl\{ (\omega_{1}, \omega_{2}); \psi_{f,g}(\omega_{1},\omega _{2})=0\bigr\} . $$

Hence by the theory of multipliers, there exists a $\theta\in\mathbb {R}$ such that

$$ I'_{f,g}\bigl(\omega^{0}_{1}, \omega^{0}_{2}\bigr)=\theta\psi'_{f,g} \bigl(\omega ^{0}_{1},\omega^{0}_{2} \bigr). $$

This implies that $0=I'_{f,g}(\omega^{0}_{1},\omega^{0}_{2})(\omega ^{0}_{1},\omega^{0}_{2})=\theta\psi'_{f,g}(\omega^{0}_{1},\omega^{0}_{2})(\omega ^{0}_{1},\omega^{0}_{2})$. Moreover, noting (2.9), we get $\theta=0$, and so $I'_{f,g}(\omega^{0}_{1},\omega^{0}_{2})=0$. This completes the proof. □

Lemma 2.4

For each $(\omega_{1},\omega_{2})\in X$ with $\int_{\Omega\times\{0\} }h(x)(\omega_{1})_{+}^{\alpha}(\omega_{2})_{+}^{\beta}\,dx>0$, there is a $t_{(\omega_{1},\omega_{2})}$ such that $(t_{(\omega_{1},\omega_{2})}\omega _{1},t_{(\omega_{1},\omega_{2})}\omega_{2})\in N_{f,g}$ and

$$ I_{f,g}(t_{(\omega_{1},\omega_{2})}\omega_{1},t_{(\omega_{1},\omega _{2})} \omega_{2})=\sup_{t\geq0} I_{f,g}(t \omega_{1},t\omega_{2}). $$

Proof

For fixed $(\omega_{1},\omega_{2})\in X$ with $\int_{\Omega\times\{0\} }h(x)(\omega_{1})_{+}^{\alpha}(\omega_{2})_{+}^{\beta}\,dx>0$, consider

$$\begin{aligned} \varphi(t)&=I_{f,g}(t\omega_{1},t\omega_{2}) \\ &=\frac{t^{2}}{2} \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}^{2}-\frac{t^{q}}{q} \int_{\Omega\times\{0\}}\bigl(f(x) (\omega _{1})_{+}^{q}+g(x) (\omega_{2})_{+}^{q}\bigr)\,dx\\ &\quad {}-\frac{t^{2_{s}^{*}}}{2_{s}^{*}} \int_{\Omega \times\{0\}} h(x) (\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx. \end{aligned}$$

Because $2< q<2_{s}^{*}$, $\sup_{t\geq0}\varphi(t)$ is achieved at some $t_{(\omega_{1},\omega_{2})}>0$. This means $\varphi'(t_{(\omega _{1},\omega_{2})})=0$, i.e. $(t_{(\omega_{1},\omega_{2})}\omega _{1},t_{(\omega_{1},\omega_{2})}\omega_{2})\in N_{f,g}$. □

Lemma 2.5

We have

$$ \alpha_{f,g}\geq d_{0}\quad\textit{for some }d_{0}>0. $$

Proof

Set $(\omega_{1},\omega_{2})\in N_{f,g}$, then we from Lemma 2.1 obtain

$$ \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}^{2}\leq C\bigl( \bigl\Vert (\omega_{1}, \omega_{2}) \bigr\Vert _{X}^{q}+ \bigl\Vert ( \omega_{1},\omega _{2}) \bigr\Vert _{X}^{2_{s}^{*}} \bigr), $$

i.e.

$$ 1\leq C\bigl( \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}^{q-2}+ \bigl\Vert (\omega_{1}, \omega_{2}) \bigr\Vert _{X}^{2_{s}^{*}-2}\bigr). $$

We deduce that

$$ \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}^{2}\geq C $$

for some $C>0$ independent of $(\omega_{1},\omega_{2})\in N_{f,g}$. Thus we have

$$ I_{f,g}(\omega_{1},\omega_{2})\geq d_{0}>0 $$

for some $d_{0}>0$ independent of $(\omega_{1},\omega_{2})\in N_{f,g}$. Consequently, we obtain the desired result. □

Next we establish that $I_{f,g}$ satisfies the $(PS)_{c}$-condition under some restriction on the level of $(PS)_{c}$-sequences in the following.

Lemma 2.6

$I_{f,g}$ satisfies the $(PS)_{c}$-condition for $c\in(-\infty,\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s})$.

Proof

Let $\{(\omega_{1,n},\omega_{2,n})\}\subset X$ be a $(PS)_{c}$-sequence for $I_{f,g}$ and $c\in(-\infty,\frac{s}{2N}(K_{s}S_{s,\alpha,\beta })^{N/s})$. Noting (2.6), it is easy to obtain $\{(\omega_{1,n},\omega _{2,n})\}$ is bounded in X. Thus, there exists a subsequence still denoted by $\{ (\omega_{1,n},\omega_{2,n})\}$ and $(\omega_{1},\omega_{2})\in X$ such that $(\omega_{1,n},\omega_{2,n})\rightharpoonup(\omega _{1},\omega_{2})$ weakly in X. Furthermore, we get $I'_{f,g}(\omega _{1},\omega_{2})=0$ and

• $\int_{\Omega\times\{0\}}(f(x)(\omega_{1,n})_{+}^{q}+g(x)(\omega _{2,n})_{+}^{q})\,dx=\int_{\Omega\times\{0\}}(f(x)(\omega _{1})_{+}^{q}+g(x)(\omega_{2})_{+}^{q})\,dx+o(1)$;

• $\Vert (\omega_{1,n}-\omega_{1},\omega_{2,n}-\omega _{2}) \Vert _{X}^{2}= \Vert (\omega_{1,n},\omega_{2,n}) \Vert _{X}^{2}- \Vert (\omega_{1},\omega_{2}) \Vert _{X}^{2}+o(1)$;

Moreover, by the Brezis-Lieb lemma, we can obtain

$$\begin{aligned} \int_{\Omega\times\{0\}}h(x) (\omega_{1,n}-\omega_{1})_{+}^{\alpha}(\omega_{2,n}-\omega_{2})_{+}^{\beta}\,dx&= \int_{\Omega\times\{0\} }h(x) (\omega_{1,n})_{+}^{\alpha}( \omega_{2,n})_{+}^{\beta}\,dx \\ &\quad {}- \int _{\Omega\times\{0\}}h(x) (\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx+o(1). \end{aligned}$$

Since $I_{f,g}(\omega_{1,n},\omega_{2,n})=c+o(1)$ and $I'_{f,g}(\omega _{1,n},\omega_{2,n})=o(1)$, we deduce that

$$\begin{aligned} &\frac{1}{2} \bigl\Vert (\omega_{1,n}-\omega_{1}, \omega_{2,n}-\omega _{2}) \bigr\Vert _{X}^{2}- \frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\} }h(x) (\omega_{1,n}-\omega_{1})_{+}^{\alpha}( \omega_{2,n}-\omega _{2})_{+}^{\beta}\,dx \\ &\quad =c-I_{f,g}(\omega_{1},\omega_{2})+o(1) \end{aligned}$$

(2.10)

and

$$\begin{aligned} o(1)&=I'_{f,g}(\omega_{1,n}, \omega_{2,n}) (\omega_{1,n}-\omega _{1}, \omega_{2,n}-\omega_{2}) \\ &=\bigl(I'_{f,g}(\omega_{1,n}, \omega_{2,n})-I'_{f,g}(\omega_{1},\omega _{2})\bigr) (\omega_{1,n}-\omega_{1}, \omega_{2,n}-\omega_{2}) \\ &= \bigl\Vert (\omega_{1,n}-\omega_{1},\omega_{2,n}- \omega_{2}) \bigr\Vert _{X}^{2}- \int_{\Omega\times\{0\}}h(x) (\omega_{1,n}-\omega _{1})_{+}^{\alpha}(\omega_{2,n}-\omega_{2})_{+}^{\beta}\,dx+o(1). \end{aligned}$$

(2.11)

Now we may assume that

$$ \begin{gathered} \bigl\Vert (\omega_{1,n}-\omega_{1},\omega_{2,n}- \omega_{2}) \bigr\Vert _{X}^{2}\rightarrow l\quad \text{and}\\ \int_{\Omega\times\{ 0\}}h(x) (\omega_{1,n}-\omega_{1})_{+}^{\alpha}( \omega_{2,n}-\omega _{2})_{+}^{\beta}\,dx\rightarrow l \quad\text{as }n\rightarrow\infty, \end{gathered} $$

for some $l\in[0,+\infty)$.

Suppose $l\neq0$ and notice that $h\leq1$, using (2.5), (2.11) and passing to the limit as $n\rightarrow\infty$, we have

$$ l\geq K_{s}S_{s,\alpha,\beta}l^{\frac{2}{2_{s}^{*}}}, $$

that is,

$$ l\geq (K_{s}S_{s,\alpha,\beta} )^{N/s}. $$

(2.12)

Then by (2.10)-(2.12) and $(\omega_{1},\omega_{2})\in N_{f,g}\cup\{ (0,0)\}$, we have

$$ c=I_{f,g}(\omega_{1},\omega_{2})+ \biggl( \frac{1}{2}-\frac{1}{2_{s}^{*}} \biggr)l\geq\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}, $$

which contradicts the definition of c. Hence $l=0$, and the proof is completed. □

3 Some technical results

In this section, we shall introduce some useful results which are crucial for the proof of Theorem 1.1.

Lemma 3.1

Let $\{(\omega_{1,n},\omega_{2,n})\}\subset X$ be a non-negative function sequence with

$$ \int_{\Omega\times\{0\}}(\omega_{1,n})_{+}^{\alpha}(\omega _{2,n})_{+}^{\beta}\,dx=1\quad\textit{and}\quad \bigl\Vert (\omega _{1,n},\omega_{2,n}) \bigr\Vert _{X}^{2} \rightarrow K_{s}S_{s,\alpha,\beta}. $$

Then there exists a sequence $\{(y_{n},\varepsilon_{n})\}\subset\mathbb {R}^{N}\times\mathbb{R}^{+}$ such that

$$ \bigl(W_{1,n}(x,y),W_{2,n}(x,y)\bigr):=(E_{s}\bigl( \varepsilon_{n}^{\frac{N-s}{2}}\omega _{1,n}( \varepsilon_{n}x+y_{n},0)\bigr),E_{s}\bigl( \varepsilon_{n}^{\frac{N-s}{2}}\omega_{2,n}( \varepsilon_{n}x+y_{n},0)\bigr) $$

contains a convergent subsequence denoted again by $\{ (W_{1,n}(x,y),W_{2,n}(x,y))\}$ such that

$$ \bigl(W_{1,n}(x,y),W_{2,n}(x,y)\bigr)\rightarrow(W_{1},W_{2}) \quad\textit{in }X. $$

Moreover, we have $\varepsilon_{n}\rightarrow0$ and $y_{n}\rightarrow y_{0}\in\overline{\Omega}$ as $n\rightarrow\infty$.

Proof

Let $Z_{n,1}(x)=\omega_{1,n}(x,0)$, $Z_{n,2}(x)=\omega_{2,n}(x,0)$, we have

$$ \int_{\Omega}(Z_{n,1})_{+}^{\alpha}(Z_{n,2})_{+}^{\beta}\,dx=1\quad\text{and}\quad \Vert Z_{n,1} \Vert ^{2}_{H_{0}^{s}(\Omega)}+ \Vert Z_{n,1} \Vert ^{2}_{H_{0}^{s}(\Omega)}\rightarrow K_{s}S_{\alpha ,\beta}\quad\text{as }n\rightarrow\infty. $$

By the proof of Lemma 2.2, we know that $\{Z_{n,1}\}$ and $\{Z_{n,2}\}$ are minimizing sequences for the critical Sobolev inequality in the form (2.2). Thus from [28], Theorem 3, and [28], Theorem 5, we deduce that there exist a sequence of points $\{y_{n}\}\subseteq \mathbb{R}^{N}$ and a sequence of numbers $\{\varepsilon_{n}\}\subset (0,\infty)$ such that $\widehat{Z}_{n,1}(x)=\varepsilon_{n}^{\frac{N-s}{2}}Z_{n,1}(\varepsilon_{n}x+y_{n})\rightarrow\widehat{Z}_{1}(x)$ and $\widehat{Z}_{n,2}(x)=\varepsilon_{n}^{\frac{N-s}{2}}Z_{n,2}(\varepsilon_{n}x+y_{n})\rightarrow\widehat{Z}_{2}(x)$ in $H^{s}(\mathbb{R}^{N})$ as $n\rightarrow\infty$. Moreover, we have $\varepsilon_{n}\rightarrow0$ and $y_{n}\rightarrow y_{0}\in\overline {\Omega}$ as $n\rightarrow\infty$. Denote $W_{1,n}=E_{s}(\widehat{Z}_{n,1})$, $W_{2,n}=E_{s}(\widehat {Z}_{n,2})$ and $W_{1}=E_{s}(\widehat{Z}_{1})$, $W_{2}=E_{s}(\widehat{Z}_{2})$. Then we obtain the result. □

Next, we will use $\omega_{\varepsilon}=E_{s}(u_{\varepsilon})$, the family of minimizers to the inequality (2.2), where $u_{\varepsilon}$ is given in (2.4). Let $\eta\in C^{\infty}(C_{\Omega})$, $0\leq\eta (x,y)\leq1$ and for small fixed $\rho_{0}$,

$$ \eta(x,y)= \textstyle\begin{cases} 1, &(x,y)\in B_{\frac{\rho_{0}}{2}}^{+}:=\{(x,y); \vert (x,y) \vert < \frac{\rho_{0}}{2},y>0\},\\ 0, &(x,y)\notin B_{\rho_{0}}^{+}:=\{(x,y); \vert (x,y) \vert >\rho_{0},y>0\}. \end{cases} $$

We take $\rho_{0}< r_{0}$ small enough such that

$$ \overline{B_{\rho_{0}}^{+}}(x-z,y)\subset\overline{C_{\Omega}} $$

for all $z\in M$, where

$$ \overline{B_{\rho_{0}}^{+}}(x-z,y):=\bigl\{ (x,y); \bigl\vert (x-z,y) \bigr\vert \leq\rho_{0},y\geq0\bigr\} . $$

For any $z\in M$, we define

$$ v_{\varepsilon,z}=\eta(x-z,y)\omega_{\varepsilon}(x-z,y)=\eta (x-z,y)E_{s}\bigl(u_{\varepsilon}(x-z)\bigr). $$

From the same argument as in [15, 29] we obtain

$$\begin{aligned}& \Vert v_{\varepsilon,z} \Vert _{X_{0}^{s}(C_{\Omega})}^{2} =K_{s} \int_{\mathbb{R}_{+}^{N+1}}y^{1-s} \vert \nabla\omega _{\varepsilon} \vert ^{2}\,dx\,dy+O\bigl(\varepsilon^{N-s} \bigr), \end{aligned}$$

(3.1)

$$\begin{aligned}& \int_{\Omega\times\{0\}} \vert v_{\varepsilon,z} \vert ^{2_{s}^{*}} \,dx = \int_{\mathbb{R}^{N}\times\{0\}} \vert \omega _{\varepsilon} \vert ^{2_{s}^{*}}\,dx+O\bigl(\varepsilon^{N}\bigr)= \int_{\mathbb {R}^{N}} \biggl(\frac{\varepsilon}{\varepsilon^{2}+ \vert x \vert ^{2}} \biggr)^{N} \,dx+O\bigl(\varepsilon^{N}\bigr). \end{aligned}$$

(3.2)

A straight calculation shows that

$$ \int_{\Omega\times\{0\}} \vert v_{\varepsilon,z} \vert ^{q}\,dx \geq C\varepsilon^{\frac{2N-(N-s)q}{2}}, $$

(3.3)

for ε small and some $C>0$. Then we have the following result.

Lemma 3.2

There exist $\varepsilon_{0}$, $\sigma(\varepsilon_{0})>0$ such that, for $\varepsilon\in(0,\varepsilon_{0})$ and $\sigma\in(0,\sigma (\varepsilon_{0}))$, we have

$$ \sup_{t\geq0}I_{f,g}(t\sqrt{\alpha}v_{\varepsilon,z},t \sqrt{\beta }v_{\varepsilon,z})< \frac{s}{2N}(K_{s}S_{s,\alpha,\beta })^{N/s}- \sigma\quad\textit{uniformly in } z\in M. $$

Furthermore, there exists $t_{z}>0$ such that

$$ (t_{z}\sqrt{\alpha}v_{\varepsilon,z},t_{z}\sqrt{ \beta}v_{\varepsilon ,z})\in N_{f,g}\quad\textit{for all } z\in M. $$

Proof

At first we shall show that

$$ \sup_{t\geq0}I_{f,g}(t\sqrt{\alpha}v_{\varepsilon,z},t \sqrt{\beta }v_{\varepsilon,z})< \frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s} \quad \text{for } \varepsilon>0 \text{ small enough} . $$

It follows from $2< q<2_{s}^{*}$ that

$$ \lim_{t\rightarrow0}I_{f,g}(t\sqrt{\alpha}v_{\varepsilon,z},t \sqrt {\beta}v_{\varepsilon,z})=0\quad\text{and}\quad\lim_{t\rightarrow +\infty}I_{f,g}(t \sqrt{\alpha}v_{\varepsilon,z},t\sqrt{\beta }v_{\varepsilon,z})=-\infty. $$

Thus, for all ε sufficiently small, there exist $t_{0}>0$ and $t_{1}>0$ such that

$$ I_{f,g}(t\sqrt{\alpha}v_{\varepsilon,z},t\sqrt{\beta }v_{\varepsilon,z})< \frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s} \quad\text{for all } t\in(0,t_{0}] $$

(3.4)

and

$$ I_{f,g}(t\sqrt{\alpha}v_{\varepsilon,z},t\sqrt{\beta }v_{\varepsilon,z})< \frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s} \quad\text{for all } t\in[t_{1},+\infty). $$

(3.5)

By the definition of $v_{\varepsilon,z}$, we get

$$\begin{aligned} \int_{\Omega\times\{0\}}h(x) (v_{\varepsilon,z})^{2_{s}^{*}}\,dx&= \int _{B_{r_{0}}(z)}h(x) \bigl(\eta(x-z,0)u_{\varepsilon}(x-z) \bigr)^{2_{s}^{*}}\,dx \\ &= \int_{\mathbb{R}^{N}}h(x+z) \bigl(\eta(x,0)u_{\varepsilon}(x) \bigr)^{2_{s}^{*}}\,dx \\ &= \int_{\mathbb{R}^{N}}h(x+z)\eta^{2_{s}^{*}}(x,0) \bigl(u_{\varepsilon}(x)\bigr)^{2_{s}^{*}}\,dx \\ &= \int_{\mathbb{R}^{N}}h(x+z)\eta^{2_{s}^{*}}(x,0)\frac{\varepsilon ^{N}}{(\varepsilon^{2}+ \vert x \vert ^{2})^{N}}\,dx. \end{aligned} $$

Thus, noting the condition (H₁), we obtain

$$\begin{aligned} 0&\leq \int_{\Omega\times\{0\}}(v_{\varepsilon,z})^{2_{s}^{*}}\,dx- \int _{\Omega\times\{0\}}h(x) (v_{\varepsilon,z})^{2_{s}^{*}}\,dx \\ &= \int_{\mathbb{R}^{N}}(v_{\varepsilon,z})^{2_{s}^{*}}\,dx- \int_{\mathbb {R}^{N}}h(x) (v_{\varepsilon,z})^{2_{s}^{*}}\,dx \\ &=\varepsilon^{N} \biggl( \int_{\mathbb{R}^{N}\backslash B_{\frac{\rho_{0}}{2}}}\frac{(1-h(x+z))\eta^{2_{s}^{*}}(x,0)}{(\varepsilon^{2}+ \vert x \vert ^{2})^{N}}\,dx+ \int_{ B_{\frac{\rho_{0}}{2}}}\frac{(1-h(x+z))}{(\varepsilon^{2}+ \vert x \vert ^{2})^{N}}\,dx \biggr) \\ &\leq\varepsilon^{N} \biggl( \int_{\mathbb{R}^{N}\backslash B_{\frac{\rho_{0}}{2}}}\frac{1}{ \vert x \vert ^{2N}}\,dx+ \int_{ B_{\frac{\rho_{0}}{2}}}\frac{ \vert x \vert ^{\rho}}{(\varepsilon^{2}+ \vert x \vert ^{2})^{N}}\,dx \biggr) \\ &\leq\varepsilon^{N} \biggl(C+C\varepsilon^{\rho-N} \int_{0}^{\frac{\rho_{0}}{2\varepsilon}} \frac{r^{\rho+N-1}}{(1+r^{2})^{N}}\,dr \biggr) \\ &\leq \textstyle\begin{cases} C\varepsilon^{N}, &\rho>N,\\ C\varepsilon^{N}\ln\frac{1}{\varepsilon}, &\rho=N, \end{cases}\displaystyle \end{aligned}$$

(3.6)

for all $z\in M$. It follows from Remark 1.1 and the definition of $v_{\varepsilon,z}$ that

$$ h(x)>0\quad\text{for all } x\in B_{r_{0}}(z)\quad\text{and} \quad v_{\varepsilon,z}=0\quad\text{for all } x\notin B_{r_{0}}(z). $$

(3.7)

From (3.1)-(3.7) and $q>\frac{2s}{N-s}$ we deduce that

$$\begin{aligned} &I_{f,g}(t\sqrt{\alpha}v_{\varepsilon,z},t\sqrt{\beta }v_{\varepsilon,z}) \\ &\quad \leq\frac{2_{s}^{*}}{2}t^{2} \Vert v_{\varepsilon,z} \Vert _{X_{0}^{s}(C_{\Omega})}^{2}-\frac{\alpha^{\frac{\alpha}{2}}\beta^{\frac{\beta}{2}}}{2_{s}^{*}}t^{2_{s}^{*}} \int_{\Omega\times\{0\} }h(x) (v_{\varepsilon,z})^{2_{s}^{*}}\,dx-C \varepsilon^{\frac{2N-(N-s)q}{2}} \\ &\quad \leq\frac{s}{2N} \biggl(\frac{(\alpha+\beta) \Vert v_{\varepsilon,z} \Vert _{X_{0}^{s}(C_{\Omega})}^{2}}{ (\int _{\Omega\times\{0\}}\alpha^{\frac{\alpha}{2}}\beta^{\frac{\beta}{2}}h(x)(v_{\varepsilon,z})^{2_{s}^{*}}\,dx )^{\frac{2}{2_{s}^{*}}}} \biggr)^{\frac{N}{s}}-C \varepsilon^{\frac{2N-(N-s)q}{2}} \\ &\quad =\frac{s}{2N} \biggl( \biggl( \biggl(\frac{\alpha}{\beta} \biggr)^{\frac{\beta}{\alpha+\beta}}+ \biggl(\frac{\beta}{\alpha} \biggr)^{\frac{\alpha}{\alpha+\beta}} \biggr)\frac{K_{s}\int_{\mathbb {R}_{+}^{N+1}}y^{1-s} \vert \nabla\omega_{\varepsilon} \vert ^{2}\,dx\,dy+O(\varepsilon^{N-s})}{ (\int_{\mathbb{R}^{N}} (\frac{\varepsilon}{\varepsilon^{2}+ \vert x \vert ^{2}} )^{N}\,dx+O(\varepsilon^{N}\ln\frac{1}{\varepsilon}) )^{\frac{2}{2_{s}^{*}}}} \biggr)^{\frac{N}{s}} \\ &\qquad {}-C\varepsilon^{\frac{2N-(N-s)q}{2}} \\ &\begin{aligned}[b] &\quad =\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{\frac{N}{s}}+O \bigl(\varepsilon ^{N-s}\bigr)-C\varepsilon^{\frac{2N-(N-s)q}{2}} \\ &\quad < \frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{\frac{N}{s}} \end{aligned} \end{aligned}$$

(3.8)

for ε sufficiently small and $t\in[t_{0},t_{1}]$. Note the compactness of M; it follows from (3.4)-(3.5) and (3.8) that there exist $\varepsilon_{0}$, $\sigma(\varepsilon)>0$ such that, for $\varepsilon\in(0,\varepsilon_{0})$ and $\sigma\in(0,\sigma (\varepsilon))$, we have

$$ \sup_{t\geq0}I_{f,g}(t\sqrt{\alpha}v_{\varepsilon,z},t \sqrt{\beta }v_{\varepsilon,z})< \frac{s}{2N}(K_{s}S_{s,\alpha,\beta })^{N/s}- \sigma\quad\text{uniformly in } z\in M. $$

By Lemma 2.4, we conclude that there exists $t_{z}>0$ such that

$$ (t_{z}\sqrt{\alpha}v_{\varepsilon,z},t_{z}\sqrt{ \beta}v_{\varepsilon ,z})\in N_{f,g}\quad\text{for all } z\in M. $$

 □

Related to $I_{f,g}$ and $N_{f,g}$, we define

$$ J_{h}(\omega_{1},\omega_{2})=\frac{1}{2} \bigl\Vert (\omega_{1},\omega _{2}) \bigr\Vert _{X}^{2}-\frac{1}{2_{s}^{*}} \int_{\Omega\times\{0\}} h(\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx $$

and

$$ N_{h}=\bigl\{ (\omega_{1},\omega_{2})\in X \backslash\bigl\{ (0,0)\bigr\} ;(J_{h})'(\omega _{1},\omega_{2}) (\omega_{1},\omega_{2})=0 \bigr\} . $$

Then we have the following.

Lemma 3.3

We have

$$ \inf_{(\omega_{1},\omega_{2})\in N_{h}} J_{h}(\omega_{1}, \omega_{2})=\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}. $$

Proof

Let $(\omega_{1},\omega_{2})\in N_{h}$, then

$$ \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}^{2}= \int_{\Omega\times \{0\}}h(x) (\omega_{1})_{+}^{\alpha}(( \omega_{2})_{+}^{\beta}\,dx. $$

(3.9)

By (2.5), we see

$$ \begin{aligned} K_{s}S_{s,\alpha,\beta} \biggl( \int_{\Omega\times\{0\}}h(x) (\omega _{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx \biggr)^{\frac{2}{2_{s}^{*}}}&\leq K_{s}S_{s,\alpha,\beta} \biggl( \int_{\Omega\times\{0\}}(\omega _{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx \biggr)^{\frac{2}{2_{s}^{*}}}\\ &\leq \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}^{2}, \end{aligned} $$

i.e.

$$ \int_{\Omega\times\{0\}}h(x) (\omega_{1})_{+}^{\alpha}(\omega _{2})_{+}^{\beta}\,dx\leq \biggl(\frac{1}{K_{s}S_{s,\alpha,\beta}} \biggr)^{\frac{2_{s}^{*}}{2}} \bigl\Vert (\omega_{1}, \omega_{2}) \bigr\Vert _{X}^{2_{s}^{*}}. $$

(3.10)

From (3.9) and (3.10) we deduce that

$$ \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}^{2}\geq(K_{s}S_{s,\alpha ,\beta})^{N/s}. $$

Then

$$ J_{h}(\omega_{1},\omega_{2})=\frac{s}{2N} \bigl\Vert (\omega_{1},\omega _{2}) \bigr\Vert _{X}^{2}\geq\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}, $$

and thus

$$ \inf_{(\omega_{1},\omega_{2})\in N_{h}} J_{h}(\omega_{1}, \omega_{2})\geq\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}. $$

(3.11)

Since

$$ \max_{t\geq0} \biggl(\frac{a}{2}t^{2}- \frac{b}{2_{s}^{*}}t^{2_{s}^{*}} \biggr)=\frac{s}{2N} \biggl( \frac{a}{b^{2/2_{s}^{*}}} \biggr)^{N/2}\quad\text{for any }a>0\text{ and } b>0, $$

by (3.1)-(3.2) and (3.6), we deduce that

$$\begin{aligned} \sup_{t\geq0}J_{h}(t\sqrt{\alpha}v_{\varepsilon,z},t \sqrt{\beta }v_{\varepsilon,z})&=\frac{s}{2N} \biggl(\frac{(\alpha+\beta)\int _{C_{\Omega}} \vert \nabla v_{\varepsilon,z} \vert ^{2}\,dx\,dy}{ (\alpha^{\frac{\alpha}{2}}\beta^{\frac{\beta}{2}}\int _{\Omega\times\{0\}} h(v_{\varepsilon,z})^{2_{s}^{*}}\,dx )^{2/2_{s}^{*}}} \biggr)^{N/s} \\ &=\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}+O\bigl( \varepsilon ^{N-s}\bigr). \end{aligned}$$

Then we obtain

$$ \inf_{(\omega_{1},\omega_{2})\in N_{h}} J_{h}(\omega_{1}, \omega_{2})\leq\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}, \quad\text{as }\varepsilon \rightarrow0^{+}. $$

(3.12)

The desired result follows from (3.11) and (3.12). □

4 Proof of Theorem 1.1

In this section, we use the idea of category to get positive solutions of $(E_{f,g})$ and give the proof of Theorem 1.1.

Initially, we give the following two propositions related to the category.

Proposition 4.1

Let R be a $C^{1,1}$ complete Riemannian manifold (modeled on a Hilbert space) and assume $F\in C^{1}(R,\mathbb{R})$ bounded from below. Let $-\infty<\inf_{R} F<a<b<+\infty$. Suppose that F satisfies (PS)-condition on the sublevel $\{u\in R; F(u)\leq b\}$ and that a is not a critical level for F. Then

$$ \sharp\bigl\{ u\in F^{a}; \nabla F(u)=0\bigr\} \geq \operatorname{cat}_{F^{a}} \bigl(F^{a}\bigr), $$

where $F^{a}\equiv\{u\in R; F(u)\leq a\}$.

Proof

See [30], Theorem 2.1. □

Proposition 4.2

Let Q, $\Omega^{+}$ and $\Omega^{-}$ be closed sets with $\Omega ^{-}\subset\Omega^{+}$; Let $\phi: Q\rightarrow\Omega^{+}$, $\varphi: \Omega^{-}\rightarrow Q$ be two continuous maps such that $\phi\circ \varphi$ is homotopically equivalent to the embedding $j:\Omega ^{-}\rightarrow\Omega^{+}$. Then $\operatorname{cat}_{Q}(Q)\geq \operatorname{cat}_{\Omega^{+}}(\Omega^{-})$.

Proof

See [30], Lemma 2.2. □

The proof of Theorem 1.1 is based on Propositions 4.1 and 4.2. Next, we define the continuous map $\Phi:X\backslash G\rightarrow\mathbb{R}^{N}$ by

$$ \Phi(\omega_{1},\omega_{2}):=\frac{\int_{\Omega\times\{0\}} x(\omega _{1})_{+}^{\alpha}(\omega_{2})_{+}^{\beta}\,dx}{\int_{\Omega\times\{0\}} (\omega_{1})_{+}^{\alpha}(\omega_{2})_{+}^{\beta}\,dx}, $$

where $G=\{(\omega_{1},\omega_{2})\in X;\int_{\Omega\times\{0\}} (\omega_{1})_{+}^{\alpha}(\omega_{2})_{+}^{\beta}\,dx=0\}$.

Lemma 4.1

For each $0<\delta<r_{0}$, there exists $\delta_{0}>0$ such that if $(\omega_{1},\omega_{2})\in N_{h}$, $J_{h}(\omega_{1},\omega_{2})<\frac{s}{2N}(K_{s}S_{s,\alpha,\beta })^{N/s}+\delta_{0}$, then

$$ \Phi(\omega_{1},\omega_{2})\in M_{\delta}. $$

Proof

Suppose the contrary. Then there exists a function sequence $\{(\omega _{1,n},\omega_{2,n})\}\subset N_{h}$ such that $J_{h}(\omega_{1,n},\omega _{2,n})=\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}+o(1)$, and

$$ \Phi(\omega_{1,n},\omega_{2,n})\notin M_{\delta}\quad \text{for all } n. $$

It is easy to see that $\{(\omega_{1,n},\omega_{2,n})\}$ is bounded in X. Furthermore, by Lemma 3.3 we have

$$\begin{aligned} \frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}&\leq\lim _{n\rightarrow \infty}J_{h}(\omega_{1,n}, \omega_{2,n})=\lim_{n\rightarrow\infty }\frac{s}{2N} \bigl\Vert (\omega_{1,n},\omega_{2,n}) \bigr\Vert _{X}^{2} \\ &=\lim_{n\rightarrow\infty}\frac{s}{2N} \int_{\Omega \times\{0\}}h(x) (\omega_{1,n})_{+}^{\alpha}( \omega_{2,n})_{+}^{\beta}\,dx\leq\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}. \end{aligned}$$

(4.1)

Define

$$ (W_{1,n},W_{2,n})= \biggl(\frac{(\omega_{1,n})_{+}}{ (\int_{\Omega \times\{0\}}(\omega_{1,n})_{+}^{\alpha}(\omega_{2,n})_{+}^{\beta}\,dx )^{1/(\alpha+\beta)}}, \frac{(\omega_{2,n})_{+}}{ (\int_{\Omega \times\{0\}}(\omega_{1,n})_{+}^{\alpha}(\omega_{2,n})_{+}^{\beta}\,dx )^{1/(\alpha+\beta)}} \biggr), $$

we see that $\int_{\Omega\times\{0\}}(W_{1,n})_{+}^{\alpha}(W_{2,n})_{+}^{\beta}\,dx=1$. It follows from (4.1) and the definition of $S_{s,\alpha,\beta}$ that

$$\begin{aligned} K_{s}S_{s,\alpha,\beta}&\leq \bigl\Vert (W_{1,n},W_{2,n}) \bigr\Vert _{X}^{2}=\frac{ \Vert (\omega_{1,n},\omega_{2,n}) \Vert _{X}^{2}}{ (\int_{\Omega\times\{0\}}(\omega_{1,n})_{+}^{\alpha}(\omega_{2,n})_{+}^{\beta}\,dx )^{\frac{2}{2_{s}^{*}}}} \\ &\leq\frac{ \Vert (\omega_{1,n},\omega_{2,n}) \Vert _{X}^{2}}{ (\int_{\Omega\times\{0\}}h(x)(\omega_{1,n})_{+}^{\alpha}(\omega_{2,n})_{+}^{\beta}\,dx )^{\frac{2}{2_{s}^{*}}}} \\ &= \bigl\Vert (\omega_{1,n},\omega_{2,n}) \bigr\Vert _{X}^{\frac{2s}{N}}\leq K_{s}S_{s,\alpha,\beta}. \end{aligned}$$

Hence we obtain

$$ \lim_{n\rightarrow\infty} \bigl\Vert (W_{1,n},W_{2,n}) \bigr\Vert _{X}^{2}=K_{s}S_{s,\alpha,\beta} $$

(4.2)

and

$$ \lim_{n\rightarrow\infty} \int_{\Omega\times\{0\}}h(x) (\omega _{1,n})_{+}^{\alpha}( \omega_{2,n})_{+}^{\beta}\,dx=\lim_{n\rightarrow\infty } \int_{\Omega\times\{0\}}(\omega_{1,n})_{+}^{\alpha}(\omega _{2,n})_{+}^{\beta}\,dx . $$

(4.3)

By Lemma 3.1, there is a sequence $\{(y_{n},\varepsilon_{n})\}\in\mathbb {R}^{N}\times\mathbb{R}^{+}$ such that $\varepsilon_{n}\rightarrow0$, $y_{n}\rightarrow y_{0}\in\overline{\Omega }$ and $(U_{1,n}(x,y),U_{2,n}(x,y))=(E_{s}(\varepsilon_{n}^{\frac{N-s}{2}}W_{1,n}(\varepsilon_{n}x+y_{n})),E_{s}(\varepsilon_{n}^{\frac{N-s}{2}}W_{2,n}(\varepsilon_{n}x+y_{n})))\rightarrow(U_{1},U_{2})$ in X as $n\rightarrow\infty$. Then by (4.1)-(4.3), we have

$$\begin{aligned} 1&=o(1)+ \int_{\Omega\times\{0\}}h(x) (W_{1,n})_{+}^{\alpha}(W_{2,n})_{+}^{\beta}\,dx \\ &=\varepsilon_{n}^{-N} \int_{\Omega\times\{0\}}h(x) \biggl(U_{1,n} \biggl( \frac{x-y_{n}}{\varepsilon_{n}},0 \biggr) \biggr)_{+}^{\alpha}\biggl(U_{2,n} \biggl(\frac{x-y_{n}}{\varepsilon_{n}},0 \biggr) \biggr)_{+}^{\beta}\,dx+o(1) \\ &=h(y_{0}), \end{aligned}$$

as $n\rightarrow\infty$, which implies $y_{0}\in M$. Considering $\varphi\in C_{0}^{\infty}(\mathbb{R}^{N})$ such that $\varphi (x)=x$ in Ω, we infer

$$\begin{aligned} &\Phi(\omega_{1,n},\omega_{2,n})\\ &\quad =\frac{\int_{\Omega\times\{0\}} x(\omega_{1,n})_{+}^{\alpha}(\omega_{2,n})_{+}^{\beta}\,dx}{\int_{\Omega \times\{0\}}(\omega_{1,n})_{+}^{\alpha}(\omega_{2,n})_{+}^{\beta}\,dx} \\ &\quad =\frac{\int_{\mathbb{R}^{N}\times\{0\}}\varphi(x)(\omega _{1,n})_{+}^{\alpha}(\omega_{2,n})_{+}^{\beta}\,dx}{\int_{\mathbb{R}^{N}\times \{0\}}(\omega_{1,n})_{+}^{\alpha}(\omega_{2,n})_{+}^{\beta}\,dx} \\ &\quad =\frac{\int_{\mathbb{R}^{N}\times\{0\}}\varphi(\varepsilon _{n}x+y_{n}) \vert E_{s}(\varepsilon_{n}^{\frac{N-s}{2}}W_{1,n}(\varepsilon_{n}x+y_{n})) \vert ^{\alpha} \vert E_{s}(\varepsilon_{n}^{\frac{N-s}{2}}W_{1,n}(\varepsilon_{n}x+y_{n})) \vert ^{\beta}\,dx}{\int_{\mathbb{R}^{N}\times\{0\}} \vert E_{s}(\varepsilon_{n}^{\frac{N-s}{2}}W_{1,n}(\varepsilon_{n}x+y_{n})) \vert ^{\alpha} \vert E_{s}(\varepsilon_{n}^{\frac{N-s}{2}}W_{1,n}(\varepsilon_{n}x+y_{n})) \vert ^{\beta}\,dx} \\ &\quad \rightarrow y_{0}\in M,\quad\text{as }n\rightarrow\infty, \end{aligned}$$

as $n\rightarrow\infty$, which is a contradiction. □

Lemma 4.2

There exists $\Lambda_{\delta}>0$ small enough such that if $\Vert f_{+} \Vert _{L^{q^{*}}}+ \Vert g_{+} \Vert _{L^{q^{*}}}<\Lambda_{\delta}$ and $(\omega_{1},\omega_{2})\in N_{f,g}$ with $I_{f,g}(\omega_{1},\omega_{2})<\frac{s}{2N}(K_{s}S_{s,\alpha,\beta })^{N/s}+\frac{\delta_{0}}{2}$ ($\delta_{0}$ is given in Lemma  4.1), then $\Phi(\omega_{1},\omega_{2})\in M_{\delta}$.

Proof

For $A, B>0$, consider

$$ \overline{h}(t)=At^{2}-Bt^{2_{s}^{*}}, \quad t\geq0. $$

Then

$$ \max_{t\geq0}\overline{h}(t)=\frac{{2_{s}^{*}}-2}{{2_{s}^{*}}}A \biggl( \frac{2A}{{2_{s}^{*}}B} \biggr)^{\frac{2}{{2_{s}^{*}}-2}}\rightarrow \infty,\quad \text{as }B\rightarrow0^{+}. $$

Hence for $(\omega_{1},\omega_{2})\in N_{f,g}$ with $I_{f,g}(\omega _{1},\omega_{2})<\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}+\frac{\delta_{0}}{2}$, we have

$$ \int_{\Omega\times\{0\}}h(x) (\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx>0, $$

when $\Vert f_{+} \Vert _{L^{q^{*}}}+ \Vert g_{+} \Vert _{L^{q^{*}}}$ is sufficiently small. Thus we see that there exists the unique positive number

$$ t_{h}= \biggl(\frac{ \Vert (\omega_{1},\omega_{2}) \Vert _{X}^{2}}{\int_{\Omega\times\{0\}}h(x)(\omega_{1})_{+}^{\alpha}(\omega _{2})_{+}^{\beta}\,dx} \biggr)^{\frac{1}{2_{s}^{*}-2}}>0 $$

such that $(t_{h}\omega_{1},t_{h}\omega_{2})\in N_{h}$.

We claim that $t_{h}< C$ for some C independent of $(\omega_{1},\omega _{2})\in N_{f,g}$ if $\Vert f_{+} \Vert _{L^{q^{*}}}+ \Vert g_{+} \Vert _{L^{q^{*}}}$ is small enough. Indeed,

$$\begin{aligned} \frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}+ \frac{\delta_{0}}{2}&\geq I_{f,g}(\omega_{1}, \omega_{2}) \\ &= \biggl(\frac{1}{2}-\frac{1}{q} \biggr) \bigl\Vert (\omega _{1},\omega_{2}) \bigr\Vert _{X}^{2}+ \biggl(\frac{1}{q}-\frac{1}{2_{s}^{*}} \biggr) \int_{\Omega\times\{0\}}h(x) (\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx \\ &\geq\frac{q-2}{2q} \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}^{2}. \end{aligned}$$

Thus

$$ \bigl\Vert (\omega_{1},\omega_{2}) \bigr\Vert _{X}^{2}\leq\frac{2q}{q-2} \biggl(\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}+ \frac{\delta_{0}}{2} \biggr). $$

(4.4)

Moreover,

$$\begin{aligned} I_{f,g}(\omega_{1},\omega_{2})&=\frac{1}{2} \bigl\Vert (\omega_{1},\omega _{2}) \bigr\Vert _{X}^{2}-\frac{1}{q} \int_{\Omega\times\{0\}} \bigl(f(\omega _{1})_{+}^{q}+g( \omega_{2})_{+}^{q}\bigr)\,dx\\ &\quad {}-\frac{1}{\alpha+\beta} \int_{\Omega \times\{0\}}h(x) (\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx, \\ &= \biggl(\frac{1}{2}-\frac{1}{q} \biggr) \int_{\Omega\times\{0\}} \bigl(f(\omega_{1})_{+}^{q}+g( \omega_{2})_{+}^{q}\bigr)\,dx\\ &\quad {}+ \biggl(\frac{1}{2}- \frac{1}{2_{s}^{*}} \biggr) \int_{\Omega\times\{0\}}h(x) (\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx. \end{aligned}$$

Therefore,

$$\begin{aligned} & \int_{\Omega\times\{0\}}h(x) (\omega_{1})_{+}^{\alpha}(\omega _{2})_{+}^{\beta}\,dx \\ &\quad =\frac{2N}{s} \biggl(I_{f,g}(\omega_{1}, \omega_{2})- \biggl(\frac{1}{2}-\frac{1}{q} \biggr) \int_{\Omega\times\{0\}} \bigl(f(\omega _{1})_{+}^{q}+g( \omega_{2})_{+}^{q}\bigr)\,dx \biggr) \\ &\quad \geq\frac{2N}{s} \biggl(d_{0}- \biggl(\frac{1}{2}- \frac{1}{q} \biggr)C\bigl( \Vert f_{+} \Vert _{L^{q^{*}}}+ \Vert g_{+} \Vert _{L^{q^{*}}}\bigr) \bigl\Vert (\omega_{1}, \omega_{2}) \bigr\Vert _{X}^{q} \biggr), \end{aligned}$$

(4.5)

where the last inequality follows from Lemmas 2.1 and 2.5. By (4.4) and (4.5), we get, for $\Vert f_{+} \Vert _{L^{q^{*}}}+ \Vert g_{+} \Vert _{L^{q^{*}}}$ small enough,

$$ \int_{\Omega\times\{0\}}h(x) (\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta}\,dx>C>0, $$

where C is independent of $(\omega_{1},\omega_{2})\in N_{f,g}$. Then we obtain the claim. Now,

$$\begin{aligned} \frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}+ \frac{\delta_{0}}{2}&\geq I_{f,g}(\omega_{1}, \omega_{2})=\sup_{t\geq0}I_{f,g}(t \omega_{1},t\omega _{2}) \\ &\geq I_{f,g}(t_{h}\omega_{1},t_{h} \omega_{2}) \\ &\geq J_{h}(t_{h}\omega_{1},t_{h} \omega_{2})-\frac{C}{q} \int_{\Omega\times \{0\}} \bigl(f_{+}(\omega_{1})_{+}^{q}+g_{+}( \omega_{2})_{+}^{q}\bigr)\,dx, \end{aligned}$$

which leads to

$$\begin{aligned} J_{h}(t_{h}\omega_{1},t_{h} \omega_{2})&\leq\frac{s}{2N}(K_{s}S_{s,\alpha,\beta })^{N/s}+ \frac{\delta_{0}}{2}+\frac{C}{q} \int_{\Omega\times\{0\}} \bigl(f_{+}(\omega_{1})_{+}^{q}+g_{+}( \omega_{2})_{+}^{q}\bigr)\,dx \\ &\leq\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}+ \frac{\delta _{0}}{2}+C\bigl( \Vert f_{+} \Vert _{L^{q^{*}}}+ \Vert g_{+} \Vert _{L^{q^{*}}}\bigr) \bigl\Vert (\omega_{1}, \omega_{2}) \bigr\Vert _{X}^{q}. \end{aligned}$$

Therefore, by (4.4) there exists $\Lambda_{\delta}>0$ such that, for $\Vert f_{+} \Vert _{L^{q^{*}}}+ \Vert g_{+} \Vert _{L^{q^{*}}}<\Lambda_{\delta}$,

$$ J_{h}(t_{h}\omega_{1},t_{h} \omega_{2})\leq\frac{s}{2N}(K_{s}S_{s,\alpha,\beta })^{N/s}+ \delta_{0}. $$

Since Lemma 4.1, we obtain $\Phi(t_{h}\omega_{1},t_{h}\omega_{2})\in M_{\delta}$ or $\Phi(\omega_{1},\omega_{2})\in M_{\delta}$. □

Below we denote by $I_{N_{f,g}}$ the restriction of $I_{f,g}$ on $N_{f,g}$.

Lemma 4.3

Any sequence $\{(\omega_{1,n},\omega_{2,n})\}\subset N_{f,g}$ such that $I_{N_{f,g}}(\omega_{1,n},\omega_{2,n})\rightarrow c\in(-\infty , \frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s})$ and $I'_{N_{f,g}}(\omega_{1,n},\omega_{2,n})\rightarrow0$ contains a convergent subsequence.

Proof

By hypothesis there exists a sequence $\{\theta_{n}\}\subset\mathbb {R}$ such that

$$ I'_{f,g}(\omega_{1,n},\omega_{2,n})= \theta_{n}\psi'_{f,g}(\omega _{1,n}, \omega_{2,n})+o(1), $$

where $\psi_{f,g}$ is defined in (2.7). Recall that $(\omega_{1,n},\omega_{2,n})\in N_{f,g}$ and so

$$ \psi'_{f,g}(\omega_{1,n},\omega_{2,n}) (\omega_{1,n},\omega_{2,n})< 0. $$

If $\psi'_{f,g}(\omega_{1,n},\omega_{2,n})(\omega_{1,n},\omega _{2,n})\rightarrow0$, we from (2.9) obtain

$$ \bigl\Vert (\omega_{1,n},\omega_{2,n}) \bigr\Vert _{X}^{2}\rightarrow 0,\quad\text{as }n\rightarrow\infty. $$

On the other hand, it follows from $(\omega_{1,n},\omega_{2,n})\in N_{f,g}$ that

$$ 1\leq C\bigl( \bigl\Vert (\omega_{1,n},\omega_{2,n}) \bigr\Vert _{X}^{q-2}+ \bigl\Vert (\omega_{1,n}, \omega_{2,n}) \bigr\Vert _{X}^{2_{s}^{*}-2}\bigr) \rightarrow0,\quad\text{as }n\rightarrow\infty. $$

That is,

$$ \bigl\Vert (\omega_{1,n},\omega_{2,n}) \bigr\Vert _{X}^{2}\geq C_{1}^{-\frac{2}{2_{s}^{*}-2}}+o(1) $$

Hence, we arrive at a contradiction. Thus we may assume that $\psi '_{f,g}(\omega_{1,n},\omega_{2,n})(\omega_{1,n},\omega _{2,n})\rightarrow l<0$ as $n\rightarrow\infty$. Because $I'_{f,g}(\omega_{1,n},\omega_{2,n})(\omega_{1,n},\omega_{2,n})=0$, we conclude that $\theta_{n}\rightarrow0$ and, consequently, $I'_{f,g}(\omega_{1,n},\omega_{2,n})\rightarrow0$. Using this information we have

$$ I_{f,g}(\omega_{1,n},\omega_{2,n})\rightarrow c\in \biggl(-\infty,\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s} \biggr)\quad\text{and}\quad I'_{f,g}(\omega_{1,n}, \omega_{2,n})\rightarrow0, $$

so by Lemma 2.6 the proof is over. □

Denote

$$ c_{0}:=\frac{s}{2N}(K_{s}S_{s,\alpha,\beta})^{N/s}- \sigma $$

(4.6)

and

$$ N_{f,g}(c_{0}):=\bigl\{ (\omega_{1}, \omega_{2})\in N_{f,g};I_{f,g}(\omega _{1}, \omega_{2})< c_{0}\bigr\} . $$

Lemma 4.4

Suppose (H₁)-(H₂) hold, and $\Vert f_{+} \Vert _{L^{q^{*}}(\Omega)}+ \Vert g_{+} \Vert _{L^{q^{*}}(\Omega )}\in(0,\Lambda_{\delta})$, $I_{N_{f,g}}$ has at least $\operatorname{cat}_{M_{\delta}}(M)$ critical points in $N_{f,g}(c_{0})$.

Proof

For $z\in M$, by Lemma 3.2, we can define

$$ F(z)=(t_{z}\sqrt{\alpha}v_{\varepsilon,z},t_{z}\sqrt{ \alpha }v_{\varepsilon,z})\in N_{f,g}(c_{0}). $$

By Lemma 4.3, $I_{N_{f,g}}$ satisfies $(PS)$-condition on $N_{f,g}(c_{0})$. Moreover, it follows from Lemma 4.2 that $\Phi (N_{f,g}(c_{0}))\subset M_{\delta}$ for $\Vert f_{+} \Vert _{L^{q^{*}}}+ \Vert g_{+} \Vert _{L^{q^{*}}}<\Lambda_{\delta}$. Define $\xi:[0,1]\times M\rightarrow M_{\delta}$ by

$$ \xi(\theta,z)=\Phi (t_{z}\sqrt{\alpha}v_{(1-\theta)\varepsilon ,z},t_{z} \sqrt{\beta}v_{(1-\theta)\varepsilon,z} )\in N_{f,g}(c_{0}). $$

Then straightforward calculations show that $\xi(0,z)=\Phi\circ F(z)$ and $\lim_{\theta\rightarrow1^{-}}\xi(\theta,z)=z$. Hence $\Phi\circ F$ is homotopic to the inclusion $j:M\rightarrow M_{\delta}$. By Propositions 4.1 and 4.2, $I_{f,g}$ has at least $\operatorname{cat}_{M_{\delta}}(M)$ critical points in $N_{f,g}(c_{0})$. □

Lemma 4.5

If $(\omega_{1},\omega_{2})$ is a critical point of $I_{N_{f,g}}$, then it is a critical point of $I_{f,g}$ in X.

Proof

Assume $(\omega_{1},\omega_{2})\in N_{f,g}$, then $I'_{f,g}(\omega _{1},\omega_{2})(\omega_{1},\omega_{2})=0$. On the other hand,

$$ I'_{f,g}(\omega_{1},\omega_{2})= \theta\psi'_{f,g}(\omega_{1}, \omega_{2}) $$

(4.7)

for some $\theta\in\mathbb{R}$, where $\psi_{f,g}$ is defined in (2.7).

Remark that $(\omega_{1},\omega_{2})\in N_{f,g}$, and so $\psi '_{f,g}(\omega_{1},\omega_{2})(\omega_{1},\omega_{2})<0$. Thus by (4.7),

$$ 0=\theta\psi'_{f,g}(\omega_{1}, \omega_{2}) (\omega_{1},\omega_{2}), $$

which implies that $\theta=0$, consequently $I'_{f,g}(\omega _{1},\omega_{2})=0$. □

Finally, we can give the proof of Theorem 1.1.

Proof of Theorem 1.1

It follows from Lemmas 4.4 and 4.5 that $I_{f,g}$ admits at least $\operatorname{cat}_{M_{\delta}}(M)$ non-negative critical points. Thus we see that $J_{f,g}$ has at least $\operatorname{cat}_{M_{\delta}}(M)$ non-negative critical points. By the maximum principle [31], we obtain the conclusion of Theorem 1.1. □

5 Proof of Theorem 1.2

In this section, we use the Morse theory to get positive solutions of $(E_{f,g})$ and give the proof of Theorem 1.2.

Let $\alpha>2$ and $\beta>2$, then $I_{f,g}$ is of class $C^{2}$ and for $(\omega_{1},\omega_{2}), (\varphi_{1},\varphi_{2}),(\psi_{1},\psi _{2})\in X$,

$$\begin{aligned} &I_{f,g}''(\omega_{1}, \omega_{2})\bigl[(\varphi_{1},\varphi_{2}),( \psi_{1},\psi _{2})\bigr] \\ &\quad = K_{s} \int_{C_{\Omega}} y^{1-s}(\nabla\varphi_{1}\nabla \psi_{1}+ \nabla\varphi_{2}\nabla\psi_{2})\,dx\,dy \\ &\qquad {}-(q-1) \int_{\Omega\times\{0\}}\bigl(f(x) (\omega_{1})_{+}^{q-2} \varphi _{1}\psi_{1}+g(x) (\omega_{2})_{+}^{q-2} \varphi_{2}\psi_{2}\bigr)\,dx \\ &\qquad {}-\frac{\alpha(\alpha-1)}{\alpha+\beta} \int_{\Omega\times\{0\} }h(x) (\omega_{1})_{+}^{\alpha-2}( \omega_{2})_{+}^{\beta}\varphi_{1}\psi _{1} \,dx\\ &\qquad {}-\frac{\alpha\beta}{\alpha+\beta} \int_{\Omega\times\{0\} }h(x) (\omega_{1})_{+}^{\alpha-1}( \omega_{2})_{+}^{\beta-1}\varphi_{1}\psi _{2}\,dx \\ &\qquad {}-\frac{\beta(\beta-1)}{\alpha+\beta} \int_{\Omega\times\{0\} }h(x) (\omega_{1})_{+}^{\alpha}( \omega_{2})_{+}^{\beta-2}\varphi_{2}\psi _{2} \,dx\\ &\qquad {}-\frac{\alpha\beta}{\alpha+\beta} \int_{\Omega\times\{0\} }h(x) (\omega_{1})_{+}^{\alpha-1}( \omega_{2})_{+}^{\beta-1}\varphi_{2}\psi _{1} \,dx. \end{aligned}$$

Hence $I_{f,g}''(\omega_{1},\omega_{2})$ is represented by the operator

$$ L(\omega_{1},\omega_{2}):=R(\omega_{1}, \omega_{2})-K(\omega_{1},\omega _{2}):X\rightarrow X^{-1}, $$

where $R(\omega_{1},\omega_{2})$ is the Riesz isomorphism and $K(\omega _{1},\omega_{2})$ is compact. For $a\in(0,+\infty]$, set

$$\begin{aligned}& I_{f,g}^{a}: =\bigl\{ (\omega_{1}, \omega_{2})\in X;I_{f,g}(\omega_{1},\omega _{2})\leq a\bigr\} ,\qquad N_{f,g}(a):=N_{f,g}\cap I_{f,g}^{a}, \\& \mathcal{K}: =\bigl\{ (\omega_{1},\omega_{2})\in X;I'_{f,g}(\omega _{1},\omega_{2})=0 \bigr\} ,\qquad\mathcal{K}^{a}:=\mathcal{K}\cap I_{f,g}^{a}, \\& \mathcal{K}_{a}: =\bigl\{ (\omega_{1},\omega_{2}) \in\mathcal {K};I_{f,g}(\omega_{1},\omega_{2})>a\bigr\} . \end{aligned}$$

Then $I_{f,g}$ satisfies the Palais-Smale condition on $N_{f,g}(c_{0})$, where $c_{0}$ is defined in (4.6). For a pair of topological spaces $(X,Y)$, $Y\subset X$, let $H_{*}(X;Y)$ be its singular homology and

$$ P(t) (X,Y)=\sum_{k} \operatorname{dim} H_{k}(X,Y)t^{k} $$

the Poincaré polynomial of the pair. If $Y=\emptyset$, it will be always omitted in the objects which involve the pair. In the remaining part of this section we will follow [32, 33]. We are going to prove that $I_{f,g}$ restricted to $N_{f,g}$ has at least $2P_{1}(M)-1$ critical points. Then Theorem 1.2 will follow from Lemma 4.5.

For any $z\in M$, by Lemma 3.2, we can define

$$ \Psi:z\in M\mapsto(t_{z}\sqrt{\alpha}v_{\varepsilon,z},t_{z} \sqrt {\beta}v_{\varepsilon,z})\in N_{f,g}(c_{0}), $$

for some $\varepsilon>0$ small enough. Since Ψ is injective, it induces injective homomorphisms in the homology groups, then $\operatorname{dim} H_{k}(M)\leq \operatorname{dim} H_{k}(N_{f,g}(c_{0}))$ and consequently

$$ P_{t}\bigl(N_{f,g}(c_{0})\bigr)=P_{t}(M)+Q(t), \quad Q\in\mathbb{P}, $$

(5.1)

where $\mathbb{P}$ denotes the set of polynomials with non-negative integer coefficients.

The following result is analogous to [33], Lemma 5.2, and we omit the proof.

Lemma 5.1

Let $r\in(0,\alpha_{f,g})$ and $a\in(r,+\infty]$ a regular level for $I_{f,g}$. Then

$$ P_{t}\bigl(I^{a}_{f,g},I^{r}_{f,g} \bigr)=tP_{t}\bigl(N^{a}_{f,g}\bigr). $$

(5.2)

In particular we have the following.

Lemma 5.2

Let $r\in(0,\alpha_{f,g})$. Then

$$\begin{aligned}& P_{t}\bigl(I^{c_{0}}_{f,g},I^{r}_{f,g} \bigr) =t\bigl(P_{t}(M)+Q(t)\bigr),\quad Q\in\mathbb {P}, \\& P_{t}\bigl(X,I^{r}_{f,g}\bigr) =t. \end{aligned}$$

Proof

The first identity follows by (5.1) and (5.2) by choosing $a=c_{0}$. The second one follows by (5.2) with $a=+\infty$ and noticing that $N_{f,g}$ is contractible. □

To deal with critical points above the level $c_{0}$, we need also the following.

Lemma 5.3

We have

$$\begin{aligned} P_{t}\bigl(X,I^{c_{0}}_{f,g}\bigr)&=t^{2} \bigl(P_{t}(M)+Q(t)-1\bigr),\quad Q\in\mathbb{P}. \end{aligned}$$

Proof

The proof is purely algebraic and goes exactly as in [33], Lemma 5.6; see also [32], Lemma 2.4. □

As a consequence of these facts we have the following.

Lemma 5.4

Suppose that $\mathcal{K}$ is discrete. Then

$$\begin{aligned}& \sum_{(\omega_{1},\omega_{2})\in\mathcal{K}^{c_{0}}}\mathcal {I}_{t}( \omega_{1},\omega_{2}) =t\bigl(P_{t}(M)+Q(t) \bigr)+(1+t)Q_{1}(t), \\& \sum_{(\omega_{1},\omega_{2})\in\mathcal{K}_{c_{0}}}\mathcal {I}_{t}( \omega_{1},\omega_{2}) =t^{2}\bigl(P_{t}(M)+Q(t)-1 \bigr)+(1+t)Q_{2}(t), \end{aligned}$$

where $\mathcal{I}_{t}(\omega_{1},\omega_{2})$ denotes the polynomial Morse index of $(\omega_{1},\omega_{2})$ and $Q, Q_{1}, Q_{2}\in\mathbb{P}$.

Proof

Indeed Morse theory gives

$$\begin{gathered} \sum_{(\omega_{1},\omega_{2})\in\mathcal{K}^{c_{0}}}\mathcal {I}_{t}( \omega_{1},\omega _{2}) =P_{t}\bigl(I^{c_{0}}_{f,g},I^{r}_{f,g} \bigr)+(1+t)Q_{1}(t), \\ \sum_{(\omega_{1},\omega_{2})\in\mathcal{K}_{c_{0}}}\mathcal {I}_{t}( \omega_{1},\omega_{2}) =P_{t}\bigl(X,I^{c_{0}}_{f,g} \bigr)+(1+t)Q_{2}(t). \end{gathered} $$

By using Lemmas 5.2 and 5.3, we obtain the result. □

Finally, it follows from Lemma 5.4 that

$$ \sum_{(\omega_{1},\omega_{2})\in\mathcal{K}_{c_{0}}}\mathcal {I}_{t}( \omega_{1},\omega_{2})=tP_{t}(M)+t^{2} \bigl(P_{t}(M)-1\bigr)+t(1+t)Q(t), $$

for some $Q\in\mathbb{P}$. We easily deduce that, if the critical points of $I_{f,g}$ are non-degenerate, then they are at least $2P_{1}(M)-1$, if counted with their multiplicity. Thus we see that $(J_{f,g})$ has at least $2P_{1}(M)-1$ non-negative solutions, which, if non-degenerate, are possibly counted with their multiplicity. By the maximum principle [31], we complete the proof of Theorem 1.2.