Multiplicity of solutions for a class of fractional p-Kirchhoff system with sign-changing weight functions

Abstract

In this paper, we investigate the fractional p-Kirchhoff -type system:

$$\begin{aligned} \textstyle\begin{cases} M (\int_{{ \mathbb {R} }^{2N}}\frac{\vert u(x)-u(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy )(- \Delta )^{s}_{p}u=\mu g(x)\vert u \vert ^{\beta -2}u+\frac{a}{a+b}h(x)\vert u \vert ^{a-2}u\vert v \vert ^{b},&\mbox{in } \Omega , \\ M (\int_{{ \mathbb {R} }^{2N}}\frac{\vert v(x)-v(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy )(- \Delta )^{s}_{p}v=\sigma f(x)\vert v \vert ^{\beta -2}v+\frac{b}{a+b}h(x)\vert v \vert ^{b-2}v\vert u \vert ^{a},&\mbox{in } \Omega , \\ u=v=0,&\mbox{in } { \mathbb {R} }^{N}\setminus \Omega , \end{cases}\displaystyle \end{aligned}$$

where $\Omega \subset \mathbb{R}^{N}$ is a smooth bounded domain, $(-\Delta )^{s}_{p}$ is the fractional p-Laplacian operator with $0< s<1<p$ and $ps< N$. $a>1$, $b>1$ satisfy $2< a+b< p_{s}^{*}$. $1<\beta <p_{s}^{*}$, $p_{s}^{*}=\frac{Np}{N-ps}$ is the fractional critical exponent. μ, σ are two real parameters. $M(t)=k+\lambda t^{\tau }$, $k>0$, λ, $\tau \geq 0$, $\tau =0$ if and only if $\lambda =0$. The weight functions g, f, h change sign in Ω and satisfy suitable conditions. By using the Nehari manifold method, it is proved that the system has at least two solutions provided that $2< a+b< p\leq p(\tau +1)<\beta <p_{s}^{*}$ and $(\mu ,\sigma )$ belongs to a certain subset of $\mathbb {R} ^{2}$. Also, by using the mountain pass theorem, we prove that there exist $\lambda _{1}\geq \lambda_{0}$ such that the system admits at least a nontrivial solution for $\lambda \in (0,\lambda_{0})$ and no nontrivial solution for $\lambda >\lambda_{1}$ under the assumptions $\mu =\sigma =0$ and $p< a+b<\min \{p(\tau +1),p_{s}^{*}\}$.

1 Introduction

In this paper, we investigate the multiplicity of solutions to the following fractional elliptic system:

$$\begin{aligned} \textstyle\begin{cases} M (\int_{{ \mathbb {R} }^{2N}}\frac{\vert u(x)-u(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy )(-\Delta )^{s}_{p}u=\mu g(x)\vert u \vert ^{\beta -2}u+\frac{a}{a+b}h(x)\vert u \vert ^{a-2}u\vert v \vert ^{b } &\mbox{in } \Omega , \\ M (\int_{{ \mathbb {R} }^{2N}}\frac{\vert v(x)-v(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy )(-\Delta )^{s}_{p}v=\sigma f(x)\vert v \vert ^{\beta -2}v+\frac{b}{a+b}h(x)\vert v \vert ^{b-2}v\vert u \vert ^{a } &\mbox{in } \Omega , \\ u=v=0&\mbox{in } { \mathbb {R} }^{N}\setminus \Omega , \end{cases}\displaystyle \end{aligned}$$

(1.1)

where $\Omega \subset \mathbb{R}^{N}$ is a smooth bounded domain, $0< s<1<p$ and $ps< N$. $a>1$, $b>1$ satisfy $2< a+b< p^{\ast }_{s}$. $1<\beta <p_{s}^{*}$, $p_{s}^{*}=\frac{Np}{N-ps}$ is the fractional critical exponent. μ, σ are two real parameters. $M(t)=k+\lambda t^{\tau }$, $k>0$, λ, $\tau \geq 0$, $\tau =0$ if and only if $\lambda =0$. The weight functions g, f, h change sign in Ω and satisfy further assumption which will be given later. $(-\Delta )^{s}_{p}$ is the fractional p-Laplacian operator defined on smooth functions by

$$ (-\Delta )^{s}_{p}m(x)=2\lim_{\varepsilon \rightarrow 0^{+}} \int_{{ \mathbb {R} }^{N}\setminus B_{\varepsilon }(x)} \frac{\vert m(x)-m(y) \vert ^{p-2}(m(x)-m(y))}{\vert x-y \vert ^{N+ps}}\,dy,\quad x\in { \mathbb {R} } ^{N}. $$

Problem (1.1) is related to the stationary analogue of the following Kirchhoff model:

$$ \rho u_{tt}- \biggl(\frac{p_{0}}{h}+\frac{E}{2L} \int_{0}^{L}u_{x}^{2}\,dx \biggr)u _{xx}=0 $$

which was proposed by Kirchhoff in 1883 as a generalization of the well-known D’Alembert wave equation for free vibrations of elastic strings, where ρ, $p_{0}$, h, E, L are constants which represent some physical meanings, respectively. Indeed, Kirchhoff’s model takes into account the changes in length of the string produced by transverse vibrations. In particular, Kirchhoff’s equation models several physical and biological systems, we refer to [1] for more details. The Kirchhoff type equation and system have attracted attention and have been discussed by many authors, we refer to [9, 14, 21–23, 29] and the references therein.

Up to now, a great attention has been paid to the study of the fractional Laplacian equation and system, see, for example, [4, 5, 10, 11, 13, 18, 22, 23, 29, 30, 33]. In particular, the fractional and nonlocal operators of elliptic type arise in a quite natural way in many different applications, such as continuum mechanics, phase transition phenomena, population dynamics, and game theory, as they are the typical outcome of stochastic stabilization of Lévy processes, see [2, 8]. The literature on fractional nonlocal operators and their applications is very interesting and quite large, see, for example, [4, 20, 25, 28]. For the basic properties of fractional Sobolev spaces, we refer the readers to [12].

During the past ten years, by using the Nehari manifold and Fibering maps, several authors have solved semilinear and quasilinear elliptic problems with critical nonlinearity and subcritical nonlinearity, see [6, 7, 15–17, 19, 24, 26, 31, 32, 34] and the references therein. Particularly, in [10], Chen and Deng considered the following fractional p-Laplacian system:

$$\begin{aligned} \textstyle\begin{cases} (-\Delta )^{s}_{p}u=\mu \vert u \vert ^{\beta -2}u+\frac{2a}{a+b}\vert u \vert ^{a-2}u\vert v \vert ^{b } &\mbox{in } \Omega , \\ (-\Delta )^{s}_{p}v=\sigma \vert v \vert ^{\beta -2}v+\frac{2b}{a+b}\vert v \vert ^{b-2}v\vert u \vert ^{a } &\mbox{in } \Omega , \\ u=v=0&\mbox{in } { \mathbb {R} }^{n}\setminus \Omega , \end{cases}\displaystyle \end{aligned}$$

(1.2)

where $a>1$, $b>1$, $1<\beta <p<a+b<p_{s}^{*}$, $\mu >0$, $\sigma >0$. Using the Nehari manifold method, they proved (1.2) has at least two nontrivial solutions when $0<\mu^{\frac{p}{p-q}}+\sigma^{ \frac{p}{p-q}}<C$ for some $C>0$.

In [24], Rasouli and Afrouzi investigated the following elliptic system:

$$\begin{aligned} \textstyle\begin{cases} -\Delta_{q}u+n(x)\vert u \vert ^{q-2}u=\mu g(x)\vert u \vert ^{\beta -2}u&x\in \Omega , \\ -\Delta_{q}v+n(x)\vert v \vert ^{q-2}v=\sigma f(x)\vert v \vert ^{\beta -2}v&x\in \Omega, \\ \vert \nabla u \vert ^{q-2}\frac{\partial u}{\partial n}=\frac{a}{a+b}\vert u \vert ^{a-2}u\vert v \vert ^{b},\qquad \vert \nabla v \vert ^{q-2}\frac{\partial v}{\partial n}=\frac{b}{a+b}\vert u \vert ^{a}\vert v \vert ^{b-2}v,& x\in \partial \Omega , \end{cases}\displaystyle \end{aligned}$$

(1.3)

where $a>1$, $b>1$, $2< a+b< q<\beta <p^{*}$, $(\mu ,\sigma )\in { \mathbb {R} }^{2} \setminus \{(0,0)\}$, $g(x)$, $f(x)\in C(\Omega )$ are functions which change sign in Ω. Using the Nehari manifold method, they proved (1.3) admits at least two solutions when $(\mu , \sigma )$ belongs to a certain subset of ${ \mathbb {R} }^{2}$.

However, to our best knowledge, there are few results on a fractional p-Kirchhoff system with sign-changing concave-convex nonlinearity, especially for parameters meeting $a+b< p\leq p(\tau +1)<\beta <p_{s} ^{*}$ or $p< a+b<\min \{p(\tau +1),p_{s}^{*}\}$. Motivated by the above and the idea of [6, 9, 10, 24], in this paper, we are concerned with the multiplicity of solutions for system (1.1).

To state our main result precisely, we introduce some notations. Let $0< s<1<p$ with $ps< N$. Define

$$\begin{aligned} W= \biggl\{ v| v:{ \mathbb {R} }^{N}\rightarrow { \mathbb {R} }\mbox{ is measurable}, v | _{\Omega }\in L^{p}(\Omega ), \mbox{and} { \int_{\mathcal{Q}}\frac{\vert v(x)-v(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy}< +\infty \biggr\} , \end{aligned}$$

where $\mathcal{Q}={ \mathbb {R} }^{2N}\setminus (\Omega^{c}\times \Omega^{c})$ with $\Omega^{c}={ \mathbb {R} }^{N}\setminus \Omega$, this space is endowed with the norm

$$\begin{aligned} \Vert v \Vert _{W}=\Vert v \Vert _{L^{p}(\Omega )}+ \biggl( \int_{\mathcal{Q}} \frac{\vert v(x)-v(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy \biggr)^{1/p}. \end{aligned}$$

(1.4)

We denote the space $W_{0}=\{v\mid v\in W, v=0 \mbox{ a.e. in } \Omega^{c}\}$ or equivalently the closure of $C^{\infty }_{0}( \Omega )$ in W and introduce the norm

$$\begin{aligned} \Vert v \Vert =\Vert v \Vert _{W_{0}}= \biggl( \int_{\mathcal{Q}} \frac{\vert v(x)-v(y) \vert ^{p}}{\vert x-y \vert ^{N+ps}}\,dx\,dy \biggr)^{1/p}, \quad \forall v \in W_{0}. \end{aligned}$$

(1.5)

Then $(W_{0},\Vert \cdot \Vert _{W_{0}})$ is a uniformly convex Banach space, see [33, Theorem 2.4]. By results of [12, 15, 33], $W_{0}$ is continuously embedded in $L^{r}(\Omega )$ for any $1\leq r\leq p_{s}^{*}$ and compact for whenever $1\leq r< p_{s}^{*}$, then there exists $C_{r}>0$ such that, for $r\in [1,p_{s}^{*}]$,

$$\begin{aligned} \Vert v \Vert _{r}=\Vert v \Vert _{L^{r}(\Omega )}\leq C_{r}\Vert v \Vert _{W_{0}}=C_{r} \Vert v \Vert ,\quad \forall v\in W_{0}. \end{aligned}$$

(1.6)

For convenience, for some $\beta \in (1,p_{s}^{*})$, we denote $C_{\ast }=C_{\beta }$.

For the product space $X=W_{0}\times W_{0}$, we introduce the norm

$$\begin{aligned} \bigl\Vert (u,v) \bigr\Vert = \bigl\Vert (u,v) \bigr\Vert _{X}= \bigl(\Vert u \Vert ^{p}_{W_{0}}+\Vert v \Vert ^{p}_{W_{0}} \bigr)^{1/p}= \bigl( \Vert u \Vert ^{p}+\Vert v \Vert ^{p} \bigr)^{1/p}, \quad \forall (u,v)\in X. \end{aligned}$$

(1.7)

Then $(X, \Vert \cdot \Vert _{X})$ is a reflexive Banach space.

Our main results are as follows.

Theorem 1.1

Let $0< s<1<p$ with $ps< N$, $\Omega \subset { \mathbb {R} }^{N}$ is a smooth bounded domain, $(\mu ,\sigma )\in { \mathbb {R} }^{2}\backslash \{(0,0)\}$, $M(t)=k+\lambda t^{\tau }$, $k>0$, λ, $\tau \geq 0$, $\tau =0$ if and only if $\lambda =0$. Assume

$\mathrm{(H_{0})}$ :

$a+b< p\leq p(\tau +1)<\beta <p^{*}_{s}=\frac{Np}{N-ps}$;

$\mathrm{(H_{1})}$ :

$g,f\in C(\overline{\Omega })$ with $g^{\pm }=\max \{\pm g,0\}\not \equiv 0$ and $f^{\pm }=\max \{\pm f,0\} \not \equiv 0$;

$\mathrm{(H_{2})}$ :

$h\in L^{\frac{\beta }{\beta -a-b}}(\Omega )$ with $\operatorname{meas}(\{x\in \Omega : h(x)>0\})>0$,

then there exists $\theta^{\ast }>0$ such that when μ, σ satisfy

$$ 0< \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{ \infty } \bigr)^{\frac{p}{\beta -p}}< \theta^{*}, $$

problem (1.1) admits at least two solutions in X.

Theorem 1.2

Let $0< s<1<p$ with $ps< N$, $\Omega \subset { \mathbb {R} }^{N}$ is a smooth bounded domain, $\mu =\sigma =0$, $M(t)=k+\lambda t^{\tau }$, $k>0$, $\lambda ,\tau >0$. Assume

$\mathrm{(H_{0}^{\prime})}$ :

$p< a+b<\min\{p(\tau +1),p^{\ast }_{s}\}$.

In addition, suppose that one of the following holds:

$\mathrm{(H_{3})}$ :

$h\in L^{\frac{p_{s}^{\ast }}{p_{s}^{\ast }-a-b}}(\Omega )$ with $\operatorname{meas}(\{x\in \Omega : h(x)>0\})>0$;

$\mathrm{(H_{4})}$ :

$h\in L^{\infty }(\Omega )$ with $\operatorname{meas}(\{x\in \Omega : h(x)>0\})>0$,

then there exist $\lambda_{1}\geq \lambda_{0}>0$ such that problem (1.1) admits at least a nontrivial solution in X for $\lambda \in (0,\lambda_{0})$ and no nontrivial solution in X for $\lambda >\lambda_{1}$.

The organization of this paper as follows. In Sect. 2, we give some notations and properties of the Nehari manifold. In Sect. 3, we give the proof of Theorem 1.1. In Sect. 4, by applying the mountain pass theorem, we give the proof of Theorem 1.2.

2 Nehari manifold

Throughout this section we assume that all the conditions in Theorem 1.1 hold.

To simplify notations, for $(v,w)\in X$, we set

$$\begin{aligned} \mathcal{B}(v,w)=M \bigl(\Vert v \Vert ^{p} \bigr) \int_{\mathcal{Q}} \frac{\vert v(x)-v(y) \vert ^{p-2}(v(x)-v(y))(w(x)-w(y))}{\vert x-y \vert ^{N+ps}}\,dx\,dy. \end{aligned}$$

(2.1)

Definition 2.1

We say that the couple $(u,v)\in X$ is a weak solution to (1.1) if

$$\begin{aligned}& \mathcal{B}(u,w_{1})+\mathcal{B}(v,w_{2}) \\& \quad = \int_{\Omega } \bigl(\mu g(x)\vert u \vert ^{\beta -2}uw_{1}+ \sigma f(x)\vert v \vert ^{\beta -2}vw_{2} \bigr)\,dx \\& \quad \quad {}+\frac{a}{a+b} \int_{\Omega }h(x)\vert u \vert ^{a-2}u\vert v \vert ^{b}w_{1}\,dx + \frac{b}{a+b} \int_{\Omega }h(x)\vert v \vert ^{b-2}v\vert u \vert ^{a}w_{2}\,dx \end{aligned}$$

for any $(w_{1},w_{2})\in X$.

Clearly, the weak solutions to (1.1) are exactly the critical points of the following functional:

$$\begin{aligned} \mathcal{J}_{\mu ,\sigma }(u,v)=\frac{k}{p} \bigl\Vert (u,v) \bigr\Vert ^{p}+\frac{\lambda }{\alpha }G(u,v)-\frac{1}{\beta }L(u,v)- \frac{1}{q}H(u,v), \end{aligned}$$

(2.2)

where

$$\begin{aligned}& \begin{aligned} &\alpha =p(\tau +1), \\ & G(u,v)=\Vert u \Vert ^{\alpha }+\Vert v \Vert ^{\alpha }, \\ & q=a+b, \\ &L(u,v)= \int_{\Omega } \bigl(\mu g(x)\vert u \vert ^{\beta }+ \sigma f(x)\vert v \vert ^{ \beta } \bigr)\,dx, \\ & H(u,v)= \int_{\Omega }h(x)\vert u \vert ^{a}\vert v \vert ^{b}\,dx. \end{aligned} \end{aligned}$$

(2.3)

A direct computation shows that $\mathcal{J}_{\mu ,\sigma }\in C^{1}(X, { \mathbb {R} })$ and

$$\begin{aligned}& \bigl\langle \mathcal{J}'_{\mu ,\sigma }(u,v),(w_{1},w_{2}) \bigr\rangle \\& \quad = \mathcal{B}(u,w_{1})+\mathcal{B}(v,w_{2})- \int_{\Omega } \bigl( \mu g(x)\vert u \vert ^{\beta -2}uw_{1}+ \sigma f(x)\vert v \vert ^{\beta -2}vw_{2} \bigr)\,dx \\& \quad \quad {}-\frac{a}{q} \int_{\Omega }h(x)\vert u \vert ^{a-2}u\vert v \vert ^{b}w_{1}\,dx -\frac{b}{q} \int_{\Omega }h(x)\vert v \vert ^{b-2}v\vert u \vert ^{a}w_{2}\,dx \end{aligned}$$

(2.4)

for all $(u,v), (w_{1},w_{2})\in X$.

We consider the Nehari manifold

$$ \mathcal{S}_{\mu ,\sigma }= \bigl\{ (u,v)\in X\backslash \bigl\{ (0,0) \bigr\} \mid \bigl\langle \mathcal{J}'_{\mu ,\sigma }(u,v), (u,v) \bigr\rangle =0 \bigr\} . $$

Thus, $(u,v)\in \mathcal{S}_{\mu ,\sigma }$ if and only if

$$\begin{aligned} \bigl\langle \mathcal{J}'_{\mu ,\sigma }(u,v),(u,v) \bigr\rangle =k \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda G(u,v)-L(u,v)-H(u,v)=0. \end{aligned}$$

(2.5)

Denote

$$\begin{aligned} \Psi_{\mu ,\sigma }(u,v) &= \bigl\langle \mathcal{J}'_{\mu ,\sigma }(u, v),(u, v) \bigr\rangle \\ &=k \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda G(u,v)-L(u,v)-H(u,v). \end{aligned}$$

Then, for $(u,v)\in \mathcal{S}_{\mu ,\sigma }$,

$$\begin{aligned} \bigl\langle \Psi '_{\mu ,\sigma }(u,v),(u,v) \bigr\rangle =&kp \bigl\Vert (u,v) \bigr\Vert ^{p}+ \lambda \alpha G(u,v)-\beta L(u,v)-qH(u,v) \\ =&k(p-q) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda (\alpha -q) G(u,v)-(\beta -q) L(u,v) \end{aligned}$$

(2.6)

$$\begin{aligned} =&k(p-\beta ) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda (\alpha - \beta ) G(u,v)+(\beta -q) H(u,v). \end{aligned}$$

(2.7)

Obviously, $\mathcal{S}_{\mu ,\sigma }$ can be divided into the following three parts:

$$\begin{aligned} \mathcal{S}^{+}_{\mu ,\sigma }= \bigl\{ (u,v)\in \mathcal{S}_{\mu ,\sigma } \mid \bigl\langle \Psi '_{\mu ,\sigma }(u,v),(u,v) \bigr\rangle >0 \bigr\} , \\ \mathcal{S}^{0}_{\mu ,\sigma }= \bigl\{ (u,v)\in \mathcal{S}_{\mu ,\sigma } \mid \bigl\langle \Psi '_{\mu ,\sigma }(u,v),(u,v) \bigr\rangle =0 \bigr\} , \\ \mathcal{S}^{-}_{\mu ,\sigma }= \bigl\{ (u,v)\in \mathcal{S}_{\mu ,\sigma } \mid \bigl\langle \Psi '_{\mu ,\sigma }(u,v),(u,v) \bigr\rangle < 0 \bigr\} . \end{aligned}$$

Set

$$ \theta_{0}= \biggl[\frac{k(p-q)}{(\beta -q)C_{\ast }^{\beta }} \biggr]^{\frac{p}{ \beta -p}} \biggl[ \frac{k(\beta -p)}{(\beta -q)C_{\ast }^{q}\Vert h \Vert _{ {\frac{\beta }{\beta -q}}}} \biggr]^{\frac{p}{p-q}} $$

and

$$ \Lambda_{\theta_{0}}= \bigl\{ (\mu ,\sigma )\in \mathbb{R}^{2} \backslash \bigl\{ (0,0) \bigr\} : 0< \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}< \theta_{0} \bigr\} . $$

Lemma 2.2

For any $(\mu ,\sigma )\in \Lambda_{\theta_{0}}$, we have $\mathcal{S}^{0}_{\mu ,\sigma }=\emptyset$.

Proof

We argue by contradiction, then there exists $(\mu ,\sigma )\in \Lambda_{\theta_{0}}$ such that $\mathcal{S}^{0}_{\mu ,\sigma }\neq \emptyset$. For $(u,v)\in \mathcal{S}^{0}_{\mu ,\sigma }$, from (2.6) and (2.7), we can deduce that

$$\begin{aligned} k(p-q) \bigl\Vert (u,v) \bigr\Vert ^{p}\leq k(p-q) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda (\alpha -q) G(u,v)=( \beta -q) L(u,v) \end{aligned}$$

(2.8)

and

$$\begin{aligned} k(\beta -p) \bigl\Vert (u,v) \bigr\Vert ^{p} \leq k( \beta -p) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda ( \beta - \alpha ) G(u,v) =(\beta -q) H(u,v). \end{aligned}$$

(2.9)

By the Sobolev embedding theorem and Hölder’s inequality, we get

$$\begin{aligned} L(u,v) =& \int_{\Omega } \bigl(\mu g(x)\vert u \vert ^{\beta }+ \sigma f(x)\vert v \vert ^{\beta } \bigr)\,dx \\ \leq &\vert \mu \vert \Vert g \Vert _{\infty }\Vert u \Vert ^{\beta }_{\beta }+\vert \sigma \vert \Vert f \Vert _{\infty }\Vert v \Vert ^{\beta }_{\beta } \\ \leq &C_{\ast }^{\beta } \bigl(\vert \mu \vert \Vert g \Vert _{\infty }\Vert u \Vert ^{\beta }+\vert \sigma \vert \Vert f \Vert _{\infty }\Vert v \Vert ^{\beta } \bigr) \\ \leq &C_{\ast }^{\beta } \bigl[ \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{ \beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}} \bigr]^{\frac{ \beta -p}{p}} \bigl\Vert (u,v) \bigr\Vert ^{\beta }. \end{aligned}$$

(2.10)

From (2.8) and (2.10), it follows that

$$\begin{aligned} \bigl\Vert (u,v) \bigr\Vert \geq \biggl[\frac{k(p-q)}{(\beta -q)C_{\ast }^{\beta }} \biggr]^{\frac{1}{ \beta -p}} { \bigl[ \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}} \bigr]^{-\frac{1}{p}}}. \end{aligned}$$

(2.11)

By the Sobolev embedding theorem and Hölder’s inequality, we obtain

$$\begin{aligned} H(u,v)&= \int_{\Omega } h(x)\vert u \vert ^{a}\vert v \vert ^{b}\,dx \leq \Vert h \Vert _{\frac{\beta }{\beta -q}} \Vert u \Vert ^{a}_{\beta }\Vert v \Vert ^{b}_{\beta } \leq \Vert h \Vert _{{\frac{ \beta }{\beta -q}}}C_{\ast }^{q} \bigl\Vert (u,v) \bigr\Vert ^{q}. \end{aligned}$$

(2.12)

From (2.9) and (2.12), it follows that

$$\begin{aligned} \bigl\Vert (u,v) \bigr\Vert \leq \biggl[\frac{\beta -q}{k(\beta -p)} \Vert h \Vert _{\frac{\beta }{ \beta -q}}C_{\ast }^{q} \biggr]^{\frac{1}{p-q}}. \end{aligned}$$

(2.13)

Combining (2.11) with (2.13), it yields that

$$\begin{aligned} \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}} \geq \biggl[\frac{k(p-q)}{(\beta -q)C_{\ast } ^{\beta }} \biggr]^{\frac{p}{\beta -p}} \biggl[\frac{k(\beta -p)}{( \beta -q)C_{\ast }^{q}\Vert h \Vert _{\frac{\beta }{\beta -q}}} \biggr]^{ \frac{p}{p-q}} = \theta_{0}, \end{aligned}$$

which is a contradiction. □

Lemma 2.3

The functional $\mathcal{J}_{\mu ,\sigma }$ is coercive and bounded below on $\mathcal{S}_{\mu ,\sigma }$.

Proof

For every $(u,v)\in \mathcal{S}_{\mu ,\sigma }$, using (2.2), (2.5), and (2.12) yields

$$\begin{aligned} \mathcal{J}_{\mu ,\sigma }(u,v) =&k \biggl(\frac{1}{p}- \frac{1}{\beta } \biggr) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda \biggl(\frac{1}{\alpha } -\frac{1}{\beta } \biggr)G(u,v)- \biggl( \frac{1}{q}-\frac{1}{\beta } \biggr)H(u,v) \\ \geq &k \biggl(\frac{1}{p}-\frac{1}{\beta } \biggr) \bigl\Vert (u,v) \bigr\Vert ^{p}- \biggl(\frac{1}{q} -\frac{1}{ \beta } \biggr) \Vert h \Vert _{\frac{\beta }{\beta -q}}C_{\ast }^{q} \bigl\Vert (u,v) \bigr\Vert ^{q}. \end{aligned}$$

(2.14)

Hence, $\mathcal{J}_{\mu ,\sigma }$ is coercive and bounded below on $\mathcal{S}_{\mu ,\sigma }$. □

Lemma 2.4

Assume that $(u_{0},v_{0})$ is a local minimizer of $\mathcal{J}_{ \mu ,\sigma }$ on $\mathcal{S}_{\mu ,\sigma }$ and $(u_{0},v_{0}) \notin \mathcal{S}^{0}_{\mu ,\sigma }$, then $\mathcal{J}_{\mu , \sigma }'(u_{0},v_{0})=0$.

Proof

The proof is similar to that of Theorem 2.3 in [7]. □

Lemma 2.5

We have

• (1) if $(u,v)\in \mathcal{S}^{+}_{\mu ,\sigma }$, then $H(u,v)>0$;

• (2) if $(u,v)\in \mathcal{S}^{0}_{\mu ,\sigma }$, then $H(u,v)>0$ and $L(u,v)>0$;

• (3) if $(u,v)\in \mathcal{S}^{-}_{\mu ,\sigma }$, then $L(u,v)>0$.

Proof

By using (2.6) and (2.7), we arrive at the conclusion immediately. □

By Lemmas 2.2–2.3, for any $(\mu ,\sigma )\in\Lambda_{\theta_{0}}$, we obtain $\mathcal{S}_{\mu ,\sigma }= \mathcal{S}^{+}_{\mu ,\sigma }\cup \mathcal{S}^{-}_{\mu ,\sigma }$ and $\mathcal{J}_{\mu ,\sigma }$ is coercive and bounded below on $\mathcal{S}^{+}_{\mu ,\sigma }$ and $\mathcal{S}^{-}_{\mu ,\sigma }$.

Define

$$ \varepsilon_{\mu ,\sigma }=\inf_{(u,v)\in \mathcal{S}_{\mu ,\sigma }} \mathcal{J}_{\mu ,\sigma }(u,v), \qquad \varepsilon^{+}_{\mu ,\sigma }= \inf_{(u,v)\in \mathcal{S}^{+}_{\mu ,\sigma }} \mathcal{J}_{\mu , \sigma }(u,v), \qquad \varepsilon^{-}_{\mu ,\sigma }= \inf_{(u,v)\in \mathcal{S}^{-}_{\mu ,\sigma }}\mathcal{J}_{\mu , \sigma }(u,v), $$

and set

$$ \Lambda_{\theta^{\ast }}= \bigl\{ (\mu ,\sigma )\in \mathbb{R}^{2} \backslash \bigl\{ (0,0) \bigr\} : 0< \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}< \theta^{\ast } \bigr\} , $$

where $\theta^{\ast }=(\frac{q}{p})^{\frac{p}{p-q}}\theta_{0}<\theta _{0}$. Obviously, $\Lambda_{\theta^{\ast }}\subset \Lambda_{\theta _{0}}$. Then the following result is established.

Lemma 2.6

If $(\mu ,\sigma )\in \Lambda_{\theta^{\ast }}$, then

• (i) $\varepsilon_{\mu ,\sigma }\leq \varepsilon^{+}_{\mu , \sigma }<0$;

• (ii) There exists $\eta_{0}=\eta_{0}(a,b,\beta ,p,\mu ,\sigma )>0$ such that $\varepsilon^{-}_{\mu ,\sigma }\geq \eta_{0}$.

Proof

(i) For $(u,v)\in \mathcal{S}^{+}_{\mu ,\sigma }$, using (2.6), we get

$$ \frac{k(p-q)}{\beta -q} \bigl\Vert (u,v) \bigr\Vert ^{p}+ \frac{\lambda (\alpha -q)}{\beta -q}G(u,v)> L(u,v). $$

This combined with (2.5) yields

$$\begin{aligned} \mathcal{J}_{\mu ,\sigma }(u,v) =&k \biggl(\frac{1}{p}- \frac{1}{q} \biggr) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda \biggl( \frac{1}{\alpha } -\frac{1}{q} \biggr)G(u,v)+ \biggl( \frac{1}{q}- \frac{1}{ \beta } \biggr)L(u,v) \\ < & \frac{k(q-p)(\beta -p)}{pq\beta } \bigl\Vert (u,v) \bigr\Vert ^{p} + \frac{\lambda (q- \alpha )(\beta -\alpha )}{\alpha q\beta }G(u,v) \\ < &0. \end{aligned}$$

Thus, $\varepsilon_{\mu ,\sigma }\leq \varepsilon^{+}_{\mu ,\sigma }<0$.

(ii) For $(u,v)\in \mathcal{S}^{-}_{\mu ,\sigma }$, by (2.6) and (2.10), we obtain

$$\begin{aligned} \bigl\Vert (u,v) \bigr\Vert ^{p} \leq & \bigl\Vert (u,v) \bigr\Vert ^{p}+\frac{\lambda (\alpha -q)}{k(p-q)}G(u,v) < \frac{(\beta -q)}{k(p-q)}L(u,v) \\ < &\frac{(\beta -q)}{k(p-q)}C_{\ast }^{\beta } \bigl[ \bigl(\vert \mu \vert \Vert g \Vert _{ \infty } \bigr)^{\frac{p}{\beta -p}} + \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{ \beta -p}} \bigr]^{\frac{\beta -p}{p}} \bigl\Vert (u,v) \bigr\Vert ^{\beta }. \end{aligned}$$

That is,

$$\begin{aligned} \bigl\Vert (u,v) \bigr\Vert > \biggl(\frac{k(p-q)}{(\beta -q)C_{\ast }^{\beta }} \biggr)^{\frac{1}{ \beta -p}} \bigl[ \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert f \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}} \bigr]^{-\frac{1}{p}}. \end{aligned}$$

(2.15)

By using (2.15) and (2.14) of Lemma 2.3, it follows that

$$\begin{aligned}& \mathcal{J}_{\mu ,\sigma }(u,v) \\& \quad \geq k \biggl(\frac{1}{p}-\frac{1}{\beta } \biggr) \bigl\Vert (u,v) \bigr\Vert ^{p}- \biggl(\frac{1}{q}-\frac{1}{ \beta } \biggr) \Vert h \Vert _{\frac{\beta }{\beta -q}}C_{\ast }^{q} \bigl\Vert (u,v) \bigr\Vert ^{q} \\& \quad = \bigl\Vert (u,v) \bigr\Vert ^{q} \biggl[k \biggl( \frac{1}{p}-\frac{1}{\beta } \biggr) \bigl\Vert (u,v) \bigr\Vert ^{p-q} - \biggl( \frac{1}{q}-\frac{1}{\beta } \biggr)\Vert h \Vert _{\frac{\beta }{\beta -q}}C_{\ast } ^{q} \biggr] \\& \quad > \biggl(\frac{k(p-q)}{(\beta -q)C_{\ast }^{\beta }} \biggr)^{\frac{q}{ \beta -p}} { \bigl[ \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert h \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}} \bigr]^{-\frac{q}{p}}} \biggl[\frac{k(\beta -p)}{p\beta } \\& \quad \quad {} \times \biggl(\frac{k(p-q)}{(\beta -q)C_{\ast }^{\beta }} \biggr)^{\frac{p-q}{ \beta -p}} { \bigl( \bigl(\vert \mu \vert \Vert g \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}}+ \bigl(\vert \sigma \vert \Vert h \Vert _{\infty } \bigr)^{\frac{p}{\beta -p}} \bigr)^{-\frac{p-q}{p}}} - \biggl(\frac{1}{q}- \frac{1}{\beta } \biggr)\Vert h \Vert _{\frac{\beta }{\beta -q}}C_{ \ast }^{q} \biggr] \\& \quad \geq \eta_{0}>0, \end{aligned}$$

due to $(\mu ,\sigma )\in \Lambda_{\theta^{\ast }}$. □

Fix $(u,v)\in X$ with $L(u,v)>0$, define

$$\begin{aligned} \varphi (t)=kt^{p-q} \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda t^{\alpha -q}G(u,v)-t^{ \beta -q}L(u,v), \quad t\geq 0. \end{aligned}$$

(2.16)

Obviously, $\varphi (0)=0$, $\lim_{t\rightarrow +\infty }\varphi (t)=-\infty$, $\varphi '(t)=t^{p-q-1}g(t)$, where

$$\begin{aligned} g(t)=k(p-q) \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda ( \alpha-q)t^{\alpha -p}G(u,v) -( \beta -q)t^{\beta -p}L(u,v). \end{aligned}$$

When $\lambda >0$, denote

$$ t^{*}= \biggl[\frac{\lambda (\alpha -p)(\alpha -q)G(u,v)}{(\beta -q)( \beta -p)L(u,v)} \biggr]^{\frac{1}{\beta -\alpha }}>0. $$

It is easy to see that $g(t)$ is increasing on $[0,t^{*})$ and decreasing on $(t^{*},+\infty )$. Note that $g(0)=k(p-q)\Vert (u,v) \Vert ^{p}>0$ and $\lim_{t\rightarrow +\infty }g(t)=-\infty$, so there exists a unique $t_{\lambda }>t^{*}$ such that $g(t_{\lambda })=0$. Moreover, $\varphi (t)$ reaches the maximum at $t_{\lambda }$, is increasing on $[0,t_{\lambda })$ and decreasing on $(t_{\lambda },+\infty )$.

When $\lambda =0$, we have

$$\begin{aligned} t_{\lambda }=t_{0}= \biggl[\frac{k(p-q)\Vert (u,v) \Vert ^{p}}{(\beta -q)L(u,v)} \biggr]^{\frac{1}{\beta -p}}>0. \end{aligned}$$

(2.17)

It is easy to show that $t_{\lambda }\geq t_{0}$ for $\lambda \geq 0$, thus

$$\begin{aligned} \varphi (t_{\lambda })\geq \varphi (t_{0})\geq t^{p-q}_{0}\frac{k( \beta -p)}{\beta -q} \bigl\Vert (u,v) \bigr\Vert ^{p}. \end{aligned}$$

(2.18)

Then the following lemma holds.

Lemma 2.7

Assume $(u,v)\in X$ with $L(u,v)>0$ and $(\mu ,\sigma )\in \Lambda_{\theta^{\ast }}$, we obtain

(i) if $H(u,v)\leq 0$, then there exists a unique $0< t_{\lambda }< t^{-}$ such that $(t^{-}u,t^{-}v)\in \mathcal{S}^{-}_{\mu ,\sigma }$ and

$$ \mathcal{J}_{\mu ,\sigma } \bigl(t^{-}u,t^{-}v \bigr)=\sup _{t\geq 0} \mathcal{J} _{\mu ,\sigma }(tu,tv); $$

(ii) if $H(u,v)>0$, then there exist unique $0< t^{+}< t_{\lambda }< t^{-}$ such that $(t^{+}u,t^{+}v)\in \mathcal{S}^{+}_{\mu ,\sigma }$, $(t^{-}u,t^{-}v)\in \mathcal{S}^{-}_{\mu ,\sigma }$ and

$$ \mathcal{J}_{\mu ,\sigma } \bigl(t^{+}u,t^{+}v \bigr)=\inf _{0\leq t\leq t_{\lambda }}\mathcal{J}_{\mu ,\sigma }(tu,tv),\qquad \mathcal{J}_{\mu ,\sigma } \bigl(t ^{-}u,t^{-}v \bigr)=\sup _{t\geq 0} \mathcal{J}_{\mu ,\sigma }(tu,tv). $$

Proof

(i) If $H(u,v)\leq 0$, by (2.16), there exists a unique $0< t_{\lambda }< t^{-}$ such that $\varphi (t^{-})=H(u,v)$ and $\varphi '(t^{-})<0$. Note that

$$\begin{aligned} \Psi_{\mu ,\sigma } \bigl(t^{-}u,t^{-}v \bigr) =&k \bigl(t^{-} \bigr)^{p} \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda \bigl(t^{-} \bigr)^{\alpha }G(u,v)- \bigl(t^{-} \bigr)^{\beta }L(u,v)- \bigl(t^{-} \bigr)^{q}H(u,v) \\ =& \bigl(t^{-} \bigr)^{q} \bigl[\varphi \bigl(t^{-} \bigr)-H(u,v) \bigr]=0 \end{aligned}$$

and

$$\begin{aligned} k(p-q) \bigl(t^{-} \bigr)^{p} \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda (\alpha -q) \bigl(t^{-} \bigr)^{\alpha }G(u,v)-( \beta -q) \bigl(t^{-} \bigr)^{\beta }L(u,v) = \bigl(t^{-} \bigr)^{q+1}\varphi ' \bigl(t^{-} \bigr)< 0. \end{aligned}$$

Thus, $(t^{-}u,t^{-}v)\in \mathcal{S}^{-}_{\mu ,\sigma }$. It is easy to derive

$$ \frac{d}{\,dt}\mathcal{J}_{\mu ,\sigma }(tu,tv)=t^{q-1} \bigl[ \varphi (t)-H(u,v) \bigr]. $$

Hence, $\mathcal{J}_{\mu ,\sigma }(tu,tv)$ increases for $t\in [0,t ^{-})$ and decreases for $t\in (t^{-},+\infty )$. This implies $\mathcal{J}_{\mu ,\sigma }(t^{-}u,t^{-}v)=\sup_{t\geq 0} \mathcal{J}_{\mu ,\sigma }(tu,tv)$.

(ii) If $H(u,v)>0$, it follows from (2.10), (2.12), (2.17), and (2.18) that

$$ \varphi (0)=0< H(u,v)\leq \Vert h \Vert _{\frac{\beta }{\beta -q}}C_{\ast }^{q} \bigl\Vert (u,v) \bigr\Vert ^{q}< \varphi (t_{0})\leq \varphi (t_{\lambda }) $$

for $(\mu ,\sigma )\in \Lambda_{\theta_{0}}$. Hence, there exist unique $t^{+}$, $t^{-}>0$ such that $t^{+}< t_{\lambda }< t^{-}$, $\varphi (t ^{+})=H(u,v)=\varphi (t^{-})$, $\varphi '(t^{-})<0<\varphi '(t^{+})$. Similar to the argument in (i), we get $(t^{+}u,t^{+}v)\in \mathcal{S}^{+}_{\mu ,\sigma }$, $(t^{-}u,t^{-}v)\in \mathcal{S}^{-} _{\mu ,\sigma }$, and

$$\begin{aligned}& \mathcal{J}_{\mu ,\sigma }(tu,tv)\geq \mathcal{J}_{\mu ,\sigma } \bigl(t^{+}u,t^{+}v \bigr),\quad \forall t\in \bigl[0,t^{-} \bigr], \\& \mathcal{J}_{\mu ,\sigma } \bigl(t^{-}u,t^{-}v \bigr)\geq \mathcal{J}_{\mu ,\sigma }(tu,tv),\quad \forall t \in [t^{+},+\infty ). \end{aligned}$$

Note that $\mathcal{J}_{\mu ,\sigma }(t^{-}u,t^{-}v)\geq \varepsilon ^{-}_{\mu ,\sigma }>0$. Thus

$$ \mathcal{J}_{\mu ,\sigma } \bigl(t^{+}u,t^{+}v \bigr)=\inf _{0\leq t\leq t_{\lambda }}\mathcal{J}_{\mu ,\sigma }(tu,tv),\qquad \mathcal{J}_{\mu ,\sigma } \bigl(t ^{-}u,t^{-}v \bigr)=\sup _{t\geq 0}\mathcal{J}_{\mu ,\sigma }(tu,tv). $$

So we arrive at the conclusion. □

Fix $(u,v)\in X$ with $H(u,v)>0$, define

$$\begin{aligned} \bar{\varphi }(t)=kt^{p-\beta } \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda t^{\alpha - \beta }G(u,v)-t^{q-\beta }H(u,v), \quad t>0. \end{aligned}$$

(2.19)

Obviously, $\lim_{t\rightarrow 0^{+}}\bar{\varphi }(t)=-\infty$ and $\lim_{t\rightarrow +\infty }\bar{\varphi }(t)=0$, $\bar{\varphi }'(t)=t ^{q-\beta -1}\bar{g}(t)$, where

$$\begin{aligned} \bar{g}(t)=k(p-\beta )t^{p-q} \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda (\alpha -\beta )t ^{\alpha -q}G(u,v) +(\beta -q)H(u,v). \end{aligned}$$

Clearly, $\lim_{t\rightarrow 0^{+}}\bar{g}(t)=(\beta -q)H(u,v)>0$, $\lim_{t\rightarrow +\infty }\bar{g}(t)=-\infty$ and $\bar{g}(t)$ decreases on $(0,+\infty )$. Then there exists a unique $\bar{t}_{ \lambda }>0$ such that $\bar{g}(\bar{t}_{\lambda })=0$. Moreover, $\bar{\varphi }(t)$ reaches the maximum at $\bar{t}_{\lambda }$, is increasing on $(0,\bar{t}_{\lambda })$ and decreasing on $(\bar{t} _{\lambda },+\infty )$. In particular, when $\lambda =0$, we have

$$\begin{aligned} \bar{t}_{\lambda }=\bar{t}_{0}= \biggl[ \frac{(\beta -q)H(u,v)}{k(\beta -p)\Vert (u,v) \Vert ^{p}} \biggr]^{\frac{1}{p-q}}>0. \end{aligned}$$

(2.20)

It is easy to prove that $\bar{t}_{0}\geq \bar{t}_{\lambda }$ for $\lambda \geq 0$, thus

$$\begin{aligned} \bar{\varphi }(\bar{t}_{\lambda })\geq \bar{\varphi }( \bar{t}_{0}) \geq (\bar{t}_{0})^{q-\beta } \frac{p-q}{\beta -p}H(u,v). \end{aligned}$$

(2.21)

Then the following lemma holds.

Lemma 2.8

Assume $(u,v)\in X$ with $H(u,v)>0$ and $(\mu ,\sigma )\in \Lambda_{\theta^{\ast }}$, we obtain

(i) if $L(u,v)\leq 0$, then there exists a unique $0<\bar{t} ^{+}<\bar{t}_{\lambda }$ such that $(\bar{t}^{+}u,\bar{t}^{+}v)\in \mathcal{S}^{+}_{\mu ,\sigma }$ and

$$ \mathcal{J}_{\mu ,\sigma } \bigl(\bar{t}^{+}u,\bar{t}^{+}v \bigr)=\inf_{t\geq 0} \mathcal{J}_{\mu ,\sigma }(tu,tv); $$

(ii) if $L(u,v)>0$, then there exist unique $0<\bar{t}^{+}< \bar{t}_{\lambda }<\bar{t}^{-}$ such that $(\bar{t}^{+}u,\bar{t}^{+}v) \in \mathcal{S}^{+}_{\mu ,\sigma }$, $(\bar{t}^{-}u,\bar{t}^{-}v) \in \mathcal{S}^{-}_{\mu ,\sigma }$ and

$$ \mathcal{J}_{\mu ,\sigma } \bigl(\bar{t}^{+}u,\bar{t}^{+}v \bigr)= \inf_{0\leq t\leq \bar{t}_{\lambda }}\mathcal{J}_{\mu ,\sigma }(tu,tv),\qquad \mathcal{J}_{\mu ,\sigma } \bigl(\bar{t}^{-}u,\bar{t}^{-}v \bigr)=\sup_{t \geq 0} \mathcal{J}_{\mu ,\sigma }(tu,tv). $$

Proof

Using (2.19), (2.20), and (2.21), similar to the proof of Lemma 2.7, we can get the conclusion of Lemma 2.8. □

3 Proof of Theorem 1.1

Throughout this section, we still assume that all the conditions in Theorem 1.1 hold.

To prove Theorem 1.1, we first prove the following two propositions.

Proposition 3.1

Assume $(\mu ,\sigma )\in \Lambda_{\theta^{*}}$, then $\mathcal{J} _{\mu ,\sigma }$ has a minimizer $(u^{+}_{0},v^{+}_{0})$ in $\mathcal{S}^{+}_{\mu ,\sigma }$ and satisfies

(1) $\mathcal{J}_{\mu ,\sigma }(u^{+}_{0},v^{+}_{0})=\varepsilon ^{+}_{\mu ,\sigma }$;

(2) $(u^{+}_{0},v^{+}_{0})$ is a solution of problem (1.1) such that $u^{+}_{0}\not \equiv 0, v^{+}_{0}\not \equiv 0$.

Proof

By Lemma 2.3, we have $\mathcal{J}_{\mu ,\sigma }$ is coercive and bounded below on $\mathcal{S}^{+}_{\mu ,\sigma }$. Hence, there exists a minimizing sequence $\{(u_{n},v_{n})\}\subset \mathcal{S}^{+}_{ \mu ,\sigma }$, bounded in X. Since X is reflexive, there is a subsequence, still denoted by $\{(u_{n},v_{n})\}$ and $(u^{+}_{0},v ^{+}_{0})\in X$ such that, as $n\rightarrow \infty$,

$$ u_{n}\rightharpoonup u^{+}_{0} \quad \mbox{ in } W_{0}; \qquad u_{n}\rightarrow u ^{+}_{0} \quad \mbox{ in } L^{r}(\Omega ); $$

$$ v_{n}\rightharpoonup v^{+}_{0} \quad \mbox{ in } W_{0}; \qquad v_{n}\rightarrow v ^{+}_{0} \quad \mbox{ in } L^{r}(\Omega ) $$

for all $r\in [1,p_{s}^{*})$. By [3, Theorem 1.2.7], there exists $\rho (x)\in L^{r}(\Omega )$ such that

$$\begin{aligned}& \mbox{for all } n, \quad \bigl\vert u_{n}(x) \bigr\vert \leq \rho(x), \qquad \bigl\vert v_{n}(x) \bigr\vert \leq \rho (x)\quad \mbox{a.e. in } \Omega ; \\& u_{n}\rightarrow u^{+}_{0}, v_{n} \rightarrow v^{+}_{0 } \quad \mbox{a.e. in } \Omega \mbox{ as }n\rightarrow \infty \end{aligned}$$

for all $r\in [1,p_{s}^{*})$. By the dominated convergence theorem, we obtain

$$ \lim_{n\rightarrow \infty }H(u_{n},v_{n})=H \bigl(u^{+}_{0},v^{+}_{0} \bigr), \qquad \lim_{n\rightarrow \infty }L(u_{n},v_{n})= L \bigl(u^{+}_{0},v ^{+}_{0} \bigr). $$

In view of $(u_{n},v_{n})\in \mathcal{S}^{+}_{\mu ,\sigma }$, then

$$\begin{aligned} \mathcal{J}_{\mu ,\sigma }(u_{n},v_{n}) =&k \biggl( \frac{1}{p}-\frac{1}{ \beta } \biggr) \bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{p}+\lambda \biggl(\frac{1}{\alpha } -\frac{1}{ \beta } \biggr)G(u_{n},v_{n})+ \biggl(\frac{1}{\beta }- \frac{1}{q} \biggr)H(u_{n},v_{n}) \\ >& \biggl(\frac{1}{\beta }-\frac{1}{q} \biggr)H(u_{n},v_{n}), \end{aligned}$$

and

$$ \lim_{n\rightarrow \infty }\mathcal{J}_{\mu ,\sigma }(u_{n},v_{n})= \varepsilon^{+}_{\mu ,\sigma }< 0. $$

It follows $H(u^{+}_{0},v^{+}_{0})>0$, in particular, $u^{+}_{0} \not \equiv 0$, $v^{+}_{0}\not \equiv 0$. Next, we show that $u_{n}\rightarrow u^{+}_{0}$ in $W_{0}$, $v_{n}\rightarrow v^{+}_{0}$ in $W_{0}$. If not, then either

$$\begin{aligned} \bigl\Vert u^{+}_{0} \bigr\Vert < \liminf_{n\rightarrow \infty }\Vert u_{n} \Vert \quad \mbox{or} \quad \bigl\Vert v^{+}_{0} \bigr\Vert < \liminf _{n\rightarrow \infty }\Vert v_{n} \Vert . \end{aligned}$$

(3.1)

Fix $(u,v)\in X$ with $H(u,v)>0$, denote

$$ \psi_{(u,v)}(t)=\bar{\varphi }(t)-L(u,v),\quad t>0, $$

where $\bar{\varphi }(t)$ is given by (2.19). Obviously, $\lim_{t\rightarrow 0^{+}}\psi_{(u,v)}(t)=-\infty$, $\lim_{t\rightarrow +\infty }\psi_{(u,v)}(t)=-L(u,v)$, $\psi '_{(u,v)}(t)=\bar{ \varphi }'(t)$, then we get that $\varphi_{(u,v)}(t)$ reaches the maximum at $\bar{t}_{\lambda }$, is increasing on $(0,\bar{t}_{\lambda })$ and decreasing on $(\bar{t}_{\lambda },+\infty )$. Note that $H(u^{+}_{0},v^{+}_{0})>0$, by Lemma 2.8, there exists a unique $0< t^{+}_{0}<\bar{t}_{\lambda }(u^{+}_{0},v^{+}_{0})$ such that $(t^{+}_{0}u^{+}_{0},t^{+}_{0}v^{+}_{0})\in \mathcal{S}^{+}_{\mu , \sigma }$ and

$$\begin{aligned} \mathcal{J}_{\mu ,\sigma } \bigl(t^{+}_{0}u^{+}_{0},t^{+}_{0}v^{+}_{0} \bigr)= \inf_{0\leq t\leq \bar{t}_{\lambda }(u^{+}_{0},v^{+}_{0})} \mathcal{J}_{\mu ,\sigma } \bigl(tu^{+}_{0},tv^{+}_{0} \bigr). \end{aligned}$$

(3.2)

Note that $(t^{+}_{0}u^{+}_{0},t^{+}_{0}v^{+}_{0})\in \mathcal{S}^{+} _{\mu ,\sigma }\subset \mathcal{S}_{\mu ,\sigma }$ and the definition of $\bar{\varphi }(t)$, it is easy to derive

$$\begin{aligned} \psi_{(u^{+}_{0},v^{+}_{0})} \bigl(t^{+}_{0} \bigr) =&\bar{\varphi } \bigl(t^{+}_{0} \bigr)-L \bigl(u ^{+}_{0},v^{+}_{0} \bigr)=0. \end{aligned}$$

(3.3)

It follows from (3.1) and (3.3) that

$$ \psi_{(u_{n},v_{n})} \bigl(t^{+}_{0} \bigr)>0 \quad \mbox{ for large enough } n. $$

Since $(u_{n},v_{n})\in \mathcal{S}^{+}_{\mu ,\sigma }$, we get $\bar{t}_{\lambda }(u_{n},v_{n})>1$. Moreover, we can deduce that $\psi_{(u_{n},v_{n})}(1)=\bar{\varphi }(1)-L(u_{n},v_{n})=0$ and $\psi_{(u_{n},v_{n})}(t)$ increases on $(0,\bar{t}_{\lambda }(u_{n},v _{n}))$. This implies, for all n, $\psi_{(u_{n},v_{n})}(t)\leq 0, \forall t\in (0,1]$. Thus

$$\begin{aligned} 1< t^{+}_{0}< \bar{t}_{\lambda } \bigl(u^{+}_{0},v^{+}_{0} \bigr). \end{aligned}$$

(3.4)

It follows from (3.1), (3.2), and (3.4) that

$$ \mathcal{J}_{\mu ,\sigma } \bigl(t^{+}_{0}u^{+}_{0},t^{+}_{0}v^{+}_{0} \bigr) \leq \mathcal{J}_{\mu ,\sigma } \bigl(u^{+}_{0},v^{+}_{0} \bigr)< \lim_{n\rightarrow \infty }\mathcal{J}_{\mu ,\sigma }(u_{n},v_{n}) = \varepsilon^{+}_{\mu ,\sigma }, $$

which is a contradiction. Thus

$$ u_{n}\rightarrow u^{+}_{0}\quad \mbox{in } W_{0},\qquad v_{n}\rightarrow v ^{+}_{0} \quad \mbox{in } W_{0}. $$

This implies

$$ \mathcal{J}_{\mu ,\sigma }(u_{n},v_{n})\rightarrow\mathcal{J}_{\mu ,\sigma } \bigl(u^{+}_{0},v^{+}_{0}\bigr)=\varepsilon^{+}_{\mu ,\sigma }\quad \mbox{as } n\rightarrow \infty . $$

Namely, $(u^{+}_{0},v^{+}_{0})$ is a minimizer of $\mathcal{J}_{ \mu ,\sigma }$ on $\mathcal{S}^{+}_{\mu ,\sigma }$. By Lemma 2.4, $(u^{+}_{0},v^{+}_{0})$ is a solution of problem (1.1) such that $u^{+}_{0}\not \equiv 0, v^{+}_{0}\not \equiv 0$. □

Proposition 3.2

Assume $(\mu ,\sigma )\in \Lambda_{\theta^{*}}$, then $\mathcal{J} _{\mu ,\sigma }$ has a minimizer $(u^{-}_{0},v^{-}_{0})$ in $\mathcal{S}^{-}_{\mu ,\sigma }$ and satisfies

(1) $\mathcal{J}_{\mu ,\sigma }(u^{-}_{0},v^{-}_{0})=\varepsilon ^{-}_{\mu ,\sigma }$;

(2) $(u^{-}_{0},v^{-}_{0})$ is a solution of problem (1.1) such that $(u^{-}_{0},v^{-}_{0})\not \equiv (0,0)$.

Proof

Since $\mathcal{J}_{\mu ,\sigma }$ is coercive and bounded below on $\mathcal{S}^{-}_{\mu ,\sigma }$, there exists a minimizing sequence $\{(u_{n},v_{n})\}\subset \mathcal{S}^{-}_{\mu ,\sigma }$, bounded in X. Note that X is reflexive, then there is a subsequence, still denoted by $\{(u_{n},v_{n})\}$ and $(u^{-}_{0},v^{-}_{0})\in X$ such that, as $n\rightarrow \infty$,

$$\begin{aligned}& u_{n}\rightharpoonup u^{-}_{0} \quad \mbox{in }W_{0};\qquad u_{n}\rightarrow u ^{-}_{0}\quad \mbox{in } L^{r}(\Omega ); \\& v_{n}\rightharpoonup v^{-}_{0} \quad \mbox{in }W_{0}; \qquad v_{n}\rightarrow v ^{-}_{0}\quad \mbox{in } L^{r}(\Omega ) \end{aligned}$$

for all $r\in [1,p_{s}^{*})$. By [3, Theorem 1.2.7], there exists $\varrho (x)\in L^{r}(\Omega )$ such that

$$\begin{aligned}& \mbox{for all } n, \quad \bigl\vert u_{n}(x) \bigr\vert \leq \varrho (x),\qquad \bigl\vert v_{n}(x) \bigr\vert \leq \varrho (x)\quad \mbox{a.e. in } \Omega ; \\& u_{n}\rightarrow u^{-}_{0}, \qquad v_{n}\rightarrow v^{-}_{0 } \quad\mbox{a.e. in } \Omega \mbox{ as }n\rightarrow \infty \end{aligned}$$

for all $r\in [1,p_{s}^{*})$. By the dominated convergence theorem, we obtain

$$ \lim_{n\rightarrow \infty }H(u_{n},v_{n})=H \bigl(u^{-}_{0},v^{-}_{0} \bigr), \qquad \lim_{n\rightarrow \infty }L(u_{n},v_{n})= L \bigl(u^{-}_{0},v ^{-}_{0} \bigr). $$

Moreover, by (2.6), we have

$$\begin{aligned} L(u_{n},v_{n})>\frac{k(p-q)}{\beta -q} \bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{p}. \end{aligned}$$

(3.5)

Using (2.10) and (3.5), there exists a positive constant c such that

$$ L({u_{n},v_{n}})>c>0. $$

This implies

$$ L \bigl(u^{-}_{0},v^{-}_{0} \bigr)>0. $$

In particular, $(u^{-}_{0},v^{-}_{0})\not \equiv (0,0)$. Next, we show that $u_{n}\rightarrow u^{-}_{0}$ in $W_{0}$, $v_{n}\rightarrow v^{-} _{0}$ in $W_{0}$. If not, then either

$$\begin{aligned} \bigl\Vert u^{-}_{0} \bigr\Vert < \liminf_{n\rightarrow \infty }\Vert u_{n} \Vert \quad \mbox{or} \quad \bigl\Vert v^{-}_{0} \bigr\Vert < \liminf _{n\rightarrow \infty }\Vert v_{n} \Vert . \end{aligned}$$

(3.6)

By Lemma 2.7, there exists a unique $t^{-}_{0}>t_{\lambda }(u ^{-}_{0},v^{-}_{0})>0$ such that $(t^{-}_{0}u^{-}_{0},t^{-}_{0}v^{-} _{0})\in \mathcal{S}^{-}_{\mu ,\sigma }$. In view of $(u_{n},v_{n}) \in \mathcal{S}^{-}_{\mu ,\sigma }$, it follows $\mathcal{J}_{\mu , \sigma }(tu_{n},tv_{n})\leq \mathcal{J}_{\mu ,\sigma }(u_{n},v_{n})$ for $t\geq 0$. By (3.6), we have

$$ \mathcal{J}_{\mu ,\sigma } \bigl(t^{-}_{0}u^{-}_{0},t^{-}_{0}v^{-}_{0} \bigr)< \liminf_{n\rightarrow \infty }\mathcal{J}_{\mu ,\sigma } \bigl(t^{-}_{0}u _{n},t^{-}_{0}v_{n} \bigr)\leq \lim_{n\rightarrow \infty }\mathcal{J}_{ \mu ,\sigma }(u_{n},v_{n}) =\varepsilon^{-}_{\mu ,\sigma }, $$

which is a contradiction. Thus

$$ u_{n}\rightarrow u^{-}_{0}\quad \mbox{in }W_{0},\qquad v_{n}\rightarrow v ^{-}_{0}\quad \mbox{in } W_{0}. $$

This implies

$$ \mathcal{J}_{\mu ,\sigma }(u_{n},v_{n})\rightarrow \mathcal{J}_{ \mu ,\sigma } \bigl(u^{-}_{0},v^{-}_{0} \bigr)=\varepsilon^{-}_{\mu ,\sigma }\quad \mbox{as } n\rightarrow \infty . $$

Namely, $(u^{-}_{0},v^{-}_{0})$ is a minimizer of $\mathcal{J}_{ \mu ,\sigma }$ on $\mathcal{S}^{-}_{\mu ,\sigma }$. By Lemma 2.4, $(u^{-}_{0},v^{-}_{0})$ is a solution of problem (1.1) such that $(u^{-}_{0},v^{-}_{0})\not \equiv (0,0)$. □

Proof of Theorem 1.1

By Propositions 3.1 and 3.2, we obtain that when $(\mu ,\sigma )\in \Lambda_{\theta^{\ast }}$, problem (1.1) has at least two solutions $(u^{+}_{0},v^{+}_{0})$ and $(u^{-}_{0},v^{-}_{0})$ such that $(u^{+}_{0},v^{+}_{0})\in \mathcal{S}^{+}_{\mu ,\sigma }, (u^{-}_{0},v ^{-}_{0})\in \mathcal{S}^{-}_{\mu ,\sigma }$, where $u^{+}_{0}\not \equiv 0,v^{+}_{0}\not \equiv 0$ and $(u^{-}_{0},v^{-}_{0})\not \equiv (0,0)$. Note that $\mathcal{S}^{+}_{\mu ,\sigma }\cap \mathcal{S}^{-}_{\mu ,\sigma }=\emptyset$, then these two solutions are distinct. This finishes the proof. □

4 Proof of Theorem 1.2

Throughout this section, we assume that all the conditions in Theorem 1.2 hold.

Since $\mu =\sigma =0$, then (2.2) becomes

$$\begin{aligned} \mathcal{J}(u,v)=\mathcal{J}_{0,0}(u,v)= \frac{k}{p} \bigl\Vert (u,v) \bigr\Vert ^{p}+ \frac{ \lambda }{\alpha }G(u,v)-\frac{1}{q}H(u,v), \end{aligned}$$

(4.1)

where $\alpha , G(u,v), q, H(u,v)$ are as in (2.3).

If $\mathrm{(H_{3})}$ holds, we derive from Hölder’s inequality and (1.6) that

$$\begin{aligned} H(u,v)= \int_{\Omega } h(x)\vert u \vert ^{a}\vert v \vert ^{b}\,dx \leq \Vert h \Vert _{\frac{p_{s} ^{\ast }}{p_{s}^{\ast }-q}}C_{p_{s}^{\ast }}^{q} \bigl\Vert (u,v) \bigr\Vert ^{q}. \end{aligned}$$

(4.2)

If $\mathrm{(H_{4})}$ holds, similarly, we have

$$\begin{aligned} H(u,v)= \int_{\Omega } h(x)\vert u \vert ^{a}\vert v \vert ^{b}\,dx\leq \Vert h \Vert _{\infty }C_{q} ^{q} \bigl\Vert (u,v) \bigr\Vert ^{q}. \end{aligned}$$

(4.3)

The proof of Theorem 1.2 is mainly dependent on the following mountain pass theorem.

Lemma 4.1

([27])

Let X be a real Banach space, suppose that $\mathcal{I}\in C^{1}(X, \mathbb {R} )$ satisfies $(\mathit{PS})$ with $\mathcal{I}(0)=0$. In addition,

$\mathrm{(A_{1})}$ :

there exist positive numbers δ and d such that $\mathcal{I}(u)\geq d$ if $\Vert u \Vert _{X}=\delta$;

$\mathrm{(A_{2})}$ :

there exists $v\in X$ such that $\Vert v \Vert _{X}>\delta$ and $\mathcal{I}(v)<0$.

Then there exists a critical value $e\geq d$ for $\mathcal{I}$. Moreover, e can be characterized as

$$ e=\inf_{\Phi \in \Gamma }\max_{0\leq t\leq 1}\mathcal{I} \bigl( \Phi (t) \bigr), $$

where

$$ \Gamma = \bigl\{ \Phi \in C \bigl([0,1],X \bigr) \mid \Phi (0)=0, \Phi (1)=v \bigr\} . $$

Next, we will prove that the functional $\mathcal{J}$ defined by (4.1) satisfies $(\mathit{PS})$.

Recall that we say $\mathcal{J}$ satisfies the $(\mathit{PS})$ condition at the level $c\in \mathbb {R}$ (shortly: $\mathcal{J}$ satisfies $(PS)_{c}$) if every sequence $\{(u_{n},v_{n})\}\subset X$ along with $\mathcal{J}(u_{n},v _{n})\rightarrow c$ and $\mathcal{J}'(u_{n},v_{n})\rightarrow 0$ as $n\rightarrow \infty$ has a converging subsequence (in X). We say $\mathcal{J}$ satisfies the $(\mathit{PS})$ condition (shortly: $\mathcal{J}$ satisfies $(\mathit{PS})$) if $\mathcal{J}$ satisfies $(PS)_{{c}}$ for each $c\in \mathbb {R}$.

Lemma 4.2

Any $(PS)_{c}$ sequence $\{(u_{n},v_{n})\}$ for $\mathcal{J}$ is bounded in X.

Proof

Let the sequence $\{(u_{n},v_{n})\}\subset X$ satisfy

$$\begin{aligned} \mathcal{J}(u_{n},v_{n})\rightarrow c,\qquad \mathcal{J}'(u_{n},v_{n}) \rightarrow 0\quad \mbox{as }n\rightarrow \infty . \end{aligned}$$

(4.4)

If $(H_{0}^{\prime})$ and $\mathrm{(H_{3})}$ are true, by (4.2), we take $s>\alpha$ and obtain that, for large n,

$$\begin{aligned}& c+1+ \bigl\Vert (u_{n},v_{n}) \bigr\Vert \\& \quad \geq \mathcal{J}(u_{n},v_{n})-s^{-1} \mathcal{J}'(u_{n},v_{n}) (u_{n},v_{n}) \geq \lambda \biggl(\frac{1}{\alpha }-\frac{1}{s} \biggr)G(u_{n},v_{n})+ \biggl( \frac{1}{s}-\frac{1}{q} \biggr)H(u_{n}, v_{n}) \\& \quad \geq \frac{\lambda }{2^{\tau }} \biggl(\frac{1}{\alpha }-\frac{1}{s} \biggr) \bigl\Vert (u _{n},v_{n}) \bigr\Vert ^{\alpha } + \biggl( \frac{1}{s}-\frac{1}{q} \biggr)\Vert h \Vert _{\frac{p_{s} ^{\ast }}{p_{s}^{\ast }-q}}C_{p_{s}^{\ast }}^{q} \bigl\Vert (u_{n},v_{n}) \bigr\Vert ^{q}. \end{aligned}$$

(4.5)

This implies that $\{(u_{n},v_{n})\}$ is bounded in X.

If $(H_{0}^{\prime})$ and $\mathrm{(H_{4})}$ hold, substituting $H(u_{n},v _{n})$ in (4.5) by (4.3), we conclude that $\{(u_{n},v _{n})\}$ is bounded in X. This finishes the proof. □

In view of the sequence $\{(u_{n},v_{n})\}$ given by (4.4) is a bounded sequence in X, there is a subsequence, still denoted by $\{(u_{n},v_{n})\}$ and $(u,v)\in X$ such that $\Vert (u_{n},v_{n}) \Vert \leq M, \Vert (u,v) \Vert \leq M$ with some constant $M>0$ and, as $n\rightarrow \infty$,

$$\begin{aligned} \textstyle\begin{cases} u_{n}\rightharpoonup u\quad \mbox{in } W_{0}, \qquad u_{n}\rightarrow u\quad \mbox{ in } L^{r}(\Omega ), \forall r\in [1,p_{s}^{*}); \\ v_{n}\rightharpoonup v \quad \mbox{in } W_{0},\qquad v_{n}\rightarrow v\quad \mbox{ in } L^{r}(\Omega ), \forall r\in [1,p_{s}^{*}); \\ u_{n}\rightarrow u, \qquad v_{n}\rightarrow v \quad \mbox{a.e. in } \Omega . \end{cases}\displaystyle \end{aligned}$$

(4.6)

Lemma 4.3

Let $\{(u_{n},v_{n})\}$ be a $(PS)_{c}$ sequence and satisfy (4.6). Then the following statements hold:

• (i) $(u_{n},v_{n})\rightarrow (u,v)$ in X as $n\rightarrow \infty$, that is, $\mathcal{J}$ satisfies $(\mathit{PS})$.

• (ii) $(u,v)\in X$ is a critical point for $\mathcal{J}$.

Proof

(i)

$$ \bigl\langle \mathcal{J}'(u_{n},v_{n}) (u_{n}-u,0) \bigr\rangle =P_{n}- \frac{a}{q} \int_{\Omega }h(x)\vert u_{n} \vert ^{a-2}u_{n}(u_{n}-u)\vert v_{n} \vert ^{b}\,dx, $$

where

$$\begin{aligned} P_{n} =M \bigl(\Vert u_{n} \Vert ^{p} \bigr) \int_{\mathcal{Q}}\frac{\vert \omega_{n} \vert ^{p-2} \omega_{n}(\omega_{n}-\omega )}{\vert x-y \vert ^{N+ps}}\,dx\,dy,\qquad \omega_{n}=u_{n}(x)-u_{n}(y), \omega =u(x)-u(y). \end{aligned}$$

Obviously, $\langle \mathcal{J}'(u_{n},v_{n})(u_{n}-u,0)\rangle \rightarrow 0$ as $n\rightarrow \infty$ since $\mathcal{J}'(u_{n},v _{n})\rightarrow 0$. By the dominated convergence theorem, we obtain

$$\begin{aligned} \int_{\Omega }h(x)\vert u_{n} \vert ^{a-2}u_{n}(u_{n}-u)\vert v_{n} \vert ^{b}\,dx\rightarrow 0\quad \mbox{as }n\rightarrow \infty . \end{aligned}$$

This implies $P_{n}\rightarrow 0 as n\rightarrow \infty$. Moreover, the fact $u_{n}\rightharpoonup u$ in $W_{0}$ implies $Q_{n}\rightarrow 0$, where $Q_{n}=M(\Vert u_{n} \Vert ^{p})\int_{\mathcal{Q}}\frac{\vert \omega \vert ^{p-2} \omega (\omega_{n}-\omega )}{\vert x-y \vert ^{N+ps}}\,dx\,dy$.

Therefore,

$$\begin{aligned} P_{n}-Q_{n} =M \bigl(\Vert u_{n} \Vert ^{p} \bigr) \int_{\mathcal{Q}}\frac{[\vert \omega_{n} \vert ^{p-2} \omega_{n}-\vert \omega \vert ^{p-2}\omega ](\omega_{n}-\omega )}{\vert x-y \vert ^{N+ps}}\,dx\,dy \rightarrow 0\quad \mbox{as } n\rightarrow \infty . \end{aligned}$$

(4.7)

Notice that there are the well-known vector inequalities given by

$$\begin{aligned}& \begin{aligned} &\bigl\langle \vert \eta \vert ^{p-2}\eta -\vert \xi \vert ^{p-2}\xi ,\eta -\xi \bigr\rangle \geq S _{p}\vert \eta -\xi \vert ^{p},\quad p\geq 2, \\ &\bigl\langle \vert \eta \vert ^{p-2}\eta -\vert \xi \vert ^{p-2}\xi ,\eta -\xi \bigr\rangle \geq \widetilde{S}_{p} \vert \eta -\xi \vert ^{2}\bigl(\vert \eta \vert +\vert \xi \vert \bigr)^{p-2},\quad 1< p< 2, \end{aligned} \end{aligned}$$

(4.8)

for all η, $\xi \in { \mathbb {R} }^{N}$, where $S_{p}$, $\widetilde{S}_{p}>0$ are constants depending only on p. Combining (4.7) with (4.8), it follows

$$ u_{n}\rightarrow u \quad \mbox{in } W_{0} \mbox{ as }n\rightarrow\infty . $$

Similarly, we can also prove

$$ v_{n}\rightarrow v \quad \mbox{in } W_{0} \mbox{ as }n \rightarrow\infty . $$

Therefore,

$$ (u_{n},v_{n})\rightarrow (u,v) \quad \mbox{in } X \mbox{ as } n\rightarrow\infty . $$

This implies that $\mathcal{J}$ satisfies $(\mathit{PS})$.

(ii) Note that $\mathcal{J}\in C^{1}(X, \mathbb {R} )$, using $(u_{n},v _{n})$ to be a $(PS)_{c}$ sequence and (i) of Lemma 4.3, we can prove that $(u,v)\in X$ is a critical point for $\mathcal{J}$. □

Proof of Theorem 1.2

By Lemmas 4.2–4.3, we have that the functional $\mathcal{J}$ satisfies $(\mathit{PS})$. Next, we prove that $\mathcal{J}$ satisfies $\mathrm{(A1)}$ and $\mathrm{(A2)}$. By (4.2) and (4.3), we get

$$\begin{aligned} \mathcal{J}(u,v) \geq &\frac{k}{p} \bigl\Vert (u,v) \bigr\Vert ^{p}+\frac{\lambda }{ \alpha }G(u,v)-\frac{l}{q} \bigl\Vert (u,v) \bigr\Vert ^{q} \\ \geq &\frac{k}{p} \bigl\Vert (u,v) \bigr\Vert ^{p}+ \frac{\lambda }{2^{\tau }\alpha } \bigl\Vert (u,v) \bigr\Vert ^{\alpha }- \frac{l}{q} \bigl\Vert (u,v) \bigr\Vert ^{q}, \end{aligned}$$

where $l=\max \{C^{q}_{p_{s}^{\ast }}\Vert h \Vert _{\frac{p_{s}^{\ast }}{p _{s}^{\ast }-q}}, C_{q}^{q}\Vert h \Vert _{\infty }\}$. Denote

$$ \phi (z)=z^{p} \biggl(\frac{k}{p}+\frac{b}{2^{\tau }\alpha }z^{\alpha -p}- \frac{l}{q}z^{q-p} \biggr),\quad z>0. $$

It is easy to verify that there exist $z_{1}, d>0$ such that $\phi (z_{1})\geq d$. Let $\delta =z_{1}$, we get $\mathcal{J}(u,v) \geq d$ if $\Vert (u,v) \Vert =\delta$. Thus, condition $(A_{1})$ is satisfied.

Furthermore, we verify $(A_{2})$. By $\mathrm{(H_{3})}$ or $\mathrm{(H_{4})}$, we can choose $u_{0}, v_{0}\in C_{0}^{\infty }(\Omega ), u_{0}\not \equiv 0, v_{0}\not \equiv 0$, such that $\Vert u_{0} \Vert =\Vert v_{0} \Vert =1$ and $\int_{\Omega }h(x)\vert u_{0} \vert ^{a}\vert v_{0} \vert ^{b}\,dx>0$. Let

$$\begin{aligned} \mathcal{J}(tu_{0},tv_{0})=t^{p} \gamma (t),\qquad \gamma (t)=B_{1}+\lambda B_{2}t^{\alpha -p}-B_{3}t^{q-p}, \quad t\geq 0, \end{aligned}$$

(4.9)

where

$$ B_{1}=\frac{2k}{p}>0, \qquad B_{2}= \frac{2}{\alpha }>0, \qquad B_{3}= \int_{\Omega}h(x)\vert u_{0} \vert ^{a} \vert v_{0} \vert ^{b}\,dx>0. $$

Then there exist $\lambda_{0}>0$ and large $t_{\lambda }>2^{- \frac{1}{p}}\delta$ such that $\mathcal{J}(t_{\lambda }u_{0},t_{ \lambda }v_{0})<0$ for $\lambda \in (0,\lambda_{0})$, where δ is in $\mathrm{(A1)}$. Let $q_{0}=t_{\lambda }u_{0}, w_{0}=t _{\lambda }v_{0}$, then $\Vert (q_{0},w_{0}) \Vert >\delta$ and $\mathcal{J}(q _{0},w_{0})<0$. Hence, by Lemma 4.1, there exists $(u_{1},v_{1}) \in X$ with $\mathcal{(}u_{1},v_{1})\geq d>0$ which is a solution of problem (1.1) under the assumptions in Theorem 1.2. In addition, it is easy to verify $u_{1}\not \equiv 0, v_{1}\not \equiv 0$. We now prove the second part of Theorem 1.2. If $(u,v)$ is a nontrivial solution of problem (1.1), combining (4.2)–(4.3) with Young’s inequality, we get

$$\begin{aligned} k \bigl\Vert (u,v) \bigr\Vert ^{p}+ \frac{\lambda }{2^{\tau }} \bigl\Vert (u,v) \bigr\Vert ^{\alpha } \leq & k \bigl\Vert (u,v) \bigr\Vert ^{p}+\lambda G(u,v) = \int_{\Omega }h(x)\vert u \vert ^{a}\vert v \vert ^{b}\,dx \\ \leq & l \bigl\Vert (u,v) \bigr\Vert ^{q}\leq k \bigl\Vert (u,v) \bigr\Vert ^{p}+\frac{\lambda_{1}}{2^{ \tau }} \bigl\Vert (u,v) \bigr\Vert ^{\alpha }, \end{aligned}$$

(4.10)

where $l=\max \{C^{q}_{p_{s}^{\ast }}\Vert h \Vert _{\frac{p_{s}^{\ast }}{p _{s}^{\ast }-q}}, C_{q}^{q}\Vert h \Vert _{\infty }\}$, $\lambda_{1}=2^{\tau }(\frac{q-p}{ \alpha -p})(\frac{\alpha -q}{\alpha -p})^{\frac{\alpha -q}{q-p}}l^{\frac{ \alpha -p}{q-p}}k^{-\frac{\alpha -q}{q-p}}$. This implies $0<\lambda \leq \lambda_{1}$. In other words, if $\lambda >\lambda_{1}$, there is no nontrivial solution of problem (1.1). This finishes the proof. □