1 Introduction and main results

In this paper, we consider the existence and multiplicity of positive solutions for the following truncated Kirchhoff-type system involving concave–convex nonlinearities:

figure a

where \({\varOmega} \subset\mathbb{R}^{N}\) (\(N > 4\)) is a bounded domain with smooth boundary, \(\alpha>1\) and \(\beta>1\) satisfy \(\alpha+ \beta<2^{\ast}<4\) where \(2^{\ast}:= 2N/ (N-2 )\) is the critical Sobolev exponent, \(1< q<2\), \(M_{i}= a_{i} + b_{i}t\), \(a_{i},b_{i}>0\) (\(i=1,2\)), \((\lambda,\mu )\in (0,+\infty )\times (0,+\infty )\), \(k < \min \{ \frac{a_{1} (\alpha+ \beta-2 )}{2b_{1}},\frac{a_{2} (\alpha+ \beta-2 )}{2b_{2}} \}\),

$$\begin{aligned} M_{i}^{k} (s )= \left\{\textstyle\begin{array}{l@{\quad}l} M_{i} (s ) &\text{if } s \le k, \\ M_{i} (k ) &\text{if } s > k, \end{array}\displaystyle \right .\quad i=1,2, \end{aligned}$$

and the weight functions f, g satisfy the following conditions:

\((F)\):

\(f \in C(\bar{\varOmega})\), \(f > 0\);

\((G)\):

\(g \in C(\bar{\varOmega})\), \(g > 0\).

Problem (\(E_{\lambda ,\mu ,M^{k}}\)) is called nonlocal because of the presence of \(b_{1}\int_{\varOmega} \vert \nabla u \vert ^{2}\) and \(b_{2}\int_{\varOmega} \vert \nabla v \vert ^{2}\), and \(b (\int _{\varOmega} \vert \nabla u \vert ^{2} ) \Delta u\) appears in the Kirchhoff equation

figure b

related to the stationary analogue of the equation

$$\begin{aligned} u_{tt} - \biggl(a + b \int_{\varOmega} \vert \nabla u \vert ^{2} \biggr)\Delta u = f (x,u ), \end{aligned}$$

where u is the displacement, \(f (x,t )\) is the external force, a is the initial tension, and b is related to the intrinsic properties of the string. The equation was first proposed by Kirchhoff [1] as an extension of the classical D’Alembert’s wave equation to describe free vibrations of elastic strings. Several existence results for equation (\(E_{1}\)) have been obtained in recent years; see [2–7] and references therein. Moreover, other similar arguments are also obtained; see [8–12].

When \(a_{i}=0\) and \(b_{i}=1 \) (\(i=1,2 \)), problem (\(E_{\lambda ,\mu ,M^{k}}\)) becomes

figure c

Nowadays scientists and researchers paid more attentions to problem (\(E_{2}\)) with sigh-changing weight function. For instance, the case \(\alpha+ \beta= 2^{\ast}\) is considered in [13], whereas in [14, 15] the case \(\alpha+ \beta< 2^{\ast}\) was studied, and the existence and multiplicity of positive solutions when \((\lambda ,\mu)\) belongs to a certain subset of \(\mathbb{R}^{2}\) were obtained.

Meanwhile, the problem about Kirchhoff system has been studied. In [16, 17] the Kirchhoff system with boundary value shows several physical and biological systems with u and v describing a process depending on the average of itself, such as population densities. Lv and Peng [18] established the existence of positive vector solutions and positive vector ground state solutions by using variational methods and also studied the asymptotic behavior of these solutions. In [19] the authors studied the nonlocal boundary value problem of Kirchhoff-type system, where Ω is a bounded domain in \(\mathbb{R}^{N}\), \(N=1,2,3\), \(\beta\in\mathbb{R}\), \(a_{i},b_{i},\lambda_{i}>0\) for \(i=1,2\), and p and q are two positive numbers satisfying certain conditions. They obtained the existence of positive solutions by the Nehari manifold and mountain pass lemma and the multiplicity by using cohomological index of Fadell and Rabinowitz. Also, they considered the critical case and proved the existence of positive least energy solutions when β is sufficiently large.

Inspired by the works mentioned, in this paper, we mainly study the truncated Kirchhoff-type system with concave–convex nonlinearities involving \(\alpha+ \beta<4\), since the case \(\alpha+ \beta\geq4\) is trivial, which is easy to be proved by using the method in [20]. To the best of our knowledge, the usual mountain pass theorem cannot be directly applied because the concave–convex nonlinearities do not satisfy the mountain pass geometry, so it is difficult to obtain a bounded Palais–Smale sequence (see Theorem 1.15 in [21]). Hence, in this work, by using the method of Nehari manifold, we overcome this difficulty and obtain the existence of multiple positive solutions.

Let us state our knowledge framework and main result. For \(u \in H_{0}^{1}(\varOmega)\), its usual norm is denoted by

$$\Vert u \Vert ^{2}= \int_{\varOmega} \vert \nabla u \vert ^{2}. $$

Consider system (\(E_{\lambda ,\mu ,M^{k}}\)) in the framework of the Sobolev space \(H = H_{0}^{1}(\varOmega) \times H_{0}^{1}(\varOmega)\) with the standard norm

$$\bigl\Vert (u,v ) \bigr\Vert _{H}^{2}= \int_{\varOmega} \bigl( \vert \nabla u \vert ^{2} + \vert \nabla v \vert ^{2} \bigr). $$

The energy functional associated with the equation (\(E_{\lambda ,\mu ,M^{k}}\)) is defined by

$$\begin{aligned} I_{\lambda,\mu,M^{k}} (u,v ) &= \frac {1}{2}\hat{M}_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr) + \frac {1}{2} \hat{M}_{2}^{k} \bigl( \Vert v \Vert ^{2} \bigr) - \frac {1}{q} \int_{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr) \\ &\quad- \frac{1}{\alpha+ \beta} \int_{\varOmega} \vert u \vert ^{\alpha} \vert v \vert ^{\beta}, \end{aligned}$$

where \(\hat{M}_{i}^{k} (t ) = \int_{0}^{t} {M}_{i}^{k} (s )\, ds\). It is well known that the functional \(I_{\lambda,\mu,M^{k}}\) is of class \({C}^{1}\). Further, we denote

$$\varLambda= \bigl( \bigl(\lambda \Vert f \Vert _{\infty} \bigr)^{{2}/{ (2-q )}} + \bigl(\mu \Vert g \Vert _{\infty } \bigr)^{{2}/{ (2-q )}} \bigr)^{{ (2-q )}/{2}}. $$

Theorem 1.1

Assume that conditions\((F)\), \((G)\)hold. If\(\alpha+ \beta<2^{\ast}\), then there exists\(\varLambda_{0} >0\)such that for\(0 < \varLambda< \varLambda_{0}\), equation\((E_{\lambda,\mu ,M^{k}})\)has at least two positive solutions\((u_{\lambda,\mu ,M^{k}}^{+},v_{\lambda,\mu,M^{k}}^{+} )\)and\((u_{\lambda ,\mu,M^{k}}^{-},v_{\lambda,\mu,M^{k}}^{-} )\).

2 Preliminaries

Let us introduce the Nehari manifold

$$\begin{aligned} N_{\lambda,\mu,M^{k}} = \bigl\{ (u,v )\in H \setminus (0,0 ) \mid \bigl\langle I'_{\lambda,\mu ,M^{k}} (u,v ), (u,v ) \bigr\rangle =0 \bigr\} , \end{aligned}$$

and denote \(\varPsi_{\lambda,\mu,M^{k}} (u,v )= \langle I'_{\lambda,\mu,M^{k}} (u,v ), (u,v ) \rangle\) and \(\varPhi_{\lambda,\mu,M^{k}} (u,v )= \langle\varPsi'_{\lambda,\mu,M^{k}} (u,v ), (u,v ) \rangle\). If \((u,v ) \in N_{\lambda ,\mu,M^{k}}\), then

$$\begin{aligned} \varPhi_{\lambda,\mu,M^{k}} (u,v )={}&2 \bigl(M_{1}^{k} \bigr)' \bigl( \Vert u \Vert ^{2} \bigr) \Vert u \Vert ^{4} + 2 \bigl(M_{2}^{k} \bigr)' \bigl( \Vert v \Vert ^{2} \bigr) \Vert v \Vert ^{4} + 2M_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr) \Vert u \Vert ^{2} \\ &+ 2M_{2}^{k} \bigl( \Vert v \Vert ^{2} \bigr) \Vert v \Vert ^{2} - q \int_{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr) - (\alpha+ \beta ) \int _{\varOmega} \vert u \vert ^{\alpha} \vert v \vert ^{\beta } \\ ={}& (2-q )M_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr) \Vert u \Vert ^{2}+2 \bigl(M_{1}^{k} \bigr)' \bigl( \Vert u \Vert ^{2} \bigr) \Vert u \Vert ^{4} + (2-q )M_{2}^{k} \bigl( \Vert v \Vert ^{2} \bigr) \Vert v \Vert ^{2} \\ &+ 2 \bigl(M_{2}^{k} \bigr)' \bigl( \Vert v \Vert ^{2} \bigr) \Vert v \Vert ^{4} - ( \alpha+ \beta-q ) \int_{\varOmega } \vert u \vert ^{\alpha} \vert v \vert ^{\beta} \end{aligned}$$
(2.1)
$$\begin{aligned} = {}& (2-\alpha- \beta ) M_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr) \Vert u \Vert ^{2} + 2 \bigl(M_{1}^{k} \bigr)' \bigl( \Vert u \Vert ^{2} \bigr) \Vert u \Vert ^{4} \\ &+ (2-\alpha- \beta )M_{2}^{k} \bigl( \Vert v \Vert ^{2} \bigr) \Vert v \Vert ^{2}+ 2 \bigl(M_{2}^{k} \bigr)' \bigl( \Vert v \Vert ^{2} \bigr) \Vert v \Vert ^{4} \\ &- (q-\alpha- \beta ) \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr) . \end{aligned}$$
(2.2)

We split \(N_{\lambda,\mu,M^{k}}\) into three parts:

$$\begin{gathered} N_{\lambda,\mu,M^{k}}^{+} = \bigl\{ (u,v ) \in N_{\lambda ,\mu,M^{k}} \mid \varPhi_{\lambda,\mu,M^{k}} (u,v ) >0 \bigr\} , \\ N_{\lambda,\mu,M^{k}}^{0} = \bigl\{ (u,v ) \in N_{\lambda ,\mu,M^{k}} \mid \varPhi_{\lambda,\mu,M^{k}} (u,v ) =0 \bigr\} , \\ N_{\lambda,\mu,M^{k}}^{-} = \bigl\{ (u,v ) \in N_{\lambda ,\mu,M^{k}} \mid \varPhi_{\lambda,\mu,M^{k}} (u,v ) < 0 \bigr\} .\end{gathered} $$

The best Sobolev constant \(S_{r}\) (\(1< r<2^{\ast} \)) and \(S_{\alpha,\beta}\) are respectively defined by

$$\begin{aligned}& S_{r} = \inf_{u \in H_{0}^{1}(\varOmega) \setminus\{0\}} \frac{\int_{\varOmega} \vert \nabla u \vert ^{2}}{ (\int _{\varOmega} \vert u \vert ^{r} )^{2/r}}, \\& S_{\alpha,\beta} = \inf_{u,v \in H^{1}_{0}(\varOmega) \setminus\{0\}} \frac{\int_{\varOmega} ( \vert \nabla u \vert ^{2} + \vert \nabla v \vert ^{2} )}{ (\int_{\varOmega} \vert u \vert ^{\alpha} \vert v \vert ^{\beta} )^{2/(\alpha+ \beta)}}. \end{aligned}$$

Lemma 2.1

Assume that conditions\((F)\)and\((G)\)hold. Then the energy functional\(I_{\lambda,\mu,M^{k}}\)is coercive and bounded below on\(N_{\lambda,\mu,M^{k}}\).

Proof

For \((u,v ) \in N_{\lambda,\mu,M^{k}}\),

$$\begin{aligned} &{M}_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr) \Vert u \Vert ^{2} + {M}_{2}^{k} \bigl( \Vert v \Vert ^{2} \bigr) \Vert v \Vert ^{2} \\ &\quad = \int_{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr) + \int_{\varOmega} \vert u \vert ^{\alpha } \vert v \vert ^{\beta}. \end{aligned}$$

Setting \(M_{0} = \min \{\frac{a_{1}}{2} - \frac{M_{1} (k )}{\alpha+ \beta},\frac{a_{2}}{2} - \frac{M_{2} (k )}{\alpha+ \beta} \}\), by the Sobolev and Hölder inequalities we obtain

$$\begin{aligned} I_{\lambda,\mu,M^{k}} (u,v ) &=I_{\lambda,\mu,M^{k}} (u,v ) - \frac {1}{\alpha+ \beta}\varPsi_{\lambda,\mu,M^{k}} (u,v ) \\ &= \biggl(\frac{1}{2}\hat{M}_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr)- \frac{1}{\alpha+ \beta} {M}_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr) \Vert u \Vert ^{2} \biggr) \\ &\quad+ \biggl(\frac{1}{2}\hat{M}_{2}^{k} \bigl( \Vert v \Vert ^{2} \bigr)- \frac{1}{\alpha+ \beta} {M}_{2}^{k} \bigl( \Vert v \Vert ^{2} \bigr) \Vert v \Vert ^{2} \biggr) \\ &\quad- \frac{\alpha+ \beta-q}{ (\alpha+ \beta )q} \int _{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr). \end{aligned}$$

To finish this proof, we need the following claims.

Claim 1

\(\frac{1}{2}\hat{M}_{1}^{k} ( \Vert u \Vert ^{2} )- \frac{1}{\alpha+ \beta} {M}_{1}^{k} ( \Vert u \Vert ^{2} ) \Vert u \Vert ^{2} \ge ( \frac {a_{1}}{2} - \frac{M_{1} (k )}{\alpha+ \beta} ) \Vert u \Vert ^{2}\).

Claim 2

\(\frac{1}{2}\hat{M}_{2}^{k} ( \Vert v \Vert ^{2} )- \frac{1}{\alpha+ \beta} {M}_{2}^{k} ( \Vert v \Vert ^{2} ) \Vert v \Vert ^{2} \ge ( \frac {a_{2}}{2} - \frac{M_{2} (k )}{\alpha+ \beta} ) \Vert v \Vert ^{2}\).

Claim 3

\(\int_{\varOmega} (\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} ) \le S_{q}^{-q/2}\varLambda \Vert (u,v ) \Vert _{H}^{q}\).

First, by the Sobolev and Hölder inequalities we easily obtain Claim 3. Then, since the proof of Claim 2 is the same as that of Claim 1, here we only give the proof of Claim 1. If \(\Vert u \Vert ^{2} \le k\), then we have that

$$\begin{aligned} &\frac{1}{2}\hat{M}_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr)- \frac{1}{\alpha+ \beta} {M}_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr) \Vert u \Vert ^{2} - \biggl( \frac{a_{1}}{2} - \frac{M_{1} (k )}{\alpha+ \beta} \biggr) \Vert u \Vert ^{2} \\ &\quad= \frac{1}{\alpha+ \beta} b_{1} \Vert u \Vert ^{2} \bigl(k- \Vert u \Vert ^{2} \bigr) + \frac{b_{1}}{4} \Vert u \Vert ^{4} \ge0, \end{aligned}$$

and if \(\Vert u \Vert ^{2} > k\), then we conclude that

$$\begin{aligned} &\frac{1}{2}\hat{M}_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr)- \frac{1}{\alpha+ \beta} {M}_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr) \Vert u \Vert ^{2} - \biggl( \frac{a_{1}}{2} - \frac{M_{1} (k )}{\alpha+ \beta} \biggr) \Vert u \Vert ^{2} \\ &\quad= \frac{1}{2} \int_{0}^{k}{M}_{1} (s ) \,ds + \frac {1}{2} \int_{k}^{ \Vert u \Vert ^{2}}{M}_{1} (k )\, ds - \frac{a_{1}}{2} \Vert u \Vert ^{2} \\ &\quad=\frac{1}{4}b_{1}k \bigl(2 \Vert u \Vert ^{2}-k \bigr)\ge0, \end{aligned}$$

which completes the proof of Claim 1.

Thus, we could obtain that

$$\begin{aligned} I_{\lambda,\mu,M^{k}} (u,v ) &\ge \biggl( \frac{a_{1}}{2} - \frac{M_{1} (k )}{\alpha+ \beta} \biggr) \Vert u \Vert ^{2} + \biggl( \frac{a_{2}}{2} - \frac{M_{2} (k )}{\alpha+ \beta} \biggr) \Vert v \Vert ^{2} - \frac{\alpha+ \beta-q}{ (\alpha+ \beta )q}S_{q}^{-q/2} \varLambda \bigl\Vert (u,v ) \bigr\Vert _{H}^{q} \\ & \ge M_{0} \bigl\Vert (u,v ) \bigr\Vert _{H}^{2} - \frac{\alpha + \beta-q}{ (\alpha+ \beta )q}S_{q}^{-q/2}\varLambda \bigl\Vert (u,v ) \bigr\Vert _{H}^{q}. \end{aligned}$$
(2.3)

Since \(k < \min \{\frac{a_{1} (\alpha+ \beta-2 )}{2b_{1}},\frac{a_{2} (\alpha+ \beta-2 )}{2b_{2}} \}\), we have \(M_{0}>0\). Thus \(I_{\lambda,\mu,M^{k}}\) is coercive and bounded below on \(N_{\lambda,\mu,M^{k}}\). □

Lemma 2.2

Suppose that\((u_{0},v_{0} )\)is a local minimizer for\(I_{\lambda,\mu,M^{k}}\)on\(N_{\lambda,\mu ,M^{k}}\), \((u_{0},v_{0} ) \notin N_{\lambda,\mu ,M^{k}}^{0}\). Then\(I'_{\lambda,\mu,M^{k}} (u_{0},v_{0} ) =0\)in\(H^{-1}\).

Proof

We refer to Theorem 2.3 of [22]. □

Since \(2<\alpha+ \beta< 4\), we have \(k < \frac{a_{1} (\alpha+ \beta-2 )}{2b_{1}} < \frac{a_{1} (\alpha+ \beta-2 )}{b_{1} (4-\alpha- \beta )}\) and \(k < \frac{a_{2} (\alpha+ \beta-2 )}{2b_{2}} < \frac {a_{2} (\alpha+ \beta-2 )}{b_{2} (4-\alpha- \beta )}\), so that \(a_{1} (\alpha+ \beta-2 ) - b_{1} (4-\alpha- \beta )k >0\) and \(a_{2} (\alpha+ \beta -2 ) - b_{2} (4-\alpha- \beta )k >0\).

Setting

$$\tilde{\varLambda} = \frac{\tilde{C}_{2}S_{q}^{q/2}}{\alpha+ \beta -q} \biggl(\frac{ (2-q )\tilde{C}_{1} S_{\alpha,\beta }^{ (\alpha+ \beta )/2}}{\alpha+ \beta-q} \biggr)^{ (2-q )/ (\alpha+ \beta-2 )}, $$

where \(\tilde{C}_{1} = \min \{a_{1},a_{2},M_{1} (k ),M_{2} (k ) \}\), and \(\tilde{C}_{2}= \min \{a_{1} (\alpha+ \beta-2 ) - b_{1} (4-\alpha- \beta )k, a_{2} (\alpha+ \beta-2 )- b_{2} (4-\alpha- \beta )k, M_{1} (k ) (\alpha+ \beta-2 ),M_{2} (k ) (\alpha+ \beta -2 ) \}\), we obtain the following result.

Lemma 2.3

Assume that conditions\((F)\)and\((G)\)hold. If\(\alpha+ \beta< 2^{\ast}\), then\(N_{\lambda,\mu ,M^{k}}^{0}=\varnothing\)for\(\varLambda\in (0,\tilde{\varLambda } )\).

Proof

For each \((u,v )\in N_{\lambda,\mu ,M^{k}}^{0}\), in (2.1), we discuss the problem in four cases.

Case 1: If \(\Vert u \Vert ^{2} \le k\), \(\Vert v \Vert ^{2} \le k\), and

$$\begin{aligned} \varPhi_{\lambda,\mu,M^{k}} (u,v ) &= (2-q )a_{1} \Vert u \Vert ^{2} + (4-q )b_{1} \Vert u \Vert ^{4} + (2-q )a_{2} \Vert v \Vert ^{2} + (4-q )b_{2} \Vert v \Vert ^{4} \\ &\quad- (\alpha+ \beta-q ) \int_{\varOmega} \vert u \vert ^{\alpha} \vert v \vert ^{\beta}, \end{aligned}$$

then

$$\begin{aligned} (2-q )a \bigl\Vert (u,v ) \bigr\Vert _{H}^{2} &\le (2-q )a_{1} \Vert u \Vert ^{2} + (2-q )a_{2} \Vert v \Vert ^{2} \\ &\le (\alpha+ \beta-q ) \int_{\varOmega} \vert u \vert ^{\alpha} \vert v \vert ^{\beta} \\ &\le (\alpha+ \beta-q )S_{\alpha,\beta}^{- (\alpha + \beta )/2} \bigl\Vert (u,v ) \bigr\Vert _{H}^{\alpha+ \beta} , \end{aligned}$$
(2.4)

where \(a = \min \{a_{1},a_{2} \}>0\).

Case 2: If \(\Vert u \Vert ^{2} \le k\), \(\Vert v \Vert ^{2} > k\), and

$$\begin{aligned} \varPhi_{\lambda,\mu,M^{k}} (u,v ) &= (2-q )a_{1} \Vert u \Vert ^{2} + (4-q )b_{1} \Vert u \Vert ^{4} + (2-q )M_{2} (k ) \Vert v \Vert ^{2} \\ &\quad- (\alpha+ \beta-q ) \int_{\varOmega} \vert u \vert ^{\alpha} \vert v \vert ^{\beta}, \end{aligned}$$

then

$$\begin{aligned} (2-q )a_{1}^{k} \bigl\Vert (u,v ) \bigr\Vert _{H}^{2} &\le (2-q )a_{1} \Vert u \Vert ^{2} + (2-q )M_{2} (k ) \Vert v \Vert ^{2} \\ &\le (\alpha+ \beta-q )S_{\alpha,\beta}^{- (\alpha + \beta )/2} \bigl\Vert (u,v ) \bigr\Vert _{H}^{\alpha+ \beta} , \end{aligned}$$
(2.5)

where \(a_{1}^{k} = \min \{a_{1},M_{2}(k) \}\).

Case 3: If \(\Vert u \Vert ^{2} > k\), \(\Vert v \Vert ^{2} \le k\), and

$$\begin{aligned} \varPhi_{\lambda,\mu,M^{k}} (u,v ) &= (2-q )M_{1} (k ) \Vert u \Vert ^{2} + (2-q )a_{2} \Vert v \Vert ^{2} + (4-q )b_{2} \Vert v \Vert ^{4} \\ &\quad- (\alpha+ \beta-q ) \int_{\varOmega} \vert u \vert ^{\alpha} \vert v \vert ^{\beta}, \end{aligned}$$

then

$$\begin{aligned} (2-q )a_{2}^{k} \bigl\Vert (u,v ) \bigr\Vert _{H}^{2} &\le (2-q )M_{1} (k ) \Vert u \Vert ^{2} + (2-q )a_{2} \Vert v \Vert ^{2} \\ &\le (\alpha+ \beta-q )S_{\alpha,\beta}^{- (\alpha + \beta )/2} \bigl\Vert (u,v ) \bigr\Vert _{H}^{\alpha+ \beta} , \end{aligned}$$
(2.6)

where \(a_{2}^{k} = \min \{a_{2},M_{1} (k ) \}\).

Case 4: If \(\Vert u \Vert ^{2} > k\), \(\Vert v \Vert ^{2} > k\), and

$$\begin{aligned} \varPhi_{\lambda,\mu,M^{k}} (u,v ) &= (2-q )M_{1} (k ) \Vert u \Vert ^{2} + (2-q )M_{2} (k ) \Vert v \Vert ^{2} - ( \alpha + \beta-q ) \int_{\varOmega} \vert u \vert ^{\alpha} \vert v \vert ^{\beta}, \end{aligned}$$

then

$$\begin{aligned} (2-q )M (k ) \bigl\Vert (u,v ) \bigr\Vert _{H}^{2}&\le (\alpha+ \beta-q ) \int _{\varOmega} \vert u \vert ^{\alpha} \vert v \vert ^{\beta} \\ &\le (\alpha+ \beta-q )S_{\alpha,\beta}^{- (\alpha + \beta )/2} \bigl\Vert (u,v ) \bigr\Vert _{H}^{\alpha+ \beta} , \end{aligned}$$
(2.7)

where \(M (k ) = \min \{M_{1} (k ),M_{2} (k ) \}\).

For (2.2), we also split the proof into four cases as follows.

Case 1: If \(\Vert u \Vert ^{2} \le k\), \(\Vert v \Vert ^{2} \le k\), and

$$\begin{aligned} \varPhi_{\lambda,\mu,M^{k}} (u,v ) &= \bigl(b_{1} (4-\alpha- \beta ) \Vert u \Vert ^{2} - a_{1} (\alpha+ \beta-2 ) \bigr) \Vert u \Vert ^{2} \\ &\quad+ \bigl(b_{2} (4-\alpha- \beta ) \Vert v \Vert ^{2} - a_{2} (\alpha+ \beta-2 ) \bigr) \Vert v \Vert ^{2} \\ &\quad+ (\alpha+ \beta-q ) \int_{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr), \end{aligned}$$

then since \(k < \frac{a_{1} (\alpha+ \beta-2 )}{2b_{1}} < \frac{a_{1} (\alpha+ \beta-2 )}{b_{1} (4-\alpha- \beta )}\) and \(k < \frac{a_{2} (\alpha+ \beta-2 )}{2b_{2}} < \frac{a_{2} (\alpha+ \beta-2 )}{b_{2} (4-\alpha- \beta )}\), we have \(a_{1} (\alpha+ \beta -2 ) - b_{1}(4-\alpha- \beta)k >0\text{ and }a_{2} (\alpha+ \beta-2 ) - b_{2} (4-\alpha- \beta )k >0\). Thus

$$\begin{aligned}& \bigl(a_{1} (\alpha+ \beta-2 ) -b_{1} (4-\alpha- \beta )k \bigr) \Vert u \Vert ^{2} + \bigl( a_{2} (\alpha+ \beta-2 ) -b_{2} (4-\alpha- \beta )k \bigr) \Vert v \Vert ^{2} \\& \quad\le (\alpha+ \beta-q ) \int_{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr) \\& \quad\le (\alpha+ \beta-q )S_{q}^{-q/2}\varLambda \bigl\Vert (u,v ) \bigr\Vert _{H}^{q}. \end{aligned}$$

Let \(T_{1} = \min \{a_{1} (\alpha+ \beta-2 ) - b_{1} (4-\alpha- \beta )k,a_{2} (\alpha+ \beta -2 ) - b_{2} (4-\alpha- \beta )k \}\). Then

$$\begin{aligned} T_{1} \bigl\Vert (u,v ) \bigr\Vert _{H}^{2} \le (\alpha+ \beta-q )S_{q}^{-q/2}\varLambda \bigl\Vert (u,v ) \bigr\Vert _{H}^{q}. \end{aligned}$$
(2.8)

Case 2: If \(\Vert u \Vert ^{2} \le k\) and \(\Vert v \Vert ^{2} > k\), then

$$\begin{aligned} \varPhi_{\lambda,\mu,M^{k}} (u,v ) &= \bigl(b_{1} (4-\alpha- \beta ) \Vert u \Vert ^{2} - a_{1} (\alpha+ \beta-2 ) \bigr) \Vert u \Vert ^{2} -M_{2} (k ) (\alpha+ \beta-2 ) \Vert v \Vert ^{2} \\ &\quad+ (\alpha+ \beta-q ) \int_{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr) \end{aligned}$$

and

$$\begin{aligned} T_{2} \bigl\Vert (u,v ) \bigr\Vert _{H}^{2}& \le \bigl(a_{1} (\alpha+ \beta-2 ) - b_{1} (4-\alpha- \beta )k \bigr) \Vert u \Vert ^{2} + M_{2} (k ) (\alpha+ \beta-2 ) \Vert v \Vert ^{2} \\ & \le (\alpha+ \beta-q )S_{q}^{-q/2}\varLambda \bigl\Vert (u,v ) \bigr\Vert _{H}^{q}, \end{aligned}$$
(2.9)

where \(T_{2} = \min \{a_{1} (\alpha+ \beta-2 ) - b_{1} (4-\alpha- \beta )k,M_{2} (k ) (\alpha+ \beta-2 ) \}\).

Case 3: If \(\Vert u \Vert ^{2} > k\), \(\Vert v \Vert ^{2} \le k\), then

$$\begin{aligned} \varPhi_{\lambda,\mu,M^{k}} (u,v ) &= -M_{1} (k ) (\alpha+ \beta-2 ) \Vert u \Vert ^{2} + \bigl(b_{2} (4-\alpha- \beta ) \Vert v \Vert ^{2} - a_{2} (\alpha+ \beta-2 ) \bigr) \Vert v \Vert ^{2} \\ &\quad+ (\alpha+ \beta-q ) \int_{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr)\end{aligned} $$

and

$$\begin{aligned} T_{3} \bigl\Vert (u,v ) \bigr\Vert _{H}^{2}& \le M_{1} (k ) (\alpha+ \beta-2 ) \Vert u \Vert ^{2} + \bigl(a_{2} (\alpha+ \beta-2 ) - b_{2} (4-\alpha- \beta )k \bigr) \Vert v \Vert ^{2} \\ & \le (\alpha+ \beta-q )S_{q}^{-q/2}\varLambda \bigl\Vert (u,v ) \bigr\Vert _{H}^{q}, \end{aligned}$$
(2.10)

where \(T_{3} = \min \{M_{1} (k ) (\alpha+ \beta -2 ), a_{2} (\alpha+ \beta-2 ) - b_{2} (4-\alpha- \beta )k \}\).

Case 4: If \(\Vert u \Vert ^{2} > k\), \(\Vert v \Vert ^{2} > k\), then

$$\begin{aligned} \varPhi_{\lambda,\mu,M^{k}} (u,v ) &= -M_{1} (k ) (\alpha+ \beta-2 ) \Vert u \Vert ^{2} - M_{2} (k ) (\alpha+ \beta-2 ) \Vert v \Vert ^{2} \\ &\quad+ (\alpha+ \beta-q ) \int_{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr) \end{aligned}$$

and

$$\begin{aligned} T_{4} \bigl\Vert (u,v ) \bigr\Vert _{H}^{2}& \le M_{1} (k ) (\alpha+ \beta-2 ) \Vert u \Vert ^{2} + M_{2} (k ) (\alpha+ \beta-2 ) \Vert v \Vert ^{2} \\ & \le (\alpha+ \beta-q )S_{q}^{-q/2}\varLambda \bigl\Vert (u,v ) \bigr\Vert _{H}^{q}, \end{aligned}$$
(2.11)

where \(T_{4} = \min \{M_{1} (k ) (\alpha+ \beta -2 ), M_{2} (k ) (\alpha+ \beta-2 ) \}\).

So, it follows from (2.4)–(2.7) that

$$\begin{aligned} (2-q )\tilde{C}_{1} \bigl\Vert (u,v ) \bigr\Vert _{H}^{2} \le (\alpha+ \beta-q )S_{\alpha,\beta}^{- (\alpha+ \beta )/2} \bigl\Vert (u,v ) \bigr\Vert _{H}^{\alpha+ \beta}. \end{aligned}$$
(2.12)

Similarly, by (2.8)–(2.11) we also get that

$$\begin{aligned} \tilde{C}_{2} \bigl\Vert (u,v ) \bigr\Vert _{H}^{2} \le (\alpha+ \beta-q )S_{q}^{-q/2} \varLambda \bigl\Vert (u,v ) \bigr\Vert _{H}^{q}, \end{aligned}$$
(2.13)

and by (2.12)–(2.13) we have

$$\begin{aligned} \biggl(\frac{ (2-q )\tilde{C}_{1} S_{\alpha ,\beta}^{ (\alpha+ \beta )/2}}{\alpha+ \beta-q} \biggr)^{1/ (\alpha+ \beta-2 )} \le \bigl\Vert (u,v ) \bigr\Vert _{H} \le \biggl(\frac{ (\alpha+ \beta-q )\varLambda}{\tilde{C}_{2}S_{q}^{q/2}} \biggr)^{1/ (2-q )}. \end{aligned}$$

Consequently,

$$\varLambda\ge\frac{\tilde{C}_{2}S_{q}^{q/2}}{\alpha+ \beta-q} \biggl(\frac{ (2-q )\tilde{C}_{1} S_{\alpha,\beta}^{ (\alpha+ \beta )/2}}{\alpha+ \beta-q} \biggr)^{ (2-q )/ (\alpha+ \beta-2 )} = \tilde{\varLambda}. $$

Therefore \(N_{\lambda,\mu,M^{k}}^{0}=\varnothing\) for \(\varLambda\in (0,\tilde{\varLambda} )\). □

Similarly to the argument of [20], we conclude that for \(\varLambda \in (0,\tilde{\varLambda} )\), \(N_{\lambda,\mu,M^{k}} = N_{\lambda,\mu,M^{k}}^{+} \cup N_{\lambda,\mu,M^{k}}^{-}\) and \(N_{\lambda,\mu,M^{k}}^{\pm} \ne\varnothing\). Denoting

$$\begin{aligned} \alpha_{\lambda,\mu,M^{k}}^{+} = \inf_{ (u,v ) \in N_{\lambda,\mu,M^{k}}^{+}} I_{\lambda,\mu ,M^{k}} (u,v ),\qquad \alpha_{\lambda,\mu,M^{k}}^{-}= \inf _{ (u,v ) \in N_{\lambda,\mu,M^{k}}^{-}} I_{\lambda,\mu ,M^{k}} (u,v ), \end{aligned}$$

we have the following conclusion.

Lemma 2.4

Assume that conditions\((F)\)and\((G)\)hold. If\(\alpha+ \beta< 4\), then

  1. (i)

    \(\alpha_{\lambda,\mu,M^{k}}^{+} < 0 \)for all\(\varLambda \in (0,\tilde{\varLambda} )\);

  2. (ii)

    for some\(D_{0} > 0\), \(\alpha_{\lambda,\mu,M^{k}}^{-} > D_{0} \)for all\(\varLambda\in (0,\frac{ (\alpha+ \beta )q M_{0} \tilde{\varLambda }}{\tilde{C}_{2}} )\).

In particular, for each\(0 < \varLambda< \varLambda_{0} = \min \{ 1,\frac{ (\alpha+ \beta )q M_{0}}{\tilde{C}_{2}} \} \tilde{\varLambda}\), \(\alpha_{\lambda,\mu,M^{k}}^{+} = \inf_{ (u,v ) \in N_{\lambda,\mu,M^{k}}}\)\(I_{\lambda,\mu,M^{k}} (u,v )\).

Proof

(i) For \((u,v ) \in N_{\lambda,\mu ,M^{k}}^{+}\), we prove it in four cases.

Case 1: If \(\Vert u \Vert ^{2} \le k\) and \(\Vert v \Vert ^{2} \le k\), then we obtain that

$$\begin{aligned}& \bigl(a_{1} (\alpha+ \beta-2 ) -b_{1} (4-\alpha- \beta )k \bigr) \Vert u \Vert ^{2} + \bigl( a_{2} (\alpha+ \beta-2 ) -b_{2} (4-\alpha- \beta )k \bigr) \Vert v \Vert ^{2} \\& \quad\le \bigl(a_{1} (\alpha+ \beta-2 ) -b_{1} (4-\alpha - \beta ) \Vert u \Vert ^{2} \bigr) \Vert u \Vert ^{2} \\& \qquad{}+ \bigl( a_{2} (\alpha+ \beta-2 ) -b_{2} (4-\alpha- \beta ) \Vert v \Vert ^{2} \bigr) \Vert v \Vert ^{2} \\& \quad< (\alpha+ \beta-q ) \int_{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr). \end{aligned}$$

Since \((\alpha+ \beta-2 )a_{1} -b_{1} (4-\alpha- \beta )k >0\) and \((\alpha+ \beta-2 ) a_{2} -b_{2} (4-\alpha- \beta )k >0\), we have

$$\begin{aligned} I_{\lambda,\mu,M^{k}} (u,v )= {}&\frac{1}{2}\hat {M}_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr) + \frac{1}{2}\hat {M}_{2}^{k} \bigl( \Vert v \Vert ^{2} \bigr) - \frac{1}{\alpha+ \beta}M_{1}^{k} \bigl( \Vert u \Vert ^{2} \bigr) \Vert u \Vert ^{2} \\ & - \frac{1}{\alpha+ \beta}M_{2}^{k} \bigl( \Vert v \Vert ^{2} \bigr) \Vert v \Vert ^{2} - \frac{\alpha+ \beta-q}{ (\alpha+ \beta ) q} \int_{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr) \\ ={}& \frac{\alpha+ \beta-2}{2 (\alpha+ \beta )} \bigl(a_{1} \Vert u \Vert ^{2} + a_{2} \Vert v \Vert ^{2} \bigr) + \frac{\alpha+ \beta-4}{4 (\alpha+ \beta )} \bigl(b_{1} \Vert u \Vert ^{4} + b_{2} \Vert v \Vert ^{4} \bigr) \\ &- \frac{\alpha+ \beta-q}{q (\alpha+ \beta )} \int _{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr) \\ < {}& \biggl[\frac{\alpha+ \beta-2}{2 (\alpha+ \beta )}a_{1} + \frac{\alpha+ \beta-4}{4 (\alpha+ \beta )} b_{1}k - \frac{a_{1} (\alpha+ \beta-2 )-b_{1} (4-\alpha- \beta )k}{q (\alpha+ \beta )} \biggr] \Vert u \Vert ^{2} \\ &+ \biggl[\frac{\alpha+ \beta-2}{2 (\alpha+ \beta )}a_{2} + \frac{\alpha+ \beta-4}{4 (\alpha+ \beta )} b_{2}k - \frac{a_{2} (\alpha+ \beta-2 )-b_{2} (4-\alpha- \beta )k}{q (\alpha+ \beta )} \biggr] \Vert v \Vert ^{2} \\ < {}&0. \end{aligned}$$

Case 2: If \(\Vert u \Vert ^{2} \le k\) and \(\Vert v \Vert ^{2} > k\), then we get

$$\begin{aligned}& \bigl(a_{1} (\alpha+ \beta-2 ) -b_{1} (4-\alpha- \beta )k \bigr) \Vert u \Vert ^{2} + M_{2} (k ) (\alpha+ \beta-2 ) \Vert v \Vert ^{2} \\& \quad< (\alpha+ \beta-q ) \int_{\varOmega} \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr). \end{aligned}$$

Therefore

$$\begin{aligned}& I_{\lambda,\mu,M^{k}} (u,v ) \\& \quad= \frac{\alpha+ \beta-2}{2 (\alpha+ \beta )}a_{1} \Vert u \Vert ^{2} + \frac{\alpha+ \beta-4}{4 (\alpha+ \beta )}b_{1} \Vert u \Vert ^{4} + \frac{1}{2} \bigl(\hat {M}_{2} (k )- M_{2} (k )k \bigr) \\& \quad\quad{} + \frac{M_{2} (k ) (\alpha+ \beta-2 )}{2 (\alpha+ \beta )} \Vert v \Vert ^{2}- \frac {\alpha+ \beta-q}{q (\alpha+ \beta )} \int_{\varOmega } \bigl(\lambda f \vert u \vert ^{q} + \mu g \vert v \vert ^{q} \bigr) \\& \quad< \biggl(\frac{\alpha+ \beta-2}{2 (\alpha+ \beta )}a_{1} + \frac{\alpha+ \beta-4}{4 (\alpha+ \beta )} b_{1}k - \frac{a_{1} (\alpha+ \beta-2 )-b_{1} (4-\alpha- \beta )k}{q (\alpha+ \beta )} \biggr) \Vert u \Vert ^{2} \\& \quad\quad{}+ \frac{1}{2} \biggl(a_{2}k + \frac{b_{2}k^{2}}{2} - (a_{2}+b_{2}k )k \biggr)\\& \qquad{} + \frac{M_{2} (k ) (\alpha+ \beta-2 )}{2 (\alpha+ \beta )} \Vert v \Vert ^{2} - \frac{M_{2} (k ) (\alpha+ \beta -2 )}{q (\alpha+ \beta )} \Vert v \Vert ^{2} \\& \quad< 0. \end{aligned}$$

Case 3: If \(\Vert u \Vert ^{2} > k\) and \(\Vert v \Vert ^{2} \le k\), then we have that

$$\begin{aligned}& I_{\lambda,\mu,M^{k}} (u,v ) \\& \quad< \frac {1}{2} \biggl(a_{1}k + \frac{b_{1}k^{2}}{2} - (a_{1}+b_{1}k )k \biggr) + \frac{M_{1} (k ) (\alpha+ \beta-2 )}{2 (\alpha+ \beta )} \Vert u \Vert ^{2} - \frac {M_{1} (k ) (\alpha+ \beta-2 )}{q (\alpha + \beta )} \Vert u \Vert ^{2} \\& \quad\quad{}+ \biggl(\frac{\alpha+ \beta-2}{2 (\alpha+ \beta )}a_{2} + \frac{\alpha+ \beta-4}{4 (\alpha+ \beta )} b_{2}k - \frac{a_{2} (\alpha+ \beta-2 )-b_{2} (4-\alpha- \beta )k}{q (\alpha+ \beta )} \biggr) \Vert v \Vert ^{2} \\& \quad< 0. \end{aligned}$$

Case 4: If \(\Vert u \Vert ^{2} > k\) and \(\Vert v \Vert ^{2} > k\), then we deduce that

$$\begin{aligned}& I_{\lambda,\mu,M^{k}} (u,v ) \\& \quad< -\frac{b_{1}}{4}k^{2} + \frac{M_{1} (k ) (\alpha+ \beta-2 ) (q-2 )}{22^{\ast}q} \Vert u \Vert ^{2} + \frac{M_{2} (k ) (\alpha+ \beta-2 ) (q-2 )}{2 (\alpha+ \beta )q} \Vert v \Vert ^{2} - \frac{b_{2}}{4}k^{2} \\& \quad< 0. \end{aligned}$$

Therefore \(\alpha_{\lambda,\mu,M^{k}}^{+} = \inf_{ (u,v ) \in N_{\lambda,\mu,M^{k}}^{+}} I_{\lambda,\mu,M^{k}} (u,v ) <0\).

(ii) For \((u,v ) \in N_{\lambda,\mu,M^{k}}^{-}\), by (2.4)–(2.7) we have

$$\begin{aligned} \tilde{C}_{1} (2-q ) \bigl\Vert (u,v ) \bigr\Vert _{H}^{2}& < (\alpha+ \beta-q ) \int_{\varOmega} \vert u \vert ^{\alpha} \vert v \vert ^{\beta} \\ &\le (\alpha+ \beta-q ) S_{\alpha,\beta}^{- (\alpha+ \beta )/2} \bigl\Vert (u,v ) \bigr\Vert _{H}^{\alpha+ \beta}, \end{aligned}$$

which implies that

$$\bigl\Vert (u,v ) \bigr\Vert _{H} > \biggl(\frac{\tilde {C}_{1} (2-q )S_{\alpha,\beta}^{ (\alpha+ \beta )/2}}{\alpha+ \beta-q} \biggr)^{1/ (\alpha+ \beta -2 )}. $$

From (2.3) we get that

$$\begin{aligned}& I_{\lambda,\mu,M^{k}} (u,v ) \\& \quad\ge M_{0} \bigl\Vert (u,v ) \bigr\Vert _{H}^{2} - \frac{\alpha+ \beta-q}{ (\alpha+ \beta )q}S_{q}^{-q/2}\varLambda \bigl\Vert (u,v ) \bigr\Vert _{H}^{q} \\& \quad> \biggl(\frac{\tilde{C}_{1} (2-q )S_{\alpha,\beta }^{ (\alpha+ \beta )/2}}{\alpha+ \beta-q} \biggr)^{q/ (\alpha+ \beta-2 )} \\& \quad\quad{}\cdot \biggl( M_{0} \biggl(\frac{\tilde{C}_{1} (2-q )S_{\alpha,\beta}^{ (\alpha+ \beta )/2}}{\alpha+ \beta -q} \biggr)^{ (2-q )/ (\alpha+ \beta-2 )} - \frac{\alpha+ \beta-q}{ (\alpha+ \beta )q}S_{q}^{-q/2}\varLambda \biggr). \end{aligned}$$

Thus, if \(\varLambda< (\alpha+ \beta )q M_{0} \frac{\tilde {\varLambda}}{\tilde{C}_{2}}\), then \(\alpha_{\lambda,\mu,M^{k}}^{-} > D_{2}\) for some \(D_{2} >0\). □

3 The (PS) condition

Lemma 3.1

Every bounded Palais–Smale sequence for\(I_{\lambda,\mu,M^{k}}\)onHhas a strongly convergent subsequence.

Proof

Let \(\{ (u_{n},v_{n} ) \}\) be a bounded Palais–Smale sequence for \(I_{\lambda,\mu,M^{k}}\) on H. Then the sequence \(\{u_{n} \}\) (\(\{v_{n} \} \)) is bounded on \(H_{0}^{1}(\varOmega)\). Thus there exist a subsequence \(\{u_{n} \}\) (\(\{v_{n} \} \)) and \(u_{0}\in H_{0}^{1}(\varOmega)\) (\(v_{0}\in H_{0}^{1}(\varOmega ) \)) such that

$$\begin{gathered} u_{n} \rightharpoonup u_{0} \text{ weakly in }H_{0}^{1}(\varOmega) \quad\bigl(v_{n} \rightharpoonup v_{0} \text{ weakly in }H_{0}^{1}(\varOmega ) \bigr), \\ u_{n} \rightarrow u_{0} \text{ strongly in }L^{r}(\varOmega) \quad\bigl(v_{n} \rightarrow v_{0} \text{ strongly in }L^{r}(\varOmega) \bigr) \quad\text{for }1< r< 2^{\ast}.\end{gathered} $$

Then

$$\begin{aligned}& \begin{aligned}[b] \biggl\vert \int_{\varOmega} f \vert u_{n} \vert ^{q-2}u_{n} (u_{n}-u_{0} )\, dx \biggr\vert & \le \Vert f \Vert _{\infty} \biggl( \int_{\varOmega} \vert u_{n} \vert ^{q} \,dx \biggr)^{{(q-1)}/{q}} \biggl( \int_{\varOmega} \vert u_{n}-u_{0} \vert ^{q}\, dx \biggr)^{{1}/{q}}\\&\rightarrow0, \end{aligned} \end{aligned}$$
(3.1)
$$\begin{aligned}& \begin{aligned}[b] \biggl\vert \int_{\varOmega} g \vert v_{n} \vert ^{q-2}v_{n} (v_{n}-v_{0} )\, dx \biggr\vert &\le \Vert g \Vert _{\infty} \biggl( \int_{\varOmega} \vert v_{n} \vert ^{q} \,dx \biggr)^{{(q-1)}/{q}} \biggl( \int_{\varOmega} \vert v_{n}-v_{0} \vert ^{q}\, dx \biggr)^{{1}/{q}}\\&\rightarrow0, \end{aligned} \end{aligned}$$
(3.2)
$$\begin{aligned}& \begin{aligned}[b] \biggl\vert \int_{\varOmega} \vert u_{n} \vert ^{\alpha-2}u_{n} (u_{n}-u_{0} ) \vert v_{n} \vert ^{\beta } \,dx \biggr\vert &\le \biggl( \int_{\varOmega} \vert u_{n} \vert ^{\alpha+ \beta }\,dx \biggr)^{{ (\alpha-1 )}/{ (\alpha+ \beta )}} \\ &\quad\cdot\biggl( \int_{\varOmega} \vert u_{n}-u_{0} \vert ^{\alpha+ \beta} \,dx \biggr)^{{1}/{ (\alpha+ \beta )}} \\ &\quad \cdot \biggl( \int_{\varOmega} \vert v_{n} \vert ^{\alpha+ \beta}\, dx \biggr)^{{\beta}/{ (\alpha+ \beta )}}\\&\rightarrow0, \end{aligned} \end{aligned}$$
(3.3)

and

$$\begin{aligned}& \begin{aligned}[b] \biggl\vert \int_{\varOmega} \vert u_{n} \vert ^{\alpha} \vert v_{n} \vert ^{\beta-2}v_{n} (v_{n}-v_{0} )\, dx \biggr\vert & \le \biggl( \int_{\varOmega} \vert u_{n} \vert ^{\alpha+ \beta} \,dx \biggr)^{{\alpha}/{ (\alpha+ \beta )}} \biggl( \int _{\varOmega} \vert v_{n}-v_{0} \vert ^{\alpha+ \beta} \,dx \biggr)^{{1}/{ (\alpha+ \beta )}}\hspace{-24pt} \\ &\quad \cdot \biggl( \int_{\varOmega} \vert v_{n} \vert ^{\alpha+ \beta} \,dx \biggr)^{{ (\beta-1 )}/{ (\alpha+ \beta )}}\\&\rightarrow0 \end{aligned} \end{aligned}$$
(3.4)

as \(n \rightarrow\infty\). Since \(\{ (u_{n},v_{n} ) \}\) is a Palais–Smale sequence for \(I_{\lambda,\mu,M^{k}}\), it follows that

$$\begin{aligned}& \bigl\langle I'_{\lambda,\mu,M^{k}} (u_{n},v_{n} ), (u_{n}-u_{0},0 ) \bigr\rangle \\& \quad= {M}_{1}^{k} \bigl( \Vert u_{n} \Vert ^{2} \bigr) \int_{\varOmega }\nabla u_{n}\nabla (u_{n} - u_{0} )\, dx -\lambda \int _{\varOmega} f \vert u_{n} \vert ^{q-2}u_{n} (u_{n}-u_{0} )\, dx \\& \qquad{} - \frac{\alpha}{4} \int_{\varOmega} \vert u_{n} \vert ^{\alpha -2}u_{n} (u_{n}-u_{0} ) \vert v_{n} \vert ^{\beta} \,dx \\& \quad\rightarrow0 \end{aligned}$$
(3.5)

and

$$\begin{aligned}& \bigl\langle I'_{\lambda,\mu,M^{k}} (u_{n},v_{n} ), (0, v_{n}-v_{0} ) \bigr\rangle \\& \quad= {M}_{2}^{k} \bigl( \Vert v_{n} \Vert ^{2} \bigr) \int_{\varOmega }\nabla v_{n}\nabla (v_{n} - v_{0} )\, dx-\mu \int_{\varOmega} g \vert v_{n} \vert ^{q-2}v_{n} (v_{n}-v_{0} )\, dx \\& \qquad{} - \frac{\beta}{4} \int_{\varOmega} \vert u_{n} \vert ^{\alpha } \vert v_{n} \vert ^{\beta-2}v_{n} (v_{n}-v_{0} ) \,dx \\& \quad\rightarrow0 \end{aligned}$$
(3.6)

as \(n \rightarrow\infty\). By (3.1), (3.3), and (3.5) we get that

$$\int_{\varOmega}\nabla u_{n}\nabla (u_{n} - u_{0} )\, dx \rightarrow0\quad\text{as }n \rightarrow\infty. $$

By (3.2), (3.4), and (3.6) we obtain that

$$\int_{\varOmega}\nabla v_{n}\nabla (v_{n} - v_{0} ) \,dx \rightarrow0\quad\text{as }n \rightarrow\infty. $$

Thus

$$\bigl\Vert (u_{n},v_{n} )- (u_{0},v_{0} ) \bigr\Vert _{H}^{2} = \Vert u_{n}-u_{0} \Vert ^{2} + \Vert v_{n}-v_{0} \Vert ^{2} \rightarrow0\quad\text{as }n \rightarrow\infty. $$

 □

Proof of Theorem 1.1

Take \(\varLambda< \varLambda_{0}\). By Lemma 2.1 and the Ekeland variational principle [23] there exist two bounded minimizing sequences \(\{ (u_{n}^{\pm },v_{n}^{\pm} ) \}\) for \(I_{\lambda,\mu,M^{k}}\) on \(N_{\lambda,\mu,M^{k}}^{\pm}\) such that

$$I_{\lambda,\mu,M^{k}} \bigl(u_{n}^{\pm},v_{n}^{\pm} \bigr) = \alpha_{\lambda,\mu,M^{k}}^{\pm} + o(1),\qquad I'_{\lambda,\mu ,M^{k}} \bigl(u_{n}^{\pm},v_{n}^{\pm} \bigr) = o(1)\quad\text{on }H^{-1}. $$

By Lemma 3.1 there exist subsequences \(\{ (u_{n}^{\pm },v_{n}^{\pm} ) \}\) and \((u_{\lambda,\mu ,M^{k}}^{\pm},v_{\lambda,\mu,M^{k}}^{\pm} ) \in H\), the nonzero solutions of the equation (\(E_{\lambda ,\mu ,M^{k}}\)), such that \((u_{n}^{\pm},v_{n}^{\pm} ) \rightarrow (u_{\lambda,\mu,M^{k}}^{\pm},v_{\lambda,\mu,M^{k}}^{\pm} )\) strongly in H. So \((u_{\lambda,\mu,M^{k}}^{\pm},v_{\lambda ,\mu,M^{k}}^{\pm} )\in N_{\lambda,\mu,M^{k}}^{\pm}\) and \(I_{\lambda,\mu,M^{k}} (u_{\lambda,\mu,M^{k}}^{\pm },v_{\lambda,\mu,M^{k}}^{\pm} ) = \alpha_{\lambda,\mu ,M^{k}}^{\pm}\). Since \({I_{\lambda,\mu,M^{k}}} (u_{\lambda,\mu ,M^{k}}^{\pm}, v_{\lambda,\mu,M^{k}}^{\pm} ) ={I_{\lambda,\mu,M^{k}}} ( \vert u_{\lambda,\mu ,M^{k}}^{\pm} \vert , \vert v_{\lambda,\mu,M^{k}}^{\pm} \vert )\), by Lemma 2.2 and Lemma 2.4 we could obtain that \((u_{\lambda,\mu,M^{k}}^{+}, v_{\lambda,\mu,M^{k}}^{+} ) \) and \((u_{\lambda,\mu ,M^{k}}^{-},v_{\lambda,\mu,M^{k}}^{-} ) \) are two distinct solutions of equation (\(E_{\lambda ,\mu ,M^{k}}\)) such that \(u_{\lambda,\mu,M^{k}}^{\pm} \ge0\) and \(v_{\lambda,\mu ,M^{k}}^{\pm} \ge0\) in Ω. By an argument similar to Lemma 2.6 and Theorem 3.2 in [15] we get \(u_{\lambda,\mu ,M^{k}}^{\pm} \ne0\) and \(v_{\lambda,\mu,M^{k}}^{\pm} \ne0\). By the strong maximum principle [24] we get that \(u_{\lambda,\mu ,M^{k}}^{\pm} > 0\) and \(v_{\lambda,\mu,M^{k}}^{\pm} > 0\). □