Multiplicity of nontrivial solutions for a class of superquadratic elliptic systems near resonance

Abstract

Multiple solutions for a class of superquadratic elliptic systems near resonance with high eigenvalues are obtained by using the nabla theorem due to Marino and Saccon in (Topol. Methods Nonlinear Anal. 17:213-237, 2001) and the linking theorem.

1 Introduction and main results

Consider the existence of weak solutions for the following elliptic system:

$$ \textstyle\begin{cases} -\Delta u=au+bv+F_{v}(x,u,v)& \mbox{in } \Omega, \\ -\Delta v=bu+av+F_{u}(x,u,v) & \mbox{in } \Omega, \\ u=v=0 &\mbox{on } \partial\Omega, \end{cases} $$

(1)

where $\Omega\subset R^{N}$ ($N\geq3$) is a bounded, connected open domain and a, b are real numbers. The nonlinearity $F\in C(\bar{\Omega}\times R^{2}, R)$ has continuous derivatives $F_{s}(x,s,t)$, $F_{t}(x,s,t)$ with respect to s and t for any $x\in\Omega$ and satisfies the following superquadratic condition:

$$ \frac{F(x,s,t)}{|s|^{2}+|t|^{2}}\to+\infty\quad \mbox{as }|s|+|t|\to\infty \mbox{ uniformly in }x\in\Omega. $$

(2)

Denote by $H^{1}_{0}(\Omega)$ the usual Hilbert space with the inner product and the norm

$$\langle u,v\rangle= \int_{\Omega}\nabla u\nabla v\,dx, \qquad \|u\|= \biggl( \int _{\Omega}|\nabla u|^{2}\,dx \biggr)^{1/2} $$

for any $u,v\in H^{1}_{0}(\Omega)$. By the Sobolev embedding theorem, the embedding $H^{1}_{0}(\Omega)\hookrightarrow L^{p}(\Omega)$ is continuous for any $1\leq p\leq2^{*}$, and there exists a positive constant $C=C(p,N,\Omega)$ such that

$$ \|u\|_{p}\leq C\|u\| \quad \mbox{for all }u\in H^{1}_{0}(\Omega), $$

(3)

where $2^{*}:=\frac{2N}{N-2}$ is the Sobolev critical exponent and $\| \cdot\|_{p}$ denotes the norm of $L^{p}(\Omega)$. Moreover, the embedding $H^{1}_{0}(\Omega)\hookrightarrow L^{p}(\Omega)$ is compact for any $1\leq p<2^{*}$. Let $H=H^{1}_{0}(\Omega)\times H^{1}_{0}(\Omega)$ be a Hilbert space with the norm

$$\bigl\Vert (u,v) \bigr\Vert =\bigl( \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}\bigr)^{1/2}\quad \mbox{for any }(u,v)\in H. $$

The spectrum of −Δ in $H^{1}_{0}(\Omega)$ is at most a countable set, which we denote by

$$0< \lambda_{1}< \lambda_{2}< \lambda_{3}< \cdots< \lambda_{k}< \cdots, $$

where each $\lambda_{k}$ is an isolated eigenvalue of finite multiplicity of $m_{k}$, with $\varphi_{k}$ ($k=1, 2,\ldots$) the corresponding eigenfunctions, which will be taken orthogonal and normalized with $\|\varphi_{k}\|=1$. It is well known that $\lambda_{1}$ is simple and isolated and $\varphi_{1}$ may be taken positive on Ω.

Define $B:H\times H\to R$ as follows:

$$\begin{aligned} B\bigl((u,v),(\phi,\psi)\bigr) =& \int_{\Omega}\nabla u\nabla\psi \,dx+ \int_{\Omega}\nabla v\nabla\phi \,dx \\ &{}-a \int_{\Omega}(u\psi+v\phi)\,dx -b \int_{\Omega}(u\phi+v\psi)\,dx. \end{aligned}$$

As in [2], we define an orthogonal basis for H which diagonalizes B. It follows from [2] that there exist two sequences of eigenvalues for the following eigenvalue problem:

$$B\bigl((u,v),(\phi,\psi)\bigr)=\mu\bigl\langle (u,v),(\phi,\psi)\bigr\rangle , \quad \forall(\phi ,\psi)\in H, $$

that is,

$$\mu_{i}=\frac{\lambda_{i}-(a+b)}{\lambda_{i}}\quad \mbox{and} \quad \mu_{-i}= \frac {(a-b)-\lambda_{i}}{\lambda_{i}} ,\quad i=1,2,\ldots, $$

with the corresponding normalized eigenfunctions

$$\psi_{\pm i}=\frac{(\varphi_{i}, \pm\varphi_{i})}{\sqrt{2}}, \quad i=1,2,\ldots. $$

Moreover, $\{\mu_{\pm i}\}$ and $\{\psi_{\pm i}\}$ have the following simple properties:

• (1) $\lim_{i\to+\infty}\mu_{\pm i}=\pm1$;

• (2) If $a+b>0$, $\mu_{i}=\frac{\lambda_{i}-(a+b)}{\lambda_{i}}$ converges to 1 from left, but if $a+b<0$, $\mu_{i}=\frac{\lambda_{i}-(a+b)}{\lambda_{i}}$ converges to 1 from right;

• (3) If $a-b>0$, $\mu_{-i}=\frac{(a-b)-\lambda_{i}}{\lambda_{i}}$ converges to −1 from right, but if $a-b<0$, $\mu_{-i}=\frac{(a-b)-\lambda_{i}}{\lambda _{i}}$ converges to −1 from left.

• (4) Let ${\mathbf{u}}:=(u,v)=\sum_{i\in Z\setminus\{0\}}c_{i}\psi_{i}\in H$, then

  $$\|{\mathbf{u}}\|^{2}=\sum_{i\in Z\setminus\{0\}}c_{i}^{2}, \qquad B({\mathbf {u}},{\mathbf{u}})=\sum_{i\in Z\setminus\{0\}} \mu_{i}c_{i}^{2},\qquad \|{\mathbf {u}} \|^{2}=\sum_{i\in Z\setminus\{0\}} \lambda^{-1}_{|i|}c_{i}^{2}. $$

Multiplicity of nontrivial solutions for elliptic equations or elliptic systems near resonance was considered by many authors (see [1–15] and the references therein). For the following two point boundary value problem

$$-u''-\lambda u=f(x,u)+h(x), \qquad u(0)=u(\pi)=0, $$

if f is bounded and satisfies a sign condition, Mawhin and Schmitt [3] obtained the existence of at least three nontrivial solutions under the condition that the parameter λ is sufficiently close to $\lambda_{1}$ from left by using Ekeland’s variational principle and the mountain pass theorem, where $\lambda_{1}$ is the first eigenvalue of the corresponding linear problem. For the semilinear elliptic equation

$$ \textstyle\begin{cases} -\Delta u=\lambda u+f(x,u)+h(x)&\mbox{in }\Omega, \\ u=0 & \mbox{on } \partial\Omega, \end{cases} $$

(4)

if f is sublinear at infinity and the parameter λ approaches $\lambda_{1}$ from the left, Ma et al. [4] proved the existence of at least three nontrivial solutions. Similar results were obtained for the quasilinear problem in [4] and [14], for the $(p,q)$ elliptic systems in [6].

Many results were obtained for the case that the parameter λ approaches $\lambda_{k}$ ($k\geq2$) from left (see [1, 7–15] and the references therein). de Paiva and Massa [7] obtained the existence of two solutions for problem (4) under the condition that f is sublinear at infinity by using saddle point theory. In [2], Massa and Rossato obtained the existence of at least two solutions for problem (1) by using of the saddle point theorem and Galerkin approximation, where $F_{s}(x,s,t)=F_{s}(x,s)$ and $F_{t}(x,s,t)=F_{t}(x,t)$ are the sublinear nonlinearities. In [8], under a generalized Landesman-Lazer type condition, Ke and Tang obtained the existence of at least two solutions for a class of asymptotically linear noncooperative elliptic systems by using the saddle point theorem. On the other hand, if $F(x,t)=\int_{0}^{t}f(x,s)\,ds$ satisfies the $(AR)$ condition, Mugnai [9] obtained three nontrivial solutions for problem (4) by using the ∇-theorem due to Marino and Saccon in [10], and a similar result was obtained in [15] under the following superquadratic condition:

$$\frac{F(x,t)}{|t|^{2}}\to+\infty \quad \mbox{as }|t|\to\infty \mbox{ uniformly in }x \in\Omega. $$

The ∇-theorem was widely applied to consider the multiplicity of nontrivial solutions for several other equations (see [11–14]). In [1], by using the notion of limit relative category, Marino and Saccon developed a ‘limit’ version of the ∇-theorems and found multiple solutions for some noncooperative elliptic systems where the nonlinearity satisfies the global $(AR)$ condition.

In this paper, influenced by [1] and [2], we consider the existence of multiple solutions for problem (1) near resonance at the higher eigenvalue, where $F(x,s,t)$ satisfies the superquadratic condition (2). Now we state some other conditions on $F(x,s,t)$.

• (F1) There are $c_{1}>0$, $1< r<(N+2)/(N-2)$ such that

  $$\bigl\vert F_{s}(x,s,t) \bigr\vert + \bigl\vert F_{t}(x,s,t) \bigr\vert \leq c_{1}\bigl(1+ \vert s \vert ^{r}+ \vert t \vert ^{r}\bigr),\quad \forall(x,s,t)\in \Omega\times R^{2}; $$

• (F2) $F(x,s,t)=o(|s|^{2}+|t|^{2})$ as $|s|+|t|\to0$ uniformly in $x\in\Omega$;

• (F3) There exist $N(r-1)/2<\beta<2^{*}$, $c_{2}>0$ and $L>0$ such that

  $$\begin{aligned}& F_{s}(x,s,t)s+F_{t}(x,s,t)t-2F(x,s,t)>0, \quad \forall x\in \Omega, (s,t)\neq (0,0),\quad \mbox{and} \\& F_{s}(x,s,t)s+F_{t}(x,s,t)t-2F(x,s,t)\geq c_{2} \bigl(|s|^{\beta}+|t|^{\beta}\bigr),\quad \forall x\in\Omega, |s|^{2}+|t|^{2}\geq L; \end{aligned}$$

• (F4) $F(x,s,t)\geq0$, ∀ $(x,s,t)\in\Omega\times R^{2}$.

The main results of the paper are the following theorems.

Theorem 1

Assume that (2) and (F1), (F2), (F3), (F4) hold. Let $\lambda_{k}$ be an eigenvalue of −Δ and $\lambda _{l}$ be the first eigenvalue above $a-b$. $\forall \delta>0$, $\exists \varepsilon_{0}>0$, if $\operatorname{dist}(a-b,\sigma(-\Delta))>\delta$ and $a+b\in (\lambda_{k}-\varepsilon_{0},\lambda_{k})$, then problem (1) has at least three nontrivial solutions.

Theorem 2

Assume that (2) and (F1), (F2), (F3), (F4) hold. Let $\lambda_{k}$ be an eigenvalue of −Δ and $\lambda _{l}$ be the first eigenvalue above $a+b$. $\forall\delta>0$, $\exists \varepsilon_{1}>0$, if $\operatorname{dist}(a+b,\sigma(-\Delta))>\delta$ and $a-b\in (\lambda_{k},\lambda_{k}+\varepsilon_{1})$, then problem (1) has at least three nontrivial solutions.

2 Proofs of theorems

In this section, we first recall two compactness conditions, i.e., the $(PS)^{*}_{c}$ condition, which was introduced by Li and Willem in [16], and the $(Ce)^{*}_{c}$ condition, which was introduced by Luan and Mao in [17] where they extended the local linking theorem of [16]. The $(Ce)^{*}_{c}$ condition is a version of the $(PS)^{*}_{c}$ condition and a generalization of the $(Ce)$ condition of [18], and it can ensure a deformation lemma. And then, we recall the ∇−theorem due to Marino and Saccon in [1], where the $(PS)^{*}_{c}$ condition is replaced by the $(Ce)^{*}_{c}$ condition.

Let W be a real Banach space with $W=W^{1}\oplus W^{2}$. $W^{i}_{0}\subset W^{i}_{1}\subset W^{i}_{2}\subset\cdots\subset W^{i}$ and $\bigcup_{n\in N}W^{i}_{n}$ is dense in $W^{i}$, $i=1,2$. For every multi-index $\alpha=(\alpha _{1},\alpha_{2})\in N^{2}$, $W_{\alpha}=W^{1}_{\alpha_{1}}\oplus W^{2}_{\alpha_{2}}$, and it is well known that $\alpha\leq\beta\Longleftrightarrow\alpha _{1}\leq\beta_{1}$, $\alpha_{2}\leq\beta_{2}$. A sequence $\{\alpha_{n}\}\in N^{2}$ is admissible if for every $\alpha\in N^{2}$, there is $m\in N$ such that $n\geq m\Longrightarrow\alpha_{n}\geq\alpha$.

Definition 1

see [17]

The functional $I\in C^{1}(W,R)$ satisfies the $(Ce)^{*}_{c}$ condition at the level $c\in R$, if $\{\alpha _{n}\}$ is admissible and $\{u_{\alpha_{n}}\}$ satisfies

$$u_{\alpha_{n}}\in W_{\alpha_{n}} \quad \mbox{and}\quad I_{\alpha_{n}}(u_{\alpha_{n}}) \to c,\qquad \bigl(1+ \Vert u_{\alpha_{n}} \Vert \bigr)I_{\alpha_{n}}'(u_{\alpha_{n}}) \to0\quad \mbox{as } n\to\infty, $$

$\{u_{\alpha_{n}}\}$ has a subsequence which converges to a critical point of I, where $I_{\alpha_{n}}=I|_{W_{\alpha_{n}}}$.

Let E be a real Hilbert space with $E=E^{1}\oplus E^{2}\oplus E^{3}$ and $\dim E^{2}\geq1$. Let $\{E_{n}\}$ be a sequence of finite dimension closed subspaces of E such that $E^{2}\subset E_{n}$, $E_{n}\subset E_{n+1}$, and let $\bigcup_{n\in N}E_{n}$ be dense in E. Let X be a closed subspace of E and $P_{X}$ denote the orthogonal projection onto X, and for any $u\in E$, let $[u]=\{tu:t\in R\}$.

Definition 2

see [1]

Let X be a closed subspace of E such that $P_{X}P_{E_{n}}=P_{E_{n}}P_{X}$ for all n, and let c be a real number. The functional $I\in C^{1}(E,R)$ satisfies the condition $\nabla^{*}(X,c)$ with respect to $(E_{n})_{n}$ if there exists $\gamma>0$ such that

$$\liminf_{n\to\infty}\bigl\{ \bigl\Vert P_{E_{n}}P_{X\oplus[u]} \operatorname{grad}I(u) \bigr\Vert :u\in E_{n}, \operatorname{dist}(u,X)< \gamma, \bigl\vert I(u)-c \bigr\vert < \gamma\bigr\} >0. $$

The condition $\nabla^{*}(X,c)$ implies that there are no critical points u in X with $I(u)=c$, with some uniformity. Moreover, the condition $\nabla^{*}(X,c)$ is equivalent to the following conditions:

• (a) $I|_{X}$ has no critical point u in X with $I(u)=c$;

• (b) If $(h_{n})$ is a sequence in N such that $h_{n}\to\infty$, $\{u_{n}\}$ is a sequence in E such that $u_{n}\in E_{h_{n}}$ for all n, $\operatorname{dist}(u_{n}, X)\to0$, $I(u_{n})\to c$ and $P_{E_{n}}P_{X\oplus[u_{n}]}\operatorname{grad}I(u_{n})\to0$ as $n\to\infty$, then $(u_{n})$ has a convergent subsequence.

Theorem 3

see [1]

Let E be a Hilbert space and $E^{i}$, $i=1,2,3$, be three subspaces of E such that $E=E^{1}\oplus E^{2}\oplus E^{3}$ with $1\leq\dim E^{2}<+\infty$, and $I\in C^{1,1}(E,R)$. Define

$$\begin{aligned}& T=\bigl\{ u\in E^{1}\oplus E^{2}: \Vert P_{E^{1}}u \Vert \leq R, R'\leq \Vert P_{E^{2}}u \Vert \leq R''\bigr\} \quad \textit{and} \quad \Gamma= \partial_{E^{1}\oplus E^{2}}T, \\& S_{\rho}=\bigl\{ u\in E^{2}\oplus E^{3}: \Vert u \Vert =\rho\bigr\} \quad \textit{and}\quad B_{\rho}=\bigl\{ u\in E^{2}\oplus E^{3}: \Vert u \Vert \leq\rho\bigr\} , \end{aligned}$$

where $R>0$ and $0\leq R'<\rho<R''$. Especially, if $R'=0$, we have

$$\begin{aligned} \Gamma =&\bigl\{ u\in E^{1}: \Vert u \Vert \leq R\bigr\} \cup\bigl\{ u\in E^{1}\oplus E^{2}: \Vert P_{E^{1}}u \Vert \leq R, \Vert P_{E^{2}}u \Vert =R''\bigr\} \\ &{} \cup\bigl\{ u\in E^{1}\oplus E^{2}: \Vert P_{E^{1}}u \Vert =R, \Vert P_{E^{2}}u \Vert \leq R''\bigr\} . \end{aligned}$$

Assume that

$$\sup I(\Gamma)< \inf I(S_{\rho})=\alpha_{0} \quad \textit{and} \quad \beta_{0}=\sup I(T)< +\infty. $$

If the condition $\nabla^{*}(E^{1}\oplus E^{3},c)$ and the $(Ce)^{*}_{c}$ condition hold at any $c\in[\alpha_{0}, \beta_{0}]$, then I has at least two critical points in $I^{-1}([\alpha_{0},\beta_{0}])$.

Define the functional $J: H\to R$ as follows:

$$\begin{aligned} J(u,v) =& \int_{\Omega}\nabla u\nabla v\,dx -a \int_{\Omega}uv\, dx-\frac{b}{2} \int_{\Omega}\bigl(u^{2}+v^{2}\bigr)\,dx- \int_{\Omega}F(x,u,v)\,dx \\ =&\frac{1}{2}B\bigl((u,v),(u,v)\bigr)- \int_{\Omega}F(x,u,v)\,dx. \end{aligned}$$

(5)

From the subcritical growth condition (F1), it is easy to verify that $J\in C^{1}(H,R)$ and

$$\begin{aligned} \bigl\langle J'(u,v),(\phi,\psi)\bigr\rangle =& \int_{\Omega}\nabla\phi\nabla v\,dx -a \int_{\Omega}\phi v\,dx-b \int_{\Omega}u\phi \,dx- \int_{\Omega}F_{u}(x,u,v)\phi \,dx \\ &{}+ \int_{\Omega}\nabla u\nabla\psi \,dx -a \int_{\Omega}u\psi \,dx-b \int_{\Omega}v\psi \,dx - \int_{\Omega}F_{v}(x,u,v)\psi \,dx \end{aligned}$$

for all $(\phi,\psi)\in H$. It is well known that finding weak solutions of problem (1) is equivalent to finding critical points of the functional J in H. Since Theorem 2 is similar to Theorem 1, we will be devoted to the proof of Theorem 1. Two solutions are obtained by Theorem 3, and the existence of the third solution is proved by the classical linking theorem (see [19, 20]) and Galerkin approximation. Let $\lambda_{l}$ be the first eigenvalue above $a-b$ such that dist$(a-b,\sigma(-\Delta))>\delta$ for given $\delta>0$ and $a+b\in (\lambda_{k-1},\lambda_{k})$. From the definitions of $\mu_{\pm i}$, we obtain that

$$\mu_{i}< 0,\quad i=1,2,\ldots,k-1\quad \mbox{and}\quad \mu_{i}>0,\quad i=k,k+1,\ldots, $$

and

$$\mu_{-i}>0,\quad i=1,2,\ldots,l-1 \quad \mbox{and}\quad \mu_{-i}< 0,\quad i=l,l+1,\ldots. $$

Now, define

$$H^{-}=\overline{\operatorname{span}\{\psi_{i}:\mu_{i}< 0,i= \pm1,\pm2,\ldots\}},\qquad H^{+}=\overline{\operatorname{span}\{\psi_{i}: \mu_{i}>0,i=\pm1,\pm2,\ldots\}}, $$

then $H=H^{-}\oplus H^{+}$. Especially, set

$$Y=\operatorname{span}\{\psi_{i}:\mu_{i}=\mu_{k} \},\qquad Y^{\bot}=\overline{\operatorname{span}\{\psi _{i}: \mu_{i}>0 \mbox{ and } \mu_{i} \neq\mu_{k}\}}, $$

so that $\dim Y<\infty$ and $H^{+}=Y\oplus Y^{\bot}$. Moreover, for any ${\mathbf{u}}={\mathbf{u}^{-}}\oplus{\mathbf{u}^{+}}\in H^{-}\oplus H^{+}$,

$$ B(\mathbf{u},\mathbf{u})=B\bigl(\mathbf{u}^{-},\mathbf{u}^{-}\bigr)+B \bigl(\mathbf {u}^{+},\mathbf{u}^{+}\bigr). $$

(6)

Lemma 1

see [2]

Let $a+b\notin\sigma(-\Delta)$ and $\lambda_{k}$ be the first eigenvalue above $a+b$, of multiplicity $m_{k}$, and $\lambda_{l}$ be the first eigenvalue above $a-b$. If $\operatorname{dist}(a-b,\sigma (-\Delta))>\delta>0$, then there exists a constant $K=K(a+b,\delta)>0$ such that

$$\begin{aligned}& B\bigl((u,v),(u,v)\bigr) \leq -K\bigl(\|u\|^{2}+\|v\|^{2} \bigr),\quad \forall(u,v)\in H^{-}, \\& B\bigl((u,v),(u,v)\bigr) \geq K\bigl(\|u\|^{2}+\|v\|^{2} \bigr),\quad \forall (u,v)\in H^{+}. \end{aligned}$$

Lemma 2

Assume that (2), (F1) and (F3) hold. For any real number c, the functional J satisfies the $(Ce)^{*}_{c}$ condition.

Proof

Let $W^{1}=H^{-}$, $W^{2}=H^{+}$, then $H=H^{-}\oplus H^{+}=W^{1}\oplus W^{2}$. Let $\{\alpha_{n}\}$ be admissible and $\{(u_{\alpha_{n}},v_{\alpha _{n}})\}$ be a $(Ce)^{*}_{c}$ sequence, that is,

$$ (u_{\alpha_{n}},v_{\alpha_{n}})\in H_{\alpha_{n}},\qquad J_{\alpha_{n}}(u_{\alpha _{n}},v_{\alpha_{n}})\to c, \qquad \bigl(1+ \Vert u_{\alpha_{n}} \Vert + \Vert v_{\alpha_{n}} \Vert \bigr)J_{\alpha _{n}}'(u_{\alpha_{n}},v_{\alpha_{n}}) \to0 $$

(7)

as $n\to\infty$.

We first proved that $\{(u_{\alpha_{n}},v_{\alpha_{n}})\}$ is bounded in H. Arguing by contradiction, we suppose that $\|(u_{\alpha _{n}},v_{\alpha_{n}})\|=(\|u_{\alpha_{n}}\|^{2}+\|v_{\alpha_{n}}\|^{2})^{1/2}\to \infty$ as $n\to\infty$. Define

$$\tilde{u}_{\alpha_{n}}=\frac{u_{\alpha_{n}}}{ \Vert (u_{\alpha_{n}},v_{\alpha_{n}}) \Vert },\qquad \tilde{v}_{\alpha_{n}}= \frac{v_{\alpha_{n}}}{ \Vert (u_{\alpha_{n}},v_{\alpha _{n}}) \Vert }, $$

then $\{(\tilde{u}_{\alpha_{n}},\tilde{v}_{\alpha_{n}})\}$ is bounded in H, i.e.,

$$\|\tilde{u}_{\alpha_{n}}\|^{2}+\|\tilde{v}_{\alpha_{n}} \|^{2}=1\quad \mbox{for all }n. $$

From (F3) and the continuity of F, there exists a constant $M_{1}$ such that

$$ F_{s}(x,s,t)s+F_{t}(x,s,t)t-2F(x,s,t)\geq c_{2}\bigl(|s|^{\beta}+|t|^{\beta}\bigr)-M_{1} $$

(8)

for any $(x,s,t)\in\Omega\times R^{2}$. Hence, one gets

$$\begin{aligned}& 2J_{\alpha_{n}}(u_{\alpha_{n}},v_{\alpha_{n}})-\bigl\langle J'_{\alpha _{n}}(u_{\alpha_{n}},v_{\alpha_{n}}),(u_{\alpha_{n}},v_{\alpha_{n}}) \bigr\rangle \\& \quad = \int_{\Omega}\bigl(F_{s}(x,u_{\alpha_{n}},v_{\alpha_{n}})u_{\alpha_{n}} +F_{t}(x,u_{\alpha_{n}},v_{\alpha_{n}})v_{\alpha_{n}}-2F(x,u_{\alpha _{n}},v_{\alpha_{n}}) \bigr)\,dx \\& \quad \geq c_{2} \biggl( \int_{\Omega}|u_{\alpha_{n}}|^{\beta}\,dx+ \int_{\Omega}|v_{\alpha_{n}}|^{\beta}\,dx \biggr)-M_{1}|\Omega|, \end{aligned}$$

where $|\Omega|$ denotes the Lebesgue measure of the set Ω. From (7) and the above inequality, there is a positive constant $c_{3}$ such that

$$ \Vert u_{\alpha_{n}} \Vert ^{\beta}_{\beta}+ \Vert v_{\alpha_{n}} \Vert ^{\beta}_{\beta}\leq c_{3}\quad \mbox{for any }n. $$

(9)

Let $(u_{\alpha_{n}}, v_{\alpha_{n}})=(u_{\alpha_{n}}^{-}, v_{\alpha _{n}}^{-})+(u_{\alpha_{n}}^{+}, v_{\alpha_{n}}^{+})\in H_{\alpha_{n}}=H^{-}_{\alpha_{n}}\oplus H^{+}_{\alpha_{n}}$ and note that

$$\frac{N}{2}(r-1)< \beta\quad \mbox{and}\quad \frac{N}{2}(r-1)< \frac{2N}{N+2}r. $$

We first consider the case $\frac{2N}{N+2}r<\beta$, by Hölder’s inequality, (F1), (3) and (9), we have

$$\begin{aligned}& \biggl\vert \int_{\Omega}F_{s}(x,u_{\alpha_{n}},v_{\alpha_{n}})u_{\alpha _{n}}^{+} \,dx \biggr\vert \\& \quad \leq c_{1} \int_{\Omega}\bigl( \bigl\vert u_{\alpha_{n}}^{+} \bigr\vert + \vert u_{\alpha_{n}} \vert ^{r} \bigl\vert u_{\alpha_{n}}^{+} \bigr\vert + \vert v_{\alpha_{n}} \vert ^{r} \bigl\vert u_{\alpha _{n}}^{+} \bigr\vert \bigr)\,dx \\& \quad \leq c_{1} \bigl\Vert u_{\alpha_{n}}^{+} \bigr\Vert _{1}+ c_{1} \biggl( \int_{\Omega} \vert u_{\alpha_{n}} \vert ^{r\cdot\frac{\beta}{r}}\,dx \biggr) ^{\frac{r}{\beta}}\cdot \biggl( \int_{\Omega}\bigl\vert u_{\alpha_{n}}^{+} \bigr\vert ^{\frac{\beta}{\beta-r}}\,dx \biggr) ^{\frac{\beta-r}{\beta}} \\& \qquad {} +c_{1} \biggl( \int_{\Omega} \vert v_{\alpha_{n}} \vert ^{r\cdot\frac{\beta}{r}}\,dx \biggr) ^{\frac{r}{\beta}}\cdot \biggl( \int_{\Omega}\bigl\vert u_{\alpha_{n}}^{+} \bigr\vert ^{\frac{\beta}{\beta-r}}\,dx \biggr) ^{\frac{\beta-r}{\beta}} \\& \quad \leq c_{4} \bigl\Vert u_{\alpha_{n}}^{+} \bigr\Vert . \end{aligned}$$

Similarly, we obtain

$$\biggl\vert \int_{\Omega}F_{t}(x,u_{\alpha_{n}},v_{\alpha_{n}})v_{\alpha_{n}}^{+} \,dx \biggr\vert \leq c_{4} \bigl\Vert v_{\alpha_{n}}^{+} \bigr\Vert , $$

where $c_{4}$ is a positive constant. Hence, it follows from the above two inequalities that

$$\begin{aligned}& \bigl\langle J'_{\alpha_{n}}(u_{\alpha_{n}},v_{\alpha_{n}}), \bigl(u_{\alpha_{n}}^{+}, v_{\alpha_{n}}^{+}\bigr)\bigr\rangle \\& \quad = B\bigl((u_{\alpha_{n}}, v_{\alpha_{n}}),\bigl(u_{\alpha_{n}}^{+}, v_{\alpha_{n}}^{+}\bigr)\bigr)- \int_{\Omega}\bigl(F_{s}(x,u_{\alpha_{n}},v_{\alpha_{n}})u_{\alpha_{n}}^{+} +F_{t}(x,u_{\alpha_{n}},v_{\alpha_{n}})v_{\alpha_{n}}^{+}\bigr) \,dx \\& \quad \geq K\bigl( \bigl\Vert u_{\alpha_{n}}^{+} \bigr\Vert ^{2}+ \bigl\Vert v_{\alpha_{n}}^{+} \bigr\Vert ^{2}\bigr) -c_{4}\bigl( \bigl\Vert u_{\alpha_{n}}^{+} \bigr\Vert + \bigl\Vert v_{\alpha_{n}}^{+} \bigr\Vert \bigr). \end{aligned}$$

(10)

For the case $\frac{N}{2}(r-1)<\beta<\frac{N}{2}(r-1)$, let

$$\gamma=\frac{2Nr-(N+2)\beta}{2N-(N-2)\beta}\in(0,1)\quad \mbox{and} \quad p=\frac {\beta}{r-\gamma}>1. $$

By Hölder’s inequality, (F1), (3) and (9), we have

$$\begin{aligned}& \biggl\vert \int_{\Omega}F_{s}(x,u_{\alpha_{n}},v_{\alpha_{n}})u_{\alpha _{n}}^{+} \,dx \biggr\vert \\& \quad \leq c_{1} \int_{\Omega}\bigl( \bigl\vert u_{\alpha_{n}}^{+} \bigr\vert + \vert u_{\alpha_{n}} \vert ^{\beta/p} \vert u_{\alpha_{n}} \vert ^{\gamma} \bigl\vert u_{\alpha_{n}}^{+} \bigr\vert + \vert v_{\alpha_{n}} \vert ^{\beta/p} \vert v_{\alpha_{n}} \vert ^{\gamma} \bigl\vert u_{\alpha _{n}}^{+} \bigr\vert \bigr)\,dx \\& \quad \leq c_{1} \bigl\Vert u_{\alpha_{n}}^{+} \bigr\Vert _{1}+c_{1} \biggl( \int_{\Omega} \vert u_{\alpha_{n}} \vert ^{\beta }\,dx \biggr) ^{\frac{1}{p}}\cdot \biggl( \int_{\Omega} \vert u_{\alpha_{n}} \vert ^{2^{*}}\,dx \biggr)^{\frac {\gamma}{2^{*}}} \cdot \biggl( \int_{\Omega}\bigl\vert u_{\alpha_{n}}^{+} \bigr\vert ^{2^{*}}\,dx \biggr)^{\frac {1}{2^{*}}} \\& \qquad {} +c_{1} \biggl( \int_{\Omega} \vert v_{\alpha_{n}} \vert ^{\beta}\,dx \biggr) ^{\frac{1}{p}}\cdot \biggl( \int_{\Omega} \vert v_{\alpha_{n}} \vert ^{2^{*}}\,dx \biggr)^{\frac {\gamma}{2^{*}}} \cdot \biggl( \int_{\Omega}\bigl\vert u_{\alpha_{n}}^{+} \bigr\vert ^{2^{*}}\,dx \biggr)^{\frac {1}{2^{*}}} \\& \quad \leq c_{5} \bigl\Vert u_{\alpha_{n}}^{+} \bigr\Vert \bigl( \Vert u_{\alpha_{n}} \Vert ^{\gamma}+ \Vert v_{\alpha_{n}} \Vert ^{\gamma}\bigr). \end{aligned}$$

Similarly, we obtain

$$\biggl\vert \int_{\Omega}F_{t}(x,u_{\alpha_{n}},v_{\alpha_{n}})v_{\alpha_{n}}^{+} \,dx \biggr\vert \leq c_{5} \bigl\Vert v_{\alpha_{n}}^{+} \bigr\Vert \bigl( \Vert u_{\alpha_{n}} \Vert ^{\gamma}+ \Vert v_{\alpha_{n}} \Vert ^{\gamma}\bigr), $$

where $c_{5}$ is a positive constant. Therefore, from the above two inequalities, one has

$$\begin{aligned}& \bigl\langle J'_{\alpha_{n}}(u_{\alpha_{n}},v_{\alpha_{n}}), \bigl(u_{\alpha_{n}}^{+}, v_{\alpha_{n}}^{+}\bigr)\bigr\rangle \\& \quad = B\bigl((u_{\alpha_{n}}, v_{\alpha_{n}}),\bigl(u_{\alpha_{n}}^{+}, v_{\alpha_{n}}^{+}\bigr)\bigr)- \int_{\Omega}\bigl(F_{s}(x,u_{\alpha_{n}},v_{\alpha_{n}})u_{\alpha_{n}}^{+} +F_{t}(x,u_{\alpha_{n}},v_{\alpha_{n}})v_{\alpha_{n}}^{+}\bigr)\,dx \\& \quad \geq K\bigl( \bigl\Vert u_{\alpha_{n}}^{+} \bigr\Vert ^{2}+ \bigl\Vert v_{\alpha_{n}}^{+} \bigr\Vert ^{2}\bigr) -c_{5}\bigl( \bigl\Vert u_{\alpha_{n}}^{+} \bigr\Vert + \bigl\Vert v_{\alpha_{n}}^{+} \bigr\Vert \bigr) \bigl( \Vert u_{\alpha_{n}} \Vert ^{\gamma}+ \Vert v_{\alpha_{n}} \Vert ^{\gamma}\bigr). \end{aligned}$$

(11)

Dividing (10) (or (11)) by $\|u_{\alpha_{n}}\|^{2}+\| v_{\alpha_{n}}\|^{2}$, it follows from (7) that

$$ \bigl\Vert \tilde{u}_{\alpha_{n}}^{+} \bigr\Vert + \bigl\Vert \tilde{v}_{\alpha_{n}}^{+} \bigr\Vert \to0\quad \mbox{as } n\to\infty. $$

(12)

Similarly, we obtain

$$ \bigl\Vert \tilde{u}_{\alpha_{n}}^{-} \bigr\Vert + \bigl\Vert \tilde{v}_{\alpha_{n}}^{-} \bigr\Vert \to0\quad \mbox{as } n\to\infty. $$

(13)

From (12) and (13), we have the following contradiction:

$$1= \Vert \tilde{u}_{\alpha_{n}} \Vert + \Vert \tilde{v}_{\alpha_{n}} \Vert \leq \bigl\Vert \tilde{u}_{\alpha_{n}}^{-} \bigr\Vert + \bigl\Vert \tilde{v}_{\alpha_{n}}^{-} \bigr\Vert + \bigl\Vert \tilde{u}_{\alpha_{n}}^{+} \bigr\Vert + \bigl\Vert \tilde{v}_{\alpha_{n}}^{+} \bigr\Vert \to0 $$

as $n\to\infty$. Hence, $\{(u_{\alpha_{n}},v_{\alpha_{n}})\}$ must be bounded.

It follows that there is a subsequence of $\{(u_{\alpha_{n}},v_{\alpha _{n}})\}$, still denoted by $\{(u_{\alpha_{n}},v_{\alpha_{n}})\}$, and $(u,v)\in H$ such that $(u_{\alpha_{n}},v_{\alpha_{n}})\rightharpoonup(u,v)$ weakly in H, $(u_{\alpha_{n}},v_{\alpha_{n}})\to(u,v)$ strongly in $L^{p}(\Omega)\times L^{p}(\Omega)$ for any $1\leq p<2^{*}$, and $(u_{\alpha_{n}}(x),v_{\alpha_{n}}(x))\to (u(x),v(x))$ for a.e. $x\in\Omega$. Therefore, from (7), we have

$$ \lim_{n\to\infty}\bigl\langle J'(u_{\alpha_{n}},v_{\alpha_{n}}),(v_{\alpha_{n}}-v, u_{\alpha_{n}}-u)\bigr\rangle =0. $$

(14)

Recalling the boundedness of $\{(u_{\alpha_{n}},v_{\alpha_{n}})\}$, Hölder’s inequality and (F1), we obtain

$$\begin{aligned}& \biggl\vert \int_{\Omega}\bigl(v_{\alpha_{n}}(v_{\alpha_{n}}-v)+u_{\alpha_{n}}(u_{\alpha _{n}}-u) \bigr)\,dx \biggr\vert \leq \|v_{\alpha_{n}}\|_{2} \|v_{\alpha_{n}}-v\|_{2} +\|u_{\alpha_{n}}\|_{2} \|u_{\alpha_{n}}-u\|_{2} \\& \hphantom{ \biggl\vert \int_{\Omega}\bigl(v_{\alpha_{n}}(v_{\alpha_{n}}-v)+u_{\alpha_{n}}(u_{\alpha _{n}}-u) \bigr)\,dx \biggr\vert }\to 0\quad \mbox{as } n\to\infty, \end{aligned}$$

(15)

$$\begin{aligned}& \biggl\vert \int_{\Omega}\bigl(u_{\alpha_{n}}(v_{\alpha_{n}}-v)+v_{\alpha_{n}}(u_{\alpha _{n}}-u) \bigr)\,dx \biggr\vert \leq \|u_{\alpha_{n}}\|_{2} \|v_{\alpha_{n}}-v\|_{2} +\|v_{\alpha_{n}}\|_{2} \|u_{\alpha_{n}}-u\|_{2} \\& \hphantom{\biggl\vert \int_{\Omega}\bigl(u_{\alpha_{n}}(v_{\alpha_{n}}-v)+v_{\alpha_{n}}(u_{\alpha _{n}}-u) \bigr)\,dx \biggr\vert }\to 0\quad \mbox{as } n\to\infty, \end{aligned}$$

(16)

$$\begin{aligned}& \biggl\vert \int_{\Omega}F_{s}(x,u_{\alpha_{n}},v_{\alpha_{n}}) (v_{\alpha_{n}}-v) \,dx \biggr\vert \leq c_{1} \int_{\Omega}\bigl(1+|u_{\alpha_{n}}|^{r}+|v_{\alpha_{n}}|^{r} \bigr) |v_{\alpha_{n}}-v|\,dx \\& \hphantom{\biggl\vert \int_{\Omega}F_{s}(x,u_{\alpha_{n}},v_{\alpha_{n}}) (v_{\alpha_{n}}-v) \,dx \biggr\vert } \leq c_{1}\|v_{\alpha_{n}}-v\|_{1}+c_{1} \bigl(\|u_{\alpha_{n}}\|_{\beta}^{r}+\|v_{\alpha _{n}} \|_{\beta}^{r}\bigr) \|v_{\alpha_{n}}-v\|_{{\frac{\beta}{\beta-r}}} \\& \hphantom{\biggl\vert \int_{\Omega}F_{s}(x,u_{\alpha_{n}},v_{\alpha_{n}}) (v_{\alpha_{n}}-v) \,dx \biggr\vert } \to 0\quad \mbox{as } n\to\infty. \end{aligned}$$

(17)

Similarly, we have

$$ \biggl\vert \int_{\Omega}F_{t}(x,u_{\alpha_{n}},v_{\alpha_{n}}) (u_{\alpha_{n}}-u) \,dx \biggr\vert \to 0 \quad \mbox{as } n\to\infty. $$

(18)

From (14), (15), (16), (17) and (18), it follows that

$$ \lim_{n\to\infty} \int_{\Omega}\bigl(\nabla u_{\alpha_{n}}\nabla(u_{\alpha _{n}}-u)+ \nabla v_{\alpha_{n}}\nabla(v_{\alpha_{n}}-v)\bigr)\,dx=0. $$

(19)

Similarly, we obtain

$$ \lim_{n\to\infty} \int_{\Omega}\bigl(\nabla u\nabla(u_{\alpha_{n}}-u)+\nabla v \nabla(v_{\alpha_{n}}-v)\bigr)\,dx=0. $$

(20)

From (19) and (20), we get

$$\lim_{n\to\infty}\|u_{\alpha_{n}}-u\|+\|v_{\alpha_{n}}-v\|=0, $$

which implies that $(u_{\alpha_{n}},v_{\alpha_{n}})\to(u,v)$ strongly in H. □

Let $H^{-}_{n}\subset H^{-}$, $H^{+}_{n}\subset Y^{\perp}$ for any n, $H_{n}=H^{-}_{n}\oplus Y\oplus H^{+}_{n}$ with $\dim H_{n}<\infty$, $H_{n}\subset H_{n+1}$, $\bigcup_{n\in N}H_{n}$ is dense in H. We will prove that the condition $\nabla^{*}(H^{-}\oplus Y^{\perp},c)$ holds for any real number c.

Lemma 3

Assume that (2), (F1) and (F3) hold. For any real number c and any sequence $\{(u_{n},v_{n})\}$ such that

$$ \begin{aligned} &(u_{n},v_{n})\in H_{n} \quad \textit{for any }n,\qquad J(u_{n},v_{n})\to c, \\ &P_{H^{-}\oplus Y^{\perp}\oplus[(u_{n},v_{n})]}P_{H_{n}}J'(u_{n},v_{n}) \to0 \end{aligned} $$

(21)

as $n\to\infty$, there is a subsequence of $\{(u_{n},v_{n})\}$ which converges to a point $(u,v)$ such that $J(u,v)=c$ and $P_{H^{-}\oplus Y^{\perp}\oplus[(u,v)]}J'(u,v)=0$.

Proof

We first claim that $\{(u_{n},v_{n})\}$ is bounded in H. Arguing by contradiction, we suppose that $\|(u_{n},v_{n})\|\to\infty$ as $n\to\infty$, define $\tilde{u}_{n}=u_{n}\setminus\|(u_{n},v_{n})\|$, $\tilde {v}_{n}=v_{n}\setminus\|(u_{n},v_{n})\|$, then $\{(\tilde{u}_{n},\tilde{v}_{n})\}$ is bounded in H, that is,

$$\|\tilde{u}_{n}\|^{2}+\|\tilde{v}_{n} \|^{2}=1 \quad \mbox{for all }n. $$

Hence there is a subsequence of $\{(\tilde{u}_{n},\tilde{v}_{n})\}$, still denoted by $\{(\tilde{u}_{n},\tilde{v}_{n})\}$, and $(\tilde{u},\tilde {v})\in H$ such that $(\tilde{u}_{n},\tilde{v}_{n})\rightharpoonup(\tilde{u},\tilde{v})$ weakly in H, $(\tilde{u}_{n},\tilde{v}_{n})\to(\tilde{u},\tilde{v})$ strongly in $L^{p}(\Omega)\times L^{p}(\Omega)$, and $(\tilde{u}_{n}(x),\tilde{v}_{n}(x))\to(\tilde{u}(x),\tilde{v}(x))$ for a.e. $x\in\Omega$. From (8), we obtain

$$\begin{aligned}& 2J(u_{n},v_{n})-\bigl\langle J'(u_{n},v_{n}),(u_{n},v_{n}) \bigr\rangle \\& \quad = \int_{\Omega}\bigl(F_{s}(x,u_{n},v_{n})u_{n}+F_{t}(x,u_{n},v_{n})v_{n}-2F(x,u_{n},v_{n}) \bigr)\,dx \\& \quad \geq c_{2} \biggl( \int_{\Omega}|u_{n}|^{\beta}\,dx+ \int_{\Omega}|v_{n}|^{\beta }\,dx \biggr) -M_{1}|\Omega|. \end{aligned}$$

Dividing the above expression by $\|(u_{n},v_{n})\|^{\beta}$, it follows from (21) that

$$\int_{\Omega}\bigl(|\tilde{u}_{n}|^{\beta}+| \tilde{v}_{n}|^{\beta}\bigr)\,dx\to0 \quad \mbox{as } n\to \infty, $$

which implies that $(\tilde{u},\tilde{v})=(0,0)$, that is, $(\tilde {u}_{n},\tilde{v}_{n})\rightharpoonup(0,0)$ weakly in H. Similar to the proof of Lemma 2, we can obtain that $(\tilde{u}_{n},\tilde {v}_{n})\to(0,0)$ strongly in H, which is a contradiction to $\|\tilde {u}_{n}\|^{2}+\|\tilde{v}_{n}\|^{2}=1$. Hence $\{(u_{n},v_{n})\}$ is bounded. As was already noted in the proof of Lemma 2, the same proof implies that $(u_{n},v_{n})$ converges in H. □

Lemma 4

If (F1), (F2) and (F3) hold, then there is $\varepsilon'>0$ such that J has no critical point $(u,v)$ with $0<|J(u,v)|<\varepsilon'$.

Proof

We first claim that $(0,0)$ is an isolated critical point for J. Actually, from (F1) and (F2), for any $\varepsilon >0$, there is $M_{2}=M_{2}(\varepsilon)>0$ such that

$$F(x,s,t)\leq\frac{\varepsilon}{2}\bigl(s^{2}+t^{2} \bigr)+M_{2}\bigl(|s|^{r+1}+|t|^{r+1}\bigr),\quad \forall (x,s,t)\in\Omega\times R^{2}, $$

hence, we have

$$\biggl\vert \int_{\Omega}F(x,u,v)\,dx \biggr\vert \leq\frac{\varepsilon}{2} \bigl(\|u\| ^{2}_{2}+\|v\|^{2}_{2}\bigr)+ M_{2}\bigl(\|u\|^{r+1}_{r+1}+\|v\|^{r+1}_{r+1} \bigr),\quad \forall(u,v)\in H, $$

which together with Lemma 1 and (3) implies that for any $(u,v)\in H^{+}$,

$$\begin{aligned} J(u,v) =&\frac{1}{2}B\bigl((u,v),(u,v)\bigr)- \int_{\Omega}F(x,u,v)\,dx \\ \geq&\frac{K-C^{2}\varepsilon}{2} \bigl(\|u\|^{2}+\|v\|^{2} \bigr)-C^{r+1}M_{2}\bigl(\|u\|^{r+1}+\|v \|^{r+1}\bigr), \end{aligned}$$

(22)

and for any $(u,v)\in H^{-}$,

$$\begin{aligned} \begin{aligned} J(u,v)&=\frac{1}{2}B\bigl((u,v),(u,v)\bigr)- \int_{\Omega}F(x,u,v)\,dx \\ &\leq\frac{-K+C^{2}\varepsilon}{2} \bigl(\|u\|^{2}+\|v\|^{2} \bigr)+C^{r+1}M_{2}\bigl(\|u\|^{r+1}+\|v \|^{r+1}\bigr). \end{aligned} \end{aligned}$$

Since $r+1>2$, let $\varepsilon<\frac{K}{2C^{2}}$, there exists a positive constant $R_{0}$ such that

$$\begin{aligned}& J(u,v)>0\quad \mbox{for any } (u,v)\in H^{+} \mbox{ and }0< \bigl\Vert (u,v) \bigr\Vert \leq R_{0}, \\& J(u,v)< 0\quad \mbox{for any } (u,v)\in H^{-} \mbox{ and }0< \bigl\Vert (u,v) \bigr\Vert \leq R_{0}, \end{aligned}$$

which implies that $(0,0)$ is an isolated critical point for J.

Moreover, $(0,0)$ is the only critical point $(u,v)$ of J with $J(u,v)=0$. In fact, if $(u,v)$ is such a point

$$0=\bigl\langle J'(u,v),(u,v)\bigr\rangle =2J(u,v)- \int_{\Omega}\bigl(F_{s}(x,u,v)u+F_{t}(x,u,v)v-2F(x,u,v) \bigr)\,dx, $$

we obtain

$$\int_{\Omega}\bigl(F_{s}(x,u,v)u+F_{t}(x,u,v)v-2F(x,u,v) \bigr)\,dx=0, $$

from (F3), it follows that $(u,v)=(0,0)$.

Now we will finish the argument by contradiction. If there is a sequence $\{(u_{n},v_{n})\}$ such that $J'(u_{n},v_{n})=0$, $J(u_{n},v_{n})\neq0$ for any n and $J(u_{n},v_{n})\to0$ as $n\to\infty$. Similar to the proof of Lemma 2, we have that $(u_{n},v_{n})$ converges to a critical point $(u,v)$ with $J(u,v)=0$, which implies that $(u_{n},v_{n})\to(0,0)$ as $n\to\infty$, but $(0,0)$ is an isolated critical point for J. Hence we obtain a contradiction. □

Next, we will prove that the functional J satisfies the geometry of Theorem 3. We introduce some notations for the later convenience. For $R, \rho>0$, let

$$\begin{aligned}& \Gamma_{R}=\bigl\{ {\mathbf{u}}\in H^{-}:\|{\mathbf{u}}\|\leq R\bigr\} \cup \bigl\{ {\mathbf{u}}\in H^{-}\oplus Y:\|{\mathbf{u}}\|=R\bigr\} , \\& T_{R}=\bigl\{ {\mathbf{u}}\in H^{-}\oplus Y:\|{\mathbf{u}}\|\leq R\bigr\} \quad \mbox{and}\quad \Gamma_{R}=\partial T_{R}, \\& S_{\rho}=\bigl\{ {\mathbf{u}}\in H^{+}:\|{\mathbf{u}}\|=\rho\bigr\} ,\qquad B_{\rho}=\bigl\{ {\mathbf {u}}\in H^{+}:\|{\mathbf{u}}\|\leq\rho\bigr\} . \end{aligned}$$

Lemma 5

Suppose that (2), (F1), (F2), (F3) and (F4) are satisfied. For any $a+b\in(\lambda_{k-1},\lambda_{k})$, there exist $R>\rho>0$ such that

$$0=\sup J(\Gamma_{R})< \inf J(S_{\rho}). $$

Proof

From (5), Lemma 1 and (F4), for any $(u,v)\in H^{-}$, we have

$$\begin{aligned} \begin{aligned}[b] J(u,v)&=\frac{1}{2}B\bigl((u,v),(u,v)\bigr)- \int_{\Omega}F(x,u,v)\,dx \\ &\leq-\frac{K}{2}\bigl(\|u\|^{2}+\|v\|^{2}\bigr) \\ &\leq0. \end{aligned} \end{aligned}$$

(23)

In view of (2) and the continuity of F, for any $c_{6}>0$, there is $M_{3}>0$ such that

$$F(x,s,t)\geq\frac{c_{6}}{2}\bigl(s^{2}+t^{2} \bigr)-M_{3}, \quad \forall(x,s,t)\in\Omega\times R^{2}, $$

which implies that

$$ \int_{\Omega}F(x,u,v)\,dx\geq\frac{c_{6}}{2}\bigl(\|u \|^{2}_{2}+\|v\|^{2}_{2}\bigr)- M_{3}|\Omega|, \quad \forall(u,v)\in H, $$

(24)

which together with (6) implies that for any $(u,v)=(u^{-},v^{-})\oplus(u^{\sim},v^{\sim})\in H^{-}\oplus Y$ and $a+b\in (\lambda_{k-1},\lambda_{k})$,

$$\begin{aligned} J(u,v) \leq&\frac{1}{2}B\bigl(\bigl(u^{-},v^{-}\bigr),\bigl(u^{-},v^{-}\bigr) \bigr)+\frac{1}{2}B\bigl(\bigl(u^{\sim},v^{\sim}\bigr), \bigl(u^{\sim},v^{\sim}\bigr)\bigr) \\ &{}-\frac{c_{6}}{2}\bigl( \Vert u \Vert ^{2}_{2}+ \Vert v \Vert ^{2}_{2}\bigr)+M_{3}|\Omega| \\ \leq&-\frac{K}{2}\bigl( \bigl\Vert u^{-} \bigr\Vert ^{2}+ \bigl\Vert v^{-} \bigr\Vert ^{2}\bigr)+\frac{1}{2} \mu_{k}\bigl( \bigl\Vert u^{\sim}\bigr\Vert ^{2}+ \bigl\Vert v^{\sim}\bigr\Vert ^{2}\bigr) \\ &{}-\frac{c_{6}}{2}\bigl( \bigl\Vert u^{-} \bigr\Vert ^{2}_{2}+ \bigl\Vert v^{-} \bigr\Vert ^{2}_{2}\bigr)- \frac{c_{6}}{2\lambda_{k}}\bigl( \bigl\Vert u^{\sim}\bigr\Vert ^{2}+ \bigl\Vert v^{\sim}\bigr\Vert ^{2} \bigr)+M_{3}|\Omega| \\ \leq&-c_{7}\bigl( \bigl\Vert u^{-} \bigr\Vert ^{2}+ \bigl\Vert v^{-} \bigr\Vert ^{2}\bigr)+ \frac{\lambda_{k}-(a+b)-c_{6}}{2\lambda_{k}}\bigl( \bigl\Vert u^{\sim}\bigr\Vert ^{2}+ \bigl\Vert v^{\sim}\bigr\Vert ^{2}\bigr)+M_{3}|\Omega|. \end{aligned}$$

Let $c_{6}>\lambda_{k}-(a+b)$, the above expression implies that

$$ J(u,v)\to-\infty\quad \mbox{as } \bigl\Vert (u,v) \bigr\Vert \to\infty. $$

(25)

From (22), (23) and (25), there are two constants $R>\rho>0$ such that

$$\sup J(\Gamma_{R})< \inf J(S_{\rho}). $$

 □

Lemma 6

If (F4) holds, then for $R>0$ in Lemma  5 and for any $\varepsilon''>0$, there exists $\varepsilon_{0}>0$ such that for any $a+b\in(\lambda_{k}-\varepsilon_{0},\lambda_{k})$, we have

$$\sup J(T_{R})< \varepsilon''. $$

Proof

For any $(u,v)=(u^{-},v^{-})\oplus(u^{\sim},v^{\sim})\in H^{-}\oplus Y$, in view of Lemma 1, (F4) and the fact that $a+b<\lambda _{k}$, we have

$$\begin{aligned} J(u,v) \leq&\frac{1}{2}B\bigl(\bigl(u^{-},v^{-}\bigr),\bigl(u^{-},v^{-}\bigr) \bigr)+\frac{1}{2}B\bigl(\bigl(u^{\sim},v^{\sim}\bigr), \bigl(u^{\sim},v^{\sim}\bigr)\bigr) \\ \leq&-\frac{K}{2}\bigl( \bigl\Vert u^{-} \bigr\Vert ^{2}+ \bigl\Vert v^{-} \bigr\Vert ^{2}\bigr)+ \frac{\lambda_{k}-(a+b)}{2\lambda_{k}}\bigl( \bigl\Vert u^{\sim}\bigr\Vert ^{2}+ \bigl\Vert v^{\sim}\bigr\Vert ^{2}\bigr). \end{aligned}$$

Let $\varepsilon_{0}=2\lambda_{k}\varepsilon''/R^{2}$, it is easy to see that the conclusion holds. □

Proof of Theorem 1

The argument is finished by two steps.

(a) There are two critical points $(u_{1},v_{1})$ and $(u_{2},v_{2})$ of the functional J, which satisfy

$$\inf_{(u,v)\in S_{\rho}}J(u,v)\leq J(u_{i},v_{i})\leq \sup_{(u,v)\in T_{R}}J(u,v)\quad \mbox{for } i=1,2. $$

Let $\varepsilon\in(0,\varepsilon')$, $\alpha_{0}=\inf J(S_{\rho})$ and $\beta_{0}=\sup J(T_{R})$, where $\varepsilon'$ is defined in Lemma 4. It follows from Lemmas 3 and 6 that $J|_{H^{-}\oplus Y^{\perp}}$ has no critical levels between $\alpha_{0}$ and $\beta_{0}$, hence the functional J satisfies the condition $\nabla^{*}(H^{-}\oplus Y^{\perp},c)$ for any $c\in[\alpha_{0},\beta_{0}]$. Lemma 2 shows that the $(Ce)^{*}_{c}$ condition holds true for any $c\in[\alpha_{0},\beta_{0}]$. Hence two critical points are obtained from Theorem 3.

(b) There is the third critical point, which is different from $(u_{1},v_{1})$ and $(u_{2},v_{2})$. Let $(\varphi_{k}, \varphi_{k})\in H^{+}$ be fixed, define

$$\begin{aligned}& S_{\rho_{1}} = \bigl\{ (u,v)\in H^{+}: \bigl\Vert (u,v) \bigr\Vert ={ \rho_{1}}\bigr\} , \\& B_{R_{1}} = \bigl\{ (u,v)+r(\varphi_{k}, \varphi_{k}):r \geq0,(u,v)\in H^{-}, \bigl\Vert (u+r\varphi_{k},v+r \varphi_{k}) \bigr\Vert \leq R_{1}\bigr\} , \\& \partial B_{R_{1}} = \bigl\{ (u,v)+r(\varphi_{k}, \varphi_{k}):r\geq0,(u,v)\in H^{-}, \bigl\Vert (u+r \varphi_{k},v+r\varphi_{k}) \bigr\Vert =R_{1} \bigr\} \\& \hphantom{\partial B_{R_{1}} ={}}{}\cup\bigl\{ (u,v):(u,v)\in H^{-}, \bigl\Vert (u,v) \bigr\Vert \leq R_{1}\bigr\} . \end{aligned}$$

For any $(u,v)=(u^{-},v^{-})+(r\varphi_{k}, r\varphi_{k})$, $r\geq0$, from (24), we get

$$\begin{aligned} J(u,v) \leq&\frac{1}{2}B\bigl(\bigl(u^{-},v^{-}\bigr),\bigl(u^{-},v^{-}\bigr) \bigr)+ \frac{r^{2}}{2}B\bigl((\varphi_{k}, \varphi_{k}),( \varphi_{k}, \varphi_{k})\bigr) \\ &{}-\frac{c_{6}}{2}\bigl( \Vert u+r\varphi_{k} \Vert ^{2}_{2}+ \Vert v+r\varphi_{k} \Vert ^{2}_{2}\bigr)+M_{3}|\Omega | \\ \leq&-\frac{K}{2}\bigl( \bigl\Vert u^{-} \bigr\Vert ^{2}+ \bigl\Vert v^{-} \bigr\Vert ^{2}\bigr)+r^{2} \mu_{k} \Vert \varphi_{k} \Vert ^{2} \\ &{}-\frac{c_{6}}{2}\bigl( \bigl\Vert u^{-} \bigr\Vert ^{2}_{2}+ \bigl\Vert v^{-} \bigr\Vert ^{2}_{2}\bigr)- \frac{r^{2} c_{6}}{\lambda_{k}} \Vert \varphi_{k} \Vert ^{2}+M_{3}| \Omega| \\ \leq&-c_{7}\bigl( \bigl\Vert u^{-} \bigr\Vert ^{2}+ \bigl\Vert v^{-} \bigr\Vert ^{2}\bigr)+ \frac{\lambda_{k}\mu_{k}-c_{6}}{\lambda_{k}}r^{2} \Vert \varphi_{k} \Vert ^{2}+M_{3}|\Omega |. \end{aligned}$$

(26)

Picking $c_{6}>\lambda_{k}\mu_{k}$, it follows from (22), (26) and $r+1>2$ that there exists $R_{1}>\rho_{1}>0$ such that

$$\sup J(\partial B_{R_{1}})< \inf J(S_{\rho_{1}}). $$

Let $\{H_{n}\}$ be a sequence of subspaces of H, which is defined as before, then $(\varphi_{k}, \varphi_{k})\in H_{n}$ for all n, we can prove that for any n,

$$\sup J(\partial B_{R_{1}}\cap H_{n})< \inf J(S_{\rho_{1}} \cap H_{n}). $$

Similar to the proof of Lemma 2, the functional $J|_{H_{n}}$ satisfies the $(Ce)_{c}$ condition for any $c\in R$. Hence, by linking argument, there exists a critical point $(u_{n},v_{n})$ of $J|_{H_{n}}$ with

$$\inf J(S_{\rho_{1}}\cap H_{n})\leq J(u_{n},v_{n}) \leq\sup J(B_{R_{1}}\cap H_{n}). $$

From Lemma 2, there is a subsequence of $\{(u_{n},v_{n})\}$, also denoted by $\{(u_{n},v_{n})\}$, and $(u,v)\in H$ such that $(u_{n},v_{n})\to(u,v)$ as $n\to\infty$, which implies that $(u,v)$ is a critical point for J with

$$\inf J(S_{\rho_{1}})\leq J(u,v)\leq\sup J(B_{R_{1}}). $$

 □