The Pohozaev-type inequalities and their applications for a kind of elliptic equation (system)

Abstract

In this paper, we first derive a new kind of Pohozaev-type inequalities for p-Laplacian equations in a more general class of non-star-shaped domains, and then we take two examples and their graphs to explain the shape of the new kind of the non-star-shaped domain. At last, we extend the results of Pohozaev-type inequalities to elliptic systems, which are used to derive the nonexistence of positive solutions of this type of systems in the non-star-shaped domains. On this basis, we also discuss the existence of positive solutions of a kind of elliptic systems with double critical growth.

1 Introduction

In this paper, we first consider the following elliptic equation:

$$ \textstyle\begin{cases} -\Delta _{p} u=f(u), &\text{in } \Omega , \\ u=0, &\text{on }\partial \Omega , \end{cases} $$

(1.1)

where $\Delta _{p}=\operatorname{div}( \vert \nabla u \vert ^{p-2} \nabla u)$, $f\in C(R^{1}, R^{1})$, and $\Omega \subset {\mathbb {R}}^{n}$ ($n\ge 3$) is a domain with smooth boundary. We establish the Pohozaev-type inequalities not Pohozaev identities for the solutions of (1.1), and by which we derive the nonexistence of positive solutions for the problem in non-star-shaped domains. Subsequently, we take two examples and their graphs to explain the shape of the new kind of the non-star-shaped domain. Then, we also consider a similar problem for the following elliptic system:

$$ \textstyle\begin{cases} -\operatorname{div}( \vert \nabla u \vert ^{p-2}\nabla u)=f_{u}(u,v)&x\in \Omega , \\ -\operatorname{div}( \vert \nabla v \vert ^{p-2}\nabla v)=f_{v}(u,v)&x\in \Omega , \\ u=v=0 &x\in \partial \Omega , \end{cases} $$

(1.2)

where $\Delta _{p}=\operatorname{div}( \vert \nabla u \vert ^{p-2} \nabla u)$, $f_{u}(u,v)=\frac{\partial F(u,v)}{\partial u}$, $f_{v}(u,v)= \frac{\partial F(u,v)}{\partial v}$, $f_{u}(u,v), f_{v}(u,v): \mathbb{R}\times \mathbb{R}\rightarrow \mathbb{R}$, and they are continuous. $\Omega \subset \mathbb{R}^{N}$ ($N\ge 3$) is also a domain with smooth boundary.

When $p=2$ in (1.1), Pohozaev proved the following well-known Pohozaev identity in the famous paper [1]:

$$ (2-n) \int _{\Omega }u f(u)\,dx+2n \int _{\Omega }F(u)\,dx = \int _{\partial \Omega } \bigl\langle x, \nu (x) \bigr\rangle \biggl\vert \frac{\partial u }{\partial \nu } \biggr\vert ^{2}\,ds. $$

(1.3)

Based on the identity, Pohozaev, Brezis, and Nirenberg considered the following problem in [1, 2]:

$$ \textstyle\begin{cases} - \Delta u = {{ \vert u \vert }^{p - 1}}u + \lambda u&x \in \Omega , \\ {u > 0}&{x \in \Omega }, \\ {u = 0}&{x \in \partial \Omega }, \end{cases} $$

(1.4)

where $\lambda \in \mathbb{R}$, $\Omega \subset \mathbb{R}^{N}$ ($N\geq 3$) is a bounded domain with $C^{1}$ boundary. They proved that if $\Omega \subset \mathbb{R}^{N}$ ($N\geq 3$) is a star-shaped bounded domain, problem (1.4) has no solution when both $p\ge \frac{n+2}{n-2}$, $\lambda \le 0$ and $p> \frac{n+2}{n-2}$, $\lambda \leq \delta (\Omega ,p)$.

Subsequently, Kawano and Ni discussed the general semi-linear elliptic equation when $p=2$ in [3]:

$$ \textstyle\begin{cases} { - \Delta _{p} u = f(x,u)}&{x \in \Omega }, \\ {u > 0}&{x \in \Omega } , \\ {u = 0}&{x \in \partial \Omega }. \end{cases} $$

(1.5)

By the classic Pohozaev proof method, the authors derived the following identity:

$$ \begin{aligned} & (2-n) \int _{\Omega }u f(x,u)\,dx+2n \int _{\Omega }F(x,u)\,dx+2 \int _{\Omega } \bigl(x,F_{x}(x,u) \bigr)\,dx \\ &\quad = \int _{\partial \Omega } \bigl\langle x, \nu (x) \bigr\rangle \biggl\vert \frac{\partial u }{\partial \nu } \biggr\vert ^{2}\,ds. \end{aligned} $$

(1.6)

In 1989, Guedda and Veron considered (1.1) in [4] and got the following identity:

$$ (p-n) \int _{\Omega }u f(u)\,dx+pn \int _{\Omega }F(u)\,dx =(p-1) \int _{ \partial \Omega } \bigl\langle x, \nu (x) \bigr\rangle \biggl\vert \frac{\partial u }{\partial \nu } \biggr\vert ^{p}\,ds, $$

(1.7)

where $F(u)=\int _{0}^{u}f(t)\,dt$, $f(u)$, is continuous. And they got the nonexistence results of nontrivial solutions in the case of supercritical growth and critical growth when $1< p< N$ and $\Omega \subset \mathbb{R}^{N}$ is a star-shaped domain about origin. See [5–11] and the references therein. In [12], we established more general Pohozaev-type inequalities of (1.5) and got the nonexistence of positive solutions when $\Omega \subset \mathbb{R}^{N}$ is a non-star-shaped domain.

In this paper, we discuss more general Pohozaev-type inequalities and the nonexistence of positive solutions of (1.1). At the same, we take two examples and their graphs to explain the shape of a new kind of the non-star-shaped domain.

The existence or nonexistence of nontrivial positive solutions of elliptic system (1.2) have been extensively studied in recent years, we refer to [13–16] and the references therein. Researchers usually convert the proof of the existence of nonlinear elliptic problems into finding the existence of the critical point of the variational functional corresponding to the equation, in which, one of the key issues is to prove that the (PS) sequence has a strongly convergent sub-sequence. Therefore, the first step is to prove the boundedness of the (PS) sequence. Scholars have put forward many conditions such as the famous A-R condition. However, the A-R condition cannot be satisfied in many cases. In recent years, it was shown that combining the Pohozaev identity and the Nehari manifold to create a new manifold, which is called Pohozaev–Nehari manifold, was very effective when treating nonlinearity which does not satisfy the A-R condition (see [17–19]).

In [17], Li and Ye considered the following Kirchhoff equation:

$$ \textstyle\begin{cases} -(a+b \int _{\mathbb{R}^{3}} \vert \nabla u \vert ^{2} \,dx) \triangle u+u = {{ \vert u \vert }^{p - 1}}u&x \in \mathbb{R}^{3}, \\ u \in H^{1}(\mathbb{R}^{3}), \quad\quad u>0&x \in \mathbb{R}^{3} . \end{cases} $$

They established the Pohozaev identity

$$ P(u):=\frac{a}{2} \int _{\mathbb{R}^{3}} \vert \nabla u \vert ^{2} \,dx+ \frac{3}{2} \int _{\mathbb{R}^{3}} \vert u \vert ^{2} \,dx + \frac{b}{2} \biggl( \int _{\mathbb{R}^{3}} \vert \nabla u \vert ^{2} \,dx \biggr)^{2}- \frac{3}{p+1} \int _{\mathbb{R}^{3}} \vert u \vert ^{p+1} \,dx $$

and the new Pohozaev–Nehari manifold

$$ \mathcal{M}= \bigl\{ u \in H^{1} \bigl( \mathbb{R}^{3} \bigr)\backslash {0}\mid G(u)=0 \bigr\} , $$

(1.8)

where $G(u)=\langle I'(u),u\rangle +P(u)$. The authors proved that the new manifold (1.8) was $C^{1}$ and the critical point of the limit equation on the new manifold was the critical point in the entire space. Subsequently, the solution of the equation was obtained by finding the least energy solution of the limit equation of the corresponding equation in the new manifold.

In [19], the authors studied the existence of a bound state solution for a class of quasi-linear Schrödinger equations whose nonlinear term is asymptotically linear in $\mathbb{R}^{N}$. The equation is as follows:

$$ -\triangle u+V(x)u-\triangle \bigl(u^{2} \bigr)u=g(u), \quad x\in \mathbb{R}^{N}, N \geq 3, $$

where $V(x)=W(x)-E$ is the new potential and $g(u)=l(u^{2})u$ is the nonlinear term. The authors defined the following Pohozaev manifold:

$$ \mathcal{P}= \bigl\{ v \in H^{1} \bigl(\mathbb{R}^{N} \bigr) \backslash {0}\mid \gamma (v)=0 \bigr\} $$

and

$$ \mathcal{P_{\infty }}= \bigl\{ v \in H^{1} \bigl( \mathbb{R}^{N} \bigr)\backslash {0} \mid v \text{ satisfies Pohozaev identity} \bigr\} , $$

where

$$ \begin{aligned} \gamma (v)={}&\frac{N-2}{2} \int _{\mathbb{R}^{N}} \vert \nabla v \vert ^{2} \,dx+ \frac{N}{2} \int _{\mathbb{R}^{N}} V(x)f^{2}(v)\,dx+ \frac{1}{2} \int _{\mathbb{R}^{N}} \bigl(\nabla V(x),x \bigr)f^{2}(v)\,dx \\ &{} -N \int _{ \mathbb{R}^{N}}G \bigl(f(v) \bigr)\,dx, \end{aligned} $$

and the corresponding Pohozaev identity of the limiting problem is as follows:

$$ \frac{N-2}{2} \int _{\mathbb{R}^{N}} \vert \nabla v \vert ^{2} \,dx+ \frac{N}{2} \int _{\mathbb{R}^{N}} V_{\infty }f^{2}(v)\,dx=N \int _{ \mathbb{R}^{N}}G \bigl(f(v) \bigr)\,dx. $$

Employing the minimization methods restricted to the above Pohozaev manifold, the authors obtained the existence of positive solutions for the asymptotically linear case.

In [16], Bozhkov and Mitidieri studied the existence and nonexistence of the following (p,q)-Laplacian system:

$$ \textstyle\begin{cases} { - {\Delta _{p}}u = \lambda a(x){{ \vert u \vert }^{p - 2}}u + ( \alpha + 1)c(x){{ \vert u \vert }^{\alpha - 1}}u{{ \vert v \vert }^{\beta + 1}}}&{x \in \Omega }, \\ { - {\Delta _{q}}v = \mu b(x){{ \vert v \vert }^{q - 2}}v + ( \beta + 1)c(x){{ \vert u \vert }^{\alpha + 1}}{{ \vert v \vert }^{ \beta - 1}}v}&{x \in \Omega } , \\ {u = v = 0}&{x \in \partial \Omega }. \end{cases} $$

The authors also generalized the Pohozaev identity for the following system which is based on reference [20]:

$$ \textstyle\begin{cases} -\operatorname{div}( \vert \nabla u \vert ^{p-2}\nabla u)=F_{u}(x,u,v)&x\in \Omega , \\ -\operatorname{div}( \vert \nabla v \vert ^{q-2}\nabla v)=F_{v}(x,u,v)&x\in \Omega , \\ u=v=0 &x\in \partial \Omega . \end{cases} $$

(1.9)

And the Pohozaev identity is as follows:

$$\begin{aligned} \begin{aligned} &\frac{N-p}{p} \int _{\Omega } \vert \nabla u \vert ^{p}\,dx+ \frac{N-q}{q} \int _{\Omega } \vert \nabla v \vert ^{q}\,dx-N \int _{\Omega }F(x,u,v)\,dx- \int _{\Omega }D_{x}F(x,u,v)\,dx \\ &\quad =- \biggl(1-\frac{1}{p} \biggr) \int _{\partial \Omega } \vert \nabla u \vert ^{p}(x,\nu )\,ds- \biggl(1- \frac{1}{q} \biggr) \int _{\partial \Omega } \vert \nabla v \vert ^{q}(x,\nu )\,ds. \end{aligned} \end{aligned}$$

(1.10)

By this kind of Pohozaev identity, the authors derived the nonexistence of the positive solutions on the star-shaped domain. However, on the basis of preliminary work of Pohozaev inequalities for p-Laplacian equation, we can generalize the kind of identity to the kind of Pohozaev inequalities for the p-Laplace system and (p,q)-Laplacian systems, and naturally get the nonexistence of these systems on the non-star-shaped domain, which is more general than the domain in [16]. In addition, we consider the following elliptic system with double critical growth:

$$ \textstyle\begin{cases} -\Delta _{p} u= \vert u \vert ^{p^{*}-2}u+\frac{\alpha }{p^{*}} \vert u \vert ^{\alpha -2} \vert v \vert ^{ \beta } u,&x\in \Omega, \\ -\Delta _{p} v= \vert v \vert ^{p^{*}-2}v+\frac{\beta }{p^{*}} \vert u \vert ^{\alpha } \vert v \vert ^{ \beta -2} v,&x\in \Omega , \\ u=v=0,&x\in \partial \Omega , \\ u>0,\quad\quad v>0& x\in \Omega , \end{cases} $$

(1.11)

where $\alpha , \beta >1$, $\alpha +\beta =p^{*}=\frac{Np}{N-p}$. Using the Pohozaev inequalities, we can prove that system (1.11) has no positive solution when Ω is a positive domain. It is worth to mention that the more general Pohozaev to the general elliptic equations in a non-star-shaped bounded domain can help us to obtain the necessary conditions of the existence of solutions for some special equations. See [21–27]. In [27], Peng, Peng, and Wang discussed in detail the existence and the form of positive solution, the uniqueness of the minimal energy solution of system (1.11) when $p=2$. Based on their global compactness result and the main method in [21], we consider the existence of positive solution of system (1.11) for general p.

2 The results and proofs

We need the following lemma, which is similar but also has some differences with our manuscript [12].

Lemma 2.1

Suppose $V(x)=(V_{1}(x),\ldots,V_{n}(x))$, which is a $C^{1}$ vector field on ${\mathbb {R}}^{n}$ and $u\in W^{1,p}_{0}(\Omega )\cap C^{1}(\bar{\Omega })$ is a solution of (1.1). Then

$$ \int _{\Omega }u \operatorname{div} \bigl( \vert \nabla u \vert ^{p-2}V(x) \bigr)\,dx=- \int _{ \Omega } \vert \nabla u \vert ^{p-2} \bigl\langle V(x), \nabla u \bigr\rangle \,dx $$

and

$$ \int _{\Omega }F(u)\operatorname{div} V(x)\,dx=- \int _{\Omega }f(u) \bigl\langle V(x), \nabla u \bigr\rangle \,dx. $$

Based on the above lemma, we can have a Pohozaev-type inequality for solutions of equation (1.1).

Theorem 2.2

(Pohozaev-type inequality)

Suppose that $V(x)$ is a linear vector field on ${\mathbb {R}}^{n}$ with the form

$$ V(x)= \begin{pmatrix} a_{11} & \cdots & a_{1n} \\ \vdots & \vdots & \vdots \\ a_{n1} & \cdots & a_{nn} \end{pmatrix}x, $$

and $V(x)$ satisfies ${\operatorname{div} V(x)=n}$ and $\langle V(x),x\rangle >0$ for $\forall x\in \mathbb{R}^{n}\setminus \{0\}$. If $u\in W^{1,p}_{0}(\Omega )\cap C^{1}(\bar{\Omega })$ is a solution of (1.1), then

$$ \begin{aligned} &(p\mu -n) \int _{\Omega } uf(u)\,dx+pn \int _{\Omega }F(u)\,dx \\ &\quad \geq (p-1) \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl\vert \frac{\partial u}{\partial \nu } \biggr\vert ^{p}\,ds, \end{aligned} $$

(2.1)

where $\mu =\sup_{ \vert x \vert \neq 0}\frac{\langle V(x),x\rangle }{ \vert x \vert ^{2}}$.

Proof

From $\mu =\sup_{ \vert x \vert \neq 0}\frac{\langle V(x),x\rangle }{ \vert x \vert ^{2}}$, we easily get

$$ 0< \bigl\langle V(x),x \bigr\rangle < \mu \vert x \vert ^{2}, \quad \forall x\in \mathbb{R}^{n}\setminus \{0\}. $$

Multiply the equation $-\Delta _{p} u=f(u)$ by $\langle V(x),\nabla u\rangle$ and then integrate in Ω. By Lemma 2.1 and divergence theorem, the right-hand side is

$$ \begin{aligned} - \int _{\Omega }f(u) \bigl\langle V(x),\nabla u \bigr\rangle \,dx =- \int _{\Omega }\sum_{i=1}^{n}f(u)V_{i}(x) \frac{\partial u}{\partial x_{i}} = \int _{\Omega }F(u) \operatorname{div} V(x)\,dx, \end{aligned} $$

and then the left-hand side is

$$\begin{aligned} &\int _{\Omega } \operatorname{div} \bigl( \vert \nabla u \vert ^{p-2} \nabla u \bigr) \bigl\langle V(x),\nabla u \bigr\rangle \,dx \\ &\quad =\sum_{j=1}^{n} \int _{\Omega }\frac{\partial }{\partial x_{j}} \biggl( \vert \nabla u \vert ^{p-2} \frac{\partial u}{\partial x_{j}} \biggr) \Biggl(\sum _{i=1}^{n} V_{i}(x) \frac{\partial u}{\partial x_{i}} \Biggr)\,dx \\ &\quad=\sum_{j=1}^{n} \int _{\Omega }\frac{\partial }{\partial x_{j}} \Biggl( \vert \nabla u \vert ^{p-2} \frac{\partial u}{\partial x_{j}}\sum_{i=1}^{n} V_{i}(x) \frac{\partial u}{\partial x_{i}} \Biggr)- \vert \nabla u \vert ^{p-2} \frac{\partial u}{\partial x_{j}}\frac{\partial }{\partial x_{j}} \Biggl( \sum _{i=1}^{n} V_{i}(x) \frac{\partial u}{\partial x_{i}} \Biggr)\,dx. \end{aligned}$$

We also have

$$\begin{aligned} &\int _{\Omega } \operatorname{div} \bigl( \vert \nabla u \vert ^{p-2} \nabla u \bigr) \bigl\langle V(x),\nabla u \bigr\rangle \,dx \\ &\quad = \int _{\Omega }\operatorname{div} \bigl( \vert \nabla u \vert ^{p}V(x) \bigr)\,dx \\ &\quad\quad {} -\sum_{j=1}^{n} \int _{\Omega } \vert \nabla u \vert ^{p-2} \frac{\partial u}{\partial x_{j}} \Biggl(\sum_{i=1}^{n}a_{ij} \frac{\partial u}{\partial x_{i}}+\sum_{i=1}^{n} V_{i}(x) \frac{\partial ^{2}u}{\partial x_{i}\partial x_{j}} \Biggr) \\ &\quad= \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \vert \nabla u \vert ^{p}\,ds \\ &\quad\quad {} -\sum_{j=1}^{n} \int _{\Omega } \vert \nabla u \vert ^{p-2} \frac{\partial u}{\partial x_{j}} \Biggl(\sum_{i=1}^{n}a_{ij} \frac{\partial u}{\partial x_{i}}+\sum_{i=1}^{n} V_{i}(x) \frac{\partial ^{2}u}{\partial x_{i}\partial x_{j}} \Biggr) \\ &\quad= \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \vert \nabla u \vert ^{p}\,ds+ \int _{\Omega }u \operatorname{div} \bigl( \vert \nabla u \vert ^{p-2}V(\nabla u) \bigr) \\ &\quad\quad {} -\sum_{j=1}^{n} \int _{\Omega } \vert \nabla u \vert ^{p-2} \frac{\partial u}{\partial x_{j}} \Biggl(\sum_{i=1}^{n}V_{i} \frac{\partial ^{2}u}{\partial x_{i}\partial x_{j}} \frac{\partial u}{\partial x_{j}} \Biggr) \\ &\quad = \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \vert \nabla u \vert ^{p}\,ds- \int _{\Omega } \vert \nabla u \vert ^{p-2} \bigl\langle V(\nabla u),\nabla u \bigr\rangle \,dx \\ &\quad\quad {} -\frac{1}{2}\sum_{i=1}^{n} \int _{\Omega } \vert \nabla u \vert ^{p-2} V_{i}(x) \frac{\partial }{\partial x_{i}} \bigl( \vert \nabla u \vert ^{2} \bigr)\,dx \\ &\quad= \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \vert \nabla u \vert ^{p}\,ds- \int _{\Omega } \vert \nabla u \vert ^{p-2} \bigl\langle V(\nabla u),\nabla u \bigr\rangle \,dx \\ &\quad\quad {} -\frac{1}{p}\sum_{i=1}^{n} \int _{\Omega }V_{i}(x) \frac{\partial }{\partial x_{i}} \bigl( \vert \nabla u \vert ^{p} \bigr)\,dx \\ &\quad= \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \vert \nabla u \vert ^{p}\,ds- \int _{\Omega } \vert \nabla u \vert ^{p-2} \bigl\langle V(\nabla u),\nabla u \bigr\rangle \,dx \\ & \quad\quad {} -\frac{1}{p} \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \vert \nabla u \vert ^{p}\,ds+\frac{1}{p}\sum _{i=1}^{n}a_{ii} \int _{\Omega } \vert \nabla u \vert ^{p}\,dx. \end{aligned}$$

Comparing the left- and right-hand sides, we know the following identity:

$$\begin{aligned} &\biggl(1-\frac{1}{p} \biggr) \int _{\partial \Omega } \vert \nabla u \vert ^{p} \bigl\langle V(x),\nu (x) \bigr\rangle \,ds \\ &\quad =-\frac{1}{p}\sum_{i=1}^{n}a_{ii} \int _{\Omega } \vert \nabla u \vert ^{p}\,dx+ \int _{\Omega } \vert \nabla u \vert ^{p-2} \bigl\langle V(\nabla u),\nabla u \bigr\rangle \,dx+ \int _{\Omega }F \operatorname{div}V(x)\,dx \\ &\quad =-\frac{1}{p}\sum_{i=1}^{n}a_{ii} \int _{\Omega }uf(u)\,dx+ \int _{\Omega } \vert \nabla u \vert ^{p-2} \bigl\langle V(\nabla u),\nabla u \bigr\rangle \,dx+\sum_{i=1}^{n}a_{ii} \int _{\Omega }F(u)\,dx. \end{aligned}$$

By the fact $0<\langle V(x),x\rangle \leq \mu \vert x \vert ^{2}$, we get the following result:

$$ 0< \bigl\langle V(\nabla u),\nabla u \bigr\rangle \leq \mu \vert \nabla u \vert ^{2}. $$

Thus,

$$ \begin{aligned} \int _{\Omega } \vert \nabla u \vert ^{p-2} \bigl\langle V(\nabla u),\nabla u \bigr\rangle \,dx&\leq \mu \int _{\Omega } \vert \nabla u \vert ^{p}\,dx=\mu \int _{\Omega }uf(u)\,dx \end{aligned} $$

because $\operatorname{div}V(x)=\sum_{i=1}^{n}a_{ii}=n$, we obtain the following inequality:

$$ \begin{aligned} &(p\mu -n) \int _{\Omega } uf(u)\,dx+pn \int _{\Omega }F(u)\,dx \\ &\quad \geq (p-1) \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl\vert \frac{\partial u}{\partial \nu } \biggr\vert ^{p}\,ds. \end{aligned} $$

The proof is complete. □

By the above Pohozaev-type inequality, we can immediately get the nonexistence result of the following problem:

$$ \textstyle\begin{cases} -\Delta _{p} u= \vert u \vert ^{\alpha -1}u+\lambda \vert u \vert ^{p-2}u,\quad \quad u>0, & \text{in }\Omega , \\ u=0, & \text{on }\partial \Omega . \end{cases} $$

(2.2)

Theorem 2.3

Assume that the vector field ${V(x)}$ of Theorem 2.2is transverse to ∂Ω. Then (2.2) has no positive solution if

$$ \alpha \geq \frac{np-n+p\mu }{n-p\mu }\quad \textit{and} \quad \lambda \leq 0. $$

Proof

With the similar method in the proof of Theorem 2.2, we also first consider that (1.1) has a positive solution u. Then inequality (2.1) holds,

$$ \begin{aligned} &(p\mu -n) \int _{\Omega } \bigl(u^{\alpha +1}+\lambda u^{p} \bigr)\,dx+pn \int _{\Omega } \biggl(\frac{1}{\alpha +1}u^{\alpha +1}+ \frac{\lambda }{p}u^{p} \biggr)\,dx \\ &\quad \geq (p-1) \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl\vert \frac{\partial u}{\partial \nu } \biggr\vert ^{p}\,ds. \end{aligned} $$

That is,

$$ \begin{aligned} & \biggl(p\mu -n+\frac{pn}{\alpha +1} \biggr) \int _{\Omega } u^{\alpha +1}\,dx+p \mu \lambda \int _{\Omega }u^{p}\,dx \\ &\quad \geq \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl\vert \frac{\partial u}{\partial \nu } \biggr\vert ^{p}\,ds. \end{aligned} $$

Since $\alpha \geq \frac{np-n+p\mu }{n-p\mu }$, we have $p\mu -n+\frac{pn}{\alpha +1}\leq 0$. Thus,

$$ p\mu \lambda \int _{\Omega }u^{p}\,dx\geq \int _{\partial \Omega } \bigl\langle V(x), \nu (x) \bigr\rangle \biggl\vert \frac{\partial u}{\partial \nu } \biggr\vert ^{p}\,ds. $$

(2.3)

If $\lambda < 0$, (2.3) contradicts the fact $\langle V(x),\nu (x)\rangle >0$.

If $\lambda = 0$, (2.3) implies that $\frac{\partial u}{\partial \nu }=0$. By divergence theorem,

$$ \begin{aligned} \int _{\Omega }\Delta _{p} u\,dx&= \int _{\Omega }\operatorname{div} \bigl( \vert \nabla u \vert ^{p-2} \nabla u \bigr)\,dx \\ &= \int _{\partial \Omega } \bigl\langle \vert \nabla u \vert ^{p-2} \nabla u, \nu \bigr\rangle \,ds = \int _{\partial \Omega } \vert \nabla u \vert ^{p-2} \frac{\partial u}{\partial \nu }\,ds=0. \end{aligned} $$

On the other hand, the equation in (2.2) implies

$$ \int _{\Omega }\Delta _{p} u\,dx=- \int _{\Omega }u \vert u \vert ^{\alpha -1}\,dx< 0, $$

which is a contraction. Then, we complete the proof. □

By the way, we consider a special case of $p=2$ which was discussed in [28]. Take $p=2$ in Theorem 2.2 and Theorem 2.3, we have the following conclusions.

Corollary 2.4

(Pohozaev-type inequality for $p=2$)

Suppose that ${V(x)}$ is a linear vector field on $\mathbb{R}^{n}$ satisfying ${\operatorname{div} V(x)=n}$ and $\langle V(x),x\rangle >0$ for $\forall x\in \mathbb{R}^{n}\setminus \{0\}$. If $u\in W^{1,p}_{0}(\Omega )\cap C^{1}(\bar{\Omega })$ is a solution of (1.1) when $p=2$, then

$$ \begin{aligned} &(2\mu -n) \int _{\Omega } uf(u)\,dx+2n \int _{\Omega }F(u)\,dx \\ &\quad \geq \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl\vert \frac{\partial u}{\partial \nu } \biggr\vert ^{2}\,ds, \end{aligned} $$

(2.4)

where $\mu =\sup_{ \vert x \vert \neq 0}\frac{\langle V(x),x\rangle }{ \vert x \vert ^{2}}$.

Corollary 2.5

Suppose that the vector field ${V(x)}$ of Corollary 2.4is transverse to ∂Ω. Then (1.4) has no positive solution if

$$ \alpha \geq \frac{n+2\mu }{n-2\mu }\quad \textit{and} \quad \lambda \leq 0. $$

3 Examples

In this section, we construct two examples of domains in ${\mathbb {R}}^{3}$ which satisfy the conditions of our theorems but not non-star-shaped. The first example has been discussed in [28], the second one is new.

Example 1

(see [28])

Let $\Omega _{1}\subset {\mathbb {R}}^{3}$ be a bounded domain surrounded by the following closed surface $\Sigma _{1}$:

$$ {\Sigma _{1}:}\quad \textstyle\begin{cases} x_{1}=\sqrt{1-r^{2}} e^{\sin 2\theta }\cos (\theta -\sqrt{1-r^{2}} \sin 2\theta ), \\ x_{2}=\sqrt{1-r^{2}} e^{\sin 2\theta }\sin (\theta -\sqrt{1-r^{2}} \sin 2\theta ), & \theta \in [0,2\pi ], \\ x_{3}=r, & r^{2}\le 1. \end{cases} $$

(3.1)

In [28], the second author of the paper has proved that, for any given point $P\in {\mathbb {R}}^{3}$, the closed surface ${\Sigma _{1}}$ is not star-shaped with respect to P. Thus, $\Omega _{1}$ is not star-shaped. Subsequently, it is easy to verify that $\Omega _{1}$ satisfies the conditions of Theorem 2.2.

Next, we will give the specific graphic of the planar closed curve $\Gamma _{1}$ in [28], which is generated by the intersection of ${\Sigma _{1}}$ and Π, where $\Pi =\{x=(x_{1}, x_{2}, 0)\mid x_{1}\in {\mathbb {R}}^{1}, x_{2}\in {\mathbb {R}}^{1}\}$. Figure 1 is the graph of $\Gamma _{1}$.

Figure 1

The graph of $\Gamma_{1}$

Example 2

Let $\Omega _{2}\subset {\mathbb {R}}^{3}$ be a bounded domain surrounded by the following closed surface $\Sigma _{2}$:

$$ {\Sigma _{2}:}\quad \textstyle\begin{cases} x_{1}=\sqrt{1-r^{2}} e^{\cos 2\theta }\cos (\theta +\frac{1}{2}\sqrt{1-r^{2}} \sin 4\theta ) , \\ x_{2}=\sqrt{1-r^{2}} e^{\cos 2\theta }\sin (\theta +\frac{1}{2}\sqrt{1-r^{2}} \sin 4\theta ), & \theta \in [0,2\pi ], \\ x_{3}=r, & r^{2}\le 1. \end{cases} $$

(3.2)

Define a linear vector field V on ${\mathbb {R}}^{3}$ by

$$ V(x) = \begin{pmatrix} \frac{3}{7} & 0 & 0 \\ 0 & \frac{15}{7} & 0 \\ 0 & 0 & \frac{3}{7} \end{pmatrix}x. $$

It is easy to see that V is positive and $\sum_{i=1}^{3}a_{ii}=3$. By a direct computation, we can show that $V(x)$ is transverse to $\Sigma _{2}$. Therefore, $\Omega _{2}$ satisfies the conditions of Theorem 2.2.

Let $\Gamma _{2}$ be the curve given by the intersection of plane $\Pi =\{x=(x_{1}, x_{2}, 0)\mid x_{1}\in {\mathbb {R}}^{1}, x_{2}\in {\mathbb {R}}^{1}\}$ and $\Sigma _{2}$:

$$ {\Gamma _{2}:}\quad \textstyle\begin{cases} x_{1}=e^{\cos 2\theta }\cos (\theta +\frac{1}{2} \sin 4\theta ), \\ x_{2}=e^{\cos 2\theta }\sin (\theta +\frac{1}{2} \sin 4\theta ),&\theta \in [0,2\pi ], \\ x_{3}=0. \end{cases} $$

The graph of $\Gamma _{2}$ is presented in Fig. 2.

Figure 2

The graph of $\Gamma_{2}$

By the elementary method, we can show that $\Gamma _{2}$ is not star-shaped in the plane. Hence, $\Omega _{2}$ is not star-shaped.

4 The Pohozaev inequalities and the nonexistence of positive solutions for p-Laplace elliptic systems

In this section, we generalize the results of Pohozaev inequalities of Theorem 2.2 to system (1.2) to derive the nonexistence of positive solutions for the system with double critical exponent. Firstly, define the following Sobolev space:

$$ \widetilde{W}_{\Omega }=W_{0}^{1,p}(\Omega )\times W_{0}^{1,p}(\Omega ) $$

with the norm

$$ \bigl\Vert (u,v) \bigr\Vert = \biggl( \int _{\Omega } \vert \nabla u \vert ^{p}+ \vert \nabla v \vert ^{p} \biggr)^{\frac{1}{p}}. $$

Applying Theorem 2.2 to system (1.2), we get the following theorem.

Theorem 4.1

(Pohozaev inequalities)

Assume that $V(x)$ is a linear vector field on $\mathbb{R}^{N}$ of the form

$$ V(x)= \begin{pmatrix} a_{11} & \cdots & a_{1n} \\ \vdots & \vdots & \vdots \\ a_{n1} & \cdots & a_{nn} \end{pmatrix}x $$

for $\forall x\in \mathbb{R}^{N}\setminus \{0\}$, $V(x)$ satisfies ${\operatorname{div} V(x)=N}$ and $\langle V(x),x\rangle >0$. If ${(u,v)\in \widetilde{W}(\Omega )\cap C^{1}(\bar{\Omega })}$ is a solution of (1.2), then

$$ \begin{aligned} &(p\mu -N) \int _{\Omega } (u,v)\nabla F(u,v)\,dx+pN \int _{\Omega }F(u,v)\,dx \\ &\quad \geq (p-1) \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl( \biggl\vert \frac{\partial u}{\partial \nu } \biggr\vert ^{p}+ \biggl\vert \frac{\partial v}{\partial \nu } \biggr\vert ^{p} \biggr)\,ds, \end{aligned} $$

(4.1)

where $\mu =\sup_{ \vert x \vert \neq 0}\frac{\langle V(x),x\rangle }{ \vert x \vert ^{2}}$.

Proof

Applying Theorem 2.2 to the first equation of system (1.2), we have

$$ \begin{aligned} &(p\mu -N) \int _{\Omega } uf_{u}(u,v)\,dx+pN \int _{\Omega }F_{u}(u,v)\,dx \\ &\quad \geq (p-1) \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl\vert \frac{\partial u}{\partial \nu } \biggr\vert ^{p}\,ds. \end{aligned} $$

(4.2)

Similarly, applying Theorem 2.2 to the second equation of system (1.2), we have

$$ \begin{aligned} &(p\mu -N) \int _{\Omega } vf_{v}(u,v)\,dx+pN \int _{\Omega }F_{v}(u,v)\,dx \\ &\quad \geq (p-1) \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl\vert \frac{\partial v}{\partial \nu } \biggr\vert ^{p}\,ds. \end{aligned} $$

(4.3)

Adding (4.2) and (4.3), we obtain the desired result. □

Similarly, applying Theorem 2.2 and Theorem 2.2 in [12] to two equations in the system of (1.9), we can generalize the Pohozaev identity (1.10) of (p,q)-Laplace system in [16] to Pohozaev inequalities.

Theorem 4.2

(Pohozaev inequalities)

Suppose that $V(x)$ is a linear vector field on $\mathbb{R}^{N}$, and let

$$ V(x)= \begin{pmatrix} a_{11} & \cdots & a_{1n} \\ \vdots & \vdots & \vdots \\ a_{n1} & \cdots & a_{nn} \end{pmatrix}x. $$

$\forall x\in \mathbb{R}^{N}\setminus \{0\}$, $V(x)$ satisfies ${\operatorname{div} V(x)=N}$ and $\langle V(x),x\rangle >0$. If ${(u,v)\in \widetilde{W}(\Omega )\cap C^{1}(\bar{\Omega })}$ is a solution of (1.9), then

$$ \begin{aligned} &\frac{(p\mu -N)}{p} \int _{\Omega } (u,v)\nabla F(x,u,v)\,dx+ \frac{(q\mu -N)}{q} \int _{\Omega } (u,v)\nabla F(x,u,v)\,dx \\ &\quad\quad{} +N \int _{\Omega }F(x,u,v)\,dx+ \int _{\Omega }F_{1}(x,u,v)\,dx \\ &\quad \geq \biggl(1-\frac{1}{p} \biggr) \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl\vert \frac{\partial u}{\partial \nu } \biggr\vert ^{p}\,ds+ \biggl(1-\frac{1}{q} \biggr) \int _{ \partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl\vert \frac{\partial v}{\partial \nu } \biggr\vert ^{q}\,ds, \end{aligned} $$

(4.4)

where $\mu =\sup_{ \vert x \vert \neq 0}\frac{\langle V(x),x\rangle }{ \vert x \vert ^{2}}$.

Proof

Based on Theorem 2.2 and [12], and the equations of system (1.9), we have

$$ \begin{aligned} &(p\mu -N) \int _{\Omega } uF_{u}(x,u,v)\,dx+pN \int _{\Omega }F^{1}_{u}(x,u,v)\,dx+p \int _{\Omega }F_{1,u}(x,u,v)\,dx \\ &\quad \geq (p-1) \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl\vert \frac{\partial u}{\partial \nu } \biggr\vert ^{p}\,ds \end{aligned} $$

(4.5)

and

$$ \begin{aligned} &(q\mu -N) \int _{\Omega } vF_{v}(x,u,v)\,dx+qN \int _{\Omega }F^{1}_{v}(x,u,v)\,dx+q \int _{\Omega }F_{1,v}(x,u,v)\,dx \\ &\quad \geq (q-1) \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl\vert \frac{\partial v}{\partial \nu } \biggr\vert ^{q}\,ds, \end{aligned} $$

(4.6)

where $\frac{\partial F^{1}_{u}(x,u,v)}{\partial u}=f_{u}(x,u,v)$, and $\frac{\partial F^{1}_{v}(x,u,v)}{\partial u}=f_{v}(x,u,v)$. Adding (4.5) and (4.6), we complete the proof. □

Remark 1

The Pohozaev inequalities in Theorem 4.2 can also be written in the following forms:

$$ \begin{aligned} &\frac{(p\mu -N)}{p} \int _{\Omega } \vert \nabla u \vert ^{p}\,dx+ \frac{(q\mu -N)}{q} \int _{\Omega } \vert \nabla v \vert ^{q} \,dx+N \int _{\Omega }F(x,u,v)\,dx+ \int _{\Omega }F_{1}(x,u,v)\,dx \\ &\quad \geq \biggl(1-\frac{1}{p} \biggr) \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \vert \nabla u \vert ^{p}\,ds+ \biggl(1-\frac{1}{q} \biggr) \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \vert \nabla v \vert ^{q}\,ds. \end{aligned} $$

Now, we discuss the nonexistence of positive solutions for system (1.11).

Theorem 4.3

Suppose that the vector field ${V(x)}$ of Theorem 4.1is transverse to ∂Ω. Then (1.11) has no positive solution.

Proof

Let $F(u,v)=\frac{1}{p^{*}}\int _{\Omega }( \vert u \vert ^{p^{*}}+ \vert v \vert ^{p^{*}}+ \vert u \vert ^{ \alpha } \vert v \vert ^{\beta })\,dx$, by Theorem 4.3, the right-hand side of the Pohozaev inequalities can be written as follows:

$$ \biggl(1-\frac{1}{p} \biggr) \int _{\partial \Omega } \bigl\langle V(x),\nu (x) \bigr\rangle \biggl( \biggl\vert \frac{\partial u}{\partial \nu } \biggr\vert ^{p}+ \biggl\vert \frac{\partial v}{\partial \nu } \biggr\vert ^{p} \biggr)\,ds \geq 0, $$

the left-hand side is

$$ \begin{aligned} &= \biggl(\mu -\frac{N}{p} \biggr) \int _{\Omega } \bigl( \vert \nabla u \vert ^{p}+ \vert \nabla v \vert ^{p} \bigr)\,dx+\frac{N}{p^{*}} \int _{\Omega } \bigl( \vert u \vert ^{p^{*}}+ \vert v \vert ^{p^{*}}+ \vert u \vert ^{ \alpha } \vert v \vert ^{\beta } \bigr)\,dx \\ &\leq \biggl(\mu -\frac{N}{p} \biggr) \int _{\Omega } \bigl( \vert \nabla u \vert ^{p}+ \vert \nabla v \vert ^{p} \bigr)\,dx \\ &< 0. \end{aligned} $$

This is a contradiction, so (1.11) has no positive solution. □

5 The existence of positive solutions for a kind of p-Laplace elliptic system

This section studies the existence of positive solutions of (1.11).

Consider the functional $J: \widetilde{W}_{\Omega }\rightarrow \mathbb{R}$, $(u,v)\in \widetilde{W}_{\Omega }$,

$$ J(u,v)=\frac{1}{p} \int _{\Omega } \bigl( \vert \nabla u \vert ^{p}+ \vert \nabla v \vert ^{p}\,dx \bigr)- \frac{1}{p^{*}} \int _{\Omega } \bigl( \bigl(u^{+} \bigr)^{p^{*}}+ \bigl(v^{+} \bigr)^{p^{*}}+ \bigl(u^{+} \bigr)^{ \alpha } \bigl(v^{+} \bigr)^{\beta } \bigr)\,dx, $$

and $(u,v)\in \widetilde{W}_{\mathbb{R}^{N}}$,

$$ J^{\infty }(u,v)=\frac{1}{p} \int _{\mathbb{R}^{N}} \bigl( \vert \nabla u \vert ^{p}+ \vert \nabla v \vert ^{p}\,dx \bigr)-\frac{1}{p^{*}} \int _{\mathbb{R}^{N}} \bigl( \bigl(u^{+} \bigr)^{p^{*}}+ \bigl(v^{+} \bigr)^{p^{*}}+ \bigl(u^{+} \bigr)^{ \alpha } \bigl(v^{+} \bigr)^{\beta } \bigr)\,dx. $$

By Theorem 1.7 in [27] and Lemma 3.4.5 in [29], we have the global compactness results as follows.

Lemma 5.1

(Global compactness result, see [27])

Let ${(u_{n},v_{n})}$ be a Palais–Smale sequence for $\widetilde{W}_{\Omega }$. Then there exists a solution $(u_{0},v_{0})$ of system (1.11), m sequences of positive numbers ${r_{n}^{i}}$ ($1 \leq i \leq m$), and m sequences of points ${x_{n}^{i}}$ ($1 \leq i \leq m$), when $n\rightarrow \infty$, such that up to a subsequence:

$$\begin{aligned}& (1)\quad (u_{n},v_{n})=(u_{0},v_{0})+ \sum_{i = 1}^{m} {{{\bigl(r_{n}^{i} \bigr)}^{ \frac{{N - p}}{p}}}\bigl({U_{i}}\bigl(r_{n}^{i} \bigl(x - x_{n}^{i}\bigr)\bigr),{V_{i}} \bigl(r_{n}^{i}\bigl(x - x_{n}^{i}\bigr) \bigr)\bigr)} + \bigl(\sigma _{n}^{1} + \sigma _{n}^{2}\bigr), \end{aligned}$$

where $r_{n}^{i}\rightarrow \infty$, $\Vert (\sigma _{n}^{1},\sigma _{n}^{2}) \Vert \rightarrow 0$;

$$\begin{aligned}& (2)\quad J(u_{n},v_{n})=J(u_{0},v_{0})+ \sum_{i = 1}^{m} {{E^{\infty }}({U_{i}},{V_{i}})} + o(1). \end{aligned}$$

By Lemma 5.1, we have the existence of positive solutions of (1.11).

Theorem 5.2

Assume that $\Omega \subset \mathbb{R}^{N}$ is a bounded domain and there exists $0< R_{1}< R_{2}<+\infty$ satisfying

$$\begin{aligned}& \bigl\{ x\in \mathbb{R}^{N}:R_{1}< \vert x \vert < R_{2} \bigr\} \subset \Omega , \\& \bigl\{ x\in \mathbb{R}^{N}: \vert x \vert < R_{1} \bigr\} \not \subset \overline{\Omega }. \end{aligned}$$

Then problem (1.11) admits a positive solution when $\frac{R_{1}}{R_{2}}$ is sufficiently large.

Proof

Using a similar method as that in [27], let

$$\begin{aligned}& J_{1}(u,v)= \int _{\Omega } \vert \nabla u \vert ^{p}+ \vert \nabla v \vert ^{p} \\& S: = \inf_{u \in {W^{1,p}}(\mathbb{R}^{N}) \backslash \{ 0\} } \frac{{{{ \Vert u \Vert }^{p}}}}{{ \vert u \vert _{{p^{*}}}^{p}}} \qquad S_{0}:=f( \tau )S, \end{aligned}$$

where $f(\tau ):= \frac{1+\tau ^{p}}{(1+\tau ^{\beta }+\tau ^{p^{*}})^{\frac{p}{p^{*}}}}$.

$$ \widetilde{M}: = \biggl\{ (u,v)\in \widetilde{W}_{\Omega }: \int _{\Omega } \bigl( \bigl(u^{+} \bigr)^{p^{*}}+ \bigl(v^{+} \bigr)^{p^{*}} + \bigl(u^{+} \bigr)^{\alpha } \bigl(v^{+} \bigr)^{\beta } \bigr)\,dx=1 \biggr\} . $$

By similar methods as those in [27], if we choose $\varepsilon >0$ small enough, $J_{1}$ has no critical value in $(S_{0},S_{0}+ \varepsilon ]$. However, if $J_{1}$ does not have a critical point, then the unit sphere is contractible in Ω, which contradicts $\{x\in \mathbb{R}^{N}: \vert x \vert < R_{1}\}\not \subset \overline{\Omega }$. This implies that $J_{1}$ has critical points, and so the system has nontrivial positive solutions. Without loss of generality, let $R_{1}=\frac{1}{4R}<1$, $R_{2}=4R>1$. For $\forall \sigma \in S_{n}:=\{x\in \mathbb{R}^{N}\mid \vert x \vert =1\}$ (unit sphere), $0\leq t\leq 1$, we denote

$$ u_{t}^{\sigma }(x)= \biggl( \frac{(1-t)^{\frac{1}{p-1}}}{(1-t)^{\frac{p}{p-1}}+(x-\sigma t)^{\frac{p}{p-1}}} \biggr)^{ \frac{N-p}{p}}\quad\quad \in W^{1,p} \bigl(\mathbb{R}^{N} \bigr). $$

In order to make $\omega _{t}^{\sigma }\in W_{0}^{1,p}(\Omega )$ for every $u_{t}^{\sigma }(x)$, and $\int _{\mathbb{R}^{N}} \vert \nabla (u_{t}^{\sigma },\tau u_{t}^{\sigma })-( \omega _{t}^{\sigma },\tau \omega _{t}^{\sigma }) \vert ^{p}\,dx$ is small enough, we choose a radially symmetric function $\phi \in C_{0}^{\infty }$ as follows:

$$ {\phi _{R}}(x) = \textstyle\begin{cases} {\phi (Rx)}&{0 \le \vert x \vert \le \frac{1}{R}}, \\ 1&{\frac{1}{R} \le \vert x \vert \le R} , \\ {\phi (\frac{x}{R})}&{ \vert x \vert \ge R}, \end{cases} $$

where ϕ satisfies: $0\leq \phi \leq 1$, when $x\in \Omega$; $\phi =1$, when $\frac{1}{2}< \vert x \vert <2$; $\phi =0$, when x is not in $\frac{1}{4}< \vert x \vert <4$. So $\phi _{R}(x)\in C_{0}^{\infty }(\Omega )$, and set $\omega _{t}^{\sigma }=u_{t}^{\sigma }(x)\phi _{R}(x)$, then

$$\begin{aligned} & \bigl\Vert \bigl(u_{t}^{\sigma }, \tau u_{t}^{\sigma } \bigr)- \bigl(\omega _{t}^{ \sigma }, \tau \omega _{t}^{\sigma } \bigr) \bigr\Vert _{\widetilde{W}(\mathbb{R}^{N})}^{p} \\ &\quad = \bigl(1+\tau ^{2} \bigr)^{\frac{p}{2}} \int _{\mathbb{R}^{N}} \bigl\vert \nabla \bigl(u_{t}^{ \sigma }( \phi _{R}-1) \bigr) \bigr\vert ^{p}\,dx \\ &\quad \leq 2^{p} \bigl(1+\tau ^{2} \bigr)^{\frac{p}{2}} \biggl[ \int _{\mathbb{R}^{N}} \bigl\vert \nabla u_{t}^{ \sigma } \bigr\vert ^{p}(\phi _{R}-1)^{p}+ \int _{\mathbb{R}^{N}} \bigl\vert u_{t}^{\sigma } \bigr\vert ^{p} \vert \nabla \phi _{R} \vert ^{p} \biggr] \\ &\quad \leq C[ \int _{B_{2R}^{C}\cup B_{\frac{1}{2R}}} \bigl\vert \nabla u_{t}^{ \sigma } \bigr\vert ^{p}\,dx+\frac{1}{R^{p}} \int _{B_{4R}\setminus B_{2R}} \bigl\vert \nabla u_{t}^{ \sigma } \bigr\vert ^{p}\,dx+ \int _{B_{\frac{1}{2R}}} \bigl\vert u_{t}^{\sigma } \bigr\vert ^{p}\,dx \\ &\quad :=A+B+C. \end{aligned}$$

Obviously, $A\rightarrow 0$, when $R\rightarrow \infty$.

$$ B\leq \frac{C}{R^{p}} \biggl( \int _{B_{4R}\setminus B_{2R}} \bigl\vert u_{t}^{\sigma } \bigr\vert ^{p^{*}}\,dx \biggr)R^{1- \frac{p}{p^{*}}}\rightarrow 0. $$

Denote

$$ v_{t}^{\sigma }= \frac{\omega _{t}^{\sigma }}{ \Vert (\omega _{t}^{\sigma },\tau \omega _{t}^{\sigma }) \Vert _{L^{p^{*}}}}, $$

where $\Vert (\omega _{t}^{\sigma },\tau \omega _{t}^{\sigma }) \Vert _{L^{p^{*}}}^{p^{*}}= \int _{\Omega }(u^{+})^{p^{*}} +(v^{+})^{p^{*}}+(u^{+})^{\alpha }(v^{+})^{ \beta }$, $\lim_{R \to \infty } \int _{\mathbb{R}^{N}} \vert \nabla v_{t}^{\sigma }-\nabla \frac{u_{t}^{\sigma }}{ \Vert u_{t}^{\sigma } \Vert _{L^{p^{*}}}} \vert \,dx\rightarrow 0$, $(v_{t}^{\sigma }, \tau v_{t}^{\sigma })(v_{0},\tau v_{0})\in \widetilde{M}$; hence $J_{1}(v_{t}^{\sigma }, \tau v_{t}^{\sigma })\rightarrow S_{0}$.

Therefore, we just have to prove:

There exists $(u_{0},v_{0})\in \widetilde{M}$ such that $J_{1}(u_{0},v_{0})< S_{0}+\varepsilon$, $J'_{1}(u_{0},v_{0})=0$.

We prove the problem by reduction to absurdity. By Lemma 5.1, $J_{1}$ does not have any Palais–Smale sequence at level $(S_{0}, S_{0}+\varepsilon ]$. By the deformation lemma in [21], $\exists \varepsilon >0$, and a flow $\Phi :\widetilde{M}\times [0,1]\rightarrow \widetilde{M}$, such that

$$ \Phi (\widetilde{M}_{c+\varepsilon },1)\subset \widetilde{M}_{c- \varepsilon }, $$

where $\widetilde{M}_{c}=\{(u,v)\in \widetilde{M}:J_{1}(u,v)< c\}$.

By the deformation lemma, for given $\delta >0$, there exists a flow $\Phi :\widetilde{M}\times [0,1]\rightarrow \widetilde{M}$ such that

$$\begin{aligned}& \Phi (\widetilde{M}_{S_{1}},1)\subset \widetilde{M}_{S_{0}+\delta } \\& \Phi \bigl((u,v),t \bigr)=(u,v), \quad \forall (u,v)\in \widetilde{M}_{S_{0}+ \frac{\delta }{2}}. \end{aligned}$$

Let

$$ J_{2}(u,v)= \int _{\Omega }x \bigl( \bigl(u^{+} \bigr)^{p^{*}}+ \bigl(v^{+} \bigr)^{p^{*}} + \bigl(u^{+} \bigr)^{ \alpha } \bigl(v^{+} \bigr)^{\beta } \bigr)\,dx. $$

By Ekeland’s variational principle, there exists $x_{1}\in \overline{\Omega }$ such that, as $n\rightarrow \infty$,

$$\begin{aligned}& \bigl( \bigl(u_{n}^{+} \bigr)^{p^{*}}+ \bigl(v_{n}^{+} \bigr)^{p^{*}} + \bigl(u_{n}^{+} \bigr)^{\alpha } \bigl(v_{n}^{+} \bigr)^{ \beta } \bigr)\rightharpoonup \delta _{x_{1}}, \\& \vert \nabla u \vert ^{p}+ \vert \nabla u \vert ^{p} \rightharpoonup S_{0}\delta _{x_{1}}. \end{aligned}$$

That shows $J_{2}(\widetilde{M}_{S_{0}+\delta })\subset U$ for any given neighborhood $U \subset \Omega$.

Assume that the projection E of a neighborhood U of Ω̅ is continuous, and $E:\overline{U}\rightarrow \overline{\Omega }$. Let

$$ h:\quad S_{n}\times [0,1]\rightarrow \Omega , \qquad h(\sigma , t)=E \bigl(J_{2} \bigl( \Phi \bigl( \bigl(v_{t}^{\sigma }, \tau v_{t}^{\sigma } \bigr),1 \bigr) \bigr) \bigr) $$

because

$$ \lim_{t \to 1} { \int _{\Omega }{ \bigl\vert {\nabla v_{t}^{\sigma }} \bigr\vert } ^{p}}\,dx = {S_{0}}. $$

Hence, $v_{t}^{\sigma }\in \widetilde{M}_{S_{0}+\frac{\delta }{2}}$, so we have

$$ \Phi \bigl(v_{t}^{\sigma },1 \bigr) = v_{t}^{\sigma }. $$

Therefore,

$$ \lim_{t \to 1} {J_{2}} \bigl(\Phi \bigl(v_{t}^{\sigma },1 \bigr) \bigr) = \lim_{t \to 1} {J_{2}} \bigl(v_{t}^{\sigma } \bigr) = \sigma \in S_{n}\subset \Omega . $$

It shows that h is continuous and satisfies $h(\sigma ,1)=\sigma$, and for some $x_{2}$,

$$ h(\sigma , 0)=E \bigl(J_{2} \bigl(\Phi (v_{0},1) \bigr) \bigr)=x_{2}\in \Omega . $$

Hence $S_{n}$ can contract to $x_{2}$ in Ω̅, it is a contradiction. By the strong maximum principle, we have $u_{0}>0$, $v_{0}>0$. So, we have completed the proof. □