Positive symmetric solutions for a class of critical quasilinear elliptic problems in ${\mathbb{R}}^N$

Abstract

This paper deals with the critical quasilinear elliptic problem $-{\mathrm{\Delta }}_{p}u=\mu \frac{{|u|}^{p-2}u}{{|x|}^p}+Q(x)\frac{{|u|}^{p^{\ast }(s)-2}u}{{|x|}^s}+h(x){|u|}^{q-2}u$ in ${\mathbb{R}}^N$, where ${\mathrm{\Delta }}_{p}u=div({|\mathrm{\nabla }u|}^{p-2}u)$ is the p-Laplacian, $1<p<N$, $0\le \mu <\overline{\mu }$ with $\overline{\mu }={(\frac{N-p}{p})}^p$, $0\le s<p<q<p^{\ast }(s)$, $p^{\ast }(s)=\frac{(N-s)p}{N-p}$, and Q and h are measurable functions satisfying some symmetry conditions with respect to a closed subgroup G of $O(\mathbb{N})$. By variational methods and the symmetric criticality principle of Palais, we establish several existence and multiplicity results of positive G-symmetric solutions under certain appropriate hypotheses on Q, h, and q.

MSC: 35J25, 35J60, 35J65.

1 Introduction

In recent years, considerable attention has been paid to the following nonlinear elliptic problem with singular potential and critical Sobolev exponent:

$$\{\begin{array}{cc}-\mathrm{\Delta }u=\mu \frac{u}{{|x|}^2}+{|u|}^{2^{\ast }-2}u+f(x,u),\hfill & \text{in }\mathrm{\Omega },\hfill \\ u=0,\hfill & \text{on }\partial \mathrm{\Omega },\hfill \end{array}$$

(1.1)

where $\mathrm{\Omega }\subset {\mathbb{R}}^N$ ($N\ge 3$) is a smooth domain (bounded or unbounded) containing the origin, $0\le \mu <{(\frac{N-2}{2})}^2$, $2^{\ast }\triangleq \frac{2N}{N-2}$ is the critical Sobolev exponent, and $f:\mathrm{\Omega }\times \mathbb{R}\mapsto \mathbb{R}$ is a measurable function with subcritical growth. The main reason of interest in singular potentials relies in their criticality: they have the same homogeneity as the Laplacian and the critical Sobolev exponent and do not belong to the Kato class, hence they cannot be regarded as the lower order perturbation terms. We also mention that (1.1) is related to applications in many physical contexts: fluid mechanics, glaciology, molecular physics, quantum cosmology and linearization of combustion models (see [1] for example). So for this reason, many existence, nonexistence, and multiplicity results for equations like (1.1) have been obtained with various hypotheses on the measurable function $f(x,u)$; we refer the readers to [2]–[8] and the references therein. Moreover, for other results on this aspect, see [9] for boundary singularities, [10] for high-order nonlinearity, [11] for non-autonomous Schrödinger-Poisson systems in ${\mathbb{R}}^3$, [12] for singular elliptic systems in ${\mathbb{R}}^2$, and [13] for large singular sensitivity etc.

Recently, Deng and Jin [14] studied the existence of nontrivial solutions of the following singular semilinear elliptic problem:

$$-\mathrm{\Delta }u=\mu \frac{u}{{|x|}^2}+k(x)\frac{u^{2^{\ast }(s)-1}}{{|x|}^s},\text{and}u>0\text{in }{\mathbb{R}}^N,$$

(1.2)

where $\mu \in [0,{(\frac{N-2}{2})}^2)$, $s\in [0,2)$, are constants, $N>2$, $2^{\ast }(s)=\frac{2(N-s)}{N-2}$, and k fulfills certain symmetry conditions with respect to a subgroup G of $O(\mathbb{N})$. By the concentration-compactness principle in Refs. [15], [16] and variational methods, the authors obtained the existence and multiplicity of G-symmetric solutions under some assumptions on k. Very recently, Deng and Huang [17]–[19] extended the results in Ref. [14] to nonlinear singular elliptic problems in a bounded G-symmetric domain. We also mention that when $\mu =s=0$ and the right-hand side term ${|x|}^{-s}{u}^{2^{\ast }(s)-1}$ is replaced by $u^{r-1}$ ($1<r<\frac{2N}{N-2}$ or $r=\frac{2N}{N-2}$) in (1.2), the existence and multiplicity of G-symmetric solutions of (1.2) were obtained in Refs. [20]–[22]. Finally, when $G=O(\mathbb{N})$, we remark that Su and Wang [23] established the existence of nontrivial radial solutions for a class of quasilinear singular equations such as (1.2) by proving several new embedding theorems.

Motivated by Deng and Jin [14], Bianchi et al.[20], and Su and Wang [23], in this work we investigate the following critical quasilinear problem with singular potential:

$$-{\mathrm{\Delta }}_{p}u=\mu \frac{{|u|}^{p-2}u}{{|x|}^p}+Q(x)\frac{{|u|}^{p^{\ast }(s)-2}u}{{|x|}^s}+h(x){|u|}^{q-2}u\text{in }{\mathbb{R}}^N,$$

(1.3)

where ${\mathrm{\Delta }}_{p}u=div({|\mathrm{\nabla }u|}^{p-2}u)$ is the p-Laplacian, $1<p<N$, $0\le \mu <\overline{\mu }$, with $\overline{\mu }\triangleq {(\frac{N-p}{p})}^p$, $0\le s<p<q<p^{\ast }(s)$, $p^{\ast }(s)\triangleq \frac{(N-s)p}{N-p}$ is the critical Hardy-Sobolev exponent and $p^{\ast }(0)=p^{\ast }\triangleq \frac{Np}{N-p}$ is the critical Sobolev exponent; Q and h are G-symmetric functions (see Section 2 for details) satisfying some appropriate conditions which will be specified later. Problem (1.3) is in fact a continuation of (1.2). However, due to the nonlinear perturbation $h(x){|u|}^{q-2}u$ and the singularities caused by the terms $\frac{1}{{|x|}^p}$ and $\frac{1}{{|x|}^s}$, compared with the semilinear equation (1.2), the critical quasilinear equation (1.3) becomes more complicated to deal with and we have to overcome more difficulties in the study of G-symmetric solutions. As far as we know, there are few results on the existence of G-symmetric solutions for (1.3) as $\mu \ne 0$, $p\ne 2$, and $h\not\equiv 0$. Hence, it makes sense for us to investigate problem (1.3) thoroughly. Let $\overline{Q}>0$ be a constant. Note that here we will try to treat both the cases of $h=0$, $Q(x)\not\equiv \overline{Q}$, and $h\ne 0$, $Q(x)\equiv \overline{Q}$.

This paper is organized as follows. In Section 2, we will establish the appropriate Sobolev space which is applicable to the study of problem (1.3), and we will state the main results of this paper. In Section 3, we detail the proofs of some existence and multiplicity results for the cases $h=0$ and $Q(x)\not\equiv \overline{Q}$ in (1.3). In Section 4, we give the proofs of existence results for the cases $h\ne 0$ and $Q(x)\equiv \overline{Q}$ in (1.3). Our methods in this paper are mainly based upon the symmetric criticality principle of Palais (see [24]) and variational arguments.

2 Preliminaries and main results

Let $O(\mathbb{N})$ be the group of orthogonal linear transformations of ${\mathbb{R}}^N$ with natural action and let $G\subset O(\mathbb{N})$ be a closed subgroup. For $x\ne 0$ we denote the cardinality of $G_x=\{gx;g\in G\}$ by $|G_x|$ and set $|G|={inf}_{0\ne x\in {\mathbb{R}}^N}|G_x|$. Note that here $|G|$ may be +∞. For any function $f:{\mathbb{R}}^N\to \mathbb{R}$, We call $f(x)$ a G-symmetric function if for all $g\in G$ and $x\in {\mathbb{R}}^N$, $f(gx)=f(x)$ holds. In particular, if f is radially symmetric, then the corresponding group G is $O(\mathbb{N})$ and $|G|=+\mathrm{\infty }$. Other further examples of G-symmetric functions can be found in Ref. [14].

Let ${\mathcal{D}}^{1,p}({\mathbb{R}}^N)$ denote the closure of ${\mathcal{C}}_0^{\mathrm{\infty }}({\mathbb{R}}^N)$ functions with respect to the norm ${({\int }_{{\mathbb{R}}^N}{|\mathrm{\nabla }u|}^{p}dx)}^{1/p}$. We recall that the well-known Hardy-Sobolev inequality (see [3], [25]) asserts that for all $u\in {\mathcal{D}}^{1,p}({\mathbb{R}}^N)$, there is a constant $C=C(N,p,s)>0$ such that

$${\left({\int }_{{\mathbb{R}}^N}\frac{{|u|}^{p^{\ast }(s)}}{{|x|}^s}dx\right)}^{p/p^{\ast }(s)}\le C{\int }_{{\mathbb{R}}^N}{|\mathrm{\nabla }u|}^{p}dx,$$

(2.1)

where $1<p<N$, $0\le s\le p$, and $p^{\ast }(s)=\frac{(N-s)p}{N-p}$. If $s=p$, then $p^{\ast }(s)=p$ and the following Hardy inequality holds (see [2], [25]):

$${\int }_{{\mathbb{R}}^N}\frac{{|u|}^p}{{|x|}^p}dx\le \frac{1}{\overline{\mu }}{\int }_{{\mathbb{R}}^N}{|\mathrm{\nabla }u|}^{p}dx,\mathrm{\forall }u\in {\mathcal{D}}^{1,p}\left({\mathbb{R}}^N\right),$$

(2.2)

where $\overline{\mu }={(\frac{N-p}{p})}^p$. Now we employ the following norm in ${\mathcal{D}}^{1,p}({\mathbb{R}}^N)$:

$${\parallel u\parallel }_{\mu }\triangleq {\left[{\int }_{{\mathbb{R}}^N}({|\mathrm{\nabla }u|}^p-\mu \frac{{|u|}^p}{{|x|}^p})dx\right]}^{1/p},0\le \mu <\overline{\mu }.$$

By the Hardy inequality (2.2), we easily see that the above norm is equivalent to the usual norm ${({\int }_{{\mathbb{R}}^N}{|\mathrm{\nabla }u|}^{p}dx)}^{1/p}$.

The natural functional space to study problem (1.3) is the Banach space ${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$ which is the subspace of ${\mathcal{D}}^{1,p}({\mathbb{R}}^N)$ consisting of all G-symmetric functions. In this paper we consider the following problems:

$$\left({\mathcal{P}}_h^Q\right)\{\begin{array}{cc}-{\mathrm{\Delta }}_{p}u=\mu \frac{{|u|}^{p-2}u}{{|x|}^p}+Q(x)\frac{{|u|}^{p^{\ast }(s)-2}u}{{|x|}^s}+h(x){|u|}^{q-2}u,\hfill & \text{in }{\mathbb{R}}^N,\hfill \\ u\in {\mathcal{D}}_G^{1,p}({\mathbb{R}}^N),\text{ and }u>0,\hfill & \text{in }{\mathbb{R}}^N.\hfill \end{array}$$

To mention our main results, we need to introduce two notations ${\mathcal{A}}_{\mu }$ and $y_{ϵ}(x)$, which are, respectively, defined by

$${\mathcal{A}}_{\mu }\triangleq \underset{u\in {\mathcal{D}}^{1,p}({\mathbb{R}}^N)\mathrm{\setminus }\{0\}}{inf}\frac{{\int }_{{\mathbb{R}}^N}({|\mathrm{\nabla }u|}^p-\mu \frac{{|u|}^p}{{|x|}^p})dx}{{({\int }_{{\mathbb{R}}^N}{|x|}^{-s}{|u|}^{p^{\ast }(s)}dx)}^{\frac{p}{p^{\ast }(s)}}}$$

(2.3)

and

$$y_{ϵ}(x)=C{ϵ}^{-\frac{N-p}{p}}{U}_{\mu }\left(\frac{|x|}{ϵ}\right),$$

(2.4)

where $ϵ>0$, and the constant $C=C(N,p,\mu ,s)>0$, depending only on N, p, μ, and s. From Kang [5], we see that $y_{ϵ}(x)$ satisfies the equations

$${\int }_{{\mathbb{R}}^N}({|\mathrm{\nabla }{y}_{ϵ}|}^p-\mu {|x|}^{-p}{|y_{ϵ}|}^p)dx=1$$

(2.5)

and

$${\int }_{{\mathbb{R}}^N}{|x|}^{-s}{y}_{ϵ}^{p^{\ast }(s)-1}\nu dx={\mathcal{A}}_{\mu }^{-\frac{p^{\ast }(s)}{p}}{\int }_{{\mathbb{R}}^N}({|\mathrm{\nabla }{y}_{ϵ}|}^{p-2}\mathrm{\nabla }{y}_{ϵ}\mathrm{\nabla }\nu -\mu {|x|}^{-p}{|y_{ϵ}|}^{p-2}{y}_{ϵ}\nu )dx$$

for all $\nu \in {\mathcal{D}}^{1,p}({\mathbb{R}}^N)$. In particular, we have (let $\nu =y_{ϵ}$)

$${\int }_{{\mathbb{R}}^N}{|x|}^{-s}{y}_{ϵ}^{p^{\ast }(s)}dx={\mathcal{A}}_{\mu }^{-\frac{N-s}{N-p}}.$$

(2.6)

The function $U_{\mu }(x)=U_{\mu }(|x|)$ in (2.4) is the unique radial solution of the following limiting problem (see [5], Lemma 2.3]):

$$\{\begin{array}{cc}-{\mathrm{\Delta }}_{p}u=\mu \frac{u^{p-1}}{{|x|}^p}+\frac{u^{p^{\ast }(s)-1}}{{|x|}^s},\hfill & \text{in }{\mathbb{R}}^N\mathrm{\setminus }\{0\},\hfill \\ u\in {\mathcal{D}}^{1,p}({\mathbb{R}}^N),\text{ and }u>0,\hfill & \text{in }{\mathbb{R}}^N\mathrm{\setminus }\{0\},\hfill \end{array}$$

satisfying

$$U_{\mu }(1)={\left(\frac{(N-s)(\overline{\mu }-\mu )}{N-p}\right)}^{\frac{1}{p^{\ast }(s)-p}}.$$

Moreover, the following asymptotic properties at the origin and infinity for $U_{\mu }(r)$ and $U_{\mu }^{\prime }(r)$ hold [5]:

$$\underset{r\to 0}{lim}{r}^{{\eta }_1}{U}_{\mu }(r)={\overline{C}}_1>0,\underset{r\to 0}{lim}{r}^{{\eta }_1+1}|U_{\mu }^{\prime }(r)|={\overline{C}}_1{\eta }_1>0,$$

(2.7)

$$\underset{r\to +\mathrm{\infty }}{lim}{r}^{{\eta }_2}{U}_{\mu }(r)={\overline{C}}_2>0,\underset{r\to +\mathrm{\infty }}{lim}{r}^{{\eta }_2+1}|U_{\mu }^{\prime }(r)|={\overline{C}}_2{\eta }_2>0,$$

(2.8)

where ${\overline{C}}_1$, ${\overline{C}}_2$ are positive constants and ${\eta }_1={\eta }_1(N,p,\mu )$, ${\eta }_2={\eta }_2(N,p,\mu )$ are the zeros of the function

$$\eta (t)=(p-1)t^p-(N-p)t^{p-1}+\mu ,t\ge 0,0\le \mu <\overline{\mu },$$

which satisfy

$$0\le {\eta }_1<\frac{N-p}{p}<{\eta }_2\le \frac{N-p}{p-1}.$$

(2.9)

We suppose that $Q(x)$ and $h(x)$ fulfill the following conditions.

(q.1)$Q\in \mathcal{C}({\mathbb{R}}^N)\cap L^{\mathrm{\infty }}({\mathbb{R}}^N)$, and $Q(x)$ is G-symmetric.

(q.2)$Q_{+}\not\equiv 0$, where $Q_{+}=max\{0,Q\}$.

(h.1)$h(x)$ is G-symmetric.

(h.2)$h(x)$ is nonnegative and locally bounded in ${\mathbb{R}}^N\mathrm{\setminus }\{0\}$, $h(x)=O({|x|}^{-s})$ in the bounded neighborhood $\mathcal{O}$ of the origin, $h(x)=O({|x|}^{-\vartheta })$ as $|x|\to \mathrm{\infty }$, $0\le s<\vartheta <p$, $p^{\ast }(\vartheta )<q<p^{\ast }(s)$, where $p^{\ast }(\vartheta )=\frac{(N-\vartheta )p}{N-p}$.

The main results of this paper are the following.

Theorem 2.1

Suppose that (q.1) and (q.2) hold. If

$${\int }_{{\mathbb{R}}^N}Q(x)\frac{y_{ϵ}^{p^{\ast }(s)}}{{|x|}^s}dx\ge max\{\frac{Q_{+}(0)}{{\mathcal{A}}_{\mu }^{\frac{N-s}{N-p}}},\frac{Q_{+}(\mathrm{\infty })}{{\mathcal{A}}_{\mu }^{\frac{N-s}{N-p}}},\frac{{\parallel Q_{+}\parallel }_{\mathrm{\infty }}}{{|G|}^{\frac{p-s}{N-p}}{\mathcal{A}}_0^{\frac{N-s}{N-p}}}\}>0$$

(2.10)

for some$ϵ>0$, where$Q_{+}(\mathrm{\infty })={lim\hspace{0.17em}sup}_{|x|\to \mathrm{\infty }}{Q}_{+}(x)$, then problem (${\mathcal{P}}_0^Q$) has at least one positive solution in${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$.

Corollary 2.1

Suppose that (q.1) and (q.2) hold. Then we have the following statements.

• (1) Problem (${\mathcal{P}}_0^Q$) has a positive solution if

  $$Q(0)>0,Q(0)\ge max\{Q_{+}(\mathrm{\infty }),{({\mathcal{A}}_{\mu }/{\mathcal{A}}_0)}^{\frac{N-s}{N-p}}{|G|}^{\frac{s-p}{N-p}}{\parallel Q_{+}\parallel }_{\mathrm{\infty }}\}$$

and either (i) $Q(x)\ge Q(0)+\sigma {|x|}^{-N+s+{\eta }_2{p}^{\ast }(s)}$for some$\sigma >0$and$|x|$small or (ii) $|Q(x)-Q(0)|\le C{|x|}^{\alpha }$for some constant$C>0$, $\alpha >-N+s+{\eta }_2{p}^{\ast }(s)$, $|x|$small, and

$${\int }_{{\mathbb{R}}^N}(Q(x)-Q(0)){|x|}^{-s-{\eta }_2{p}^{\ast }(s)}dx>0.$$

(2.11)

• (2) Problem (${\mathcal{P}}_0^Q$) admits at least one positive solution if ${lim}_{|x|\to \mathrm{\infty }}Q(x)=Q(\mathrm{\infty })$ exists and is positive,

  $$Q(\mathrm{\infty })\ge max\{Q_{+}(0),{({\mathcal{A}}_{\mu }/{\mathcal{A}}_0)}^{\frac{N-s}{N-p}}{|G|}^{\frac{s-p}{N-p}}{\parallel Q_{+}\parallel }_{\mathrm{\infty }}\}$$

and either (i) $Q(x)\ge Q(\mathrm{\infty })+\sigma {|x|}^{-N+s+{\eta }_1{p}^{\ast }(s)}$for some$\sigma >0$and large$|x|$or (ii) $|Q(x)-Q(\mathrm{\infty })|\le C{|x|}^{-\alpha }$for some constants$C>0$, $\alpha >N-s-{\eta }_1{p}^{\ast }(s)$, large$|x|$, and

$${\int }_{{\mathbb{R}}^N}(Q(x)-Q(\mathrm{\infty })){|x|}^{-s-{\eta }_1{p}^{\ast }(s)}dx>0.$$

(2.12)

• (3) If $Q(x)\ge Q(\mathrm{\infty })=Q(0)>0$ on ${\mathbb{R}}^N$ and

  $$Q(\mathrm{\infty })=Q(0)\ge {({\mathcal{A}}_{\mu }/{\mathcal{A}}_0)}^{\frac{N-s}{N-p}}{|G|}^{\frac{s-p}{N-p}}{\parallel Q_{+}\parallel }_{\mathrm{\infty }},$$

then problem (${\mathcal{P}}_0^Q$) has at least one positive solution.

Theorem 2.2

Suppose that$Q_{+}(0)=Q_{+}(\mathrm{\infty })=0$and$|G|=+\mathrm{\infty }$. Then problem (${\mathcal{P}}_0^Q$) has infinitely many G-symmetric solutions.

Theorem 2.3

Let$\overline{Q}>0$be a constant. Suppose that$Q(x)\equiv \overline{Q}$and (h.1) and (h.2) hold. If

$$q>max\{p^{\ast }(\vartheta ),\frac{N-s}{{\eta }_2},\frac{(2N-p-s-p{\eta }_2)p}{N-p}\},$$

(2.13)

then problem (${\mathcal{P}}_h^{\overline{Q}}$) possesses at least one positive solution in${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$.

Throughout this paper, we denote by ${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$ the subspace of ${\mathcal{D}}^{1,p}({\mathbb{R}}^N)$ consisting of all G-symmetric functions. The dual space of ${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$ (${\mathcal{D}}^{1,p}({\mathbb{R}}^N)$, resp.) is denoted by ${\mathcal{D}}_G^{-1,p^{\prime }}({\mathbb{R}}^N)$ (${\mathcal{D}}^{-1,p^{\prime }}({\mathbb{R}}^N)$, resp.), where $\frac{1}{p}+\frac{1}{p^{\prime }}=1$. In a similar manner, we define ${\mathcal{D}}_G^{1,p}(\mathrm{\Omega })$ for an open and G-symmetric subset of ${\mathbb{R}}^N$, that is, if $x\in \mathrm{\Omega }$, then $gx\in \mathrm{\Omega }$ for all $g\in G$. In the case where Ω is bounded, we set $W_{0,G}^{1,p}(\mathrm{\Omega })={\mathcal{D}}_G^{1,p}(\mathrm{\Omega })$. The ball of center x and radius r is denoted by $B(x,r)$. We employ C, $C_i$ ($i=1,2,\dots$) to denote the positive constants, and denote by ‘→’ convergence in norm in a given Banach space X and by ‘⇀’ weak convergence. A functional $J\in {\mathcal{C}}^1(X,\mathbb{R})$ is said to satisfy the ${(PS)}_c$ condition if each sequence $\{u_n\}$ in X satisfying $J(u_n)\to c$, $J^{\prime }(u_n)\to 0$ in $X^{\ast }$ has a subsequence which strongly converges to some element in X. Hereafter, $L^r(\mathrm{\Omega },{|x|}^{-\varsigma })$ denotes the weighted $L^r(\mathrm{\Omega })$ space with the norm ${({\int }_{\mathrm{\Omega }}{|x|}^{-\varsigma }{|u|}^{r}dx)}^{1/r}$. Also, for nonnegative measurable function $k(x)$, we denote by $L^r({\mathbb{R}}^N,k(x))$ the space of measurable functions u satisfying ${({\int }_{{\mathbb{R}}^N}k(x){|u|}^{r}dx)}^{1/r}<\mathrm{\infty }$.

3 Existence and multiplicity results for problem (${\mathcal{P}}_0^Q$)

We associate with problem (${\mathcal{P}}_0^Q$) a functional $\mathcal{F}:{\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)\to \mathbb{R}$ given by

$$\mathcal{F}(u)=\frac{1}{p}{\int }_{{\mathbb{R}}^N}({|\mathrm{\nabla }u|}^p-\mu \frac{{|u|}^p}{{|x|}^p})dx-\frac{1}{p^{\ast }(s)}{\int }_{{\mathbb{R}}^N}Q(x)\frac{{|u|}^{p^{\ast }(s)}}{{|x|}^s}dx.$$

(3.1)

By (q.1) and (2.1), we easily see that the functional $\mathcal{F}\in {\mathcal{C}}^1({\mathcal{D}}_G^{1,p}({\mathbb{R}}^N),\mathbb{R})$. Now it is well known that there exists a one-to-one correspondence between the weak solutions of problem (${\mathcal{P}}_0^Q$) and the critical points of ℱ. More precisely, the weak solutions of (${\mathcal{P}}_0^Q$) are exactly the critical points of ℱ by the principle of symmetric criticality of Palais (see Lemma 3.1), namely $u\in {\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$ satisfies (${\mathcal{P}}_0^Q$) if and only if for all $\nu \in {\mathcal{D}}^{1,p}({\mathbb{R}}^N)$, there holds

$${\int }_{{\mathbb{R}}^N}({|\mathrm{\nabla }u|}^{p-2}\mathrm{\nabla }u\mathrm{\nabla }\nu -\mu \frac{{|u|}^{p-2}u\nu }{{|x|}^p})dx-{\int }_{{\mathbb{R}}^N}Q(x)\frac{{|u|}^{p^{\ast }(s)-2}u\nu }{{|x|}^s}dx=0.$$

(3.2)

Lemma 3.1

Let$Q(x)$be a G-symmetric function; ${\mathcal{F}}^{\prime }(u)=0$in${\mathcal{D}}_G^{-1,p^{\prime }}({\mathbb{R}}^N)$implies${\mathcal{F}}^{\prime }(u)=0$in${\mathcal{D}}^{-1,p^{\prime }}({\mathbb{R}}^N)$.

Proof

Similar to the proof of [20], Lemma 1] (see also [17], Lemma 3.1]). □

Lemma 3.2

Let$\{u_n\}$be a weakly convergent sequence to u in${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$such that${|\mathrm{\nabla }{u}_n|}^p\rightharpoonup \varsigma$, ${|x|}^{-s}{|u_n|}^{p^{\ast }(s)}\rightharpoonup \nu$, and${|x|}^{-p}{|u_n|}^p\rightharpoonup \tilde{\nu }$in the sense of measures. Then there exists some at most countable set$\mathcal{J}$, ${\{{\varsigma }_j\ge 0\}}_{j\in \mathcal{J}\cup \{0\}}$, ${\{{\nu }_j\ge 0\}}_{j\in \mathcal{J}\cup \{0\}}$, ${\tilde{\nu }}_0\ge 0$, ${\{x_j\}}_{j\in \mathcal{J}}\subset {\mathbb{R}}^N\mathrm{\setminus }\{0\}$such that

• (a) $\varsigma \ge {|\mathrm{\nabla }u|}^p+{\sum }_{j\in \mathcal{J}}{\varsigma }_j{\delta }_{x_j}+{\varsigma }_0{\delta }_0$,

• (b) $\nu ={|x|}^{-s}{|u|}^{p^{\ast }(s)}+{\sum }_{j\in \mathcal{J}}{\nu }_j{\delta }_{x_j}+{\nu }_0{\delta }_0$,

• (c) $\tilde{\nu }={|x|}^{-p}{|u|}^p+{\tilde{\nu }}_0{\delta }_0$,

• (d) ${\mathcal{A}}_0{\nu }_j^{p/p^{\ast }(s)}\le {\varsigma }_j$,

• (e) ${\mathcal{A}}_{\mu }{\nu }_0^{p/p^{\ast }(s)}\le {\varsigma }_0-\mu {\tilde{\nu }}_0$,

where${\delta }_{x_j}$, $j\in \mathcal{J}\cup \{0\}$, is the Dirac mass of 1 concentrated at$x_j\in {\mathbb{R}}^N$.

Proof

The proof is similar to that of the concentration-compactness principle in Refs. [15], [16] and is omitted here. □

To find critical points of ℱ we need the following local ${(PS)}_c$ condition which is crucial for the proof of Theorem 2.1.

Lemma 3.3

Suppose that (q.1) and (q.2) hold. Then the${(PS)}_c$condition in${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$holds for$\mathcal{F}(u)$if

$$c<c_0^{\ast }\triangleq \frac{p-s}{(N-s)p}min\{\frac{{\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}}{Q_{+}{(0)}^{\frac{N-p}{p-s}}},\frac{{\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}}{Q_{+}{(\mathrm{\infty })}^{\frac{N-p}{p-s}}},\frac{|G|{\mathcal{A}}_0^{\frac{N-s}{p-s}}}{{\parallel Q_{+}\parallel }_{\mathrm{\infty }}^{\frac{N-p}{p-s}}}\}.$$

(3.3)

Proof

The proof is similar to that in [20], Proposition 2]. We sketch the argument here for completeness. Let $\{u_n\}$ be a ${(PS)}_c$ sequence for ℱ with $c<c_0^{\ast }$. Then we easily deduce from (2.1) and (3.3) that $\{u_n\}$ is bounded in ${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$ and we may assume that $u_n\rightharpoonup u$ in ${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$. By Lemma 3.2 there exist measures ς, ν, and $\tilde{\nu }$ such that relations (a)-(e) of this lemma hold. Let $x_j\ne 0$ be a singular point of measures ς and ν. As in paper [14], we define a function ${\varphi }_{ϵ}\in {\mathcal{C}}^1({\mathbb{R}}^N)$ such that ${\varphi }_{ϵ}=1$ in $B(x_j,ϵ/2)$, ${\varphi }_{ϵ}=0$ on ${\mathbb{R}}^N\mathrm{\setminus }B(x_j,ϵ)$ and $|\mathrm{\nabla }{\varphi }_{ϵ}|\le 4/ϵ$. By Lemma 3.1, ${lim}_{n\to \mathrm{\infty }}〈{\mathcal{F}}^{\prime }(u_n),u_n{\varphi }_{ϵ}〉=0$, hence, using (2.1), the Hölder inequality, and the fact that $p^{\ast }(0)=p^{\ast }$, we get

$$\begin{array}{r}{\int }_{{\mathbb{R}}^N}{\varphi }_{ϵ}d\varsigma -{\int }_{{\mathbb{R}}^N}\mu {\varphi }_{ϵ}d\tilde{\nu }-{\int }_{{\mathbb{R}}^N}Q(x){\varphi }_{ϵ}d\nu \\ \le \underset{n\to \mathrm{\infty }}{lim\hspace{0.17em}sup}\left|{\int }_{{\mathbb{R}}^N}{u}_n{|\mathrm{\nabla }{u}_n|}^{p-2}\mathrm{\nabla }{u}_n\mathrm{\nabla }{\varphi }_{ϵ}dx\right|\\ \le \underset{n\ge 1}{sup}{\left({\int }_{{\mathbb{R}}^N}{|\mathrm{\nabla }{u}_n|}^{p}dx\right)}^{\frac{p-1}{p}}\underset{n\to \mathrm{\infty }}{lim\hspace{0.17em}sup}{\left({\int }_{{\mathbb{R}}^N}{|u_n|}^p{|\mathrm{\nabla }{\varphi }_{ϵ}|}^{p}dx\right)}^{\frac{1}{p}}\\ \le C{\left({\int }_{{\mathbb{R}}^N}{|u|}^p{|\mathrm{\nabla }{\varphi }_{ϵ}|}^{p}dx\right)}^{\frac{1}{p}}\le C{\left({\int }_{B(x_j,ϵ)}{|u|}^{p^{\ast }}dx\right)}^{\frac{1}{p^{\ast }}}{\left({\int }_{{\mathbb{R}}^N}{|\mathrm{\nabla }{\varphi }_{ϵ}|}^{N}dx\right)}^{\frac{1}{N}}\\ \le C{\left({\int }_{B(x_j,ϵ)}{|\mathrm{\nabla }u|}^{p}dx\right)}^{\frac{1}{p}}.\end{array}$$

(3.4)

Passing to the limit as $ϵ\to 0$, we deduce from (3.4) and Lemma 3.2 that

$$Q(x_j){\nu }_j\ge {\varsigma }_j.$$

(3.5)

The above inequality says that the concentration of the measure ν cannot occur at points where $Q(x_j)\le 0$, that is, if $Q(x_j)\le 0$ then ${\varsigma }_j={\nu }_j=0$. Combining (3.5) and (d) of Lemma 3.2 we find that either (i) ${\nu }_j=0$ or (ii) ${\nu }_j\ge {({\mathcal{A}}_0/{\parallel Q_{+}\parallel }_{\mathrm{\infty }})}^{\frac{N-s}{p-s}}$. For the point $x=0$, similarly to the case $x_j\ne 0$, we have ${\varsigma }_0-\mu {\tilde{\nu }}_0-Q(0){\nu }_0\le 0$. This, combined with (e) of Lemma 3.2, implies that either (iii) ${\nu }_0=0$ or (iv) ${\nu }_0\ge {({\mathcal{A}}_{\mu }/Q_{+}(0))}^{\frac{N-s}{p-s}}$. To study the concentration of the sequence $\{u_n\}$ at infinity we need to consider the following quantities:

$$\begin{array}{r}{\varsigma }_{\mathrm{\infty }}=\underset{R\to \mathrm{\infty }}{lim}\underset{n\to \mathrm{\infty }}{lim\hspace{0.17em}sup}{\int }_{|x|>R}({|\mathrm{\nabla }{u}_n|}^p-\mu {|x|}^{-p}{|u_n|}^p)dx,\\ {\nu }_{\mathrm{\infty }}=\underset{R\to \mathrm{\infty }}{lim}\underset{n\to \mathrm{\infty }}{lim\hspace{0.17em}sup}{\int }_{|x|>R}{|x|}^{-s}{|u_n|}^{p^{\ast }(s)}dx.\end{array}$$

Obviously, ${\varsigma }_{\mathrm{\infty }}$ and ${\nu }_{\mathrm{\infty }}$ both exist and are finite. For $R>1$, let ${\psi }_R$ be a regular function such that $0\le {\psi }_R\le 1$, ${\psi }_R(x)=1$ for $|x|>R+1$, ${\psi }_R(x)=0$ for $|x|<R$ and $|\mathrm{\nabla }{\psi }_R|\le 4/R$. Then we deduce from the definition of ${\mathcal{A}}_{\mu }$ that

$${\int }_{{\mathbb{R}}^N}({|{\psi }_R\mathrm{\nabla }{u}_n+u_n\mathrm{\nabla }{\psi }_R|}^p-\mu \frac{{|u_n{\psi }_R|}^p}{{|x|}^p})dx\ge {\mathcal{A}}_{\mu }{\left({\int }_{{\mathbb{R}}^N}\frac{{|u_n{\psi }_R|}^{p^{\ast }(s)}}{{|x|}^s}dx\right)}^{\frac{p}{p^{\ast }(s)}}.$$

(3.6)

We now claim that

$$\underset{R\to \mathrm{\infty }}{lim}\underset{n\to \mathrm{\infty }}{lim\hspace{0.17em}sup}{\int }_{{\mathbb{R}}^N}({|{\psi }_R\mathrm{\nabla }{u}_n+u_n\mathrm{\nabla }{\psi }_R|}^p-{\psi }_R^p{|\mathrm{\nabla }{u}_n|}^p)dx=0.$$

(3.7)

In fact, using the elementary inequality $|{|z+w|}^p-{|z|}^p|\le C({|z|}^{p-1}|w|+{|w|}^p)$ for all $z,w\in {\mathbb{R}}^N$ and $p>1$, we get

$${\int }_{{\mathbb{R}}^N}({|{\psi }_R\mathrm{\nabla }{u}_n+u_n\mathrm{\nabla }{\psi }_R|}^p-{\psi }_R^p{|\mathrm{\nabla }{u}_n|}^p)\le C{\int }_{{\mathbb{R}}^N}({|{\psi }_R\mathrm{\nabla }{u}_n|}^{p-1}|u_n\mathrm{\nabla }{\psi }_R|+{|u_n\mathrm{\nabla }{\psi }_R|}^p).$$

On the other hand, by the Hölder inequality and the Sobolev inequality, we have

$$\begin{array}{r}\underset{R\to \mathrm{\infty }}{lim}\underset{n\to \mathrm{\infty }}{lim\hspace{0.17em}sup}{\int }_{{\mathbb{R}}^N}{|{\psi }_R\mathrm{\nabla }{u}_n|}^{p-1}|u_n\mathrm{\nabla }{\psi }_R|dx\\ \le \underset{R\to \mathrm{\infty }}{lim}\underset{n\to \mathrm{\infty }}{lim\hspace{0.17em}sup}{\left({\int }_{R<|x|<R+1}{|u_n|}^p{|\mathrm{\nabla }{\psi }_R|}^{p}dx\right)}^{\frac{1}{p}}{\left({\int }_{{\mathbb{R}}^N}{|\mathrm{\nabla }{u}_n|}^{p}dx\right)}^{\frac{p-1}{p}}\\ \le C\underset{R\to \mathrm{\infty }}{lim}{\left({\int }_{R<|x|<R+1}{|u|}^p{|\mathrm{\nabla }{\psi }_R|}^{p}dx\right)}^{\frac{1}{p}}\\ \le C\underset{R\to \mathrm{\infty }}{lim}{\left({\int }_{R<|x|<R+1}{|u|}^{p^{\ast }}dx\right)}^{\frac{1}{p^{\ast }}}{\left({\int }_{{\mathbb{R}}^N}{|\mathrm{\nabla }{\psi }_R|}^{N}dx\right)}^{\frac{1}{N}}\\ \le C\underset{R\to \mathrm{\infty }}{lim}{\left({\int }_{R<|x|<R+1}{|\mathrm{\nabla }u|}^{p}dx\right)}^{\frac{1}{p}}=0.\end{array}$$

Similarly, we have ${lim}_{R\to \mathrm{\infty }}{lim\hspace{0.17em}sup}_{n\to \mathrm{\infty }}{\int }_{{\mathbb{R}}^N}{|u_n|}^p{|\mathrm{\nabla }{\psi }_R|}^{p}dx=0$. The claim (3.7) is thereby proved. From (3.6) and (3.7), we derive ${\varsigma }_{\mathrm{\infty }}\ge {\mathcal{A}}_{\mu }{\nu }_{\mathrm{\infty }}^{p/p^{\ast }(s)}$. Moreover, since ${lim}_{R\to \mathrm{\infty }}{lim\hspace{0.17em}sup}_{n\to \mathrm{\infty }}〈{\mathcal{F}}^{\prime }(u_n),u_n{\psi }_R〉=0$, we get $Q_{+}(\mathrm{\infty }){\nu }_{\mathrm{\infty }}\ge {\varsigma }_{\mathrm{\infty }}$. Therefore we conclude that either (v) ${\nu }_{\mathrm{\infty }}=0$ or (vi) ${\nu }_{\mathrm{\infty }}\ge {({\mathcal{A}}_{\mu }/Q_{+}(\mathrm{\infty }))}^{\frac{N-s}{p-s}}$. We now rule out the cases (ii), (iv), and (vi). For every continuous nonnegative function ψ such that $0\le \psi (x)\le 1$ on ${\mathbb{R}}^N$, we obtain from (3.1) and (3.2) that

$$\begin{array}{rl}c& =\underset{n\to \mathrm{\infty }}{lim}(\mathcal{F}(u_n)-\frac{1}{p^{\ast }(s)}〈{\mathcal{F}}^{\prime }(u_n),u_n〉)\\ =(\frac{1}{p}-\frac{1}{p^{\ast }(s)})\underset{n\to \mathrm{\infty }}{lim}{\int }_{{\mathbb{R}}^N}({|\mathrm{\nabla }{u}_n|}^p-\mu \frac{{|u_n|}^p}{{|x|}^p})dx\\ \ge \frac{p-s}{(N-s)p}\underset{n\to \mathrm{\infty }}{lim\hspace{0.17em}sup}{\int }_{{\mathbb{R}}^N}\psi (x)({|\mathrm{\nabla }{u}_n|}^p-\mu \frac{{|u_n|}^p}{{|x|}^p})dx.\end{array}$$

If (ii) occurs, then the set $\mathcal{J}$ must be finite because the measure ν is bounded. Since functions $u_n$ are G-symmetric, the measure ν must be G-invariant. This means that if $x_j\ne 0$ is a singular point of ν, so is $g{x}_j$ for each $g\in G$, and the mass of ν concentrated at $g{x}_j$ is the same for each $g\in G$. If we assume the existence of $j\in \mathcal{J}$ with $x_j\ne 0$ such that (ii) holds, then we choose ψ with compact support so that $\psi (g{x}_j)=1$ for each $g\in G$ and we obtain

$$c\ge \frac{p-s}{(N-s)p}|G|{\varsigma }_j\ge \frac{p-s}{(N-s)p}|G|{\mathcal{A}}_0{\nu }_j^{\frac{p}{p^{\ast }(s)}}\ge \frac{p-s}{(N-s)p}|G|{\mathcal{A}}_0^{\frac{N-s}{p-s}}{\parallel Q_{+}\parallel }_{\mathrm{\infty }}^{\frac{p-N}{p-s}},$$

a contradiction with (3.3). Similarly, if (iv) holds for $x=0$, we choose ψ with compact support, so that $\psi (0)=1$, and we obtain

$$c\ge \frac{p-s}{(N-s)p}({\varsigma }_0-\mu {\tilde{\nu }}_0)\ge \frac{p-s}{(N-s)p}{\mathcal{A}}_{\mu }{\nu }_0^{\frac{p}{p^{\ast }(s)}}\ge \frac{p-s}{(N-s)p}{\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}{Q}_{+}{(0)}^{\frac{p-N}{p-s}},$$

which contradicts (3.3). Finally, if (vi) holds we take $\psi ={\psi }_R$ to get

$$c\ge \frac{p-s}{(N-s)p}{\varsigma }_{\mathrm{\infty }}\ge \frac{p-s}{(N-s)p}{\mathcal{A}}_{\mu }{\nu }_{\mathrm{\infty }}^{\frac{p}{p^{\ast }(s)}}\ge \frac{p-s}{(N-s)p}{\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}{Q}_{+}{(\mathrm{\infty })}^{\frac{p-N}{p-s}},$$

which is impossible. Hence ${\nu }_j=0$ for all $j\in \mathcal{J}\cup \{0,\mathrm{\infty }\}$, and consequently we have $u_n\to u$ in $L^{p^{\ast }(s)}({\mathbb{R}}^N,{|x|}^{-s})$. Finally, observe that ${\mathcal{F}}^{\prime }(u)=0$ and, thus by ${lim}_{n\to \mathrm{\infty }}〈{\mathcal{F}}^{\prime }(u_n)-{\mathcal{F}}^{\prime }(u),u_n-u〉=0$ we obtain $u_n\to u$ in ${\mathcal{D}}^{1,p}({\mathbb{R}}^N)$. □

As an easy consequence of Lemma 3.3 we obtain the following result.

Corollary 3.1

If$Q_{+}(0)=0$and$|G|=+\mathrm{\infty }$, then the functional ℱ satisfies${(PS)}_c$condition for every$c\in \mathbb{R}$.

Proof of Theorem 2.1

Let $y_{ϵ}$ be the extremal function satisfying (2.4)-(2.9). We choose $ϵ>0$ such that the assumption (2.10) holds. It is easy to check that there exist constants ${\alpha }_0>0$ and $\rho >0$ such that $\mathcal{F}(u)\ge {\alpha }_0$ for all ${\parallel u\parallel }_{\mu }=\rho$. Simple arithmetic shows that there exists $\overline{t}>0$ such that

$$\underset{t\ge 0}{max}\mathcal{F}(t{y}_{ϵ})=\mathcal{F}(\overline{t}{y}_{ϵ})=\frac{p-s}{(N-s)p}{({\int }_{{\mathbb{R}}^N}Q(x)\frac{y_{ϵ}^{p^{\ast }(s)}}{{|x|}^s}dx)}^{\frac{p-N}{p-s}}.$$

(3.8)

We now choose $t_0>0$ such that $\mathcal{F}(t_0{y}_{ϵ})<0$ and ${\parallel t_0{y}_{ϵ}\parallel }_{\mu }>\rho$ and set

$$c_0=\underset{\gamma \in \mathrm{\Gamma }}{inf}\underset{t\in [0,1]}{max}\mathcal{F}(\gamma (t)),$$

(3.9)

where $\mathrm{\Gamma }=\{\gamma \in \mathcal{C}([0,1],{\mathcal{D}}_G^{1,p}({\mathbb{R}}^N));\gamma (0)=0,\mathcal{F}(\gamma (1))<0,{\parallel \gamma (1)\parallel }_{\mu }>\rho \}$. From (2.10), (3.8), (3.9), and the definition of $c_0^{\ast }$, we deduce that

$$\begin{array}{rl}{c}_0& \le \mathcal{F}(\overline{t}{y}_{ϵ})=\frac{p-s}{(N-s)p}{({\int }_{{\mathbb{R}}^N}Q(x)\frac{y_{ϵ}^{p^{\ast }(s)}}{{|x|}^s}dx)}^{\frac{p-N}{p-s}}\\ \le \frac{p-s}{(N-s)p}{(max\{\frac{Q_{+}(0)}{{\mathcal{A}}_{\mu }^{\frac{N-s}{N-p}}},\frac{Q_{+}(\mathrm{\infty })}{{\mathcal{A}}_{\mu }^{\frac{N-s}{N-p}}},\frac{{\parallel Q_{+}\parallel }_{\mathrm{\infty }}}{{|G|}^{\frac{p-s}{N-p}}{\mathcal{A}}_0^{\frac{N-s}{N-p}}}\})}^{\frac{p-N}{p-s}}\\ =\frac{p-s}{(N-s)p}min\{\frac{{\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}}{Q_{+}{(0)}^{\frac{N-p}{p-s}}},\frac{{\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}}{Q_{+}{(\mathrm{\infty })}^{\frac{N-p}{p-s}}},\frac{|G|{\mathcal{A}}_0^{\frac{N-s}{p-s}}}{{\parallel Q_{+}\parallel }_{\mathrm{\infty }}^{\frac{N-p}{p-s}}}\}=c_0^{\ast }.\end{array}$$

If $c_0<c_0^{\ast }$, then by Lemma 3.3, the ${(PS)}_c$ condition holds and the conclusion follows from the mountain pass theorem in Ref. [26] (see also [27]). If $c_0=c_0^{\ast }$, then $\gamma (t)=t{t}_0{y}_{ϵ}$, with $0\le t\le 1$, is a path in Γ such that ${max}_{t\in [0,1]}\mathcal{F}(\gamma (t))=c_0$. Consequently, either ${\mathcal{F}}^{\prime }(\overline{t}{y}_{ϵ})=0$ and we are done, or γ can be deformed to a path $\tilde{\gamma }\in \mathrm{\Gamma }$ with ${max}_{t\in [0,1]}\mathcal{F}(\tilde{\gamma }(t))<c_0$ and we get a contradiction. This part of the proof shows that a nontrivial solution $u_0\in {\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$ of (${\mathcal{P}}_0^Q$) exists. We now show that the solution $u_0$ can be chosen to be positive on ${\mathbb{R}}^N$. Since $\mathcal{F}(u_0)=\mathcal{F}(|u_0|)$ and $0=〈{\mathcal{F}}^{\prime }(u_0),u_0〉={\parallel u_0\parallel }_{\mu }^p-{\int }_{{\mathbb{R}}^N}Q(x){|x|}^{-s}{|u_0|}^{p^{\ast }(s)}dx$, we obtain ${\int }_{{\mathbb{R}}^N}Q(x){|x|}^{-s}{|u_0|}^{p^{\ast }(s)}dx={\parallel u_0\parallel }_{\mu }^p>0$, which implies $c_0=\mathcal{F}(|u_0|)={max}_{t\ge 0}\mathcal{F}(t|u_0|)$. Hence, either $|u_0|$ is a critical point of ℱ or $\gamma (t)=t{t}_0|u_0|$, with $\mathcal{F}(t_0|u_0|)<0$, can be deformed, as above of the proof, to a path $\tilde{\gamma }(t)$ with ${max}_{t\in [0,1]}\mathcal{F}(\tilde{\gamma }(t))<c_0$, which is impossible. Therefore, we may assume that $u_0$ is nonnegative on ${\mathbb{R}}^N$ and the fact that $u_0>0$ on ${\mathbb{R}}^N$ follows by the strong maximum principle. □

Proof of Corollary 2.1

First of all, we observe that due to the identity (2.6), inequality (2.10) is equivalent to ${\int }_{{\mathbb{R}}^N}(Q(x)-\tilde{Q}){|x|}^{-s}{y}_{ϵ}^{p^{\ast }(s)}dx\ge 0$ for some $ϵ>0$, or equivalently

$${ϵ}^{-\frac{N-p}{p}\cdot p^{\ast }(s)}{\int }_{{\mathbb{R}}^N}(Q(x)-\tilde{Q}){|x|}^{-s}{U}_{\mu }^{p^{\ast }(s)}\left(\frac{|x|}{ϵ}\right)dx\ge 0$$

(3.10)

for some $ϵ>0$, where

$$\tilde{Q}=max\{Q_{+}(0),Q_{+}(\mathrm{\infty }),{({\mathcal{A}}_{\mu }/{\mathcal{A}}_0)}^{\frac{N-s}{N-p}}{|G|}^{\frac{s-p}{N-p}}{\parallel Q_{+}\parallel }_{\mathrm{\infty }}\}.$$

Part (1), case (i). According to (3.10), we need to show that

$${ϵ}^{-{\eta }_2{p}^{\ast }(s)}{\int }_{{\mathbb{R}}^N}(Q(x)-Q(0)){|x|}^{-s}{U}_{\mu }^{p^{\ast }(s)}\left(\frac{|x|}{ϵ}\right)dx\ge 0$$

(3.11)

for some $ϵ>0$. We choose $\delta >0$ so that $Q(x)\ge Q(0)+\sigma {|x|}^{-N+s+{\eta }_2{p}^{\ast }(s)}$ for $|x|\le \delta$. This, combined with (2.8), implies that

$$\begin{array}{r}{ϵ}^{-{\eta }_2{p}^{\ast }(s)}{\int }_{|x|\le \delta }(Q(x)-Q(0)){|x|}^{-s}{U}_{\mu }^{p^{\ast }(s)}\left(\frac{|x|}{ϵ}\right)dx\\ ={\int }_{|x|\le \delta }\frac{Q(x)-Q(0)}{{|x|}^{s+{\eta }_2{p}^{\ast }(s)}}{\left[{\left(\frac{|x|}{ϵ}\right)}^{{\eta }_2}{U}_{\mu }\left(\frac{|x|}{ϵ}\right)\right]}^{p^{\ast }(s)}dx\\ \ge \sigma {\int }_{|x|\le \delta }{|x|}^{-N}{\left[{\left(\frac{|x|}{ϵ}\right)}^{{\eta }_2}{U}_{\mu }\left(\frac{|x|}{ϵ}\right)\right]}^{p^{\ast }(s)}dx\to \mathrm{\infty }\end{array}$$

(3.12)

as $ϵ\to 0$. On the other hand, for all $ϵ>0$, we obtain from (2.8) and the fact $s+{\eta }_2{p}^{\ast }(s)>N$ that

$$\begin{array}{r}\left|{ϵ}^{-{\eta }_2{p}^{\ast }(s)}{\int }_{|x|>\delta }(Q(x)-Q(0)){|x|}^{-s}{U}_{\mu }^{p^{\ast }(s)}\left(\frac{|x|}{ϵ}\right)dx\right|\\ \le {\int }_{|x|>\delta }\frac{|Q(x)-Q(0)|}{{|x|}^{s+{\eta }_2{p}^{\ast }(s)}}{\left[{\left(\frac{|x|}{ϵ}\right)}^{{\eta }_2}{U}_{\mu }\left(\frac{|x|}{ϵ}\right)\right]}^{p^{\ast }(s)}dx\le {\overline{C}}_3\end{array}$$

(3.13)

for some constant ${\overline{C}}_3>0$ independent of ϵ. Combining (3.12) and (3.13), we get (3.11) for ϵ sufficiently small.

Part (1), case (ii). We choose $\delta >0$ so that $|Q(x)-Q(0)|\le C{|x|}^{\alpha }$ for $|x|\le \delta$. Since $\alpha >-N+s+{\eta }_2{p}^{\ast }(s)>0$, we deduce from (2.8) and the fact $-s-{\eta }_2{p}^{\ast }(s)<-N$ that

$$\begin{array}{r}{ϵ}^{-{\eta }_2{p}^{\ast }(s)}{\int }_{{\mathbb{R}}^N}\frac{|Q(x)-Q(0)|}{{|x|}^s}{U}_{\mu }^{p^{\ast }(s)}\left(\frac{|x|}{ϵ}\right)dx\\ ={\int }_{{\mathbb{R}}^N}\frac{|Q(x)-Q(0)|}{{|x|}^{s+{\eta }_2{p}^{\ast }(s)}}{\left[{\left(\frac{|x|}{ϵ}\right)}^{{\eta }_2}{U}_{\mu }\left(\frac{|x|}{ϵ}\right)\right]}^{p^{\ast }(s)}dx\le C{\int }_{{\mathbb{R}}^N}\frac{|Q(x)-Q(0)|}{{|x|}^{s+{\eta }_2{p}^{\ast }(s)}}dx\\ \le C({\int }_{|x|\le \delta }{|x|}^{\alpha -s-{\eta }_2{p}^{\ast }(s)}dx+{\int }_{|x|>\delta }|Q(x)-Q(0)|{|x|}^{-s-{\eta }_2{p}^{\ast }(s)}dx)\le C.\end{array}$$

So by (2.8), (2.11), and the Lebesgue dominated convergence theorem we obtain

$$\underset{ϵ\to 0}{lim}{ϵ}^{-{\eta }_2{p}^{\ast }(s)}{\int }_{{\mathbb{R}}^N}\frac{Q(x)-Q(0)}{{|x|}^s}{U}_{\mu }^{p^{\ast }(s)}\left(\frac{|x|}{ϵ}\right)dx=C{\int }_{{\mathbb{R}}^N}\frac{Q(x)-Q(0)}{{|x|}^{s+{\eta }_2{p}^{\ast }(s)}}dx>0.$$

Thus (3.11) holds for ϵ sufficiently small.

Part (2), case (i). From (3.10) it is sufficient to show that

$${ϵ}^{-{\eta }_1{p}^{\ast }(s)}{\int }_{{\mathbb{R}}^N}(Q(x)-Q(\mathrm{\infty })){|x|}^{-s}{U}_{\mu }^{p^{\ast }(s)}\left(\frac{|x|}{ϵ}\right)dx\ge 0$$

(3.14)

for some $ϵ>0$. We choose $R>0$ such that $Q(x)\ge Q(\mathrm{\infty })+\sigma {|x|}^{-N+s+{\eta }_1{p}^{\ast }(s)}$ for all $|x|\ge R$. This, combined with (2.7), implies that

$$\begin{array}{r}{ϵ}^{-{\eta }_1{p}^{\ast }(s)}{\int }_{|x|\ge R}(Q(x)-Q(\mathrm{\infty })){|x|}^{-s}{U}_{\mu }^{p^{\ast }(s)}\left(\frac{|x|}{ϵ}\right)dx\\ ={\int }_{|x|\ge R}\frac{Q(x)-Q(\mathrm{\infty })}{{|x|}^{s+{\eta }_1{p}^{\ast }(s)}}{\left[{\left(\frac{|x|}{ϵ}\right)}^{{\eta }_1}{U}_{\mu }\left(\frac{|x|}{ϵ}\right)\right]}^{p^{\ast }(s)}dx\\ \ge \sigma {\int }_{|x|\ge R}{|x|}^{-N}{\left[{\left(\frac{|x|}{ϵ}\right)}^{{\eta }_1}{U}_{\mu }\left(\frac{|x|}{ϵ}\right)\right]}^{p^{\ast }(s)}dx\to \mathrm{\infty }\end{array}$$

as $ϵ\to \mathrm{\infty }$. Moreover, for all $ϵ>0$, we get from (2.7) and the fact that $-s-{\eta }_1{p}^{\ast }(s)>-N$

$$\begin{array}{r}{ϵ}^{-{\eta }_1{p}^{\ast }(s)}{\int }_{|x|\le R}(Q(x)-Q(\mathrm{\infty })){|x|}^{-s}{U}_{\mu }^{p^{\ast }(s)}\left(\frac{|x|}{ϵ}\right)dx\\ ={\int }_{|x|\le R}\frac{Q(x)-Q(\mathrm{\infty })}{{|x|}^{s+{\eta }_1{p}^{\ast }(s)}}{\left[{\left(\frac{|x|}{ϵ}\right)}^{{\eta }_1}{U}_{\mu }\left(\frac{|x|}{ϵ}\right)\right]}^{p^{\ast }(s)}dx\\ \le C{\int }_{|x|\le R}(Q(x)-Q(\mathrm{\infty })){|x|}^{-s-{\eta }_1{p}^{\ast }(s)}dx\le {\overline{C}}_4\end{array}$$

for some constant ${\overline{C}}_4>0$ independent of $ϵ>0$. These two estimates combined together give (3.14) for $ϵ>0$ large.

Part (2), case (ii). We choose $R>0$ such that $|Q(x)-Q(\mathrm{\infty })|\le C{|x|}^{-\alpha }$ for all $|x|\ge R$. Since $\alpha >N-s-{\eta }_1{p}^{\ast }(s)>0$, we have

$$\begin{array}{r}{\int }_{{\mathbb{R}}^N}|Q(x)-Q(\mathrm{\infty })|{|x|}^{-s-{\eta }_1{p}^{\ast }(s)}dx\\ \le C{\int }_{|x|\ge R}{|x|}^{-\alpha -s-{\eta }_1{p}^{\ast }(s)}dx+{\int }_{|x|\le R}|Q(x)-Q(\mathrm{\infty })|{|x|}^{-s-{\eta }_1{p}^{\ast }(s)}dx<\mathrm{\infty }.\end{array}$$

Thus by (2.7), (2.12), and the Lebesgue dominated convergence theorem, we obtain

$$\begin{array}{r}\underset{ϵ\to \mathrm{\infty }}{lim}{ϵ}^{-{\eta }_1{p}^{\ast }(s)}{\int }_{{\mathbb{R}}^N}(Q(x)-Q(\mathrm{\infty })){|x|}^{-s}{U}_{\mu }^{p^{\ast }(s)}\left(\frac{|x|}{ϵ}\right)dx\\ =\underset{ϵ\to \mathrm{\infty }}{lim}{\int }_{{\mathbb{R}}^N}(Q(x)-Q(\mathrm{\infty })){|x|}^{-s-{\eta }_1{p}^{\ast }(s)}{\left[{\left(\frac{|x|}{ϵ}\right)}^{{\eta }_1}{U}_{\mu }\left(\frac{|x|}{ϵ}\right)\right]}^{p^{\ast }(s)}dx\\ =C{\int }_{{\mathbb{R}}^N}(Q(x)-Q(\mathrm{\infty })){|x|}^{-s-{\eta }_1{p}^{\ast }(s)}dx>0\end{array}$$

and (3.14) holds for $ϵ>0$ large. Similarly to above, we find that part (3) holds. □

To prove Theorem 2.2 we need the following version of the symmetric mountain pass theorem (cf.[28], Theorem 9.12]).

Lemma 3.4

Let E be an infinite dimensional Banach space and let$\mathcal{F}\in {\mathcal{C}}^1(E,\mathbb{R})$be an even functional satisfying${(PS)}_c$condition for each c and$\mathcal{F}(0)=0$. Further, we suppose that:

• (i) there exist constants $\overline{\alpha }>0$ and $\rho >0$ such that $\mathcal{F}(u)\ge \overline{\alpha }$ for all $\parallel u\parallel =\rho$;

• (ii) there exists an increasing sequence of subspaces $\{E_m\}$ of E, with $dim{E}_m=m$, such that for every m one can find a constant $R_m>0$ such that $\mathcal{F}(u)\le 0$ for all $u\in E_m$ with $\parallel u\parallel \ge R_m$.

Then ℱ possesses a sequence of critical values$\{c_m\}$tending to ∞ as$m\to \mathrm{\infty }$.

Proof of Theorem 2.2

Applying Lemma 3.4 with $E={\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$, we deduce from (q.1), (2.3), and (3.1) that

$$\mathcal{F}(u)\ge \frac{1}{p}{\parallel u\parallel }_{\mu }^p-\frac{{\parallel Q\parallel }_{\mathrm{\infty }}}{p^{\ast }(s)}{\mathcal{A}}_{\mu }^{-\frac{p^{\ast }(s)}{p}}{\parallel u\parallel }_{\mu }^{p^{\ast }(s)}.$$

Since $p^{\ast }(s)>p>1$, there exists $\overline{\alpha }>0$ and $\rho >0$ such that $\mathcal{F}(u)\ge \overline{\alpha }$ for all u with ${\parallel u\parallel }_{\mu }=\rho$. To find a suitable sequence of finite dimensional subspaces of ${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$, we set $\mathrm{\Omega }=\{x\in {\mathbb{R}}^N;Q(x)>0\}$. Since the set Ω is G-symmetric, we can define ${\mathcal{D}}_G^{1,p}(\mathrm{\Omega })$, which is the subspace of G-symmetric functions of ${\mathcal{D}}^{1,p}(\mathrm{\Omega })$ (see Section 2). By extending functions in ${\mathcal{D}}_G^{1,p}(\mathrm{\Omega })$ by 0 outside Ω we can assume that ${\mathcal{D}}_G^{1,p}(\mathrm{\Omega })\subset {\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$. Let $\{E_m\}$ be an increasing sequence of subspaces of ${\mathcal{D}}_G^{1,p}(\mathrm{\Omega })$ with $dim{E}_m=m$ for each m. Then there exists a constant $\xi (m)>0$ such that

$$\frac{1}{p^{\ast }(s)}{\int }_{\mathrm{\Omega }}Q(x)\frac{{|\nu |}^{p^{\ast }(s)}}{{|x|}^s}dx\ge \xi (m)\text{for all }\nu \in E_m,\text{ with }{\parallel \nu \parallel }_{\mu }=1.$$

Consequently, if $u\in E_m$, with $u\ne 0$, then we write $u=t\nu$, with $t={\parallel u\parallel }_{\mu }$ and ${\parallel \nu \parallel }_{\mu }=1$. Therefore we obtain

$$\mathcal{F}(u)=\frac{1}{p}{t}^p-\frac{1}{p^{\ast }(s)}{t}^{p^{\ast }(s)}{\int }_{\mathrm{\Omega }}Q(x)\frac{{|\nu |}^{p^{\ast }(s)}}{{|x|}^s}dx\le \frac{1}{p}{t}^p-\xi (m)t^{p^{\ast }(s)}\le 0$$

for t large enough. By Corollary 3.1 and Lemma 3.4 we conclude that there exists a sequence of critical values $c_m\to \mathrm{\infty }$ as $m\to \mathrm{\infty }$ and the result follows. □

4 Existence results for problem (${\mathcal{P}}_h^{\overline{Q}}$)

The aim of this section is to discuss problem (${\mathcal{P}}_h^{\overline{Q}}$) and prove Theorem 2.3; here $Q(x)\equiv \overline{Q}>0$ is a constant. First, we give the following compact embedding result which is indispensable for the proof of Theorem 2.3.

Lemma 4.1

Suppose that (h.2) holds. Then${\mathcal{D}}^{1,p}({\mathbb{R}}^N)$is compactly embedded in$L^q({\mathbb{R}}^N,h(x))$. Furthermore, if h satisfies (h.1) and (h.2) and$G\subset O(\mathbb{N})$is closed, then the inclusion of${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$in$L^q({\mathbb{R}}^N,h(x))$is compact.

Proof

We follow the argument of [4], Lemma 2.2]. Let $R_1>0$ and $R_2>0$ such that $0<R_1<R_2$. By (h.2), we can define the following integrals:

$$\begin{array}{r}I(u)={\int }_{{\mathbb{R}}^N}h(x){|u|}^{q}dx,I_1(u)={\int }_{|x|<R_1}h(x){|u|}^{q}dx,\\ I_2(u)={\int }_{|x|>R_2}h(x){|u|}^{q}dx,I_3(u)={\int }_{R_1\le |x|\le R_2}h(x){|u|}^{q}dx.\end{array}$$

For $R_1>0$ sufficiently small, we deduce from (h.2), (2.1), (2.2), the Hölder inequality, and the fact $N-1-\frac{s{p}^{\ast }}{p^{\ast }-q}>-1$, $N(1-\frac{q}{p^{\ast }})-s>0$ that

$$\begin{array}{rl}{I}_1(u)& \le C{\int }_{|x|<R_1}\frac{{|u|}^q}{{|x|}^s}dx\le C{\left({\int }_{|x|<R_1}{|u|}^{p^{\ast }}dx\right)}^{\frac{q}{p^{\ast }}}{\left({\int }_{|x|<R_1}\frac{1}{{|x|}^{\frac{s{p}^{\ast }}{p^{\ast }-q}}}dx\right)}^{\frac{p^{\ast }-q}{p^{\ast }}}\\ \le C{\left({\int }_{|x|<R_1}{|\mathrm{\nabla }u|}^{p}dx\right)}^{\frac{q}{p}}{\left({\int }_{|x|<R_1}\frac{1}{{|x|}^{\frac{s{p}^{\ast }}{p^{\ast }-q}}}dx\right)}^{\frac{p^{\ast }-q}{p^{\ast }}}\\ \le C{\parallel u\parallel }_{\mu }^q{\left({\int }_0^{R_1}{r}^{N-1-\frac{s{p}^{\ast }}{p^{\ast }-q}}dr\right)}^{\frac{p^{\ast }-q}{p^{\ast }}}\le C{\parallel u\parallel }_{\mu }^q{R}_1^{N(1-\frac{q}{p^{\ast }})-s}\to 0\end{array}$$

(4.1)

as $R_1\to 0$. Also, for $R_2>0$ large enough, we obtain from (h.2), the Hölder inequality, and the fact $N-1-\frac{\vartheta p^{\ast }}{p^{\ast }-q}<-1$, $N(1-\frac{q}{p^{\ast }})-\vartheta <0$ that

$$\begin{array}{rl}{I}_2(u)& \le C{\int }_{|x|>R_2}\frac{{|u|}^q}{{|x|}^{\vartheta }}dx\le C{\left({\int }_{|x|>R_2}{|u|}^{p^{\ast }}dx\right)}^{\frac{q}{p^{\ast }}}{\left({\int }_{|x|>R_2}\frac{1}{{|x|}^{\frac{\vartheta p^{\ast }}{p^{\ast }-q}}}dx\right)}^{\frac{p^{\ast }-q}{p^{\ast }}}\\ \le C{\left({\int }_{|x|>R_2}{|\mathrm{\nabla }u|}^{p}dx\right)}^{\frac{q}{p}}{\left({\int }_{|x|>R_2}\frac{1}{{|x|}^{\frac{\vartheta p^{\ast }}{p^{\ast }-q}}}dx\right)}^{\frac{p^{\ast }-q}{p^{\ast }}}\\ \le C{\parallel u\parallel }_{\mu }^q{\left({\int }_{R_2}^{+\mathrm{\infty }}{r}^{N-1-\frac{\vartheta p^{\ast }}{p^{\ast }-q}}dr\right)}^{\frac{p^{\ast }-q}{p^{\ast }}}\le C{\parallel u\parallel }_{\mu }^q{R}_2^{N(1-\frac{q}{p^{\ast }})-\vartheta }\to 0\end{array}$$

(4.2)

as $R_2\to \mathrm{\infty }$. Suppose that $\omega =\{x\in {\mathbb{R}}^N;R_1<|x|<R_2\}$ and $\{u_n\}$ is bounded in ${\mathcal{D}}^{1,p}({\mathbb{R}}^N)$. We may assume $u_n\rightharpoonup u$ in ${\mathcal{D}}^{1,p}({\mathbb{R}}^N)$. By the compactness of the inclusion of ${\mathcal{D}}^{1,p}(\omega )$ in $L^q(\omega )$ and the local boundedness of $h(x)$, we easily see that ${lim}_{n\to \mathrm{\infty }}{I}_3(u_n-u)=0$. Therefore, by taking $R_1\to 0$ and $R_2\to \mathrm{\infty }$, we conclude from (4.1) and (4.2) that ${lim}_{n\to \mathrm{\infty }}I(u_n-u)=0$. This implies the compactness of the inclusion of ${\mathcal{D}}^{1,p}({\mathbb{R}}^N)$ in $L^q({\mathbb{R}}^N,h(x))$.

On the other hand, since $G\subset O(\mathbb{N})$ is closed and $O(\mathbb{N})$ is a compact Lie group, G is compact. Consequently, by using the first part of the proof and the methods in Schneider [29], Corollaries 3.4 and 3.2], we deduce that ${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$ is compactly embedded in $L^q({\mathbb{R}}^N,h(x))$ and the results follow. □

Since we are interested in positive G-symmetric solutions of (${\mathcal{P}}_h^{\overline{Q}}$), we define a functional $J_h:{\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)\to \mathbb{R}$ given by

$$J_h(u)=\frac{1}{p}{\parallel u\parallel }_{\mu }^p-\frac{\overline{Q}}{p^{\ast }(s)}{\int }_{{\mathbb{R}}^N}\frac{{|u^{+}|}^{p^{\ast }(s)}}{{|x|}^s}dx-\frac{1}{q}{\int }_{{\mathbb{R}}^N}h(x){\left|u^{+}\right|}^{q}dx,$$

(4.3)

where $u^{+}=max\{0,u\}$. By (2.1), (h.1), (h.2), and Lemma 4.1, we easily see that $J_h$ is well defined and of ${\mathcal{C}}^1$. Thus there exists a one-to-one correspondence between the weak solutions of (${\mathcal{P}}_h^{\overline{Q}}$) and the critical points of $J_h$. Moreover, an analogously symmetric criticality principle of Lemma 3.1 clearly holds; hence the weak solutions of problem (${\mathcal{P}}_h^{\overline{Q}}$) are exactly the critical points of $J_h$.

Recall that the extremal function $y_{ϵ}(x)$ satisfies (2.4)-(2.9). By (h.2), we can choose $\varrho >0$ such that $B(0,2\varrho )\subset \mathcal{O}$ and define a function $\varphi \in {\mathcal{C}}^1({\mathbb{R}}^N)$ such that $0\le \varphi (x)\le 1$, $\varphi (x)=1$ for $|x|\le \varrho$, $\varphi (x)=0$ for $|x|\ge 2\varrho$ and $|\mathrm{\nabla }\varphi |\le 4/\varrho$ on ${\mathbb{R}}^N$. Using the methods in [3], [27], we deduce from (2.4)-(2.9) that

$${\parallel \varphi y_{ϵ}\parallel }_{\mu }^p={\int }_{{\mathbb{R}}^N}({|\mathrm{\nabla }(\varphi y_{ϵ})|}^p-\mu \frac{{|\varphi y_{ϵ}|}^p}{{|x|}^p})dx=1+O\left({ϵ}^{-N+p+p{\eta }_2}\right),$$

(4.4)

$${\int }_{{\mathbb{R}}^N}\frac{{|\varphi y_{ϵ}|}^{p^{\ast }(s)}}{{|x|}^s}dx={\mathcal{A}}_{\mu }^{-\frac{N-s}{N-p}}+O\left({ϵ}^{-N+s+{\eta }_2{p}^{\ast }(s)}\right),$$

(4.5)

$${\int }_{{\mathbb{R}}^N}\frac{{|\varphi y_{ϵ}|}^q}{{|x|}^s}dx=\{\begin{array}{cc}O({ϵ}^{q({\eta }_2+1-\frac{N}{p})}),\hfill & 1\le q<\frac{N-s}{{\eta }_2},\hfill \\ O({ϵ}^{N-s+(1-\frac{N}{p})q}|lnϵ|),\hfill & q=\frac{N-s}{{\eta }_2},\hfill \\ O({ϵ}^{N-s+(1-\frac{N}{p})q}),\hfill & \frac{N-s}{{\eta }_2}<q<p^{\ast }(s).\hfill \end{array}$$

(4.6)

Set $V_{ϵ}=\varphi y_{ϵ}/{\parallel \varphi y_{ϵ}\parallel }_{\mu }$; then by (4.4), (4.5), and (4.6) we have

$${\int }_{{\mathbb{R}}^N}{|x|}^{-s}{|V_{ϵ}|}^{p^{\ast }(s)}dx={\int }_{{\mathbb{R}}^N}\frac{{|x|}^{-s}{|\varphi y_{ϵ}|}^{p^{\ast }(s)}}{{\parallel \varphi y_{ϵ}\parallel }_{\mu }^{p^{\ast }(s)}}dx={\mathcal{A}}_{\mu }^{-\frac{N-s}{N-p}}+O\left({ϵ}^{-N+p+p{\eta }_2}\right),$$

(4.7)

$$\{\begin{array}{cc}{C}_1{ϵ}^{q({\eta }_2+1-\frac{N}{p})}\le {\int }_{{\mathbb{R}}^N}\frac{{|V_{ϵ}|}^q}{{|x|}^s}dx\le C_2{ϵ}^{q({\eta }_2+1-\frac{N}{p})},\hfill & 1\le q<\frac{N-s}{{\eta }_2},\hfill \\ C_3{ϵ}^{N-s+(1-\frac{N}{p})q}|lnϵ|\le {\int }_{{\mathbb{R}}^N}\frac{{|V_{ϵ}|}^q}{{|x|}^s}dx\le C_4{ϵ}^{N-s+(1-\frac{N}{p})q}|lnϵ|,\hfill & q=\frac{N-s}{{\eta }_2},\hfill \\ C_5{ϵ}^{N-s+(1-\frac{N}{p})q}\le {\int }_{{\mathbb{R}}^N}\frac{{|V_{ϵ}|}^q}{{|x|}^s}dx\le C_6{ϵ}^{N-s+(1-\frac{N}{p})q},\hfill & \frac{N-s}{{\eta }_2}<q<p^{\ast }(s).\hfill \end{array}$$

(4.8)

Lemma 4.2

Suppose that (h.1), (h.2), and (2.13) hold. Then there exists some$v_0\in {\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)\mathrm{\setminus }\{0\}$, $v_0\ge 0$and$v_0\not\equiv 0$on${\mathbb{R}}^N$, such that

$$\underset{t\ge 0}{sup}{J}_h(t{v}_0)<\frac{p-s}{(N-s)p}{\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}{\overline{Q}}^{\frac{p-N}{p-s}}.$$

(4.9)

Proof

Recall that $V_{ϵ}=\varphi y_{ϵ}/{\parallel \varphi y_{ϵ}\parallel }_{\mu }$, which satisfies (4.7) and (4.8). In the following, we will show that $V_{ϵ}$ satisfies (4.9) for $ϵ>0$ sufficiently small. Set

$$\mathrm{\Psi }(t)=J_h(t{V}_{ϵ})=\frac{t^p}{p}-\frac{\overline{Q}}{p^{\ast }(s)}{t}^{p^{\ast }(s)}{\int }_{{\mathbb{R}}^N}{|x|}^{-s}{|V_{ϵ}|}^{p^{\ast }(s)}dx-\frac{t^q}{q}{\int }_{{\mathbb{R}}^N}h(x){|V_{ϵ}|}^{q}dx$$

and

$$\tilde{\mathrm{\Psi }}(t)=\frac{t^p}{p}-\frac{\overline{Q}}{p^{\ast }(s)}{t}^{p^{\ast }(s)}{\int }_{{\mathbb{R}}^N}{|x|}^{-s}{|V_{ϵ}|}^{p^{\ast }(s)}dx$$

with $t\ge 0$; we deduce from $p^{\ast }(s)>q>p>1$ and (h.2) that $\mathrm{\Psi }(0)=0$, $\mathrm{\Psi }(t)>0$ for $t\to 0^{+}$, and ${lim}_{t\to +\mathrm{\infty }}\mathrm{\Psi }(t)=-\mathrm{\infty }$. Therefore ${sup}_{t\ge 0}\mathrm{\Psi }(t)$ can be achieved at some $t_{ϵ}>0$ for which we obtain

$$t_{ϵ}^{p-1}-t_{ϵ}^{p^{\ast }(s)-1}\overline{Q}{\int }_{{\mathbb{R}}^N}{|x|}^{-s}{|V_{ϵ}|}^{p^{\ast }(s)}dx-t_{ϵ}^{q-1}{\int }_{{\mathbb{R}}^N}h(x){|V_{ϵ}|}^{q}dx=0.$$

(4.10)

Consequently, we deduce from (h.2), (4.7), and (4.10) that

$$0<t_{ϵ}\le {\left(\overline{Q}{\int }_{{\mathbb{R}}^N}{|x|}^{-s}{|V_{ϵ}|}^{p^{\ast }(s)}dx\right)}^{\frac{1}{p-p^{\ast }(s)}}\triangleq t_{ϵ}^0\le C.$$

(4.11)

On the other hand, the function $\tilde{\mathrm{\Psi }}(t)$ attains its maximum at $t_{ϵ}^0$ and is increasing on the interval $[0,t_{ϵ}^0]$, together with (4.3), (4.7), (4.11), and $h(x)\ge C{|x|}^{-s}$ which is directly got from (h.2), we obtain

$$\begin{array}{rl}\underset{t\ge 0}{sup}{J}_h(t{V}_{ϵ})& =\mathrm{\Psi }(t_{ϵ})=\tilde{\mathrm{\Psi }}(t_{ϵ})-\frac{t_{ϵ}^q}{q}{\int }_{{\mathbb{R}}^N}h(x){|V_{ϵ}|}^{q}dx\le \tilde{\mathrm{\Psi }}\left(t_{ϵ}^0\right)-\frac{t_{ϵ}^q}{q}{\int }_{{\mathbb{R}}^N}h(x){|V_{ϵ}|}^{q}dx\\ \le (\frac{1}{p}-\frac{1}{p^{\ast }(s)}){\left(\overline{Q}{\int }_{{\mathbb{R}}^N}{|x|}^{-s}{|V_{ϵ}|}^{p^{\ast }(s)}dx\right)}^{\frac{p}{p-p^{\ast }(s)}}-C{\int }_{{\mathbb{R}}^N}{|x|}^{-s}{|V_{ϵ}|}^{q}dx\\ =\frac{p-s}{(N-s)p}{\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}{\overline{Q}}^{\frac{p-N}{p-s}}+O\left({ϵ}^{-N+p+p{\eta }_2}\right)-C{\int }_{{\mathbb{R}}^N}{|x|}^{-s}{|V_{ϵ}|}^{q}dx.\end{array}$$

(4.12)

Furthermore, we easily check from (2.13) that

$$-N+p+p{\eta }_2>N-s+(1-\frac{N}{p})q.$$

(4.13)

Consequently, choosing $ϵ>0$ small enough, we deduce from (4.8), (4.12), and (4.13) that

$$\underset{t\ge 0}{sup}{J}_h(t{V}_{ϵ})=\mathrm{\Psi }(t_{ϵ})<\frac{p-s}{(N-s)p}{\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}{\overline{Q}}^{\frac{p-N}{p-s}}.$$

Therefore we conclude that $V_{ϵ}$ satisfies (4.9) for $ϵ>0$ sufficiently small and the result follows. □

Lemma 4.3

Suppose that (h.1) and (h.2) hold. Then the${(PS)}_c$condition in${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$holds for$J_h$if

$$c<\frac{p-s}{(N-s)p}{\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}{\overline{Q}}^{\frac{p-N}{p-s}}.$$

(4.14)

Proof

Let $\{u_n\}\subset {\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$ be a ${(PS)}_c$ sequence for $J_h$ with c satisfying (4.14). Then by (4.3) and the fact that $1<p<p^{\ast }(\vartheta )<q<p^{\ast }(s)$, there exists $n_0\ge 1$ such that for $n\ge n_0$, we have

$$\begin{array}{rl}c+1& \ge J_h(u_n)-\frac{1}{q}〈J_h^{\prime }(u_n),u_n〉+\frac{1}{q}〈J_h^{\prime }(u_n),u_n〉\\ =(\frac{1}{p}-\frac{1}{q}){\parallel u_n\parallel }_{\mu }^p+\overline{Q}(\frac{1}{q}-\frac{1}{p^{\ast }(s)}){\int }_{{\mathbb{R}}^N}\frac{{|u_n^{+}|}^{p^{\ast }(s)}}{{|x|}^s}dx+\frac{1}{q}〈J_h^{\prime }(u_n),u_n〉\\ \ge (\frac{1}{p}-\frac{1}{q}){\parallel u_n\parallel }_{\mu }^p+o(1){\parallel u_n\parallel }_{\mu }.\end{array}$$

This implies that $\{u_n\}$ is bounded in ${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$. Consequently, just as in Lemma 3.3, we may assume that $u_n\rightharpoonup u$ in ${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$ and in $L^{p^{\ast }(s)}({\mathbb{R}}^N,{|x|}^{-s})$; moreover, $u_n\to u$ in $L^q({\mathbb{R}}^N,h(x))$ (see Lemma 4.1) and a.e. on ${\mathbb{R}}^N$. A standard argument shows that u is a critical point of $J_h$, and hence

$$J_h(u)=\overline{Q}(\frac{1}{p}-\frac{1}{p^{\ast }(s)}){\int }_{{\mathbb{R}}^N}\frac{{|u^{+}|}^{p^{\ast }(s)}}{{|x|}^s}dx+(\frac{1}{p}-\frac{1}{q}){\int }_{{\mathbb{R}}^N}h(x){\left|u^{+}\right|}^{q}dx\ge 0.$$

(4.15)

Now we set ${\nu }_n=u_n-u$; then we apply the Brezis-Lieb lemma [30] to the sequence ${|x|}^{-s}{|u_n^{+}|}^{p^{\ast }(s)}$ and use the condition (h.2) and the fact that u is a critical point of $J_h$ to obtain

$${\parallel {\nu }_n\parallel }_{\mu }^p=\overline{Q}{\int }_{{\mathbb{R}}^N}{|x|}^{-s}{\left|{\nu }_n^{+}\right|}^{p^{\ast }(s)}dx+o(1)$$

(4.16)

and

$$J_h(u)+\frac{1}{p}{\parallel {\nu }_n\parallel }_{\mu }^p-\frac{\overline{Q}}{p^{\ast }(s)}{\int }_{{\mathbb{R}}^N}{|x|}^{-s}{\left|{\nu }_n^{+}\right|}^{p^{\ast }(s)}dx=c+o(1).$$

(4.17)

Consequently, for a subsequence of $\{{\nu }_n\}$ one gets

$${\parallel {\nu }_n\parallel }_{\mu }^p\to k\ge 0\text{and}\overline{Q}{\int }_{{\mathbb{R}}^N}{|x|}^{-s}{\left|{\nu }_n^{+}\right|}^{p^{\ast }(s)}dx\to k\text{as }n\to \mathrm{\infty }.$$

It follows from (2.3) that $k\ge {\mathcal{A}}_{\mu }{(k/\overline{Q})}^{\frac{p}{p^{\ast }(s)}}$, which implies either $k=0$ or $k\ge {\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}{\overline{Q}}^{\frac{p-N}{p-s}}$. If $k\ge {\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}{\overline{Q}}^{\frac{p-N}{p-s}}$, we deduce from (4.15), (4.16), and (4.17) that

$$c=J_h(u)+(\frac{1}{p}-\frac{1}{p^{\ast }(s)})k\ge \frac{p-s}{(N-s)p}{\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}{\overline{Q}}^{\frac{p-N}{p-s}},$$

which contradicts (4.14). Therefore, we obtain ${\parallel {\nu }_n\parallel }_{\mu }^p\to 0$ as $n\to \mathrm{\infty }$, and hence, $u_n\to u$ in ${\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$. The proof of this lemma is completed. □

Proof of Theorem 2.3

For any $u\in {\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)\mathrm{\setminus }\{0\}$, we obtain from (h.2), (2.1), (2.3), (4.3), and the Hölder inequality

$$J_h(u)\ge \frac{1}{p}{\parallel u\parallel }_{\mu }^p-\frac{\overline{Q}}{p^{\ast }(s)}{\mathcal{A}}_{\mu }^{-\frac{p^{\ast }(s)}{p}}{\parallel u\parallel }_{\mu }^{p^{\ast }(s)}-C{\parallel u\parallel }_{\mu }^q.$$

Therefore, there exist constants $\tilde{\alpha }>0$ and $\rho >0$ such that $J_h(u)\ge \tilde{\alpha }$ for all ${\parallel u\parallel }_{\mu }=\rho$. Moreover, since $J_h(tu)\to -\mathrm{\infty }$ as $t\to \mathrm{\infty }$, there exists $\tilde{t}>0$ such that ${\parallel \tilde{t}u\parallel }_{\mu }>\rho$ and $J_h(\tilde{t}u)<0$. Now we set

$$c_h=\underset{\gamma \in \mathrm{\Gamma }}{inf}\underset{t\in [0,1]}{max}{J}_h(\gamma (t)),$$

where $\mathrm{\Gamma }=\{\gamma \in \mathcal{C}([0,1],{\mathcal{D}}_G^{1,p}({\mathbb{R}}^N));\gamma (0)=0,J_h(\gamma (1))<0,{\parallel \gamma (1)\parallel }_{\mu }>\rho \}$. By the mountain pass theorem in Ref. [26] (see also [27]), we conclude that there exists a sequence $\{u_n\}\subset {\mathcal{D}}_G^{1,p}({\mathbb{R}}^N)$ such that $J_h(u_n)\to c_h\ge \tilde{\alpha }$, $J_h^{\prime }(u_n)\to 0$ as $n\to \mathrm{\infty }$. Let $v_0$ be the function obtained in Lemma 4.2. Then we have

$$0<\tilde{\alpha }\le c_h\le \underset{t\in [0,1]}{sup}{J}_h(t{v}_0)<\frac{p-s}{(N-s)p}{\mathcal{A}}_{\mu }^{\frac{N-s}{p-s}}{\overline{Q}}^{\frac{p-N}{p-s}}.$$

Combining the above inequality and Lemma 4.3, we obtain a critical point $u_1$ of $J_h$ satisfying (${\mathcal{P}}_h^{\overline{Q}}$). Taking $u_1^{-}=min\{0,u_1\}$ as the test function, we get $0=〈J_h^{\prime }(u_1),u_1^{-}〉={\parallel u_1^{-}\parallel }_{\mu }^p$. This implies $u_1\ge 0$ in ${\mathbb{R}}^N$. By the strong maximum principle, we obtain $u_1>0$ in ${\mathbb{R}}^N$. This, combined with the symmetric criticality principle, implies that $u_1$ is a positive G-symmetric solution of (${\mathcal{P}}_h^{\overline{Q}}$). □