1 Introduction

In this paper, we consider the following quasilinear elliptic problem with critical Sobolev exponent:

$$ \textstyle\begin{cases} -\varepsilon^{p}\Delta_{p}u+V(x)\vert u\vert ^{p-2}u=Q(x)\vert u\vert ^{p^{*}-2}u +P(x)\vert u\vert ^{ {q-2}}u, & x\in \Omega , \\ \vert \nabla u\vert ^{p-2}\frac{\partial u}{\partial \nu }=0, &x\in \partial \Omega , \end{cases} $$
(1.1)

where \(\Omega \subset R^{N}\) is a bounded domain with smooth boundary, \(\Delta_{p}u=\operatorname{div}(\vert \nabla u\vert ^{p-2}\nabla u)\), \(\varepsilon >0\), \(1< p<N\), \(p < q < p^{*}=\frac{Np}{N-p}\), ν denotes the unit outward normal vector with respect to ∂Ω. The weight functions \(V(x)\), \(Q(x)\) and \(P(x)\) are continuous on Ω. Such problems arise in the theory of quasiregular and quasiconformal mapping or in the study of non-Newtonian fluids. In the latter case, the p is a characteristic of the medium. Media with \(p > 2\) are called dilatant fluids and those with \(p < 2\) are called pseudoplastics. If \(p = 2\), they are Newtonian fluids.

The early study of Laplacian elliptic equation with critical Sobolev exponent was Pohozaev [1], the author established the nonexistence of nontrivial solution to the Dirichlet problems when Ω is a star-shaped domain with respect to the origin. Later, Brézis and Nirenberg [2] showed the existence of positive solutions by introducing the low-order perturbation terms, and Struwe [3] also obtained the global compactness result. Since then, the study of these elliptic problems with critical growth terms have been paid wide attentions in recent years (see [4–7]). Set \(p=2\), \(\varepsilon =1\), \(P(x)=0\), \(V(x)=\lambda \), then Problem (1.1) reduces to the following semilinear elliptic problem:

$$ \textstyle\begin{cases} -\Delta u+\lambda u=Q(x)\vert u\vert ^{2^{*}-2}u,& x\in \Omega , \\ \frac{\partial u}{\partial v}=0,& x\in \Omega . \end{cases} $$
(1.2)

Comte and Knaap [8] proved that there exists a nontrivial solution of problem (1.2) by variational method if \(Q(x)=1\) and \(\lambda =-\mu \). Chabrowski and Willem [9] studied this problem with the assumption that the function \(Q(x)\) is nonnegative and Hölder continuous, they obtained the existence of least energy solutions by solving minimization problem corresponding to

$$ \begin{aligned} S_{\lambda }= \inf_{u\in H^{1}(\Omega ), \int_{\Omega }Q(x)\vert u\vert ^{2^{*}}\,\mathrm{d}x \neq 0} \frac{\int_{\Omega }(\vert \nabla u\vert ^{2}+\lambda u^{2})\,\mathrm{d}x}{( \int_{\Omega }Q(x)\vert u\vert ^{2^{*}}\,\mathrm{d}x)^{\frac{2}{2^{*}}}}. \end{aligned} $$

Subsequently, Chabrowski and Girão [10] investigated the existence and nonexistence of least energy solutions when the function \(Q(x)\) has some symmetry properties. For more relevant information as regards the corresponding problems, the interested reader may refer to [11–21] and the references therein.

As for quasilinear elliptic problems with critical Sobolev exponent, the existence and multiplicity of solutions have also been studied extensively. Abreu et al. [22] studied the following nonhomogeneous Neumann boundary problems:

$$ \textstyle\begin{cases} -\Delta_{p} u+\lambda u^{p-1}=u^{q}, &x\in \Omega , \\ u>0, &x\in \Omega , \\ \vert \nabla u\vert ^{p-2}\frac{\partial u}{\partial v}=\varphi , &x\in \partial \Omega , \end{cases} $$
(1.3)

where \(p-1< q\leq p^{*}-1\), \(\varphi \in C^{\alpha }(\overline{\Omega })\), \(0<\alpha <1\), \(\varphi \not \equiv 0\). They proved that there exists a \(\lambda^{*}>0\) such that problem (1.3) has at least two positive solutions if \(\lambda >\lambda^{*}\), has at least one positive solution if \(\lambda =\lambda^{*}\) and has no positive solution if \(\lambda <\lambda^{*}\) relying on the lower and upper solutions method and variational approach. Zhao et al. [23] discussed the quasilinear elliptic problem of the form

$$ \textstyle\begin{cases} -\Delta_{p} u+\lambda (x) \vert u\vert ^{p-2}u=\vert u\vert ^{p^{*}-2}u + \vert u\vert ^{r-2}u, &x \in \Omega , \\ \vert \nabla u\vert ^{p-2}\frac{\partial u}{\partial v}=\eta \vert u\vert ^{p-2}u , &x \in \partial \Omega , \end{cases} $$
(1.4)

they showed that there exists at least a nontrivial solution when \(p < r < p^{*}\) and there exist infinitely many solutions when \(1 < r < p\) by using the Mountain pass theorem and the concentration-compactness principle. Some authors also studied the critical Sobolev exponent for quasilinear equations and the corresponding evolution problems with Neumann boundary conditions, the reader may also refer to [24–37].

Motivated by the results of the above papers, we discuss the existence of nontrivial nonnegative solutions to Problem (1.1) by a variational method. The special features of this problem are the following. Firstly, due to the lack of compactness of the embedding of \(W^{1,p}(\Omega ) \hookrightarrow L^{p^{*}}(\Omega )\), we cannot use the standard variational argument directly. In order to overcome this difficulty and obtain the existence of solutions, we have to add restrictions on the weight functions \(Q(x)\) and \(P(x)\) to prove the corresponding functional of Problem (1.1) satisfies \((PS)_{c}\)-condition in a suitable range by the Lions concentration-compactness principle. Secondly, the weight function \(V(x)\) may be unbounded near the boundary ∂Ω, which leads to the space \(W^{1,p}(\Omega )\) is not suitable for our problem. To solve such problem, we have to introduce a suitable weighted Sobolev space.

For the sake of convenience, we introducing a new parameter \(\lambda =\varepsilon^{-p}\), then Problem (1.1) may be rewritten as the following problem:

$$ \textstyle\begin{cases} -\Delta_{p} u+\lambda V(x)\vert u\vert ^{p-2}u=\lambda Q(x)\vert u\vert ^{p^{*}-2}u+ \lambda P(x)\vert u\vert ^{q-2}u,&x\in \Omega , \\ \vert \nabla u\vert ^{p-2}\frac{\partial u}{\partial \nu }=0,&x\in \partial \Omega . \end{cases} $$
(1.5)

Throughout this paper, we make some assumptions on the weight functions \(Q(x)\), \(P(x)\), \(V(x)\) as the following:

  1. (A1)

    \(Q(x)\), \(P(x)\) are continuous on Ω̅, and \(Q(x)>0\), \(P(x)\geq 0\) for \(x\in \overline{\Omega }\);

  2. (A2)

    \(V(x)\) is continuous in Ω, and \(V(x)\geq 0\), \(V(x)\not \equiv 0\) for \(x\in \Omega \).

Set \(Q_{m}=\max_{x\in \partial \Omega }Q(x)\), \(Q_{M}= \max_{x\in \overline{\Omega }}Q(x)\), \(P_{M}=\max_{x\in \overline{ \Omega }}P(x)\). The main results of this paper are the following.

Theorem 1.1

Suppose that (A1), (A2) hold, \(H(0)>0\) and \(Q_{m}=Q(0)\). If functions \(Q(x)\), \(V(x)\) satisfy

  1. (A3)

    \(Q_{M}\leq 2^{\frac{p}{N-p}}Q_{m}\), and \(\vert Q(x)-Q(0)\vert =o(\vert x\vert ^{ \alpha })\) as \(x\rightarrow 0\), where \(1<\alpha <\frac{N}{p-1}\);

  2. (A4)

    \(\int_{\Omega \cap B(0,\delta )}V^{r^{\prime }}\,\mathrm{d}x< \infty \), where \(\frac{1}{r}+\frac{1}{r^{\prime }}=1\), \(1< r<\frac{N(p-1)}{Np+2p-N-p ^{2}-1}\), \(\delta >0\).

Then Problem (1.5) has at least one nontrivial solution for every \(\lambda >0\) and \(N\geq 2p\), where \(H(0)\) will be later determined.

Theorem 1.2

Suppose that (A1), (A2) hold. If \(Q_{M}>2^{\frac{p}{N-p}}Q_{m}\) and functions \(P(x)\), \(V(x)\) satisfy

  1. (A5)

    \(P(x)\not \equiv 0\) for \(x\in \Omega \), and \(V\in L^{1}(\Omega )\).

Then there exists a \(\lambda_{*}>0\) such that Problem (1.5) has at least one nontrivial solution for \(0<\lambda <\lambda_{*}\).

Theorem 1.3

Suppose that (A1), (A2) hold. If \(Q_{M}>2^{\frac{p}{N-p}}Q_{m}\) and functions \(P(x)\), \(V(x)\) satisfy

  1. (A6)

    \(P(x)>0\) for \(x\in \overline{\Omega }\);

  2. (A7)

    there exist \(x_{0}\in \Omega \) and constant \(\delta >0\) such that \(V(x)=0\) for \(x\in B(x_{0},\delta )\subset \Omega \).

Then there exists a \(\lambda^{*}>0\) such that Problem (1.5) has at least one nontrivial solution for \(\lambda >\lambda^{*}\).

Theorem 1.4

Suppose that (A1), (A2) hold. If \(Q_{M}>2^{\frac{p}{N-p}}Q_{m}\) and functions \(P(x)\), \(V(x)\) satisfy the conditions (A6) and (A7). Then, for every integer n, there exists a constant \(\Lambda_{n}>0\) such that Problem (1.5) has at least n pairs of nontrivial solutions for \(\lambda >\Lambda_{n}\).

2 Preliminaries

Firstly, we define the weighted Sobolev space

$$\begin{aligned} W_{\lambda ,V}^{1,p}(\Omega )= \biggl\{ u; D_{i}u\in L^{p}(\Omega ), i=1, 2, \ldots , N, \int_{\Omega }V(x)\vert u\vert ^{p}\,\mathrm{d}x< + \infty \biggr\} \end{aligned}$$

with norm \(\Vert u\Vert _{\lambda ,V}=(\int_{\Omega }(\vert \nabla u\vert ^{p}+\lambda V(x)\vert u\vert ^{p})\,\mathrm{d}x)^{\frac{1}{p}}\). Obviously, norms \(\Vert u\Vert _{ \lambda ,V}\) and \(\Vert u\Vert _{V}\) are equivalent, \(W^{1,p}_{\lambda ,V}( \Omega )\hookrightarrow W^{1,p}(\Omega )\), where \(\Vert u\Vert _{V}= ( \int_{\Omega }(\vert \nabla u\vert ^{p}+ V(x)\vert u\vert ^{p})\,\mathrm{d}x )^{ \frac{1}{p}}\).

By using the Sobolev embedding theorem, we know that there exists a constant \(C_{q}>0\) such that

$$ \begin{aligned} \vert u\vert _{q}\leq C_{q}\Vert u\Vert _{V}\leq C_{q}\Vert u \Vert _{\lambda V},\quad \mbox{for } \lambda \geq 1, u\in W_{\lambda ,V}^{1,p}( \Omega ), \end{aligned} $$
(2.1)

and

$$ \begin{aligned} \vert u\vert _{q}\leq C_{q}\Vert u\Vert _{V}\leq \lambda^{-\frac{1}{p}}C_{q} \Vert u\Vert _{ \lambda V},\quad \mbox{for }0< \lambda < 1, u\in W_{\lambda ,V}^{1,p}( \Omega ), \end{aligned} $$
(2.2)

where \(\vert u\vert _{q}= (\int_{\Omega }\vert u\vert ^{q}\,\mathrm{d}x )^{ \frac{1}{q}}\), \(q\in (p, p^{*})\).

Next, we give the definition of weak solution to Problem (1.5).

Definition 2.1

A function \(u\in W_{\lambda ,V}^{1,p}(\Omega )\) is said to be a weak solution of Problem (1.5) if it satisfies

$$\begin{aligned}& \int_{\Omega }\vert \nabla u\vert ^{p-2}\nabla u\nabla \psi \,\mathrm{d}x+ \lambda \int_{\Omega }V(x)\vert u\vert ^{p-2}u \psi \, \mathrm{d}x \\& \quad =\lambda \int_{\Omega }Q(x)\vert u\vert ^{p^{*}-2}u \psi \, \mathrm{d}x+\lambda \int_{\Omega }P(x)\vert u\vert ^{q-2} u \psi \, \mathrm{d}x,\quad \forall \psi \in W_{\lambda ,V}^{1,p}(\Omega ). \end{aligned}$$

Thus, the corresponding energy functional of Problem (1.5) is defined in \(W_{\lambda ,V}^{1,p}(\Omega )\) by

$$\begin{aligned} J_{\lambda }(u)=\frac{1}{p} \int_{\Omega } \bigl(\vert \nabla u\vert ^{p}+\lambda V(x)\vert u\vert ^{p} \bigr) \,\mathrm{d}x - \frac{\lambda }{p^{*}} \int_{\Omega }Q(x)\vert u\vert ^{p^{*}} \,\mathrm{d}x - \frac{\lambda }{q} \int_{\Omega }P(x)\vert u\vert ^{q} \,\mathrm{d}x. \end{aligned}$$

Let S be the best Sobolev constants, namely

$$ \begin{aligned} S=\inf_{D^{1,p}(R^{N})\setminus \{0\}}\frac{\int_{\Omega }\vert \nabla u\vert ^{p} \,\mathrm{d}x}{(\int_{\Omega }\vert u\vert ^{p^{*}}\,\mathrm{d}x)^{\frac{p}{p^{*}}}}, \end{aligned} $$
(2.3)

where \(D^{1,p}(R^{N})=\{u\in L^{p^{*}}(R^{N}): \vert \nabla u\vert \in L^{p}(R ^{N})\}\). This constant S is achieved by the functional \(u_{\varepsilon }\) given by

$$\begin{aligned} u_{\varepsilon }(x)=C_{Np}\varepsilon^{\frac{N-p}{p^{2}}} \bigl(\varepsilon +\vert x\vert ^{\frac{p}{p-1}} \bigr)^{\frac{p-N}{p}}, \end{aligned}$$

where the constant \(C_{Np}\) is chosen such that \(-\Delta_{p} u_{ \varepsilon }=\vert u_{\varepsilon }\vert ^{p^{*}-1}\) in \(R^{N}\) (see [22] for details).

In order to obtain the existence of solutions to Problem (1.5), we need the following lemma.

Lemma 2.1

For each \(\lambda >0\),

  1. (i)

    there exist constants \(\beta_{\lambda }, \rho_{\lambda }>0\) such that \(J_{\lambda }(u)\geq \beta_{\lambda }\) for \(\Vert u\Vert _{\lambda V}= \rho_{\lambda }\);

  2. (ii)

    there exists an \(u_{0}\in W_{\lambda ,V}^{1,p}(\Omega )\) with \(u_{0}\not \equiv 0\) such that \(J_{\lambda }(u_{0})<0\) for \(\Vert u_{0}\Vert _{\lambda V}>\rho_{\lambda }\).

Proof

(i) Firstly, we consider the case \(\lambda \geq 1\). let \(\Vert u\Vert _{V} ^{p}=\rho^{p}\), then \(\rho_{\lambda }^{p}=\Vert u\Vert _{\lambda V}^{p}\leq \lambda \rho^{p}\). Using (2.1) and (2.3), we have

$$\begin{aligned} J_{\lambda }(u) \geq &\frac{1}{p}\Vert u\Vert _{\lambda V}^{p}-\frac{\lambda }{p^{*}}Q_{M}S^{-\frac{p^{*}}{p}} \biggl( \int_{\Omega }\vert \nabla u\vert ^{p} \,\mathrm{d}x \biggr)^{\frac{p^{*}}{p}}-\frac{\lambda }{q}P_{M}C_{q}^{q} \Vert u\Vert _{\lambda V}^{q} \\ \geq &\frac{1}{p}\Vert u\Vert _{\lambda V}^{p}- \frac{\lambda }{p^{*}}Q_{M}S ^{-\frac{p^{*}}{p}}\Vert u\Vert _{\lambda V}^{p^{*}} -\frac{\lambda }{q}P_{M}C _{q}^{q}\Vert u\Vert _{\lambda V}^{q} \\ \geq &\rho_{\lambda }^{p} \biggl(\frac{1}{p}- \frac{\lambda^{ \frac{p^{*}}{p}}}{p^{*}}Q_{M}S^{-\frac{p^{*}}{p}}\rho^{p^{*}-p}- \frac{ \lambda^{\frac{q}{p}}}{q}P_{M}C_{q}^{q} \rho^{q-p} \biggr). \end{aligned}$$

Since \(p< q< p^{*}\), taking \(\rho >0\) small enough, there exists a \(\beta_{\lambda }>0\) such that \(J_{\lambda }(u)\geq \beta_{\lambda }\) for \(\Vert u\Vert _{\lambda V}=\rho_{\lambda }\).

If \(0<\lambda <1\), let \(\Vert u\Vert _{V}^{p}=\rho^{p}\), then \(\rho > \rho_{\lambda }>\lambda^{\frac{1}{p}} \rho \). Combining (2.2) with (2.3), we see that

$$\begin{aligned} J_{\lambda }(u) \geq &\frac{1}{p}\Vert u\Vert _{\lambda V}^{p}-\frac{\lambda }{p^{*}}Q_{M}S^{-\frac{p^{*}}{p}} \Vert u\Vert _{V}^{p^{*}}-\frac{\lambda }{q}P _{M}C_{q}^{q}\Vert u\Vert _{V}^{q} \\ >&\lambda \rho^{p} \biggl(\frac{1}{p}-\frac{1}{p^{*}}Q_{M}S^{- \frac{p^{*}}{p}} \rho^{p^{*}-p}-\frac{1}{q}P_{M}C_{q}^{q} \rho^{q-p} \biggr). \end{aligned}$$

Since \(p< q< p^{*}\), taking \(\rho >0\) small enough, there exists a \(\beta_{\lambda }>0\) such that \(J_{\lambda }(u)\geq \beta_{\lambda }\) for \(\Vert u\Vert _{\lambda V}=\rho_{\lambda }\).

(ii) For \(u\in W_{\lambda ,V}^{1,p}(\Omega )\) and \(u\not \equiv 0\), we define

$$\begin{aligned} J_{\lambda }(t u)=\frac{t^{p}}{p}\Vert u\Vert _{\lambda V}^{p}- \frac{t^{p^{*}}}{p ^{*}}\lambda \int_{\Omega }Q(x)\vert u\vert ^{p^{*}}\,\mathrm{d}x - \frac{t^{q}}{q}\lambda \int_{\Omega }P(x)\vert u\vert ^{q}\,\mathrm{d}x,\quad t>0 , \end{aligned}$$

it follows from \(\lim_{t\rightarrow +\infty }J_{\lambda }(t u)=- \infty \) that there exists a \(t_{0}>0\) such that \(\Vert t_{0}u\Vert _{\lambda V}>\rho_{\lambda }\) and \(J_{\lambda }(t_{0}u)<0\). Letting \(u_{0}=t _{0}u\), then condition (ii) holds. The proof of Lemma 2.1 is completed. □

Define

$$\begin{aligned} c=\inf_{h\in \Gamma }\sup_{t\in [0,1]}J_{\lambda } \bigl(h(t) \bigr), \end{aligned}$$

where \(\Gamma =\{h\in C([0,1], W_{\lambda ,V}^{1,p}(\Omega ))\mid h(0)=0, h(1)=t_{0}u=u_{0}\}\). Using Lemma 2.1, we know that the energy functional \(J_{\lambda }(u)\) satisfies the geometry of the mountain pass lemma, then there exists a \((PS)_{c}\)-sequence \(\{u_{n}\}\subset W _{\lambda ,V}^{1,p}(\Omega )\) such that \(J_{\lambda }(u_{n})\rightarrow c\), \(J_{\lambda }^{\prime }(u_{n})\rightarrow 0\) as \(n\rightarrow \infty \).

Lemma 2.2

Assume (A1), (A2) hold, and \(\{u_{n}\}\) be a \((PS)_{c}\)-sequence at the level of c for \(J_{\lambda }\) with \(c< c^{*}=\min \{\frac{S ^{\frac{N}{p}}}{N\lambda^{\frac{N-p}{p}}Q_{M}^{\frac{N-p}{p}}}, \frac{S ^{\frac{N}{p}}}{2N\lambda^{\frac{N-p}{p}}Q_{m}^{\frac{N-p}{p}}} \}\), then \(\{u_{n}\}\) is relatively compact in \(W_{\lambda ,V}^{1,p}( \Omega )\).

Proof

Firstly, we prove that \(\{u_{n}\}\) is bounded. Since \(J_{\lambda }(u _{n})\rightarrow c\), \(J_{\lambda }^{\prime }(u_{n})\rightarrow 0\) as \(n\rightarrow \infty \), we have

$$\begin{aligned}& J_{\lambda }(u_{n}) =\frac{1}{p} \int_{\Omega } \bigl(\vert \nabla u_{n}\vert ^{p}+ \lambda V(x)\vert u_{n}\vert ^{p} \bigr) \,\mathrm{d}x - \frac{\lambda }{p^{*}} \int_{ \Omega }Q(x)\vert u_{n}\vert ^{p^{*}} \,\mathrm{d}x -\frac{\lambda }{q} \int_{ \Omega }P(x)\vert u_{n}\vert ^{q} \,\mathrm{d}x \\& \hphantom{J_{\lambda }(u_{n})}=c+o(1)\Vert u_{n}\Vert , \\& \int_{\Omega } \bigl(\vert \nabla u_{n}\vert ^{p}+\lambda V(x)\vert u_{n}\vert ^{p} \bigr) \,\mathrm{d}x- \lambda \int_{\Omega }Q(x)\vert u_{n}\vert ^{p^{*}} \,\mathrm{d}x -\lambda \int_{\Omega }P(x)\vert u_{n}\vert ^{q} \,\mathrm{d}x \\& \quad =o(1)\Vert u_{n}\Vert . \end{aligned}$$

Combining (A1) and (A2), one has

$$\begin{aligned} c+o(1)\Vert u_{n}\Vert =& \biggl(\frac{1}{p}- \frac{1}{q} \biggr) \int_{\Omega } \bigl(\vert \nabla u_{n}\vert ^{p}+\lambda V(x)\vert u_{n}\vert ^{p} \bigr) \,\mathrm{d}x +\lambda \biggl(\frac{1}{q}-\frac{1}{p ^{*}} \biggr)\int_{\Omega } Q(x) \vert u_{n}\vert ^{p^{*}}\,\mathrm{d}x \\ \geq & \biggl(\frac{1}{p}-\frac{1}{q} \biggr)\Vert u_{n}\Vert _{\lambda V}^{p}. \end{aligned}$$

Thus, we can find that \(\{u_{n}\}\) is bounded in \(W_{\lambda ,V}^{1,p}( \Omega )\).

Next, we prove that \(\{u_{n}\}\) is relatively compact in \(W_{\lambda ,V}^{1,p}(\Omega )\). Since \(\{u_{n}\}\) is bounded in \(W_{\lambda ,V} ^{1,p}(\Omega )\), there exists a subsequence, still denoted by \(\{u_{n}\}\) and \(u \in W_{\lambda ,V}^{1,p}(\Omega )\) such that

$$\begin{aligned}& u_{n}\rightharpoonup u\quad \mbox{weakly in } W_{\lambda ,V}^{1,p}( \Omega ), \\& u_{n}\rightharpoonup u \quad \mbox{weakly in } L^{p^{*}}( \Omega ), \\& u_{n}\rightarrow u \quad \mbox{strongly in } L^{q}(\Omega ), p \leq q< p ^{*} , \\& u_{n}\rightarrow u\quad \mbox{a.e. in } \Omega . \end{aligned}$$

By the Lions concentration-compactness principle [38], there exists at most set J, a set of different points \(\{x_{j}\}_{j\in J}\subset \overline{ \Omega }\), sets of nonnegative real numbers \(\{\mu_{j}\}_{j\in J}\), \(\{\nu_{j}\}_{j\in J}\) such that

$$ \begin{aligned} &\vert \nabla u_{n}\vert ^{p}\rightharpoonup \,\mathrm{d}\mu \geq \vert \nabla u\vert ^{p}+ \sum_{j\in J}\mu_{j} \delta_{x_{j}}, \\ &\vert u_{n}\vert ^{p^{*}}\rightharpoonup \,\mathrm{d}\nu = \vert u\vert ^{p^{*}}+\sum_{j\in J} \nu_{j}\delta_{x_{j}}, \end{aligned} $$
(2.4)

where \(\delta_{x}\) is the Dirac mass at x, and the constants \(\mu_{j}\), \(\nu_{j}\) satisfying

$$\begin{aligned}& S\nu_{j}^{\frac{p}{p^{*}}}\leq \mu_{j},\quad \mbox{where } x_{j} \in \Omega , \end{aligned}$$
(2.5)
$$\begin{aligned}& \frac{S}{2^{\frac{p}{N}}}\nu_{j}^{\frac{p}{p^{*}}} \leq \mu_{j}, \quad \mbox{where } x_{j}\in \partial \Omega . \end{aligned}$$
(2.6)

Next, we prove \(\mu_{j}=0\) and \(\nu_{j}=0\), where \(j\in J\). In fact, choosing \(\varepsilon >0\) sufficiently small such that \(B_{\varepsilon }(x_{i})\cap B_{\varepsilon }(x_{j})=\varnothing \) for \(i\neq j\), \(i,j \in J\). Let \(\phi_{\varepsilon }^{j}(x)\) be a smooth cut off function centered at \(x_{j}\) such that

$$\begin{aligned} 0\leq \phi_{\varepsilon }^{j}(x)\leq 1 \quad \mbox{for } \vert x-x_{j}\vert < \varepsilon ,\quad \phi_{\varepsilon }^{j}(x)= \textstyle\begin{cases} 1, \vert x-x_{j}\vert \leq \frac{\varepsilon }{2}, \\ 0, \vert x-x_{j}\vert \geq \varepsilon , \end{cases}\displaystyle \mbox{and}\quad \bigl\vert \nabla \phi_{\varepsilon }^{j} \bigr\vert \leq \frac{4}{\varepsilon }. \end{aligned}$$

Noting that

$$\begin{aligned}& \bigl\langle J_{\lambda }^{\prime }(u_{n}),u_{n} \phi_{\varepsilon }^{j}(x) \bigr\rangle \\& \quad = \int_{\Omega }\vert \nabla u_{n}\vert ^{p} \phi_{\varepsilon }^{j}(x) \,\mathrm{d}x+ \int_{\Omega }\vert \nabla u_{n}\vert ^{p-2} \nabla u_{n}\nabla \phi_{\varepsilon }^{j}(x)u_{n} \,\mathrm{d}x \\& \qquad {}+\lambda \int_{\Omega }V(x)\vert u_{n}\vert ^{p} \phi_{\varepsilon }^{j}(x) \,\mathrm{d}x -\lambda \int_{\Omega }Q(x)\vert u_{n}\vert ^{p^{*}} \phi_{\varepsilon }^{j}(x)\,\mathrm{d}x \\& \qquad {}-\lambda \int_{\Omega }P(x)\vert u_{n}\vert ^{q} \phi_{\varepsilon }^{j}(x) \,\mathrm{d}x, \end{aligned}$$

and by (2.4), we have

$$\begin{aligned}& \lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \int_{\Omega }\vert \nabla u_{n}\vert ^{p} \phi_{\varepsilon }^{j}(x)\,\mathrm{d}x \geq \lim _{\varepsilon \rightarrow 0} \biggl[ \int_{\Omega }\vert \nabla u\vert ^{p} \phi_{\varepsilon }^{j}(x)\,\mathrm{d}x+ \int_{\Omega }\sum_{j\in J}\mu _{j}\delta_{x_{j}}\phi_{\varepsilon }^{j}(x) \, \mathrm{d}x \biggr] \geq \mu _{j}, \\& \lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \int_{\Omega }\vert \nabla u_{n}\vert ^{p-2} \nabla u_{n}\nabla \phi_{\varepsilon }^{j}(x)u_{n} \,\mathrm{d}x=0, \\& \lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \int_{\Omega }V(x)\vert u_{n}\vert ^{p} \phi_{\varepsilon }^{j}(x)\,\mathrm{d}x=0, \\ & \lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \int_{\Omega }Q(x)\vert u_{n}\vert ^{p^{*}} \phi_{\varepsilon }^{j}(x) \,\mathrm{d}x=Q(x_{j}) \nu_{j}, \\ & \lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \int_{\Omega }P(x)\vert u_{n}\vert ^{q} \phi_{\varepsilon }^{j}(x)\,\mathrm{d}x=0. \end{aligned}$$

Thus,

$$ 0=\lim_{\varepsilon \rightarrow 0}\lim_{n\rightarrow \infty } \bigl\langle J_{\lambda }^{\prime }(u_{n}),u_{n} \phi_{\varepsilon }^{j}(x) \bigr\rangle \geq \mu_{j}- \lambda Q(x_{j})\nu_{j}. $$

If \(\nu_{j}\neq 0\), by (2.5) and (2.6), we find that

$$\begin{aligned}& \nu_{j}\geq \frac{S^{\frac{N}{p}}}{\lambda^{\frac{N}{p}}Q^{ \frac{N}{p}}(x_{j})},\quad x_{j}\in \Omega , \\ & \nu_{j}\geq \frac{S^{\frac{N}{p}}}{2\lambda^{\frac{N}{p}}Q^{ \frac{N}{p}}(x_{j})},\quad x_{j}\in \partial \Omega . \end{aligned}$$

On the other hand,

$$\begin{aligned} c =&\lim_{n\rightarrow \infty } \biggl(J_{\lambda }(u_{n})- \frac{1}{p} \bigl\langle J_{\lambda }^{\prime }(u_{n}),u_{n} \bigr\rangle \biggr) \\ =& \biggl(\frac{1}{p}-\frac{1}{p^{*}} \biggr)\lambda \int_{\Omega }Q(x)\vert u\vert ^{p^{*}} \,\mathrm{d}x + \biggl(\frac{1}{p}-\frac{1}{q} \biggr)\lambda \int_{\Omega }P(x) \vert u\vert ^{q} \,\mathrm{d}x+ \biggl(\frac{1}{p}-\frac{1}{p^{*}} \biggr)\lambda \sum _{j\in J}Q(x_{j}) \nu_{j} \\ \geq &\frac{1}{N}\lambda \sum_{j\in J}Q(x_{j}) \nu_{j}, \end{aligned}$$

consequently,

$$\begin{aligned}& c\geq \frac{1}{N}\lambda Q(x_{j})\nu_{j}\geq \frac{S^{\frac{N}{p}}}{N \lambda^{\frac{N-p}{p}}Q^{\frac{N-p}{p}}_{M}},\quad x_{j}\in \Omega , \\ & c\geq \frac{1}{N}\lambda Q(x_{j})\nu_{j}\geq \frac{S^{\frac{N}{p}}}{2N \lambda^{\frac{N-p}{p}}Q^{\frac{N-p}{p}}_{m}}, \quad x_{j}\in \partial \Omega , \end{aligned}$$

which is a contradiction. Hence, \(\mu_{j}=0\), \(\nu_{j}=0\) and we find that \(u_{n}\rightarrow u\) strongly in \(L^{p^{*}}(\Omega )\).

Now, we prove that \(u_{n}\rightarrow u\) strongly in \(W_{\lambda ,V} ^{1,p}(\Omega )\). We have

$$\begin{aligned}& \bigl\langle J_{\lambda }^{\prime }(u_{n})-J_{\lambda }^{\prime }(u),u_{n}-u \bigr\rangle \\& \quad =\Vert u_{n}\Vert _{\lambda ,V}^{p}+ \Vert u \Vert _{\lambda ,V}^{p}- \int_{ \Omega } \bigl(\vert \nabla u_{n}\vert ^{p-2}\nabla u_{n}\nabla u+\lambda V(x)\vert u_{n}\vert ^{p-2}u _{n}u \bigr)\,\mathrm{d}x \\& \qquad {}- \int_{\Omega } \bigl(\vert \nabla u\vert ^{p-2}\nabla u \nabla u_{n}+\lambda V(x)\vert u\vert ^{p-2}uu _{n} \bigr)\,\mathrm{d}x-I-II, \end{aligned}$$

where

$$\begin{aligned}& I =\lambda \int_{\Omega }Q(x) \bigl(\vert u_{n}\vert ^{p^{*}-2}u_{n}-\vert u\vert ^{p^{*}-2}u \bigr) (u _{n}-u)\,\mathrm{d}x, \\& II =\lambda \int_{\Omega }P(x) \bigl(\vert u_{n}\vert ^{q-2}u_{n}-\vert u\vert ^{q-2}u \bigr) (u_{n}-u) \,\mathrm{d}x. \end{aligned}$$

By the Hölder inequality and Jensen’s inequality

$$\begin{aligned} (a+b)^{\alpha }(c+d)^{1-\alpha }\geq a^{\alpha }c^{1-\alpha }+b^{ \alpha }d^{1-\alpha }, \end{aligned}$$

where \(\alpha \in (0,1)\), \(a>0\), \(b>0\), \(c>0\), \(d>0\), we have

$$\begin{aligned}& \int_{\Omega } \bigl(\vert \nabla u_{n}\vert ^{p-2}\nabla u_{n}\nabla u+\lambda V(x)\vert u_{n}\vert ^{p-2}u_{n}u \bigr)\,\mathrm{d}x \\& \quad \leq \biggl( \int_{\Omega }\vert \nabla u_{n}\vert ^{p} \,\mathrm{d}x \biggr)^{ \frac{p-1}{p}} \biggl( \int_{\Omega }\vert \nabla u\vert ^{p}\,\mathrm{d}x \biggr)^{ \frac{1}{p}} \\& \qquad {}+ \biggl(\lambda \int_{\Omega }V(x)\vert u_{n}\vert ^{p} \, \mathrm{d}x \biggr)^{\frac{p-1}{p}} \biggl(\lambda \int_{\Omega }V(x)\vert u\vert ^{p} \,\mathrm{d}x \biggr)^{\frac{1}{p}} \\& \quad \leq \biggl( \int_{\Omega }\vert \nabla u_{n}\vert ^{p}+\lambda V(x)\vert u_{n}\vert ^{p} \, \mathrm{d}x \biggr)^{\frac{p-1}{p}} \biggl( \int_{\Omega }\vert \nabla u\vert ^{p}+ \lambda V(x) \vert u\vert ^{p}\,\mathrm{d}x \biggr)^{\frac{1}{p}} =\Vert u_{n}\Vert _{\lambda V}^{p-1}\Vert u \Vert _{\lambda V}. \end{aligned}$$

Similarly, we get

$$\begin{aligned}& \int_{\Omega } \bigl(\vert \nabla u\vert ^{p-2}\nabla u \nabla u_{n}+\lambda V(x)\vert u\vert ^{p-2}uu _{n} \bigr)\,\mathrm{d}x\leq \Vert u\Vert _{\lambda V}^{p-1} \Vert u_{n}\Vert _{\lambda V}, \\& \vert I\vert \leq \lambda Q_{M} \biggl[ \int_{\Omega }\vert u_{n}\vert ^{p^{*}-1} \vert u_{n}-u\vert \,\mathrm{d}x+ \int_{\Omega }\vert u\vert ^{p^{*}-1}\vert u_{n}-u\vert \,\mathrm{d}x \biggr] \\& \hphantom{\vert I\vert } \leq \lambda Q_{M} \biggl( \int_{\Omega }\vert u_{n}\vert ^{p^{*}} \, \mathrm{d}x \biggr)^{\frac{p^{*}-1}{p^{*}}} \biggl( \int_{\Omega }\vert u_{n}-u\vert ^{p^{*}} \, \mathrm{d}x \biggr)^{\frac{1}{p^{*}}} \\& \hphantom{\vert I\vert \leq} {}+\lambda Q_{M} \biggl( \int_{\Omega }\vert u\vert ^{p^{*}}\,\mathrm{d}x \biggr)^{\frac{p ^{*}-1}{p^{*}}} \biggl( \int_{\Omega }\vert u_{n}-u\vert ^{p^{*}} \, \mathrm{d}x \biggr)^{\frac{1}{p ^{*}}}, \\& \vert II\vert \leq \lambda P_{M} \biggl[ \int_{\Omega }\vert u_{n}\vert ^{q-1} \vert u_{n}-u\vert \,\mathrm{d}x+ \int_{\Omega }\vert u\vert ^{q-1}\vert u_{n}-u\vert \,\mathrm{d}x \biggr] \\& \hphantom{\vert II\vert } \leq \lambda P_{M} \biggl( \int_{\Omega }\vert u_{n}\vert ^{q} \, \mathrm{d}x \biggr)^{ \frac{q-1}{q}} \biggl( \int_{\Omega }\vert u_{n}-u\vert ^{q} \, \mathrm{d}x \biggr)^{ \frac{1}{q}} \\& \hphantom{\vert II\vert \leq} {}+\lambda P_{M} \biggl( \int_{\Omega }\vert u\vert ^{q}\,\mathrm{d}x \biggr)^{ \frac{q-1}{q}} \biggl( \int_{\Omega }\vert u_{n}-u\vert ^{q} \, \mathrm{d}x \biggr)^{ \frac{1}{q}}. \end{aligned}$$

We have

$$\begin{aligned} 0 =&\lim_{n\rightarrow \infty } \bigl\langle J_{\lambda }^{\prime }(u_{n})-J_{\lambda }^{\prime }(u),u_{n}-u \bigr\rangle \geq \lim_{n\rightarrow \infty } \bigl(\Vert u_{n}\Vert _{\lambda V}^{p-1}-\Vert u\Vert _{\lambda V}^{p-1} \bigr) \bigl(\Vert u_{n}\Vert _{\lambda V}-\Vert u\Vert _{\lambda V} \bigr) \geq 0. \end{aligned}$$

Hence, \(u_{n}\rightarrow u\) strongly in \(W_{\lambda ,V}^{1,p}(\Omega )\). □

Since \(0\in \partial \Omega \) and \(\partial \Omega \in C^{2}\), the boundary ∂Ω near the origin can be represented \(x_{N}=h(x^{\prime })=\frac{1}{2}\sum_{i=1}^{N-1}\lambda_{i}x_{i}^{2}+o(\vert x^{\prime }\vert ^{2})\), where \(x^{\prime }=(x_{1},x_{2},\ldots ,x_{N-1}) \in D(0,\delta )=B(0,\delta )\cap \{x_{N}=0\}, \lambda_{i}\) (\(i=1,2, \ldots ,N-1\)) are the principal curvatures of ∂Ω at 0 and the mean curvatures \(H(0)=\frac{1}{N-1}\sum_{i=1}^{N-1}\lambda _{i}>0\). Then the following lemma holds.

Lemma 2.3

([22])

(1) For \(N> 2p-1\) and \(\varepsilon >0\) small enough,

$$\begin{aligned}& \int_{\Omega }\vert \nabla u_{\varepsilon }\vert ^{p}\,\mathrm{d}x = \int_{R_{+} ^{N}}\vert \nabla u_{\varepsilon }\vert ^{p}\,\mathrm{d}x-K_{1}(\varepsilon )+o \bigl( \varepsilon^{\frac{p-1}{p}} \bigr), \\& \int_{\Omega }\vert u_{\varepsilon }\vert ^{p ^{*}} \, \mathrm{d}x= \int_{R_{+}^{N}}\vert u_{\varepsilon }\vert ^{p^{*}} \, \mathrm{d}x-K_{2}(\varepsilon )+o \bigl(\varepsilon^{\frac{p-1}{p}} \bigr), \end{aligned}$$

where \(K_{1}(\varepsilon )\), \(K_{2}(\varepsilon )\) satisfy

$$\begin{aligned}& \lim_{\varepsilon \rightarrow 0}\varepsilon^{-\frac{p-1}{p}}K_{1}( \varepsilon ) =\frac{1}{2}H(0)C_{Np}^{p} \biggl( \frac{N-p}{p-1} \biggr)^{p} \int_{R^{N-1}} \bigl(1+ \bigl\vert x^{\prime } \bigr\vert ^{\frac{p}{p-1}} \bigr)^{-N} \bigl\vert x^{\prime } \bigr\vert ^{ \frac{3p-2}{p-1}}\,\mathrm{d}x^{\prime }=K_{1}, \\& \lim_{\varepsilon \rightarrow 0}\varepsilon^{-\frac{p-1}{p}}K_{2}( \varepsilon ) =\frac{1}{2}H(0)C_{Np}^{p^{*}} \int_{R^{N-1}} \bigl(1+ \bigl\vert x^{\prime } \bigr\vert ^{\frac{p}{p-1}} \bigr)^{-N} \bigl\vert x^{\prime } \bigr\vert ^{2}\,\mathrm{d}x^{\prime } =K_{2}. \end{aligned}$$

(2)

$$\begin{aligned}& \int_{\Omega }\vert u_{\varepsilon }\vert ^{p} \, \mathrm{d}x= \textstyle\begin{cases} O(\varepsilon^{\frac{N-p}{p}}), & N< p^{2}, \\ O(\varepsilon^{\frac{N-p}{p}}\vert \ln \varepsilon \vert ), & N=p^{2}, \\ O(\varepsilon^{p-1}), & N>p^{2}. \end{cases}\displaystyle \end{aligned}$$

(3)

$$\begin{aligned}& \int_{\Omega }\vert u_{\varepsilon }\vert ^{q} \, \mathrm{d}x= \textstyle\begin{cases} O(\varepsilon^{\frac{q(N-p)}{p^{2}}}), & q< \frac{N(p-1)}{N-p}, \\ O(\varepsilon^{\frac{q(N-p)}{p^{2}}}\vert \ln \varepsilon \vert ), & q= \frac{N(p-1)}{N-p}, \\ O(\varepsilon^{\frac{(p-1)(Np-q(N-p))}{p^{2}}}), & q> \frac{N(p-1)}{N-p}. \end{cases}\displaystyle \end{aligned}$$

3 Proof of main results

Let \(\varphi (x)\in C_{0}^{\infty }(R^{N})\) be a smooth cut off function such that

$$\begin{aligned}& 0\leq \varphi (x)\leq 1,\quad \frac{\delta }{2}\leq \vert x\vert \leq \delta ; \\& \varphi (x)=1, \quad \vert x\vert < \frac{\delta }{2}; \\& \varphi (x)=0,\quad \vert x\vert >\delta . \end{aligned}$$

Define \(\omega_{\varepsilon }=\varphi u_{\varepsilon }\), then we have the following lemma about \(\omega_{\varepsilon }\).

Lemma 3.1

Suppose \(N\geq 2p\), \(0\in \partial \Omega \). If the function \(V(x)\) satisfies \(\int_{\Omega \cap B(0,\delta )}V^{r^{\prime }}\,\mathrm{d}x<\infty \), then

$$\begin{aligned}& \int_{\Omega \cap B(0,\delta )}V\omega_{\varepsilon }^{p}\,\mathrm{d}x=O \bigl( \varepsilon^{\frac{N-p}{p}+p-N+\frac{N(p-1)}{pr}} \bigr), \end{aligned}$$

where \(\frac{1}{r}+\frac{1}{r^{\prime }}=1\), \(1< r<\frac{N(p-1)}{Np+2p-N-p ^{2}-1}\).

Proof

According to the Hölder inequality and the definition of \(\omega_{\varepsilon }\), we have

$$\begin{aligned} \int_{\Omega \cap B(0,\delta )}V\omega_{\varepsilon }^{p}\,\mathrm{d}x \leq & \biggl( \int_{\Omega \cap B(0,\delta )}V^{r^{\prime }}\,\mathrm{d}x \biggr)^{\frac{1}{r^{\prime }}} \biggl( \int_{\Omega \cap B(0,\delta )} \omega_{\varepsilon }^{pr}\,\mathrm{d}x \biggr)^{\frac{1}{r}} \\ \leq & \varepsilon^{\frac{(N-p)}{p}}C_{Np}^{p} \biggl( \int_{\Omega \cap B(0,\delta )}V^{r^{\prime }}\,\mathrm{d}x \biggr)^{\frac{1}{r ^{\prime }}} \biggl( \int_{ B(0,\delta )} \bigl(\varepsilon +\vert x\vert ^{\frac{p}{p-1}} \bigr)^{r(p-N)} \,\mathrm{d}x \biggr)^{\frac{1}{r}} \\ =&\varepsilon^{{\frac{N-p}{p}+p-N+\frac{N(p-1)}{pr}}}C_{Np}^{p} \biggl( \int_{\Omega \cap B(0,\delta )} V^{r^{\prime }}\,\mathrm{d}x \biggr)^{\frac{1}{r^{\prime }}} \\ &{}\times \biggl( \int_{ B(0,\delta \varepsilon^{-\frac{p-1}{p}})} \bigl(1+\vert x\vert ^{ \frac{p}{p-1}} \bigr)^{r(p-N)}\,\mathrm{d}x \biggr)^{\frac{1}{r}} . \end{aligned}$$

Noting that \(\frac{N(p-1)}{(N-p)p}\leq 1< r\), a series of computations yield

$$\begin{aligned} & \int_{\Omega \cap B(0,\delta )}V\omega_{\varepsilon }^{p}\,\mathrm{d}x = O \bigl(\varepsilon^{\frac{N-p}{p}+p-N+\frac{N(p-1)}{pr}} \bigr). \end{aligned}$$

 □

Lemma 3.2

Suppose that (A1), (A2) hold and \(0\in \partial \Omega \), \(H(0)>0\), \(Q _{m}=Q(0)\). If the functions \(Q(x)\), \(V(x)\) satisfy the conditions (A3), (A4), then there exists a nonnegative function \(v\in W_{\lambda ,V}^{1,p}(\Omega )\), \(v\not \equiv 0\), such that

$$\begin{aligned} \sup_{t\geq 0}J_{\lambda }(tv)< c^{*} \end{aligned}$$
(3.1)

for each \(\lambda >0\), \(N\geq 2p\).

Proof

We divide the proof into three steps.

(i) We consider the functional

$$\begin{aligned} g(t) =&J_{\lambda }(t\omega_{\varepsilon }) \\ =&\frac{t^{p}}{p} \int_{\Omega } \bigl(\vert \nabla \omega_{\varepsilon }\vert ^{p}+ \lambda V(x)\vert \omega_{\varepsilon }\vert ^{p} \bigr)\,\mathrm{d}x-\frac{t^{p*}}{p ^{*}}\lambda \int_{\Omega }Q(x)\vert \omega_{\varepsilon }\vert ^{p^{*}} \,\mathrm{d}x \\ &{}-\frac{t^{q}}{q}\lambda \int_{\Omega }P(x)\vert \omega_{\varepsilon }\vert ^{q}\,\mathrm{d}x, \quad t>0. \end{aligned}$$

Noting that \(\lim_{t\rightarrow \infty }g(t)=-\infty \), \(g(0)=0\), \(g(t)>0\) for \(t\rightarrow 0^{+}\), we know that there exists a \(t_{\varepsilon }>0\) such that \(\sup_{t>0}g(t)\) is attained for \(t_{\varepsilon }\) and \(t_{\varepsilon }\) is uniformly bounded for \(\varepsilon >0\) sufficiently small. Thus,

$$\begin{aligned} g(t_{\varepsilon }) =&\sup_{t\geq 0} J_{\lambda }(t\omega_{\varepsilon }) \\ \leq &\sup_{t\geq 0} \biggl[\frac{t^{p}}{p} \int_{\Omega }\vert \nabla\omega_{\varepsilon }\vert ^{p}\,\mathrm{d}x-\frac{t^{p*}}{p^{*}}\lambda \int_{\Omega }Q(x)\vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x \biggr] \\ &{}+\frac{t_{\varepsilon }^{p}}{p} \int_{\Omega }\lambda V(x)\vert \omega_{\varepsilon }\vert ^{p}\,\mathrm{d}x-\frac{t_{\varepsilon }^{q}}{q} \lambda \int_{\Omega }P(x)\vert \omega_{\varepsilon }\vert ^{q}\,\mathrm{d}x \\ =&\frac{1}{N} \biggl[\frac{\int_{\Omega }\vert \nabla \omega_{\varepsilon }\vert ^{p} \,\mathrm{d}x}{(\lambda \int_{\Omega }Q(x)\vert \omega_{\varepsilon }\vert ^{p ^{*}}\,\mathrm{d}x)^{\frac{N-p}{N}}} \biggr]^{\frac{N}{p}}+ \frac{t_{\varepsilon }^{p}}{p} \int_{\Omega }\lambda V(x)\vert \omega_{\varepsilon }\vert ^{p} \,\mathrm{d}x \\ &{}-\frac{t_{\varepsilon }^{q}}{q}\lambda \int_{\Omega }P(x)\vert \omega_{\varepsilon }\vert ^{q}\,\mathrm{d}x. \end{aligned}$$
(3.2)

(ii) When \(\varepsilon >0\) is sufficiently small, we have

$$\begin{aligned}& \begin{aligned} & \int_{\Omega }Q(x)\vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x =Q_{m} \int_{\Omega }\vert u_{\varepsilon }\vert ^{p^{*}} \, \mathrm{d}x+o \bigl( \varepsilon^{\frac{p-1}{p}} \bigr), \\ & \int_{\Omega }\vert \nabla \omega_{\varepsilon }\vert ^{p}\,\mathrm{d}x\leq \int_{\Omega }\vert \nabla u_{\varepsilon }\vert ^{p}\,\mathrm{d}x+o \bigl(\varepsilon^{ \frac{p-1}{p}} \bigr), \\ &\int_{\Omega }\vert \omega_{\varepsilon }\vert ^{q} \,\mathrm{d}x= \int_{\Omega }\vert u_{\varepsilon }\vert ^{q} \, \mathrm{d}x+o \bigl(\varepsilon^{\frac{p-1}{p}} \bigr), \\ &\int_{\Omega }\vert \omega_{\varepsilon }\vert ^{p^{*}} \,\mathrm{d}x= \int_{ \Omega }\vert u_{\varepsilon }\vert ^{p^{*}} \, \mathrm{d}x+o \bigl( \varepsilon^{\frac{p-1}{p}} \bigr). \end{aligned} \end{aligned}$$
(3.3)

We firstly prove the first formula. Since \(\vert Q(x)-Q(0)\vert =o(\vert x\vert ^{\alpha })\) for \(x\rightarrow 0\), there exists a \(0<\delta_{0}\leq \delta \) such that \(\vert Q(x)-Q(0)\vert \leq C\vert x\vert ^{\alpha }\) for \(\vert x\vert <\delta_{0}\), where \(C>0\) is constant. Moreover

$$\begin{aligned}& \int_{\Omega } \bigl\vert Q(x)-Q(0) \bigr\vert \vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x \\& \quad \leq \int_{\Omega \cap \vert x\vert \leq \delta_{0}} \bigl\vert Q(x)-Q(0) \bigr\vert \vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x+ \int_{\Omega \cap \vert x\vert \geq \delta_{0}} \bigl\vert Q(x)-Q(0) \bigr\vert \vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x \\& \quad \leq C \int_{\vert x\vert \leq \delta_{0}} \vert x\vert ^{\alpha }\vert \omega_{\varepsilon} \vert ^{p^{*}}\,\mathrm{d}x +2Q_{M} \int_{\Omega \cap \vert x\vert \geq \delta_{0}} \vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x \\& \quad \leq C C_{Np}^{p^{*}}\varepsilon^{\frac{(p-1)\alpha }{p}} \int_{\vert x\vert \leq \frac{\delta_{0}}{\varepsilon^{\frac{p-1}{p}}}}\vert x\vert ^{ \alpha } \bigl(1+\vert x \vert ^{\frac{p}{p-1}} \bigr)^{-N}\,\mathrm{d}x \\& \qquad {}+2Q_{M}C_{Np}^{p^{*}}\varepsilon^{\frac{N}{p}} \int_{\Omega \cap \vert x\vert \geq \delta_{0}} \bigl(\varepsilon +\vert x\vert ^{\frac{p}{p-1}} \bigr)^{-N} \,\mathrm{d}x \\& \quad =O \bigl(\varepsilon^{\frac{(p-1)\alpha }{p}} \bigr)+O \bigl(\varepsilon^{\frac{N}{p}} \bigr). \end{aligned}$$

Since \(N\geq 2p\), \(1<\alpha <\frac{N}{p-1}\), \(\int_{\Omega }\vert Q(x)-Q(0)\vert \vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x=o(\varepsilon^{ \frac{p-1}{p}})\), which implies

$$\begin{aligned}& \int_{\Omega }Q(x)\vert \omega_{\varepsilon }\vert ^{p^{*}}\,\mathrm{d}x \\& \quad =Q_{m} \int_{\Omega }\vert \omega_{\varepsilon }\vert ^{p^{*}} \,\mathrm{d}x+ \int_{\Omega } \bigl(Q(x)-Q(0) \bigr)\vert \omega_{\varepsilon } \vert ^{p^{*}}\,\mathrm{d}x \\& \quad =Q_{m} \int_{\Omega }\vert \omega_{\varepsilon }\vert ^{p^{*}} \,\mathrm{d}x+o \bigl( \varepsilon^{\frac{p-1}{p}} \bigr). \end{aligned}$$

Similarly, we can evaluate the rest of formulas and omit the details here.

(iii) \(\sup_{t\geq 0}J_{\lambda }(t\omega_{\varepsilon })< c^{*}\).

Combining (3.3) with Lemma 2.3, one has

$$\begin{aligned}& \int_{\Omega }\vert \nabla \omega_{\varepsilon }\vert ^{p}\,\mathrm{d}x \le M _{1}\bigl(1-M_{1}^{-1}K_{1}\bigl( \varepsilon )+o \bigl(\varepsilon^{\frac{p-1}{p}}\bigr) \bigr), \\& \int_{\Omega }\vert \omega_{\varepsilon }\vert ^{p^{*}} \,\mathrm{d}x = M_{2}\bigl(1-M _{2}^{-1}K_{2}\bigl( \varepsilon )+o \bigl(\varepsilon^{\frac{p-1}{p}}\bigr)\bigr), \end{aligned}$$

where \(M_{1}=\frac{1}{2}\int_{R^{N}}\vert \nabla u_{\varepsilon }\vert ^{p} \,\mathrm{d}x\), \(M_{2}=\frac{1}{2}\int_{R^{N}}\vert u_{\varepsilon }\vert ^{p^{*}} \,\mathrm{d}x\). Then, using (3.2), (3.3), Lemma 2.3 and Lemma 3.1, we see that

$$\begin{aligned} \sup_{t\geq 0}J_{\lambda }(t\omega_{\varepsilon }) \leq & \frac{S^{ \frac{N}{p}}}{2N(\lambda Q_{m})^{\frac{N-p}{p}}} \biggl[1+\frac{N-p}{p}M _{2}^{-1}K_{2}( \varepsilon ) -\frac{N}{p}M_{1}^{-1}K_{1}( \varepsilon )+o \bigl(\varepsilon^{\frac{p-1}{p}} \bigr) \biggr] \\ &{}+O \bigl( \varepsilon^{\frac{N-p}{p}+p-N+\frac{(p-1)N}{pr}} \bigr). \end{aligned}$$

Next, we claim that

$$ \begin{aligned} \lim_{\varepsilon \rightarrow 0} \varepsilon^{\frac{p-1}{p}} \biggl[ \frac{N-p}{p}M_{2}^{-1}K_{2}( \varepsilon ) -\frac{N}{p}M_{1}^{-1}K _{1}( \varepsilon ) \biggr]< 0 \end{aligned} $$
(3.4)

for \(\varepsilon > 0\) small enough, which implies (3.1) holds. According to \(\lim_{\varepsilon \rightarrow 0} \varepsilon^{\frac{p-1}{p}}K_{1}(\varepsilon ) =K_{1}\), \(\lim_{\varepsilon \rightarrow 0}\varepsilon^{\frac{p-1}{p}}K_{2}( \varepsilon ) =K_{2}\), we know that (3.4) is equivalent to \(\frac{K_{1}}{K_{2}} >\frac{N-p}{N}\frac{M_{1}}{M_{2}}\).

From the expressions of \(K_{1}\), \(K_{2}\), \(M_{1}\), \(M_{2}\) and \(u_{\varepsilon }\), a series of computations yield

$$\begin{aligned}& \frac{K_{1}}{K_{2}} = \frac{\frac{1}{2}H(0)C_{Np}^{p}(\frac{N-p}{p-1})^{p} \int_{R^{N-1}} (1+\vert x^{\prime }\vert ^{\frac{p}{p-1}})^{-N}\vert x^{\prime }\vert ^{ \frac{3p-2}{p-1}}\,\mathrm{d}x^{\prime }}{\frac{1}{2}H(0)C_{Np}^{p^{*}} \int_{R^{N-1}} (1+\vert x^{\prime }\vert ^{\frac{p}{p-1}})^{-N}\vert x^{\prime }\vert ^{2} \,\mathrm{d}x^{\prime }} \\& \hphantom{\frac{K_{1}}{K_{2}}}=C_{Np}^{p-p^{*}} \biggl(\frac{N-p}{p-1} \biggr)^{p} \frac{\int_{0}^{\infty }(1+r ^{2})^{-N}r^{\frac{2Np+3p-2N-2}{p}}\,\mathrm{d}r}{\int_{0}^{\infty }(1+r ^{2})^{-N}r^{\frac{2Np+p-2N-2}{p}}\,\mathrm{d}r}, \\& \frac{N-p}{N}\frac{M_{1}}{M_{2}} =\frac{N-p}{N}\frac{\int_{R^{N}}\vert \nabla u_{\varepsilon }\vert ^{p}\,\mathrm{d}x}{\int_{R^{N}}\vert u_{\varepsilon} \vert ^{p^{*}}\,\mathrm{d}x} \\& \hphantom{\frac{N-p}{N}\frac{M_{1}}{M_{2}}}=\frac{N-p}{N}C_{Np}^{p-p^{*}} \biggl(\frac{N-p}{p-1} \biggr)^{p}\frac{\int_{0} ^{\infty }(1+r^{2})^{-N}r^{\frac{2Np+p-2N}{p}}\,\mathrm{d}r}{\int_{0} ^{\infty }(1+r^{2})^{-N}r^{\frac{2Np-p-2N}{p}}\,\mathrm{d}r}. \end{aligned}$$

Integrating by parts, we have

$$\int_{0}^{\infty }\frac{r^{\beta }}{(1+r^{2})^{n}}\,\mathrm{d}r= \frac{ \beta -1}{2n-\beta -1} \int_{0}^{\infty } \frac{r^{\beta -2}}{(1+r^{2})^{n}}\,\mathrm{d}r\quad \mbox{for }2\leq \beta < 2n-1 . $$

Then

$$\begin{aligned}& \frac{K_{1}}{K_{2}} =C_{Np}^{p-p^{*}} \biggl( \frac{N-p}{p-1} \biggr)^{p} \frac{(p-1)(N+1)}{N-2p+1}, \\& \frac{N-p}{N}\frac{M_{1}}{M_{2}}=C_{Np}^{p-p^{*}} \biggl( \frac{N-p}{p-1} \biggr)^{p}(p-1). \end{aligned}$$

This implies \(\frac{K_{1}}{K_{2}} >\frac{N-p}{N}\frac{M_{1}}{M_{2}}\). Thus

$$ \begin{aligned} \sup_{t\geq 0}J_{\lambda }(t \omega_{\varepsilon })< \frac{S^{ \frac{N}{p}}}{2N(\lambda Q_{m})^{\frac{N-p}{p}}}=c^{*}. \end{aligned} $$

The proof of Lemma 3.2 is complete. □

Proof of Theorem 1.1

Applying Lemma 2.1 and Lemma 3.2, we obtain

$$c=\inf_{h\in \Gamma }\max _{t\in [0,1]}J_{\lambda } \bigl(h(t) \bigr) \leq \sup _{t\geq 0}J_{\lambda }(t \omega_{\varepsilon })< c^{*}. $$

From Lemma 2.2 and the mountain pass theorem, we know that there exists at least one nontrivial solution to Problem (1.5). Since \(J_{\lambda }(u)\geq J_{\lambda }(\vert u\vert )\), Problem (1.5) has at least one nonnegative nontrivial solution. The proof of Theorem 1.1 is complete. □

Proof of Theorem 1.2

Consider the following function:

$$\begin{aligned} h(t) =&J_{\lambda }(t u ) \\ =&\frac{t^{p}}{p} \int_{\Omega } \bigl(\vert \nabla u\vert ^{p}+\lambda V(x)\vert u\vert ^{p} \bigr) \,\mathrm{d}x-\frac{t^{p*}}{p^{*}} \lambda \int_{\Omega }Q(x)\vert u\vert ^{p^{*}} \,\mathrm{d}x \\ &{}- \frac{t^{q}}{q}\lambda \int_{\Omega }P(x)\vert u\vert ^{q} \,\mathrm{d}x,\quad t>0. \end{aligned}$$

Since \(V\in L^{1}(\Omega )\), we find that

$$\begin{aligned} \sup_{t\geq 0}h(t) =&\sup_{t\geq 0} \biggl[ \frac{t^{p}}{p} \int_{\Omega }\lambda V(x)\vert A\vert ^{p} \, \mathrm{d}x-\frac{t^{p*}}{p^{*}}\lambda \int_{ \Omega }Q(x)\vert A\vert ^{p^{*}}\,\mathrm{d}x- \frac{t^{q}}{q}\lambda \int_{ \Omega }P(x)\vert A\vert ^{q}\,\mathrm{d}x \biggr] \\ \leq &\frac{\lambda }{N} \biggl[\frac{\int_{\Omega }V(x)\,\mathrm{d}x}{( \int_{\Omega }Q(x)\,\mathrm{d}x)^{\frac{N-p}{N}}} \biggr]^{\frac{N}{p}} \quad \mbox{for } u=A. \end{aligned}$$

Then \(\sup_{t\geq 0}J_{\lambda }(tA)< c^{*}\) for \(\lambda <\frac{S( \int_{\Omega }Q(x)\,\mathrm{d}x)^{\frac{N-p}{N}}}{Q_{M}^{\frac{N-p}{N}} \int_{\Omega }V(x)\,\mathrm{d}x}\).

Similarly,

$$\begin{aligned} J_{\lambda }(tA) =&\sup_{t\geq 0}h(t) \\ =&\sup_{t\geq 0} \biggl[ \frac{t^{p}}{p} \int_{\Omega }\lambda V(x)\vert A\vert ^{p} \, \mathrm{d}x-\frac{t ^{p*}}{p^{*}}\lambda \int_{\Omega }Q(x)\vert A\vert ^{p^{*}}\,\mathrm{d}x - \frac{t ^{q}}{q}\lambda \int_{\Omega }P(x)\vert A\vert ^{q}\,\mathrm{d}x \biggr] \\ \leq& \lambda \biggl(\frac{q-p}{pq} \biggr)\frac{(\int_{\Omega }V(x)\,\mathrm{d}x)^{ \frac{q}{q-p}}}{(\int_{\Omega }P(x)\,\mathrm{d}x)^{\frac{p}{q-p}}}< c ^{*} \end{aligned}$$

for \(\lambda <(\frac{pq}{q-p})^{\frac{p}{N}}\frac{S}{N^{\frac{p}{N}}Q _{M}^{\frac{N-p}{N}}}\frac{(\int_{\Omega }P(x)\,\mathrm{d}x)^{\frac{p ^{2}}{N(q-p)}}}{(\int_{\Omega }V(x)\,\mathrm{d}x)^{\frac{pq}{N(q-p)}}}\).

Set

$$\lambda_{*}=\max \biggl\{ \frac{S(\int_{\Omega }Q(x)\,\mathrm{d}x)^{ \frac{N-p}{N}}}{Q_{M}^{\frac{N-p}{N}}\int_{\Omega }V(x)\,\mathrm{d}x}, \biggl( \frac{pq}{q-p} \biggr)^{\frac{p}{N}}\frac{S}{N^{\frac{p}{N}}Q_{M}^{ \frac{N-p}{N}}} \frac{(\int_{\Omega }P(x)\,\mathrm{d}x)^{ \frac{p^{2}}{N(q-p)}}}{(\int_{\Omega }V(x)\,\mathrm{d}x)^{ \frac{pq}{N(q-p)}}} \biggr\} , $$

then we have \(\sup_{t\geq 0}J_{\lambda }(tA)< c^{*}\) for \(0<\lambda <\lambda_{*}\). Similar to the proof of Theorem 1.1, Problem (1.5) has at least one nonnegative nontrivial solution. The proof of Theorem 1.2 is complete. □

Proof of Theorem 1.3

Define

$$ \begin{aligned} K=\inf_{u \in W_{0}^{1,p}(B(x_{0} ,\delta ))\setminus \{0\}}\frac{ \int_{B(x_{0},\delta )}\vert \nabla u\vert ^{p}\,\mathrm{d}x}{( \int_{B( x_{0},\delta )}\vert u\vert ^{q}\,\mathrm{d}x)^{\frac{p}{q}}}. \end{aligned} $$

Since \(p< q< p^{*}\), as is well known, there exists a function \(w\in W _{0}^{1,p}(B(x_{0} ,\delta ))\) such that

$$ \begin{aligned} K=\frac{\int_{B(x_{0},\delta )}\vert \nabla w\vert ^{p}\,\mathrm{d}x}{( \int_{B( x_{0},\delta )}\vert w\vert ^{q}\,\mathrm{d}x)^{\frac{p}{q}}}. \end{aligned} $$

Thus,

$$\begin{aligned} \sup_{t\geq 0}J_{\lambda }(t w) \leq &\sup _{t\geq 0} \biggl[\frac{t^{p}}{p} \int_{B(x_{0} ,\delta )} \bigl(\vert \nabla w\vert ^{p}+\lambda V(x)\vert w\vert ^{p} \bigr) \,\mathrm{d}x-\frac{t^{q}}{q} \lambda \int_{B(x_{0} ,\delta )}P(x)\vert w\vert ^{q} \,\mathrm{d}x \biggr] \\ \leq &\frac{q-p}{pq}\frac{(\int_{B(x_{0} ,\delta )}\vert \nabla w\vert ^{p} \,\mathrm{d}x)^{\frac{q}{q-p}}}{P_{m}^{\frac{p}{q-p}}( \int_{B(x_{0} ,\delta )}\lambda \vert w\vert ^{q}\,\mathrm{d}x)^{\frac{p}{q-p}}} \\ =&\frac{q-p}{pq}\frac{K^{\frac{q}{q-p}}}{\lambda^{\frac{p}{q-p}}P _{m}^{\frac{p}{q-p}}}. \end{aligned}$$

Let \(\lambda^{*}= (\frac{N(q-p)K^{\frac{q}{q-p}}Q_{M}^{ \frac{N-p}{p}}}{pqS^{\frac{N}{p}}P_{m}^{\frac{p}{q-p}}} )^{ \frac{p(q-p)}{Np+pq-Nq}}\), where \(P_{m}=\min_{x\in B(x_{0} ,\delta )}P(x)\), then \(\sup_{t\geq 0}J _{\lambda }(tw)< c^{*}\) for \(\lambda >\lambda^{*}\). Similar to the proof of Theorem 1.1, Problem (1.5) has at least one nonnegative nontrivial solution for \(\lambda >\lambda^{*}\). The proof of Theorem 1.3 is complete. □

Proof of Theorem 1.4

Fix \(n\in N\), let \(\varphi_{1}\), \(\varphi_{2},\ldots ,\varphi_{n}\in C_{0}^{\infty }(R^{N})\) be smooth functions such that \(\operatorname{supp}\varphi_{j}\subset B(x_{0},\delta )\), \(j=1, 2, \ldots , n\), \(\operatorname{supp}\varphi_{i}\cap \operatorname{supp}\varphi_{j}= \varnothing \), \(i\neq j\).

We define \(E_{n}=\operatorname{Span}\{\varphi_{1}, \varphi_{2}, \ldots , \varphi _{n}\}\), Σ is the set of all symmetric and closed subsets of \(W _{V}^{1,p}(\Omega )\), \(\gamma (A)\) is the Krasnoselski genus,

$$i(A)=\min_{h\in \Gamma }\gamma \bigl(h(A) \bigr) \cap \partial B_{ \beta_{\lambda }} ),\quad A\in \Sigma , $$

where Γ is the set of all odd homomorphisms \(C^{1}(W_{V}^{1,p}( \Omega ), W_{V}^{1,p}(\Omega ))\).

Set

$$ \begin{aligned} c_{j}=\inf_{i(A)\geq j}\sup _{u\in A}J_{\lambda }(u),\quad j=1, 2, \ldots , n. \end{aligned} $$

Since \(i(E_{n})=\dim E_{n}=n\) and \(J_{\lambda }(u)\geq \beta_{\lambda }\) for \(\Vert u\Vert _{\lambda V}=\rho_{\lambda }\) in Lemma 2.1, we find that

$$ \begin{aligned} \beta_{\lambda }\leq c_{1}\leq c_{2}\leq \cdots \leq c_{n}\leq \sup_{u\in E_{n}}J_{\lambda }(u). \end{aligned} $$

We now estimate \(\sup_{u\in E_{n}}J_{\lambda }(u)\). If \(u\in E_{n}\), one has \(u=\sum_{j=1}^{n}\tau_{j}\varphi_{j}\) for \(\tau_{j}\in R\). From the properties of \(\varphi_{j}\), we obtain

$$\begin{aligned} \sup_{u\in E_{n}}J_{\lambda }(u) =&\sup_{u\in E_{n}} \Biggl(\sum_{j=1} ^{n}J_{\lambda }( \tau_{j}\varphi_{j}) \Biggr) \\ \leq &\sup_{u\in E_{n}}\sum_{j=1}^{n} \biggl[\frac{\tau_{j}^{p}}{p} \int_{B(x_{0} ,\delta )} \bigl(\vert \nabla \varphi_{j}\vert ^{p}+\lambda V(x)\vert \varphi_{j}\vert ^{p} \bigr)\,\mathrm{d}x-\frac{\tau_{j}^{q}}{q}\lambda \int_{B(x_{0} ,\delta )}P(x)\vert \varphi_{j}\vert ^{q}\,\mathrm{d}x \biggr] \\ \leq &\frac{q-p}{pq}\sum_{j=1}^{n} \frac{(\int_{B(x_{0} ,\delta )}\vert \nabla \varphi_{j}\vert ^{p}\,\mathrm{d}x)^{\frac{q}{q-p}}}{ \lambda^{\frac{p}{q-p}}P_{m}^{\frac{p}{q-p}}(\int_{B(x_{0} ,\delta )}\vert \varphi_{j}\vert ^{q}\,\mathrm{d}x)^{\frac{p}{q-p}}}. \end{aligned}$$

Consequently, there exists a \(\Lambda_{n}>0\) such that \(\sup_{u\in E_{n}}J_{\lambda }(u)< c^{*}\) for \(\lambda >\Lambda_{n}\). Similar to the proof of Theorem 1.1, Problem (1.5) has at least n pairs of nonnegative nontrivial solutions. The proof of Theorem 1.4 is complete. □

4 Conclusion

In this paper, we study the following quasilinear Neumann problem with critical Sobolev exponent:

$$ \textstyle\begin{cases} -\varepsilon^{p}\Delta_{p}u+V(x)\vert u\vert ^{p-2}u=Q(x)\vert u\vert ^{p^{*}-2}u +P(x)\vert u\vert ^{ {q-2}}u, & x\in \Omega , \\ \vert \nabla u\vert ^{p-2}\frac{\partial u}{\partial \nu }=0, &x\in \partial \Omega , \end{cases} $$

where the weight functions \(V(x)\) is continuous in Ω and \(Q(x)\), \(P(x)\) are continuous on Ω̅. Due to the lack of compactness of the embedding of \(W^{1,p}(\Omega )\hookrightarrow L ^{p^{*}}(\Omega )\) and the fact that the weight function \(V(x)\) may be unbounded close to the boundary ∂Ω, some classical methods may not directly be applied to our problem. We introduce a suitable weighted Sobolev space and add restrictions on the weight functions \(Q(x)\) and \(P(x)\) to prove the corresponding functional of problem satisfies \((PS)_{c}\)-condition in a suitable range by the Lions concentration-compactness principle, then apply the mountain pass lemma, the existence and multiplicity of nontrivial solutions are obtained.