1 Introduction and main result

In this paper, we consider the existence and multiplicity of solutions for the following elliptic problem:

$$ (-\Delta)_{p}^{s}u+V(x) \vert u \vert ^{p-2}u=h_{1}(x) \vert u \vert ^{q-2}u+h_{2}(x) \vert u \vert ^{r-2}u\quad \mbox{in } \mathbb{R}^{N}, $$
(1.1)

where \((-\Delta)_{p}^{s}\) is the fractional p-Laplacian operator with \(0< s<1<p\) and \(sp< N\), \(1< q< p< r< p_{s}^{*}:=\frac{Np}{N-sp}\) and potential function \(V(x)>0\), \(h_{1}\) and \(h_{2}\) are sign-changing weight functions. The exact assumptions will be given below.

The fractional p-Laplacian operator \((-\Delta)_{p}^{s}\) is defined along a function \(u\in C_{0}^{\infty}(\mathbb{R}^{N})\) as follows:

$$ (-\Delta)_{p}^{s}u(x)=2\lim _{\varepsilon\rightarrow0^{+}} \int_{ \mathbb{R}^{N}\setminus B_{\varepsilon}(x)} \frac{ \vert u(x)-u(y) \vert ^{p-2}(u(x)-u(y))}{ \vert x-y \vert ^{N+ps}}\,dy $$
(1.2)

for \(x\in\mathbb{R}^{N}\), where \(B_{\varepsilon}(x)=\{y\in\mathbb{R}^{N}:|x-y|<\varepsilon\}\), see [1–3] and the references therein.

In the last years, since the nonlinear equations involving fractional powers of the Laplacian played an increasingly important role in physics, probability, and finance, a great attention has been focused on the study of problems involving the fractional Laplacian. So there has been a lot of interest in the study of the fractional Schrödinger equation

$$ (-\Delta)^{s} u+V(x)u=f(x,u),\quad x\in \mathbb{R}^{N}, $$
(1.3)

where the nonlinearity f satisfies some general conditions, see for instance [4–19] and the references therein.

More recently, Xiang et al. [3] investigated the fractional p-Laplacian equation

$$ (-\Delta)_{p}^{s}u+V(x) \vert u \vert ^{p-2}u=\lambda a(x) \vert u \vert ^{r-2}u-b(x) \vert u \vert ^{q-2}u \quad \mbox{in } \mathbb{R}^{N}, $$
(1.4)

where \(\lambda>0\), \(p< r<\min\{q,p_{s}*\}\), and \(a(x)\) and \(b(x)\) are related by the condition \(a(a/b)^{(r-p)(q-r)}\in L^{N/ps}(\mathbb{R}^{N})\). By using a direct variational method and the mountain pass theorem, the authors proved the existence of two nontrivial weak solutions of (1.4) for \(\lambda>\lambda^{*}\) (\(\lambda^{*}>0\) is a given constant).

There are also a lot of works about problem (1.3) with concave-convex nonlinearities. Goyal and Sreenadh in [20] considered the following p-fractional Laplace equation:

$$ \textstyle\begin{cases} (-\Delta)_{p}^{s}u= \lambda h(x) \vert u \vert ^{q-2}u+b(x) \vert u \vert ^{r-2}u \quad \mbox{in } \varOmega, \\ u=0\quad \mbox{on } \partial\varOmega, \end{cases} $$
(1.5)

where Ω is a bounded domain in \(\mathbb{R}^{N}\) with Lipschitz boundary, \(p\geq2\), \(n>p\alpha\), \(1< q< p< r< p_{s}^{*}\), \(\lambda>0\), h and b are sign-changing smooth functions. They proved that there exists \(\lambda_{0}>0\) such that problem (1.5) has at least two nonnegative solutions for all \(\lambda\in(0,\lambda_{0})\).

In [21], the authors considered the problem as follows:

$$ M\bigl([u]_{s,p}^{p}\bigr) (-\Delta )_{p}^{s}u=\lambda h_{1}(x) \vert u \vert ^{q-2}u+h_{2}(x) \vert u \vert ^{r-2}u+h(x) \quad \mbox{in } \mathbb{R}^{N}, $$
(1.6)

where \(M(t)=a+bt^{\theta-1}\), \(\theta>1\), \(a,b\geq0\), \(a+b>0\), \(\lambda >0\), and \(1< q< p<\theta p<r<p_{s}^{*}\). The functions \(h_{1}(x)\), \(h_{2}(x)\), and \(h(x)\) may change sign on \(\mathbb{R}^{N}\). Note that problem (1.6) is reduced to the fractional p-Laplacian equation with \(a=1\) and \(b=0\). Under some suitable conditions, they obtained the existence of two nontrivial entire solutions by applying the mountain pass theorem and Ekeland’s variational principle.

In this paper, we are interested in the multiplicity of solutions to equation (1.1) and find sufficient conditions to guarantee the existence of infinitely many solutions.

The present article is motivated by the papers [6, 7], as well as by the fact that we do not find in the literature any paper dealing with the existence of infinitely many solutions to equation (1.1). The main tools employed in our works are the variant fountain theorems established in [22]. They are effective tools for studying the existence of infinitely many large or small energy solutions. Moreover, the results about the existence of solutions in the above papers are all related to the number λ. So we are also interested in whether the restriction on λ can be taken out.

Throughout this paper, we make the following assumptions:

\((H_{1})\):

\(V(x)\in C(\mathbb{R}^{N})\) and there exists \(V_{0}>0\) such that \(V(x)\geq V_{0}\) in \(\mathbb{R}^{N}\);

\((H_{2})\):

\(h_{1}(x)\in L^{\tau}(\mathbb{R}^{N})\), where \(\tau\in[\frac{p_{s}^{*}}{p_{s}^{*}-q},\frac{p}{p-q}]\) is a positive constant;

\((H_{3})\):

There exists \(\mu\in[\frac{p_{s}^{*}}{p_{s}^{*}-r},\infty)\) such that \(h_{2}(x)\in L^{\mu}(\mathbb{R}^{N})\);

\((H_{3}')\):

\(h_{2}(x)\in L^{\infty}(\mathbb{R}^{N})\) and \(h_{2}(x)\rightarrow0\) as \(|x|\rightarrow\infty\).

Our main results in this paper are stated as follows.

Theorem 1.1

Assume\((H_{1})\), \((H_{2})\), \((H_{3})\) (or\((H_{3}')\)), and\(h_{1}(x)>0\), then problem (1.1) admits infinitely many small energy solutions\(u_{k}\in E\)satisfying\(I(u_{k})\rightarrow0^{-}\)as\(k\rightarrow\infty\).

Theorem 1.2

Let\((H_{1})\), \((H_{2})\), and\((H_{3})\) (or\((H_{3}')\)) hold. If\(h_{2}(x)>0\)is satisfied, then problem (1.1) possesses infinitely many high energy solutions\(u_{k}\in E\)such that\(I(u_{k})\rightarrow\infty\)as\(k\rightarrow\infty\).

The functional I that appeared in Theorems 1.1–1.2 is the energy functional for problem (1.1), which will be given below.

By combining Theorems 1.1–1.2, the following corollary is immediate.

Corollary 1.3

Let\((H_{1})\), \((H_{2})\), and\((H_{3})\) (or\((H_{3}')\)) hold. If\(h_{1}(x),h_{2}(x)>0\), then problem (1.1) has two sequences\(\{u_{k}\}\)and\(\{\overline{u}_{k}\}\)of nontrivial solutions such that

$$ I(u_{k})\rightarrow0^{-},\qquad I(\overline{u}_{k}) \rightarrow\infty\quad \textit{as } k\rightarrow\infty. $$

The rest of the paper is organized as follows. In the forthcoming section, we set up the variational framework for (1.1) and state the variant fountain theorems that will be used later. In Sect. 3, we study problem (1.1) and give the proof of Theorem 1.1. Section 4 is devoted to the proof of Theorem 1.2.

2 Preliminaries

First of all, we give some basic results of fractional Sobolev space that will be used in the next sections. Let \(0< s<1<p\) be real numbers. The fractional Sobolev space \(W^{s,p}(\mathbb{R}^{N})\) is defined as follows:

$$ W^{s,p}\bigl(\mathbb{R}^{N}\bigr)= \biggl\{ u\in L^{p}\bigl(\mathbb{R}^{N}\bigr): \int_{ \mathbb{R}^{N}\times\mathbb{R}^{N}} \frac{ \vert u(x)-u(y) \vert ^{p}}{ \vert x-y \vert ^{N+sp}}\,dx\,dy< \infty\biggr\} $$
(2.1)

equipped with the norm

$$ \Vert u \Vert _{W^{s,p}(\mathbb{R}^{N})}= \biggl( \Vert u \Vert _{L^{p}(\mathbb{R}^{N})}^{p}+ \int_{\mathbb{R}^{N}\times\mathbb{R}^{N}} \frac{ \vert u(x)-u(y) \vert ^{p}}{ \vert x-y \vert ^{N+sp}}\,dx\,dy \biggr)^{\frac{1}{p}}. $$
(2.2)

Then \((W^{s,p}(\mathbb{R}^{N}), \|u\|_{W^{s,p}(\mathbb{R}^{N})})\) is a uniformly convex Banach space and the embedding

$$ W^{s,p}\bigl(\mathbb{R}^{N}\bigr)\hookrightarrow L^{t}\bigl(\mathbb{R}^{N}\bigr) $$

is continuous for \(t\in[p,p_{s}^{*}]\). Moreover, the embedding is locally compact whenever \(1\leq t< p_{s}^{*}\), see [1] for details.

For our problem (1.1), consider the subspace \(X\subset W^{s,p}(\mathbb{R}^{N})\) given by

$$ X= \biggl\{ u\in W^{s,p}\bigl(\mathbb{R}^{N} \bigr): \int_{\mathbb{R}^{N}}V(x) \vert u \vert ^{p}\,dx< \infty\biggr\} . $$
(2.3)

Then X is a separable Banach space with the norm

$$ \Vert u \Vert = \biggl( \int_{\mathbb{R}^{N}\times\mathbb{R}^{N}} \frac{ \vert u(x)-u(y) \vert ^{p}}{ \vert x-y \vert ^{N+sp}}\,dx\,dy+ \int_{\mathbb{R}^{N}}V(x) \vert u \vert ^{p}\,dx \biggr)^{\frac{1}{p}}. $$
(2.4)

As usual, for \(t\geq1\), we let

$$ \vert u \vert _{t}= \biggl( \int_{\mathbb{R}^{N}} \vert u \vert ^{t}\,dx \biggr)^{\frac{1}{t}},\quad u\in L^{t}\bigl(\mathbb{R}^{N} \bigr). $$
(2.5)

By the embedding \(X\hookrightarrow L^{t}(\mathbb{R}^{N})\), we know that there exists a constant \(S_{t}>0\) such that

$$ \vert u \vert _{t}\leq S_{t} \Vert u \Vert ,\quad t\in\bigl[p,p_{s}^{*}\bigr]. $$
(2.6)

Definition 2.1

A function \(u\in X\) is said to be a (weak) solution of (1.1) if, for any \(v\in X\), we have

$$ \begin{aligned}[b] &\int_{\mathbb{R}^{N}\times\mathbb{R}^{N}} \frac{ \vert u(x)-u(y) \vert ^{p-2}(u(x)-u(y))(v(x)-v(y))}{ \vert x-y \vert ^{N+sp}}\, dx\,dy+ \int_{\mathbb{R}^{N}}V(x) \vert u \vert ^{p-2}uv\,dx \\ &\quad = \int_{\mathbb{R}^{N}}h_{1}(x) \vert u \vert ^{q-2}uv\,dx+ \int_{\mathbb{R}^{N}}h_{2}(x) \vert u \vert ^{r-2}uv\,dx. \end{aligned} $$
(2.7)

Let \(I(u): X\rightarrow\mathbb{R}\) be the energy functional associated with (1.1) defined by

$$ I(u)=\frac{1}{p} \Vert u \Vert ^{p}- \frac{1}{q} \int_{\mathbb{R}^{N}}h_{1}(x) \vert u \vert ^{q}\,dx- \frac{1}{r} \int_{\mathbb{R}^{N}}h_{2}(x) \vert u \vert ^{r}\,dx. $$
(2.8)

Using (2.6), it follows from conditions \((H_{1})\), \((H_{2})\), and \((H_{3})\) (or \((H'_{3})\)) that the functional I is well defined and \(I\in C^{1}(X,\mathbb{R})\) with

$$ \begin{aligned}[b] I'(u)v&= \int_{\mathbb{R}^{N}\times\mathbb{R}^{N}} \frac{ \vert u(x)-u(y) \vert ^{p-2}(u(x)-u(y))(v(x)-v(y))}{ \vert x-y \vert ^{N+sp}}\, dx\,dy \\ &\quad {}+ \int_{\mathbb{R}^{N}}V(x) \vert u \vert ^{p-2}uv\,dx- \int_{\mathbb{R}^{N}}h_{1}(x) \vert u \vert ^{q-2}uv\,dx- \int_{\mathbb{R}^{N}}h_{2}(x) \vert u \vert ^{r-2}uv\,dx \end{aligned} $$
(2.9)

for any \(v\in X\). It is standard to verify that the weak solutions of (1.1) correspond to the critical points of I.

Finally, we give the variant fountain theorems.

Let X be a Banach space with the norm \(\|\cdot\|\) and \(X=\overline{\bigoplus_{j=1}^{\infty}X_{j}}\) with \(\dim X_{j}<\infty\), \(j\in\mathbb{Z}\). Define

$$ Y_{k}=\bigoplus_{j=1}^{k} X_{j}, \qquad Z_{k}= \overline{\bigoplus_{j=k}^{\infty}X_{j}}. $$
(2.10)

Consider the following \(C^{1}\)-functional \(I_{\lambda}: X\rightarrow\mathbb{R}\) defined by

$$ I_{\lambda}(u):= A(u)-\lambda B(u),\quad \lambda\in[1,2]. $$
(2.11)

The following two variant fountain theorems were established in [22].

Theorem 2.2

Assume that the functional\(I_{\lambda}\)defined above satisfies

\((A_{1})\):

\(I_{\lambda}\)maps bounded sets into bounded sets uniformly for\(\lambda\in[1,2]\), and\(I_{\lambda}(-u)=I_{\lambda}(u)\)for all\((\lambda,u)\in[1,2]\times X\);

\((A_{2})\):

\(B(u)\geq0\)for all\(u\in X\), and\(B(u)\rightarrow\infty\)as\(\|u\|\rightarrow\infty\)on any finite dimensional subspace ofX;

\((A_{3})\):

There exist\(\rho_{k}>r_{k}>0\)such that

$$ a_{k}(\lambda):=\inf_{u\in Z_{k}, \|u\|=\rho_{k}}I_{\lambda}(u) \geq0,\qquad b_{k}(\lambda):=\max_{u\in Y_{k}, \|u\|=r_{k}}I_{\lambda}(u)< 0, \quad \forall\lambda\in[1,2] $$

and

$$ d_{k}(\lambda):=\inf_{u\in Z_{k}, \|u\|\leq\rho_{k}}I_{\lambda}(u) \rightarrow0,\quad \textit{as } k\rightarrow\infty\textit{ uniformly for } \lambda \in[1,2]. $$

Then there exist\(\lambda_{n}\rightarrow1\), \(u_{n}(\lambda_{n})\in Y_{n}\)such that

$$ I_{\lambda_{n}}'|_{Y_{n}}\bigl(u(\lambda _{n})\bigr)=0, \quad \textit{and}\quad I_{ \lambda_{n}}\bigl(u(\lambda _{n})\bigr)\rightarrow c_{k}\quad \textit{as } n \rightarrow\infty, $$

where\(c_{k}\in[d_{k}(2),b_{k}(1)]\). In particular, if\(\{u(\lambda_{n})\}\)has a convergent subsequence for everyk, then\(I_{1}\)has infinitely many nontrivial critical points\(\{u_{k}\}\in X\setminus \{0\}\)satisfying\(I_{1}(u_{k})\rightarrow0^{-}\)as\(k\rightarrow\infty\).

Theorem 2.3

Assume that the functional\(I_{\lambda}\)defined above satisfies:

\((B_{1})\):

\(I_{\lambda}\)maps bounded sets into bounded sets uniformly for\(\lambda\in[1,2]\), and\(I_{\lambda}(-u)=I_{\lambda}(u)\)for all\((\lambda,u)\in[1,2]\times X\);

\((B_{2})\):

\(B(u)\geq0\), \(A(u)\rightarrow\infty\)or\(B(u)\rightarrow\infty\)as\(\|u\|\rightarrow\infty\) (or\(B(u)\leq0\), \(B(u)\rightarrow-\infty\)as\(\|u\|\rightarrow\infty\));

\((B_{3})\):

There exist\(\rho_{k}>r_{k}>0\)such that

$$ b_{k}(\lambda)=\inf_{u\in Z_{k}, \|u\|=r_{k}}I_{\lambda}(u)>a_{k}( \lambda)=\max_{u\in Y_{k}, \|u\|=\rho_{k}}I_{\lambda}(u),\quad \forall \lambda \in[1,2]. $$

Then

$$ b_{k}(\lambda)\leq c_{k}(\lambda):=\inf _{\gamma\in\varGamma_{k}} \max_{u\in B_{k}}I_{\lambda}\bigl( \gamma(u)\bigr),\quad \forall\lambda\in[1,2], $$

where\(B_{k}=\{u\in Y_{k}: \|u\|\leq\rho_{k}\}\)and\(\varGamma_{k}=\{\gamma\in C(B_{k},X): \gamma\textit{ is odd}, \gamma|_{\partial B_{k}=\mathrm{id}}\}\) (\(k\geq2\)). Moreover, for almost every\(\lambda\in[1,2]\), there exists a sequence\(u_{n}^{k}(\lambda)\)such that

$$ \sup_{n} \bigl\Vert u_{n}^{k}( \lambda) \bigr\Vert < \infty, \qquad I_{\lambda}' \bigl(u_{n}^{k}( \lambda)\bigr)\rightarrow0 \quad \textit{and}\quad I_{\lambda}\bigl(u_{n}^{k}( \lambda)\bigr) \rightarrow c_{k}(\lambda) \quad \textit{as } n \rightarrow\infty. $$

3 Proof of Theorem 1.1

In this section, we use Theorem 2.2 to complete the proof.

For the notation in Theorem 2.2, we define the functional A, B, and \(I_{\lambda}\) on our working space X by

$$ A(u)=\frac{1}{p} \Vert u \Vert ^{p}- \frac{1}{r} \int_{\mathbb{R}^{N}}h_{2}(x) \vert u \vert ^{r}\,dx,\qquad B(u)=\frac{1}{q} \int_{\mathbb{R}^{N}}h_{1}(x) \vert u \vert ^{q}\,dx, $$
(3.1)

and

$$ \begin{aligned}[b] I_{\lambda}(u)&=A(u)-\lambda B(u) \\ &=\frac{1}{p} \Vert u \Vert ^{p}- \frac{1}{r} \int_{\mathbb{R}^{N}}h_{2}(x) \vert u \vert ^{r}\,dx- \frac{\lambda}{q} \int_{\mathbb{R}^{N}}h_{1}(x) \vert u \vert ^{q}\,dx,\quad \lambda\in[1,2]. \end{aligned} $$
(3.2)

Since X is a separable and reflexive Banach space, then there exist \(\{e_{i}\}_{i=1}^{\infty}\subset X\) and \(\{e_{i}^{*}\}_{i=1}^{\infty}\subset X^{*}\) such that

$$ X=\overline{\operatorname{span}\{e_{1},e_{2}, \ldots\}},\qquad X^{*}= \overline{\operatorname{span}\bigl\{ e_{1}^{*},e_{2}^{*},\ldots \bigr\} } $$
(3.3)

and

$$ \bigl\langle e_{i}^{*},e_{j} \bigr\rangle = \textstyle\begin{cases} 1,& \mbox{if } i=j, \\ 0,& \mbox{if } i\neq j. \end{cases} $$
(3.4)

Let \(X_{i}=\mathbb{R}e_{i}\), and \(Y_{k}\) and \(Z_{k}\) be defined as (2.10).

In the proof of our results, we need the following limits.

Lemma 3.1

Assume\((H_{1})\), \((H_{2})\), and\((H_{3})\) (or\((H'_{3})\)), and let

$$ \alpha_{k}:=\sup_{u\in Z_{k}, \Vert u \Vert =1}\biggl( \int_{\mathbb{R}^{N}} \vert h_{1} \vert \vert u \vert ^{q}\,dx\biggr)^{ \frac{1}{q}},\qquad \beta_{k}:= \sup_{u\in Z_{k}, \Vert u \Vert =1}\biggl( \int_{ \mathbb{R}^{N}} \vert h_{2} \vert \vert u \vert ^{r}\,dx\biggr)^{\frac{1}{r}}, $$
(3.5)

then\(\alpha_{k},\beta_{k}\rightarrow0\)as\(k\rightarrow\infty\).

Proof

It is clear that \(0<\alpha_{k+1}\leq\alpha_{k}\), so \(\alpha_{k}\rightarrow\alpha_{0}\geq0\) as \(k\rightarrow\infty\). For every \(k\in\mathbb{N}^{+}\), taking \(u_{k}\in Z_{k}\) such that \(\|u_{k}\|=1\) and

$$ \biggl( \int_{\mathbb{R}^{N}} \vert h_{1} \vert \vert u_{k} \vert ^{q}\,dx\biggr)^{\frac{1}{q}}> \frac{1}{2}\alpha_{k}>0. $$
(3.6)

As X is reflexive, \(\{u_{k}\}\) has a weakly convergent subsequence, without loss of generality, suppose \(u_{k}\rightharpoonup u\) weakly in X. Then, for every \(i\in\mathbb{N}^{+}\), we have

$$ \bigl\langle e_{i}^{*},u\bigr\rangle = \lim_{k\rightarrow\infty}\bigl\langle e_{i}^{*},u_{k} \bigr\rangle =0, $$
(3.7)

which implies that \(u=0\) and \(u_{k}\rightharpoonup0\) weakly in X.

Let \((H_{2})\) hold, then for any given small \(\varepsilon>0\), we may find \(R>0\) big enough such that

$$ \vert h_{1} \vert _{L^{\tau}(B_{R}^{c})}< \frac{\varepsilon}{2S_{q\tau'}^{q}}, $$
(3.8)

where \(B_{R}=\{x\in\mathbb{R}^{N}: |x|< r\}\), \(B_{R}^{c}=\mathbb{R}^{N}\setminus B_{R}\), and \(S_{q\tau'}\) is the embedding constant in (2.6).

If \(\frac{p_{s}^{*}}{p_{s}^{*}-q}<\tau\leq\frac{p}{p-q}\), since the embedding \(X\hookrightarrow L^{q\tau'}(B_{R})\) is compact, then \(u_{k}\rightarrow0\) in \(L^{q\tau'}(B_{R})\) and hence there exists \(K_{1}>0\) such that

$$ \vert u_{k} \vert _{L^{q\tau'}(B_{R})}< \frac{\varepsilon}{2 \vert h_{1} \vert _{\tau}} $$
(3.9)

for \(k>K_{1}\).

Using (3.8) and (3.9), for all \(k>K_{1}\), we get

$$ \begin{aligned}[b] \int_{\mathbb{R}^{N}} \vert h_{1} \vert \vert u_{k} \vert ^{q}\,dx&= \int_{B_{R}} \vert h_{1} \vert \vert u_{k} \vert ^{q}\,dx+ \int_{B_{R}^{c}} \vert h_{1} \vert \vert u_{k} \vert ^{q}\,dx \\ &\leq \vert h_{1} \vert _{L^{\tau}(B_{R})} \vert u_{k} \vert _{L^{q\tau'}(B_{R})}^{q}+ \vert h_{1} \vert _{L^{ \tau}(B_{R}^{c})} \vert u_{k} \vert _{L^{q\tau'}(B_{R}^{c})}^{q} \\ &\leq \vert h_{1} \vert _{\tau} \vert u_{k} \vert _{L^{q\tau'}(B_{R})}^{q}+ \vert h_{1} \vert _{L^{ \tau}(B_{R}^{c})} \vert u_{k} \vert _{q\tau'}^{q} \\ &\leq\varepsilon. \end{aligned} $$
(3.10)

If \(\tau=\frac{p_{s}^{*}}{p_{s}^{*}-q}\), since \(u_{k}\rightharpoonup0\) in E and the compact embedding \(X\hookrightarrow L^{t}(B_{R})\) (\(1\leq t< p_{s}^{*}\)), we can assume that \(u_{k}\rightarrow0\) a.e. in \(B_{R}\). For each measurable subset \(\varOmega\subset B_{R}\), we have

$$ \int_{\varOmega} \vert h_{1} \vert \vert u_{k} \vert ^{q}\,dx\leq \vert h_{1} \vert _{L^{\tau}(\varOmega)} \vert u_{k} \vert _{L^{p_{s}^{*}}( \varOmega)}^{q} \leq C \vert h_{1} \vert _{L^{\tau}(\varOmega)}. $$
(3.11)

Then \(\{|h_{1}||u_{k}|^{q}\}\) is uniformly integrable, and the Vitali convergence theorem implies

$$ \lim_{k\rightarrow\infty} \int_{B_{R}} \vert h_{1} \vert \vert u_{k} \vert ^{q}\,dx=0. $$
(3.12)

So, for k big enough, (3.10) still holds.

Then, from (3.6) and (3.10), we conclude that \(\alpha_{k}\rightarrow0\) as \(k\rightarrow\infty\).

Assume \((H_{3})\). Since \(\mu\in[\frac{p_{s}^{*}}{p_{s}^{*}-r},\infty)\) implies \(p< r\mu'\leq p_{s}^{*}\), arguing as in the above proof, one has \(\beta_{k}\rightarrow0\) as \(k\rightarrow\infty\).

Similarly, if assumption \((H_{3}')\) holds, it follows

$$ \begin{aligned}[b] \int_{\mathbb{R}^{N}} \vert h_{2} \vert \vert u_{k} \vert ^{r}\,dx&= \int_{B_{R}} \vert h_{2} \vert \vert u_{k} \vert ^{r}\,dx+ \int_{B_{R}^{c}} \vert h_{2} \vert \vert u_{k} \vert ^{r}\,dx \\ &\leq \vert h_{2} \vert _{L^{\infty}(\mathbb{R}^{N})} \vert u_{k} \vert _{L^{r}(B_{R})}^{r}+ \vert h_{2} \vert _{L^{\infty}(B_{R}^{c})} \vert u_{k} \vert _{L^{r}(B_{R}^{c})}^{r}. \end{aligned} $$
(3.13)

Since \(h_{2}(x)\in L^{\infty}(\mathbb{R}^{N})\) and \(h_{2}(x)\rightarrow0\) as \(|x|\rightarrow\infty\), \(\beta_{k}\rightarrow0\) can be obtained in a similar way, and we complete the proof. □

In order to apply Theorem 2.2, we give the following lemma first.

Lemma 3.2

Let\((H_{1})\), \((H_{2})\), and\((H_{3})\) (or\((H'_{3})\)) hold. Then there exist two sequences\(0< r_{k}<\rho_{k}\rightarrow0\)as\(k\rightarrow\infty\)such that

$$ a_{k}(\lambda)=\inf_{u\in Z_{k}, \|u\|=\rho_{k}}I_{\lambda}(u) \geq0,\qquad b_{k}(\lambda)=\max_{u\in Y_{k}, \|u\|=r_{k}}I_{\lambda}(u)< 0,\quad \forall\lambda\in[1,2] $$

and

$$ d_{k}(\lambda)=\inf_{u\in Z_{k}, \|u\|\leq\rho_{k}}I_{\lambda}(u) \rightarrow0,\quad \textit{as } k\rightarrow\infty\textit{ uniformly for } \lambda \in[1,2]. $$

Proof

From Lemma 3.1 we see that, for every \(u\in Z_{k}\), it holds

$$ \int_{\mathbb{R}^{N}} \vert h_{1} \vert \vert u \vert ^{q}\,dx\leq\bigl(\alpha_{k} \Vert u \Vert \bigr)^{q},\qquad \int_{\mathbb{R}^{N}} \vert h_{2} \vert \vert u \vert ^{r}\,dx\leq\bigl(\beta_{k} \Vert u \Vert \bigr)^{r}. $$
(3.14)

Thus

$$ I_{\lambda}(u)\geq\frac{1}{p} \Vert u \Vert ^{p}-\frac{1}{r}\beta_{k}^{r} \Vert u \Vert ^{r}- \frac{\lambda}{q}\alpha _{k}^{q} \Vert u \Vert ^{q}. $$
(3.15)

Fix \(K_{2}>0\) big enough such that \(\frac{1}{r}\beta_{k}^{r}<\frac{1}{2p}\) for \(k>K_{2}\), then for \(u\in Z_{k}\) and \(\|u\|<1\), we have

$$ I_{\lambda}(u)\geq\frac{1}{p} \Vert u \Vert ^{p}-\frac{1}{2p} \Vert u \Vert ^{r}- \frac{\lambda}{q}\alpha_{k}^{q} \Vert u \Vert ^{q}\geq\frac{1}{2p} \Vert u \Vert ^{p}- \frac{2}{q}\alpha_{k}^{q} \Vert u \Vert ^{q}. $$
(3.16)

If we choose \(\rho_{k}=(8p\alpha_{k}^{q}/q)^{\frac{1}{p-q}}\), then \(\rho_{k}\rightarrow0^{+}\) as \(k\rightarrow\infty\) and for any \(u\in Z_{k}\) with \(\|u\|=\rho_{k}\), we get that

$$ I_{\lambda}(u)\geq\frac{1}{4p}\bigl(8p\alpha _{k}^{q}/q\bigr)^{\frac{p}{p-q}}>0. $$
(3.17)

This inequality implies that

$$ a_{k}(\lambda)=\inf_{u\in Z_{k}, \|u\|=\rho _{k}}I_{\lambda}(u)> 0,\quad \forall\lambda\in[1,2]. $$
(3.18)

In addition, for all \(\lambda\in[1,2]\), \(k>K_{2}\) and \(u\in Z_{k}\) with \(\|u\|\leq\rho_{k}\), we have

$$ I_{\lambda}(u)\geq-\frac{2}{q}\alpha _{k}^{q} \Vert u \Vert ^{q}\geq- \frac{2}{q}\alpha_{k}^{q}\rho _{k}^{q}\rightarrow0, \quad \text{as } k \rightarrow \infty. $$
(3.19)

Obviously,

$$ d_{k}(\lambda)=\inf_{u\in Z_{k}, \|u\|\leq\rho _{k}}I_{\lambda}(u) \leq I_{\lambda}(0)=0. $$
(3.20)

So we have \(d_{k}(\lambda)\rightarrow0\) as \(k\rightarrow\infty\) uniformly for \(\lambda\in[1,2]\).

For all \(u\in Y_{k}\), \(\lambda\in[1,2]\), by the equivalence of any norm in a finite dimensional space, we can derive

$$ \begin{aligned}[b] I_{\lambda}(u)&\leq \frac{1}{p} \Vert u \Vert ^{p}+\frac{1}{r} \int_{\mathbb{R}^{N}} \vert h_{2} \vert \vert u \vert ^{r}\,dx- \frac{1}{q} \int_{\mathbb{R}^{N}}h_{1}(x) \vert u \vert ^{q}\,dx \\ &\leq\frac{1}{p} \Vert u \Vert ^{p}+ \frac{C_{1}}{r} \Vert u \Vert ^{r}-\frac{C_{2}}{q} \Vert u \Vert ^{q}, \end{aligned} $$
(3.21)

where \(C_{1}\geq0\), \(C_{2}>0\). Notice \(q< p< r\), so we can choose \(r_{k}>0\) small enough satisfying \(r_{k}<\rho_{k}\) such that

$$ b_{k}(\lambda)=\max_{u\in Y_{k}, \|u\|=r_{k}}I_{\lambda}(u)< 0. $$
(3.22)

The proof is completed. □

Proof of Theorem 1.1

At first, we confirm conditions \((A_{1})\)–\((A_{2})\) in Theorem 2.2. It follows from \((H_{1})\)–\((H_{3})\) that \(I_{\lambda}\) maps bounded sets into bounded sets uniformly for \(\lambda\in[1,2]\). Evidently, \(I_{\lambda}(-u)=I_{\lambda}(u)\) for all \((\lambda,u)\in[1,2]\times X\). \(B(u)\geq0\) for all \(u\in X\), and by the equivalence of any norm in a finite dimensional space, we know that \(B(u)\rightarrow\infty\) as \(\|u\|\rightarrow\infty\) on any finite dimensional subspace of X. So \((A_{1})\) and \((A_{2})\) hold.

From Lemma 3.2, we see that \((A_{3})\) in Theorem 2.2 are verified. Consequently, for any \(k\in\mathbb{Z}^{+}\), there exist \(\lambda_{n}\rightarrow1\), \(u(\lambda_{n})\in Y_{n}\) such that

$$ I_{\lambda_{n}}'|_{Y_{n}}\bigl(u( \lambda_{n})\bigr)=0, \quad \text{and} \quad I_{ \lambda_{n}}\bigl(u( \lambda_{n})\bigr)\rightarrow c_{k}\in \bigl[d_{k}(2),b_{k}(1)\bigr]\quad \text{as } n \rightarrow\infty. $$
(3.23)

Equation (3.23) implies \(I_{\lambda_{n}}'(u(\lambda_{n}))\rightarrow0\) as \(n\rightarrow\infty\). Then, for large n, from \((H_{1})\)–\((H_{3})\) and (2.6) we have

$$ \begin{aligned}[b] c+1+ \bigl\Vert u(\lambda_{n}) \bigr\Vert &\geq I_{\lambda_{n}}\bigl(u(\lambda_{n})\bigr)- \frac{1}{r}I'_{\lambda_{n}}\bigl(u(\lambda _{n})\bigr)u(\lambda_{n}) \\ &=\biggl(\frac{1}{p}-\frac{1}{r}\biggr) \bigl\Vert u( \lambda_{n}) \bigr\Vert ^{p}-\lambda _{n}\biggl( \frac{1}{q}-\frac{1}{r}\biggr) \int_{\mathbb{R}^{N}} h_{1}(x) \bigl\vert u(\lambda _{n}) \bigr\vert ^{q} \,dx \\ &\geq\biggl(\frac{1}{p}-\frac{1}{r}\biggr) \bigl\Vert u( \lambda_{n}) \bigr\Vert ^{p}-2S_{q\tau'}^{q} \biggl( \frac{1}{q}-\frac{1}{r}\biggr) \vert h_{1} \vert _{\tau} \bigl\Vert u(\lambda _{n}) \bigr\Vert ^{q}. \end{aligned} $$
(3.24)

Since \(p>1\), \(p>q>0\), the above inequality implies that \(\{u(\lambda_{n})\}\) is bounded in X.

Then there exist a constant \(M>0\) and \(u\in X\), and a subsequence \(\{u(\lambda_{n})\}\), denoted by \(\{u_{n}\}\), such that \(\|u\|,\|u_{n}\|\leq M\) and \(u_{n}\rightharpoonup u\) weakly in X. Arguing as in the proof of Lemma 3.1, we have

$$ \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \vert h_{1} \vert \vert u_{n}-u \vert ^{q}\,dx=0,\qquad \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \vert h_{2} \vert \vert u_{n}-u \vert ^{r}\,dx=0. $$
(3.25)

Using the Hölder inequality, we have

$$ \int_{\mathbb{R}^{N}} \vert h_{1} \vert \vert u_{n} \vert ^{q-2}u_{n}(u_{n}-u) \,dx\leq\biggl( \int_{\mathbb{R}^{N}} \vert h_{1} \vert \vert u_{n} \vert ^{q}\,dx\biggr)^{\frac{q-1}{q}}\biggl( \int_{\mathbb{R}^{N}} \vert h_{1} \vert \vert u_{n}-u \vert ^{q}\,dx\biggr)^{\frac{1}{q}} $$
(3.26)

and

$$ \int_{\mathbb{R}^{N}} \vert h_{2} \vert \vert u_{n} \vert ^{r-2}u_{n}(u_{n}-u) \,dx\leq\biggl( \int_{\mathbb{R}^{N}} \vert h_{2} \vert \vert u_{n} \vert ^{q}\,dx\biggr)^{\frac{r-1}{r}}\biggl( \int_{\mathbb{R}^{N}} \vert h_{2} \vert \vert u_{n}-u \vert ^{r}\,dx\biggr)^{\frac{1}{r}}. $$
(3.27)

Then (3.25) implies

$$ \begin{aligned} &\lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}}h_{1}(x) \vert u_{n} \vert ^{q-2}u_{n}(u_{n}-u)\,dx=0, \\ &\lim _{n\rightarrow\infty} \int_{\mathbb{R}^{N}}h_{2}(x) \vert u_{n} \vert ^{r-2}u_{n}(u_{n}-u)\,dx=0. \end{aligned} $$
(3.28)

Denote

$$ \begin{aligned}[b] P_{n}&:=I'_{\lambda_{n}}(u_{n}) (u_{n}-u) \\ &= \int_{\mathbb{R}^{N}\times\mathbb{R}^{N}} \frac{ \vert u_{n}(x)-u_{n}(y) \vert ^{p-2}(u_{n}(x)-u_{n}(y))((u_{n}(x)-u(x))-(u_{n}(y)-u(y)))}{ \vert x-y \vert ^{N+sp}}\,dx\,dy \\ &\quad {}+ \int_{\mathbb{R}^{N}}V(x) \vert u_{n} \vert ^{p-2}u_{n}(u_{n}-u)\,dx+ \int_{ \mathbb{R}^{N}}h_{1}(x) \vert u_{n} \vert ^{q-2}u_{n}(u_{n}-u)\,dx \\ &\quad {}+ \int_{\mathbb{R}^{N}}h_{2}(x) \vert u_{n} \vert ^{r-2}u_{n}(u_{n}-u)\,dx \end{aligned} $$
(3.29)

and

$$ \begin{aligned}[b] Q_{n}&:= \int_{\mathbb{R}^{N}\times\mathbb{R}^{N}} \frac{ \vert u(x)-u(y) \vert ^{p-2}(u(x)-u(y))((u_{n}(x)-u(x))-(u_{n}(y)-u(y)))}{ \vert x-y \vert ^{N+sp}}\,dx\,dy \\ &\quad {}+ \int_{\mathbb{R}^{N}}V(x) \vert u \vert ^{p-2}u(u_{n}-u) \,dx. \end{aligned} $$
(3.30)

Then the fact \(I_{\lambda_{n}}'(u(\lambda_{n}))\rightarrow0\) shows that \(P_{n}\rightarrow0\) as \(n\rightarrow\infty\). Moreover, \(\{u_{n}\}\) is a bounded sequence and \(u_{n}\rightharpoonup u\) in X, which imply \(Q_{n}\rightarrow0\).

Equations (3.29) and (3.30) show that, for large n,

$$ \begin{aligned}[b] & \int_{\mathbb{R}^{N}\times\mathbb{R}^{N}} \frac{ \vert u_{n}(x)-u_{n}(y) \vert ^{p-2}(u_{n}(x)-u_{n}(y))- \vert u(x)-u(y) \vert ^{p-2}(u(x)-u(y))}{ \vert x-y \vert ^{N+sp}} \\ &\qquad {}\times \bigl( \bigl(u_{n}(x)-u(x) \bigr)- \bigl(u_{n}(y)-u(y) \bigr) \bigr)\,dx\,dy \\ &\qquad {}+ \int_{\mathbb{R}^{N}}V(x) \bigl( \vert u_{n} \vert ^{p-2}u_{n}- \vert u \vert ^{p-2}u\bigr) (u_{n}-u)\,dx \\ &\quad =P_{n}-Q_{n}+o_{n}(1)\rightarrow0. \end{aligned} $$
(3.31)

By using the standard inequalities (see [23]) given by

$$ \bigl\langle \vert \xi \vert ^{p-2}\xi- \vert \eta \vert ^{p-2}\eta, \xi-\eta\bigr\rangle \geq C_{p} \vert \xi-\eta \vert ^{p},\quad p\geq2, \forall\xi,\eta\in \mathbb{R}^{N} $$
(3.32)

and

$$ \bigl\langle \vert \xi \vert ^{p-2}\xi- \vert \eta \vert ^{p-2}\eta, \xi-\eta\bigr\rangle \geq C_{p} \vert \xi-\eta \vert ^{2}\bigl( \vert \xi \vert + \vert \eta \vert \bigr)^{p-2},\quad 1< p< 2, \forall\xi,\eta\in \mathbb{R}^{N}, $$
(3.33)

where \(C_{p}\) is a positive constant and \(\langle\cdot,\cdot\rangle\) denotes the inner product in \(\mathbb{R}^{N}\), we can easily deduce that \(\|u_{n}-u\|\rightarrow0\) as \(n\rightarrow\infty\). Now, from Theorem 2.2, we see that \(I=I_{1}\) possesses infinitely many nontrivial critical points \(u_{k}\) for \(k\in\mathbb{Z}^{+}\) satisfying \(I(u_{k})\rightarrow0^{-}\) as \(k\rightarrow\infty\). Therefore, problem (1.1) possesses infinitely many nontrivial solutions, the proof of Theorem 1.1 is completed. □

4 Proof of Theorem 1.2

For the notation in Theorem 2.2, we define the functional A, B and \(I_{\lambda}\) on our working space X by

$$ A(u)=\frac{1}{p} \Vert u \Vert ^{p}- \frac{1}{q} \int_{\mathbb{R}^{N}}h_{1}(x) \vert u \vert ^{q}\,dx, \qquad B(u)=\frac{1}{r} \int_{\mathbb{R}^{N}}h_{2}(x) \vert u \vert ^{r}\,dx, $$
(4.1)

and

$$ \begin{aligned}[b] I_{\lambda}(u)&=A(u)-\lambda B(u) \\ &=\frac{1}{p} \Vert u \Vert ^{p}- \frac{1}{q} \int_{\mathbb{R}^{N}}h_{1}(x) \vert u \vert ^{q}\,dx- \frac{\lambda}{r} \int_{\mathbb{R}^{N}}h_{2}(x) \vert u \vert ^{r}\,dx,\quad \lambda\in[1,2]. \end{aligned} $$
(4.2)

Lemma 4.1

Let\((H_{1})\)–\((H_{3})\)hold and\(h_{2}(x)>0\). Then there exist two sequences\(\rho_{k}>r_{k}>0\)such that

$$ b_{k}(\lambda)=\inf_{u\in Z_{k}, \|u\|=r_{k}}I_{\lambda}(u)>a_{k}( \lambda)=\max_{u\in Y_{k}, \|u\|=\rho_{k}}I_{\lambda}(u),\quad \forall \lambda \in[1,2]. $$

Proof

Similar to the beginning of the proof of Lemma 3.2, by (3.14) we have

$$ I_{\lambda}(u)\geq\frac{1}{p} \Vert u \Vert ^{p}-\frac{1}{q}\alpha_{k}^{q} \Vert u \Vert ^{q}- \frac{2}{r}\beta _{k}^{r} \Vert u \Vert ^{r}. $$
(4.3)

Fix \(K_{3}>0\) big enough such that \(\frac{2}{q}\alpha_{k}^{q}<\frac{1}{2p}\) for \(k>K_{3}\), then for \(u\in Z_{k}\) and \(\|u\|>1\), we have

$$ I_{\lambda}(u)\geq\frac{1}{2p} \Vert u \Vert ^{p}-\frac{2}{r}\beta_{k}^{r} \Vert u \Vert ^{r}. $$
(4.4)

If we choose \(r_{k}=(\frac{r}{8p\beta_{k}^{r}})^{\frac{1}{r-p}}\), then \(r_{k}\rightarrow\infty\) as \(k\rightarrow\infty\), and for any \(u\in Z_{k}\) with \(\|u\|=r_{k}\), we get that

$$ I_{\lambda}(u)\geq\frac{1}{4p}r_{k}^{p}>0. $$
(4.5)

This inequality implies that

$$ b_{k}(\lambda)=\inf_{u\in Z_{k}, \|u\|=r_{k}}I_{\lambda}(u) \geq\frac{1}{4p}r_{k}^{p}>0, \quad \forall\lambda \in[1,2]. $$
(4.6)

For all \(u\in Y_{k}\), \(\lambda\in[1,2]\), by the equivalence of any norm in a finite dimensional space, we can derive

$$ I_{\lambda}(u) \leq\frac{1}{p} \Vert u \Vert ^{p}+C_{1} \Vert u \Vert ^{q}-C_{2} \Vert u \Vert ^{r}, $$
(4.7)

where \(C_{1}\geq0\), \(C_{2}>0\). Notice \(q< p< r\), then \(I_{\lambda}(u)\rightarrow-\infty\) as \(\|u\|\rightarrow\infty\). So we can choose \(\rho_{k}>r_{k}\) big enough such that

$$ a_{k}(\lambda)=\max_{u\in Y_{k}, \|u\|=\rho _{k}}I_{\lambda}(u)< 0. $$
(4.8)

The proof is completed. □

Proof of Theorem 1.2

We complete the proof by Theorem 2.3. Let us verify the conditions in Theorem 2.3 firstly.

\((H_{1})\)–\((H_{3})\) imply that \(I_{\lambda}\) maps bounded sets into bounded sets uniformly for \(\lambda\in[1,2]\). Moreover, \(I_{\lambda}(-u)=I_{\lambda}(u)\) for all \(u\in X\) and \(\lambda\in[1,2]\). So condition \((B_{1})\) of Theorem 2.3 holds.

Evidently, \(B(u)\geq0\) for all \(u\in X\), and \(A(u)\rightarrow\infty\) as \(\|u\|\rightarrow\infty\). \((B_{2})\) in Theorem 2.3 is verified.

From Lemma 4.1, we see that condition \((B_{3})\) has been verified. Therefore, by Theorem 2.3, for a.e. \(\lambda\in[1,2]\), there exists a sequence \(u_{n}^{k}(\lambda)_{n=1}^{\infty}\) such that

$$ \sup_{n} \bigl\Vert u_{n}^{k}(\lambda) \bigr\Vert < \infty,\qquad I_{\lambda}'\bigl(u_{n}^{k}( \lambda)\bigr)\rightarrow0,\quad \mbox{and}\quad I_{\lambda}\bigl(u_{n}^{k}( \lambda)\bigr)\rightarrow c_{k}(\lambda)\quad \mbox{as } n\rightarrow \infty. $$
(4.9)

Moreover, by Theorem 2.3 and (4.6) we see that

$$ c_{k}(\lambda)\geq b_{k}(\lambda) \geq\frac{1}{4p}\biggl( \frac{r}{8p\beta_{k}^{r}}\biggr)^{\frac{p}{r-p}}:= \overline{b}_{k} \rightarrow\infty\quad \mbox{as } k\rightarrow\infty. $$
(4.10)

Since

$$ c_{k}(\lambda)=\inf_{\gamma\in\varGamma_{k}}\max _{u\in B_{k}}I_{\lambda}\bigl(\gamma(u)\bigr)\leq\max _{u\in B_{k}}I_{1}(u):=\overline{c}_{k}. $$
(4.11)

Then we have

$$ \overline{b}_{k}\leq c_{k}\leq \overline{c}_{k}. $$
(4.12)

If we choose a sequence \(\lambda_{m}\rightarrow1\), then (4.9) implies that the sequence \(\{u_{n}^{k}(\lambda_{m})\}_{n=1}^{\infty}\) is bounded. Using similar arguments as those in the proof of Theorem 1.1, we can prove that the sequence \(\{u_{n}^{k}(\lambda_{m})\}_{n=1}^{\infty}\) has a strong convergent subsequence as \(n\rightarrow\infty\). Thus we may assume that \(u_{n}^{k}(\lambda_{m})\rightarrow u^{k}(\lambda_{m})\) in X as \(n\rightarrow\infty\) for every \(m\in\mathbb{N}\). Moreover, by (4.9) and (4.12) we have

$$ I_{\lambda}'\bigl(u^{k}( \lambda_{m})\bigr)=0 \quad \mbox{and}\quad I_{\lambda_{m}} \bigl(u^{k}( \lambda_{m})\bigr)\in[ \overline{b}_{k},\overline{c}_{k}]. $$
(4.13)

As in the proof of Theorem 1.1, we can get the boundedness of \(\{u^{k}(\lambda_{m})\}_{m=1}^{\infty}\) and prove it possesses a strong convergent subsequence with the limit \(u^{k}\in X\). Therefore, the limit \(u^{k}\in X\) is a critical point of I with \(I(u^{k})\in[\overline{b}_{k},\overline{c}_{k}]\). Since \(\overline{b}_{k}\rightarrow\infty\) as \(k\rightarrow\infty\), we get infinitely many nontrivial critical points of I. Consequently, problem (1.1) possesses infinitely many nontrivial solutions with high energy. The proof is completed. □