1 Introduction and preliminaries

Consider the following fractional Schrödinger equation with electromagnetic fields and critical or supercritical nonlinearity:

$$ (-\Delta )_{A}^{s}u+V(x)u=f\bigl(x, \vert u \vert ^{2}\bigr)u+\lambda \vert u \vert ^{p-2}u, \quad x \in \mathbb{R}^{N}, $$
(1.1)

where \((-\Delta )_{A}^{s}\) is the fractional magnetic operator with \(0< s<1\), \(N>2s\), \(\lambda >0\), \(2_{s}^{*}=\frac{2N}{N-2s}\), \(p\geq 2_{s}^{*}\), f is a subcritical nonlinearity, and \(V \in C(\mathbb{R}^{N},\mathbb{R})\) and \(A \in C(\mathbb{R}^{N}, \mathbb{R}^{N})\) are the electric and magnetic potentials, respectively.

The fractional magnetic Laplacian is defined by

$$ (-\Delta )_{A}^{s}u(x)=C_{N,s}\lim _{r\rightarrow 0} \int _{B_{r}^{c}(x)} \frac{u(x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}u(y)}{ \vert x-y \vert ^{N+2s}}\,dy,\quad C_{N,s}= \frac{4^{s}\varGamma (\frac{N+2s}{2})}{\pi ^{\frac{N}{2}} \vert \varGamma (-s) \vert }. $$

This nonlocal operator has been defined in [4] as a fractional extension (for any \(s \in (0,1)\)) of the magnetic pseudorelativistic operator or Weyl pseudodifferential operator defined with midpoint prescription [1]. As stated in [17], up to correcting the operator by the factor \((1-s)\), it follows that \((-\Delta )_{A}^{s}u\) converges to \(-(\nabla u-iA)^{2}u\) as \(s\rightarrow 1\). Thus, up to normalization, the nonlocal case can be seen as an approximation of the local one. The motivation for its introduction is described in more detail in [4, 17] and relies essentially on the Lévy–Khintchine formula for the generator of a general Lévy process.

The main driving force for the study of problem (1.1) arises in the following time-dependent Schrödinger equation when \(s=1\):

$$ i\hbar \frac{\partial \psi }{\partial t}=\frac{1}{2m}\bigl(-i\hbar \nabla +A(x) \bigr)^{2} \psi +P(x)\psi -\rho \bigl(x, \vert \psi \vert \bigr) \psi, $$
(1.2)

where ħ is the Planck constant, m is the particle mass, \(A:\mathbb{R}^{N}\rightarrow \mathbb{R}^{N}\) is the magnetic potential, \(P:\mathbb{R}^{N}\rightarrow \mathbb{R}^{N}\) is the electric potential, ρ is the nonlinear coupling, and ψ is the wave function representing the state of the particle. This equation arises in quantum mechanics and describes the dynamics of the particle in a nonrelativistic setting [2, 15]. Clearly, the form \(\psi (x,t):=e^{-i\varpi th^{-1}}u(x)\) is a standing wave solution of (1.2) if and only if \(u(x)\) satisfies the following stationary equation:

$$ (-i\varepsilon \nabla +A)^{2}u+V(x)u=f\bigl(x, \vert u \vert \bigr)u, $$

where \(\varepsilon =\hbar \), \(V(x)=2m(P(x)-\varpi )\), and \(f=2m\rho \); see [3, 5, 7, 8]. By applying variational methods and Lyusternik–Schnirelmann theory Ambrosio and d’Avenia [1] proved the existence and multiplicity of solutions for the equation

$$ \varepsilon ^{2s}(-\Delta )_{A/\varepsilon }^{s}u+V(x)u=f \bigl( \vert u \vert ^{2}\bigr)u $$

when \(\varepsilon >0\) is small. Recently, Liang et al. [14] obtained the existence and multiplicity of solutions for the fractional Schrödinger–Kirchhoff equation

$$ \varepsilon ^{2s}M\bigl([u]_{s,A_{\varepsilon }}^{2}\bigr) (- \Delta )_{A_{\varepsilon }}^{s}u+V(x)u= \vert u \vert ^{2_{s}^{*}-2}u+h \bigl(x, \vert u \vert ^{2}\bigr)u $$

with the help of fractional version of the concentration compactness principle and variational methods. If the magnetic field \(A\equiv 0\), then the operator \((-\Delta )_{A}^{s}\) can be reduced to the fractional Laplacian operator \((-\Delta )^{s}\), which is defined as

$$ (-\Delta )^{s}u(x)=C_{N,s}\mathrm{P.V.} \int _{\mathbb{R}^{N}} \frac{u(x)-u(y)}{ \vert x-y \vert ^{N+2s}}\,dy =C_{N,s}\lim _{\varepsilon \rightarrow 0^{+}} \int _{\mathbb{R}^{N} \setminus B_{\varepsilon }(x)} \frac{u(x)-u(y)}{ \vert x-y \vert ^{N+2s}}\,dy. $$

The symbol \(\mathrm{P.V}.\) stands for the Cauchy principal value, and \(C_{N,s}\) is a dimensional constant that depends on \(N, s\), precisely given by

$$ C_{N,s}= \biggl( \int _{\mathbb{R}^{N}} \frac{1-\cos \zeta _{1}}{ \vert \zeta \vert ^{N+2s}}\,d\zeta \biggr)^{-1}. $$

It is well known that the fractional Laplacian \((-\Delta )^{s}\) can be viewed as a pseudodifferential operator of symbol \(|\xi |^{2s}\), as stated in Lemma 1.1 in [6]. Simultaneously, problem (1.1) becomes the classical Schrödinger equation

$$ (-\Delta )^{s}u+V(x)u=f\bigl(x, \vert u \vert ^{2} \bigr)u+\lambda \vert u \vert ^{p-2}u,\quad x \in \mathbb{R}^{N}. $$
(1.3)

Recently, there has been a lot of interest in the study of equation (1.3) and other related nonlocal problems. See, for instance, [6, 1013, 16, 2123] and the references therein. For more results about dealing with magnetic operators, see [9, 20]. Nonlocal problems also appear in other mathematical research fields. We refer the interested readers to [18, 19] for mathematical researches on Kirchhoff-type nonlocal equations, where Tang and Cheng [19] proposed a new approach to recover compactness for the (PS)-sequence, and Tang and Chen [18] proposed a new approach to recover compactness for the minimizing sequence.

Most of the works mentioned are set in \(\mathbb{R}^{N}\), \(N>2s\), with subcritical or critical growth, and to the best of our knowledge, no results are available on the existence for problem (1.1) with supercritical exponent. In this paper, we aim at studying the existence of nontrivial solutions for critical or supercritical problem (1.1).

To reduce the statements of the main result, we introduce the following assumptions:

(V):

\(V \in C(\mathbb{R}^{N},\mathbb{R})\), \(0< V_{0}:=\inf_{x\in \mathbb{R}^{N}}V(x)\), and \(\lim_{|x|\rightarrow +\infty }V(x)=+\infty \).

\((f_{1})\):

\(f\in C(\mathbb{R}^{N}\times \mathbb{R},\mathbb{R})\), and there exists \(2< q< 2_{s}^{*}\) such that

$$ \bigl\vert f(x,t) \bigr\vert \leq C\bigl(1+ \vert t \vert ^{\frac{q-2}{2}}\bigr) $$

for all \((x,t) \in \mathbb{R}^{N}\times \mathbb{R}\), where C is a positive constant.

\((f_{2})\):

\(f(x,t)=o(1)\) as \(|t|\to 0\) uniformly in \(x \in \mathbb{R}^{N}\);

\((f_{3})\):

\(f(x,t)t\geq \frac{q}{2}F(x,t):=\frac{q}{2}\int _{0}^{t}f(x, \tau )\,d\tau \) for all \((x,t) \in \mathbb{R}^{N}\times \mathbb{R}\);

\((f_{4})\):

\(c_{0}:=\inf_{x\in \mathbb{R}^{N},|t|=1}F(x,t)>0\).

For a function \(u:\mathbb{R}^{N}\rightarrow \mathbb{C}\), we set

$$ [u]_{A}^{2}= \int _{\mathbb{R}^{2N}} \frac{ \vert u(x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}u(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx\,dy $$

and

$$ D_{A}^{s}\bigl(\mathbb{R}^{N}, \mathbb{C}\bigr)= \bigl\{ u \in L^{2_{s}^{*}}\bigl( \mathbb{R}^{N}, \mathbb{C} \bigr):[u]_{A}^{2}< +\infty \bigr\} . $$

Then we may introduce the Hilbert space

$$ H_{A}^{s}\bigl(\mathbb{R}^{N}, \mathbb{C}\bigr)= \biggl\{ u \in D_{A}^{s}\bigl( \mathbb{R}^{N}, \mathbb{C}\bigr): \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx< +\infty \biggr\} $$

endowed with the scalar product

$$\begin{aligned} \langle u,v \rangle _{A}:={}&\frac{C_{N,s}}{2}\mathcal{R} \int _{\mathbb{R}^{2N}} \frac{[u(x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}u(y)]\cdot \overline{[v(x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}v(y)]}}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &{}+\mathcal{R} \int _{\mathbb{R}^{N}}u\bar{v}\,dx \end{aligned}$$

and norm

$$ \Vert u \Vert _{A}^{2}=\langle u,u \rangle _{A}, $$

where \(\mathcal{R}(z)\) is the real part of a complex number z. By Lemma 3.5 in [4] the embedding \(H_{A}^{s}(\mathbb{R}^{N}, \mathbb{C})\hookrightarrow L^{t}( \mathbb{R}^{N}, \mathbb{C})\) is continuous for any \(t \in [2,2_{s}^{*}]\), and the embedding \(H_{A}^{s}(\mathbb{R}^{N}, \mathbb{C})\hookrightarrow L_{loc}^{t}( \mathbb{R}^{N}, \mathbb{C})\) is compact for any \(t \in [1,2_{s}^{*})\). Moreover, set

$$ E= \biggl\{ u \in H_{A}^{s}\bigl(\mathbb{R}^{N}, \mathbb{C}\bigr): \int _{ \mathbb{R}^{N}}V(x) \vert u \vert ^{2}\,dx< +\infty \biggr\} $$

with the norm

$$ \Vert u \Vert ^{2}=\frac{C_{N,s}}{2}[u]_{A}^{2}+ \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{2}\,dx. $$

By assumption \((V)\) the embedding \(E\hookrightarrow H_{A}^{s}(\mathbb{R}^{N}, \mathbb{C})\) is continuous.

For convenience, we define the homogeneous fractional Sobolev space

$$ \mathcal{D}^{s,2}\bigl(\mathbb{R}^{N}\bigr):= \bigl\{ u \in L^{2_{s}^{*}}\bigl( \mathbb{R}^{N}\bigr): \vert \xi \vert ^{s}\hat{u}(\xi ) \in L^{2}\bigl(\mathbb{R}^{N} \bigr) \bigr\} , $$

which is the completion of \(C_{0}^{\infty }(\mathbb{R}^{N})\) under the norm

$$ \Vert u \Vert _{\mathcal{D}^{s,2}(\mathbb{R}^{N})}^{2}:= \int _{\mathbb{R}^{N}} \bigl\vert (- \Delta )^{\frac{s}{2}}u \bigr\vert ^{2}\,dx = \int _{\mathbb{R}^{N}} \vert \xi \vert ^{2s} \bigl\vert \hat{u}(\xi ) \bigr\vert ^{2}\,d\xi. $$

Define the norm on \(H^{s}(\mathbb{R}^{N})\) as follows:

$$ \Vert u \Vert _{H^{s}(\mathbb{R}^{N})}:= \biggl[ \int _{\mathbb{R}^{N}} \vert \xi \vert ^{2s} \bigl\vert \hat{u}(\xi ) \bigr\vert ^{2}\,d\xi + \int _{\mathbb{R}^{N}}u^{2}\,dx \biggr]^{ \frac{1}{2}} = \bigl[ \Vert u \Vert _{\mathcal{D}^{s,2}(\mathbb{R}^{N})}^{2}+ \Vert u \Vert _{L^{2}(\mathbb{R}^{N})}^{2} \bigr]^{\frac{1}{2}}. $$

Moreover, the best fractional critical Sobolev constant is given by

$$ S:=\inf_{u \in \mathcal{D}^{s,2}(\mathbb{R}^{N})\setminus \{0\}} \frac{ \Vert u \Vert _{\mathcal{D}^{s,2}(\mathbb{R}^{N})}^{2}}{ \Vert u \Vert _{2_{s}^{*}}^{2}}. $$

Our main result is the following:

Theorem 1.1

Suppose that\((V)\)and\((f_{1})\)\((f_{4})\)are satisfied. Then there exists\(\lambda _{0}>0\)such that for each\(\lambda \in (0,\lambda _{0}]\), problem (1.1) has a nontrivial solution\(u_{\lambda }\).

As a complement of Theorem 1.1, by the Pohozaev identity we can deduce that the equation

$$ (-\Delta )_{A}^{s}u+\mu u=\lambda \vert u \vert ^{p-2}u,\quad x \in \mathbb{R}^{N}, $$

with\(p \geq 2_{s}^{*}\)and\(\mu >0\)has no nontrivial solution for all\(\lambda >0\). Indeed, let\(u \in E\)be a weak solution of the problem. Then we have the following Pohozaev identity:

$$ \frac{1}{2_{s}^{*}}\cdot \frac{C_{N,s}}{2} \int _{\mathbb{R}^{2N}} \frac{ \vert u(x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}u(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx\,dy+ \frac{1}{2}\mu \int _{\mathbb{R}^{N}} \vert u \vert ^{2}\,dx = \frac{\lambda }{p} \int _{\mathbb{R}^{N}} \vert u \vert ^{p}\,dx. $$
(1.4)

Moreover, takinguas the test function, we have

$$ \frac{C_{N,s}}{2} \int _{\mathbb{R}^{2N}} \frac{ \vert u(x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}u(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx\,dy+ \mu \int _{\mathbb{R}^{N}}u^{2}\,dx =\lambda \int _{\mathbb{R}^{N}} \vert u \vert ^{p}\,dx. $$
(1.5)

Taking into account (1.4) and (1.5), we can derive that

$$ \frac{p-2_{s}^{*}}{2_{s}^{*}}\cdot \frac{C_{N,s}}{2} \int _{\mathbb{R}^{2N}} \frac{ \vert u(x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}u(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx\,dy+ \frac{p-2}{2}\mu \int _{\mathbb{R}^{N}}u^{2}\,dx =0, $$

which implies the conclusion.

2 Proof of Theorem 1.1

It is well known that a weak solution of problem (1.1) is a critical point of the following functional:

$$\begin{aligned} I_{\lambda }(u)={}&\frac{1}{2} \Vert u \Vert ^{2}- \frac{1}{2} \int _{ \mathbb{R}^{N}}F\bigl(x, \vert u \vert ^{2}\bigr)\,dx- \frac{\lambda }{p} \int _{\mathbb{R}^{N}} \vert u \vert ^{p}\,dx \\ ={}&\frac{1}{2}\cdot \frac{C_{N,s}}{2}[u]_{A}^{2}+ \frac{1}{2} \int _{ \mathbb{R}^{N}}V(x) \vert u \vert ^{2}\,dx- \frac{1}{2} \int _{\mathbb{R}^{N}}F\bigl(x, \vert u \vert ^{2}\bigr)\,dx- \frac{\lambda }{p} \int _{\mathbb{R}^{N}} \vert u \vert ^{p}\,dx \\ ={}&\frac{1}{2}\cdot \frac{C_{N,s}}{2} \int _{\mathbb{R}^{2N}} \frac{ \vert u(x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}u(y) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx\,dy+ \frac{1}{2} \int _{\mathbb{R}^{N}}V(x) \vert u \vert ^{2}\,dx \\ &{}-\frac{1}{2} \int _{\mathbb{R}^{N}}F\bigl(x, \vert u \vert ^{2}\bigr)\,dx -\frac{\lambda }{p} \int _{\mathbb{R}^{N}} \vert u \vert ^{p}\,dx. \end{aligned}$$

Clearly, we cannot apply variational methods directly because the functional \(I_{\lambda }\) is not well defined on E unless \(p=2_{s}^{*}\). To overcome this difficulty, we define the function

$$ \phi (t)= \textstyle\begin{cases} \vert t \vert ^{p-2}t & \text{if } \vert t \vert \leq M, \\ M^{p-q} \vert t \vert ^{q-2}t & \text{if } \vert t \vert >M, \end{cases} $$

where \(M>0\). Then \(\phi \in C(\mathbb{R},\mathbb{R})\), \(\phi (t)t\geq q\varPhi (t):=q\int _{0}^{t}\phi (\tau )\,d\tau \geq 0\), and \(|\phi (t)|\leq M^{p-q}|t|^{q-1}\) for all \(t \in \mathbb{R}\). Set \(h_{\lambda }(x,t)=\lambda \phi (t)+f(x,|t|^{2})t\) for \((x,t) \in \mathbb{R}^{N} \times \mathbb{R}\). Then \(h_{\lambda }(x,t)\) admits the following properties:

\((h_{1})\):

\(h_{\lambda }\in C(\mathbb{R}^{N} \times \mathbb{R}, \mathbb{R})\), and \(|h_{\lambda }(x,t)|\leq \lambda M^{p-q}|t|^{q-1}+C(|t|+|t|^{q-1})\) for all \((x,t) \in \mathbb{R}^{N} \times \mathbb{R}\).

\((h_{2})\):

\(h_{\lambda }(x,t)t\geq qH_{\lambda }(x,t):=q\int _{0}^{t}h_{\lambda }(x,\tau )\,d\tau \geq 0\) for all \((x,t) \in \mathbb{R}^{N} \times \mathbb{R}\).

\((h_{3})\):

\(\inf_{x\in \mathbb{R}^{N},|t|=1}H_{\lambda }(x,t) \geq \frac{c_{0}}{2}>0\).

Let

$$\begin{aligned} J_{\lambda }(u)={}&\frac{1}{2} \Vert u \Vert ^{2}- \int _{\mathbb{R}^{N}}H_{\lambda }(x,u)\,dx \\ ={}&\frac{1}{2}\cdot \frac{C_{N,s}}{2}[u]_{A}^{2}+ \frac{1}{2} \int _{ \mathbb{R}^{N}}V(x) \vert u \vert ^{2}\,dx -\lambda \int _{\mathbb{R}^{N}}\varPhi (u)\,dx- \frac{1}{2} \int _{\mathbb{R}^{N}}F\bigl(x, \vert u \vert ^{2}\bigr)\,dx. \end{aligned}$$

By \((h_{1})\)\((h_{3})\), \((V)\), and the mountain pass theorem, using a standard argument, we easily see that the equation

$$ (-\Delta )_{A}^{s}u+V(x)u=h_{\lambda }(x,u) $$

has a nontrivial solution \(u_{\lambda }\in E\) with \(J_{\lambda }^{\prime }(u_{\lambda })=0\) and \(J_{\lambda }(u_{\lambda })=c_{\lambda }:=\inf_{\gamma \in \varGamma _{\lambda }}\sup_{t \in [0,1]}J_{\lambda }(\gamma (t))\), where

$$ \varGamma _{\lambda }:= \bigl\{ \gamma \in C\bigl([0,1], E\bigr):\gamma (0)=0, J_{\lambda }\bigl(\gamma (1)\bigr)< 0 \bigr\} . $$

We further set

$$\begin{aligned} &J(u)=\frac{1}{2} \Vert u \Vert ^{2}-\frac{1}{2} \int _{\mathbb{R}^{N}}F\bigl(x, \vert u \vert ^{2}\bigr)\,dx \\ &\phantom{J(u)}=\frac{1}{2}\cdot \frac{C_{N,s}}{2}[u]_{A}^{2}+ \frac{1}{2} \int _{ \mathbb{R}^{N}}V(x) \vert u \vert ^{2}\,dx - \frac{1}{2} \int _{\mathbb{R}^{N}}F\bigl(x, \vert u \vert ^{2}\bigr)\,dx, \\ &\varGamma:= \bigl\{ \gamma \in C\bigl([0,1], E\bigr):\gamma (0)=0, J\bigl(\gamma (1) \bigr)< 0 \bigr\} \end{aligned}$$

and

$$ c:=\inf_{\gamma \in \varGamma }\sup_{t \in [0,1]}J\bigl( \gamma (t) \bigr). $$

Then \(\varGamma \subset \varGamma _{\lambda }\) and \(c_{\lambda }\leq c\).

Lemma 2.1

The solution\(u_{\lambda }\)satisfies\(\| u_{\lambda }\|^{2}\leq \frac{2q}{q-2}c_{\lambda }\), and there exists a constant\(A>0\)independent onλsuch that\(\| u_{\lambda }\|^{2}\leq A\).

Proof

By \((h_{2})\) we know that

$$\begin{aligned} qc_{\lambda }={}&qJ_{\lambda }(u_{\lambda })=qJ_{\lambda }(u_{\lambda })- \bigl\langle J_{\lambda }^{\prime }(u_{\lambda }),u_{\lambda } \bigr\rangle \\ ={}& \biggl(\frac{q}{2}-1 \biggr)\cdot \frac{C_{N,s}}{2}[u_{\lambda }]_{A}^{2}+ \biggl(\frac{q}{2}-1 \biggr) \int _{\mathbb{R}^{N}}V(x) \vert u_{\lambda } \vert ^{2} \,dx \\ &{}+ \int _{\mathbb{R}^{N}}\bigl[h_{\lambda }(x,u_{\lambda })u_{\lambda }-qH_{\lambda }(x,u_{\lambda }) \bigr]\,dx \\ \geq {}& \biggl(\frac{q}{2}-1 \biggr) \Vert u_{\lambda } \Vert ^{2}, \end{aligned}$$

which means that \(\| u_{\lambda }\|^{2}\leq \frac{2q}{q-2}c_{\lambda }\leq \frac{2q}{q-2}c:=A>0\). This completes the proof. □

Lemma 2.2

There exist two constants\(B, D>0\)independent onλsuch that\(\Vert \vert u_{\lambda } \vert \Vert _{\infty }\leq B(1+\lambda )^{D}\).

Proof

For any \(L>0\) and \(\beta >1\), set \(\gamma (a)=aa_{L}^{2(\beta -1)}, a \in \mathbb{R}\), where \(a_{L}:=\min \{|a|,L\}\). Since γ is an increasing function, we have

$$ (a-b)\bigl[\gamma (a)-\gamma (b)\bigr]\geq 0,\quad \forall a, b \in \mathbb{R}. $$

Let \(\varPhi (t)=\frac{|t|^{2}}{2}\) and \(\varGamma (t)=\int _{0}^{t} (\gamma ^{\prime }(\tau ) )^{ \frac{1}{2}}\,d\tau \) for \(t\geq 0\). Then if \(a>b\), then we have

$$\begin{aligned} \varPhi ^{\prime }(a-b)\bigl[\gamma (a)-\gamma (b)\bigr]={}&(a-b)\bigl[\gamma (a)- \gamma (b)\bigr]=(a-b) \int _{b}^{a}\gamma ^{\prime }(t)\,dt \\ ={}&(a-b) \int _{b}^{a} \bigl(\varGamma ^{\prime }(t) \bigr)^{2}\,dt\geq \biggl( \int _{b}^{a} \varGamma ^{\prime }(t)\,dt \biggr)^{2} \\ ={}& \bigl\vert \varGamma (a)-\varGamma (b) \bigr\vert ^{2}. \end{aligned}$$

If \(a\leq b\), then we can use a similar argument to obtain the conclusion. It follows that

$$ (a-b)\bigl[\gamma (a)-\gamma (b)\bigr]\geq \bigl\vert \varGamma (a)-\varGamma (b) \bigr\vert ^{2} $$

for all \(a, b \in \mathbb{R}\), which implies that

$$\begin{aligned} &\bigl\vert \varGamma \bigl( \bigl\vert u_{\lambda }(x) \bigr\vert \bigr)- \varGamma \bigl( \bigl\vert u_{\lambda }(y) \bigr\vert \bigr) \bigr\vert ^{2} \\ &\quad\leq \bigl[ \bigl\vert u_{\lambda }(x) \bigr\vert - \bigl\vert u_{\lambda }(y) \bigr\vert \bigr]\cdot \bigl[\bigl( \vert u_{\lambda } \vert u_{ \lambda,L}^{2(\beta -1)}\bigr) (x)-\bigl( \vert u_{\lambda } \vert u_{\lambda,L}^{2(\beta -1)}\bigr) (y) \bigr]. \end{aligned}$$
(2.1)

Choosing \(u_{\lambda }u_{\lambda,L}^{2(\beta -1)}\) as a test function, where \(u_{\lambda,L}:=\min \{|u_{\lambda }|,L\}\), we obtain

$$\begin{aligned} &\mathcal{R} \biggl[ \int _{\mathbb{R}^{N}}\bigl[f\bigl(x, \vert u_{\lambda } \vert ^{2}\bigr)u_{\lambda }+\lambda \phi (u_{\lambda })\bigr] \overline{u_{\lambda }u_{\lambda,L}^{2(\beta -1)}}\,dx \biggr] \\ &\quad=\frac{C_{N,s}}{2}\mathcal{R}\\ &\qquad{}\times \int _{\mathbb{R}^{2N}} \frac{[u_{\lambda }(x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}u_{\lambda }(y)]\cdot \overline{[(u_{\lambda }u_{\lambda,L}^{2(\beta -1)})(x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}(u_{\lambda }u_{\lambda,L}^{2(\beta -1)})(y)]}}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad{}+ \int _{\mathbb{R}^{N}}V(x) \vert u_{\lambda } \vert ^{2}u_{\lambda,L}^{2(\beta -1)}\,dx. \end{aligned}$$

Note that

$$\begin{aligned} &\bigl[u_{\lambda }(x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}u_{\lambda }(y)\bigr] \cdot \overline{\bigl[\bigl(u_{\lambda }u_{\lambda,L}^{2(\beta -1)}\bigr) (x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}\bigl(u_{\lambda }u_{\lambda,L}^{2(\beta -1)}\bigr) (y)\bigr]} \\ &\quad =\bigl[u_{\lambda }(x)-e^{i(x-y)\cdot A(\frac{x+y}{2})}u_{\lambda }(y)\bigr]\cdot \bigl[ \overline{u_{\lambda }(x)}u_{\lambda,L}^{2(\beta -1)}(x)-e^{-i(x-y) \cdot A(\frac{x+y}{2})} \overline{u_{\lambda }(y)}u_{\lambda,L}^{2( \beta -1)}(y)\bigr] \\ &\quad = \bigl\vert u_{\lambda }(x) \bigr\vert ^{2}u_{\lambda,L}^{2(\beta -1)}(x)-u_{\lambda }(x) \overline{u_{\lambda }(y)}e^{-i(x-y)\cdot A(\frac{x+y}{2})}u_{\lambda,L}^{2( \beta -1)}(y) \\ &\qquad{}-u_{\lambda }(y)\overline{u_{\lambda }(x)}e^{i(x-y)\cdot A(\frac{x+y}{2})}u_{ \lambda,L}^{2(\beta -1)}(x)+ \bigl\vert u_{\lambda }(y) \bigr\vert ^{2}u_{\lambda,L}^{2( \beta -1)}(y) \\ &\quad\geq \bigl\vert u_{\lambda }(x) \bigr\vert ^{2}u_{\lambda,L}^{2(\beta -1)}(x)- \bigl\vert u_{\lambda }(x) \bigr\vert \bigl\vert u_{\lambda }(y) \bigr\vert u_{\lambda,L}^{2(\beta -1)}(y)- \bigl\vert u_{\lambda }(y) \bigr\vert \bigl\vert u_{\lambda }(x) \bigr\vert u_{\lambda,L}^{2(\beta -1)}(x) \\ &\qquad{}+ \bigl\vert u_{\lambda }(y) \bigr\vert ^{2}u_{\lambda,L}^{2(\beta -1)}(y) \\ &\quad = \bigl[ \bigl\vert u_{\lambda }(x) \bigr\vert - \bigl\vert u_{\lambda }(y) \bigr\vert \bigr]\cdot \bigl[ \bigl\vert u_{\lambda }(x) \bigr\vert u_{ \lambda,L}^{2(\beta -1)}(x)- \bigl\vert u_{\lambda }(y) \bigr\vert u_{\lambda,L}^{2(\beta -1)}(y) \bigr]. \end{aligned}$$

Consequently, by (2.1) we have

$$\begin{aligned} &\mathcal{R} \biggl[ \int _{\mathbb{R}^{N}}\bigl[f\bigl(x, \vert u_{\lambda } \vert ^{2}\bigr)u_{\lambda }+\lambda \phi (u_{\lambda })\bigr] \overline{u_{\lambda }u_{\lambda,L}^{2(\beta -1)}}\,dx \biggr] \\ &\quad \geq \frac{C_{N,s}}{2} \int _{\mathbb{R}^{2N}} \frac{[ \vert u_{\lambda }(x) \vert - \vert u_{\lambda }(y) \vert ]\cdot [ \vert u_{\lambda }(x) \vert u_{\lambda,L}^{2(\beta -1)}(x)- \vert u_{\lambda }(y) \vert u_{\lambda,L}^{2(\beta -1)}(y)]}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad{}+ \int _{\mathbb{R}^{N}}V(x) \vert u_{\lambda } \vert ^{2}u_{\lambda,L}^{2(\beta -1)}\,dx \\ &\quad \geq \frac{C_{N,s}}{2} \int _{\mathbb{R}^{2N}} \frac{ \vert \varGamma ( \vert u_{\lambda }(x) \vert )-\varGamma ( \vert u_{\lambda }(y) \vert ) \vert ^{2}}{ \vert x-y \vert ^{N+2s}}\,dx\,dy + \int _{\mathbb{R}^{N}}V(x) \vert u_{\lambda } \vert ^{2}u_{\lambda,L}^{2(\beta -1)}\,dx. \end{aligned}$$
(2.2)

For any \(\varepsilon >0\), by \((f_{1})\)\((f_{2})\) and properties of ϕ, there exists \(C_{\varepsilon }>0\) such that

$$ \bigl\vert f\bigl(x, \vert t \vert ^{2}\bigr) \bigr\vert \leq \varepsilon +C_{\varepsilon } \vert t \vert ^{q-2} $$

and

$$ \bigl\vert \phi (t) \bigr\vert \leq \varepsilon \vert t \vert +C_{\varepsilon } \vert t \vert ^{q-1} $$

for all \((x,t) \in \mathbb{R}^{N} \times \mathbb{R}\). Thereby, for fixed \(\lambda >0\) and small \(\varepsilon >0\), we have

$$ \bigl\vert f\bigl(x, \vert t \vert ^{2}\bigr)t+\lambda \phi (t) \bigr\vert \leq V_{0} \vert t \vert +(1+\lambda )C \vert t \vert ^{q-1} $$
(2.3)

for all \((x,t) \in \mathbb{R}^{N} \times \mathbb{R}\). Simultaneously, \(\varGamma (|u_{\lambda }|)\geq \frac{1}{\beta }|u_{\lambda }| u_{\lambda,L}^{ \beta -1}\), and

$$ \frac{C_{N,s}}{2}\bigl[\varGamma \bigl( \vert u_{\lambda } \vert \bigr) \bigr]_{H^{s}(\mathbb{R}^{N})}^{2}= \bigl\Vert \varGamma \bigl( \vert u_{\lambda } \vert \bigr) \bigr\Vert _{\mathcal{D}^{s,2}(\mathbb{R}^{N})}^{2} \geq S \bigl\Vert \varGamma \bigl( \vert u_{\lambda } \vert \bigr) \bigr\Vert _{2_{s}^{*}}^{2}\geq \frac{1}{\beta ^{2}}S \bigl\Vert \vert u_{\lambda } \vert u_{\lambda,L}^{\beta -1} \bigr\Vert _{2_{s}^{*}}^{2}. $$
(2.4)

Therefore, taking into account (2.2)–(2.4) and condition \((V)\), we can see that

$$\begin{aligned} &\frac{1}{\beta ^{2}}S \bigl\Vert \vert u_{\lambda } \vert u_{\lambda,L}^{\beta -1} \bigr\Vert _{2_{s}^{*}}^{2}\\ &\quad \leq \frac{C_{N,s}}{2}\bigl[\varGamma \bigl( \vert u_{\lambda } \vert \bigr) \bigr]_{H^{s}( \mathbb{R}^{N})}^{2} \\ &\quad \leq \mathcal{R} \int _{\mathbb{R}^{N}}\bigl[f\bigl(x, \vert u_{\lambda } \vert ^{2}\bigr)u_{\lambda }+\lambda \phi (u_{\lambda })\bigr] \overline{u_{\lambda }u_{\lambda,L}^{2(\beta -1)}}\,dx- \int _{\mathbb{R}^{N}}V(x) \vert u_{\lambda } \vert ^{2} u_{\lambda,L}^{2(\beta -1)}\,dx \\ &\quad \leq \int _{\mathbb{R}^{N}}V_{0} \vert u_{\lambda } \vert ^{2} u_{\lambda,L}^{2( \beta -1)}\,dx+(1+\lambda )C \int _{\mathbb{R}^{N}} \vert u_{\lambda } \vert ^{q} u_{ \lambda,L}^{2(\beta -1)}\,dx- \int _{\mathbb{R}^{N}}V(x) \vert u_{\lambda } \vert ^{2} u_{\lambda,L}^{2(\beta -1)}\,dx \\ &\quad \leq C(1+\lambda ) \int _{\mathbb{R}^{N}} \vert u_{\lambda } \vert ^{q} u_{ \lambda,L}^{2(\beta -1)}\,dx, \end{aligned}$$

which implies that

$$ \bigl\Vert \vert u_{\lambda } \vert u_{\lambda,L}^{\beta -1} \bigr\Vert _{2_{s}^{*}}^{2}\leq C(1+ \lambda )\beta ^{2} \int _{\mathbb{R}^{N}} \vert u_{\lambda } \vert ^{q} u_{\lambda,L}^{2( \beta -1)}\,dx. $$

Setting \(w_{\lambda,L}=|u_{\lambda }| u_{\lambda,L}^{\beta -1}\), by the Hölder inequality we can derive that

$$\begin{aligned} \Vert w_{\lambda,L} \Vert _{2_{s}^{*}}^{2}&\leq C(1+\lambda )\beta ^{2} \int _{\mathbb{R}^{N}} \vert u_{\lambda } \vert ^{q-2} \vert u_{\lambda } \vert ^{2} u_{\lambda,L}^{2( \beta -1)} \,dx \\ &\leq C(1+\lambda )\beta ^{2} \biggl( \int _{\mathbb{R}^{N}} \vert u_{\lambda } \vert ^{2_{s}^{*}}\,dx \biggr)^{\frac{q-2}{2_{s}^{*}}} \cdot \biggl( \int _{\mathbb{R}^{N}} \vert w_{ \lambda,L} \vert ^{\alpha _{s}^{*}}\,dx \biggr)^{\frac{2}{\alpha _{s}^{*}}}, \end{aligned}$$

where \(\alpha _{s}^{*}=\frac{22_{s}^{*}}{2_{s}^{*}-(q-2)} \in (2,2_{s}^{*})\).

By Lemma 2.1 we have

$$ \Vert w_{\lambda,L} \Vert _{2_{s}^{*}}^{2}\leq C(1+\lambda ) \beta ^{2} \Vert w_{ \lambda,L} \Vert _{\alpha _{s}^{*}}^{2}. $$
(2.5)

Now we observe that if \(|u_{\lambda }|^{\beta }\in L^{\alpha _{s}^{*}}(\mathbb{R}^{N})\), then from the definition of \(\{u_{\lambda, L}\}\), the inequality \(u_{\lambda,L}\leq |u_{\lambda }|\), and (2.5) we obtain

$$ \Vert w_{\lambda,L} \Vert _{2_{s}^{*}}^{2}\leq C(1+\lambda ) \beta ^{2} \biggl( \int _{\mathbb{R}^{N}} \vert u_{\lambda } \vert ^{\beta \alpha _{s}^{*}}\,dx \biggr)^{ \frac{2}{\alpha _{s}^{*}}}< +\infty. $$
(2.6)

Passing to the limit in (2.6) as \(L\rightarrow +\infty \), by the Fatou lemma we deduce that

$$ \bigl\Vert \vert u_{\lambda } \vert \bigr\Vert _{\beta 2_{s}^{*}}\leq C^{\frac{1}{\beta }}(\sqrt{1+ \lambda })^{\frac{1}{\beta }}\beta ^{\frac{1}{\beta }} \bigl\Vert \vert u_{\lambda }\vert \bigr\Vert _{\beta \alpha _{s}^{*}} $$
(2.7)

whenever \(|u_{\lambda }|^{\beta \alpha _{s}^{*}} \in L^{1}(\mathbb{R}^{N})\).

Now set \(\beta:=\frac{2_{s}^{*}}{\alpha _{s}^{*}}>1\). Since \(|u_{\lambda }| \in L^{2_{s}^{*}}(\mathbb{R}^{N})\), the inequality holds for this choice of β. Then, since \(\beta ^{2}\alpha _{s}^{*}=\beta 2_{s}^{*}\), it follows that (2.7) holds with β replaced by \(\beta ^{2}\). Consequently,

$$\begin{aligned} \bigl\Vert \vert u_{\lambda } \vert \bigr\Vert _{\beta ^{2}2_{s}^{*}}&\leq C^{ \frac{1}{\beta ^{2}}}(\sqrt{1+\lambda })^{\frac{1}{\beta ^{2}}}\beta ^{ \frac{2}{\beta ^{2}}} \bigl\Vert \vert u_{\lambda } \vert \bigr\Vert _{\beta ^{2}\alpha _{s}^{*}} \\ &= C^{\frac{1}{\beta ^{2}}}(\sqrt{1+\lambda })^{\frac{1}{\beta ^{2}}} \beta ^{\frac{2}{\beta ^{2}}} \bigl\Vert \vert u_{\lambda } \vert \bigr\Vert _{\beta 2_{s}^{*}} \\ &\leq C^{\frac{1}{\beta ^{2}}}(\sqrt{1+\lambda })^{ \frac{1}{\beta ^{2}}}\beta ^{\frac{2}{\beta ^{2}}}C^{\frac{1}{\beta }}( \sqrt{1+\lambda })^{\frac{1}{\beta }}\beta ^{\frac{1}{\beta }} \bigl\Vert \vert u_{\lambda } \vert \bigr\Vert _{\beta \alpha _{s}^{*}} \\ &=C^{\frac{1}{\beta }+\frac{1}{\beta ^{2}}}(\sqrt{1+\lambda })^{ \frac{1}{\beta }+\frac{1}{\beta ^{2}}}\beta ^{\frac{1}{\beta }+ \frac{2}{\beta ^{2}}} \bigl\Vert \vert u_{\lambda } \vert \bigr\Vert _{\beta \alpha _{s}^{*}}. \end{aligned}$$

Iterating this process and recalling that \(\beta \alpha _{s}^{*}=2_{s}^{*}\), we conclude that for every \(m \in \mathbb{N}\),

$$ \bigl\Vert \vert u_{\lambda } \vert \bigr\Vert _{\beta ^{m}2_{s}^{*}}\leq C^{\sum _{i=1}^{m} \frac{1}{\beta ^{i}}}(\sqrt{1+\lambda })^{\sum _{i=1}^{m} \frac{1}{\beta ^{i}}}\beta ^{\sum _{i=1}^{m} \frac{i}{\beta ^{i}}} \bigl\Vert \vert u_{\lambda } \vert \bigr\Vert _{2_{s}^{*}}. $$
(2.8)

Set \(d_{m}=\sum_{i=1}^{m}\frac{1}{\beta ^{i}}\) and \(e_{m}=\sum_{i=1}^{m}\frac{i}{\beta ^{i}}\). Then \(d_{m}\rightarrow \sigma _{1}>0\) and \(e_{m}\rightarrow \sigma _{2}>0\) as \(m\rightarrow \infty \). Then, taking the limit in (2.8) as \(m\rightarrow +\infty \), by Lemma 2.1 we have

$$ \bigl\Vert \vert u_{\lambda } \vert \bigr\Vert _{L^{\infty }}\leq C^{\sigma _{1}}(\sqrt{1+\lambda })^{ \sigma _{1}}\beta ^{\sigma _{2}}C:=B(1+ \lambda )^{D}, $$

where \(B:=C^{\sigma _{1}}\beta ^{\sigma _{2}}C>0\) and \(D:=\frac{\sigma _{1}}{2}\). This completes the proof. □

Proof of Theorem 1.1

By Lemma 2.2, for large \(M>0\), we can choose small \(\lambda _{0}>0\) such that \(\Vert \vert u_{\lambda } \vert \Vert _{L^{\infty }}\leq B(1+\lambda )^{D}\leq M\) for all \(\lambda \in (0,\lambda _{0}]\). Consequently, \(u_{\lambda }\) is a nontrivial solution of (1.1) with \(\lambda \in (0,\lambda _{0}]\). This completes the proof. □