1 Introduction and main result

In this paper, we consider the existence of the solutions for the following problem:

$$ \textstyle\begin{cases} (-\Delta)^{s}u-\mu \frac{u}{\vert z\vert ^{2s}}= \frac{\vert u\vert ^{2_{s}^{*}(\alpha)-2}u}{\vert z\vert ^{\alpha }}+\lambda f(x,u), & \mbox{in }\Omega, \\ u>0, & \mbox{in }\Omega, \\ u=0, & \mbox{in }\mathbb{R}^{N}\backslash \Omega, \end{cases} $$
(1)

where Ω is a smooth bounded domain in \(\mathbb{R}^{N}= \mathbb{R}^{k}\times \mathbb{R}^{N-k}\) with \(N\geq 3\) and \(2\leq k< N\), \(0\leq \mu < a_{k,s}:=2^{2s}\Gamma^{2}(\frac{k+2s}{4})/\Gamma^{2}( \frac{k-2s}{4})\), \(s\in (0,1)\), \(\Gamma (t)=\int_{0}^{+\infty }\tau ^{t-1}e^{-\tau }\,d\tau \). A point \(x\in \mathbb{R}^{N}\) is denoted as \(x=(z,w)\in \mathbb{R}^{N}=\mathbb{R}^{k}\times \mathbb{R}^{N-k}\). \(\lambda >0\) is a real parameter, the number \(2_{s}^{*}(\alpha)=2(N-\alpha)/(N-2s)\) is a critical Hardy-Sobolev exponent with \(s\in (0,1)\) and \(\alpha \in [0,2s)\). The nonlinearity term f is continuous function and satisfies suitable hypotheses. Here, \((-\Delta)^{s}\) is the fractional Laplace operator (see [1, 2]) defined, up to a normalization factor, by

$$\begin{aligned}& (-\Delta)^{s}u(x):= \int_{\mathbb{R}^{N}} \frac{u(x)-u(y)}{\vert x-y\vert ^{N+2s}}\,dy, \quad x\in \mathbb{R}^{N}. \end{aligned}$$

In recent years, much attention has been focused on the study of the problems involving fractional operators. The fractional operators appear in several applications to some models related to probability, mathematical, finances or fluid mechanics, soft thin films, stratified materials, multiple scattering and minimal surfaces (see [3–6]). When \(\mu =0\) and \(\alpha =0\), problem (1) reduces to critical fractional equation. Abundant results have been accumulated (see [7–12]).

For a class of fractional elliptic problems with the Hardy potential

$$ \textstyle\begin{cases} (-\Delta)^{s}u-\mu \frac{u}{\vert x\vert ^{2s}}= g(x,u), & \mbox{in }\Omega, \\ u>0, & \mbox{in }\Omega, \\ u=0, & \mbox{in }\mathbb{R}^{N}\backslash \Omega, \end{cases} $$
(2)

Abdellaoui and Medina et al. in [13] gave the solvability of the problem (2) for the linear case \(g(x,t)=g(x)\) and the nonlinear case \(g(x,t)=\frac{h(x)}{t^{\sigma }}\), respectively. For critical case, a positive solution was obtained in [14] with by the Lagrange multipliers technique. Moreover, the authors in [15] have studied the solvability of problem (2) for the case \(g(x,t)\) involving concave-convex nonlinearities.

Recently, Jiang and Tang in [16] had considered the problem (1) for the case \(s=1\), they supposed the nonlinearity term \(f\in C(\overline{\Omega }\times \mathbb{R}^{+},\mathbb{R}^{+})\) satisfies the following conditions:

\((f_{1})\) :

\(f(x,t)=0\) for \(t\leq 0\) uniformly for \(x\in \overline{ \Omega }\). There exists a nonempty open subset \(\Omega_{0}\subset \Omega \) with \((0,w^{0})\in \mathbb{R}^{k}\times \mathbb{R}^{N-k} \in \Omega_{0}\), such that \(f(x,t)\geq 0\) for almost everywhere \(x\in \Omega \) and all \(t>0\); \(f(x,t)>0\) for almost \(x\in \Omega_{0}\) and all \(t>0\).

\((f_{2})\) :

\(\lim_{t\rightarrow 0^{+}}\frac{f(x,t)}{t}=0\) and \(\lim_{t\rightarrow +\infty }\frac{f(x,t)}{t^{2_{s}^{*}(\alpha)-1}}=0\) uniformly for \(x\in \overline{\Omega }\).

For \(\lambda >0\) large enough, they obtained the existence of positive solutions of problem (1) for \(s=1\) by using variational methods. For the case \(s=1\) and \(\lambda =1\), Ding and Tang in [17] obtained the existence of positive solutions for problem (1) by the variational methods and some analysis techniques with f satisfying the (AR) condition. For related papers on the semilinear elliptic equations with Hardy-Sobolev critical exponents of (1) for \(s=1\), we just mention [18, 19] and the references therein.

To the best of our knowledge, there is no result in the literature on the fractional elliptic problem with Hardy-Sobolev-Maz’ya potential and critical nonlinearities. Motivated by the above papers, our aim is to study the existence of positive solutions for problem (1) and our main result of this paper is as follows.

Theorem 1

Assume that conditions \((f_{1})\) and \((f_{2})\) hold. Then there exists \(\lambda^{*}>0\) such that \(\lambda \geq \lambda^{*}\), problem (1) admits a positive solution.

2 Functional setting and useful tools

We will denote by \(H^{s}(\mathbb{R}^{N})\) the usual fractional Sobolev space endowed with the natural norm

$$\Vert u\Vert _{ H^{s}(\mathbb{R}^{N})}=\Vert u\Vert _{L^{2}(\mathbb{R}^{N})}+ \biggl( \iint _{\mathbb{R}^{2N}}\frac{\vert u(x)-u(y)\vert ^{2}}{\vert x-y\vert ^{N+2s}}\,dx\,dy \biggr) ^{ \frac{1}{2}}. $$

We consider the function space

$$X_{0}^{s}= \bigl\{ u\in H^{s} \bigl( \mathbb{R}^{N} \bigr):u=0 \mbox{ a.e. in }\mathbb{R}^{N} \backslash \Omega \bigr\} , $$

with the norm

$$\Vert u\Vert _{X_{0}^{s}}= \biggl( \int_{Q}\frac{\vert u(x)-u(y)\vert ^{2}}{\vert x-y\vert ^{N+2s}}\,dx\,dy-\mu \int_{ \Omega }\frac{u^{2}}{\vert z\vert ^{2s}}\,dx \biggr) ^{\frac{1}{2}}, $$

which is equivalent to its general norm due to the Hardy inequality

$$ a_{k,s} \int_{\mathbb{R}^{N}}\frac{u^{2}}{\vert z\vert ^{2s}}\leq \int_{Q} \frac{\vert u(x)-u(y)\vert ^{2}}{\vert x-y\vert ^{N+2s}}\,dx\,dy, $$
(3)

where \(Q=\mathbb{R}^{2N}\setminus (C\Omega \times C\Omega)\) with \(C\Omega =\mathbb{R}^{N}\setminus \Omega \). We can introduce the best fractional critical Hardy-Sobolev constant \(S_{\mu,\alpha }\), given by

$$ S_{\mu,\alpha } = \inf_{u\in X_{0}^{s}({\mathbb{R}^{N}}\setminus (0,w^{0})),u\neq 0}\frac{ \int_{Q}\frac{\vert u(x)-u(y)\vert ^{2}}{\vert x-y\vert ^{N+2s}}\,dx\,dy- \mu \int_{\Omega }\frac{u ^{2}}{\vert z\vert ^{2s}}\,dx}{ ( \int_{\mathbb{R}^{n}}\frac{\vert u(x)\vert ^{2_{s}^{*}( \alpha)}}{\vert z\vert ^{\alpha }}\,dx) ^{2/ 2_{s}^{*}(\alpha)}}. $$
(4)

From [20], we know that \(S_{\mu,\alpha }\) is attained by functions

$$U_{\varepsilon }(x)= \frac{\varepsilon^{\frac{(N-2s)}{2}}}{(\varepsilon^{2}+\vert x-x_{0}\vert ^{2})^{ \frac{(N-2s)}{2}}}. $$

Let \(u^{+}=\max \{u,0\}\), the energy functional \(J_{\lambda }:X_{0} ^{s}\rightarrow \mathbb{R}\) associated to the problem (1) is defined as

$$\begin{aligned} J_{\lambda }(u) =& \frac{1}{2} \int_{Q}\frac{\vert u(x)-u(y)\vert ^{2}}{\vert x-y\vert ^{N+2s}}\,dx\,dy- \frac{ \mu }{2} \int_{\Omega }\frac{(u^{+})^{2}}{\vert z\vert ^{2s}}\,dx \\ &{}- \frac{1}{2_{s}^{*}(\alpha)} \int_{\Omega }\frac{(u^{+})^{2_{s}^{*}( \alpha)}}{\vert z\vert ^{\alpha }}\,dx -\lambda \int_{\Omega }F \bigl(x,u^{+} \bigr)\,dx, \end{aligned}$$
(5)

for all \(u\in X_{0}^{s}\), where \(F(x,t)\) is a primitive function of \(f(x,t)\) defined by \(F(x,t)=\int_{0}^{t}f(x,\tau)\,d\tau \). Obviously, \(J_{\lambda }\) is a \(C^{1}(X_{0}^{s})\) functional, and it is well known that the solutions of problem (1) are the critical points of the energy functional \(J_{\lambda }\). In fact, if u is a weak solution of problem (1), we have

$$\begin{aligned} \bigl\langle J'_{\lambda }(u),\varphi \bigr\rangle =& \int_{Q}\frac{(u(x)-u(y))(\varphi (x)-\varphi (y))}{\vert x-y\vert ^{N+2s}}\,dx\,dy -\mu \int_{\Omega }\frac{u^{+}\varphi }{\vert z\vert ^{2s}}\,dx \\ &{}- \int_{\Omega }\frac{(u^{+})^{2_{s}^{*}(\alpha)-1}\varphi }{\vert z\vert ^{ \alpha }}\,dx-\lambda \int_{\Omega } f \bigl(x,u^{+} \bigr)\varphi \,dx \\ =&0, \end{aligned}$$
(6)

for all \(\varphi \in X_{0}^{s}\). Now, we will give some essential lemmas as follows.

Lemma 1

Let \(\lambda >0\) and f satisfies assumptions \((f_{1})\) and \((f_{2})\). We can deduce that:

  1. (i)

    there exist \(\varsigma,\rho >0\) such that \(J_{\lambda }(u)\geq \varsigma >0\) for any \(u\in X_{0}^{s}\), with \(\Vert u\Vert _{X_{0} ^{s}}=\rho \);

  2. (ii)

    there exists \(e\in X_{0}^{s}\), with \(e\geq 0\) in \(\mathbb{R}^{N}\) such that \(J_{\lambda }(e)<0\) and \(\Vert e\Vert _{X_{0}^{s}} > \rho \).

Proof of Lemma 1

(i) Fixing \(\lambda >0\), from \((f_{2})\), for \(\varepsilon >0\), there exists \(C_{1}>0\), one has

$$ \bigl\vert F(x,t) \bigr\vert \leq \varepsilon \vert t\vert ^{2}+C_{1} \vert t\vert ^{2_{s}^{*}(\alpha)},\quad \forall (x,t)\in \Omega \times \mathbb{R}^{+}. $$
(7)

It is evident that \(X_{0}^{s}\hookrightarrow L^{q}(\Omega)\) for \(1\leq q\leq 2_{s}^{*}(\alpha)\), then there exists \(C_{2}>0\) such that

$$ \int_{\Omega }\vert u\vert ^{q}\,dx\leq C_{2} \Vert u\Vert _{X_{0}^{s}}^{q}. $$
(8)

Take \(u\in X_{0}^{s}\). Combining (4), (5), (7) and (8), we have

$$\begin{aligned} J_{\lambda }(u) =& \frac{1}{2}\Vert u\Vert _{X_{0}^{s}}^{2}- \frac{1}{2^{*}(s)} \int_{\Omega }\frac{\vert u\vert ^{2_{s} ^{*}(\alpha)}}{\vert z\vert ^{\alpha }}\,dx -\lambda \int_{\Omega }F(x,u)\,dx \\ \geq & \frac{1}{2}\Vert u\Vert _{X_{0}^{s}}^{2}- \frac{1}{2_{s}^{*}(\alpha)} \int_{\Omega }\frac{\vert u\vert ^{2_{s}^{*}(\alpha)}}{\vert x\vert ^{\alpha }}\,dx- \lambda \int_{\Omega } \bigl(\varepsilon \vert u\vert ^{2}+C_{1}\vert u\vert ^{2_{s}^{*}(\alpha)} \bigr)\,dx \\ \geq & \frac{1}{2}\Vert u\Vert _{X_{0}^{s}}^{2}- \frac{C_{3}}{2_{s}^{*}(\alpha)}\Vert u \Vert _{X_{0}^{s}}^{2_{s}^{*}(\alpha)}-\lambda \varepsilon C_{4} \Vert u\Vert _{X_{0}^{s}}^{2}- \lambda C_{5} \Vert u\Vert _{X_{0}^{s}}^{2_{s}^{*}(\alpha)}, \end{aligned}$$

where \(C_{i}\), \(i=3,4,5\), are positive constants. For \(\varepsilon >0\) small and according to the fact \(2< 2^{*}(s)\), then there exists \(\rho >0\) small enough such that \(J_{\lambda }(u)\geq \varsigma >0\), for any \(\Vert u\Vert _{X_{0}^{s}}=\rho \).

(ii) Given \(\lambda >0\). Take \(v \in X_{0}^{s}\), with \(v\geq 0\) in \(\mathbb{R}^{N}\) and \(\Vert v\Vert _{X_{0}^{s}}=1\). From \((f_{1})\), we get

$$\begin{aligned} J_{\lambda }(tv) =& \frac{1}{2}\Vert tv\Vert _{X_{0}^{s}}^{2}- \frac{1}{2_{s}^{*}(\alpha)} \int_{\Omega }\frac{\vert tv\vert ^{2_{s}^{*}(\alpha)}}{\vert x\vert ^{\alpha }}\,dx- \lambda \int_{\Omega }F(x,tv)\,dx \\ \leq& \frac{t^{2}}{2}\Vert v\Vert _{X_{0}^{s}}^{2}- \frac{t^{2_{s}^{*}(\alpha)}}{2_{s}^{*}(\alpha)} \int_{\Omega }\frac{\vert v\vert ^{2_{s} ^{*}(\alpha)}}{\vert x\vert ^{\alpha }}\,dx, \end{aligned}$$

then \(J_{\lambda }(tv)\rightarrow -\infty \) as \(t\rightarrow +\infty \). Choosing \(e=t_{*}v\) with \(t_{*}>0\) large enough, we get \(\Vert e\Vert _{X _{0}^{s}}>\rho \) and \(J_{\lambda }(e)<0\). This completes the proof of the Lemma 1. □

We recall that a sequence \(\{u_{j}\}_{j\in \mathbb{N}} \subset X_{0} ^{s}\) is a Palais-Smale sequence for the functional \(J_{\lambda }\) at level \(c_{\lambda }\) if

$$J_{\lambda }(u_{j})\rightarrow c_{\lambda } \quad \mbox{and} \quad J'_{\lambda }(u_{j})\rightarrow 0\quad \mbox{in } \bigl(X_{0}^{s} \bigr)', $$

as \(j\rightarrow \infty \). We say that \(J_{\lambda }\) satisfies the Palais-Smale condition if every Palais-Smale sequence of \(J_{\lambda }\) has a convergent subsequence in \(X_{0}^{s}\). Now put

$$c_{\lambda }=\inf_{g \in \Gamma } \max_{t\in [0,1]} J_{\lambda } \bigl(g(t) \bigr), $$

where

$$\Gamma = \bigl\{ g \in C \bigl([0,1],X_{0}^{s} \bigr):g(0)=0,J_{\lambda } \bigl(g(1) \bigr)< 0 \bigr\} . $$

Obviously, \(c_{\lambda }>0\) from Lemma 1. Next, we introduce an asymptotic condition for the level \(c_{\lambda }\).

Lemma 2

Under the conditions of Lemma 1, \(\lim_{\lambda \rightarrow \infty } c_{\lambda }=0\).

Proof of Lemma 2

Fix \(\lambda >0\). Since the functional \(J_{\lambda }\) satisfies the Mountain pass geometry, there exists \(t_{\lambda }>0\) verifying \(J_{\lambda }(t_{\lambda }e)=\max_{t\geq 0}J _{\lambda }(te)\), where \(e \in X_{0}^{s}\) is the function given in Lemma 1. Hence, by (6), we have \(\langle J'_{\lambda }(t_{\lambda }e),t _{\lambda }e\rangle =0\), that is,

$$ t_{\lambda }^{2}\Vert e\Vert _{X_{0}^{s}}^{2}= t_{\lambda }^{2_{s}^{*}(\alpha)} \int_{\Omega }\frac{(e^{+})^{2_{s}^{*}(\alpha)}}{\vert z\vert ^{\alpha }}\,dx+ \lambda \int_{\Omega }f \bigl(x,t_{\lambda }e^{+} \bigr)t_{\lambda }e^{+}\,dx. $$
(9)

From \((f_{1})\), we have

$$t_{\lambda }^{2}\Vert e\Vert _{X_{0}^{s}}^{2}\geq t_{\lambda }^{2_{s}^{*}( \alpha)} \int_{\Omega }\frac{(e^{+})^{2_{s}^{*}(\alpha)}}{\vert z\vert ^{ \alpha }}\,dx, $$

which implies that \(\{t_{\lambda }\}\) is bounded. Hence, there exist a number \(t_{0}\geq 0\) and a subsequence of \(\{\lambda_{j}\}_{j\in \mathbb{N}}\), which we still denote by \(\{\lambda_{j}\}_{j\in \mathbb{N}}\), such that \(\lambda_{j} \rightarrow +\infty \) and \(t_{\lambda_{j}}\rightarrow t_{0}\) as \(j\rightarrow \infty \). So by (9) there exists \(D>0\) such that \(t_{\lambda }^{2}\Vert e\Vert _{X_{0} ^{s}}^{2}\leq D \) for any \(j\in \mathbb{N}\), then

$$ \lambda_{j} \int_{\Omega }f \bigl(x,t_{\lambda_{j}} e^{+} \bigr)t_{\lambda_{j}} e ^{+}\,dx+t_{\lambda_{j}}^{2_{s}^{*}(\alpha)} \int_{\Omega }\frac{(e^{+})^{2_{s} ^{*}(\alpha)}}{\vert z\vert ^{\alpha }}\,dx \leq D. $$
(10)

If \(t_{0}>0\), by \((f_{1})\) and the Lebesgue dominated convergence theorem, we obtain

$$\lim_{j\rightarrow \infty } \int_{\Omega }f \bigl(x,t_{\lambda_{j}} e^{+} \bigr)t _{\lambda_{j}} e^{+}\,dx = \int_{\Omega }f \bigl(x,t_{0} e^{+} \bigr)t_{0}e^{+}\,dx>0. $$

Recalling that \(\lambda_{j}\rightarrow \infty \), we have

$$\lim_{j\rightarrow \infty } \biggl( \lambda_{j} \int_{\Omega }f \bigl(x,t_{ \lambda_{j}}e^{+} \bigr)t_{\lambda_{j}}e^{+}\,dx+t_{\lambda_{j}}^{p_{s}^{*}( \alpha)} \int_{\Omega }\frac{(e^{+})^{2_{s}^{*}(\alpha)}}{\vert z\vert ^{ \alpha }}\,dx \biggr) =\infty, $$

which contradicts (10). Thus \(t_{0}=0\) for \(\lambda_{j}\rightarrow \infty \). Now, let us consider the path \(g(t)=te\), for \(t\in [0,1]\), which belongs to Γ. By Lemma 1 and \((f_{1})\), we get

$$0< c_{\lambda }\leq \max_{t \in [0,1]} J_{\lambda } \bigl(g(t) \bigr)\leq J_{ \lambda }(t_{\lambda }e)\leq \frac{t_{\lambda }^{2}}{2}\Vert e \Vert _{X_{0} ^{s}}^{2}. $$

Notice that \(t_{\lambda_{j}}\rightarrow t_{0}=0\) as \(j\rightarrow \infty \), one has

$$\lim_{\lambda \rightarrow +\infty }\frac{t_{\lambda }^{2}}{2}\Vert e\Vert _{X_{0}^{s}}^{2}=0, $$

which leads to \(\lim_{\lambda \rightarrow \infty } c_{\lambda }=0\). This completes the proof of the Lemma 2. □

Lemma 3

Assume that conditions \((f_{1})\) and \((f_{2})\) hold. If \(\{u_{j}\}\subset X_{0}^{s}\) is a \((PS)_{c_{\lambda }}\) condition of \(J_{\lambda }\), then \(\{u_{j}\}\) is bounded in \(X_{0}^{s}\).

Proof of Lemma 3

By \((f_{2})\) and the boundedness of Ω, for any \(\varepsilon >0\), there exists \(T>0\), such that

$$\begin{aligned}& \bigl\vert F(x,t) \bigr\vert \leq \varepsilon \vert t\vert ^{2_{s}^{*}(\alpha)},\quad x\in \Omega, t\geq T;\qquad \bigl\vert F(x,t) \bigr\vert \leq C_{6}(\varepsilon),\quad t\in (0,T]; \\& \bigl\vert f(x,t)t \bigr\vert \leq \varepsilon \vert t\vert ^{2_{s}^{*}(\alpha)}, \quad x\in \Omega, t\geq T; \qquad \bigl\vert f(x,t)t \bigr\vert \leq C_{7} (\varepsilon),\quad t\in (0,T], \end{aligned}$$

for \(C_{i}(\varepsilon)>0\), \(i=6,7\). Furthermore, for any \((x,t)\in \Omega \times \mathbb{R}^{+}\), we have

$$\bigl\vert F(x,t) \bigr\vert \leq C_{6}(\varepsilon)+\varepsilon \vert t\vert ^{2_{s}^{*}(\alpha)},\qquad \bigl\vert f(x,t)t \bigr\vert \leq C_{7}(\varepsilon)+\varepsilon \vert t\vert ^{2_{s}^{*}(\alpha)}. $$

Then, for \(\xi \in (2,2_{s}^{*}(\alpha))\), one has

$$ F(x,t)-\frac{1}{2}f(x,t)t\leq F(x,t)-\frac{1}{\xi }f(x,t)t \leq C_{8}( \varepsilon)+\varepsilon \vert t\vert ^{2_{s}^{*}(\alpha)}, $$
(11)

for \(C_{8}(\varepsilon)>0\) and any \((x,t)\in \overline{\Omega } \times \mathbb{R}^{+}\). Set \(l(x,t):=\frac{\vert t\vert ^{2_{s}^{*}(\alpha)-1}}{\vert z\vert ^{ \alpha }}+\lambda f(x,t)\), we claim that \(l(x,t)\) satisfies the \(\mathrm{(AR)}\) condition. By (11), one easily gets

$$\begin{aligned} \xi L(x,t)-l(x,y)t =& \biggl(\frac{\xi }{2_{s}^{*}(\alpha)}-1 \biggr)\frac{\vert t\vert ^{2_{s}^{*}( \alpha)}}{\vert z\vert ^{\alpha }}+ \lambda \bigl( \xi F(x,t)-f(x,t)t \bigr) \\ \leq & \biggl(\frac{\xi }{2_{s}^{*}(\alpha)}-1 \biggr)\frac{\vert t\vert ^{2_{s}^{*}( \alpha)}}{\vert z\vert ^{\alpha }}+\lambda \xi C_{8}(\varepsilon)+\lambda \xi \varepsilon \vert t\vert ^{2_{s}^{*}(\alpha)} \\ =& \biggl( \biggl(\frac{\xi }{2_{s}^{*}(\alpha)}-1 \biggr)\vert z\vert ^{-\alpha }+ \lambda \xi \varepsilon \biggr)\vert t\vert ^{2_{s}^{*}(\alpha)}+\lambda \xi C_{8}( \varepsilon), \end{aligned}$$

where \(L(x,t)=\int_{0}^{t}l(x,\tau)\,d\tau \). Thus, for a fixed \(\lambda >0\) and \(\varepsilon >0\) sufficiently small, there exists \(T'_{\lambda }>0\), such that

$$0\leq \xi L(x,t)\leq l(x,t)t, \quad t\geq T'_{\lambda }. $$

Moreover, by \((f_{2})\), we obtain

$$L(x,t)-\frac{1}{\xi }l(x,t)t\leq \max_{x\in \Omega,0\leq t\leq T'_{\lambda }} \biggl( F(x,t)-\frac{1}{ \xi }f(x,t)t \biggr):=T_{\lambda }, $$

for any \(0\leq t\leq T'_{\lambda }\). Notice that \(\xi <2_{s}^{*}( \alpha)\), we obtain \(T_{\lambda }>0\). It follows from the above inequalities that

$$ L(x,t)-\frac{1}{\xi }l(x,t)t\leq T_{\lambda },\quad \mbox{for all } x\in \overline{\Omega }\backslash \bigl\{ \bigl(0,w^{0} \bigr) \bigr\} , t \geq 0. $$
(12)

Combining \((f_{2})\), (6) and (12), it follows that

$$\begin{aligned} c_{\lambda }+1 \geq& J_{\lambda }(u_{j})-\frac{1}{\xi } \bigl\langle J'_{\lambda }(u_{j}),u_{j} \bigr\rangle \\ \geq& \biggl(\frac{1}{2}-\frac{1}{\xi } \biggr)\Vert u_{j}\Vert _{X_{0}^{s}}^{2}+ \biggl( \frac{1}{ \xi }-\frac{1}{2_{s}^{*}(\alpha)} \biggr) \int_{\Omega }\frac{(u_{j}^{+})^{2_{s} ^{*}(\alpha)}}{\vert z\vert ^{\alpha }}\,dx \\ &{}-\lambda \int_{\Omega } \biggl(F \bigl(x,u_{j}^{+} \bigr)-\frac{1}{\xi }f \bigl(x,u_{j} ^{+} \bigr)u_{j}^{+} \biggr)\,dx \\ \geq & \biggl( \frac{1}{2}-\frac{1}{\xi } \biggr) \Vert u_{j}\Vert _{X_{0}^{s}}^{2} - \int_{\Omega } \biggl(L \bigl(x,u_{j}^{+} \bigr)-\frac{1}{\xi }l \bigl(x,u _{j}^{+} \bigr)u_{j}^{+} \biggr)\,dx \\ \geq& \biggl( \frac{1}{2}-\frac{1}{\xi } \biggr) \Vert u_{j}\Vert _{X_{0}^{s}}^{2}-T _{\lambda }\vert \Omega \vert . \end{aligned}$$

Hence, we obtain \(\{u_{j}\}_{j\in \mathbb{N}}\) is bounded in \(X_{0}^{s}\). This completes the proof of Lemma 3. □

Lemma 4

Assume that conditions \((f_{1})\) and \((f_{2})\) hold. Then \(J_{\lambda }\) satisfies the \((PS)_{c_{\lambda }}\) condition with \(c_{\lambda }<\frac{2s-\alpha }{2(N-\alpha)} S_{ \mu,\alpha }^{\frac{N-\alpha }{2s-\alpha }} -T_{\lambda }\vert \Omega \vert \), where \(T_{\lambda }\) is a bounded constant given in Lemma 3.

Proof of Lemma 4

Let \(\{u_{j}\}_{j\in \mathbb{N}}\subset X_{0}^{s}\) be a \((PS)_{c_{\lambda }}\) sequence of \(J_{\lambda }\). From Lemma 3, we know that \(\{u_{j}\}_{j\in \mathbb{N}}\) is bounded. Thus there exist a subsequence (still denoted by \(\{u_{j}\}_{j\in \mathbb{N}}\)) and \(u_{\lambda }\in X_{0}^{s}\) such that

$$\begin{aligned} \textstyle\begin{cases} u_{j}\rightharpoonup u_{\lambda }, & \mbox{weakly in }X_{0}^{s}, \\ u_{j}\rightarrow u_{\lambda }, & \mbox{strongly in }L^{p},2\leq p< 2_{s}^{*}(\alpha), \\ u_{j}\rightharpoonup u_{\lambda }, & \mbox{weakly in }L^{2_{s}^{*}(\alpha)}(\Omega, \vert z\vert ^{-\alpha }), \\ u_{j}\rightarrow u_{\lambda }, & \mbox{a.e. in } \mathbb{R}^{n}. \end{cases}\displaystyle \end{aligned}$$
(13)

Due to the continuity of embedding \(X_{0}^{s}\hookrightarrow H^{s}( \Omega)\hookrightarrow L^{\nu }(\Omega)\) and the boundedness of \(\{u_{j}\}_{j\in \mathbb{N}}\), there exists a constant \(C_{\nu }\) such that \(\Vert u\Vert _{\nu }\leq \Vert u\Vert _{X_{0}^{s}}\leq C_{\nu }\) for all \(u\in X_{0}^{s}\) and \(\nu \in [2,2_{s}^{*}(\alpha)]\). Now, according to \((f_{2})\), for any \(\varepsilon >0\), there exists a \(a(\varepsilon)>0\) such that

$$\bigl\vert F(x,t) \bigr\vert \leq \frac{1}{2C_{\nu }}\varepsilon \vert t \vert ^{2_{s}^{*}(\alpha)}+a( \varepsilon)\quad \mbox{for }(x,t)\in \Omega \times \mathbb{R}^{+}. $$

Set \(\delta =\frac{\varepsilon }{2a(\varepsilon)}>0\). When \(E\subset \Omega \), meas \(E<\delta \), one gets

$$\begin{aligned} \biggl\vert \int_{E}F \bigl(x,u_{j}^{+} \bigr)\,dx \biggr\vert \leq& \int_{E} a(\varepsilon)\,dx+ \frac{1}{2C}\varepsilon \int_{E} \bigl\vert u_{j}^{+} \bigr\vert ^{2_{s}^{*}(\alpha)}\,dx \\ \leq& a(\varepsilon) \operatorname{meas} E+ \frac{1}{2C_{\nu }} \varepsilon C _{\nu }\leq \varepsilon. \end{aligned}$$

Obviously, \(\{\int_{\Omega }F(x,u_{j}^{+})\,dx,j\in N\}\) is equi-absolutely continuous. It follows from Vitali’s convergence theorem that

$$ \int_{\Omega }F \bigl(x,u_{j}^{+} \bigr)\,dx \rightarrow \int_{\Omega }F \bigl(x,u_{\lambda }^{+} \bigr)\,dx, \quad \mbox{as } j\rightarrow \infty. $$
(14)

Similarly, we get

$$ \int_{\Omega }f \bigl(x,u_{j}^{+} \bigr)u_{j}\,dx\rightarrow \int_{\Omega }f \bigl(x,u _{\lambda }^{+} \bigr)u_{\lambda }\,dx,\quad \mbox{as } j\rightarrow \infty. $$
(15)

By (13) and (15) we obtain

$$\begin{aligned}& \lim_{j\rightarrow \infty }\bigl\langle J'_{\lambda }(u_{j}),v \bigr\rangle \\& \quad = \int_{Q} \frac{(u_{\lambda }(x)-u_{\lambda }(y))(v(x)-v(y))}{\vert x-y\vert ^{N+2s}}\,dx\,dy -\mu \int_{\Omega }\frac{u_{\lambda }^{+}v}{\vert z\vert ^{2s}}\,dx \\& \qquad {}- \int_{\Omega }\frac{(u_{\lambda }^{+})^{2_{s}^{*}(\alpha)-1}v}{\vert z\vert ^{ \alpha }}\,dx-\lambda \int_{\Omega } f\bigl(x,u_{\lambda }^{+}\bigr)v \,dx \\& \quad =0, \end{aligned}$$

for any \(v \in X_{0}^{s}\). That is, \(\langle J'_{\lambda }(u_{\lambda }),v\rangle =0\) for any \(v \in X_{0}^{s}\). Then \(u_{\lambda }\) is a critical point of \(J_{\lambda }\), thus \(u_{\lambda }\) is a solution of problem (1). It follows from (6) and (12) that

$$\begin{aligned} J_{\lambda }(u_{\lambda }) =& J_{\lambda }(u_{\lambda })- \frac{1}{\xi }\bigl\langle J'_{\lambda }(u_{ \lambda }),u_{\lambda } \bigr\rangle \\ \geq & \biggl(\frac{1}{2}-\frac{1}{\xi } \biggr)\Vert u_{\lambda }\Vert _{X_{0}^{s}}^{2}+ \biggl( \frac{1}{\xi }-\frac{1}{2_{s}^{*}(\alpha)} \biggr) \int_{\Omega }\frac{\vert u_{\lambda }\vert ^{2_{s}^{*}(\alpha)}}{\vert z\vert ^{\alpha }}\,dx \\ &{} -\lambda \int_{\Omega } \biggl(F(x,u_{\lambda })-\frac{1}{\xi }f(x,u_{ \lambda })u_{\lambda } \biggr)\,dx \\ \geq & \biggl( \frac{1}{2}-\frac{1}{\xi } \biggr) \Vert u_{\lambda }\Vert _{X_{0}^{s}} ^{2} - \int_{\Omega } \biggl(L(x,u_{\lambda })-\frac{1}{\xi }l(x,u_{\lambda })u _{\lambda }\biggr)\,dx \\ \geq & \biggl( \frac{1}{2}-\frac{1}{\xi } \biggr) \Vert u_{\lambda }\Vert _{X_{0}^{s}} ^{2}-T_{\lambda }\vert \Omega \vert \\ \geq & -T_{\lambda }\vert \Omega \vert , \end{aligned}$$

for \(\xi \in (2,2_{s}^{*}(\alpha))\). Now, let \(w_{j}=u_{j}-u_{\lambda }\), by the Brezis-Lieb lemma [21], we have

$$\begin{aligned}& \int_{Q}\frac{\vert u_{j}(x)-u_{j}(y)\vert ^{2}}{\vert x-y\vert ^{N+2s}}\,dx\,dy \\& \quad = \int_{Q}\frac{\vert w_{j}(x)-w_{j}(y)\vert ^{2}}{\vert x-y\vert ^{N+2s}}\,dx\,dy+ \int_{Q}\frac{\vert u_{\lambda }(x)-u_{\lambda }(y)\vert ^{2}}{\vert x-y\vert ^{N+2s}}\,dx\,dy+o(1), \end{aligned}$$
(16)
$$\begin{aligned}& \int_{\Omega }\frac{u_{j}^{2}}{\vert z\vert ^{2s}}\,dx= \int_{\Omega }\frac{w_{j} ^{2}}{\vert z\vert ^{2s}}\,dx+ \int_{\Omega }\frac{u_{\lambda }^{2}}{\vert z\vert ^{2s}}\,dx+o(1), \end{aligned}$$
(17)
$$\begin{aligned}& \int_{\Omega }\frac{(u_{j}^{+})^{2_{s}^{*}(\alpha)}}{\vert z\vert ^{\alpha }}\,dx= \int_{\Omega }\frac{(w_{j}^{+})^{2_{s}^{*}(\alpha)}}{\vert z\vert ^{\alpha }}\,dx+ \int_{\Omega }\frac{(u_{\lambda }^{+})^{2_{s}^{*}(\alpha)}}{\vert z\vert ^{ \alpha }}\,dx+o(1). \end{aligned}$$
(18)

Since \(J_{\lambda }(u_{j})=c_{\lambda }+o(1)\), by (14) and (16)-(18), we obtain

$$ J_{\lambda }(u_{j})=J_{\lambda }(u_{\lambda })+ \frac{1}{2}\Vert w_{j}\Vert _{X_{0}^{s}}^{2}- \frac{1}{2_{s}^{*}(\alpha)} \int_{\Omega }\frac{(w _{j}^{+})^{2_{s}^{*}(\alpha)}}{\vert z\vert ^{\alpha }}\,dx=c_{\lambda }+o(1). $$
(19)

According to \(\langle J'_{\lambda }(u_{j}),u_{j}\rangle =o(1)\), (15) and (16)-(18), we get

$$ \Vert w_{j}\Vert _{X_{0}^{s}}^{2}- \int_{\Omega }\frac{(w_{j}^{+})^{2_{s}^{*}( \alpha)}}{\vert z\vert ^{2s}}\,dx=o(1). $$
(20)

Assume that \(\Vert w_{j}\Vert _{X_{0}^{s}}\rightarrow l\), it follows from (20) that

$$\int_{\Omega }\frac{(w_{j}^{+})^{2_{s}^{*}(\alpha)}}{\vert z\vert ^{2s}}\,dx \rightarrow l^{2}, $$

as \(j\rightarrow \infty \). From (4), one has

$$\Vert w_{j}\Vert _{X_{0}^{s}}^{2}\geq S_{\mu,\alpha } \biggl( \int_{\Omega }\frac{(w _{j}^{+})^{2_{s}^{*}(\alpha)}}{\vert z\vert ^{2s}}\,dx \biggr) ^{\frac{2}{2_{s} ^{*}(\alpha)}}. $$

We get \(l\geq S_{\mu,\alpha }^{\frac{N-\alpha }{2s-\alpha }}\). It follows from (19) and (20) that

$$\begin{aligned} c_{\lambda }+o(1) =& J_{\lambda }(u_{\lambda })+\frac{1}{2} \Vert w_{j}\Vert _{X_{0}^{s}}^{2}-\frac{1}{2_{s} ^{*}(\alpha)} \int_{\Omega }\frac{(w_{j}^{+})^{2_{s}^{*}(\alpha)}}{\vert z\vert ^{ \alpha }}\,dx+o(1) \\ \geq & \frac{2s-\alpha }{2(N-\alpha)} S_{\mu,\alpha }^{\frac{N-\alpha }{2s- \alpha }} -T_{\lambda } \vert \Omega \vert +o(1), \end{aligned}$$

which contradicts \(c_{\lambda }<\frac{2s-\alpha }{2(N-\alpha)} S_{ \mu,\alpha }^{\frac{N-\alpha }{2s-\alpha }}-T_{\lambda }\vert \Omega \vert \). Therefore, we have \(l=0\), which implies that \(u_{j}\rightarrow u_{ \lambda }\) in \(X_{0}^{s}\). This completes the proof of the Lemma 4. □

3 Proof of Theorem 1

Thanks to Lemmas 1, 2, 3 and Lemma 4, the functional \(J_{\lambda }\) satisfies all the assumptions of the mountain pass theorem for any \(\lambda \geq \lambda^{*}\), with \(\lambda^{*}>0\). This guarantees the existence of a critical point \(u_{\lambda }\in X_{0}^{s}\) for \(J_{\lambda }\) at level \(c_{\lambda }\). Since \(J_{\lambda }(u_{\lambda })=c_{\lambda }>0=J_{\lambda }(0)\) we have \(u_{\lambda }\not \equiv 0\). By [22, Lemma 8], we have \(u_{\lambda }^{-}\in X_{0}^{s}\) and we let \(\varphi =u_{\lambda }^{-}\) in (6), we get

$$\begin{aligned}& \int_{\mathbb{R}^{N}} \frac{(u_{\lambda }(x)-u_{\lambda }(y))(u_{ \lambda }^{-}(x)-u_{\lambda }^{-}(y))}{\vert x-y\vert ^{N+2s}}\,dy -\mu \int_{ \Omega }\frac{(u_{\lambda }^{-})^{2}}{\vert z\vert ^{2s}}\,dx \\& \quad = \int_{\Omega }\frac{(u_{\lambda }^{-})^{2^{*}(s)}}{\vert z\vert ^{\alpha }}\,dx + \lambda \int_{\Omega }f(x,u_{\lambda })u_{\lambda }^{-} \,dx. \end{aligned}$$

Moreover, for a.e. \(x,y\in \mathbb{R}^{N}\), one has

$$\begin{aligned}& \bigl(u_{\lambda }(x)-u_{\lambda }(y)\bigr) \bigl(u_{\lambda }^{-}(x)-u_{\lambda } ^{-}(y)\bigr) \\& \quad =-u_{\lambda }^{+}(x)u_{\lambda }^{-}(y)-u_{\lambda }^{-}(x)u_{ \lambda }^{+}(y) -\bigl(u_{\lambda }^{-}(x)-u_{\lambda }^{-}(y) \bigr)^{2} \\& \quad \leq -\bigl\vert u_{\lambda }^{-}(x)-u_{\lambda }^{-}(y) \bigr\vert ^{2}. \end{aligned}$$
(21)

From (3) and (21), we have

$$\begin{aligned}& \int_{Q} \frac{(u_{\lambda }(x)-u_{\lambda }(y))(u_{\lambda }^{-}(x)-u _{\lambda }^{-}(y))}{\vert x-y\vert ^{N+2s}}\,dx\,dy -\mu \int_{\Omega }\frac{(u _{\lambda }^{-})^{2}}{\vert z\vert ^{2s}}\,dx \\& \quad \leq \biggl(1-\frac{\mu }{a_{k,s}}\biggr) \int_{\mathbb{R}^{N}} \frac{ -\vert u_{\lambda }^{-}(x)-u_{\lambda }^{-}(y)\vert ^{2}}{\vert x-y\vert ^{N+2s}}\,dy\leq 0, \end{aligned}$$

by the fact that \(\mu < a_{k,s}\). Hence, according to \((f_{1})\), \(f(x,u_{\lambda }(x))u_{\lambda }^{-}(x)=0\) for \(x\in \mathbb{R}^{N}\), we obtain

$$\int_{\Omega }\frac{(u_{\lambda }^{-})^{2_{s}^{*}(\alpha)}}{\vert z\vert ^{ \alpha }}\,dx\leq 0, $$

which implies that \(u_{\lambda }^{-}\equiv 0\). Hence, \(u_{\lambda } \geq 0\). It implies \((-\Delta)^{s} u_{\lambda }\geq 0\). Then, by the strong maximum principle, we obtain \(u_{\lambda }\) is a positive solution of problem (1). This completes the proof of Theorem 1.

4 Conclusion

In this paper, we devoted our study to the existence of solutions for a fractional elliptic problems with the Hardy-Sobolev-Maz’ya potential and critical nonlinearities. The approach of this paper is by the well-known mountain pass theorem. The nonlinear term f satisfies assumptions \((f_{1})\), \((f_{2})\) without the (AR) conditions. We established a new term \(l(x,t)\), which satisfies the (AR) conditions combined with the critical term \(\frac{\vert t\vert ^{2_{s}^{*}(\alpha)-1}}{\vert z\vert ^{ \alpha }}\) by using some analysis techniques. Then we overcame the compactness and obtained a positive solution of problem (1). Our results are new and the work established in this paper is of quite a general nature.