1 Introduction

In this work, we consider the existence of positive solution of the following nonlocal quasilinear system:

$$\begin{aligned} \left\{ \begin{array}{llll} (-\Delta )_{p}^{s_1} u &{}=&{}\dfrac{1}{v^{\alpha _1}}+v^{\beta _1}&{} \text { in }\Omega , \\ (-\Delta )_{q}^{s_2} v &{}=&{}\dfrac{1}{u^{\alpha _2}}+u^{\beta _2}&{} \text { in }\Omega , \\ u,v&{}=&{}0 &{} \text { in } ({I\!\!R}^N\setminus \Omega ) , \\ u,v&{}>&{}0 &{} \text { in } \Omega , \end{array} \right. \end{aligned}$$
(1.1)

where \(\Omega \) be a smooth bounded domain of \({I\!\!R}^N\), \(s_1, s_2\in (0,1)\) with \(s_1\ne s_1\), \(\alpha _1\), \(\alpha _2\), \(\beta _1\) and \(\beta _2\) are positive constants. Here, \((-\Delta )_{p}^{s_2}\)( resp. \((-\Delta )_{q}^{s_2}\)) is the fractional \(p-\text {Laplacian}\) (resp. \(q-\text {Laplacian}\)), defined by

$$\begin{aligned} (-\Delta )^{t_i}_{g_i}u(x):= \text{ P.V. }\int _{{I\!\!R}^N}{\frac{|u(x)-u(y)|^{g_i-2}(u(x)-u(y))}{|x-y|^{N+t_ig_i}}\, \mathrm{d}y},\, \quad i=1,2, \end{aligned}$$
(1.2)

where \((t_1,g_1)=(s_1,p)\), \((t_2,g_2)=(s_2,q)\) with \(p,q>1\) and \(\text {P.V}\) is the principal value .

In the local case, \(t_i=1\) for \(i=1,2\), the operator defined in (1.2) is reduced to \(\Delta _{g_i}u=div(|\nabla u|^{g_i-2}\nabla u)\) that the well-known \(g_i-\text {Laplacian}\) operator with \(g_i>1\) and \(g_i\in \{p,q\}\).

Before giving our main results, let us briefly recall literature.

\(\bullet \) Equation: Notice that System (1.1) can be seen as a version of the singular scalar equations

$$\begin{aligned} \left\{ \begin{array}{llll} (-\Delta )_{p}^{s} u &{}=&{}\dfrac{\lambda }{u^{\alpha }}+u^{\beta }&{} \text { in }\Omega , \\ u&{}=&{}0 &{} \text { in } ({I\!\!R}^N\setminus \Omega ) , \\ u&{}>&{}0 &{} \text { in } \Omega .^{2}, \end{array} \right. \end{aligned}$$
(1.3)

where \(s\in (0,1)\), \(\alpha ,\beta >0\), \(p>1\) and \(\lambda \) is real positive parameter. Several works are devoted to classes of problems (1.3).

For \(s=1\), \(1<p<N\) and \(\lambda =0\), the existence of weak solution and regularity of solutions have been widely studied in [6, 7, 9, 11, 21, 29] and the references therein. In the case \(s=1\), \(p=2\), and \(\lambda \ne 0\), problem (1.3) has been treated in [18], where the author has used the variational method to show that for \(0<\lambda<\Lambda <\infty \), problem (1.3) has two solutions. This paper was generalized for \(p-\text {Laplacian}\) operator in [16] where the authors have showed the existence of two solutions using the variational method for \(0<\alpha <1\) and \(p-1<\beta <\frac{PN}{N-p}-1\) (see also [4]). Other related works can be found [3, 14, 15, 32] and their corresponding references.

Recently, the study of fractional elliptic equations with singular nonlinearity attracted lot of interests by researchers in nonlinear analysis. In [5], for \(p=2\) and \(0<s<1\), the authors studied the existence of distributional solutions of problem (1.3) using the uniform estimates of \(\{u_n\}\) which are solutions of the regularized problems with singular term \(u^{-\alpha }\) replaced by \((u_n+\frac{1}{n})^{-\alpha }\) (see also [12, 27, 30]) for more general context. The cases, when \(0<s<1\) and \(p\ne 2\), have been considered in [26] where the authors have showed the existence of multiple solutions to (1.3) using variational methods. Readers may refer to the work in [13, 31] and the references therein.

Needless to say, the references mentioned above do not exhaust the rich literature on the subject.

\(\bullet \) System : The case of systems with \(p,q-\text {Laplacians}\) and \(s_1=s_2=1\), System (1.1) with singular nonlinearities was treated in [1], the authors have showed using Rabinowitz bifurcation theorem and a Hardy–Sobolev inequality the existence of the weak solution, for every \((\alpha _i,\beta _i)\in (0,\theta _i)\) with \(i=1,2\) and

$$\begin{aligned} \theta _1=\min \Bigg \{\frac{p'}{q'},p-1, 1\Bigg \} \quad \hbox { and } \theta _2=\min \Bigg \{\frac{q'}{p'},q-1, 1\Bigg \}. \end{aligned}$$

We refer the readers, [2, 17, 22, 25] for more general context and the references therein.

Recently, System (1.1) has been treated by another type of operator, notably an anisotropic operator; see [8].

Our main interest in this work is to analyze System (1.1). We will consider principally nonlinearities with concave–convex structure. It is clear that one of the main difficulties to show some control of the singular term near the boundary of the domain. The existence of solutions will be proved using approximation technics, the classical Rabinowitz bifurcation Theorem, and Hopf’s lemma. Our main existence result is stated in the following theorem.

Theorem 1.1

Let \(\Omega \) be a bounded regular domain in \({I\!\!R}^N\), \(s_1,s_2\in (0,1)\), \(p\in [1, \frac{N}{s_1})\), \(q\in [1, \frac{N}{s_2})\) \(p'\in [1,p^{*}_{s_1}]\), \(q'\in [1,q^{*}_{s_2}]\) where \(p'\) and \(q'\) are conjugate exponents of p and q, respectively. Assume that \(\alpha _i,\beta _i \in (0,\gamma _i)\) for \(i=1,2\), such that

$$\begin{aligned} \gamma _1=\min \Bigg \{\frac{p'}{q'},p-1, \frac{s_1}{s_2}\Bigg \} \quad \hbox {and} \quad \gamma _2=\min \Bigg \{\frac{q'}{p'},q-1, \frac{s_2}{s_1}\Bigg \}. \end{aligned}$$

Then, System (1.1) possesses a nontrivial solution in \(W^{s_1,p}_0(\Omega )\times W^{s_2,q}_0(\Omega )\).

The paper is organized as follows. In the next section, we recall some basic notions and properties like fractional Sobolev spaces, notion of solution, and beside that some inequalities and useful lemmas are included, as well as strong maximum principle and Rabinowitz bifurcation Theorem that will be used along in this paper. In the last section, we prove the main existence results of this work.

2 The functional setting and tools

In this section, we collect some well-known results on Sobolev spaces and give some tools as they are needed to prove our main results.

Let \(\Omega \subset {I\!\!R}^N\) be an arbitrary open-bounded set . For \(p>1\) and \(s\in (0,1)\), we denoted by

$$\begin{aligned} W^{s,p}(\Omega ):= \left\{ u \in L^p(\Omega ): \iint _{\Omega \times \Omega } \frac{|u(x)-u(y)|^p}{|x-y|^{N+sp}}\mathrm{d}x \mathrm{d}y < \infty \right\} , \end{aligned}$$

the fractional order Sobolev space endowed with the norm

$$\begin{aligned} \Vert u\Vert _{W^{s,p}(\Omega )} := \left( \Vert u\Vert _{L^p(\Omega )}^p + \iint _{\Omega \times \Omega } \frac{|u(x)-u(y)|^p}{|x-y|^{N+sp}} \mathrm{d}x \mathrm{d}y \right) ^{\frac{1}{p}}. \end{aligned}$$

We set

$$\begin{aligned} W_0^{s,p}(\Omega ) := \left\{ u \in W^{s,p}({I\!\!R}^N): u = 0 in {I\!\!R}^N\setminus \Omega \right\} . \end{aligned}$$

Then, \(W_0^{s,p}(\Omega )\) endowed with the norm

$$\begin{aligned} \Vert u\Vert _{W^{s,p}_0(\Omega )} := \left( \iint _{D_{\Omega }} \frac{|u(x)-u(y)|^p}{|x-y|^{N+sp}} \mathrm{d}x \mathrm{d}y \right) ^{1/p}, \end{aligned}$$

where

$$\begin{aligned} D_{\Omega } := ({I\!\!R}^N\times {I\!\!R}^N) \setminus (\mathcal {C}\Omega \times \mathcal {C}\Omega ) = (\Omega \times {I\!\!R}^N) \cup (\mathcal {C}\Omega \times \Omega ), \end{aligned}$$

is Banach space; we refer to [10, 23] for more details and properties of the fractional Sobolev spaces.

Theorem 2.1

(Fractional Sobolev inequality [10]) Assume that \(0<s<1, p>1\) satisfy \(ps<N\). Then, there exists a positive constant \(S\equiv S(N,s,p)\), such that for all \(v\in C_{0}^{\infty }({I\!\!R}^N)\)

$$\begin{aligned} \iint _{{I\!\!R}^{2N}}\dfrac{|v(x)-v(y)|^{p}}{|x-y|^{N+ps}}\,\mathrm{d}x \mathrm{d}y \ge S \Big (\displaystyle \int _{\mathbb {R}^{N}}|v(x)|^{p_{s}^{*}}\mathrm{d}x\Big )^{\frac{p}{p^{*}_{s}}}, \end{aligned}$$

where \(p^{*}_{s}= \dfrac{pN}{N-ps}\) is critical Sobolev exponent.

Let us consider now the following quasilinear problem:

$$\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )_p^s u=f &{}\hbox { in } \Omega ,\\ u=0 &{}\hbox { in } \mathbb {R}^N\setminus \Omega , \end{array}\right. } \end{aligned}$$
(2.1)

where \(\Omega \subset {I\!\!R}^N\) be open-bounded domain, \(1<p<\infty \), \(0<s<1\), \(f\in W^{-s,p'}(\Omega )\) (we shall denoted \(W^{-s,p'}(\Omega )\) is the dual of the reflexive Banach space of \(W_0^{s,p}(\Omega )\) and \(p'=\frac{p}{p-1}\).

Definition 2.2

For \(f\in W^{-s,p'}(\Omega )\), we say that \(u\in W_0^{s,p}(\Omega )\) is a weak solution to (2.1) if

$$\begin{aligned} \begin{array}{lll}&\displaystyle \dfrac{C_{N,p,s}}{2}\iint _{D_{\Omega }}\dfrac{|u(x)-u(y)|^{p-2}(u(x)-u(y)(\phi (x)-\phi (y))}{|x-y|^{N+sp}}\mathrm{d}x\mathrm{d}y = \int \limits _\Omega f v\mathrm{d}x, \end{array} \end{aligned}$$

for all \(v\in W^{s,p}_0(\Omega )\).

Proposition 2.3

[33]. Let \(s\in (0,1)\) and \(1<p<\infty \). Then, for every \(f\in W^{-s,p'}(\Omega )\), the Dirichlet problem (2.1) has a unique weak solution \(u\in W_0^{s,p}(\Omega )\). Moreover

$$\begin{aligned} ||u||_{W_0^{s,p}(\Omega )}\le ||f||^{\frac{1}{p-1}}_{W^{-s,p'}(\Omega )}. \end{aligned}$$
(2.2)

Proposition 2.4

[33]. Let \(s\in (0,1)\) and \(1<p<\infty \). Then, we have that

  1. (1)

    \((-\Delta )^s_p : W_0^{s,p}(\Omega ) \rightarrow W^{-s,p'}(\Omega )\) is strictly monotone, continuous, coercive, and bounded.

  2. (2)

    \(((-\Delta )^s_p)^{-1} : W^{-s,p'}(\Omega )\rightarrow W_0^{s,p}(\Omega )\) is locally Lipschitz continuous if \(p\in (1,2)\) and is Lipschitz continuous if \(p\ge 2\).

  3. (3)

    The composed operator \(W^{-s,p'}(\Omega )\hookrightarrow W_0^{s,p}(\Omega )\hookrightarrow L^{q}(\Omega )\) is compact if \(1\le q<\frac{pN}{N-ps}\).

Lemma 2.5

(Strong maximum principle [24]). Let \(u\in W_0^{s,p}(\Omega )\) satisfy

$$\begin{aligned}{} {\left\{ \begin{array}{ll} (-\Delta )_p^s u\ge 0 &{}\hbox { in } \Omega ,\\ u\ge 0 &{}\hbox { in } \mathbb {R}^N\setminus \Omega . \end{array}\right. } \end{aligned}$$
(2.3)

Then, u has lower semi-continuous representative in \(\Omega \), which is either identically 0 or positive.

Theorem 2.6

(Hardy inequality [19]). Let \(0<s<1\) and \(1<p<\infty \) be such that \(sp<N\). Assume that \(\Omega \subset {I\!\!R}^N\) is a (bounded) uniform domain with a (locally) \((s, p)-\text {uniformly fat boundary}\). Then, \(\Omega \) admits an \((s, p)-\text {Hardy inequality}\), that is, there is a constant \(C>0\), such that

$$\begin{aligned} C \displaystyle \int _{\Omega }\dfrac{|u(x)|^{p}}{d^{sp}(x)}\mathrm{d}x\le \iint _{{I\!\!R}^{2N}}\dfrac{|u(x)-u(y)|^{p}}{|x-y|^{N+ps}}\,\mathrm{d}x \mathrm{d}y\quad \hbox { for every } u\in W^{s,p}_0(\Omega ). \end{aligned}$$
(2.4)

Finally, we recall the classical Rabinowitz result, see [28], that will be used systematically in this paper.

Theorem 2.7

Let E be a Banach space and \(T: \mathbb {R}^{+}\times E\rightarrow E\) a continuous and compact operator, such that \(T(0,u)=0\) for all \(u\in E\). Then, the equation

$$\begin{aligned} u=T(\lambda ,u), \end{aligned}$$

possesses an unbounded continuum \(F\subset \mathbb {R}^{+}\times E\) of solutions with \((0,0)\in F\).

3 Proof of the main result

In this section, we focus to prove the existence of nontrivial solution to (1.1) under some hypothesis on \(\alpha _1\), \(\alpha _2\), \(\beta _1\), \(\beta _2\), \(s_1\), and \(s_2\)

Before proving Theorem 1.1, we begin with the following auxiliary system:

$$\begin{aligned} \left\{ \begin{array}{llll} (-\Delta )_{p}^{s_1} u &{}=&{}\lambda \Big [\dfrac{1}{\sigma +v^{\alpha _1}}+v^{\beta _1}\Big ]&{} \text { in }\Omega , \\ (-\Delta )_{q}^{s_2} v &{}=&{}\lambda \Big [\dfrac{1}{\delta +u^{\alpha _2}}+u^{\beta _2}\Big ]&{} \text { in }\Omega , \\ u,v&{}=&{}0 &{} \text { in } ({I\!\!R}^N\setminus \Omega ) , \\ u,v&{}>&{}0 &{} \text { in } \Omega . \end{array} \right. \end{aligned}$$
(3.1)

First, we begin by the following Lemma.

Lemma 3.1

Under the hypothesis of Theorem 1.1. Then, for \(\delta ,\sigma >0\) fixed and for all \(\lambda >0\), System (3.1) possesses a nontrivial solution in \(W^{s_1,p}_0(\Omega )\times W^{s_2,q}_0(\Omega )\).

Proof

We consider now the following approximating system:

$$\begin{aligned} \left\{ \begin{array}{llll} (-\Delta )_{p}^{s_1} u &{}=&{}\lambda \Big [\dfrac{1}{|\psi |^{\alpha _1}+\sigma }+|\psi |^{\beta _1}\Big ]&{} \text { in }\Omega , \\ (-\Delta )_{q}^{s_2} v &{}=&{}\lambda \Big [\dfrac{1}{|\phi |^{\alpha _2}+\delta }+|\phi |^{\beta _2}\Big ]\Omega , \\ u,v&{}=&{}0 &{} \text { in } ({I\!\!R}^N\setminus \Omega ) , \\ u,v&{}>&{}0 &{} \text { in } \Omega , \end{array} \right. \end{aligned}$$
(3.2)

where \((\phi \,\,, \psi )\in L^{p'}(\Omega )\times L^{q'}(\Omega )\) are be fixed.

First of all, we observe that:

For \(\psi \in L^{q'}(\Omega )\) and \(\sigma >0\), we have that \(\frac{1}{|\psi |^{\alpha _1}+\sigma }\in L^{p'}(\Omega )\).

On the other hand, by hypothesis, \(p'\beta _1<q'\) , then we get,

$$L^{q'}(\Omega )\hookrightarrow L^{p'\beta _1}(\Omega ).$$

Hence, \(|\psi |^{\beta _1}\in L^{p'}(\Omega )\).

By same way as before, we obtain that \(\frac{1}{|\phi |^{\alpha _2}+\delta }\in L^{q'}(\Omega ) \) and \(|\phi |^{\beta _2}\in L^{q'}(\Omega )\) for \(\beta _2q'<p'\).

Now, using Proposition 2.3 for each \((\lambda ,\phi ,\psi )\in {I\!\!R}^{+}\times L^{p'}(\Omega )\times L^{q'}(\Omega )\), system (3.2) possesses a unique weak solution (u, v) in \(W^{s_1,p}_0(\Omega )\times W^{s_2,q}_0(\Omega )\), that is

$$\begin{aligned} \dfrac{C_{N,p,s}}{2}\iint _{D_{\Omega }}\dfrac{|u(x)-u(y)|^{p-2}(u(x)-u(y)(\xi (x)-\xi (y))}{|x-y|^{N+sp}}\mathrm{d}x\mathrm{d}y=\lambda \int \limits _\Omega \Big [\dfrac{1}{|\psi |^{\alpha _1}+\sigma }+|\psi |^{\beta _1}\Big ] \xi \mathrm{d}x, \end{aligned}$$

for all \(\xi \in W^{s_1,p}_0(\Omega )\) and

$$\begin{aligned} \dfrac{C_{N,q,s}}{2}\iint _{D_{\Omega }}\dfrac{|u(x)-u(y)|^{q-2}(u(x)-u(y)(\zeta (x)-\zeta (y))}{|x-y|^{N+sq}}\mathrm{d}x\mathrm{d}y=\lambda \int \limits _\Omega \Big [\dfrac{1}{|\phi |^{\alpha _2}+\delta }+|\phi |^{\beta _2}\Big ] \zeta \mathrm{d}x, \end{aligned}$$

for all \(\zeta \in W^{s_2,q}_0(\Omega )\).

Hence, the following operator:

$$\begin{aligned} S:&\mathbb {R}^{+}\times L^{p'}(\Omega )\times L^{q'}(\Omega )\rightarrow W^{s_1,p}_0(\Omega )\times W^{s_2,q}_0(\Omega ) \\&(\lambda ,\phi ,\psi )\mapsto T(\lambda ,\phi ,\psi )=(u,v). \end{aligned}$$

is well defined. Let us show that

$$\begin{aligned} S:&\mathbb {R}^{+}\times L^{p'}(\Omega )\times L^{q'}(\Omega )\rightarrow L^{p'}(\Omega )\times L^{q'}(\Omega ) \\&(\lambda ,\phi ,\psi )\mapsto T(\lambda ,\phi ,\psi )=(u,v) \end{aligned}$$

is compact.

In fact, let \(\{(\lambda _n,\phi _n,\psi _n)_n\}\) be a bounded sequence in \(\mathbb {R}^{+}\times L^{p'}(\Omega )\times L^{q'}(\Omega )\), such that

$$\begin{aligned} \left\{ \begin{array}{llll} (-\Delta )_{p}^{s_1} u_n &{}=&{}\lambda _n\Big [\dfrac{1}{|\psi _n|^{\alpha _1}+\sigma }+|\psi _n|^{\beta _1}\Big ]&{} \text { in }\Omega , \\ (-\Delta )_{q}^{s_2} v_n &{}=&{}\lambda _n\Big [\dfrac{1}{|\phi _n|^{\alpha _2}+\delta }+|\phi _n|^{\beta _2}\Big ]\Omega , \\ u_n,v_n&{}=&{}0 &{} \text { in } ({I\!\!R}^N\setminus \Omega ) , \\ u_n,v_n&{}>&{}0 &{} \text { in } \Omega . \end{array} \right. \end{aligned}$$
(3.3)

Therefore, it follows that \(\{u_n\}\) and \(\{v_n\}\) are bounded, respectively, in \(W^{s_1,p}_0(\Omega )\) and \(W^{s_2,q}_0(\Omega )\).

Thus, we get the existence a subsequence \((u_n, v_n)_n\) and \((u,v)\in W^{s_1,p}_0(\Omega )\times W^{s_2,q}_0(\Omega )\), such that

  1. 1.

    \(u_n\rightharpoonup u\) weakly in \(W^{s_1,p}_0(\Omega )\),

  2. 2.

    \(u_n \rightarrow u\) in \(L^{\tau }\) strongly for every \(\tau \in [1, p^{*}_{s_1})\),

  3. 3.

    \(v_n \rightharpoonup v\) weakly in \(W^{s_2,q}_0(\Omega )\),

  4. 4.

    \(v_n \rightarrow v\) in \(L^{\theta }(\Omega )\) strongly for every \(\theta \in [1, q^{*}_{s_2})\).

Now, using \(\xi =u_n-u\) as test function in first equation of system (3.3), we get

$$\begin{aligned} \int \limits _\Omega (u_n-u )((-\Delta )_{p}^{s_1} u_n -(-\Delta )_{p}^{s_1} u )\mathrm{d}x= & {} \lambda _n\Big [\int \limits _\Omega \dfrac{u_n-u}{|\psi _n|^{\alpha _1}+\sigma }\mathrm{d}x+\int \limits _\Omega |\psi _n|^{\beta _1}(u_n-u) \mathrm{d}x\Big ]+o(1)\\\le & {} C||u_n-u||_{L^{p}(\Omega )}+C||u_n-u||_{L^{p}(\Omega )}\left\| \psi _n^{\beta _1}\right\| _{L^{p'}(\Omega )}+o(1),\\\le & {} C||u_n-u||_{L^{p}(\Omega )}+C||u_n-u||_{L^{p}(\Omega )}\left\| \psi _n\right\| ^{\beta _1}_{L^{q'}(\Omega )}+o(1),\\\le & {} C||u_n-u||_{L^{p}(\Omega )}+o(1). \end{aligned}$$

On the other hand, if \(p\ge 2\), we obtain that

$$\begin{aligned} C||u_n-u||^{p}_{W^{s_1,p}_0(\Omega )}\le \int \limits _\Omega (u_n-u )((-\Delta )_{p}^{s_1} u_n -(-\Delta )_{p}^{s_1} u )\mathrm{d}x. \end{aligned}$$

Thus

$$\begin{aligned} ||u_n-u||^{p}_{W^{s_1,p}_0(\Omega )}\le C ||u_n-u||_{L^{p}(\Omega )}. \end{aligned}$$

Since \(W^{s_1,p}_0(\Omega )\hookrightarrow L^p(\Omega )\) is compact, hence, it follows that \(u_n\rightarrow u\) strongly in \(W^{s_1,p}_0(\Omega )\). As before, by similar reasoning , we get \(v_n\rightarrow v\) strongly in \(W^{s_2,q}_0(\Omega )\). The case \(1<q<2\) and \(1<p<2\) is made using similar arguments, and we will omit its proof.

Therefore

$$\begin{aligned} ||(u_n,v_n)-(u,v)||\rightarrow 0, \quad \hbox { as } n\rightarrow +\infty , \quad \hbox { in } W^{s_1,p}_0(\Omega )\times W^{s_2,q}_0(\Omega ). \end{aligned}$$

Consequently, mapping S is compact. and claim follows.

Hence, using the same computation, we get easily that S is continuous.

On the other hand, we observe that \((0,0,0)\in F\) and \(S(0,u,v)=(0,0)\), then we are in the conditions of Theorem 2.7. Hence, we get an component F of solutions to \((u,v)=S(\lambda ,u,v)\), which means that, if \((\lambda ,u,v)\in F\), then, \((\lambda ,u,v)\) solve (3.1). It remains only to show that \((1,u,v)\in F\) which corresponds to our required solution.

It clear that for every \(\lambda >0\) by strong maximum principle \(u,v>0\) in \(\Omega \).

We argue by contradiction. Assume that there exist \(\lambda ^{*}\), such that for all \(\lambda \le \lambda ^{*}\), we have \((\lambda ,\, u,\,v)\in F\).

Using u as test function of the first equation of (3.1), we obtain that

$$\begin{aligned} ||u||^{p}_{W^{s_1,p}_0(\Omega )}= & {} \lambda \Big [\int \limits _\Omega \dfrac{u}{v^{\alpha _1}+\sigma }\mathrm{d}x+\int \limits _\Omega v^{\beta _1}u \mathrm{d}x\Big ]\\\le & {} C||u||_{L^{p}(\Omega )}+C||u||_{L^{p}(\Omega )}\left\| v\right\| ^{\beta _1}_{L^{q'}(\Omega )},\\\le & {} C||u||_{W^{s_1,p}_0(\Omega )}+C||u||_{W^{s_1,p}_0(\Omega )}\left\| v\right\| ^{\beta _1}_{W^{s_2,q}_0(\Omega )},\\ \end{aligned}$$

where in the last inequalities, we have used the fact

$$\begin{aligned} W^{s_1,p}_0(\Omega )\hookrightarrow L^{p}(\Omega ) \hbox { and } W^{s_2,q}_0(\Omega )\hookrightarrow L^{q'}(\Omega ). \end{aligned}$$

Similarly, using v as test function in the second equation of (3.1) and by taking into consideration the following immersions:

$$\begin{aligned} W^{s_2,q}_0(\Omega )\hookrightarrow L^{q}(\Omega ) \hbox { and } W^{s_1,p}_0(\Omega )\hookrightarrow L^{p'}(\Omega ), \end{aligned}$$

it follows that:

$$\begin{aligned} ||v||^{q}_{W^{s_2,q}_0(\Omega )}\le C||v||_{W^{s_2,q}_0(\Omega )}+C||v||_{W^{s_2,q}_0(\Omega )}\left\| u\right\| ^{\beta _2}_{W^{s_1,p}_0(\Omega )}. \end{aligned}$$

Hence, combining the above estimate, we obtain that

Since \(p,q>1\), \(\beta _1<p-1\) and \(\beta _2<q-1\), then from the last inequality, we get \(||u||^{p}_{W^{s_1,p}_0(\Omega )}\) and \(||v||^{q}_{W^{s_2,q}_0(\Omega )}\) are bounded and this provides the contradiction, and consequently, F must be unbounded with respect to \(\lambda \) and in particular for \(\lambda =1\), on have \((1,\,u,\,v)\in F\) which gives a solution to (3.1). \(\square \)

Now, we are able to prove our main result.

Proof of Theorem 1.1. Using Lemma 3.1, we deduce that the system

$$\begin{aligned} \left\{ \begin{array}{llll} (-\Delta )_{p}^{s_1} u_n &{}=&{}\dfrac{1}{v_n^{\alpha _1}+\frac{1}{n}}+v_n^{\beta _1}&{} \text { in }\Omega , \\ (-\Delta )_{q}^{s_2} v_n &{}=&{}\dfrac{1}{u_n^{\alpha _2}+\frac{1}{n}}+u_n^{\beta _2}&{} \text { in }\Omega , \\ u_n,v_n&{}=&{}0 &{} \text { in } ({I\!\!R}^N\setminus \Omega ) , \\ u_n,v_n&{}>&{}0 &{} \text { in } \Omega \end{array} \right. \end{aligned}$$
(3.4)

has a solution \((u_n,v_n)\in W^{s_1,p}_0(\Omega )\times W^{s_2,q}_0(\Omega )\).

Now, we claim

$$\begin{aligned} u_n>0, v_n >0 \hbox { for all } n\in {I\!\!N}. \end{aligned}$$

In fact, let \((w_1, w_2)\in W^{s_1,p}_0(\Omega )\times W^{s_2,q}_0(\Omega ) \) are the nontrivial solutions to

$$\begin{aligned} \left\{ \begin{array}{llll} (-\Delta )_{p}^{s_1} w_1 &{}=&{} m_1, \text { in }\Omega , \\ w_1 &{}=&{}0 &{} \text { in } ({I\!\!R}^N\setminus \Omega ) , \\ w_1 &{}>&{}0 &{} \text { in } \Omega . \end{array} \right. \end{aligned}$$
(3.5)

and

$$\begin{aligned} \left\{ \begin{array}{llll} (-\Delta )_{q}^{s_2} w_2 &{}=&{} m_2, \text { in }\Omega ,\\ w_2 &{}=&{}0 &{} \text { in } ({I\!\!R}^N\setminus \Omega ) , \\ w_2 &{}>&{}0 &{} \text { in } \Omega , \end{array} \right. \end{aligned}$$
(3.6)

where \(0<m_1=\min _{t\ge 0}\Big \{\frac{1}{t^{\alpha _1+1}}+t^{\beta _1} \Big \} \) and \(0<m_2=\min _{t\ge 0}\Big \{\frac{1}{t^{\alpha _2}+1}+t^{\beta _2}\Big \}\).

Since \(p,q>1\), then using the comparison principle (see [20] ) and Lemma 2.5, we get

$$\begin{aligned} u_n\ge w_1>0 \quad \hbox { and } v_n\ge w_2>0\quad \hbox { for every } n\in {I\!\!N}\end{aligned}$$

as desired. Let us show that the sequences \(\{u_n\}_n\) and \(\{v_n\}_n\) are bounded in \(W^{s_1,p}_0(\Omega )\) and \( W^{s_2,q}_0(\Omega )\), respectively.

First, we take \(u_n\) as test function in first equation of (3.4), and we get

$$\begin{aligned} ||u_n||^{p}_{W^{s_1,p}_0(\Omega )}\le \int \limits _\Omega \frac{u_n}{v_n^{\alpha _1}}\mathrm{d}x+\int \limits _\Omega u_nv_n^{\beta _1}\mathrm{d}x. \end{aligned}$$
(3.7)

Since \(\beta _1 p'<q'\) and \(q'\in [1, q^*_{s_2}]\), thus, using Hölder and Sobolev inequalities , it follows that:

$$\begin{aligned} \int \limits _\Omega u_nv_n^{\beta _1}\mathrm{d}x\le & {} C||u_n||_{L^{p}(\Omega )}\left\| v_n\right\| ^{\beta _1}_{L^{q'}(\Omega )}\le C||u_n||_{W^{s_1,p}_0(\Omega )}||v_n||^{\beta _1}_{W^{s_1,q}_0(\Omega )}.\\ \end{aligned}$$

Now, we will estimate the first integral in the right-hand side of inequality (3.7). By Hopf’s lemma (see [24]), we get that, \(w_2(x)\ge Cd^{s_2}(x)\); therefore, \(v_n\ge w_2(x)\ge Cd^{s_2}(x)\) in \(\Omega \).

Therefore, we get

$$\begin{aligned} \int \limits _\Omega \frac{u_n}{v_n^{\alpha _1}}\mathrm{d}x \le \int \limits _\Omega \frac{u_n}{w_2^{\alpha _1}}\mathrm{d}x\le C\int \limits _\Omega \frac{u_n}{d^{s_2\alpha _1}(x)}\mathrm{d}x. \end{aligned}$$

Since \(s_1>\alpha _1 s_2\), then, using Hölder and Hardy inequalities, we reach that

$$\begin{aligned} \int \limits _\Omega \frac{u_n}{v_n^{\alpha _1}}\mathrm{d}x\le & {} C\int \limits _\Omega \frac{u_n}{d^{s_1}(x)}d^{s_1-s_2\alpha _1}(x)\mathrm{d}x \le C \left( \int \limits _\Omega \frac{u^p_n}{d^{ps_1}(x)}\mathrm{d}x\right) ^{\frac{1}{p}}\le C ||u_n||_{W^{s_1,p}_0(\Omega )} . \end{aligned}$$

Therefore, we conclude that

$$\begin{aligned} ||u_n||^{p}_{W^{s_1,p}_0(\Omega )}\le C ||u_n||_{W^{s_1,p}_0(\Omega )}+C||u_n||_{W^{s_1,p}_0(\Omega )}||v_n||^{\beta _1}_{W^{s_2,q}_0(\Omega )}. \end{aligned}$$
(3.8)

By the same computation as in above, if \(s_2>\alpha _2 s_1\), \(\beta _2 q'<p'\) and \(p'\in [1, p^{*}_{s_1}]\) are satisfies, we can show that

$$\begin{aligned} ||v_n||^{q}_{W^{s_2,q}_0(\Omega )}\le C ||v_n||_{W^{s_2,q}_0(\Omega )}+C||v_n||_{W^{s_2,q}_0(\Omega )}||u_n||^{\beta _2}_{W^{s_1,p}_0(\Omega )}, \end{aligned}$$
(3.9)

Since \(p,q> 1\) and \(\beta _1,\beta _2 <\min \{p-1, q-1\}\) and from (3.8) and (3.9), we reach that \(\{u_n\}_n\) and \(\{v_n\}_n\) are bounded in \(W^{s_1,p}_0(\Omega )\) and in \( W^{s_2,q}_0(\Omega )\), respectively. Therefore, there exist two measurable functions \(u\in W^{s_1,p}_0(\Omega ) \) and \(v\in W^{s_2,q}_0(\Omega )\), such that

  1. (1)

    \(u_n\rightharpoonup u\) weakly in \(W^{s_1,p}_0(\Omega )\),

  2. (2)

    \(u_n \rightarrow u\) in \(L^{\tau }\) strongly for every \(\tau \in [1, p^{*}_{s_1})\),

  3. (3)

    \(v_n \rightharpoonup v\) weakly in \(W^{s_2,q}_0(\Omega )\),

  4. (4)

    \(v_n \rightarrow v\) in \(L^{\theta }(\Omega )\) strongly for every \(\theta \in [1, q^{*}_{s_2})\).

  5. (5)

    \(u_n (x) \rightarrow u(x)\) a.e in \(\Omega \),

  6. (6)

    \(v_n (x)\rightarrow v (x)\) a.e in \(\Omega \).

Hence, using classical arguments, we get the desired result.

A direct consequence of our result in the case where \(s_1=s_2=s\) is the following.

Corollary 3.2

Let \(\Omega \) be a bounded regular domain of \({I\!\!R}^N\), \(s\in (0,1)\), \(p\in [1, \frac{N}{s})\), \(q\in [1, \frac{N}{s})\), \(p'\in [1,p^{*}_{s}]\), \(q'\in [1,q^{*}_{s}]\) where \(p'\) and \(q'\) are conjugate exponents of p and q, respectively. Assume that \(\alpha _i,\beta _i \in (0,\gamma _i)\) for \(i=1,2\), such that

$$\begin{aligned} \gamma _1=\min \left\{ \frac{p'}{q'},p-1, 1\right\} \quad \hbox { and } \gamma _2=\min \left\{ \frac{q'}{p'},q-1, 1\right\} . \end{aligned}$$

Then, System (1.1) possesses a nontrivial solution in \(W^{s_1,p}_0(\Omega )\times W^{s_2,q}_0(\Omega )\).

Remark 1

Notice that, if we take, \(s_1=s_2=1\) in Theorem 1.1, we get the result obtained in [1].