Liouville-type theorems for fractional Hardy–Hénon systems

Abstract

In this paper, we study Liouville-type theorems for fractional Hardy–Hénon elliptic systems with weights. Because the weights are singular at zero, we firstly prove that classical solutions for systems in ${\mathbb {R}}^N \backslash \{0\}$ are also distributional solutions in ${\mathbb {R}}^N$. Then we study the equivalence between the fractional Hardy–Hénon system and a proper integral system, and we obtain new Liouville-type theorems for supersolutions and solutions by the method of integral estimates and scaling spheres respectively.

1 Introduction

In recent years, the fractional Laplacian operator as a nonlocal operator has been widely used to model diverse physical phenomena, such as anomalous diffusion and quasi-geostrophic flows, turbulence and water waves, molecular dynamics and relativistic quantum mechanics of stars [8, 9, 37]. Moreover, they also appear in mathematical finance [2, 15], elasticity problems [34], obstacle problems [3, 4], phase transition [1] and crystal dislocation [18, 38].

The aim of this paper is to investigate the fractional Hardy–Hénon system

$$\begin{aligned} \left\{ \begin{array}{ll} (-\Delta )^{s_1} u = |x|^a v^p,&{}\quad x \in {\mathbb {R}}^N, \\ (-\Delta )^{s_2} v = |x|^b u^q,&{}\quad x \in {\mathbb {R}}^N, \end{array}\right. \end{aligned}$$

(1.1)

where $(-\Delta )^{s_i}(i=1,2)$ is the fractional Laplacian operator. Throughout this paper, we always suppose $s_1, s_2 \in (0,1)$, $a, b \in {\mathbb {R}}$, $p, q > 0$ and $N \ge 2$. The fractional Laplacian operator $( - \Delta )^s$, $0< s < 1$ is defined as

$$\begin{aligned} (-\Delta )^s u(x) = C(N,s) \int \nolimits _{{\mathbb {R}}^N} \frac{ u(x) - u(y) }{ |x-y|^{N+2s} } \,\textrm{d}y, \end{aligned}$$

(1.2)

where $C(N,s) > 0$ is a normalization constant that is often omitted for convenience. The integral in (1.2) is interpreted in the Cauchy principal value sense, that is, as the limit as $\varepsilon \rightarrow 0$ of the same integral taken in ${\mathbb {R}}^N \backslash B_\varepsilon (x)$, i.e., the complementary of the ball of center x and radius $\varepsilon$. This operator is well defined in ${\mathcal {L}}_{2\,s} \cap C_{loc}^{1,1}$, where

$$\begin{aligned} {\mathcal {L}}_{2s} := \left\{ u : {\mathbb {R}}^N \rightarrow {\mathbb {R}} : \int \nolimits _{{\mathbb {R}}^N} \frac{ |u(x)| }{ 1 + |x|^{N+2s} } < \infty \right\} . \end{aligned}$$

The system (1.1) is a natural generalization of the fractional Hardy–Hénon equation

$$\begin{aligned} ( - \Delta )^s u = |x|^a u^p,\quad x \in {\mathbb {R}}^N, \end{aligned}$$

(1.3)

where $s \in (0,1)$, $a \in {\mathbb {R}}$, $p > 0$.

As for Liouville-type theorems of (1.3), in the local case $s = 1$, the equation (1.3) is the famous Hardy–Hénon equation

$$\begin{aligned} - \Delta u = |x|^a u^p,\quad x \in {\mathbb {R}}^N. \end{aligned}$$

(1.4)

Define the Hardy–Sobolev exponent

$$\begin{aligned} p_S(a) = \left\{ \begin{array}{ll} \frac{N+2+2a}{N-2},&{} \ N \ge 3, \\ +\infty ,&{} \ N = 2. \end{array}\right. \end{aligned}$$

When $a \le -2$, Dupaigne et al. [21] proved that Eq. (1.4) has no positive solutions in $C^2({\mathbb {R}}^N \backslash \{0\}) \cap C({\mathbb {R}}^N)$. As for the case $a > -2$, Phan and Souplet proposed in [29] the following conjecture:

Conjecture 1.1

Let $N \ge 2$ and $a > -2$. Then Eq. (1.4) has no positive solutions if $0< p < p_S(a)$.

In [29], authors proved this conjecture for bounded solutions. Moreover, in [25], Li and Zhang solved this conjecture for $N = 3$ and Guo and Wan [20] completely solved this conjecture for $N \ge 3$.

In the nonlocal case $0< s < 1$, authors in [19] established some Liouville-type theorems for positive solutions when $a > 0$ and $\frac{N+a}{N-2\,s}< p < \frac{N+a+2\,s}{N-2\,s}$ via the method of moving planes in integral forms. In [5], authors proved a nonexistence result for positive solutions in the optimal range of the nonlinearity, that is, when $a > -2\,s$ and $1< p < \frac{N+2a+2\,s}{N-2\,s}$, and additionally assume $a > -N/2$ if $N < 4s$. Furthermore, they used reduction argument to prove that a bubble solution, which is a fast decay positive radially symmetric solution, exists when $p = \frac{N+2a+2\,s}{N-2\,s}$, $\frac{1}{2} \le s < 1$, $a > 0$ or $0< s < \frac{1}{2}$, $0< a < \frac{2\,s(N-1)}{1-2\,s}$.

Returning to the system (1.1), when $s_1 = s_2 = 1$, (1.1) becomes famous Hénon–Lane–Emden elliptic system

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u = |x|^a v^p,&{}\quad x \in {\mathbb {R}}^N, \\ -\Delta v = |x|^b u^q,&{}\quad x \in {\mathbb {R}}^N. \end{array}\right. \end{aligned}$$

(1.5)

Define critical Sobolev hyperbola (see [7, 10])

$$\begin{aligned} \frac{N+a}{p+1} + \frac{N+b}{q+1} = N -2, \end{aligned}$$

and call the pair (p, q) subcritical if $\frac{N+a}{p+1} + \frac{N+b}{q+1} > N -2$. In [6], authors proved that the system (1.5) has positive radially symmetric solutions if and only if the pair (p, q) is above or on the critical Sobolev hyperbola, which implies the following well-known Hénon–Lane–Emden conjecture is valid if all positive solutions of (1.5) are radially symmetric:

Conjecture 1.2

Suppose that (p, q) is subcritical, then system (1.5) has no positive classical solutions.

When $a = b = 0$, this is celebrated Lane–Emden conjecture which has been extensively studied and solved for $N \le 4$ in [27, 30, 31, 33, 36]. It still remains open for $N \ge 5$. Recently, Li and Zhang [25] proved the Hénon–Lane–Emden conjecture for $N = 3$.

When $s_1, s_2 \in (0,1)$, there are very few results about Liouville-type theorems for (1.1). In [19], Dou and Zhou proved that when $s_1 = s_2 =: s$, $a, b \ge 0$, $\frac{N+a}{N-2\,s} < p \le \frac{N+a+2\,s}{N-2\,s}$, $\frac{N+b}{N-2\,s} < q \le \frac{N+b+2\,s}{N-2\,s}$, the system (1.1) has no nontrivial nonnegative solutions. In [26], authors used moving plane method to prove that when $s_1 = s_2 = s$, $a, b \ge 0$, $1< p < \frac{N+2\,s+a}{N-2\,s}$, $1< q < \frac{N+2\,s+b}{N-2\,s}$, the system (1.1) has no nontrivial nonnegative solutions. In [28], authors used scaling sphere method to prove Liouville theorems for the system (1.3) with $s_1 = s_2 =s$. They proved that when $a, b > -2\,s$ and $0< p < \frac{N+2\,s+2a}{N-2\,s}$, $0< q < \frac{N+2\,s+2b}{N-2b}$, the system (1.3) has no nontrivial nonnegative solutions. Moreover, they also prove Liouville theorems for higher-order Hénon–Hardy systems, i.e., s is a positive integer. However, since this is out of the scope of our paper, we refer interested readers to that paper.

In this paper, we study Liouville-type theorems of system (1.1) in a more general setting. That is, we assume that a, b might be negative and $s_1, s_2$ does not necessarily equal, which are not considered in the references mentioned above. We stress that when $a, b < 0$, the weights $|x|^a$ and $|x|^b$ in (1.1) are singular at zero which makes the problem more difficult. Specifically, the regularity of u and v at the origin is not clear which makes the techniques used in the references mentioned above invalid. In order to overcome this difficulty, we firstly need to choose an appropriate function space to give definitions of classical (super-)solutions.

Definition 1.3

Suppose $s_1, s_2 \in (0,1)$, $a, b \in {\mathbb {R}}$, $p, q > 0$. We call (u, v) is a positive classical (super-)solution of system (1.1), if $u \in {\mathcal {L}}_{2s_1} \cap C({\mathbb {R}}^N) \cap C_{loc}^{1,1} ({\mathbb {R}}^N \backslash \{0\})$, $v \in {\mathcal {L}}_{2s_2} \cap C({\mathbb {R}}^N) \cap C_{loc}^{1,1} ({\mathbb {R}}^N \backslash \{0\})$ with $u,v > 0$ in ${\mathbb {R}}^N \backslash \{0\}$ and (u, v) satisfies the system

$$\begin{aligned} \left\{ \begin{array}{ll} (-\Delta )^{s_1} u (\ge )= |x|^a v^p,&{}\quad x \in {\mathbb {R}}^N \backslash \{0\}, \\ (-\Delta )^{s_2} v (\ge )= |x|^b u^q,&{}\quad x \in {\mathbb {R}}^N \backslash \{0\}, \end{array}\right. \end{aligned}$$

(1.6)

In other words, solutions which blow up at zero are not in our consideration. And it can be easily seen from definition that positive classical solutions of system (1.1) are also positive classical super-solutions. So, our first main result is concerned with Liouville-type theorems for super-solutions.

Theorem 1.4

Suppose that $a, b > -N$, $p,q > 0$, $a \le -2s_1$ or $b \le -2s_2$, then system (1.1) has no positive classical super-solutions.

Now we only need consider the case $a > -2s_1$ and $b > -2s_2$. In this case, if $pq \ne 1$, we define

$$\begin{aligned} \alpha = \frac{2(s_1 + ps_2) + a + pb}{pq-1},\quad \beta =\frac{2(s_2 + q s_1) + b + q a}{pq-1}. \end{aligned}$$

(1.7)

Note that if $pq \ne 1$ and (u, v) is a solution of system (1.1), then $(R^\alpha u(Rx), R^\beta v(Rx))$ is also a solution of system (1.1) for any $R > 0$, this scaling invariance property is very crucial to our proof.

Theorem 1.5

Suppose that $a > -2s_1$ and $b > -2s_2$, $p > 1$ and $q > 1$, $\alpha > N-2s_1$ or $\beta > N-2s_2$. Then system (1.1) has no positive classical super-solutions.

Moreover, if we assume stronger in Theorem 1.5 that $s_1 = s_2$, $p = q$ and $a = b$, we find that if u is a positive classical super-solution of Eq. (1.3), then (u, v) with $u=v$ is a positive classical super-solution of system (1.1). As a result, we have the following Liouville-type theorem for super-solutions to Eq. (1.3), which is also a new result.

Corollary 1.6

Suppose that $a > -2\,s$, $p > 1$, $\alpha > N-2\,s$. Then Eq. (1.3) has no positive classical super-solutions.

Our next result is a Liouville-type theorem for classical solutions of system (1.1).

Theorem 1.7

Assume that $N \ge 2$, $s_1, s_2 \in (0,1)$, $a>-2s_1$, $b> -2s_2$, $0 < p \le \frac{ N + 2 s_1 + 2 a }{ N - 2 s_2 }$ and $0 < q \le \frac{ N + 2 s_2 + 2 b }{ N - 2 s_1 }$, $(p,q) \ne ( \frac{ N + 2 s_1 + 2 a }{ N - 2 s_2 }, \frac{ N + 2 s_2 + 2 b }{ N - 2 s_1 } )$. Suppose (u, v) is a pair of nonnegative classical solution to system (1.1), then $(u,v) \equiv (0,0)$ in ${\mathbb {R}}^N$.

Now we sketch ideas of proofs of main theorems. We argue by contradiction and assume that system (1.1) have nonnegative classical (super-)solutions. As mentioned before, in order to overcome the difficulties arising from singular weights of system (1.1), we firstly prove that nonnegative classical (super-)solutions for system (1.1) are also distributional (super-)solutions (see Lemma 2.1). Then we can prove the equivalence between the system (1.1) and a proper integral system (see Lemma 2.2 for super-solutions and Lemma 3.1 for solutions). Thus, we can directly obtain Theorem 1.4. To prove Theorem 1.5, we utilize the scaling invariance of system (1.1) to establish integral estimates of nonlinearities in a ball (see Lemma 2.3). We emphasize here that, because $(-\Delta )^s$ is a nonlocal operator, we need choose very suitable test functions when estimating integrals. This appears strikingly different from classical Laplace operator and also becomes more difficult. As for Theorem 1.7, we use scaling spheres method for integral systems to derive better lower bounds for u and v(see Theorem 3.3) and use them to get contradiction. The method of scaling spheres, developed in [16], comes from the method of moving spheres. The method of moving spheres was firstly used by Chen and Li [12] and then developed in [14, 32] and references therein. The method of scaling spheres is essentially a frozen variant of the method of moving spheres. It makes use of the integral representation formula of solutions and the “Bootstrap” technique to derive lower bound estimates of solutions at infinity. This method has been widely used to deal with varieties of problems such as Liouville-type theorem and classification of solutions to many equations or systems. For instance, in [11], authors use this method to establish Liouville-type theorems, integral representation formula and classification results for nonnegative solutions to fractional higher-order equations with general nonlinearities. For more applications, we refer interested readers to Le [22,23,24], Dai et al. [17], Peng [28] and references therein.

Throughout this paper, $C,C_1,C_2...$ denotes positive constant independent of (u, v) which may be different from line to line. $B_R(x)$ is the open ball with radius R and center $x_0$, supp f denotes the support of a Lebesgue measurable function f. ${\mathcal {F}}[f]$ denotes the Fourier transform of f.

The rest of this paper is organized as follows. In Sect. 2, we establish the equivalence between the system (1.1) and a proper integral system and prove Theorem 1.4 and Theorem 1.5 by integral estimates. Section 3 is devoted to prove Theorem 1.7 by combining the equivalence between the system (1.1) and an integral system with scaling spheres method.

2 Non-existence results for super-solutions

Lemma 2.1

Let $a, b > -N$ and (u, v) is a positive classical super-solution of system (1.1), then it is also a super-solution in the sense of distribution, i.e., for any $\varphi \in C_c^\infty ({\mathbb {R}}^N)$ with $\varphi \ge 0$,

$$\begin{aligned}&\int \nolimits _{{\mathbb {R}}^N} (-\Delta )^{s_1} u \varphi = \int \nolimits _{{\mathbb {R}}^N} u (-\Delta )^{s_1} \varphi \ge \int \nolimits _{{\mathbb {R}}^N} |x|^a v^p \varphi , \end{aligned}$$

(2.1)

$$\begin{aligned}&\int \nolimits _{{\mathbb {R}}^N} (-\Delta )^{s_2} v \varphi = \int \nolimits _{{\mathbb {R}}^N} v (-\Delta )^{s_2} \varphi \ge \int \nolimits _{{\mathbb {R}}^N} |x|^b u^q \varphi \end{aligned}$$

(2.2)

Proof

We only prove (2.1), since the proof of (2.2) is similar. Firstly, we claim that, for any $\varphi \in C_c^{\infty } ({\mathbb {R}}^N \backslash \{0\} )$ with $\varphi \ge 0$ and $u \in {\mathcal {L}}_{2s_1} \cap C_{loc}^{1,1}({\mathbb {R}}^N \backslash \{0\}) \cap C({\mathbb {R}}^N)$,

$$\begin{aligned} \int \nolimits _{{\mathbb {R}}^N} (-\Delta )^{s_1} u \varphi = \int \nolimits _{{\mathbb {R}}^N} u (-\Delta )^{s_1} \varphi \ge \int \nolimits _{{\mathbb {R}}^N} |x|^a v^p \varphi . \end{aligned}$$

(2.3)

In fact, we only need to show the first equality holds in (2.3). Take any arbitrary open set $\Omega _0$ compactly contained in ${\mathbb {R}}^N \backslash \{0\}$, there exists a sequence $\{ u_k \} \subset C_c^\infty ({\mathbb {R}}^N)$ uniformly bounded in $C^{1,1}({\mathbb {R}}^N \backslash \{0\})$, converging uniformly to u in $\Omega _0$ and also converging to u in the norm of ${\mathcal {L}}_{2s_1}$. By the uniform bound on the $C^{1,1}$-norm of $u_k$ in $\Omega _0$, it can be shown that the integrals converge uniformly in $\Omega _0$,

$$\begin{aligned} (-\Delta )^s u_k \rightarrow \int \nolimits _{{\mathbb {R}}^N} \frac{u(x)-u(y)}{|x-y|^{N+2s}} \textrm{d}y. \end{aligned}$$

But $(-\Delta )^s u_k \rightarrow (-\Delta )^s u$ in the topology of $C_c^\infty ({\mathbb {R}}^N)^\prime$, the dual of $C_c^\infty ({\mathbb {R}}^N)$. By the uniqueness of the limits, $\lim _{k \rightarrow +\infty } (-\Delta )^s u_k = (-\Delta )^s u$. Since $\Omega _0$ is arbitrary, this holds for any $x \in {\mathbb {R}}^N \backslash \{0\}$. By Parseval’s theorem and Fourier definition of the fractional Laplacian [13, p. 3], we have

$$\begin{aligned} \int \nolimits _{{\mathbb {R}}^N} (-\Delta )^s u \varphi&= \lim _{k \rightarrow +\infty } \int \nolimits _{{\mathbb {R}}^N} (-\Delta )^s u_k \varphi \nonumber \\&= \lim _{k \rightarrow +\infty } \int \nolimits _{{\mathbb {R}}^N} {\mathcal {F}} [ (-\Delta )^s u_k ] {\mathcal {F}}^{-1} [\varphi ] \nonumber \\&= \lim _{k \rightarrow +\infty } \int \nolimits _{{\mathbb {R}}^N} {\mathcal {F}} [u_k] \left( |\xi |^{2s} {\mathcal {F}}^{-1} [\varphi ] \right) \nonumber \\&= \lim _{k \rightarrow +\infty } \int \nolimits _{{\mathbb {R}}^N} u_k {\mathcal {F}} [ |\xi |^{2s} {\mathcal {F}}^{-1} [\varphi ]]\nonumber \\&= \lim _{k \rightarrow +\infty } \int \nolimits _{{\mathbb {R}}^N} u_k (-\Delta )^s \varphi \nonumber \\&= \int \nolimits _{{\mathbb {R}}^N} u (-\Delta )^s \varphi , \end{aligned}$$

(2.4)

which concludes (2.3).

Now, take $\phi \in C_c^\infty ({\mathbb {R}}^N)$ with $\phi \ge 0$, $\eta \in C_c^\infty ({\mathbb {R}}^N)$, $0 \le \eta \le 1$, and $\eta = 1$ in $B_1$, $\eta = 0$ in $B_2^c$. Let $\eta _j = 1 - \eta (jx)$ and $\phi _j = \phi \eta _j$. Then $\phi _j \in C_c^{\infty } ({\mathbb {R}}^N \backslash \{0\})$,

$$\begin{aligned} \int \nolimits _{{\mathbb {R}}^N} (-\Delta )^{s_1} u \phi _j = \int \nolimits _{{\mathbb {R}}^N} u (-\Delta )^{s_1} \phi _j \ge \int \nolimits _{{\mathbb {R}}^N} |x|^a v^p \phi _j. \end{aligned}$$

(2.5)

Firstly, we intend to prove

$$\begin{aligned} \lim _{j \rightarrow \infty } \int \nolimits _{{\mathbb {R}}^N} (-\Delta )^{s_1} u \phi _j =\int \nolimits _{{\mathbb {R}}^N} (-\Delta )^{s_1} u \phi . \end{aligned}$$

(2.6)

It is simply followed by Beppo–Levi monotone convergence theorem by observing that the sequence $\phi _j(x) \nearrow \phi (x)$.

Next, we prove

$$\begin{aligned} \lim _{j \rightarrow \infty } \int \nolimits _{{\mathbb {R}}^N} u (-\Delta )^{s_1} \phi _j = \int \nolimits _{{\mathbb {R}}^N} u (-\Delta )^{s_1} \phi . \end{aligned}$$

(2.7)

By calculation,

$$\begin{aligned} (-\Delta )^{s_1} \phi _j&= (-\Delta )^{s_1} ( \phi \eta _j ) (x) \nonumber \\&= \int \nolimits _{{\mathbb {R}}^N} \frac{ (\eta _j \phi )(x) -(\eta _j \phi )(y) }{ |x-y|^{N+2s_1} } \textrm{d}y\nonumber \\&= \int \nolimits _{{\mathbb {R}}^N} \frac{ \phi (x) \eta _j(x) -\phi (x) \eta _j(y) + \phi (x) \eta _j(y) - \phi (y) \eta _j(y)}{ |x-y|^{N+2s_1} } \textrm{d}y \nonumber \\&= \phi (x) (-\Delta )^{s_1} \eta _j (x) +\int \nolimits _{{\mathbb {R}}^N} \frac{\eta _j(y) (\phi (x) - \phi (y) )}{ |x-y|^{N+2s_1} } \textrm{d}y \nonumber \\&=: I_1 + I_2 \end{aligned}$$

(2.8)

For $x \in {\mathbb {R}}^N \backslash \{0\}$, we choose $R < |x|$, then

$$\begin{aligned} (-\Delta )^{s_1} \eta _j (x)&= \int \nolimits _{{\mathbb {R}}^N} \frac{\eta _j(x) - \eta _j(y)}{|x-y|^{N+2s_1}} \textrm{d}y \nonumber \\&= \int \nolimits _{ B_R(x) } \frac{\eta _j(x) - \eta _j(y)}{|x-y|^{N+2s_1}} \textrm{d}y + \int \nolimits _{{\mathbb {R}}^N \backslash B_R(x)} \frac{\eta _j(x) - \eta _j(y)}{|x-y|^{N+2s_1}} \textrm{d}y \nonumber \\&=: I_{11} + I_{12} \end{aligned}$$

(2.9)

For j sufficiently large and $x \in {\mathbb {R}}^N \backslash \{0\}$, $\eta _j = 1$ in $B_R(x)$, so

$$\begin{aligned} I_{11} = \int \nolimits _{B_R(x)} \frac{1-1}{ |x-y|^{N+2s_1} } \textrm{d}y = 0. \end{aligned}$$

(2.10)

As for $I_{12}$,

$$\begin{aligned} \left| \frac{ \eta _j(x) - \eta _j(y) }{ |x-y|^{N+2s_1} } \right| \le \frac{2}{ |x-y|^{N+2s_1} } \in L^1({\mathbb {R}}^N \backslash B_R(x)), \end{aligned}$$

(2.11)

Combining with the fact $\lim _{j \rightarrow \infty } \eta _j (x) = 1$ in ${\mathbb {R}}^N \backslash \{0\}$ and the Lebesgue dominated convergence theorem, we have

$$\begin{aligned} \lim _{j \rightarrow \infty } (-\Delta )^{s_1} \eta _j (x) = 0, \quad x \in {\mathbb {R}}^N \backslash \{0\}, \end{aligned}$$

(2.12)

which implies that $I_1 \rightarrow 0~\text {a.e.}~x \in {\mathbb {R}}^N$ as $j \rightarrow + \infty$. Similar to the discussion above, we have $I_2 \rightarrow (- \Delta )^{s_1} \phi ~ \text {a.e.}~ x \in {\mathbb {R}}^N$ as $j \rightarrow + \infty$. Thus, $(-\Delta )^{s_1} \phi _j \rightarrow (-\Delta )^{s_1} \phi ~ \text {a.e.}~ x \in {\mathbb {R}}^N$ as $j \rightarrow + \infty$.

For R sufficiently large,

$$\begin{aligned} \left| \int \nolimits _{B_R(0)} u (-\Delta )^{s_1} \phi _j - u (-\Delta )^{s_1} \phi \right|&\le \int \nolimits _{B_R(0)} \left| u (-\Delta )^{s_1} \phi _j - u (-\Delta )^{s_1} \phi \right| \nonumber \\&\le \Vert u \Vert _{L^1 ( B_R (0) ) } \Vert (-\Delta )^{s_1} \phi _j -(-\Delta )^{s_1} \phi \Vert _{L^\infty } \nonumber \\&\rightarrow 0,\quad \text {as}\quad j \rightarrow \infty . \end{aligned}$$

(2.13)

For $x \in B_R(0)^c$, recall that $u \in {\mathcal {L}}_{2{s_1}}$,

$$\begin{aligned} \left| u (-\Delta )^{s_1} \phi _j \right|&= \left| u \int \nolimits _{{\mathbb {R}}^N} \frac{ (\phi \eta _j)(x) - (\phi \eta _j)(y) }{ |x-y|^{N+2{s_1}} } \textrm{d}y \right| \nonumber \\&= \left| u \int \nolimits _{supp~\phi } \frac{ (\phi \eta _j)(x) -(\phi \eta _j)(y) }{ |x-y|^{N+2{s_1}} } \textrm{d}y \right| \nonumber \\&\le |u| \int \nolimits _{supp~\phi } \frac{C}{|x-y|^{N+2{s_1}}} \textrm{d}y \nonumber \\&\le \frac{C |u|}{|x|^{N+2{s_1}}} \in L^1({\mathbb {R}}^N \backslash B_R(0)). \end{aligned}$$

(2.14)

By Legesgue dominated convergence theorem,

$$\begin{aligned} \int \nolimits _{{\mathbb {R}}^N \backslash B_R(0)} u (-\Delta )^{s_1} \phi _j \rightarrow \int \nolimits _{{\mathbb {R}}^N \backslash B_R(0)} u (-\Delta )^{s_1} \phi , \quad \text {as}\quad j \rightarrow \infty . \end{aligned}$$

(2.15)

By (2.13) and (2.15), we can conclude (2.7). Finally, by $a > -N$, we can conclude that

$$\begin{aligned} \left| \int \nolimits _{{\mathbb {R}}^N} |x|^a v^p \phi _j - |x|^a v^p \phi \right|&\le \int \nolimits _{supp~\phi }\left| |x|^a v^p \phi _j - |x|^a v^p \phi \right| \nonumber \\&\le \Vert |x|^a \Vert _{L^1(supp~\phi )} \Vert v^p \phi _j - v^p \phi \Vert _{L^{\infty }}\nonumber \\&\rightarrow 0,\quad \text {as}\quad j \rightarrow +\infty . \end{aligned}$$

(2.16)

That is,

$$\begin{aligned} \lim _{j \rightarrow \infty } \int \nolimits _{{\mathbb {R}}^N} |x|^a v^p \phi _j =\int \nolimits _{{\mathbb {R}}^N} |x|^a v^p \phi . \end{aligned}$$

(2.17)

Combining (2.5)–(2.7) with (2.17), we can conclude (2.1). $\square$

Lemma 2.2

Assume $a, b > -N$, $u \in {\mathcal {L}}_{2s_1}$ and $v \in {\mathcal {L}}_{2s_2}$ are lower semi-continuous. Suppose that (u, v) is a positive classical super-solution of system (1.1), then

$$\begin{aligned} u(x) \ge C \int \nolimits _{{\mathbb {R}}^N} \frac{|y|^a v^p(y) }{|x-y|^{N-2s_1}} \textrm{d}y,\quad v(x) \ge C \int \nolimits _{{\mathbb {R}}^N} \frac{|y|^b u^q(y) }{|x-y|^{N-2s_2}} \textrm{d}y,\quad x \in {\mathbb {R}}^N \backslash \{0\}. \end{aligned}$$

(2.18)

For writing convenience, we will choose $C=1$ in the following use of system (2.18) since it only depends on N, $s_1$, $s_2$ and thus does not affect our proof.

Proof of Lemma 2.2

Let

$$\begin{aligned} u_R (x) = \int \nolimits _{B_R} G_R(x,y) |y|^a v^p \textrm{d}y, \end{aligned}$$

(2.19)

where $G_R (x,y)$ is the Green’s function on the ball $B_R(0)$ taking the form of

$$\begin{aligned} G_R(x,y) = \frac{C_{N,s}}{|x-y|^{N-2s_1}} \int \nolimits _{0}^{\frac{t_R}{s_R}} \frac{ b^{s - 1} }{ (1+b)^{N/2} } \textrm{d}b, \end{aligned}$$

(2.20)

with $s_R = \frac{ |x-y|^2 }{ R^2 }$ and $t_R = \left( 1 - \left| \frac{x}{R} \right| ^2 \right) \left( 1 - \left| \frac{y}{R} \right| ^2 \right)$. It is easy to see that

$$\begin{aligned} \left\{ \begin{array}{ll} (-\Delta )^{s_1} u_R (x) = |x|^a v^p,\quad &{}x \in B_R(0) , \\ u_R(x) = 0,\quad &{}x \notin B_R(0), \end{array}\right. \end{aligned}$$

(2.21)

in the sense of distribution. Let

$$\begin{aligned} w_R (x) = u(x) - u_R(x). \end{aligned}$$

Then

$$\begin{aligned} \left\{ \begin{array}{ll} (-\Delta )^{s_1} w_R (x) \ge 0,\quad &{}x \in B_R(0), \\ w_R(x) \ge 0,\quad &{}x \notin B_R(0), \end{array}\right. \end{aligned}$$

(2.22)

in the sense of distribution. By the maximum principle [35, Proposition 2.17], we have

$$\begin{aligned} w_R(x) \ge 0, \quad x \in B_R(0). \end{aligned}$$

(2.23)

As $R \rightarrow \infty$, $s_R \rightarrow 0$, $t_R \rightarrow 1$,

$$\begin{aligned} \int \nolimits _{0}^{\frac{t_R}{s_R}} \frac{ b^{s - 1} }{ (1+b)^{N/2} } \textrm{d}b \rightarrow \int \nolimits _{0}^{\infty } \frac{ b^{s - 1} }{ (1+b)^{N/2} } \textrm{d}b = C. \end{aligned}$$

From

$$\begin{aligned} G_R(x,y) \rightarrow G(x,y) = \frac{ C(N,s) }{ |x-y|^{N-2s_1} }, \quad R \rightarrow \infty , \end{aligned}$$

and

$$\begin{aligned} u_R(x) \rightarrow C(N,s) \int \nolimits _{{\mathbb {R}}^N} \frac{ |y|^a u^p (y) }{ |x-y|^{N-2s_1} } \textrm{d}y, \quad R \rightarrow \infty , \end{aligned}$$

we can get the first inequality of (2.18) and the second inequality is followed by the same argument as above. $\square$

Proof of Theorem 1.4

Suppose that (u, v) is a positive super-solution, by Lemma 2.2, u and v satisfy (2.18). When $p,q > 0$ and $a \le -2s_1$ or $b \le -2s_2$, $u(x) \ge \int \nolimits _{{\mathbb {R}}^N} \frac{|y|^a v^p(y) }{|x-y|^{N-2s_1}} \textrm{d}y = \infty$ or $v(x) \ge \int \nolimits _{{\mathbb {R}}^N} \frac{|y|^b u^q(y) }{|x-y|^{N-2s_2}} \textrm{d}y = \infty$, which is a contradiction. $\square$

In the following, we establish an integral estimate lemma which plays a key role in the proof of Theorem 1.5. Notice that the scaling invariance of (1.1) stated before Theorem 1.5 is very important to the following lemma.

Lemma 2.3

Suppose that $a > -2s_1$ and $b > -2s_2$, $p, q >1$ and (u, v) is a positive super-solution of system (1.1). Then there exists a positive constant $C = C(p,q,a,b,N,s)$ such that when $\max \{ a,b\} \le 0$,or $\max \{a,b\} > 0$ and $s_1 = s_2 =:s$,

$$\begin{aligned} \int \nolimits _{B_R} |x|^a v^p \le C R^{N-2s_1-\alpha },\quad \int \nolimits _{B_R} |x|^b u^q \le C R^{N-2s_2-\beta }, \end{aligned}$$

(2.24)

when $\max \{a,b\} > 0$ and $s_1 \ne s_2$,

$$\begin{aligned} \int \nolimits _{B_R \backslash B_{\frac{R}{2}}} |x|^a v^p \le C R^{N-2s_1-\alpha },\quad \int \nolimits _{B_R \backslash B_{\frac{R}{2}}} |x|^b u^q \le C R^{N-2s_2-\beta }. \end{aligned}$$

(2.25)

Proof

We first prove the case of $R = 1$.

Case (1). $\max \{ a,b \} > 0$ and $s_1 = s_2 = s$.

Without loss of generality, we assume $a = \max \{ a,b \} > 0$. Choose $\tilde{\varphi } > 0$ is the eigenfunction corresponding to the first eigenvalue of the equation

$$\begin{aligned} \left\{ \begin{array}{ll} (-\Delta )^s \varphi = \lambda |x|^a \varphi , \quad &{}\text {in}\quad B_2, \\ \varphi = 0, \quad &{}\text {in}~ {\mathbb {R}}^N \backslash B_2. \end{array}\right. \end{aligned}$$

(2.26)

On the one hand, by Lemma 2.1 and Hölder inequality,

$$\begin{aligned} \int \nolimits _{B_2} |x|^b u^q \tilde{\varphi }&\le \int \nolimits _{B_2} (-\Delta )^s v \tilde{\varphi } = \int \nolimits _{B_2} v (-\Delta )^s \tilde{\varphi } = \lambda _1 \int \nolimits _{B_2} |x|^a v \tilde{\varphi } \nonumber \\&\le C \left( \int \nolimits _{B_2} |x|^{a} v^p \tilde{\varphi } \right) ^{1/p} \left( \int \nolimits _{B_2} |x|^{a} \tilde{\varphi } \right) ^{1-1/p} \nonumber \\&\le C \left( \int \nolimits _{B_2} |x|^a v^p \tilde{\varphi } \right) ^{1/p}. \end{aligned}$$

(2.27)

On the other hand, in the similar way, we have

$$\begin{aligned} \int \nolimits _{B_2} |x|^a v^p \tilde{\varphi }&\le \int \nolimits _{B_2} (-\Delta )^s u \tilde{\varphi } = \int \nolimits _{B_2} u (-\Delta )^s \tilde{\varphi } = \lambda _1 \int \nolimits _{B_2} |x|^a u \tilde{\varphi } \nonumber \\&\le C \left( \int \nolimits _{B_2} |x|^b u^q \tilde{\varphi } \right) ^{1/q} \left( \int \nolimits _{B_2} |x|^{(a-b/q) (q/q-1)} \tilde{\varphi } \right) ^{1-1/q} \nonumber \\&\le C \left( \int \nolimits _{B_2} |x|^b u^q \tilde{\varphi } \right) ^{1/q}. \end{aligned}$$

(2.28)

By (2.27) and (2.28), we have

$$\begin{aligned} \int \nolimits _{B_2} |x|^b u^q \tilde{\varphi } \le C,\quad \int \nolimits _{B_2} |x|^a v^p \tilde{\varphi } \le C. \end{aligned}$$

(2.29)

Furthermore,

$$\begin{aligned} \int \nolimits _{B_1} |x|^a v^p \le C \int \nolimits _{B_1} |x|^a v^p \tilde{\varphi } \le C \int \nolimits _{B_2} |x|^a v^p \tilde{\varphi } \le C, \end{aligned}$$

(2.30)

and

$$\begin{aligned} \int \nolimits _{B_1} |x|^b u^q \le C \int \nolimits _{B_1} |x|^b u^q \tilde{\varphi } \le C \int \nolimits _{B_2} |x|^b u^q \tilde{\varphi } \le C. \end{aligned}$$

(2.31)

Case (2). $\max \{ a,b \} \le 0$.

Choose $m \in {\mathbb {N}}$ sufficiently large and $\eta \in C_c^\infty (B_2)$ with $0 \le \eta \le 1$ and $\eta = 1$ in $B_1$. By Lemma 2.1 and convexity of $\eta \mapsto \eta ^m$, we have

$$\begin{aligned}&\int \nolimits _{B_2} |x|^a v^p \eta ^m&\le \int \nolimits _{B_2} (-\Delta )^{s_1} u \eta ^m = \int \nolimits _{B_2} u (-\Delta )^{s_1} \eta ^m \le C \int \nolimits _{B_2} u \eta ^{m-1}, \nonumber \\&\int \nolimits _{B_2} |x|^b u^q \eta ^m&\le \int \nolimits _{B_2} (-\Delta )^{s_2} v \eta ^m = \int \nolimits _{B_2} v (-\Delta )^{s_2} \eta ^m \le C \int \nolimits _{B_2} v \eta ^{m-1}. \end{aligned}$$

(2.32)

By Hölder inequality and (2.32), we have

$$\begin{aligned} \int \nolimits _{B_2} |x|^a v^p \eta ^m&\le C \int \nolimits _{B_2} u \eta ^{m-1} \nonumber \\&\le C \left( \int \nolimits _{B_2} |x|^b u^q \eta ^m \right) ^{1/q} \left( \int \nolimits _{B_2} |x|^{-b/(q-1)} \eta ^{m - q/(q-1)}\right) ^{1 - 1/q} \nonumber \\&\le C \left( \int \nolimits _{B_2} |x|^b u^q \eta ^m \right) ^{1/q}. \end{aligned}$$

(2.33)

and

$$\begin{aligned} \int \nolimits _{B_2} |x|^b u^q \eta ^m&\le C \int \nolimits _{B_2} v \eta ^{m-1} \nonumber \\&\le C \left( \int \nolimits _{B_2} |x|^a v^p \eta ^m \right) ^{1/p} \left( \int \nolimits _{B_2} |x|^{-a/(p-1)} \eta ^{m - p/(p-1)}\right) ^{1 - 1/p} \nonumber \\&\le C \left( \int \nolimits _{B_2} |x|^a v^p \eta ^m \right) ^{1/p}. \end{aligned}$$

(2.34)

Combining (2.33) with (2.34), we obtain

$$\begin{aligned} \left( \int \nolimits _{B_2} |x|^a v^p \eta ^m \right) ^{1 - \frac{1}{pq}} \le C, \quad \left( \int \nolimits _{B_2} |x|^b u^q \eta ^m \right) ^{1 - \frac{1}{pq}} \le C, \end{aligned}$$

(2.35)

which implies

$$\begin{aligned} \int \nolimits _{B_1} |x|^a v^p \le \int \nolimits _{B_2} |x|^a v^p \eta ^m \le C, \quad \int \nolimits _{B_1} |x|^b u^q \le \int \nolimits _{B_2} |x|^b u^q \eta ^m \le C. \end{aligned}$$

(2.36)

Case (3). $\max \{ a,b \} > 0$ and $s_1 \ne s_2$.

Here, we choose $m \in {\mathbb {N}}$ sufficiently large and $\eta \in C_c^\infty (B_\frac{5}{4} \backslash B_\frac{1}{4})$ with $0 \le \eta \le 1$ and $\eta = 1$ in $B_1 \backslash B_\frac{1}{2}$ and repeat the argument in case (2).

Since $q \alpha = \beta + 2s_2 +b$, $p \beta = \alpha + 2s_1 + a$ and $( R^\alpha u(Rx), R^\beta v(Rx) )$ is also a super-solution, replacing u and v by $R^\alpha u(Rx)$ and $R^\beta v (Rx)$ in the above inequality respectively and changing variables yield

$$\begin{aligned} \int \nolimits _{B_R(\backslash B_\frac{R}{2})} |x|^b u^q = R^{N-2s_2-\beta } \int \nolimits _{B_1(\backslash B_\frac{1}{2})} |x|^b \left( R^\alpha u (Rx) \right) ^q \le C R^{N - 2s_2 - \beta }, \nonumber \\ \int \nolimits _{B_R(\backslash B_\frac{R}{2})} |x|^a v^p = R^{N-2s_1-\alpha } \int \nolimits _{B_1(\backslash B_\frac{1}{2})} |x|^a \left( R^\beta v (Rx) \right) ^p \le C R^{N - 2s_1 - \alpha }. \end{aligned}$$

(2.37)

$\square$

Proof of Theorem 1.5

Suppose that (u, v) is a positive super-solution and $\alpha > N-2s_1$. By Lemma 2.3, we can get two kinds of integral estimates (2.24) or (2.25).

In the case of (2.24), recalling that $\alpha > N-2s_1$,

$$\begin{aligned} \lim _{R \rightarrow \infty } \int \nolimits _{B_R} |x|^a v^p = 0. \end{aligned}$$

It follows that $v \equiv 0$ and $u \equiv 0$, which is a contradiction.

In the case of (2.25), by (2.18), we have, for $|x| \ge 1$

$$\begin{aligned} u(x)&\ge \int \nolimits _{|y| \le \frac{1}{2}} \frac{|y|^a}{|x-y|^{N-2s_1}} v^p(y) \textrm{d}y \nonumber \\&\ge \frac{C}{|x|^{N-2s_1}} \int \nolimits _{|y| \le \frac{1}{2}} |y|^a v^p(y) \textrm{d}y \nonumber \\&=: \frac{C}{|x|^{N-2s_1}}. \end{aligned}$$

(2.38)

Furthermore, again by (2.18),

$$\begin{aligned} v(x)&\ge \int \nolimits _{{\mathbb {R}}^N} \frac{ |y|^b u^q(y)}{|x-y|^{N-2s_2} } \textrm{d}y \nonumber \\&\ge \int \nolimits _{B_\frac{|x|}{2} (x)} \frac{ |y|^b u^q(y)}{|x-y|^{N-2s_2} } \textrm{d}y \nonumber \\&\ge C |x|^{2s_2 -N} \int \nolimits _{B_\frac{|x|}{2} (x)} |x|^b |y|^{q (2s_1 - N)} \textrm{d}y\nonumber \\&= C |x|^{ b - q (N-2s_1) + 2s_2}. \end{aligned}$$

(2.39)

Then,

$$\begin{aligned} \int \nolimits _{R \le |x|< 2R} |x|^a v^p \textrm{d}x&\ge \int \nolimits _{R \le |x| <2R} |x|^a |x|^{ pb - pq (N-2s_1) + 2ps_2 } \textrm{d}x \nonumber \\&\ge C R^{(pq-1) (\alpha -N +2s_1)} \rightarrow +\infty ,\quad \text {as}\quad R \rightarrow +\infty , \end{aligned}$$

(2.40)

which contradicts with (2.25). $\square$

3 Non-existence result for solutions

In this section, we only need consider the case of $a> -2s_1, b> -2s_2$ and $p, q > 0$.

Lemma 3.1

Assume that $u \in {\mathcal {L}}_{2s_1}$, $v \in {\mathcal {L}}_{2s_2}$ is lower semi-continuous. Suppose that (u, v) is a non-negative classical solution of system (1.1). Then

$$\begin{aligned} u(x) = \int \nolimits _{{\mathbb {R}}^N} \frac{|y|^a v^p(y) }{|x-y|^{N-2s_1}} \textrm{d}y,\quad v(x) = \int \nolimits _{{\mathbb {R}}^N} \frac{|y|^b u^q(y) }{|x-y|^{N-2s_2}} \textrm{d}y,\quad x \in {\mathbb {R}}^N \backslash \{0\}. \end{aligned}$$

(3.1)

Remark 3.2

Assume that $u \in C^{1,1}_{loc} ({\mathbb {R}}^N \backslash \{0\}) \cap C({\mathbb {R}}^N) \cap {\mathcal {L}}_{2\,s}$ with $u \ge 0$ and $(-\Delta )^s u \ge 0$ in ${\mathbb {R}}^N \backslash \{0\}$, then maximum principle implies that $u \equiv 0$ in ${\mathbb {R}}^N$ or $u > 0$ in ${\mathbb {R}}^N \backslash \{0\}$. By Lemma 3.1, we can get that $u(0) > 0$ in ${\mathbb {R}}^N$ if $u > 0$ in ${\mathbb {R}}^N \backslash \{0\}$. As a result, $u > 0$ in ${\mathbb {R}}^N$ or $u \equiv 0$ in ${\mathbb {R}}^N$.

Proof

With minor changes of the proof of Lemma 2.2, we can get that there exist constants $C_1, C_2 \ge 0$ such that

$$\begin{aligned} u(x)&= C_1 + \int \nolimits _{{\mathbb {R}}^N} \frac{|y|^a v^p(y)}{|x-y|^{N-2s_1}} \textrm{d}y,\quad x \in {\mathbb {R}}^N \backslash \{0\} \nonumber \\ v(x)&= C_2 + \int \nolimits _{{\mathbb {R}}^N} \frac{|y|^b u^q(y) }{|x-y|^{N-2s_2}} \textrm{d}y,\quad x \in {\mathbb {R}}^N \backslash \{0\}. \end{aligned}$$

(3.2)

Next, we will show $C_1 = 0$. If not, we assume that $C_1 > 0$, by (3.2), $u \ge C_1$ and

$$\begin{aligned} v \ge C_2 + C_1^p \int \nolimits _{{\mathbb {R}}^N} \frac{ |y|^b }{ |x-y|^{N-2s_2} } \textrm{d}y = + \infty , \end{aligned}$$

which is a contradiction. Similarly, we can show that $C_2 = 0$. $\square$

3.1 Scaling Sphere Methods

We consider the following integral system

$$\begin{aligned} \left\{ \begin{array}{l} u = \int \nolimits _{{\mathbb {R}}^N} \frac{ |y|^a v^p(y) }{ |x-y|^{N-2s_1} } \textrm{d}y, \\ v = \int \nolimits _{{\mathbb {R}}^N} \frac{ |y|^b u^q(y) }{ |x-y|^{N-2s_2} } \textrm{d}y. \end{array}\right. \end{aligned}$$

(3.3)

Suppose that $u, v \ge 0$ satisfy the system (3.3) but u, v is not identically zero. From Remark 3.2, we can see that (u, v) is actually a positive solution in ${\mathbb {R}}^N$. Moreover, we can show that (u, v) satisfy the following lower bound:

$$\begin{aligned} u(x) \ge \frac{C}{|x|^{N-2s_1}}, \quad v(x) \ge \frac{C}{|x|^{N-2s_2}} \quad \text {for} ~ |x| \ge 1. \end{aligned}$$

(3.4)

Indeed, since $u, v > 0$ satisfy the system (3.3), we have

$$\begin{aligned} u(x)&\ge C \int \nolimits _{|y| \le \frac{1}{2}} \frac{|y|^a}{|x-y|^{N-2s_1}} v^p(y) \textrm{d}y \nonumber \\&\ge \frac{C}{|x|^{N-2s_1}} \int \nolimits _{|y| \le \frac{1}{2}} |y|^a v^p(y) \textrm{d}y \nonumber \\&= \frac{C}{|x|^{N-2s_1}} \end{aligned}$$

(3.5)

for all $|x| \ge 1$. Similarly, we also have

$$\begin{aligned} v(x) \ge \frac{C}{|x|^{N-2s_2}} \quad \text {for~all}~ |x| \ge 1. \end{aligned}$$

(3.6)

Next, we will apply the scaling spheres method for integral systems to show the following lower bound estimates for a positive solution (u, v) which contradict with the integral system (3.3).

Theorem 3.3

Assume that $N \ge 2$, $s_1, s_2 \in (0,1)$, $a>-2s_1$, $b> -2s_2$, $0 < p \le \frac{ N + 2 s_1 + 2 a }{ N - 2 s_2 }$ and $0 < q \le \frac{ N + 2 s_2 + 2 b }{ N - 2 s_1 }$, $(p,q) \ne ( \frac{ N + 2 s_1 + 2 a }{ N - 2 s_2 }, \frac{ N + 2 s_2 + 2 b }{ N - 2 s_1 } )$. Suppose that (u, v) is a pair of nonnegative solution to system (3.3), then it satisfies the following lower bound estimates: for $|x| \ge 1$, if $pq < 1$,

$$\begin{aligned}&u(x) \ge C_{\kappa _u} |x|^{\kappa _u},\quad \forall ~ \kappa _u<\frac{(a+2s_1) + p(b+2s_2)}{1-pq}, \nonumber \\&v(x) \ge C_{\kappa _v} |x|^{\kappa _v},\quad \forall ~ \kappa _v < \frac{ (b+2s_2) + q(a+2s_1)}{1-pq}. \end{aligned}$$

(3.7)

If $pq \ge 1$,

$$\begin{aligned} u(x) \ge C_\kappa |x|^\kappa , \quad v(x) \ge C_\kappa |x|^\kappa , \quad \forall ~ \kappa < +\infty . \end{aligned}$$

(3.8)

Proof

Given any $\lambda > 0$, define the Kelvin transform of u, v centered at 0 by

$$\begin{aligned} u_\lambda (x) = \left( \frac{\lambda }{|x|} \right) ^{ N-2s_1 } u \left( \frac{ \lambda ^2 x }{ |x|^2 } \right) , \quad v_\lambda (x) = \left( \frac{\lambda }{|x|} \right) ^{ N-2s_2 } v \left( \frac{ \lambda ^2 x }{ |x|^2 } \right) . \end{aligned}$$

(3.9)

For $x \in {\mathbb {R}}^N \backslash \{0\}$, define

$$\begin{aligned} w_{\lambda ,u} (x) := u_\lambda (x) - u (x), \quad w_{\lambda ,v} (x) := v_\lambda (x) - v(x). \end{aligned}$$

(3.10)

We will first show that, for $\lambda > 0$ sufficiently small,

$$\begin{aligned} w_{\lambda ,u} \ge 0, \quad w_{\lambda ,v} \ge 0,\quad \forall x \in B_\lambda (0) \backslash \{0\}. \end{aligned}$$

(3.11)

Step 1. Start dilating the sphere from near $\lambda =0$

Define

$$\begin{aligned} B_{\lambda ,u}^{-} := \{ x \in B_\lambda (0) \backslash \{0\} | w_{\lambda ,u}< 0 \}, \quad B_{\lambda ,v}^{-} := \{ x \in B_\lambda (0) \backslash \{0\} | w_{\lambda ,v} < 0 \}. \end{aligned}$$

(3.12)

We will show that, for $\lambda > 0$ sufficiently small,

$$\begin{aligned} B_{ \lambda , u }^{-} = B_{ \lambda , v }^{-} = \emptyset . \end{aligned}$$

(3.13)

Since $u,v \in C( {\mathbb {R}}^N )$ is a positive solution to integral system (3.3), we get

$$\begin{aligned} u(x)&= \int \nolimits _{B_\lambda (0)} \frac{|y|^a v^p(y)}{|x-y|^{N-2 s_1}} \textrm{d}y + \int \nolimits _{B_\lambda (0)} \frac{|y|^a }{ \left| \frac{|y|}{\lambda } x - \frac{\lambda }{|y|} y \right| ^{N-2s_1} } \left( \frac{\lambda }{|y|} \right) ^{\tau _a} v_\lambda ^{p} \textrm{d}y,\nonumber \\ v(x)&= \int \nolimits _{B_\lambda (0)} \frac{|y|^b u^q(y)}{|x-y|^{N-2 s_2}} \textrm{d}y + \int \nolimits _{B_\lambda (0)} \frac{|y|^b }{ \left| \frac{|y|}{\lambda } x - \frac{\lambda }{|y|} y \right| ^{N-2s_2} } \left( \frac{\lambda }{|y|} \right) ^{\tau _b} u_\lambda ^{q} \textrm{d}y, \end{aligned}$$

(3.14)

where $\tau _a:= N + 2s_1 + 2a - p(N-2s_2)$, $\tau _b:= N + 2s_2 + 2b - q(N-2s_1)$. $u_\lambda$ and $v_\lambda$ satisfy the integral system for $x \in {\mathbb {R}}^N \backslash \{0\}$

$$\begin{aligned} \left\{ \begin{array}{ll} &{}u_\lambda (x) = \int \nolimits _{{\mathbb {R}}^N} \frac{|y|^a}{|x-y|^{N-2s_1}} \left( \frac{\lambda }{|y|} \right) ^{\tau _a} v_\lambda ^p \textrm{d}y,\\ &{}v_\lambda (x) = \int \nolimits _{{\mathbb {R}}^N} \frac{|y|^b}{|x-y|^{N-2s_2}} \left( \frac{\lambda }{|y|} \right) ^{\tau _b} u_\lambda ^q \textrm{d}y. \end{array}\right. \end{aligned}$$

(3.15)

It follows that

$$\begin{aligned} u_\lambda (x)&= \int \nolimits _{B_\lambda (0)} \frac{|y|^a}{\left| \frac{|y|}{\lambda } x - \frac{\lambda }{|y|} y \right| ^{N-2s_1} } v^p(y) + \int \nolimits _{B_\lambda (0)} \frac{|y|^a}{|x-y|^{N-2s_1}} \left( \frac{\lambda }{|y|} \right) ^{\tau _a} v_\lambda ^p (y) \textrm{d}y, \nonumber \\ v_\lambda (x)&= \int \nolimits _{B_\lambda (0)} \frac{|y|^b}{\left| \frac{|y|}{\lambda } x - \frac{\lambda }{|y|} y \right| ^{N-2s_2} } u^q(y) + \int \nolimits _{B_\lambda (0)} \frac{|y|^b}{|x-y|^{N-2s_2}} \left( \frac{\lambda }{|y|} \right) ^{\tau _b} u_\lambda ^q (y) \textrm{d}y. \end{aligned}$$

(3.16)

By (3.14) and (3.16), we can derive that, for $x \in B_{\lambda ,u}^{-}$,

$$\begin{aligned} 0&> w_{\lambda ,u} (x) = u_\lambda (x) - u (x) \nonumber \\&= \int \nolimits _{B_\lambda (0)} \left( \frac{|y|^a}{|x-y|^{N-2s_1}} - \frac{|y|^a}{\left| \frac{|y|}{\lambda } x - \frac{\lambda }{|y|} y \right| ^{N-2s_1}} \right) \nonumber \\&\quad \times \left( \left( \frac{\lambda }{|y|} \right) ^{\tau _a} v_\lambda ^p (y) - v^p(y) \right) \textrm{d}y \nonumber \\&\ge \int \nolimits _{B_{\lambda ,v}^{-}} \left( \frac{|y|^a}{|x-y|^{N-2s_1}} - \frac{|y|^a}{\left| \frac{|y|}{\lambda } x - \frac{\lambda }{|y|} y \right| ^{N-2s_1}} \right) \nonumber \\&\quad \max \{ v^{p-1} (y) , v_\lambda ^{p-1} (y)\} w_{\lambda ,v} (y) \textrm{d}y \nonumber \\&\ge \int \nolimits _{B_{\lambda ,v}^{-}} \frac{|y|^a}{|x-y|^{N-2s_1}} \max \{ v^{p-1} (y) , v_\lambda ^{p-1} (y) \} w_{\lambda ,v} (y) \textrm{d}y. \end{aligned}$$

(3.17)

In the third line of (3.17), we have used the fact that if $y \in B_{\lambda ,v}^{-}$, then

$$\begin{aligned} \max \{ v^{p-1}(y) , v_\lambda ^{p-1}(y) \} = v^{p-1}(y) \end{aligned}$$

if $p \ge 1$ and $\max \{ v^{p-1}(y), v_\lambda ^{p-1}(y) \} =v_\lambda ^{p-1}(y)$ if $p \le 1$. Similarly, for $x \in B_{\lambda ,v}^{-}$,

$$\begin{aligned} 0&> w_{\lambda ,v} (x) = v_\lambda (x) - v (x) \nonumber \\&= \int \nolimits _{B_\lambda (0)} \left( \frac{|y|^b}{|x-y|^{N-2s_2}} - \frac{|y|^b}{\left| \frac{|y|}{\lambda } x -\frac{\lambda }{|y|} y \right| ^{N-2s_2}}\right) \nonumber \\&\quad \times \left( \left( \frac{\lambda }{|y|} \right) ^{\tau _b} u_\lambda ^p (y) - u^p(y) \right) \textrm{d}y \nonumber \\&\ge \int \nolimits _{B_{\lambda ,u}^{-}} \left( \frac{|y|^b}{|x-y|^{N-2s_2}} - \frac{|y|^b}{\left| \frac{|y|}{\lambda } x -\frac{\lambda }{|y|} y \right| ^{N-2s_2}} \right) \nonumber \\&\quad \max \{ u^{q-1} (y) , u_\lambda ^{q-1} (y) \} w_{\lambda ,u} (y) \textrm{d}y \nonumber \\&\ge \int \nolimits _{B_{\lambda ,u}^{-}} \frac{|y|^b}{|x-y|^{N-2s_2}} \max \{u^{q-1} (y) , u_\lambda ^{q-1} (y) \} w_{\lambda ,u} (y) \textrm{d}y. \end{aligned}$$

(3.18)

By Hardy–Littlewood–Sobolev inequality, (3.17) and (3.18), we have, for any $\max \{ \frac{N}{N-2s_1}, \frac{N}{N-2s_2} \}< r < +\infty$,

$$\begin{aligned} \Vert w_{\lambda ,u} \Vert _{L^{r} (B_{\lambda ,u}^{-})}&\le C \Vert |x|^a \max \{ v^{p-1} (y) , v_\lambda ^{p-1} (y) \} w_{\lambda ,v} \Vert _{L^{ \frac{N r}{ N + 2r s_{1}} }(B_{\lambda ,v}^{-}) } \nonumber \\&\le C \Vert |x|^a \max \{ v^{p-1} (y) , v_\lambda ^{p-1} (y) \} \Vert _{L^{\frac{N}{2s_1}}(B_{\lambda ,v}^{-})} \Vert w_{\lambda ,v} \Vert _{ L^{r}(B_{\lambda ,v}^{-}) }, \end{aligned}$$

(3.19)

$$\begin{aligned} \Vert w_{\lambda ,v} \Vert _{L^{r} (B_{\lambda ,v}^{-})}&\le C \Vert |x|^b \max \{ u^{q-1} (y) , u_\lambda ^{q-1} (y) \} w_{\lambda ,u} \Vert _{L^{ \frac{N r }{N + 2 r s_2} }(B_{\lambda ,u}^{-}) } \nonumber \\&\le C \Vert |x|^b \max \{ u^{q-1} (y) , u_\lambda ^{q-1} (y) \} \Vert _{L^{\frac{N}{2s_2}}(B_{\lambda ,u}^{-})} \Vert w_{\lambda ,u} \Vert _{ L^{r}(B_{\lambda ,u}^{-}) }. \end{aligned}$$

(3.20)

Since (3.4) implies that

$$\begin{aligned} \inf _{ x \in B_{\lambda ,u} (0) \backslash \{0\} } u_\lambda (x) \ge C \quad \text {and} \quad \inf _{ x \in B_{\lambda ,v} (0) \backslash \{0\} } v_\lambda (x) \ge C \end{aligned}$$

(3.21)

for any $\lambda \le 1$, and so

$$\begin{aligned} \sup _{ x \in B_{\lambda ,u} (0) \backslash \{0\} } u_\lambda ^{q-1} (x) \le C \quad \text {and} \quad \sup _{ x \in B_{\lambda ,v} (0) \backslash \{0\} } v_\lambda ^{p-1} (x) \le C \end{aligned}$$

(3.22)

whenever $p \le 1$, $q \le 1$ and $\lambda \le 1$.

It follows from (3.19) and (3.20) that,

$$\begin{aligned} \Vert w_{\lambda ,v} \Vert _{L^{r} (B_{\lambda ,v}^{-})}&\le C \Vert |x|^b \max \{ u^{q-1} (y) , u_\lambda ^{q-1} (y) \} \Vert _{L^{\frac{N}{2s_2}}(B_{\lambda ,u}^{-})} \nonumber \\&\quad \times \Vert |x|^a \max \{ v^{p-1} (y) , v_\lambda ^{p-1} (y) \} \Vert _{L^{\frac{N}{2s_1}}(B_{\lambda ,v}^{-})} \Vert w_{\lambda ,v} \Vert _{ L^{r}(B_{\lambda ,v}^{-}) }. \end{aligned}$$

(3.23)

Since $a, b > -2s$, there exists $\epsilon _0 > 0$ small enough such that

$$\begin{aligned}&C \Vert |x|^b \max \{ u^{q-1} (y) , u_\lambda ^{q-1} (y) \} \Vert _{L^{\frac{N}{2s_2}}(B_{\lambda ,u}^{-})} \nonumber \\&\quad \times \Vert |x|^a \max \{ v^{p-1} (y) , v_\lambda ^{p-1} (y) \} \Vert _{L^{\frac{N}{2s_1}}(B_{\lambda ,v}^{-})} \le \frac{1}{2} \end{aligned}$$

(3.24)

for all $0< \lambda < \epsilon _0$. Thus,

$$\begin{aligned} \Vert w_{\lambda ,u} \Vert _{ L^r (B_{\lambda ,u}^{-}) } = \Vert w_{\lambda ,v} \Vert _{ L^r (B_{\lambda ,v}^{-}) } = 0, \end{aligned}$$

(3.25)

which implies that $B_{\lambda ,u}^{-} = B_{\lambda ,v}^{-} = \emptyset$ for all $0 < \lambda \le \epsilon _0$.

Step 2. Dilate the sphere $S_\lambda$ until $\lambda = + \infty$ to derive lower bound estimates on (u, v)

Based on Step 1, now we dilate the sphere $S_\lambda$ outward as long as (3.11) holds. Let

$$\begin{aligned} \lambda _0 := \sup \{ \lambda >0 | w_{\mu ,u} \ge 0\quad \text {and} \quad w_{\mu ,v} \ge 0\quad \text {in} \quad B_{\mu } (0) \backslash \{0\},~ \forall ~0 < \mu \le \lambda \}. \end{aligned}$$

(3.26)

In the following, we will prove that $\lambda _0 = +\infty$. If not, $\lambda _0 < +\infty$, we first prove that

$$\begin{aligned} w_{\lambda _0,u}> 0 \quad \text {and} \quad w_{\lambda _0,v} > 0 \end{aligned}$$

(3.27)

for all $x \in B_{\lambda _0} (0) \backslash \{0\}$. Indeed, if we assume that $w_{\lambda _0,u} = w_{\lambda _0,v} = 0$, we have, for any $x \in B_{\lambda _0} (0) \backslash \{0\}$,

$$\begin{aligned} 0&= w_{\lambda _0, u}(x) = u_{\lambda _0} (x) - u(x) \nonumber \\&= \int \nolimits _{B_{\lambda _0}} \left( \frac{|y|^a}{|x-y|^{N-2s_1}} - \frac{|y|^a}{\left| \frac{|y|}{\lambda } x - \frac{\lambda }{|y|} y \right| ^{N-2s_1}} \right) \left( \left( \frac{\lambda _0}{|y|} \right) ^{\tau _a} - 1 \right) v^p(y) \textrm{d}y > 0, \end{aligned}$$

(3.28)

$$\begin{aligned} 0&= w_{\lambda _0, v}(x) = v_{\lambda _0} (x) - v(x) \nonumber \\&= \int \nolimits _{B_{\lambda _0}} \left( \frac{|y|^b}{|x-y|^{N-2s_2}} - \frac{|y|^b}{\left| \frac{|y|}{\lambda } x - \frac{\lambda }{|y|} y \right| ^{N-2s_2}} \right) \left( \left( \frac{\lambda _0}{|y|} \right) ^{\tau _b} - 1 \right) u^q(y) \textrm{d}y > 0. \end{aligned}$$

(3.29)

So, there exists a $x_0 \in B_{\lambda _0} \backslash \{0\}$ such that $w_{\lambda _0,u} (x_0) > 0$ or $w_{\lambda _0,v} (x_0) > 0$. Assume that $w_{\lambda _0,u} (x_0) > 0$, by the second equality of (3.29), we have $w_{\lambda _0,v} (x_0) > 0$. By continuity, there exists $\delta > 0$, a constant $l_u, l_v > 0$ such that

$$\begin{aligned} B_\delta (x_0) \subset B_{\lambda _0} (0) \backslash \{0\},\quad w_{\lambda _0,u} \ge l_u> 0 \quad \text {and}\quad w_{\lambda _0,v} \ge l_v > 0 \end{aligned}$$

(3.30)

From (3.30) and the system (3.14) and (3.16), we have, for any $x \in B_{\lambda _0} (0) \backslash \{0\}$,

$$\begin{aligned} w_{\lambda _0,u} (x)&= u_{\lambda _0} (x) - u (x) \nonumber \\&= \int \nolimits _{B_{\lambda _0} (0)} \left( \frac{|y|^a}{|x-y|^{N-2s_1}} - \frac{|y|^a}{\left| \frac{|y|}{\lambda _0} x - \frac{\lambda _0}{|y|} y \right| ^{N-2s_1}} \right) \nonumber \\&\quad \times \left( \left( \frac{\lambda _0}{|y|} \right) ^{\tau _a} v_{\lambda _0}^p (y) - v^p(y) \right) \textrm{d}y \nonumber \\&\ge \int \nolimits _{B_{\lambda _0} (0)} \left( \frac{|y|^a}{|x-y|^{N-2s_1}} - \frac{|y|^a}{\left| \frac{|y|}{\lambda _0} x -\frac{\lambda _0}{|y|} y \right| ^{N-2s_1}} \right) \nonumber \\&\quad \min \{ v^{p-1} (y) , v_{\lambda _0}^{p-1} (y) \} w_{\lambda _0,v} (y) \textrm{d}y \nonumber \\&\ge \int \nolimits _{B_\delta (x_0)} \left( \frac{|y|^a}{|x-y|^{N-2s_1}} - \frac{|y|^a}{\left| \frac{|y|}{\lambda _0} x - \frac{\lambda _0}{|y|} y \right| ^{N-2s_1}} \right) \nonumber \\&\quad \min \{ v^{p-1} (y) , v_{\lambda _0}^{p-1} (y) \} w_{\lambda _0,v} (y) \textrm{d}y \nonumber \\&>0, \end{aligned}$$

(3.31)

$$\begin{aligned} w_{\lambda _0,v} (x)&= v_{\lambda _0} (x) - v (x) \nonumber \\&= \int \nolimits _{B_{\lambda _0} (0)} \left( \frac{|y|^b}{|x-y|^{N-2s_2}} - \frac{|y|^b}{\left| \frac{|y|}{\lambda _0} x - \frac{\lambda _0}{|y|} y \right| ^{N-2s_2}} \right) \nonumber \\&\quad \times \left( \left( \frac{\lambda _0}{|y|} \right) ^{\tau _b} u_{\lambda _0}^q (y) - u^q(y) \right) \textrm{d}y \nonumber \\&\ge \int \nolimits _{B_{\lambda _0} (0)} \left( \frac{|y|^b}{|x-y|^{N-2s_2}} - \frac{|y|^b}{\left| \frac{|y|}{\lambda _0} x - \frac{\lambda _0}{|y|} y \right| ^{N-2s_2}} \right) \nonumber \\&\quad min \{ u^{q-1} (y) , u_{\lambda _0}^{q-1} (y) \} w_{\lambda _0,u} (y) \textrm{d}y \nonumber \\&\ge \int \nolimits _{B_\delta (x_0)} \left( \frac{|y|^b}{|x-y|^{N-2s_2}} - \frac{|y|^b}{\left| \frac{|y|}{\lambda _0} x - \frac{\lambda _0}{|y|} y \right| ^{N-2s_2}} \right) \nonumber \\&\quad \min \{ u^{q-1} (y) , u_{\lambda _0}^{q-1} (y) \} \ w_{\lambda _0,u} (y) \textrm{d}y \nonumber \\&>0, \end{aligned}$$

(3.32)

which verify (3.27).

Next, we will show that there exists $\varepsilon > 0$ small enough such that $w_{\lambda ,u} \ge 0$ and $w_{ \lambda , v} \ge 0$ in $B_\lambda (0) \backslash \{0\}$ for all $\lambda \in [ \lambda _0, \lambda _0+\varepsilon ]$. Once this is proved, we can obtain a contradiction with (3.26).

By (3.31) and (3.32), there exists a $0< \eta < \lambda _0$ small enough and $\tilde{l}_u, \tilde{l}_v > 0$, such that, for any $x \in \overline{B_\eta (0)} \backslash \{0\}$,

$$\begin{aligned} w_{\lambda _0,u} (x)&\ge \tilde{l}_u> 0, \nonumber \\ w_{\lambda _0,v} (x)&\ge \tilde{l}_v > 0 \end{aligned}$$

(3.33)

We fix $0< r_0 < \frac{\lambda _0}{2}$ small enough, such that

$$\begin{aligned}&C \Vert |x|^a \max \{ v^{p-1},v_\lambda ^{p-1} \} \Vert _{L^{ \frac{N}{2s_1}} (A_{\lambda _0 + r_0 , 2r_0}) } \nonumber \\&\quad \times \Vert |x|^b \max \{ u^{q-1},u_\lambda ^{q-1} \} \Vert _{L^{ \frac{N}{2s_2} } (A_{\lambda _0 + r_0 , 2r_0}) } \le \frac{1}{2}, \end{aligned}$$

(3.34)

where the constant C is the same as in (3.23) and $A_{\lambda _0 + r_0, 2r_0}$ is defined as

$$\begin{aligned} A_{\lambda _0 + r_0 , 2r_0} := \{ x \in B_{\lambda _0 + r_0} (0) | |x| > \lambda _0 - r_0 \}. \end{aligned}$$

(3.35)

By (3.17) and (3.18), we can also deduce that for any $\lambda \in [ \lambda _0, \lambda _0 + r_0 ]$ and $\max \{ \frac{N}{N-2s_1}, \frac{N}{N-2s_2} \}< r < \infty$,

$$\begin{aligned} \Vert w_{\lambda ,u} \Vert _{ L^r (B_{\lambda ,u}^{-}) }&\le C \Vert |x|^a \max \{ v^{p-1},v_\lambda ^{p-1} \} \Vert _{ L^\frac{N}{2s_1} (B_{\lambda ,v}^{-}) } \Vert w_{\lambda ,v} \Vert _{ L^r (B_{\lambda ,v}^{-})}, \nonumber \\ \Vert w_{\lambda ,v} \Vert _{ L^r (B_{\lambda ,v}^{-}) }&\le C \Vert |x|^b \max \{ u^{q-1},u_\lambda ^{q-1} \} \Vert _{ L^\frac{N}{2s_2} (B_{\lambda ,u}^{-}) } \Vert w_{\lambda ,u} \Vert _{ L^r (B_{\lambda ,u}^{-})}. \end{aligned}$$

(3.36)

Furthermore,

$$\begin{aligned} \Vert w_{\lambda ,u} \Vert _{ L^r (B_{\lambda ,u}^{-}) }&\le C \Vert |x|^a \max \{ v^{p-1},v_\lambda ^{p-1} \} \Vert _{ L^\frac{N}{2s_1} (B_{\lambda ,v}^{-}) } \nonumber \\&\quad \Vert |x|^b \max \{ u^{q-1},u_\lambda ^{q-1} \} \Vert _{ L^\frac{N}{2s_2} (B_{\lambda ,u}^{-}) } \Vert w_{\lambda ,u} \Vert _{ L^r (B_{\lambda ,u}^{-}) }, \nonumber \\ \Vert w_{\lambda ,v} \Vert _{ L^r (B_{\lambda ,v}^{-})}&\le C \Vert |x|^a \max \{ v^{p-1},v_\lambda ^{p-1} \} \Vert _{ L^\frac{N}{2s_1} (B_{\lambda ,v}^{-}) } \nonumber \\&\quad \Vert |x|^b \max \{ u^{q-1},u_\lambda ^{q-1} \} \Vert _{ L^\frac{N}{2s_2} (B_{\lambda ,u}^{-}) } \Vert w_{\lambda ,v} \Vert _{ L^r (B_{\lambda ,v}^{-}) }. \end{aligned}$$

(3.37)

From (3.30) and (3.33), we can deduce that

$$\begin{aligned} m_u := \inf _{ x \in \overline{ B_{\lambda _0 - r_0} (0) } \backslash \{0\} } w_{\lambda _0,u} (x)> 0,\quad \text {and} \quad m_v := \inf _{ x \in \overline{ B_{\lambda _0 - r_0} (0) } \backslash \{0\} } w_{\lambda _0,v} (x) > 0. \end{aligned}$$

(3.38)

Since u and v are uniformly continuous on arbitrary $K \subset {\mathbb {R}}^N$, for example, $K = \overline{ B_{4 \lambda _0} (0) }$, we claim that, there exists a $0< \varepsilon _1 < r_0$ small enough, satisfying that for any $\lambda \in [ \lambda _0, \lambda _0 + \varepsilon _1 ]$,

$$\begin{aligned} w_{ \lambda ,u } (x) \ge \frac{m_u}{2}> 0 \quad \text {and} \quad w_{ \lambda ,v } (x) \ge \frac{m_v}{2} > 0,\quad x \in \overline{ B_{\lambda _0 - r_0} (0)} \backslash \{0\}. \end{aligned}$$

(3.39)

In order to prove this, we observe that (3.38) is equivalent to

$$\begin{aligned} |x|^{N-2s_1} u(x) - \lambda _0^{N-2s_1} u(x^{\lambda _0})&\ge m_u \lambda _0^{N-2s_1},\quad |x| \ge \frac{\lambda _0^2}{\lambda _0 - r_0}, \nonumber \\ |x|^{N-2s_2} v(x) - \lambda _0^{N-2s_2} v(x^{\lambda _0})&\ge m_v \lambda _0^{N-2s_2},\quad |x| \ge \frac{\lambda _0^2}{\lambda _0 - r_0}. \end{aligned}$$

(3.40)

where the reflection of x about the sphere $\partial B_{\lambda }(0)$ is defined by $x^\lambda := \frac{\lambda ^2 x}{|x|^2}$ for any $\lambda > 0$. Since u, v are uniformly continuous on $\overline{ B_{4\lambda _0} (0) }$, by (3.40), there exists a $0< \varepsilon _1 < r_0$ small enough such that for any $\lambda \in [ \lambda _0, \lambda _0 + \varepsilon _1 ]$,

$$\begin{aligned} |x|^{N-2s_1} u(x) - \lambda ^{N-2s_1} u(x^{\lambda })&\ge \frac{m_u}{2} \lambda ^{N-2s_1},\quad |x| \ge \frac{\lambda ^2}{\lambda _0 - r_0}, \nonumber \\ |x|^{N-2s_2} v(x) - \lambda ^{N-2s_2} v(x^{\lambda })&\ge \frac{m_v}{2} \lambda ^{N-2s_2},\quad |x| \ge \frac{\lambda ^2}{\lambda _0 - r_0}. \end{aligned}$$

(3.41)

which verify (3.39). By (3.41), we can see that for any $\lambda \in [ \lambda _0, \lambda _0 + \varepsilon _1 ]$,

$$\begin{aligned} B_{\lambda ,u}^{-} \subset A_{\lambda _0+r_0 , 2r_0} \quad \text {and} \quad B_{\lambda ,v}^{-} \subset A_{\lambda _0+r_0 , 2r_0}. \end{aligned}$$

(3.42)

Therefore, by (3.34) and (3.37),

$$\begin{aligned} \Vert w_{\lambda , u} \Vert _{L^r (B_{\lambda ,u}^{-})} = \Vert w_{\lambda , v} \Vert _{L^r (B_{\lambda ,v}^{-})} = 0, \end{aligned}$$

(3.43)

which means for $\lambda \in [ \lambda _0, \lambda _0 + \varepsilon _1 ]$, $B_{\lambda ,u}^{-} = B_{\lambda ,v}^{-} = \emptyset$. That is,

$$\begin{aligned} w_{\lambda ,u} \ge 0 \quad \text {and} \quad w_{\lambda ,v} \ge 0 \quad x \in B_{\lambda (0)} \backslash \{0\}, \end{aligned}$$

(3.44)

which contradicts with (3.26). Now, there holds for any $0< \lambda < +\infty$ and $|x| \ge \lambda$,

$$\begin{aligned} u(x)&\ge \left( \frac{\lambda }{|x|} \right) ^{N-2s_1} u \left( \frac{\lambda ^2 x}{|x|^2} \right) ,\nonumber \\ v(x)&\ge \left( \frac{\lambda }{|x|} \right) ^{N-2s_2} v \left( \frac{\lambda ^2 x}{|x|^2} \right) . \end{aligned}$$

(3.45)

For $|x| \ge 1$, choose $\lambda := \sqrt{ |x| }$, then for any $0< \lambda < +\infty$ and $|x| \ge \lambda$,

$$\begin{aligned} u(x)&\ge \frac{1}{ |x|^{ \frac{N-2s_1}{2} } } u \left( \frac{x}{|x|} \right) , \nonumber \\ v(x)&\ge \frac{1}{ |x|^{ \frac{N-2s_2}{2} } } v \left( \frac{x}{|x|} \right) . \end{aligned}$$

(3.46)

There exists a $C_0 > 0$ such that for $|x| \ge 1$,

$$\begin{aligned} u(x)&\ge \left( \min _{ |x|=1 } u(x) \right) \frac{1}{ |x|^{ \frac{N-2s_1}{2} } } =: \frac{C_0}{ |x|^{ \frac{N-2s_1}{2} } }, \nonumber \\ v(x)&\ge \left( \min _{ |x|=1 } v(x) \right) \frac{1}{ |x|^{ \frac{N-2s_2}{2} } } =: \frac{C_0}{ |x|^{ \frac{N-2s_2}{2} } }. \end{aligned}$$

(3.47)

Now, we use the bootstrap iteration argument to improve the lower bound estimate (3.47). Let $\mu _{0,u}:= \frac{N-2s_1}{2}$ and $\mu _{0.v}:= \frac{N-2s_2}{2}$, by (3.47) and (3.16), for $|x| \ge 1$,

$$\begin{aligned} u(x)&\ge \int \nolimits _{2|x| \le |y| \le 4|x|} \frac{1}{ |x-y|^{N-2s_1} |y|^{ p \mu _{0,v} - a } } \textrm{d}y \nonumber \\&\ge \frac{C_0}{|x|^{N-2s_1}} \int \nolimits _{2|x| \le |y| \le 4|x|} \frac{1}{|y|^{ p \mu _{0,v} - a }} \textrm{d}y \nonumber \\&= \frac{C_0}{|x|^{N-2s_1}} \int \nolimits _{2|x|}^{4|x|} r^{ N - 1 -p\mu _{0,v} + a } \textrm{d}r \nonumber \\&\ge \frac{C_1}{ |x|^{p \mu _{0,v} - (a + 2s_1 )} }, \end{aligned}$$

(3.48)

$$\begin{aligned} v(x)&\ge \int \nolimits _{2|x| \le |y| \le 4|x|} \frac{1}{ |x-y|^{N-2s_2} |y|^{ q \mu _{0,u} - b } } \textrm{d}y \nonumber \\&\ge \frac{C_0}{|x|^{N-2s_2}} \int \nolimits _{2|x| \le |y| \le 4|x|} \frac{1}{|y|^{ q \mu _{0,u} - b }} \textrm{d}y \nonumber \\&= \frac{C_0}{|x|^{N-2s_2}} \int \nolimits _{2|x|}^{4|x|} r^{ N - 1 - q\mu _{0,u} + b } \textrm{d}r \nonumber \\&\ge \frac{C_1}{ |x|^{q \mu _{0,u} - (b + 2s_2 )} }, \end{aligned}$$

(3.49)

Let $\mu _{1,u}:= p \mu _{0,v} - ( a + 2s_1 )$, $\mu _{1,v}:= q \mu _{0,u} - ( b + 2s_2 )$. Because $0< p < \frac{ N + 2 s_1 + 2 a }{ N - 2 s_2 }$ and $0< q < \frac{ N + 2 s_2 + 2 b }{ N - 2 s_1 }$, we have $\mu _{1,u} < \mu _{0,u}$ and $\mu _{1,v} < \mu _{0,v}$. So, we have obtained a better lower bound estimate

$$\begin{aligned} u(x) \ge \frac{ C_1 }{ |x|^{ \mu _{1,u} } }\quad \text {and} \quad v(x) \ge \frac{ C_1 }{ |x|^{ \mu _{1,v} } },\quad \forall |x| \ge 1. \end{aligned}$$

(3.50)

For $k = 0,1,2,\ldots$, define

$$\begin{aligned} \mu _{ k+1 , u } := p \mu _{ k,v } - ( a + 2s_1 )\quad \text {and} \quad \mu _{ k+1, v } := q \mu _{ k,u } - ( b+ 2s_2 ). \end{aligned}$$

(3.51)

Repeating the iteration process above, we can obtain

$$\begin{aligned} u(x) \ge \frac{ C_k }{ |x|^{ \mu _{k,u} } }\quad \text {and} \quad v(x) \ge \frac{ C_k }{ |x|^{ \mu _{k,v} } },\quad \forall ~|x| \ge 1. \end{aligned}$$

(3.52)

Moreover,

$$\begin{aligned} \mu _{ k+2 , u }&= pq \mu _{ k ,u } - p(b+2s_2) - (a+2s_1), \nonumber \\ \mu _{ k+2 , v }&= pq \mu _{ k ,v } - q(a+2s_1) - (b+2s_2). \end{aligned}$$

(3.53)

For $pq < 1$, as $k \rightarrow + \infty$,

$$\begin{aligned}&\mu _{ 2k , u} \rightarrow \frac{(a+2s_1) + p(b+s_2)}{pq-1},\nonumber \\&\mu _{ 2k , v} \rightarrow \frac{(b+2s_2) + q(a+s_1)}{pq-1}, \end{aligned}$$

(3.54)

and for $pq \ge 1$, we have $\mu _{ 2k, u} \rightarrow - \infty$ and $\mu _{ 2k, v} \rightarrow - \infty$ as $k \rightarrow +\infty$. This finishes the proof of Theorem 3.3. $\square$

Proof of Theorem 1.7

Assume that system (1.1) has a pair of nonnegative solution (u, v), for $0 < p \le \frac{ N + 2 s_1 + 2 a }{ N - 2 s_2 }$ and $0 < q \le \frac{ N + 2 s_2 + 2 b }{ N - 2 s_1 }$, $(p,q) \ne (\frac{ N + 2 s_1 + 2 a }{ N - 2 s_2 }, \frac{ N + 2 s_2 + 2 b }{ N - 2 s_1 } )$, the lower bound estimates in Theorem 3.3 contradicts with the following integrability of (u, v)

$$\begin{aligned} \int \nolimits _{{\mathbb {R}}^N} \frac{v^p(x)}{|x|^{N-2s_1-a}} \textrm{d}x&= u(0)< +\infty , \nonumber \\ \int \nolimits _{{\mathbb {R}}^N} \frac{u^q(x)}{|x|^{N-2s_2-b}} \textrm{d}x&= u(0) < +\infty . \end{aligned}$$

(3.55)

Therefore, we must have $(u,v) \equiv (0,0)$ in ${\mathbb {R}}^N$. $\square$