1 Introduction

In this paper, our main research is the existence and nonexistence of positive solutions for the following fractional coupled system with generalized p-Laplacian involving Riemann-Stieltjes integral conditions.

$$\begin{aligned} \textstyle\begin{cases} D^{\beta_{1} }_{0^{+}}(\phi(D^{\alpha_{1} }_{0^{+}}u(t)))+\lambda _{1} f_{1}(t,u(t), v(t))=0, \\ D^{\beta_{2} }_{0^{+}}(\phi(D^{\alpha_{2} }_{0^{+}}v(t)))+\lambda _{2} f_{2}(t,u(t), v(t))=0,\quad 0< t< 1, \\ u(0)=u'(0)=\cdots u^{(n-2)}=0,\quad\quad \phi(D^{\alpha_{1} }_{0^{+}}u(0))=( \phi(D^{\alpha_{1} }_{0^{+}}u(1)))'=0, \\ v(0)=v'(0)=\cdots v^{(m-2)}=0,\quad\quad \phi(D^{\alpha_{2} }_{0^{+}}v(0))=( \phi(D^{\alpha_{2} }_{0^{+}}v(1)))'=0, \\ u(1)=\mu_{1}\int_{0}^{1}g_{1}(s)v(s)\,dA_{1}(s), \quad\quad v(1)=\mu_{2}\int_{0} ^{1}g_{2}(s)u(s)\,dA_{2}(s), \end{cases}\displaystyle \end{aligned}$$
(1)

where \(\lambda_{i}>0\) (\(i=1, 2\)) is a parameter, \(1<\beta_{i}\leq2\), \(n-1<\alpha_{1} \leq n \), \(m-1<\alpha_{2} \leq m \), \(n, m\geq2\), \(D^{\alpha_{i}}_{0^{+}}\), \(D^{\beta_{i}}_{0^{+}}\) are the standard Riemann-Liouville derivatives. \(\mu_{i}>0\) is a constant, \(g_{i}:(0,1) \rightarrow[0, +\infty)\) is continuous with \(g_{i}\in L^{1}(0, 1)\), \(A_{i}\) is right continuous on \([0, 1)\), left continuous at \(t = 1\), and nondecreasing on \([0, 1]\), \(A_{i}(0) = 0\), \(\int_{0}^{1}x(s)\,dA _{i}(s)\) denotes the Riemann-Stieltjes integrals of x with respect to \(A_{i}\), ϕ is a generalized p-Laplacian operator and satisfies the following condition \((\mathbf{H}_{0})\).

The positive solution \((u,v)\) of system (1) means that \((u, v) \in C[0, 1]\times C[0, 1]\), \((u, v)\) satisfies system (1) and \(u(t)>0\), \(v(t)>0\) for all \(t\in(0, 1]\).

\((\mathbf{H}_{0})\) :

\(\phi:\mathbb{R}\rightarrow\mathbb{R}\) is an odd, increasing homeomorphism, and there exist two increasing homeomorphisms \(\psi_{1}, \psi_{2}:(0, +\infty)\rightarrow(0, +\infty)\) such that

$$ \psi_{1}(x)\phi(y)\leq\phi(xy)\leq\psi_{2}(x)\phi(y), \quad x, y>0. $$

Moreover, \(\phi,\phi^{-1}\in C^{1}(\mathbb{R})\), where \(\phi^{-1}\) denotes the inverse of ϕ and \(\mathbb{R}=(-\infty, +\infty)\).

Lemma 1.1

([1])

Assume that \((\mathbf{H}_{0})\) holds. Then

$$ \psi^{-1}_{2}(x)y\leq\phi^{-1}\bigl(x\phi(y) \bigr)\leq\psi^{-1}_{1}(x)y,\quad x, y>0. $$

For ϕ satisfying \((\mathbf{H}_{0})\), we call it a generalized p-Laplacian operator, it contains two important special cases: \(\phi(u)=u\) and \(\phi(u)=\vert u\vert ^{p-2}u\) (\(p>1\)) (see [1]). Many researchers have studied the existence of positive solutions for two above cases due to their great application background (see [215]). Combined with the fractional calculus, the application of the above two kinds of special circumstances becomes more extensive and practical. For the sake of considering the turbulent flow in a porous medium, the governing equation

$$\begin{aligned} \frac{\partial}{\partial x} \biggl( \frac{\partial u^{m}}{\partial x} \biggl\vert \frac{\partial u^{m}}{\partial x} \biggr\vert ^{p-1} \biggr) =g \biggl( t, u, \frac{\partial u}{\partial t} \biggr) ,\quad m\leq2,\frac{1}{2} \leq p\leq1 \end{aligned}$$
(2)

was presented by Leibenson (see [2]). If \(p = 1\), \(m > 0\), it is used as a nonlinear model for the dispersion of animals and insects (see [3]).

In [4], Lu et al. studied the existence of positive solution for the fractional boundary value problem with a p-Laplacian operator:

$$\begin{aligned} \textstyle\begin{cases} D^{\beta}_{0^{+}}(\phi_{p}(D^{\alpha}_{0^{+}}u(t)))=f(t,u(t)), \quad 0< t< 1, \\ u(0)=u'(0)=u {(1)}=0,\quad\quad D^{\alpha}_{0^{+}}u(0) = D^{\alpha}_{0^{+}}u(1) =0, \end{cases}\displaystyle \end{aligned}$$
(3)

where \(2 <\alpha\leq3\), \(1 <\beta\leq2\), \(D^{\alpha}_{0^{+}} \), \(D^{\beta}_{0^{+}} \) are the standard Riemann-Liouville fractional derivatives, \(\phi_{p}(s) =\vert s\vert ^{p-2}s\), \(p> 1\), \(\phi^{-1}_{p} =\phi _{q}\), \(\frac{1}{p}+ \frac{1}{q} = 1\), \(f: [0,1]\times[0,+\infty) \rightarrow[0, +\infty)\) is a continuous function. By the properties of Green’s function and the Guo-Krasnosel’skii fixed point theorem, some results on the existence of positive solutions are obtained.

In [5], Wang et al. investigated the same equation as (3) for \(1 <\alpha\leq2\), \(0 <\beta\leq1\), with boundary value condition \(u(0) = 0\), \(D^{\alpha}_{0^{+}}u(0)= 0\), \(u(1)=au( \xi)\), where \(0\leq a\leq1\), \(0<\xi<1\), \(f: [0,1]\times[0,+\infty )\rightarrow[0, +\infty)\) is a continuous function. Through the application of the Guo-Krasnosel’skii fixed point theorem and the Leggett-Williams theorem, sufficient conditions for the existence of positive solutions are received.

In system (1), \(\int_{0}^{1}g_{1}(s)v(s)\,dA_{1}(s)\), \(\int_{0} ^{1}g_{2}(s)u(s)\,dA_{2}(s)\) denote the Riemann-Stieltjes integrals, and \(A_{i}\) is a function of bounded variation, which implies that \(dA_{i}\) can be a signed measure. Then, a multipoint boundary value problem and an integral boundary value problem are included in our study, that is to say, system (1) includes more generalized boundary value conditions. Henderson and Luca in [16] considered the following system:

$$\begin{aligned} \textstyle\begin{cases} D^{\alpha_{1} }_{0^{+}}u(t)+\lambda_{1} f_{1}(t,u(t), v(t))=0, \\ D^{\alpha_{2} }_{0^{+}}v(t)+\lambda_{2} f_{2}(t,u(t), v(t))=0,\quad 0< t< 1, \\ u(0)=u'(0)=\cdots u^{(n-2)}=0, \quad\quad u(1)=\sum_{i=1}^{p}a_{i}u(\xi_{i}), \\ v(0)=v'(0)=\cdots v^{(m-2)}=0, \quad\quad v(1)=\sum_{i=1}^{q}b_{i}v(\eta_{i}), \end{cases}\displaystyle \end{aligned}$$
(4)

where \(\lambda_{i}>0\) (\(i=1, 2\)) is a parameter, \(n-1<\alpha_{1} \leq n \), \(m-1<\alpha_{2} \leq m \), \(n, m\geq2\), \(D^{\alpha_{i}}_{0^{+}}\), \(D^{\beta_{i}}_{0^{+}}\) are the standard Riemann-Liouville derivatives. \(a_{i}>0\), \(b_{i}>0\) are constants, \(f_{i}: [0,1]\times[0,+\infty)\times[0,+\infty) \rightarrow[0, + \infty)\) is a continuous function. By the Guo-Krasnosel’skii fixed point theorem, the authors in [16] got the existence of positive solutions on system (4). System (4) with uncoupled and coupled multi-point boundary value conditions

$$\begin{aligned}& \textstyle\begin{cases} u(0)=u'(0)=\cdots u^{(n-2)}=0, \quad\quad u(1)=\mu_{1}\int_{0}^{1}u(s)\,dA_{1}(s), \\ v(0)=v'(0)=\cdots v^{(m-2)}=0, \quad\quad v(1)=\mu_{2}\int_{0}^{1}v(s)\,dA_{2}(s), \end{cases}\displaystyle \\& \textstyle\begin{cases} u(0)=u'(0)=\cdots u^{(n-2)}=0, \quad\quad u(1)=\mu_{1}\int_{0}^{1}v(s)\,dA_{1}(s), \\ v(0)=v'(0)=\cdots v^{(m-2)}=0, \quad\quad v(1)=\mu_{2}\int_{0}^{1}u(s)\,dA_{2}(s) \end{cases}\displaystyle \end{aligned}$$

has been studied in many papers, where \(\mu_{i}>0\) is a constant, for \(\mu_{i}=1\) as an exceptional case ( see [1723] and the references therein). However, these articles only study the existence of positive solutions for the system, and do not relate to the nonexistence of positive solutions.

Up to now, coupled boundary value conditions for a fractional differential system with generalized p-Laplacian like system (1) have seldom been considered when \(\lambda_{1}\), \(\lambda_{2}\) are different. Motivated by the results mentioned above, in this paper, we obtain several new existence and nonexistence results for positive solutions in terms of different values of the parameter \(\lambda_{i}\) by using the properties of Green’s function and the Guo-Krasnosel’skii fixed point theorem on cone. Especially, paying attention to the nonlinear operator \(D^{\beta}_{0^{+}}(\phi(D^{\alpha}_{0^{+}}))\) with the discussion in (1), we can convert it to the linear operator \(D^{\beta}_{0^{+}}D^{\alpha}_{0^{+}}\), if \(\phi(u)=u\), and the additive index law

$$ D^{\beta}_{0^{+}}D^{\alpha}_{0^{+}}u(t)=D^{\alpha+\beta}_{0^{+}}u(t) $$

holds under some reasonable constraints on the function u (see [24]). Therefore, our article promotes, includes and improves the previous results in a certain degree.

2 Preliminaries and lemmas

For convenience of the reader, we present some necessary definitions about fractional calculus theory.

Definition 2.1

([24, 25])

Let \(\alpha>0\) and u be piecewise continuous on \((0, +\infty)\) and integrable on any finite subinterval of \([0, +\infty)\). Then, for \(t>0\), we call

$$ I^{\alpha}_{0^{+}}u(t)=\frac{1}{\Gamma(\alpha)} \int_{0}^{t}(t-s)^{ \alpha-1}u(s)\,ds, $$

the Riemann-Liouville fractional integral of u of order α.

Definition 2.2

([24, 25])

The Riemann-Liouville fractional derivative of order \(\alpha>0\), \({n-1}\leq\alpha< n\), \(n\in\mathbb{N}\), is defined as

$$ D^{\alpha}_{0^{+}}u(t)=\frac{1}{\Gamma(n-\alpha)} \biggl( \frac {d}{dt} \biggr) ^{n} \int_{0}^{t}(t-s)^{n-\alpha-1}u(s)\,ds, $$

where \(\mathbb{N}\) denotes the natural number set, the function \(u(t)\) is n times continuously differentiable on \([0, +\infty)\).

Lemma 2.1

([24, 25])

Let \(\alpha>0\), if the fractional derivatives \(D^{\alpha-1}_{0^{+}}u(t)\) and \(D^{\alpha} _{0^{+}}u(t)\) are continuous on \([0, +\infty)\), then

$$ I^{\alpha}_{0^{+}} D^{\alpha}_{0^{+}}u(t)=u(t)+c_{1}t^{\alpha-1}+c _{2}t^{\alpha-2}+\cdots+c_{n}t^{\alpha-n}, $$

where \(c_{1}, c_{2}, \ldots, c_{n}\in(-\infty, +\infty )\), n is the smallest integer greater than or equal to α.

Similarly to the proof in [22], it enables us to obtain the following Lemmas 2.2, 2.3 and Remark 2.1.

Lemma 2.2

Assume that the following condition \(( \mathbf{H}_{1})\) holds.

\((\mathbf{H}_{1})\) :
$$ \begin{aligned} &k_{1}= \int_{0}^{1}g_{1}(t)t^{\alpha_{2}-1}\,dA_{1}(t)>0, \quad\quad k_{2}= \int _{0}^{1}g_{2}(t)t^{\alpha_{1}-1}\,dA_{2}(t)>0, \\ & 1-\mu_{1}\mu_{2}k_{1}k _{2}>0. \end{aligned} $$

Let \(h_{i}\in C(0, 1)\cap L(0, 1)\) (\(i=1, 2\)), then the system with the coupled boundary conditions

$$\begin{aligned} \textstyle\begin{cases} D^{\alpha_{1}}_{0^{+}}u(t)+h_{1}(t)=0, \quad\quad D^{\alpha_{2}}_{0^{+}}v(t)+h_{2}(t)=0,\quad 0< t< 1, \\ u(0)=u'(0)=\cdots u^{(n-2)}=0,\quad\quad u(1)=\mu_{1}\int_{0}^{1}g _{1}(s)v(s)\,dA_{1}(s), \\ v(0)=v'(0)=\cdots v^{(n-2)}=0,\quad\quad v(1)=\mu_{2}\int_{0}^{1}g _{2}(s)u(s)\,dA_{2}(s) \end{cases}\displaystyle \end{aligned}$$
(5)

has a unique integral representation

$$\begin{aligned} \textstyle\begin{cases} u(t)=\int_{0}^{1} K_{1}(t, s)h_{1}(s)\,ds+\int_{0}^{1} H_{1}(t, s)h_{2}(s)\,ds, \\ v(t)=\int_{0}^{1} K_{2}(t, s)h_{2}(s)\,ds+\int_{0}^{1} H_{2}(t, s)h_{1}(s)\,ds, \end{cases}\displaystyle \end{aligned}$$
(6)

where

$$\begin{aligned} \begin{gathered} K_{1}(t, s)= \frac{\mu_{1}\mu_{2}k_{1}t^{\alpha_{1}-1}}{1-\mu_{1}\mu _{2}k_{1}k_{2}} \int_{0}^{1} g_{2}(t) \overline{G}_{1}(t, s)\,dA_{2}(t)+ \overline{G}_{1}(t, s), \\ H_{1}(t, s)= \frac{\mu_{1}t^{\alpha_{1}-1}}{1-\mu_{1}\mu_{2}k_{1}k _{2}} \int_{0}^{1} g_{1}(t)\overline{G}_{2}(t, s)\,dA_{1}(t), \\ K_{2}(t, s)=\frac{\mu_{2}\mu_{1}k_{2}t^{\alpha_{2}-1}}{1-\mu_{1}\mu _{2}k_{1}k_{2}} \int_{0}^{1}g_{1}(t) \overline{G}_{2}(t, s)\,dA_{1}(t)+ \overline{G}_{2}(t, s), \\ H_{2}(t, s)= \frac{\mu_{2}t^{\alpha_{2}-1}}{1-\mu_{1}\mu_{2}k_{1}k _{2}} \int_{0}^{1} g_{2}(t) \overline{G}_{1}(t, s)\,dA_{2}(t), \end{gathered} \end{aligned}$$
(7)

and

$$\begin{aligned} \overline{G}_{i}(t,s)=\frac{1}{\Gamma(\alpha_{i})} \textstyle\begin{cases} [t(1-s)]^{\alpha_{i}-1}-(t-s)^{\alpha_{i}-1},&0\leq s\leq t \leq1, \\ [t(1-s)]^{\alpha_{i}-1},&0\leq t\leq s\leq1, \end{cases}\displaystyle \quad i=1,2. \end{aligned}$$

Lemma 2.3

For \(t, s\in[0, 1]\), the functions \(K_{i}(t, s)\) and \(H_{i}(t, s)\) (\(i=1, 2\)) defined as (7) satisfy

$$\begin{aligned}& K_{1}(t, s),H_{2}(t, s)\leq\rho s(1-s)^{\alpha_{1}-1},\quad\quad K_{2}(t, s),H_{1}(t, s)\leq\rho s(1-s)^{\alpha_{2}-1}, \\& K_{1}(t, s),H_{1}(t, s)\leq\rho t^{\alpha_{1}-1},\quad\quad K_{2}(t, s), H_{2}(t, s)\leq\rho t^{\alpha_{2}-1}, \\& K_{1}(t, s)\geq\varrho t^{{\alpha_{1}-1}} s(1-s)^{\alpha_{1}-1}, \quad\quad H_{2}(t, s)\geq\varrho t^{{\alpha_{2}-1}} s(1-s)^{\alpha_{1}-1}, \\& K_{2}(t, s)\geq\varrho t^{{\alpha_{2}-1}} s(1-s)^{\alpha_{2}-1}, \quad\quad H_{1}(t, s)\geq\varrho t^{{\alpha_{1}-1}} s(1-s)^{\alpha_{2}-1}, \end{aligned}$$

where

$$\begin{aligned}& \begin{aligned} \rho&=\max \biggl\{ \frac{1}{\Gamma(\alpha_{1}-1)} \biggl( \frac{\mu_{1}\mu_{2}k_{1}}{1-\mu _{1}\mu_{2}k_{1}k_{2}} \int_{0}^{1} g_{2}(t)\,dA_{2}(t)+ 1 \biggr) , \\ &\quad \frac{ \mu_{1}}{\Gamma(\alpha_{2}-1)(1-\mu_{1}\mu_{2}k_{1}k_{2})} \int_{0} ^{1}g_{1}(t)\,dA_{1}(t), \\ &\quad\frac{1}{\Gamma(\alpha_{2}-1)} \biggl( \frac{\mu_{2}\mu _{1}k_{2}}{1-\mu _{1}\mu_{2}k_{1}k_{2}} \int_{0}^{1}g_{1}(t) \,dA_{1}(t)+1 \biggr) , \\ &\quad \frac{ \mu_{2}}{\Gamma(\alpha_{1}-1)(1-\mu_{1}\mu_{2}k_{1}k_{2})} \int_{0} ^{1}g_{2}(t) \,dA_{2}(t) \biggr\} , \end{aligned} \\& \begin{aligned} \varrho&=\min \biggl\{ \frac{\mu_{1}\mu_{2}k_{1}}{\Gamma(\alpha_{1})( 1-\mu_{1}\mu_{2}k_{1}k _{2})} \int_{0}^{1} g_{2}(t) (1-t)t^{\alpha_{1}-1}\,dA_{2}(t), \\ &\quad\frac{\mu_{1}}{\Gamma(\alpha_{2})(1-\mu_{1}\mu _{2}k_{1}k_{2})} \int _{0}^{1} g_{1}(t) (1-t)t^{\alpha_{2}-1}\,dA_{1}(t), \\ &\quad\frac{\mu_{2}\mu_{1}k_{2}}{\Gamma(\alpha_{2})(1-\mu_{1}\mu_{2}k_{1}k _{2})} \int_{0}^{1} g_{1}(t) (1-t)t^{\alpha_{2}-1}\,dA_{1}(t), \\ &\quad\frac{\mu_{2}}{\Gamma(\alpha_{1})(1-\mu_{1}\mu _{2}k_{1}k_{2})} \int _{0}^{1} g_{2}(t) (1-t)t^{\alpha_{1}-1}\,dA_{2}(t) \biggr\} . \end{aligned} \end{aligned}$$

Remark 2.1

From Lemma 2.3, for \(t,\widetilde{t},s \in[0, 1]\), we have

$$\begin{aligned}& K_{i}(t, s)\geq\omega t^{{\alpha_{i}-1}} K_{i}( \widetilde{t}, s), \quad\quad H _{i}(t, s)\geq\omega t^{{\alpha_{i}-1}} H_{i}(\widetilde{t}, s),\quad i=1, 2, \\& K_{1}(t, s)\geq\omega t^{{\alpha_{1}-1}} H_{2}(\widetilde{t}, s),\quad\quad H _{2}(t, s)\geq\omega t^{{\alpha_{2}-1}} K_{1}( \widetilde{t}, s), \\& K_{2}(t, s)\geq\omega t^{{\alpha_{2}-1}} H_{1}(\widetilde{t}, s), \quad\quad H _{1}(t, s)\geq\omega t^{{\alpha_{1}-1}} K_{2}( \widetilde{t}, s), \end{aligned}$$

where \(\omega=\frac{\varrho}{\rho}\), ϱ, ρ are defined as in Lemma 2.3, \(0<\omega<1\).

From Lemmas 2.1 and 2.2, we obtain the following Lemma 2.4.

Lemma 2.4

Let \(1<\beta_{i}\leq2\), \(n-1<\alpha_{1} \leq n \), \(m-1<\alpha_{2} \leq m \), \(h_{i}\in C(0, 1)\cap L(0, 1)\) (\(i=1, 2\)), the following system of fractional differential equations

$$\begin{aligned} \textstyle\begin{cases} D^{\beta_{1} }_{0^{+}}(\phi(D^{\alpha_{1} }_{0^{+}}u(t)))+\lambda _{1} h_{1}(t)=0, \quad\quad D^{\beta_{2} }_{0^{+}}(\phi(D^{\alpha_{2} }_{0^{+}}v(t)))+\lambda _{2} h_{2}(t)=0,\quad 0< t< 1, \\ u(0)=u'(0)=\cdots u^{(n-2)}=0,\quad\quad \phi(D^{\alpha_{1} }_{0^{+}}u(0))=( \phi(D^{\alpha_{1} }_{0^{+}}u(1)))'=0, \\ v(0)=v'(0)=\cdots v^{(m-2)}=0,\quad\quad \phi(D^{\alpha_{2} }_{0^{+}}v(0))=( \phi(D^{\alpha_{2} }_{0^{+}}v(1)))'=0, \\ u(1)=\mu_{1}\int_{0}^{1}v(s)\,dA_{1}(s),\quad\quad v(1)=\mu_{2}\int_{0}^{1}u(s)\,dA _{2}(s) \end{cases}\displaystyle \end{aligned}$$

has a unique integral representation

$$\begin{aligned} \textstyle\begin{cases} u(t)=\int_{0}^{1} K_{1}(t, s)\phi^{-1} ( \lambda_{1}\int_{0}^{1}G _{1}(s,\tau)h_{1}(\tau)\,d\tau ) \,ds \\ \hphantom{u(t)=}{}+ \int_{0}^{1} H_{1}(t, s) \phi^{-1} ( \lambda_{2}\int_{0}^{1}G_{2}(s,\tau)h_{2}(\tau)\,d\tau ) \,ds, \\ v(t)=\int_{0}^{1} K_{2}(t, s)\phi^{-1} ( \lambda_{2}\int_{0}^{1}G _{2}(s,\tau)h_{2}(\tau)\,d\tau ) \,ds \\ \hphantom{v(t)=}{}+\int_{0}^{1} H_{2}(t, s)\phi ^{-1} ( \lambda_{1}\int_{0}^{1}G_{1}(s,\tau)h_{1}(\tau)\,d\tau ) \,ds, \end{cases}\displaystyle \end{aligned}$$

where

$$\begin{aligned} {G}_{i}(s,\tau)=\frac{1}{\Gamma(\beta_{i})} \textstyle\begin{cases} s[s(1-\tau)]^{\beta_{i}-2}-(s-\tau)^{\beta_{i}-1},&0\leq \tau\leq s\leq1, \\ s[s(1-\tau)]^{\beta_{i}-2},&0\leq s\leq\tau\leq1, \end{cases}\displaystyle \quad i=1,2. \end{aligned}$$
(8)

Lemma 2.5

([26])

The function \({G}_{i}(s, \tau)\) defined as (8) is continuous on \([0,1]\times[0,1]\), and for \(s, \tau\in[0,1]\), \({G}_{i}(s,\tau)\) satisfies

$$(1)\quad {G}_{i}(s,\tau)\geq0 ;\quad\quad(2) \quad {G}_{i}(s,\tau )\leq{G}_{i}( \tau,\tau) ; \quad\quad(3) \quad {G}_{i}(s,\tau)\geq s^{\beta _{i}-1}{G}_{i}(1, \tau) . $$

In the rest of the paper, we always suppose that the following assumption holds:

\((\mathbf{H}_{2})\) :

\(f_{i}: [0,1]\times[0,+\infty)\times[0,+ \infty) \rightarrow[0, +\infty)\) is continuous.

Let \(X=C[0, 1]\times C[0, 1]\), then X is a Banach space with the norm

$$ \bigl\Vert (u,v)\bigr\Vert =\max \bigl\{ \Vert u\Vert , \Vert v \Vert \bigr\} ,\quad\quad \Vert u\Vert =\max_{t\in[0, 1]}\bigl\vert u(t) \bigr\vert , \quad\quad \Vert v\Vert =\max_{t\in[0, 1]}\bigl\vert v(t) \bigr\vert . $$

Denote

$$ K= \bigl\{ (u,v)\in X: u(t)\geq\omega t^{{\alpha_{1}-1}}\bigl\Vert (u,v)\bigr\Vert , v(t) \geq\omega t^{{\alpha_{2}-1}}\bigl\Vert (u,v)\bigr\Vert ,t \in[0, 1] \bigr\} , $$

where ω is defined as Remark 2.1. It is easy to see that K is a positive cone in X. Under the above conditions \((\mathbf{H}_{0})( \mathbf{H}_{1})(\mathbf{H}_{2})\), for any \((u,v)\in K\), we can define an integral operator \(T: K\to X\) by

$$\begin{aligned}& T(u,v) (t)=\bigl(T_{1}(u,v) (t), T_{2}(u,v) (t)\bigr), \quad 0\leq t \leq1, \end{aligned}$$
(9)
$$\begin{aligned}& \begin{gathered} T_{1}(u,v) (t)= \int_{0}^{1} K_{1}(t, s) \phi^{-1} \biggl( \lambda_{1} \int _{0}^{1}G_{1}(s,\tau)f_{1} \bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ \hphantom{T_{1}(u,v) (t)}\quad {} + \int_{0}^{1} H_{1}(t, s) \phi^{-1} \biggl( \lambda_{2} \int_{0}^{1}G _{2}(s, \tau)f_{2}\bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds, \\ T_{2}(u,v) (t)= \int_{0}^{1} K_{2}(t, s) \phi^{-1} \biggl( \lambda_{2} \int _{0}^{1}G_{2}(s, \tau)f_{2}\bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ \hphantom{T_{2}(u,v) (t)}\quad{} + \int_{0}^{1} H_{2}(t, s) \phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}G _{1}(s,\tau)f_{1} \bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds, \end{gathered} \end{aligned}$$
(10)

we know that \((u, v)\) is a positive solution of system (1) if and only if \((u, v)\) is a fixed point of T in K.

Lemma 2.6

Assume that \((\mathbf{H}_{0})(\mathbf{H}_{1})( \mathbf{H}_{2})\) hold. Then \(T: K\to K\) is a completely continuous operator.

Proof

By the routine discussion, we know that \(T: K\rightarrow X\) is well defined, so we only prove \(T(K)\subseteq K\). For any \((u, v)\in K\), \(0\leq t\), \(\widetilde{t}\leq1\), by Remark 2.1, we have

$$\begin{aligned} T_{1}(u,v) (t) =& \int_{0}^{1} K_{1}(t, s) \phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}G_{1}(s,\tau)f_{1} \bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ &{} + \int_{0}^{1} H_{1}(t, s) \phi^{-1} \biggl( \lambda_{2} \int_{0}^{1}G _{2}(s, \tau)f_{2}\bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ \geq & \int_{0}^{1} \omega t^{\alpha_{1}-1} K_{1}(\widetilde{t}, s) \phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}G_{1}(s,\tau)f_{1} \bigl(\tau, u( \tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ &{}+ \int_{0}^{1} \omega t^{\alpha_{1}-1} H_{1}(\widetilde{t}, s)\phi ^{-1} \biggl( \lambda_{2} \int_{0}^{1}G_{2}(s,\tau)f_{2} \bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ \geq & \omega t^{\alpha_{1}-1} \biggl( \int_{0}^{1} K_{1}( \widetilde{t}, s) \phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}G_{1}(s,\tau)f _{1} \bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ &{} + \int_{0}^{1} H_{1}(\widetilde{t}, s) \phi^{-1} \biggl( \lambda _{2} \int_{0}^{1}G_{2}(s,\tau)f_{2} \bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \biggr) \\ \geq &\omega t^{\alpha_{1}-1} T_{1}(u,v) (\widetilde{t}). \end{aligned}$$
(11)

On the other hand,

$$\begin{aligned} T_{1}(u,v) (t) \geq & \int_{0}^{1} \omega t^{\alpha_{1}-1} H_{2}( \widetilde{t}, s)\phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}G_{1}(s,\tau)f _{1} \bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ &{}+ \int_{0}^{1} \omega t^{\alpha_{1}-1} K_{2}(\widetilde{t}, s)\phi ^{-1} \biggl( \lambda_{2} \int_{0}^{1}G_{2}(s,\tau)f_{2} \bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ \geq & \omega t^{\alpha_{1}-1} \biggl( \int_{0}^{1} H_{2}( \widetilde{t}, s) \phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}G_{1}(s,\tau)f _{1} \bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ &{} + \int_{0}^{1} K_{2}(\widetilde{t}, s) \phi^{-1} \biggl( \lambda _{2} \int_{0}^{1}G_{2}(s,\tau)f_{2} \bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \biggr) \\ \geq &\omega t^{\alpha_{1}-1} T_{2}(u,v) (\widetilde{t}). \end{aligned}$$
(12)

Then we have

$$ T_{1}(u,v) (t)\geq\omega t^{\alpha_{1}-1} \bigl\Vert T_{1}(u,v)\bigr\Vert , \quad\quad T_{1}(u,v) (t) \geq\omega t^{\alpha_{1}-1} \bigl\Vert T_{2}(u,v)\bigr\Vert , $$

i.e.,

$$ T_{1}(u,v) (t)\geq\omega t^{\alpha_{1}-1} \bigl\Vert \bigl(T_{1}(u,v), T_{2}(u,v)\bigr) \bigr\Vert . $$

In the same way as (11) and (12), we can prove that

$$ T_{2}(u,v) (t)\geq\omega t^{\alpha_{2}-1} \bigl\Vert \bigl(T_{1}(u,v), T_{2}(u,v)\bigr) \bigr\Vert . $$

Therefore, we have \(T ( K ) \subseteq K\).

According to the Ascoli-Arzela theorem, we can easily get that \(T: K\rightarrow K\) is completely continuous. The proof is completed. □

In order to obtain the existence of the positive solutions of system (1), we will use the following cone compression and expansion fixed point theorem.

Lemma 2.7

([27])

Let P be a positive cone in a Banach space E, \(\Omega_{1}\) and \(\Omega_{2}\) are bounded open sets in E, \(\theta\in\Omega_{1}\), \(\overline{\Omega}_{1}\subset\Omega _{2}\), \(A :P\cap\overline{\Omega}_{2}\backslash\Omega_{1} \rightarrow P\) is a completely continuous operator. If the following conditions are satisfied:

$$\Vert Ax \Vert \leq \Vert x \Vert , \quad\forall{x}\in{P\cap\partial \Omega _{1}} ,\quad \quad \Vert Ax \Vert \geq \Vert x \Vert , \quad\forall{x}\in{P\cap\partial \Omega _{2}} , $$

or

$$\Vert Ax \Vert \geq \Vert x \Vert , \quad\forall{x}\in{P\cap\partial \Omega _{1}} ,\quad \quad \Vert Ax \Vert \leq \Vert x \Vert , \quad \forall{x}\in{P\cap\partial \Omega _{2}} , $$

then A has at least one fixed point in \(P\cap(\overline{\Omega} _{2}\backslash\Omega_{1})\).

3 Main results

Denote

$$\begin{aligned}& f_{10}=\liminf_{x\to0^{+}} \inf_{\substack{t\in[a, b]\subset(0, 1)\\ y\in[0, +\infty)}} \frac{f_{1}(t, x, y)}{ \phi(x)},\quad \quad f_{1}^{0}=\limsup _{x\to0^{+}} \sup_{\substack{t\in[0, 1]\\ y\in[0, +\infty)}}\frac{f_{1}(t, x, y)}{\phi(x)}, \\& f_{20}=\liminf_{y \to0^{+}} \inf_{\substack{t\in[a, b]\subset(0, 1)\\ x\in[0, +\infty)}} \frac{f_{2}(t, x, y)}{ \phi(y)},\quad \quad f_{2}^{0}=\limsup _{ y\to0^{+}} \sup_{\substack{t\in[0, 1]\\ x\in[0, +\infty)}}\frac{f_{2}(t, x, y)}{ \phi(y) }, \\& f_{1\infty}=\liminf_{x\to+\infty} \inf_{\substack{t\in[a, b] \subset(0, 1)\\ y\in[0, +\infty) }} \frac{f_{1}(t, x, y)}{ \phi(x)},\quad\quad f_{1}^{\infty}=\limsup _{x\to+\infty} \sup_{\substack{t\in[0, 1]\\ y\in[0, +\infty)}}\frac{f_{1}(t, x, y)}{\phi(x) }, \\& f_{2\infty}=\liminf_{ y\to+\infty} \inf_{\substack{t\in[a, b]\subset(0, 1)\\ x\in[0, +\infty) }} \frac{f_{2}(t, x, y)}{ \phi(y) },\quad\quad f_{2}^{\infty}=\limsup _{y\to+\infty} \sup_{\substack{t\in[0, 1]\\ x\in[0, +\infty)}}\frac{f_{2}(t, x, y)}{ \phi(y) }, \\& L_{1}=\max \biggl\{ 2\rho\varphi_{1}^{-1} \biggl( \int_{0}^{1}G_{1}( \tau,\tau)\,d\tau \biggr) , 2\rho\varphi_{1}^{-1} \biggl( \int_{0}^{1}G _{2}(\tau,\tau)\,d\tau \biggr) \biggr\} , \\& \begin{aligned} L_{2}&= \min \biggl\{ 2\varrho\omega\theta^{2} \int_{0}^{1}\varphi _{2}^{-1} \bigl( s^{\beta_{1}-1} \bigr) s^{\beta_{1}-1}\varphi_{2}^{-1} \biggl( \int_{0}^{1}G_{1}(1,\tau)\,d\tau \biggr) \,ds, \\ &\quad{} 2\varrho\omega\theta^{2} \int_{0}^{1}\varphi_{2}^{-1} \bigl( s ^{\beta_{2}-1} \bigr) s^{\beta_{2}-1}\varphi_{2}^{-1} \biggl( \int_{0} ^{1}G_{2}(1,\tau)\,d\tau \biggr) \,ds \biggr\} , \end{aligned} \\& \begin{aligned} L_{3} &= \min \biggl\{ 2\varrho\theta \int_{0}^{1} \varphi_{2}^{-1} \bigl( s^{\beta_{1}-1} \bigr) s(1-s)^{\alpha_{1}-1}\varphi_{2}^{-1} \biggl( \int_{0}^{1}G_{1}(1,\tau) \,d\tau \biggr) \,ds, \\ &\quad{} 2\varrho\theta \int_{0}^{1} \varphi_{2}^{-1} \bigl( s^{\beta _{2}-1} \bigr) s(1-s)^{\alpha_{2}-1}\varphi_{2}^{-1} \biggl( \int_{0}^{1}G _{2}(1,\tau)\,d\tau \biggr) \,ds \biggr\} ,\quad\theta=\min_{t\in[a, b]} \bigl\{ t^{\alpha_{1}-1}, t^{\alpha_{2}-1} \bigr\} . \end{aligned} \end{aligned}$$

3.1 Existence of system (1)

Theorem 3.1

Assume that \((\mathbf{H}_{0})(\mathbf{H} _{1})(\mathbf{H}_{2})\) hold and \(f_{i\infty}\varphi_{1}(L_{1}^{-1})>f _{i}^{0}\varphi_{2}(L_{2}^{-1})\), then system (1) has at least one positive solution for

$$\begin{aligned} \lambda_{i}\in \biggl( \frac{\varphi_{2}(L_{2}^{-1})}{f_{i\infty}}, \frac{ \varphi_{1}(L_{1}^{-1})}{f_{i}^{0}} \biggr) , \end{aligned}$$
(13)

where we impose \(\frac{1}{f_{i\infty}}=0\) if \(f_{i\infty}=+\infty\) and \(\frac{1}{f_{i}^{0}}=+\infty\) if \(f_{i}^{0}=0\) (\(i=1, 2\)).

Proof

For any \(\lambda_{i}\) satisfying (13), there exists \(\varepsilon_{0}>0\) such that

$$\begin{aligned} \frac{\varphi_{2}(L_{2}^{-1})}{f_{i\infty}-\varepsilon_{0}}\leq \lambda_{i}\leq\frac{\varphi_{1}(L_{1}^{-1})}{f_{i}^{0}+\varepsilon _{0}}. \end{aligned}$$
(14)

By the definition of \(f_{i}^{0}\), there exists \(r_{1}>0\) such that

$$\begin{aligned} f_{i}(t, x, y)\leq\bigl(f_{i}^{0}+ \varepsilon_{0}\bigr)\max\bigl\{ \phi(x), \phi(y) \bigr\} , \quad0\leq x, y\leq r_{1}, t\in[0, 1]. \end{aligned}$$
(15)

Let \(K_{r_{1}}=\{(u, v)\in K: \Vert (u, v)\Vert < r_{1} \}\). For any \((u, v)\in\partial K_{r_{1}}\), \(t\in[0, 1]\), by the definition of \(\Vert \cdot \Vert \), we know that

$$\begin{aligned} \begin{gathered} u(t)\leq\bigl\vert u(t)\bigr\vert \leq \Vert u\Vert \leq\bigl\Vert (u, v)\bigr\Vert \leq r_{1},\\ v(t)\leq\bigl\vert v(t)\bigr\vert \leq \Vert v\Vert \leq\bigl\Vert (u, v)\bigr\Vert \leq r_{1}, \quad t \in[0, 1]. \end{gathered} \end{aligned}$$
(16)

Thus, for any \((u, v)\in\partial K_{r_{1}}\), by (15), (16) and \((\mathbf{H}_{0})\), we have

$$\begin{aligned} f_{i}\bigl(t, u(t), v(t)\bigr)\leq\bigl(f_{i}^{0}+ \varepsilon_{0}\bigr)\phi(r_{1}), \quad t\in[0, 1]. \end{aligned}$$
(17)

Hence, for any \((u, v)\in\partial K_{r_{1}}\), by Lemmas 1.1, 2.3, 2.5 and (17), we conclude that

$$\begin{aligned} \bigl\Vert T_{1}(u,v) (t)\bigr\Vert =&\max_{t\in[0, 1]} \biggl\vert \int_{0}^{1} K_{1}(t, s) \phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}G_{1}(s,\tau)f_{1} \bigl(\tau, u( \tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ &{} + \int_{0}^{1} H_{1}(t, s) \phi^{-1} \biggl( \lambda_{2} \int_{0} ^{1}G_{2}(s, \tau)f_{2}\bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds\biggr\vert \\ \leq&\max_{t\in[0, 1]}\biggl\vert \int_{0}^{1} \rho t^{\alpha_{1}-1}\phi ^{-1} \biggl( \lambda_{1} \int_{0}^{1}G_{1}(\tau,\tau) \bigl(f_{1}^{0}+ \varepsilon_{0}\bigr) \phi(r_{1})\,d\tau \biggr) \,ds \\ & {}+ \int_{0}^{1} \rho t^{\alpha_{1}-1} \phi^{-1} \biggl( \lambda _{2} \int_{0}^{1}G_{2}(\tau,\tau) \bigl(f_{2}^{0}+\varepsilon_{0}\bigr)\phi(r _{1})\,d\tau \biggr) \,ds\biggr\vert \\ \leq&\rho r_{1}\varphi_{1}^{-1}\bigl( \lambda_{1}\bigl(f_{1}^{0}+\varepsilon _{0}\bigr)\bigr)\varphi_{1}^{-1} \biggl( \int_{0}^{1}G_{1}(\tau,\tau)\,d\tau \biggr) \\ &{}+ \rho r_{1}\varphi_{1}^{-1}\bigl( \lambda_{2}\bigl(f_{2}^{0}+\varepsilon_{0} \bigr)\bigr) \varphi_{1}^{-1} \biggl( \int_{0}^{1}G_{2}(\tau,\tau)\,d\tau \biggr) \\ \leq&r_{1}\varphi_{1}^{-1}\bigl( \lambda_{1}\bigl(f_{1}^{0}+\varepsilon _{0}\bigr)\bigr)\frac{L _{1}}{2}+r_{1} \varphi_{1}^{-1}\bigl(\lambda_{2} \bigl(f_{2}^{0}+\varepsilon _{0}\bigr)\bigr) \frac{L _{1}}{2} \\ \leq&r_{1}=\bigl\Vert (u, v)\bigr\Vert . \end{aligned}$$
(18)

Similarly to (18), for any \((u, v)\in\partial K_{r_{1}}\), we also have

$$ \bigl\Vert T_{2}(u,v)\bigr\Vert \leq r_{1}=\bigl\Vert (u, v)\bigr\Vert . $$

Consequently, we have

$$\begin{aligned} \bigl\Vert T(u, v)\bigr\Vert =\max \bigl\{ \bigl\Vert T_{1}(u, v)\bigr\Vert ,\bigl\Vert T_{2}(u, v)\bigr\Vert \bigr\} \leq r_{1}=\bigl\Vert (u, v)\bigr\Vert , \quad (u, v)\in\partial K_{r_{1}}. \end{aligned}$$
(19)

On the other hand, by the definition of \(f_{i\infty}\), there exist \(r_{1}', r_{2}'>0\) such that

$$\begin{aligned} \begin{gathered} f_{1}(t, x, y)\geq(f_{1\infty}-\varepsilon_{0}) \phi(x), \quad x \geq r',y\geq0,t\in[a,b]\subset(0, 1), \\ f_{2}(t, x, y)\geq(f_{2\infty}-\varepsilon_{0}) \phi(y), \quad y \geq r',x\geq0,t\in[a,b]\subset(0, 1). \end{gathered} \end{aligned}$$
(20)

Choose \(r_{2}=\max \{ \frac{r_{1}'}{\omega\theta},\frac{r_{2}'}{ \omega\theta},2r_{1} \} \). Let \(K_{r_{2}}=\{(u, v)\in K:\Vert (u, v)\Vert < r_{2}\}\). For any \((u, v)\in\partial K_{r_{2}} \), by the definition of \(\Vert \cdot \Vert \), we have

$$\begin{aligned} \begin{gathered} u(t)\geq\omega t^{\alpha_{1}-1} \bigl\Vert (u, v)\bigr\Vert \geq\omega\theta r_{2} \geq r_{1}',\\ v(t)\geq\omega t^{\alpha_{2}-1} \bigl\Vert (u, v)\bigr\Vert \geq \omega\theta r_{2}\geq r_{2}', \quad t\in[a, b]\subset(0, 1). \end{gathered} \end{aligned}$$
(21)

Thus, for any \((u, v)\in\partial K_{r_{2}}\), by (20), (21) and \((\mathbf{H}_{0})\), we have

$$\begin{aligned} \begin{gathered} f_{1}\bigl(t, u(t), v(t)\bigr)\geq(f_{1\infty}- \varepsilon_{0})\phi\bigl(u(t)\bigr) \geq(f_{1\infty}- \varepsilon_{0})\phi(\omega\theta r_{2}), \quad t\in[a,b] \subset(0, 1), \\ f_{2}\bigl(t, u(t), v(t)\bigr)\geq(f_{2\infty}- \varepsilon_{0})\phi\bigl(v(t)\bigr) \geq(f_{2\infty}- \varepsilon_{0})\phi(\omega\theta r_{2}), \quad t\in[a,b] \subset(0, 1). \end{gathered} \end{aligned}$$
(22)

Hence, for any \((u, v)\in\partial K_{r_{2}}\), by Lemmas 1.1, 2.3, 2.5 and (22), we have

$$\begin{aligned} &\bigl\Vert T_{1}(u,v) (t)\bigr\Vert \\ & \quad \geq \min _{t\in[a, b]}\biggl\vert \int_{0}^{1} \varrho t^{\alpha_{1}-1}s(1-s)^{\alpha_{1}-1} \phi^{-1} \biggl( \lambda_{1} \int _{0}^{1}s^{\beta_{1}-1}G_{1}(1,\tau) (f_{1\infty}-\varepsilon_{0}) \phi(\omega\theta r_{2})\,d\tau \biggr) \,ds \\ & \quad {} + \int_{0}^{1}\varrho t^{\alpha_{1}-1}s(1-s)^{\alpha_{2}-1} \phi^{-1} \biggl( \lambda_{2} \int_{0}^{1}s^{\beta_{2}-1}G_{2}(1,\tau) (f _{2\infty}-\varepsilon_{0})\phi(\omega\theta r_{2})\,d\tau \biggr) \,ds\biggr\vert \\ & \quad \geq \varrho\omega\theta^{2} r_{2} \int_{0}^{1}\varphi_{2}^{-1} \bigl(s ^{\beta_{1}-1}\bigr)s^{\beta_{1}-1}\varphi_{2}^{-1} \biggl( \int_{0}^{1}G_{1}(1, \tau)\,d\tau \biggr) \,ds \psi_{2}^{-1} \bigl(\lambda_{1}(f_{1\infty}- \varepsilon_{0})\bigr) \\ & \quad {} +\varrho\omega\theta^{2} r_{2} \int_{0}^{1}\varphi_{2}^{-1} \bigl(s^{ \beta_{2}-1}\bigr)s^{\beta_{2}-1}\varphi_{2}^{-1} \biggl( \int_{0}^{1}G_{2}(1, \tau)\,d\tau \biggr) \,ds\psi_{2}^{-1} \bigl(\lambda_{2}(f_{2\infty}- \varepsilon_{0})\bigr) \\ & \quad \geq r_{2}\varphi_{2}^{-1}\bigl( \lambda_{1}(f_{1\infty}-\varepsilon _{0})\bigr) \frac{L _{2}}{2}+r_{2}\varphi_{2}^{-1}\bigl( \lambda_{2}(f_{2\infty}-\varepsilon _{0})\bigr) \frac{L_{2}}{2} \\ & \quad \geq r_{2}=\bigl\Vert (u, v)\bigr\Vert . \end{aligned}$$
(23)

Therefore, we obtain

$$\begin{aligned} \bigl\Vert T(u, v)\bigr\Vert =\max \bigl\{ \bigl\Vert T_{1}(u, v)\bigr\Vert ,\bigl\Vert T_{2}(u, v)\bigr\Vert \bigr\} \geq r_{2}=\bigl\Vert (u, v)\bigr\Vert \quad \mbox{for any } (u, v)\in \partial K_{r_{2}}. \end{aligned}$$
(24)

It follows from the above discussion, (18), (24), Lemmas 2.6 and 2.7 that, for any \(\lambda_{i}\in ( \frac {\varphi_{2}(L _{2}^{-1})}{f_{i\infty}}, \frac{\varphi_{1}(L_{1}^{-1})}{f_{i}^{0}} ) \), T has a fixed point \((u, v)\in\overline{K}_{r_{2}}\setminus K_{r_{1}}\), so system (1) has at least one positive solution \((u,v)\); moreover, \((u,v)\) satisfies \(r_{1}\leq \Vert (u,v)\Vert \leq r_{2}\). The proof is completed. □

Remark 3.1

From the proof of Theorem 3.1, if we choose

$$\begin{aligned} \overline{L}_{2}=\varrho\omega\theta^{2} \int_{0}^{1}\varphi_{2} ^{-1} \bigl(s^{\beta_{1}-1}\bigr)s^{\beta_{1}-1}\varphi_{2}^{-1} \biggl( \int_{0} ^{1}G_{1}(1,\tau)\,d\tau \biggr) \,ds,\quad\theta=\min_{t\in[a, b]} \bigl\{ t^{\alpha_{1}-1}, t^{\alpha_{2}-1} \bigr\} , \end{aligned}$$
(25)

then for \(\lambda_{1}\in ( \frac{\varphi_{2}(\overline{L}_{2}^{-1})}{f _{1\infty}}, \frac{\varphi_{1}(L_{1}^{-1})}{f_{1}^{0}} ) \), \(\lambda_{2}\in ( 0, \frac{\varphi_{1}(L_{1}^{-1})}{f_{2}^{0}} ) \), the conclusion of Theorem 3.1 is valid.

Or we choose

$$\begin{aligned} \widetilde{{L}}_{2}=\varrho\omega\theta^{2} \int_{0}^{1}\varphi_{2} ^{-1} \bigl(s^{\beta_{2}-1}\bigr)s^{\beta_{2}-1}\varphi_{2}^{-1} \biggl( \int_{0} ^{1}G_{2}(1,\tau)\,d\tau \biggr) \,ds,\quad\theta=\min_{t\in[a, b]} \bigl\{ t^{\alpha_{1}-1}, t^{\alpha_{2}-1} \bigr\} , \end{aligned}$$
(26)

then, for \(\lambda_{1}\in ( 0, \frac{\varphi_{1}(L_{1}^{-1})}{f _{1}^{0}} ) \), \(\lambda_{2}\in ( \frac{\varphi_{2}( \widetilde{{L}}_{2}^{-1})}{f_{2\infty}}, \frac{\varphi_{1}(L_{1}^{-1})}{f _{2}^{0}} ) \), the conclusion of Theorem 3.1 is valid.

Theorem 3.2

Assume that \((\mathbf{H}_{0})(\mathbf{H} _{1})(\mathbf{H}_{2})\) hold and \(f_{i0}\varphi_{1}(L_{1}^{-1})>f_{i} ^{\infty}\varphi_{2}(L_{2}^{-1})\), then system (1) has at least one positive solution for

$$\begin{aligned} \lambda_{i}\in \biggl( \frac{\varphi_{2}(L_{2}^{-1})}{f_{i0}}, \frac{ \varphi_{1}(L_{1}^{-1})}{f_{i}^{\infty}} \biggr) , \end{aligned}$$

where we impose \(\frac{1}{f_{i0}}=0\) if \(f_{i0}=+\infty\) and \(\frac{1}{f_{i}^{\infty}}=+\infty\) if \(f_{i}^{\infty}=0\), \(i=1, 2\).

The proof of Theorem 3.2 is similar to that of Theorem 3.1, and so we omit it.

Remark 3.2

Similar to Remark 3.1, if we choose \(\overline{L}_{2}\) as (25), then for \(\lambda_{1}\in ( \frac{ \varphi_{2}(\overline{L}_{2}^{-1})}{f_{10}}, \frac{\varphi_{1}(L_{1} ^{-1})}{f_{1}^{\infty}} ) \), \(\lambda_{2}\in ( 0, \frac{ \varphi_{1}(L_{1}^{-1})}{f_{2}^{\infty}} ) \), the conclusion of Theorem 3.2 is valid.

Or we choose \(\widetilde{{L}}_{2}\) as (26), then for \(\lambda _{1}\in ( 0, \frac{\varphi_{1}(L_{1}^{-1})}{f_{1}^{\infty}} ) \), \(\lambda_{2} \in ( \frac{\varphi_{2}(\widetilde{{L}}_{2}^{-1})}{f_{20}}, \frac{ \varphi_{1}(L_{1}^{-1})}{f_{2}^{\infty}} ) \), the conclusion of Theorem 3.2 is valid.

Theorem 3.3

Assume that \((\mathbf{H}_{0})(\mathbf{H} _{1})(\mathbf{H}_{2})\) hold and there exist \(R>r>0\) such that

$$\begin{aligned} \lambda_{i} \min_{\substack{t\in[a,b]\subset(0,1)\\ \omega\theta r\leq x, y\leq r}}f _{i}(t, x, y) \geq\phi \biggl( \frac{r}{L_{3}} \biggr) , \quad \quad\lambda _{i}\max _{\substack{t\in[0,1]\\0\leq x, y\leq R}}f_{i}(t, x, y)\leq \phi \biggl( \frac{R}{L_{1}} \biggr) , \quad i=1, 2. \end{aligned}$$
(27)

Then system (1) has at least one positive solution \((u, v)\); moreover, \((u,v)\) satisfies \(r\leq \Vert (u,v)\Vert \leq R\).

Proof

Set \(K_{r}=\{(u, v)\in K:\Vert (u, v)\Vert < r\}\). For any \((u, v)\in\partial K_{r} \), by the definition of \(\Vert \cdot \Vert \), we have

$$\begin{aligned} \begin{gathered} \omega\theta r\leq\omega t^{\alpha_{1}-1} r=\omega t^{\alpha_{1}-1} \bigl\Vert (u, v)\bigr\Vert \leq u(t)\leq r, \\ \omega t^{\alpha_{2}-1} r=\omega t ^{\alpha_{2}-1} \bigl\Vert (u, v)\bigr\Vert \leq v(t)\leq r, \quad t\in[a,b] \subset(0,1). \end{gathered} \end{aligned}$$

Thus, for any \((u, v)\in\partial K_{r}\), by the first inequality of (27), we have

$$\begin{aligned} \lambda_{i} \min_{\substack {t\in[a,b]\subset(0,1)\\ \omega\theta r\leq u(t), v(t)\leq r}}f _{i}\bigl(t, u(t), v(t)\bigr)\geq\phi \biggl( \frac{r}{L_{3}} \biggr) , \quad i=1,2. \end{aligned}$$
(28)

Hence, for any \((u, v)\in\partial K_{r}\), by Lemmas 1.1, 2.3, 2.5 and (28), we have

$$\begin{aligned}& \bigl\Vert T_{1}(u,v) (t)\bigr\Vert \\& \quad \geq\min_{t\in[a, b]}\biggl\vert \int_{0}^{1} \varrho t^{\alpha_{1}-1}s(1-s)^{ \alpha_{1}-1} \phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}s^{\beta_{1}-1}G _{1}(1, \tau)f_{1}\bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\& \quad \quad{} + \int_{0}^{1}\varrho t^{\alpha_{1}-1}s(1-s)^{\alpha _{2}-1} \phi^{-1} \biggl( \lambda_{2} \int_{0}^{1}s^{\beta_{2}-1}G_{2}(1, \tau)f_{2}\bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds\biggr\vert \\& \quad \geq\varrho\theta \int_{0}^{1} s(1-s)^{\alpha_{1}-1} \phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}s^{\beta_{1}-1}G_{1}(1,\tau) \min_{ \substack{\tau\in[a,b]\subset(0,1) \\ \quad \omega r\leq u(\tau), v(\tau)\leq r}}f _{1}\bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\& \quad \quad{}+ \varrho\theta \int_{0}^{1} s(1-s)^{\alpha_{2}-1} \phi^{-1} \biggl( \lambda_{2} \int_{0}^{1}s^{\beta_{2}-1}G_{2}(1,\tau) \min_{ \substack{\tau\in[a,b]\subset(0,1) \\ \quad \omega r\leq u(\tau), v(\tau)\leq r}}f _{2}\bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\& \quad \geq\phi^{-1} \Bigl( \lambda_{1} \min_{ \substack{\tau\in[a,b]\subset(0,1) \\ \quad \omega r\leq u(\tau), v(\tau)\leq r}}f _{1}\bigl(\tau, u(\tau), v(\tau)\bigr) \Bigr) \frac{L_{3}}{2} \\& \quad\quad{} + \phi^{-1} \Bigl( \lambda _{2} \min_{ \substack{\tau\in[a,b]\subset(0,1) \\ \quad \omega r\leq u(\tau), v(\tau)\leq r}}f _{2}\bigl(\tau, u(\tau), v(\tau)\bigr) \Bigr) \frac{L_{3}}{2} \\& \quad \geq r=\bigl\Vert (u, v)\bigr\Vert . \end{aligned}$$
(29)

Therefore, we obtain

$$\begin{aligned} \bigl\Vert T(u, v)\bigr\Vert =\max \bigl\{ \bigl\Vert T_{1}(u, v)\bigr\Vert ,\bigl\Vert T_{2}(u, v)\bigr\Vert \bigr\} \geq r= \bigl\Vert (u, v)\bigr\Vert \quad \mbox{for any } (u, v)\in\partial K_{r}. \end{aligned}$$
(30)

Choose \(K_{R}=\{(u, v)\in K: \Vert (u, v)\Vert < R \}\). For any \((u, v)\in \partial K_{R}\), \(t\in[0, 1]\), by the definition of \(\Vert \cdot \Vert \), we know that

$$\begin{aligned} \begin{gathered} u(t)\leq\bigl\vert u(t)\bigr\vert \leq \Vert u\Vert \leq\bigl\Vert (u, v)\bigr\Vert \leq R, \\ v(t)\leq\bigl\vert v(t)\bigr\vert \leq \Vert v \Vert \leq\bigl\Vert (u, v)\bigr\Vert \leq R, \quad t\in[0, 1]. \end{gathered} \end{aligned}$$
(31)

Thus, for any \((u, v)\in\partial K_{R}\), by the first inequality of (27) and (31), we have

$$\begin{aligned} \lambda_{i}\max_{\substack{t\in[0,1]\\0\leq u(t), v(t)\leq R}}f_{i}\bigl(t, u(t), v(t)\bigr)\leq\phi \biggl( \frac{R}{L_{1}} \biggr) , \quad i=1,2. \end{aligned}$$
(32)

Hence, for any \((u, v)\in\partial K_{R}\), by Lemmas 1.1, 2.3, 2.5 and (32), we can gain

$$\begin{aligned} \bigl\Vert T_{1}(u,v) (t)\bigr\Vert \leq&\max _{t\in[0, 1]}\biggl\vert \int_{0}^{1} \rho t ^{\alpha_{1}-1} \phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}G_{1}(\tau, \tau)\max _{\substack{\tau\in[0,1]\\0\leq u(\tau), v(\tau)\leq R}} f_{1}\bigl( \tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ &{} + \int_{0}^{1} \rho t^{\alpha_{1}-1} \phi^{-1} \biggl( \lambda _{2} \int_{0}^{1}G_{2}(\tau,\tau) \max _{\substack{\tau\in[0,1]\\0\leq u(\tau), v(\tau)\leq R}}f_{2}\bigl( \tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds\biggr\vert \\ \leq&\phi^{-1}\Bigl(\lambda_{1} \max_{\substack {\tau\in[0,1]\\0\leq u(\tau), v(\tau)\leq R}}f_{1} \bigl( \tau, u(\tau), v(\tau)\bigr)\Bigr)\frac{L_{1}}{2} \\ &{}+\phi_{1}^{-1} \Bigl(\lambda_{2} \max_{\substack{\tau\in[0,1]\\0\leq u(\tau), v(\tau)\leq R}}f_{i}\bigl( \tau, u(\tau), v(\tau)\bigr)\Bigr)\frac{L_{1}}{2} \\ \leq&R=\bigl\Vert (u, v)\bigr\Vert . \end{aligned}$$
(33)

Similarly to (33), for any \((u, v)\in\partial K_{R}\), we also have

$$ \bigl\Vert T_{2}(u,v)\bigr\Vert < R=\bigl\Vert (u, v)\bigr\Vert . $$

Consequently, we have

$$\begin{aligned} \bigl\Vert T(u, v)\bigr\Vert =\max \bigl\{ \bigl\Vert T_{1}(u, v)\bigr\Vert ,\bigl\Vert T_{2}(u, v)\bigr\Vert \bigr\} < R= \bigl\Vert (u, v)\bigr\Vert , \quad (u, v)\in\partial K_{R}. \end{aligned}$$
(34)

It follows from the above discussion, (30), (34), Lemmas 2.6 and 2.7 that T has a fixed point \((u, v)\in \overline{K}_{R} \setminus K_{r}\), so system (1) has at least one positive solution \((u,v)\); moreover, \((u,v)\) satisfies \(r\leq \Vert (u,v)\Vert \leq R\). The proof is completed. □

Remark 3.3

From the proof of Theorem 3.3, if we choose

$$\begin{aligned} \begin{aligned}[b] & \overline{L}_{3}=\varrho\theta \int_{0}^{1} \varphi_{2}^{-1} \bigl( s ^{\beta_{1}-1} \bigr) s(1-s)^{\alpha_{1}-1}\varphi_{2}^{-1} \biggl( \int _{0}^{1}G_{1}(1,\tau) \,d\tau \biggr) \,ds,\\ &\quad\theta=\min_{t\in[a, b]} \bigl\{ t^{\alpha_{1}-1}, t^{\alpha_{2}-1} \bigr\} ,\end{aligned} \end{aligned}$$
(35)

then for

$$\begin{aligned} \lambda_{1}\min_{ \substack{t\in[a,b]\subset(0,1)\\\omega r\leq x, y\leq r}}f_{1}(t, x, y) \geq \phi \biggl( \frac{r}{\overline{L}_{3}} \biggr) ,\quad \quad\lambda_{i} \max _{\substack{t\in[0,1]\\0\leq x, y\leq R}}f_{i}(t, x, y)\leq\phi \biggl( \frac{R}{L_{1}} \biggr) , \quad i=1, 2, \end{aligned}$$

the conclusion of Theorem 3.3 is valid.

Or we choose

$$\begin{aligned} \begin{aligned}[b] & \widetilde{L}_{3}=\varrho\theta \int_{0}^{1} \varphi_{2}^{-1} \bigl( s ^{\beta_{2}-1} \bigr) s(1-s)^{\alpha_{2}-1}\varphi_{2}^{-1} \biggl( \int _{0}^{1}G_{2}(1,\tau)\,d\tau \biggr) \,ds,\\&\quad\theta=\min_{t\in[a, b]} \bigl\{ t^{\alpha_{1}-1}, t^{\alpha_{2}-1} \bigr\} . \end{aligned} \end{aligned}$$
(36)

Then, for

$$\begin{aligned} \lambda_{2}\min_{ \substack{t\in[a,b]\subset(0,1)\\\omega r\leq x, y\leq r}}f_{2}(t, x, y) \geq \phi \biggl( \frac{r}{\widetilde{L}_{3}} \biggr) , \quad \quad\lambda_{i} \max _{\substack{t\in[0,1]\\0\leq x, y\leq R}}f_{i}(t, x, y)\leq\phi \biggl( \frac{R}{L_{1}} \biggr) , \quad i=1, 2, \end{aligned}$$

the conclusion of Theorem 3.3 is valid.

Theorem 3.4

Assume that \((\mathbf{H}_{0})(\mathbf{H} _{1})(\mathbf{H}_{2})\) hold and \(f_{i0}=f_{i\infty}=+\infty\), then there exists \(\lambda_{i}^{*}>0\) such that system (1) has at least two positive solutions for \(\lambda_{i}\in(0, \lambda_{i}^{*})\), \(i=1, 2\).

Proof

Choose \(r>0\), define

$$ \chi_{i}(r)=\sup_{r>0}\frac{\phi(r)}{\varphi_{2}(L_{1}) \max_{\substack{t\in[0, 1]\\0\leq x, y\leq r}}f_{i}(t, x, y)},\quad i=1, 2. $$

In view of the continuity of \(f_{i}\) and \(f_{i0}=f_{i\infty}=+\infty \), we know \(\chi_{i}(r): (0, +\infty)\rightarrow(0, +\infty)\) is continuous and

$$ \lim_{t\rightarrow0^{+}}\chi_{i}(r)=\lim_{t\rightarrow+\infty} \chi _{i}(r)=0. $$

So, there exists \(r^{*}\in(0, +\infty)\) such that \(\chi_{i}(r^{*})= \sup_{r>0}\chi_{i}(r)=\lambda_{i}^{*}\). Therefore, for \(\lambda_{i} \in(0, \lambda_{i}^{*})\), we can find \(r_{1}\), \(r_{2}\) (\(0< r_{1}< r ^{*}< r_{2}<+\infty\)) satisfying \(\chi_{i}(r_{1})=\lambda_{1}\), \(\chi _{i}(r_{2})=\lambda_{2}\). Thus, by \((\mathbf{H}_{0})\), we have

$$\begin{aligned}& \lambda_{1}\max_{\substack{t\in[0, 1]\\0\leq x, y\leq r_{1}}} f_{i}(t, x, y)\leq \frac{\phi(r_{1})}{\varphi_{2}(L_{1})}\leq\phi \biggl( \frac{r _{1}}{L_{1}} \biggr) , \end{aligned}$$
(37)
$$\begin{aligned}& \lambda_{2}\max_{\substack{t\in[0, 1]\\0\leq x, y\leq r_{2}}} f_{i}(t, x, y)\leq \frac{\phi(r_{2})}{\varphi_{2}(L_{1})}\leq\phi \biggl( \frac{r _{2}}{L_{1}} \biggr) . \end{aligned}$$
(38)

From the condition \(f_{i0}=f_{i\infty}=+\infty\), there exist \(R_{1}\), \(R_{2}\) (\(0< R_{1}< r_{1}< r^{*}< r_{2}< R_{2}<+\infty\)) satisfying

$$\begin{aligned}& \frac{f_{1}(t, x, y)}{\phi(x)}\geq\frac{1}{\lambda_{1}\varphi_{1}( \omega\theta)\varphi_{1}(L_{3})}, \quad(x, y)\in[0, R_{1}] \cup[R _{2}, +\infty), t\in[a, b]\subset(0, 1), \\& \frac{f_{2}(t, x, y)}{\phi(y)}\geq\frac{1}{\lambda_{2}\varphi_{1}( \omega\theta)\varphi_{1}(L_{3})}, \quad(x, y)\in[0, R_{1}] \cup[R _{2}, +\infty), t\in[a, b]\subset(0, 1). \end{aligned}$$

Hence, by \((\mathbf{H}_{0})\), we get

$$\begin{aligned}& \lambda_{i} \min_{ \substack{t\in[a, b]\subset(0, 1)\\\omega\theta R_{1}\leq x, y\leq R_{1}}} f_{i}(t, x, y) \geq\phi \biggl( \frac{R_{1}}{L_{3}} \biggr) , \end{aligned}$$
(39)
$$\begin{aligned}& \lambda_{i} \min_{ \substack{t\in[a, b]\subset(0, 1)\\\omega\theta R_{2}\leq x, y\leq R_{2}}} f_{i}(t, x, y) \geq\phi \biggl( \frac{R_{1}}{L_{3}} \biggr) . \end{aligned}$$
(40)

By (37) and (39), (38) and (40), combining with Lemmas 2.6, 2.7 and Theorem 3.3, system (1) has at least two positive solutions for \(\lambda_{i}\in(0, \lambda_{i}^{*})\), \(i=1, 2\). The proof is completed. □

Remark 3.4

From the proof of Theorem 3.4, assume that \((\mathbf{H}_{0})(\mathbf{H}_{1})(\mathbf{H}_{2})\) hold, if \(f_{i0}=+\infty\) or \(f_{i\infty}=+\infty\), then there exists \(\lambda_{i}^{*}>0\) such that system (1) has at least one positive solution for \(\lambda_{i}\in(0, \lambda_{i}^{*})\), \(i=1, 2\).

3.2 Nonexistence of system (1)

Theorem 3.5

Assume that \((\mathbf{H}_{0})(\mathbf{H} _{1})(\mathbf{H}_{2})\) hold and \(f_{i}^{\infty}<+\infty\), \(f_{i}^{0}<+\infty\), then there exists \(\lambda_{i0}>0\) such that for \(\lambda_{i}\in(0, \lambda_{i0})\) (\(i=1, 2\)), system (1) has no positive solution.

Proof

From the definitions of \(f_{i}^{\infty}\), \(f_{i}^{0}\), which are finite, there exist positive constants \(M_{i}^{1}\), \(M_{i}^{2}\) and \(R_{1}\), \(R_{2}\) (\(R_{1}< R_{2}\)) such that

$$\begin{aligned}& f_{i}(t, x, y)\leq M_{i}^{1}\max\bigl\{ \phi(x), \phi(y)\bigr\} , \quad0 \leq x, y\leq R_{1}, t\in[0, 1], \\& f_{i}(t, x, y)\leq M_{i}^{2}\max\bigl\{ \phi(x), \phi(y)\bigr\} , \quad x, y \geq R_{2}, t\in[0, 1]. \end{aligned}$$

Set \(M_{i}^{0}=\max \{ M_{i}^{1},M_{i}^{2},\max_{ { t\in[0, 1], R_{1}\leq x, y\leq R_{2}}}\frac{f_{i}(t, x, y)}{ \max\{\phi(x), \phi(y)\}} \} \), we have

$$ f_{i}(t, x, y)\leq M_{i}^{0}\max\bigl\{ \phi(x), \phi(y)\bigr\} , \quad x, y \geq0, t\in[0, 1]. $$

Assume that \((u, v)\) is a positive solution of system (1), we will show that this leads to a contradiction. Define \(\lambda_{i0}= {(M_{i}^{0})}^{-1}\varphi_{1}(L_{1}^{-1})\), since \(\lambda_{i}\in(0, \lambda_{i0})\), by Lemmas 1.1, 2.3 and 2.5, we conclude that

$$\begin{aligned} u(t) =&T_{1}(u,v) (t) \\ =& \int_{0}^{1} K_{1}(t, s) \phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}G _{1}(s,\tau)f_{1} \bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ &{} + \int_{0}^{1} H_{1}(t, s) \phi^{-1} \biggl( \lambda_{2} \int_{0}^{1}G _{2}(s, \tau)f_{2}\bigl(\tau, u(\tau), v(\tau)\bigr)\,d\tau \biggr) \,ds \\ \leq& \int_{0}^{1} \rho t^{\alpha_{1}-1} \phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}G_{1}(\tau, \tau)M_{1}^{0}\max\bigl\{ \phi\bigl(u(\tau)\bigr), \phi \bigl(v( \tau)\bigr)\bigr\} \,d\tau \biggr) \,ds \\ &{}+ \int_{0}^{1} \rho t^{\alpha_{1}-1} \phi^{-1} \biggl( \lambda_{2} \int _{0}^{1}G_{2}(\tau, \tau)M_{2}^{0}\max\bigl\{ \phi\bigl(u(\tau)\bigr), \phi \bigl(v( \tau)\bigr)\bigr\} \,d\tau \biggr) \,ds \\ \leq&\rho\bigl\Vert (u, v)\bigr\Vert \varphi_{1}^{-1} \bigl(\lambda_{1}M_{1}^{0}\bigr)\varphi _{1}^{-1} \biggl( \int_{0}^{1}G_{1}(\tau,\tau)\,d\tau \biggr) \\ &{} +\rho\bigl\Vert (u, v)\bigr\Vert \varphi_{1}^{-1}\bigl( \lambda_{2}M_{2}^{0}\bigr)\varphi_{1}^{-1} \biggl( \int _{0}^{1}G_{2}(\tau,\tau)\,d\tau \biggr) \\ \leq& \bigl\Vert (u, v)\bigr\Vert \varphi_{1}^{-1} \bigl(\lambda_{1}M_{1}^{0}\bigr) \frac{L_{1}}{2}+ \bigl\Vert (u, v)\bigr\Vert \varphi_{1}^{-1} \bigl(\lambda_{2}M_{2}^{0}\bigr) \frac{L _{1}}{2}. \end{aligned}$$
(41)

Therefore, we conclude

$$\begin{aligned} \Vert u\Vert \leq&\bigl\Vert (u, v)\bigr\Vert \varphi_{1}^{-1} \bigl(\lambda_{1}M_{1}^{0}\bigr)\frac{L _{1}}{2}+ \bigl\Vert (u, v)\bigr\Vert \varphi_{1}^{-1}\bigl( \lambda_{2}M_{2}^{0}\bigr) \frac{L_{1}}{2} \\ < &\bigl\Vert (u, v)\bigr\Vert \varphi_{1}^{-1}\bigl( \lambda_{10}M_{1}^{0}\bigr)\frac{L_{1}}{2}+ \bigl\Vert (u, v)\bigr\Vert \varphi_{1}^{-1}\bigl( \lambda_{20}M_{2}^{0}\bigr)\frac{L_{1}}{2}= \bigl\Vert (u, v)\bigr\Vert . \end{aligned}$$
(42)

Similarly to (41) (42), we also have

$$\begin{aligned} \Vert v \Vert < \bigl\Vert (u, v)\bigr\Vert . \end{aligned}$$
(43)

Hence, by (42) (43), we get

$$\begin{aligned} \bigl\Vert (u, v)\bigr\Vert =\max \bigl\{ \Vert u\Vert ,\Vert v \Vert \bigr\} < \bigl\Vert (u, v)\bigr\Vert , \end{aligned}$$
(44)

which is a contradiction. Therefore, system (1) has no positive solution. The proof is completed. □

Theorem 3.6

Assume that \((\mathbf{H}_{0})(\mathbf{H} _{1})(\mathbf{H}_{2})\) hold and \(f_{i\infty}>0\), \(f_{i 0}>0\), \(f_{i}(t, x, y)>0\) for \(t\in[a, b]\subset(0, 1)\), \(x\geq0\), \(y>0\) or \(t\in[a, b]\subset(0, 1)\), \(x>0\), \(y\geq0\), then there exists \(\lambda_{i*}>0\) such that for \(\lambda_{i}\in(\lambda_{i*}, +\infty )\) (\(i=1, 2\)), system (1) has no positive solution.

Proof

From the definitions of \(f_{i\infty}\), \(f_{i0}\), which are finite, there exist positive constants \(m_{i}^{1}\), \(m_{i}^{2}\) and \(R_{3}\), \(R_{4}\) (\(R_{3}< R_{4}\)) such that

$$\begin{aligned}& f_{1}(t, x, y)\geq m_{1}^{1}\phi(x), \quad 0\leq x, y\leq R_{3}, t \in[a, b]\subset(0, 1), \\& f_{1}(t, x, y)\geq m_{1}^{2}\phi(x), \quad x, y \geq R_{4}, t\in[a, b]\subset(0, 1). \end{aligned}$$

Set \(m_{1}^{0}=\min \{ m_{1}^{1},m_{1}^{2}, \min_{{ t\in[a, b]\subset(0, 1), R_{3}\leq x, y\leq R_{4}}}\frac{f _{1}(t, x, y)}{\phi(x)} \} \), we have

$$ f_{1}(t, x, y)\geq m_{1}^{0}\phi(x), \quad x, y \geq0,t\in[a, b] \subset(0, 1). $$

Similarly, set \(m_{2}^{0}=\min \{ m_{2}^{1},m_{2}^{2}, \min_{{ t\in[a, b]\subset(0, 1), R_{3}\leq x, y\leq R_{4}}}\frac{f _{2}(t, x, y)}{\phi(y)} \} \), we have

$$ f_{2}(t, x, y)\geq m_{2}^{0}\phi(y), \quad x, y \geq0,t\in[a, b] \subset(0, 1). $$

Assume that \((u, v)\) is a positive solution of system (1), we will show that this leads to a contradiction. Define \(\lambda_{i*}= {(m_{i}^{0})}^{-1}\varphi_{2}(L_{2}^{-1})\), since \(\lambda_{i}\in( \lambda_{i*}, +\infty)\), by Lemmas 1.1, 2.3 and 2.5, we conclude that

$$\begin{aligned} \Vert u\Vert =& \bigl\Vert T_{1}(u,v)\bigr\Vert \\ \geq&\min_{t\in[a, b]} \int_{0}^{1} \varrho t^{\alpha_{1}-1}s(1-s)^{ \alpha_{1}-1} \phi^{-1} \biggl( \lambda_{1} \int_{0}^{1}s^{\beta_{1}-1}G _{1}(1,\tau) m_{1}^{0}\phi\bigl(u(\tau)\bigr)\,d\tau \biggr) \,ds \\ &{}+ \int_{0}^{1}\varrho t^{\alpha_{1}-1}s(1-s)^{\alpha_{2}-1} \phi^{-1} \biggl( \lambda_{2} \int_{0}^{1}s^{\beta_{2}-1}G_{2}(1, \tau)m_{2}^{0} \phi\bigl(v(\tau)\bigr)\,d\tau \biggr) \,ds \\ \geq&\varrho\omega\theta^{2} \bigl\Vert (u, v)\bigr\Vert \int_{0}^{1}\varphi_{2} ^{-1} \bigl(s^{\beta_{1}-1}\bigr)s^{\beta_{1}-1}\varphi_{2}^{-1} \biggl( \int_{0} ^{1}G_{1}(1,\tau)\,d\tau \biggr) \,ds \psi_{2}^{-1} \bigl(\lambda_{1}m_{1}^{0}\bigr) \\ &{} +\varrho\omega\theta^{2} \bigl\Vert (u, v)\bigr\Vert \int_{0}^{1}\varphi_{2}^{-1} \bigl(s ^{\beta_{2}-1}\bigr)s^{\beta_{2}-1}\varphi_{2}^{-1} \biggl( \int_{0}^{1}G_{2}(1, \tau)\,d\tau \biggr) \,ds\psi_{2}^{-1} \bigl(\lambda_{2}m_{2}^{0}\bigr) \\ \geq&\bigl\Vert (u, v)\bigr\Vert \varphi_{2}^{-1} \bigl(\lambda_{1}m_{1}^{0}\bigr) \frac{L_{2}}{2}+\bigl\Vert (u, v)\bigr\Vert \varphi_{2}^{-1} \bigl(\lambda_{2}m_{2}^{0}\bigr) \frac{L _{2}}{2}>\bigl\Vert (u, v)\bigr\Vert . \end{aligned}$$
(45)

Similarly to (45), we also have

$$\begin{aligned} \Vert v \Vert > \bigl\Vert (u, v)\bigr\Vert . \end{aligned}$$
(46)

Hence, by (45) (46), we get

$$\begin{aligned} \bigl\Vert (u, v)\bigr\Vert =\max \bigl\{ \Vert u\Vert ,\Vert v \Vert \bigr\} >\bigl\Vert (u, v)\bigr\Vert , \end{aligned}$$
(47)

which is a contradiction. Therefore, system (1) has no positive solution. The proof is completed. □

Similar to the proof of Theorem 3.6, we obtain the following Theorems 3.7 and 3.8.

Theorem 3.7

Assume that \((\mathbf{H}_{0})(\mathbf{H} _{1})(\mathbf{H}_{2})\) hold and \(f_{1\infty}>0\), \(f_{10}>0\), \(f_{1}(t, x, y)>0\) for \(t\in[a, b]\subset(0, 1)\), \(x\geq0\), \(y>0\) or \(t\in[a, b]\subset(0, 1)\), \(x>0\), \(y\geq0\), then there exists \(\lambda_{1*}>0\) such that for \(\lambda_{1}\in(\lambda_{1*}, +\infty )\), \(\lambda_{2}\in(0, +\infty)\), system (1) has no positive solution.

Theorem 3.8

Assume that \((\mathbf{H}_{0})(\mathbf{H} _{1})(\mathbf{H}_{2})\) hold and \(f_{2\infty}>0\), \(f_{2 0}>0\), \(f_{2}(t, x, y)>0\) for \(t\in[a, b]\subset(0, 1)\), \(x\geq0\), \(y>0\) or \(t\in[a, b]\subset(0, 1)\), \(x>0\), \(y\geq0\), then there exists \(\lambda_{2*}>0\) such that for \(\lambda_{2}\in(\lambda_{2*}, +\infty )\), \(\lambda_{1}\in(0, +\infty)\), system (1) has no positive solution.

Remark 3.5

From the proof of Theorems 3.1-3.8, if we choose

$$\begin{aligned}& f_{10}=\liminf_{y\to0^{+}} \inf_{\substack{t\in[a, b]\subset(0, 1)\\ x\in[0, +\infty)}} \frac{f_{1}(t, x, y)}{ \phi(y)},\quad \quad f_{1}^{0}=\limsup _{y\to0^{+}} \sup_{\substack{t\in[0, 1]\\ x\in[0, +\infty)}}\frac{f_{1}(t, x, y)}{\phi(y)}, \\& f_{20}=\liminf_{x \to0^{+}} \inf_{\substack{t\in[a, b]\subset(0, 1)\\ y\in[0, +\infty)}} \frac{f_{2}(t, x, y)}{ \phi(x)},\quad \quad f_{2}^{0}=\limsup _{ x\to0^{+}} \sup_{\substack{t\in[0, 1]\\ y\in[0, +\infty)}}\frac{f_{2}(t, x, y)}{ \phi(x) }, \\& f_{1\infty}=\liminf_{y\to+\infty} \inf_{\substack{t\in[a, b] \subset(0, 1)\\ x\in[0, +\infty) }} \frac{f_{1}(t, x, y)}{ \phi(y)},\quad \quad f_{1}^{\infty}=\limsup _{y\to+\infty} \sup_{\substack{t\in[0, 1]\\ x\in[0, +\infty)}}\frac{f_{1}(t, x, y)}{\phi(y) }, \\& f_{2\infty}=\liminf_{ x\to+\infty} \inf_{\substack{t\in[a, b]\subset(0, 1)\\ y\in[0, +\infty) }} \frac{f_{2}(t, x, y)}{ \phi(x) },\quad \quad f_{2}^{\infty}=\limsup _{x\to+\infty} \sup_{\substack{t\in[0, 1]\\ y\in[0, +\infty)}}\frac{f_{2}(t, x, y)}{ \phi(x) }, \end{aligned}$$

then all the conclusion of Theorems 3.1-3.8 are valid.

4 Example

Consider the fractional differential system

$$\begin{aligned} \textstyle\begin{cases} D^{\frac{3}{2}} ( D^{\frac{5}{2}}_{0^{+}}u(t) ) +\lambda_{1} f _{1}(t,u(t), v(t)) =0, \\ D^{\frac{3}{2}} ( D^{\frac{5}{2}}_{0^{+}}v(t) ) +\lambda_{2} f _{2}(t,u(t), v(t))=0, \quad 0< t< 1, \\ u(0)=u'(0)=0,\quad\quad D^{\frac{5}{2}}_{0^{+}}u(0)= ( D^{\frac {5}{2}}_{0^{+}}u(1) ) '=0, \quad\quad u(1)=\frac{1}{2}\int_{0}^{1}t^{-\frac{1}{2}}v(t)\,dt, \\ v(0)=v'(0)=0,\quad\quad D^{\frac{5}{2}}_{0^{+}}v(0)= ( D^{\frac {5}{2}}_{0^{+}}v(1) ) '=0, \quad\quad v(1)=\int_{0}^{1}u(t)\,dt^{\frac{1}{2}}, \end{cases}\displaystyle \end{aligned}$$
(48)

where \(\lambda_{i}>0\) (\(i=1, 2\)) is a parameter, \(\alpha_{1}=\alpha _{2}=\frac{5}{2}\), \(\beta_{1}=\beta_{2}=\frac{3}{2}\), \(\mu_{1}= \frac{1}{2}\), \(\mu_{2}=1\), \(A_{1}(t)=t\), \(A_{2}(t)=t^{\frac{1}{2}}\), \(g_{1}(t)=t^{-\frac{1}{2}}\), \(g_{2}(t)=1\), \(\phi(x)=x\), choose \(\varphi_{1}(x)=\varphi_{2}(x)=x\). Then we have

$$\begin{aligned}& k_{1}= \int_{0}^{1}g_{1}(t)t^{\alpha_{2}-1}\,dA_{1}(t)= \int_{0}^{1}t^{- \frac{1}{2}}t^{\frac{3}{2}}\,dt= \frac{1}{2}>0, \\& k_{2}= \int_{0}^{1}g_{2}(t)t^{\alpha_{1}-1}\,dA_{2}(t)= \int_{0}^{1}t^{ \frac{3}{2}}\,dt^{\frac{1}{2}} = \frac{1}{2} \int_{0}^{1}t\,dt=\frac{1}{4}>0, \\& 1-\mu_{1}\mu_{2}k_{1}k_{2}= \frac{15}{16}>0. \end{aligned}$$

So, condition \((\mathbf{H}_{1})\) holds. Next, in order to demonstrate the application of our main results obtained in Section 3, we choose two different sets of functions \(f_{i}\) (\(i= 1,2\)) such that \(f_{i}\) satisfies the conditions of Theorems 3.1 and 3.5.

Case 1. Let \(f_{1}(t, x, y)=\frac{x^{2}}{1+t}+x\sin y\), \(f_{2}(t, x, y)=\frac{y ^{2}}{e^{t}}+y\sin x\), choose \([ \frac{1}{4}, \frac{2}{2} ] \subset[0,1]\), we know \(f_{i\infty}=+\infty\), \(f_{i}^{0}=0\). Then, by Theorem 3.1, system (48) has at least one positive solution for \(\lambda_{i}\in(0, +\infty)\) (\(i=1, 2\)).

Case 2. Let \(f_{1}(t, x, y)=\frac{(10x^{2}+x)(3+\sin y)}{(1+t)(x+1)}\), \(f_{2}(t, x, y)=\frac{(10y^{2}+y)(2+\sin x)}{e^{t}(y+1)}\), therefore, we have \(f_{1}^{\infty}=40\), \(f_{1}^{0}=3\), \(f_{2}^{\infty}=30\), \(f_{2}^{0}=2\), and for \(x, y\leq0\), we get \(x\leq f_{1}(t, x, y) \leq40x\), \(y\leq f_{2}(t, x, y)\leq30y\). By calculation, we obtain \(L_{1} =2.2445\int_{0}^{1}\frac{\tau^{\frac{1}{2}}(1-\tau)^{- \frac{1}{2}}}{\Gamma(\frac{3}{2})}\,d\tau\approx0.6632\). Then, by Theorem 3.5, system (48) has no positive solution for \(\lambda_{1}\in(0, 0.0377)\), \(\lambda_{2}\in(0, 0.0503)\).