1 Introduction

Fractional differential equations have attracted the attention of many researchers working in a variety of disciplines due to the development and applications of these equations in many fields such as engineering, mathematics, physics, chemistry, etc. For recent development of the topic, we refer the reader to a series of books and papers [111]. The study of boundary value problems of coupled systems of fractional order differential equations is also very important as such systems appear in a variety of problems of applied nature, especially in biosciences, for instance, see [1222].

The Langevin equation (first formulated by Langevin in 1908) is found to be an effective tool to describe the evolution of physical phenomena in fluctuating environments [23]. For some new developments on the fractional Langevin equation, see, for example, [2432]. The Sturm-Liouville problem has many applications in different areas of science, for example, engineering and mathematics. The classical Sturm-Liouville problem for a linear differential equation of second order is a boundary-value problem as the following one:

$$ \textstyle\begin{cases} -\frac{d}{dt} [p(t)\frac{dx}{dt} ]+v(t)x=\lambda r(t)x,& t \in[a,b], \\ a_{1} x(a)+a_{2}x'(a)=0, \\ b_{1} x(b)+b_{2} x'(b)=0. \end{cases} $$
(1.1)

Recently in [33], the authors proposed an approach to the fractional version of the Sturm-Liouville problem. They investigated the eigenvalues and eigenfunctions associated with these operators and also their properties with the objective of applying this generalized Sturm-Liouville theory to fractional partial differential equations.

This paper investigates the existence of solutions for the following system of fractional differential equations:

$$ \textstyle\begin{cases} D^{\alpha_{2}}([p(t)D^{\alpha_{1}}+r(t)]x(t))=f(t,x(t),y(t)),& 0< t< T, \\ D^{\beta_{2}}([q(t)D^{\beta_{1}}+s(t)]y(t))=g(t,x(t),y(t)),& 0< t< T, \end{cases} $$
(1.2)

subject to anti-periodic boundary conditions

$$ \textstyle\begin{cases} x(0)=-x(T),\qquad D^{\alpha_{1}}x(0)=-D^{\alpha_{1}}x(T), \\ y(0)=-y(T),\qquad D^{\beta_{1}}y(0)=-D^{\beta_{1}}y(T), \end{cases} $$
(1.3)

where \(D^{\theta}\) is the Caputo fractional derivative of orders \(\theta\in\{\alpha_{1},\alpha_{2},\beta_{1},\beta_{2}\}\) with \(0<\alpha_{1},\alpha_{2},\beta_{1},\beta_{2}< 1\), \(f,g\in C([0,T] \times\mathbb{R}^{2},\mathbb{R})\), \(p,q\in C([0,T],\mathbb{R}\setminus \{0\})\) with \(\vert p(t)\vert \geq K_{1}\), \(\vert q(t)\vert \geq K_{2}\), \(K_{1},K_{2}>0\) and \(r,s\in C([0,T],\mathbb{R})\).

Note that system (1.2) is a generalization of Sturm-Liouville and Langevin fractional differential systems. If \(r(t), s(t)\equiv 0\) for all \(t\in[0,T]\), then (1.2) is reduced to

$$ \textstyle\begin{cases} D^{\alpha_{2}}( p(t) D^{\alpha_{1}}x(t))=f(t,x(t),y(t)),& 0< t< T, \\ D^{\beta_{2}}( q(t) D^{\beta_{1}}y(t))=g(t,x(t),y(t)),& 0< t< T, \end{cases} $$
(1.4)

which are Sturm-Liouville fractional differential equations. If \(p(t)=q(t)\equiv1\) and \(r(t)\equiv\lambda_{1}\), \(s(t)=\lambda_{2}\) for all \(t\in[0, T]\), then system (1.2) is reduced to

$$ \textstyle\begin{cases} D^{\alpha_{2}}[ D^{\alpha_{1}}+\lambda_{1} ] x(t)=f(t,x(t),y(t)),& 0< t< T, \\ D^{\beta_{2}}[ D^{\beta_{1}}+\lambda_{2} ] y(t)=g(t,x(t),y(t)), & 0< t< T, \end{cases} $$
(1.5)

which are Langevin fractional differential equations.

The paper is organized as follows. In Section 2, we recall definitions from fractional calculus and present an auxiliary lemma. The main results for the coupled system of generalized Sturm-Liouville and Langevin fractional differential equations with anti-periodic boundary conditions are discussed in Section 3. We give an existence and uniqueness result with the help of Banach’s contraction mapping principle and an existence result via the Leray-Schauder alternative. Our results are well illustrated with the aid of examples presented in Section 4.

2 Preliminaries

In this section, we introduce some notations and definitions of fractional calculus (see [2]) and present preliminary results needed in our proofs later.

Definition 2.1

For an \((n-1)\)-times absolutely continuous function \(f: [0,\infty) \to\mathbb{R}\), the Caputo derivative of fractional order \(\alpha>0\) is defined as

$$D^{\alpha} f(t)=\frac{1}{\Gamma(n-\alpha)}\int_{0}^{t}(t-s)^{n- \alpha-1}f^{(n)}(s)\,ds, \quad n-1 < \alpha< n, $$

where \(n=[\alpha]+1\), \([\alpha]\) denotes the integer part of the positive real number α, and \(\Gamma(\cdot)\) is the gamma function.

Definition 2.2

The Riemann-Liouville fractional integral of order α of a function \(f:[0,\infty)\to\mathbb{R}\) is defined as

$$ I^{\alpha}f(t)=\frac{1}{\Gamma(\alpha)} \int_{0}^{t}\frac{f(s)}{(t-s)^{1- \alpha}}\,ds,\quad \alpha>0, $$
(2.1)

provided the integral exists.

Lemma 2.1

For \(\alpha>0\), the general solution of the fractional differential equation \(D^{\alpha}x(t)=0\) is given by

$$ x(t)=c_{0}+c_{1}t+\cdots+c_{n-1}t^{n-1}, $$
(2.2)

where \(c_{i}\in\mathbb{R}\), \(i=0,1,2,\ldots,n-1\) (\(n=[\alpha]+1\)).

In view of Lemma 2.1, it follows that

$$ I^{\alpha}D^{\alpha}x(t)=x(t)+c_{0}+c_{1}t+ \cdots+c_{n-1}t^{n-1} $$
(2.3)

for some \(c_{i}\in\mathbb{R}\), \(i=0,1,2,\ldots,n-1\).

Lemma 2.2

Let \(u,v\in C( [0,T],\mathbb{R} )\) be two given functions. Then the following linear system of fractional differential equations subject to anti-periodic boundary conditions

$$\begin{aligned} \textstyle\begin{cases} D^{\alpha_{2}}([p(t)D^{\alpha_{1}}+r(t)]x(t))=u(t),\quad 0< t< T, \\ D^{\beta_{2}}([q(t)D^{\beta_{1}}+s(t)]y(t))=v(t),\quad 0< t< T, \\ x(0)=-x(T),\qquad D^{\alpha_{1}}x(0)=-D^{\alpha_{1}}x(T), \\ y(0)=-y(T),\qquad D^{\beta_{1}}y(0)=-D^{\beta_{1}}y(T), \end{cases}\displaystyle \end{aligned}$$
(2.4)

is equivalent to the following integral equations:

$$\begin{aligned} x(t) =& I^{\alpha_{1}} \biggl( \frac{1}{p}I^{\alpha_{2}}u \biggr) (t)-I ^{\alpha_{1}} \biggl( \frac{r}{p}x \biggr) (t) \\ &{}+ \biggl( \frac{-\gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}u(T)+\frac {\eta _{1}}{\gamma_{1}+1}x(T) \biggr) I^{\alpha_{1}} \biggl( \frac{1}{p} \biggr) (t) \\ &{}-\frac{1}{2} \biggl[I^{\alpha_{1}} \biggl( \frac{1}{p}I^{\alpha_{2}}u \biggr) (T)-I ^{\alpha_{1}} \biggl( \frac{r}{p}x \biggr) (T) \\ &{}+ \biggl( \frac{-\gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}u(T)+\frac {\eta _{1}}{\gamma_{1}+1}x(T) \biggr) \rho_{1} \biggr] \end{aligned}$$
(2.5)

and

$$\begin{aligned} y(t) =&I^{\beta_{1}} \biggl( \frac{1}{q}I^{\beta_{2}}v \biggr) (t)-I^{\beta _{1}} \biggl( \frac{s}{q}y \biggr) (t) \\ &{}+ \biggl( \frac{-\gamma_{2}}{\gamma_{2}+1}I^{\beta_{2}}v(T)+\frac{\eta _{2}}{\gamma_{2}+1}y(T) \biggr) I^{\beta_{1}} \biggl( \frac{1}{q} \biggr) (t) \\ &{}-\frac{1}{2} \biggl[I^{\beta_{1}} \biggl( \frac{1}{q}I^{\beta_{2}}v \biggr) (T)-I ^{\beta_{1}} \biggl( \frac{s}{q}y \biggr) (T) \\ &{}+ \biggl( \frac{-\gamma_{2}}{\gamma_{2}+1}I^{\beta_{2}}v(T)+\frac{\eta _{2}}{\gamma_{2}+1}y(T) \biggr) \rho_{2} \biggr], \end{aligned}$$
(2.6)

where

$$\begin{aligned}& \gamma_{1} = \frac{p(0)}{p(T)},\qquad \gamma_{2}= \frac{q(0)}{q(T)},\qquad \eta_{1} = \gamma_{1}r(T)-r(0),\qquad \eta_{2}=\gamma_{2}s(T)-s(0), \\& \rho_{1} = I^{\alpha_{1}} \biggl( \frac{1}{p} \biggr) (T),\qquad \rho_{2}=I^{\beta_{1}} \biggl( \frac{1}{q} \biggr) (T). \end{aligned}$$

Proof

Taking the Riemann-Liouville fractional integral of orders \(\alpha_{2}\), \(\beta_{2}\) into the first two equations of system (2.4), we have

$$\begin{aligned}& D^{\alpha_{1}}x(t) = \frac{I^{\alpha_{2}}u(t)-r(t)x(t)+c_{0}}{p(t)}, \end{aligned}$$
(2.7)
$$\begin{aligned}& D^{\beta_{1}}y(t) = \frac{I^{\beta_{2}}v(t)-s(t)y(t)+k_{0}}{q(t)}, \end{aligned}$$
(2.8)

where \(c_{0},k_{0}\in\mathbb{R}\). From the boundary conditions of (2.4), we obtain

$$c_{0} =\frac{-\gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}u(T)+\frac{\eta _{1}}{\gamma_{1}+1}x(T) $$

and

$$k_{0} =\frac{-\gamma_{2}}{\gamma_{2}+1}I^{\beta_{2}}v(T)+\frac{\eta _{2}}{\gamma_{2}+1}y(T). $$

Taking the Riemann-Liouville fractional integral of orders \(\alpha _{1}\), \(\beta_{1}\) into (2.7), (2.8), respectively, we get

$$ x(t)=I^{\alpha_{1}} \biggl( \frac{1}{p}I^{\alpha_{2}}u \biggr) (t)-I^{ \alpha_{1}} \biggl( \frac{r}{p}x \biggr) (t)+c_{0}I^{\alpha_{1}} \biggl( \frac{1}{p} \biggr) (t)+c _{1} $$
(2.9)

and

$$ y(t)=I^{\beta_{1}} \biggl( \frac{1}{q}I^{\beta_{2}}v \biggr) (t)-I^{\beta _{1}} \biggl( \frac{s}{q}y \biggr) (t)+k_{0}I^{\beta_{1}} \biggl( \frac{1}{q} \biggr) (t)+k _{1}, $$
(2.10)

where \(c_{1},k_{1}\in\mathbb{R}\). Using the boundary conditions of (2.4), we have

$$\begin{aligned} c_{1} =&-\frac{1}{2} \biggl[ I^{\alpha_{1}} \biggl( \frac{1}{p}I^{\alpha _{2}}u \biggr) (T)-I^{\alpha_{1}} \biggl( \frac{r}{p}x \biggr) (T)+ \biggl( \frac{- \gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}u(T)+ \frac{\eta_{1}}{\gamma _{1}+1}x(T) \biggr) \rho_{1} \biggr] \end{aligned}$$

and

$$\begin{aligned} k_{1} =&-\frac{1}{2} \biggl[ I^{\beta_{1}} \biggl( \frac{1}{q}I^{\beta_{2}}v \biggr) (T)-I ^{\beta_{1}} \biggl( \frac{s}{q}y \biggr) (T)+ \biggl( \frac{-\gamma_{2}}{ \gamma_{2}+1}I^{\beta_{2}}v(T)+ \frac{\eta_{2}}{\gamma_{2}+1}y(T) \biggr) \rho_{2} \biggr] . \end{aligned}$$

Substituting the values of constants \(c_{0}\), \(c_{1}\), \(k_{0}\) and \(k_{1}\) into (2.9) and (2.10), we obtain the integral equations (2.5) and (2.6), respectively. The converse follows by a direct computation. This completes the proof. □

3 Main results

Throughout this paper, for convenience, we use the following expression:

$$I^{\phi}h\bigl(s,x(s),y(s)\bigr) (i)=\frac{1}{\Gamma(\phi)} \int_{0}^{i}\frac{h(s,x(s),y(s))}{(i-s)^{1- \phi}}\,ds, $$

where \(\phi\in\{\alpha_{1},\alpha_{2},\beta_{1}\beta_{2}\}\), \(i\in\{t,T\}\), \(h=\{f,g\}\). Let us introduce the space \(X=\{x(t)\vert x(t)\in C([0,T], {\mathbb{R}})\}\) endowed with the norm \(\Vert x\Vert =\sup\{\vert x(t)\vert , t\in[0,T]\}\). It is obvious that \((X,\Vert \cdot \Vert )\) is a Banach space. In addition, the product space \((X\times X,\Vert (x,y)\Vert )\) is a Banach space with the norm \(\Vert (x,y)\Vert =\Vert x\Vert +\Vert y\Vert \). In view of Lemma (2.2), we define an operator \(\mathcal{A}:X\times X \rightarrow X\times X\) by

A(x,y)(t)= ( A 1 ( x , y ) ( t ) A 2 ( x , y ) ( t ) ) ,

where

$$\begin{aligned} \mathcal{A}_{1}(x,y) (t) =&I^{\alpha_{1}} \biggl( \frac{1}{p}I^{\alpha _{2}}f\bigl(s,x(s),y(s)\bigr) \biggr) (t)-I^{\alpha_{1}} \biggl( \frac{r}{p}x \biggr) (t) \\ &{}+ \biggl( \frac{-\gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}f\bigl(s,x(s),y(s)\bigr) (T)+ \frac{ \eta_{1}}{\gamma_{1}+1}x(T) \biggr) I^{\alpha_{1}} \biggl( \frac{1}{p} \biggr) (t) \\ &{}-\frac{1}{2} \biggl[I^{\alpha_{1}} \biggl( \frac{1}{p}I^{\alpha_{2}}f \bigl(s,x(s),y(s)\bigr) \biggr) (T)-I ^{\alpha_{1}} \biggl( \frac{r}{p}x \biggr) (T) \\ &{}+ \biggl( \frac{-\gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}f\bigl(s,x(s),y(s)\bigr) (T)+ \frac{ \eta_{1}}{\gamma_{1}+1}x(T) \biggr) \rho_{1} \biggr] \end{aligned}$$

and

$$\begin{aligned} \mathcal{A}_{2}(x,y) (t) =&I^{\beta_{1}} \biggl( \frac{1}{q}I^{\beta_{2}}g\bigl(s,x(s),y(s)\bigr) \biggr) (t)-I ^{\beta_{1}} \biggl( \frac{s}{q}y \biggr) (t) \\ &{}+ \biggl( \frac{-\gamma_{2}}{\gamma_{2}+1}I^{\beta_{2}}g\bigl(s,x(s),y(s)\bigr) (T)+ \frac{ \eta_{2}}{\gamma_{2}+1}y(T) \biggr) I^{\beta_{1}} \biggl( \frac{1}{q} \biggr) (t) \\ &{}-\frac{1}{2} \biggl[I^{\beta_{1}} \biggl( \frac{1}{q}I^{\beta_{2}}g \bigl(s,x(s),y(s)\bigr) \biggr) (T)-I ^{\beta_{1}} \biggl( \frac{s}{q}y \biggr) (T) \\ &{}+ \biggl( \frac{-\gamma_{2}}{\gamma_{2}+1}I^{\beta_{2}}g\bigl(s,x(s),y(s)\bigr) (T)+ \frac{ \eta_{2}}{\gamma_{2}+1}y(T) \biggr) \rho_{2} \biggr]. \end{aligned}$$

We set the following constants:

$$p^{*}=\inf_{t\in[0,T]}\bigl\vert p(t)\bigr\vert ,\qquad q^{*}=\inf_{t\in[0,T]}\bigl\vert q(t)\bigr\vert ,\qquad r^{*}=\sup_{t\in[0,T]}\bigl\vert r(t)\bigr\vert ,\qquad s^{*}=\sup_{t\in[0,T]}\bigl\vert s(t)\bigr\vert $$

and

$$\begin{aligned}& h_{1}=\frac{T^{\alpha_{1}+\alpha_{2}}}{p^{*}\Gamma (1+\alpha_{1}+ \alpha_{2})},\qquad h_{2}= \frac{r^{*}T^{\alpha_{1}}}{p^{*}\Gamma(1+\alpha_{1})}, \\& h_{3}=\frac {\gamma_{1} T^{\alpha_{1}+\alpha_{2}}}{p^{*}(\gamma_{1}+1) \Gamma(1+\alpha_{1})\Gamma(1+\alpha_{2})},\qquad h_{4}=\frac{\vert \eta_{1}\vert T^{\alpha_{1}}}{p^{*}(\gamma _{1}+1)\Gamma (1+\alpha_{1})}, \\& h_{5}=\frac{T^{\beta_{1}+\beta_{2}}}{q^{*}\Gamma (1+\beta_{1}+\beta _{2})},\qquad h_{6}=\frac{s^{*}T^{\beta_{1}}}{q^{*}\Gamma{(1+\beta_{1})}}, \\& h_{7}=\frac{\gamma_{2} T^{\beta_{1}+\beta_{2}}}{q^{*}(\gamma_{2}+1) \Gamma{(1+\beta_{1})}\Gamma{(1+\beta_{2})}},\qquad h_{8}=\frac{\vert \eta _{2}\vert T^{\beta_{1}}}{q^{*}(\gamma_{2}+1)\Gamma {(1+\beta_{1})}}. \end{aligned}$$

Theorem 3.1

Assume that \(f,g:[0,T]\times\mathbb{R}^{2}\rightarrow\mathbb{R}\) are continuous functions, and there exist constants \(m_{i}\), \(n_{i}\), \(i=1,2\) such that for all \(t\in[0,T]\) and \(x_{i},y_{i}\in \mathbb{R}\), \(i=1,2\),

(\(H_{1}\)):

\(\vert f(t,x_{1},y_{1})-f(t,x_{2},y_{2})\vert \leq m_{1} \vert x_{1}-x_{2}\vert +m_{2}\vert y_{1}-y_{2}\vert \),

(\(H_{2}\)):

\(\vert g(t,x_{1},y_{1})-g(t,x_{2},y_{2})\vert \leq n_{1} \vert x_{1}-x_{2}\vert +n_{2}\vert y_{1}-y_{2}\vert \).

In addition, let

$$L_{1}+L_{2}< 1, $$

where

$$\begin{aligned}& M_{1} = \frac{3}{2}(h_{1}+h_{3}),\qquad M_{2} = \frac{3}{2}(h_{2}+h_{4}),\qquad M_{3} = \frac{3}{2}(h_{5}+ h_{7}),\qquad M_{4} = \frac{3}{2}(h_{6}+h_{8}), \\& L_{1} =(m_{1} +m_{2})M_{1}+M_{2},\qquad L_{2} = (n_{1} +n_{2})M_{3}+M_{4}. \end{aligned}$$

Then problem (1.2)-(1.3) has a unique solution on \([0, T]\).

Proof

To show that problem (1.2)-(1.3) has a unique solution on \([0,T]\), we will use Banach’s contraction mapping principle. In the first step, we define \(\sup_{t\in[0,T]}f(t,0,0)=N _{1}<\infty\), \(\sup_{t\in[0,T]}g(t,0,0)=N_{2}<\infty\) and choose a positive real number w such that

$$w\geq\frac{N_{1}M_{1}+N_{2}M_{2}}{1-L_{1}-L_{2}}. $$

Now, we show that \(\mathcal{A}B_{w}\subset B_{w}\), where \(B_{w}=\{(x,y) \in X\times X:\Vert (x,y)\Vert \leq w\}\). For any \((x,y)\in B_{w}\), we have

$$\begin{aligned}& \bigl\vert \mathcal{A}_{1}(x,y) (t)\bigr\vert \\& \quad \leq \sup_{t\in[0,T]} \biggl\{ I^{\alpha_{1}} \biggl( \frac{1}{p^{*}}I ^{\alpha_{2}}\bigl\vert f\bigl(s,x(s),y(s)\bigr)\bigr\vert \biggr) (t)+I^{\alpha_{1}} \biggl( \frac{r ^{*}}{p^{*}}\bigl\vert x(s) \bigr\vert \biggr) (t) \\& \qquad {}+ \biggl( \frac{\gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}\bigl\vert f\bigl(s,x(s),y(s) \bigr)\bigr\vert (T)+\frac{\vert \eta_{1}\vert }{\gamma_{1}+1}\bigl\vert x(T)\bigr\vert \biggr) \biggl\vert I^{\alpha_{1}} \biggl( \frac{1}{p} \biggr) (t)\biggr\vert \\& \qquad {}+\frac{1}{2} \biggl[I^{\alpha_{1}} \biggl( \frac{1}{p^{*}}I^{\alpha_{2}} \bigl\vert f\bigl(s,x(s),y(s)\bigr)\bigr\vert \biggr) (T)+I ^{\alpha_{1}} \biggl( \frac{r^{*}}{p^{*}}\bigl\vert x(s)\bigr\vert \biggr) (T) \\& \qquad {}+ \biggl( \frac{\gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}\bigl\vert f\bigl(s,x(s),y(s) \bigr)\bigr\vert (T)+\frac{\vert \eta_{1}\vert }{\gamma_{1}+1}\bigl\vert x(T)\bigr\vert \biggr) \vert \rho_{1}\vert \biggr] \biggr\} \\& \quad \leq I^{\alpha_{1}} \biggl( \frac{1}{p^{*}}I^{\alpha_{2}}\bigl\vert f \bigl(s,x(s),y(s)\bigr)-f(s,0,0)\bigr\vert +\bigl\vert f(s,0,0)\bigr\vert \biggr) (T) \\& \qquad {}+I^{\alpha_{1}} \biggl( \frac{r^{*}}{p^{*}}\bigl\vert x(s)\bigr\vert \biggr) (T)+ \biggl(\frac{ \gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}\bigl(\bigl\vert f \bigl(s,x(s),y(s)\bigr)-f(s,0,0)\bigr\vert \\& \qquad {}+\bigl\vert f(s,0,0)\bigr\vert \bigr) (T)+\frac{\vert \eta _{1}\vert }{\gamma_{1}+1}\bigl\vert x(T)\bigr\vert \biggr)\biggl\vert I ^{\alpha_{1}} \biggl( \frac{1}{p} \biggr) (T)\biggr\vert \\& \qquad {}+\frac{1}{2} \biggl[I^{\alpha_{1}} \biggl( \frac{1}{p^{*}}I^{\alpha_{2}} \bigl\vert f\bigl(s,x(s),y(s)\bigr)-f(s,0,0)\bigr\vert +\bigl\vert f(s,0,0) \bigr\vert \biggr) (T) \\& \qquad {}+I^{\alpha_{1}} \biggl( \frac{r^{*}}{p^{*}}\bigl\vert x(s)\bigr\vert \biggr) (T)+ \biggl(\frac{ \gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}\bigl(\bigl\vert f \bigl(s,x(s),y(s)\bigr)-f(s,0,0)\bigr\vert \\& \qquad {}+\bigl\vert f(s,0,0)\bigr\vert \bigr) (T)+\frac{\vert \eta _{1}\vert }{\gamma_{1}+1}\bigl\vert x(T)\bigr\vert \biggr) \vert \rho_{1}\vert \biggr] \\& \quad \leq I^{\alpha_{1}} \biggl( \bigl(m_{1}\Vert x\Vert +m_{2}\Vert y\Vert +N_{1}\bigr)\frac{1}{p ^{*}}I^{\alpha_{2}}(1) \biggr) (T)+ \biggl( \frac{r^{*}}{p^{*}} \biggr) \Vert x\Vert I^{\alpha_{1}}(1) (T) \\& \qquad {}+ \biggl(\frac{\gamma_{1}}{\gamma_{1}+1}\bigl(m_{1}\Vert x\Vert +m_{2}\Vert y\Vert +N_{1}\bigr)I ^{\alpha_{2}}(1) (T)+ \frac{\vert \eta_{1}\vert }{\gamma_{1}+1}\Vert x\Vert \biggr)\biggl\vert I ^{\alpha_{1}} \biggl( \frac{1}{p} \biggr) (T)\biggr\vert \\& \qquad {}+\frac{1}{2} \biggl[I^{\alpha_{1}} \biggl( \bigl(m_{1} \Vert x\Vert +m_{2}\Vert y\Vert +N_{1}\bigr) \frac{1}{p ^{*}}I^{\alpha_{2}}(1) \biggr) (T)+ \biggl( \frac{r^{*}}{p^{*}} \Vert x\Vert \biggr) I ^{\alpha_{1}}(1) (T) \\& \qquad {}+ \biggl(\frac{\gamma_{1}}{\gamma_{1}+1}\bigl(m_{1}\Vert x\Vert +m_{2}\Vert y\Vert +N_{1}\bigr)I ^{\alpha_{2}}(1) (T)+ \frac{\vert \eta_{1}\vert }{\gamma_{1}+1}\Vert x\Vert \biggr) \vert \rho_{1}\vert \biggr] \\& \quad \leq\bigl(m_{1}\Vert x\Vert +m_{2}\Vert y\Vert +N_{1}\bigr)\frac{T^{\alpha_{1}+\alpha_{2}}}{p ^{*}\Gamma(1+\alpha_{1}+\alpha_{2})}+\frac{r^{*}T^{\alpha_{1}}}{p ^{*}\Gamma{(1+\alpha_{1})}}\Vert x\Vert \\& \qquad {}+\bigl(m_{1}\Vert x\Vert +m_{2}\Vert y\Vert +N_{1}\bigr)\frac{\gamma_{1} T^{\alpha_{1}+\alpha _{2}}}{p^{*}(\gamma_{1}+1)\Gamma{(1+\alpha_{1})}\Gamma{(1+\alpha _{2})}} \\& \qquad {}+\frac{\vert \eta_{1}\vert T^{\alpha_{1}}}{p^{*}(\gamma _{1}+1)\Gamma {(1+\alpha_{1})}}\Vert x\Vert +\frac{1}{2}\frac{\vert \eta _{1}\vert T^{\alpha_{1}}}{p ^{*}(\gamma_{1}+1)\Gamma{(1+\alpha_{1})}} \Vert x\Vert \\& \qquad {}+\frac{1}{2}\bigl(m_{1}\Vert x\Vert +m_{2} \Vert y\Vert +N_{1}\bigr)\frac{T^{\alpha_{1}+\alpha _{2}}}{p^{*}\Gamma(1+\alpha_{1}+\alpha_{2})}+\frac{1}{2} \frac{r^{*}T ^{\alpha_{1}}}{p^{*}\Gamma{(1+\alpha_{1})}}\Vert x\Vert \\& \qquad {}+\frac{1}{2}\frac{\gamma_{1} T^{\alpha_{1}+\alpha_{2}}}{p^{*}( \gamma_{1}+1)\Gamma{(1+\alpha_{1})}\Gamma{(1+\alpha_{2})}}\bigl(m_{1}\Vert x\Vert +m_{2}\Vert y\Vert +N_{1}\bigr) \\& \quad =\frac{3}{2}\bigl(m_{1}\Vert x\Vert +m_{2} \Vert y\Vert +N_{1}\bigr) ( h_{1}+h_{3} ) + \frac{3}{2} ( h_{2}+h_{4} ) \Vert x\Vert \\& \quad =(m_{1}M_{1}+M_{2})\Vert x\Vert +m_{2}M_{1}\Vert y\Vert +N_{1}M_{1} \\& \quad \leq(m_{1}M_{1}+M_{2})w+m_{2}M_{1}w+N_{1}M_{1} \\& \quad =L_{1}w+N_{1}M_{1}. \end{aligned}$$

Similarly, we obtain

$$\begin{aligned}& \bigl\vert \mathcal{A}_{2}(x,y) (t)\bigr\vert \\& \quad \leq\bigl(n_{1}\Vert x\Vert +n_{2}\Vert y\Vert +N_{2}\bigr)\frac{T^{\beta_{1}+\beta_{2}}}{q ^{*}\Gamma(1+\beta_{1}+\beta_{2})}+\frac{s^{*} T^{\beta_{1}}}{q^{*} \Gamma{(1+\beta_{1})}}\Vert y\Vert \\& \qquad {}+\bigl(n_{1}\Vert x\Vert +n_{2}\Vert y\Vert +N_{2}\bigr)\frac{\gamma_{2}T^{\beta_{1}+\beta _{2}}}{q^{*}(\gamma_{2}+1)\Gamma{(1+\beta_{1})}\Gamma{(1+\beta_{2})}} \\& \qquad {}+\frac{\vert \eta_{2}\vert T^{\beta_{1}}}{q^{*}(\gamma _{2}+1)\Gamma{(1+\beta _{1})}}\Vert y\Vert +\frac{1}{2}\frac{\vert \eta_{2}\vert T^{\beta_{1}}}{q^{*}(\gamma _{2}+1)\Gamma{(1+\beta_{1})}} \Vert y\Vert \\& \qquad {}+\frac{1}{2}\bigl(n_{1}\Vert x\Vert +n_{2} \Vert y\Vert +N_{2}\bigr)\frac{T^{\beta_{1}+\beta _{2}}}{q^{*}\Gamma(1+\beta_{1}+\beta_{2})}+\frac{1}{2} \frac{s^{*}T ^{\beta_{1}}}{q^{*}\Gamma{(1+\beta_{1})}}\Vert y\Vert \\& \qquad {}+\frac{1}{2}\frac{\gamma_{2} T^{\beta_{1}+\beta_{2}}}{q^{*}(\gamma _{2}+1)\Gamma{(1+\beta_{1})}\Gamma{(1+\beta_{2})}}\bigl(n_{1}\Vert x\Vert +n_{2} \Vert y\Vert +N_{2}\bigr) \\& \quad =\frac{3}{2}\bigl(n_{1}\Vert x\Vert +n_{2} \Vert y\Vert +N_{2}\bigr) ( h_{5}+h_{7} ) + \frac{3}{2} ( h_{6}+h_{8} ) \Vert y\Vert \\& \quad =n_{1}M_{3}\Vert x\Vert +(n_{2}M_{3}+M_{4}) \Vert y\Vert +N_{2}M_{3} \\& \quad \leq n_{1}M_{3}w + (n_{2}M_{3}+M_{4})w+N_{2}M_{3} \\& \quad =L_{2}w+N_{2}M_{3}. \end{aligned}$$

Therefore, we deduce that

$$\Vert \mathcal{A}(x,y)\Vert =\Vert \mathcal{A}_{1}(x,y)\Vert + \Vert \mathcal{A}_{2}(x,y)\Vert \le(L_{1}+L_{2})w+N_{1}M_{1}+N_{2}M_{3}< w, $$

which implies \(\mathcal{A}B_{w}\subset B_{w}\).

Next, for \((x_{2},y_{2}),(x_{1},y_{1})\in X\times X \) and for any \(t\in[0,T]\), we have

$$\begin{aligned}& \bigl\vert \mathcal{A}_{1}(x_{2},y_{2}) (t)- \mathcal{A}_{1}(x_{1},y_{1}) (t)\bigr\vert \\& \quad \leq I^{\alpha_{1}} \biggl( \frac{1}{p}I^{\alpha_{2}}\bigl\vert f \bigl(s,x_{2}(s),y_{2}(s)\bigr)-f\bigl(s,x_{1}(s),y_{1}(s) \bigr)\bigr\vert \biggr) (T) \\& \qquad {}+I^{\alpha_{1}} \biggl( \frac{r^{*}}{p^{*}}\bigl\vert x_{2}(s)-x_{1}(s) \bigr\vert \biggr) (T) \\& \qquad {}+ (\frac{\gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}\biggl(\bigl\vert f\bigl(s,x_{2}(s),y_{2}(s) \bigr)-f\bigl(s,x_{1}(s),y_{1}(s)\bigr)\bigr\vert (T) \\& \qquad {}+\frac{\vert \eta_{1}\vert }{\gamma_{1}+1}\bigl\vert x_{2}(T)-x_{1}(T)\bigr\vert \biggr)\biggl\vert I ^{\alpha_{1}} \biggl( \frac{1}{p} \biggr) (T)\biggr\vert \\& \qquad {}+\frac{1}{2} \biggl[I^{\alpha_{1}} \biggl( \frac{1}{p}I^{\alpha_{2}} \bigl\vert f\bigl(s,x_{2}(s),y_{2}(s)\bigr)-f \bigl(s,x_{1}(s),y_{1}(s)\bigr)\bigr\vert \biggr) (T) \\& \qquad {}+I^{\alpha_{1}} \biggl( \frac{r^{*}}{p}\bigl\vert x_{2}(s)-x_{1}(s) \bigr\vert \biggr) (T)+ (\frac{\gamma_{1}}{\gamma_{1}+1}I^{\alpha _{2}} \biggl(\bigl\vert f \bigl(s,x_{2}(s),y_{2}(s)\bigr) \\& \qquad {}-f\bigl(s,x_{1}(s),y_{1}(s)\bigr)\bigr\vert (T) + \frac{\vert \eta_{1}\vert }{\gamma_{1}+1}\bigl\vert x_{2}(T)-x_{1}(T)\bigr\vert \biggr) \vert \rho_{1}\vert \biggr] \\& \quad \leq\frac{T^{\alpha_{1}+\alpha_{2}}}{p^{*}\Gamma(1+\alpha_{1}+ \alpha_{2})}\bigl(m_{1}\Vert x_{2}-x_{1} \Vert +m_{2}\Vert y_{2}-y_{1}\Vert \bigr)+ \frac{r^{*}T ^{\alpha_{1}}}{p^{*}\Gamma{(1+\alpha_{1})}}\Vert x_{2}-x_{1}\Vert \\& \qquad {}+\frac{\gamma_{1} T^{\alpha_{1}+\alpha_{2}}}{p^{*}(\gamma_{1}+1) \Gamma{(1+\alpha_{1})}\Gamma{(1+\alpha_{2})}}\bigl(m_{1}\Vert x_{2}-x_{1} \Vert +m _{2}\Vert y_{2}-y_{1}\Vert \bigr) \\& \qquad {}+\frac{\vert \eta_{1}\vert T^{\alpha_{1}}}{p^{*}(\gamma _{1}+1)\Gamma {(1+\alpha_{1})}}\Vert x_{2}-x_{1}\Vert + \frac{1}{2}\frac{\vert \eta_{1}\vert T^{ \alpha_{1}}}{p^{*}(\gamma_{1}+1)\Gamma{(1+\alpha_{1})}}\Vert x_{2}-x_{1} \Vert \\& \qquad {}+\frac{1}{2}\frac{T^{\alpha_{1}+\alpha_{2}}}{p^{*}\Gamma(\alpha _{1}+\alpha_{2})}\bigl(m_{1}\Vert x_{2}-x_{1}\Vert +m_{2}\Vert y_{2}-y_{1}\Vert \bigr)+ \frac{1}{2} \frac{r^{*}T^{\alpha_{1}}}{p^{*}\Gamma{(\alpha_{1})}}\Vert x_{2}-x_{1}\Vert \\& \qquad {}+\frac{1}{2}\frac{\gamma_{1} T^{1+\alpha_{1}+\alpha_{2}}}{p^{*}( \gamma_{1}+1)\Gamma{(\alpha_{1})}\Gamma{(\alpha_{2})}}\bigl(m_{1}\Vert x_{2}-x_{1}\Vert +m_{2} \Vert y_{2}-y_{1}\Vert \bigr) \\& \quad =\frac{3}{2} ( h_{1}+h_{3} ) \bigl(m_{1}\Vert x_{2}-x_{1}\Vert +m_{2} \Vert y_{2}-y_{1}\Vert \bigr)+ \frac{3}{2} ( h_{2}+h_{4} ) \Vert x_{2}-x_{1}\Vert \\& \quad =(m_{1}M_{1}+M_{2})\Vert x_{2}-x_{1}\Vert +m_{2}M_{1}\Vert y_{2}-y_{1}\Vert , \end{aligned}$$

which leads to

$$ \bigl\Vert \mathcal{A}_{1}(x_{2},y_{2})- \mathcal{A}_{1}(x_{1},y_{1})\bigr\Vert \leq L _{1}\bigl(\Vert x_{2}-x_{1}\Vert +\Vert y_{2}-y_{1}\Vert \bigr). $$
(3.1)

Similarly, we obtain

$$ \bigl\Vert \mathcal{A}_{2}(x_{2},y_{2})- \mathcal{A}_{2}(x_{1},y_{1})\bigr\Vert \leq L _{2}\bigl(\Vert x_{2}-x_{1}\Vert +\Vert y_{2}-y_{1}\Vert \bigr). $$
(3.2)

Hence, from (3.1) and (3.2), we deduce that

$$\bigl\Vert \mathcal{A}(x_{2},y_{2})-\mathcal{A}(x_{1},y_{1}) \bigr\Vert \leq(L_{1}+L _{2}) \bigl(\Vert x_{2}-x_{1}\Vert +\Vert y_{2}-y_{1} \Vert \bigr). $$

Since \(L_{1}+L_{2}<1\), \(\mathcal{A}\) is a contraction operator. Thus, by Banach’s fixed point theorem, the operator \(\mathcal{A}\) has a unique fixed point, which is the unique solution of problem (1.2)-(1.3) on \([0,T]\). This completes the proof. □

If \(r(t),s(t)\equiv0\) for all \(t\in[0,T]\), then we have \(\eta_{1}, \eta_{2}=0\).

Corollary 3.1

Suppose that conditions \((H_{1})\)-\((H_{2})\) hold. If \((m_{1}+m_{2})M _{1}+(n_{1}+n_{2})M_{3}<1\), then system (1.4) with (1.3) has a unique solution on \([0,T]\).

If \(p(t),q(t)\equiv1\) and \(r(t)\equiv\lambda_{1}\), \(s(t)=\lambda _{2}\) for all \(t\in[0, T]\), then we get that \(p^{\ast},q^{\ast}=1\), \(r^{\ast}=\vert \lambda_{1}\vert \), \(s^{\ast}=\vert \lambda _{2}\vert \), \(\gamma_{1}, \gamma_{2}=1\), \(\eta_{1},\eta_{2}=0\). Let

$$\begin{aligned}& a_{1}=\frac{3}{2}\frac{T^{\alpha_{1}+\alpha_{2}}}{\Gamma(1+\alpha _{1}+\alpha_{2})}+\frac{3}{4} \frac{T^{\alpha_{1}+\alpha_{2}}}{\Gamma (1+\alpha_{1})\Gamma(1+\alpha_{2})}, \\& a_{2}=\frac{3}{2}\vert \lambda_{1}\vert \frac{T^{\alpha_{1}}}{\Gamma(1+\alpha_{1})}, \\& a_{3}=\frac{3}{2}\frac{T^{\beta_{1}+\beta_{2}}}{\Gamma(1+\beta_{1}+ \beta_{2})}+\frac{3}{4} \frac{T^{\beta_{1}+\beta_{2}}}{\Gamma(1+\beta _{1})\Gamma(1+\beta_{2})},\\& a_{4}=\frac{3}{2}\vert \lambda_{2}\vert \frac{T ^{\beta_{1}}}{\Gamma(1+\beta_{1})}. \end{aligned}$$

Corollary 3.2

Assume that conditions \((H_{1})\)-\((H_{2})\) are satisfied. If

$$\bigl[a_{1}(m_{1}+m_{2})+a_{2}\bigr]+ \bigl[a_{3}(n_{1}+n_{2})+a_{4}\bigr]< 1, $$

then system (1.5) with (1.3) has a unique solution on \([0,T]\).

In the next result, we will show the existence of solutions of problem (1.2)-(1.3) by applying the Leray-Schauder alternative.

Lemma 3.1

Leray-Schauder alternative [34]

Let G be a normed linear space and \(F:G\rightarrow G\) be a completely continuous operator (i.e., a map restricted to any bounded set in G is compact). Let

$$\mathcal{J}(F)=\bigl\{ x\in G : x=\kappa F(x) \textit{ for some } 0< \kappa < 1\bigr\} . $$

Then either the set \(\mathcal{J}(F)\) is unbounded, or F has at least one fixed point.

Theorem 3.2

Assume that \(f,g:(0,T)\times\mathbb{R}^{2}\rightarrow\mathbb{R}\) are two continuous functions and there exist real constants \(C_{i},D_{i} \geq0\) (\(i=1,2\)) and \(C_{0},D_{0}>0\) such that \(\forall x_{i},y_{i} \in\mathbb{R}\), \((i=1,2)\) satisfying

(\(H_{3}\)):

\(\vert f(t,x_{1},x_{2})\vert \leq C_{0}+C_{1}\vert x_{1}\vert +C_{2} \vert x_{2}\vert \) and

(\(H_{4}\)):

\(\vert g(t,y_{1},y_{2})\vert \leq D_{0}+D_{1}\vert y_{1}\vert +D_{2} \vert y_{2}\vert \).

In addition, it is assumed that

$$J_{1}< 1\quad \textit{and}\quad J_{2}< 1, $$

where

$$J_{1} = 1-(C_{1}M_{1}+M_{2}+D_{1}M_{3}), \quad \textit{and}\quad J_{2} = 1-(C_{2}M_{1}+D_{2}M_{3}+M_{4}). $$

Then there exists at least one solution of problem (1.2)-(1.3) on \([0,T]\).

Proof

Firstly, we show that the operator \(\mathcal{A}:X\times X \rightarrow X\times X\) is completely continuous. Note that \(\mathcal{A}\) is continuous, since the functions f, g are continuous. Let \(U\subset X\times X\) be a bounded set. Then there exists a positive constant ŵ such that \(\Vert (x,y)\Vert \le\hat{w}\) for any \((x,y)\in U\). Also there exist \(S_{1}\) and \(S_{2}\) such that

$$\bigl\vert f\bigl(t,x(t),y(t)\bigr)\bigr\vert \leq S_{1},\qquad \bigl\vert g\bigl(t,x(t),y(t)\bigr)\bigr\vert \leq S_{2},\quad \forall(x,y) \in U. $$

Therefore, for any \((x,y)\in U\), we have

$$\begin{aligned}& \bigl\vert \mathcal{A}_{1}(x,y) (t)\bigr\vert \\& \quad \leq I^{\alpha_{1}} \biggl( \frac{1}{p}I^{\alpha_{2}}\bigl\vert f \bigl(s,x(s),y(s)\bigr)\bigr\vert \biggr) (T)+I ^{\alpha_{1}} \biggl( \frac{r^{*}}{p^{*}}\bigl\vert x(s)\bigr\vert \biggr) (T) \\& \qquad {}+ \bigg(\frac{\gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}\bigl\vert f\bigl(s,x(s),y(s) \bigr)\bigr\vert (T)+\frac{\vert \eta_{1}\vert }{\gamma_{1}+1}\bigl\vert x(T)\bigr\vert \biggr) \biggl\vert I^{\alpha_{1}} \biggl( \frac{1}{p} \biggr) (T)\biggr\vert \\& \qquad {}+\frac{1}{2} \biggl[I^{\alpha_{1}} \biggl( \frac{1}{p}I^{\alpha_{2}} \bigl\vert f\bigl(s,x(s),y(s)\bigr)\bigr\vert \biggr) (T)+I ^{\alpha_{1}} \biggl( \frac{r^{*}}{p^{*}}\bigl\vert x(s)\bigr\vert \biggr) (T) \\& \qquad {}+ \bigg(\frac{\gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}\bigl\vert f\bigl(s,x(s),y(s) \bigr)\bigr\vert (T)+\frac{\vert \eta_{1}\vert }{\gamma_{1}+1}\bigl\vert x(T)\bigr\vert \biggr) \vert \rho_{1}\vert \biggr] \\& \quad \leq\frac{T^{\alpha_{1}+\alpha_{2}}}{p^{*}\Gamma(1+\alpha_{1}+ \alpha_{2})}S_{1}+\frac{r^{*}T^{\alpha_{1}}}{p^{*}\Gamma{(1+\alpha _{1})}}\hat{w} + \frac{\gamma_{1} T^{\alpha_{1}+\alpha_{2}}}{p^{*}( \gamma_{1}+1)\Gamma{(1+\alpha_{1})}\Gamma{(1+\alpha_{2})}}S_{1} \\& \qquad {}+\frac{\vert \eta_{1}\vert T^{\alpha_{1}}}{p^{*}(\gamma _{1}+1)\Gamma{(1+ \alpha_{1})}}\hat{w}+\frac{1}{2}\frac{T^{\alpha_{1}+\alpha_{2}}}{p ^{*}\Gamma(1+\alpha_{1}+\alpha_{2})}S_{1} +\frac{1}{2}\frac{r^{*}T ^{\alpha_{1}}}{p^{*}\Gamma{(1+\alpha_{1})}}\hat{w} \\& \qquad {}+\frac{1}{2}\frac{\gamma_{1}T^{\alpha_{1}+\alpha_{2}}}{p^{*}(\gamma _{1}+1)\Gamma{(1+\alpha_{1})}\Gamma{(1+\alpha_{2})}}S_{1} + \frac{1}{2} \frac{\vert \eta_{1}\vert T^{\alpha_{1}}}{p^{*}(\gamma_{1}+1)\Gamma {(1+\alpha_{1})}}\hat{w} \\& \quad =\frac{3}{2} ( h_{1}+h_{3} ) S_{1}+ \frac{3}{2} ( h_{2}+h _{4} ) \hat{w} \\& \quad =M_{1}S_{1}+M_{2}\hat{w}. \end{aligned}$$

Thus \(\Vert \mathcal{A}_{1}(x,y)\Vert \le M_{1}S_{1}+M_{2}\hat{w}\).

Similarly, we deduce that

$$\begin{aligned} \bigl\vert \mathcal{A}_{2}(x,y) (t)\bigr\vert \le& \frac{T^{\beta_{1}+\beta_{2}}}{q^{*} \Gamma(1+\beta_{1}+\beta_{2})}S_{2}+\frac{s^{*}T^{\beta_{1}}}{q^{*} \Gamma{(1+\beta_{1})}}\hat{w} \\ &{}+\frac{\gamma_{2} T^{\beta_{1}+\beta_{2}}}{q^{*}(\gamma_{2}+1) \Gamma{(1+\beta_{1})}\Gamma{(1+\beta_{2})}}S_{2}+\frac{\vert \eta _{2}\vert T ^{\beta_{1}}}{q^{*}(\gamma_{2}+1)\Gamma{(1+\beta_{1})}}\hat{w} \\ &{}+\frac{1}{2}\frac{T^{\beta_{1}+\beta_{2}}}{q^{*}\Gamma(1+\beta_{1}+ \beta_{2})}S_{2}+\frac{1}{2} \frac{s^{*}T^{\beta_{1}}}{q^{*}\Gamma {(1+\beta_{1})}}\hat{w} \\ &{}+\frac{1}{2}\frac{\gamma_{2} T^{\beta_{1}+\beta_{2}}}{q^{*}(\gamma _{2}+1)\Gamma{(1+\beta_{1})}\Gamma{(1+\beta_{2})}}S_{2}+\frac{1}{2} \frac{\vert \eta_{2}\vert T^{\beta_{1}}}{q^{*}(\gamma_{2}+1)\Gamma{(1+\beta _{1})}} \hat{w} \\ =&\frac{3}{2} ( h_{5}+ h_{7} ) S_{2}+ \frac{3}{2} ( h_{6}+h _{8} ) \hat{w} \\ =&M_{3}S_{2}+M_{4}\hat{w}, \end{aligned}$$

and therefore \(\Vert \mathcal{A}_{2}(x,y)\Vert \le M_{3}S_{2}+M_{4}\hat{w}\). Consequently, \(\Vert \mathcal{A}(x,y)\Vert \le M_{1}S_{1}+M_{3}S_{2}+(M_{2}+M _{4})\hat{w}\), which means that the operator \(\mathcal{A}\) is uniformly bounded.

Next, we prove that \(\mathcal{A}\) is equicontinuous. For given \(t_{1},t_{2}\in[0,T]\), with \(t_{1}< t_{2}\), we get

$$\begin{aligned}& \bigl\vert \mathcal{A}_{1}(x,y) (t_{2})-\mathcal{A}_{1}(x,y) (t_{1})\bigr\vert \\& \quad \le\biggl\vert I^{\alpha_{1}} \biggl( \frac{1}{p^{*}}I^{\alpha_{2}}f \bigl(s,x(s),y(s)\bigr) \biggr) (t _{2})-I^{\alpha_{1}} \biggl( \frac{1}{p^{*}}I^{\alpha_{2}}f\bigl(s,x(s),y(s)\bigr) \biggr) (t_{1})\biggr\vert \\& \qquad {}+\biggl\vert I^{\alpha_{1}} \biggl( \frac{r^{*}}{p^{*}}x(s) \biggr) (t_{2})-I ^{\alpha_{1}} \biggl( \frac{r^{*}}{p^{*}}x(s) \biggr)(t_{1})\biggr\vert \\& \qquad {}+\biggl\vert \frac{\gamma_{1}}{\gamma_{1}+1}I^{\alpha_{2}}f\bigl(s,x(s),y(s)\bigr) (T)+\frac{\eta_{1}}{\gamma_{1}+1} x(T) \biggl\vert \biggl\vert I^{\alpha_{1}} \biggl( \frac{1}{p}\biggr) (t _{2})-I^{\alpha_{1}} \biggl( \frac{1}{p} \biggr)(t_{1})\biggr\vert \\& \quad \le\frac{S_{1}}{p^{*}\Gamma(\alpha_{1}+\alpha_{2})} \biggl[ \int_{0} ^{t_{1}}\bigl\vert (t_{2}-s)^{\alpha_{1}+\alpha _{2}-1}-(t_{1}-s)^{\alpha_{1}+\alpha_{2}-1}\bigr\vert \,ds+ \int_{t_{1}}^{t_{2}}(t_{2}-s)^{\alpha_{1}+\alpha_{2}-1}\,ds\biggr] \\& \qquad {}+\frac{r^{*}\hat{w}}{p^{*}\Gamma(\alpha_{1})} \biggl[ \int_{0}^{t_{1}} \bigl\vert (t_{2}-s)^{\alpha_{1}-1}-(t_{1}-s)^{\alpha_{1}-1} \bigr\vert \,ds+\int_{t_{1}}^{t_{2}}(t_{2}-s)^{\alpha_{1}-1}\,ds\biggr] \\& \qquad {}+ \biggl(\frac{\gamma_{1}T^{\alpha_{2}}S_{1}}{(\gamma_{1}+1)\Gamma(1+ \alpha_{2})}+\frac{\vert \eta_{1}\vert \hat{w}}{\gamma_{1}+1} \biggr) \frac{1}{\Gamma(\alpha_{1})}\biggl[\int_{0}^{t_{1}}\bigl\vert (t_{2}-s)^{\alpha_{1}-1}-(t _{1}-s)^{\alpha_{1}-1}\bigr\vert \,ds \\& \qquad {}+\int_{t_{1}}^{t_{2}}(t_{2}-s)^{\alpha_{1}-1}\,ds\biggr] \\& \quad \le\frac{S_{1}}{p^{*}\Gamma(1+\alpha_{1}+\alpha_{2})} \bigl[2(t_{2}-t _{1})^{\alpha_{1}+\alpha_{2}}+\bigl\vert t_{2}^{\alpha_{1}+\alpha_{2}}-t_{1}^{\alpha_{1}+\alpha _{2}}\bigr\vert \bigr] \\& \qquad {}+\frac{1}{\Gamma(1+\alpha_{1})} \biggl(\frac{r^{*}\hat {w}}{p^{*}}+\frac{\gamma_{1}T^{\alpha_{2}}S_{1}}{(\gamma_{1}+1)\Gamma(1+\alpha_{2})}+ \frac{\vert \eta_{1}\vert \hat{w}}{\gamma_{1}+1} \biggr) \bigl[2(t_{2}-t_{1})^{\alpha_{1}}+\bigl\vert t_{2}^{\alpha_{1}}-t_{1}^{\alpha_{1}}\bigr\vert \bigr]. \end{aligned}$$

Hence we have

$$\bigl\Vert \mathcal{A}_{1}(x,y) (t_{2})- \mathcal{A}_{1}(x,y) (t_{1})\bigr\Vert \to0,\quad \mbox{as } t_{1}\to t_{2}. $$

Analogously, we can obtain

$$\bigl\Vert \mathcal{A}_{2}(x,y) (t_{2})- \mathcal{A}_{2}(x,y) (t_{1})\bigr\Vert \to0, \quad \mbox{as } t_{1}\to t_{2}. $$

Therefore, the operator \({\mathcal{A}}\) is equicontinuous, and thus the operator \({\mathcal{A}}\) is completely continuous.

Finally, it will be verified that the set

$$\mathcal{J}=\{(x,y)\in X\times X \vert (x,y)=\kappa\mathcal{A}(x,y), \mbox{ for some } 0< \kappa< 1\} $$

is bounded. Let \((x,y)\in\mathcal{J}\), then \((x,y)=\kappa \mathcal{A}(x,y)\). For any \(t\in[0,T]\), we have

$$x(t)=\kappa\mathcal{A}_{1}(x,y) (t),\qquad y(t)=\kappa \mathcal{A}_{2}(x,y) (t). $$

Then

$$\begin{aligned} \bigl\vert x(t)\bigr\vert =&\bigl\vert \kappa\mathcal{A}_{1}(x,y) (t)\bigr\vert \\ \leq&I^{\alpha_{1}} \biggl( \bigl(C_{0}+C_{1}\Vert x \Vert +C_{2}\Vert y\Vert \bigr)\frac{1}{p ^{*}}I^{\alpha_{2}}(1) \biggr) (T)+ \biggl( \frac{r^{*}}{p^{*}} \biggr) \Vert x\Vert I^{\alpha_{1}}(1) (T) \\ &{}+ \biggl(\frac{\gamma_{1}}{\gamma_{1}+1}\bigl(C_{0}+C_{1}\Vert x \Vert +C_{2}\Vert y\Vert \bigr)I ^{\alpha_{2}}(1) (T)+ \frac{\vert \eta_{1}\vert }{\gamma_{1}+1}(T)\Vert x\Vert \biggr)I ^{\alpha_{1}} \biggl( \frac{1}{p^{*}} \biggr) (T) \\ &{}+\frac{1}{2} \biggl[I^{\alpha_{1}} \biggl( \bigl(C_{0}+C_{1} \Vert x\Vert +C_{2}\Vert y\Vert \bigr) \frac{1}{p}I^{\alpha_{2}}(1) (T) \biggr) + \biggl( \frac{r^{*}}{p^{*}}\Vert x\Vert \biggr) I ^{\alpha_{1}}(1) (T) \\ &{}+ \biggl(\frac{\gamma_{1}}{\gamma_{1}+1}\bigl(C_{0}+C_{1}\Vert x \Vert +C_{2}\Vert y\Vert \bigr)I ^{\alpha_{2}}(1) (T)+ \frac{\vert \eta_{1}\vert }{\gamma_{1}+1}\Vert x\Vert \biggr) \vert \rho_{1}\vert \biggr] \\ =&\bigl(C_{0}+C_{1}\Vert x\Vert +C_{2} \Vert y\Vert \bigr)M_{1}+M_{2}\Vert x\Vert \end{aligned}$$

and

$$\begin{aligned} \bigl\vert y(t)\bigr\vert =&\bigl\vert \kappa\mathcal{A}_{2}(x,y) (t)\bigr\vert \\ \leq&I^{\beta_{1}} \biggl( \bigl(D_{0}+D_{1}\Vert x \Vert +D_{2}\Vert y\Vert \bigr)\frac{1}{q}I ^{\beta_{2}}(1) (T) \biggr) + \biggl( \frac{s^{*}}{q^{*}} \biggr) \Vert y \Vert I^{ \beta_{1}}(1) (T) \\ &{}+ \biggl(\frac{\gamma_{2}}{\gamma_{2}+1}\bigl(D_{0}+D_{1}\Vert x \Vert +D_{2}\Vert y\Vert \bigr)I ^{\beta_{2}}(1) (T)+ \frac{\vert \eta_{2}\vert }{\gamma_{2}+1}(T)\Vert y\Vert \biggr)I^{ \beta_{1}} \biggl( \frac{1}{q^{*}} \biggr) (T) \\ &{}+\frac{1}{2} \biggl[I^{\beta_{1}} \biggl( \bigl(D_{0}+D_{1} \Vert x\Vert +D_{2}\Vert y\Vert \bigr)\frac{1}{q ^{*}}I^{\beta_{2}}(1) (T) \biggr) + \biggl( \frac{s^{*}}{q^{*}}\Vert y\Vert \biggr) I ^{\beta_{1}}(1) (T) \\ &{}+ \biggl(\frac{\gamma_{2}}{\gamma_{2}+1}\bigl(D_{0}+D_{1}\Vert x \Vert +D_{2}\Vert y\Vert \bigr)I ^{\beta_{2}}(1) (T)+ \frac{\vert \eta_{2}\vert }{\gamma_{2}+1}\Vert y\Vert \biggr) \vert \rho_{2}\vert \biggr] \\ =&\bigl(D_{0}+D_{1}\Vert x\Vert +D_{2} \Vert y\Vert \bigr)M_{3}+M_{4}\Vert y\Vert . \end{aligned}$$

Hence

$$\begin{aligned}& \Vert x\Vert \leq\bigl(C_{0}+C_{1}\Vert x\Vert +C_{2}\Vert y\Vert \bigr)M_{1}+M_{2}\Vert x \Vert , \\& \Vert y\Vert \leq\bigl(D_{0}+D_{1}\Vert x\Vert +D_{2}\Vert y\Vert \bigr)M_{3}+M_{4}\Vert y \Vert . \end{aligned}$$

Then we have

$$\Vert x\Vert +\Vert y\Vert \leq C_{0}M_{1}+D_{0}M_{3}+(C_{1}M_{1}+M_{2}+D_{1}M_{3}) \Vert x\Vert +(C_{2}M_{1}+D_{2}M_{3}+M_{4}) \Vert y\Vert . $$

Consequently,

$$\bigl\Vert (x,y)\bigr\Vert \leq\frac{C_{0}M_{1}+D_{0}M_{3}}{J^{*}}, $$

for any \(t\in[0,T]\), where \(J^{*}=\min\{1-J_{1}, 1-J_{2}\}\), which proves that \(\mathcal{J}\) is bounded. Hence, by Lemma 3.1, the operator \(\mathcal{A}\) has at least one fixed point. So, problem (1.2)-(1.3) has at least one solution on \([0,T]\). The proof is completed. □

Let \(J_{3}=1-(C_{1}M_{1}+D_{1}M_{3})\), \(J_{4}=1-(C_{2}M_{1}+D_{2}M _{3})\), \(J_{5}=1-(a_{1}C_{1}+a_{2}+a_{3}D_{1})\) and \(J_{6}=1-(a_{1}C _{2}+a_{4}+a_{3}D_{2})\). We have the following results.

Corollary 3.3

Suppose that conditions \((H_{3})\) and \((H_{4})\) of Theorem 3.2 hold. If \(J_{3}<1\) and \(J_{4}<1\), then system (1.4) with (1.3) has at least one solution on \([0,T]\).

Corollary 3.4

Assume that conditions \((H_{3})\) and \((H_{4})\) of Theorem 3.2 are fulfilled. If \(J_{5}<1\) and \(J_{6}<1\), then system (1.5) with (1.3) has at least one solution on \([0,T]\).

4 Examples

In this section, we present two examples to illustrate our results.

Example 4.1

Consider the following system of generalized Sturm-Liouville and Langevin fractional differential equations subject to anti-periodic boundary conditions:

$$ \textstyle\begin{cases} D^{3/5}([(t^{3/2}+8)D^{2/3}+(t^{2/7}-1)]x(t))= \frac{\vert x\vert \sin^{2}(2\pi t)}{(4-t)^{2}} ( \frac{\vert x\vert }{\vert x\vert +4}+1 ) +\frac{\vert y\vert +1}{(4-t)^{2}}, \\ \quad 0< t< 2, \\ D^{2/5}([(t^{5/3}+7)D^{3/4}+(t^{3/10}-1)]y(t))= \frac{\vert x\vert }{(7+t)^{2}}+\frac{\cos^{2}(\pi t)}{(5-t)} ( \frac{ \vert y\vert }{\vert y\vert +5}+1 ), \\ \quad 0< t< 2, \\ x(0)=-x(2),\qquad D^{2/3}x(0)=-D^{2/3}x(2), \\ y(0)=-y(2),\qquad D ^{3/4}y(0)=-D^{3/4}y(2). \end{cases} $$
(4.1)

Here \(\alpha_{1}=2/3\), \(\alpha_{2}=3/5\), \(\beta_{1}=3/4\), \(\beta_{2}=2/5\), \(T=2\), \(p(t)=t^{3/2}+8\), \(q(t)=t^{5/3}+7\), \(r(t)=t^{2/7}-1\), \(s(t)=t^{3/10}-1\), \(f(t,x,y)=((\vert x\vert \sin ^{2}(2\pi t))/(4-t)^{2})(\vert x\vert /(\vert x\vert +4)+1)+ ((\vert y\vert +1)/(4-t)^{2})\) and \(g(t,x,y)=(\vert x\vert /(7+t)^{2})+(\cos ^{2}(\pi t)/(5-t))((\vert y\vert /(\vert y\vert +5))+1)\). From the above information, we can find that \(p^{\ast}=8\), \(q^{\ast}=7\), \(r^{\ast}=0.21901\), \(s^{\ast}=0.23114\), \(\gamma_{1}=0.73879\), \(\gamma_{2}=0.6797\), \(\eta_{1}=1.16180\), \(\eta_{2}=1.15901\). Since

$$\bigl\vert f(t,x_{1},y_{1})-f(t,x_{2},y_{2}) \bigr\vert \leq\frac{1}{64} \vert x_{1}-x_{2} \vert +\frac{1}{16} \vert y_{1}-y_{2} \vert $$

and

$$\bigl\vert g(t,x_{1},y_{1})-g(t,x_{2},y_{2}) \bigr\vert \leq\frac{1}{49} \vert x_{1}-x_{2} \vert +\frac{1}{25} \vert y_{1}-y_{2} \vert , $$

the assumptions of Theorem 3.1 are satisfied with \(m_{1}=1/64\), \(m_{2}=1/16\), \(n_{1}=1/49\), \(n_{2}=1/25\), \(M_{1}= 0.63199\), \(M_{2}= 0.29248\), \(M_{3}= 0.68084\), \(M_{4}= 0.35986\). Thus

$$\begin{aligned}& L_{1} =(m_{1} +m_{2})M_{1}+M_{2}= 0.34185, \qquad L_{2} =(n_{1} +n_{2})M_{3}+M_{4}= 0.40098. \end{aligned}$$

Therefore, we have \(L_{1}+L_{2} = 0.74283 <1\). Hence, by Theorem 3.1, problem (4.1) has a unique solution on \([0,2]\).

Example 4.2

Consider the following system of generalized Sturm-Liouville and Langevin fractional differential equations subject to anti-periodic boundary conditions:

$$ \textstyle\begin{cases} D^{4/7}([(t^{3/5}+7)D^{3/5}+(t^{3/8}-1)]x(t))= 1+\frac{\sqrt{3}\vert x\vert \cos^{2}(2\pi t)}{3(27-t)}+\frac{ \sqrt{2}\pi \vert y\vert }{(7\pi-t)^{2}} ( \frac{\vert y\vert }{ \vert y\vert +3}+1 ), \\ \quad 0< t< \pi, \\ D^{3/4}([(t^{4/7}+8)D^{5/6}+(t^{2/7}-1)]y(t))= \frac{4}{3}+\frac{\sqrt{2}\pi \vert x\vert }{4(4\pi-t)^{2}} ( \frac{ \vert x\vert }{\vert x\vert +3}+1 ) +\frac{\vert y\vert \sin^{2}(2 \pi t)}{(10-t)^{2}}, \\ \quad 0< t< \pi, \\ x(0)=-x(\pi),\qquad D^{3/5}x(0)=-D^{3/5}x(\pi),\\ y(0)=-y(\pi),\qquad D^{5/6}y(0)=-D ^{5/6}y(\pi). \end{cases} $$
(4.2)

Here \(\alpha_{1}=3/5\), \(\alpha_{2}=4/7\), \(\beta_{1}=5/6\), \(\beta_{2}=3/4\), \(T=\pi\), \(p(t)=t^{3/5}+7\), \(q(t)=t^{4/7}+8\), \(r(t)= t^{3/8}-1\), \(s(t)=t^{2/7}-1\), \(f(t,x,y)=1+((\sqrt{3}\vert x\vert \cos^{2}(2\pi t))/(3(27-t)))+((\sqrt{2}\pi \vert y\vert )/(7 \pi-t)^{2})((\vert y\vert /(\vert y\vert +3)+1))\) and \(g(t,x,y)=(4/3)+(( \sqrt{2}\pi \vert x\vert )/(4(4\pi-t)^{2}))((\vert x\vert /(\vert x\vert +3)+1))+(\sin^{2}(2\pi t)/100)\). From all the information, we can find that \(p^{\ast}=7\), \(q^{\ast}=8\), \(r^{\ast}=0.53614\) and \(s^{\ast}=0.38689\). It is obvious that

$$\bigl\vert f(t,x_{1},x_{2})\bigr\vert \leq1+ \frac{\sqrt{3}}{81}\vert x_{1}\vert +\frac{\sqrt{2}}{49\pi} \vert x_{2}\vert $$

and

$$\bigl\vert g(t,x_{1},x_{2})\bigr\vert \leq \frac{4}{3}+\frac{\sqrt{2}}{64 \pi} \vert x_{1}\vert + \frac{1}{100}\vert x_{2}\vert . $$

Then the assumptions of Theorem 3.2 are satisfied with \(C_{0}=1\), \(C_{1}=\sqrt{3}/81\), \(C_{2}=\sqrt{2}/49\pi\), \(D_{0}=4/3\), \(D_{1}=\sqrt{2}/64\pi\), \(D_{2}=1/100\), and

$$\begin{aligned}& J_{1} =1-(C_{1}M_{1}+M_{2}+D_{1}M_{3})= 0.32897< 1, \\& J_{2} =1-(C_{2}M_{1}+M_{4}+D_{2}M_{3})= 0.39883< 1. \end{aligned}$$

Therefore, all the conditions of Theorem 3.2 hold true; and consequently, by the conclusion of Theorem 3.2, problem (4.2) has at least one solution on \([0,\pi]\).