1 Introduction

Nonlinear differential equations are crucial tools in the modeling of nonlinear real phenomena corresponding to a great variety of events, in relation with several fields of the physical sciences and technology. For instance, they appear in the study of the air motion or the fluids dynamics, electricity, electromagnetism, or the control of nonlinear processes, among others (see [2]). The resolution of nonlinear differential equations requires, in general, the development of different techniques in order to deduce the existence and other essential properties of the solutions. There are still many open problems related the solvability of nonlinear systems, apart form the fact that this is a field where advances are continuously taking place.

Perturbation techniques are useful in the nonlinear analysis for studying the dynamical systems represented by nonlinear differential and integral equations. Evidently, some differential equations representing a certain dynamical system have no analytical solution, so the perturbation of such problems can be helpful. The perturbed differential equations are categorized into various types. An important type of these such perturbations is called a hybrid differential equation (i.e. quadratic perturbation of a nonlinear differential equation). See [3] and the references therein.

Recently, the hybrid differential equations have been much more attractive [4–7], and then there have been many works on the theory of hybrid differential equations. Additionally, hybrid fixed point theory can be used to develop the existence theory for the hybrid equations. We refer to the articles [8–12]. Dhage and Jadhav [13] discussed the following first-order hybrid differential equation with linear perturbations of second type:

$$ \textstyle\begin{cases} \frac{d}{dt} [x(t)- f(t,x(t))] = g (t,x(t)) ,\quad \mbox{a.e. }t \in J, \\ x (t_{0}) = x_{0} \in\mathbb{R}, \end{cases} $$

where \(J=[t_{0},t_{0}+a)\), in \(\mathbb{R}\) for some fixed \(t_{0},a \in\mathbb {R}\) with \(a > 0\), and \(f, g \in\mathcal{C}(J\times\mathbb{R}, \mathbb{R})\). They proved the existence of the maximal and minimal solution for this equation. Furthermore, they established some basic results concerning the strict and nonstrict differential inequalities.

Indeed, the fractional differential equations have recently been intensively used in modeling several physical phenomena and have been studied by many researchers in recent years [14–22]; therefore they seem to deserve an independent study of their theory parallel to the theory of ordinary differential equations.

Lu et al. [23] developed this problem as regards the following FHDE involving the R-L differential operators of order \(0 < q < 1\), with linear perturbations of second type:

$$ \textstyle\begin{cases} D^{q} [x(t)-f(t,x(t))] = g (t,x(t)) ,\quad \mbox{a.e. }t \in J, \\ x (t_{0}) = x_{0} \in\mathbb{R}, \end{cases} $$

\(f, g \in\mathcal{C}(J\times\mathbb{R}, \mathbb{R})\). Beside that, using mixed Lipschitz and Carathéodory conditions allowed them to prove existence theorem for fractional hybrid differential equations.

On top of that, the study of coupled systems involving fractional differential equations is also important as such systems occur in various problems of applied nature, for instance, see [24–29]. Lately, Su [30] discussed a two-point boundary value problem for a coupled system of fractional differential equations. Gafiychuk et al. [31] analyzed the solutions of coupled nonlinear fractional reaction-diffusion equations.

In line with the above works, our purpose in this paper is to prove the existence of solution to the following system of fractional hybrid differential equations of order \(0 < p < 1\):

$$ \textstyle\begin{cases} D^{p} [x(t)- f(t,x(t))] = g (t,y(t),I^{\alpha}(y(t))) ,\quad \mbox{a.e. }t \in J, \\ D^{p} [y(t)- f(t,y(t))] = g (t,x(t),I^{\alpha}(x(t))) ,\quad \mbox{a.e. }t \in J, 0 < p < 1, \alpha>0, \\ x (0) = 0, \qquad y (0) = 0. \end{cases} $$
(1)

The proof is rooted in a coupled fixed point theorem which is a generalization of a fixed point theorem of Burton [1] in the Banach spaces.

2 Preliminaries

Let \(C(J\times \mathbb{R}\times\mathbb{R},\mathbb{R})\) denote the class of continuous functions \(f:J\times\mathbb{R}\times \mathbb{R}\rightarrow\mathbb{R}\) and let \(\mathcal{C}(J\times \mathbb{R}\times\mathbb{R},\mathbb{R})\) denote the class of functions \(g:J\times\mathbb{R}\times\mathbb{R}\rightarrow \mathbb{R}\) such that

  1. (i)

    the map \(t \rightarrow g(t,x,y)\) is measurable for each \(x,y \in\mathbb{R}\),

  2. (ii)

    the map \(x \rightarrow g(t,x,y)\) is continuous for each \(x \in \mathbb{R}\),

  3. (iii)

    the map \(y \rightarrow g(t,x,y)\) is continuous for each \(y \in\mathbb{R}\).

The class \(\mathcal{C}(J\times\mathbb{R}\times \mathbb{R},\mathbb{R})\) is called the Carathéodory class of functions on \(J\times\mathbb{R}\times\mathbb{R,}\) which are Lebesgue integrable when bounded by a Lebesgue integrable function on J.

We need some precise definitions of the basic concepts. The following is a discussion of some of the concepts we will need.

Definition 1

[32]

The form of the Riemann-Liouville fractional integral operator of order \(\alpha > 0\), of function \(f\in L^{1}(\mathbb{R}^{+})\) is defined as

$$ I^{\alpha}f(t) = \frac{1}{\Gamma(\alpha)} \int_{0}^{t} (t - s)^{\alpha - 1} f (s) \,ds. $$

Definition 2

[32]

Let α be a positive real number, such that \(m-1<\alpha\le m\), \(m\in\mathbb{N}\) and \(f^{(m)}(x)\) exists, a function of class C. Then the Caputo fractional derivative of f is defined as

$$ {}^{\mathrm{C}}D^{\alpha}f(x)=\frac{1}{\Gamma(m - \alpha)} \int_{0}^{t} (t - s)^{m - \alpha- 1} f^{(m)} (s) \,ds. $$

Definition 3

[32]

The Riemann-Liouville fractional derivative of order \(\alpha> 0\) of a continuous function \(f:(0,\infty)\rightarrow\mathbb{R}\) is defined as

$$ D^{\alpha}f(t) = \frac{1}{\Gamma(n - \alpha)} \biggl(\frac{d}{dt} \biggr)^{n} \int_{0}^{t} (t - s)^{n - \alpha- 1} f (s) \, ds, $$

where \(n=[\alpha]+1\).

Lemma 1

[32, 33]

Let \(0<\alpha<1\) and \(f\in L^{1}(0,1)\). Then:

  1. (K1)

    The equality \(D^{\alpha}I^{\alpha}f(t) = f(t)\) holds.

  2. (K2)

    The equality

    $$ I^{\alpha}D^{\alpha}f (t) = f (t) - \frac{[D^{\alpha-1}f(t)]_{t=0}}{\Gamma(\alpha)} t^{\alpha- 1} $$

    holds almost everywhere on J.

The following is a fixed point theorem in Banach spaces due to Burton [1].

Lemma 2

[1]

Let S be a nonempty, closed, convex, and bounded subset of a Banach space X and let \(A:X\rightarrow X\) and \(B:S\rightarrow X\) be two operators such that

  1. (i)

    A is a contraction with constant \(\alpha<1\),

  2. (ii)

    B is completely continuous,

  3. (iii)

    \(x=Ax+By \Rightarrow x \in S\) for all \(y \in S\).

Then the operator equation \(Ax+Bx = x\) has a solution in S.

Now, we recall the definition of a coupled fixed point for a bivariate mapping.

Definition 4

[34]

An element \((x,y)\in X\times X\) is called a coupled fixed point of a mapping \(T:X\times X \rightarrow X\) if \(T(x,y)=x\) and \(T(y,x)=y\).

Let us denote by Φ the family of all functions \(\varphi:\mathbb{R}^{+}\rightarrow\mathbb{R}^{+}\) fulfilling \(\varphi(r)< r\) for \(r>0\) and \(\varphi(0)=0\).

By a solution of the FHDEs system (1) we mean a function \((x,y) \in AC(J, \mathbb{R}\times\mathbb{R})\) such that:

  1. (i)

    the function \(t\rightarrow x- f(t,x)\) is absolutely continuous for each \(x \in\mathbb{R}\), and

  2. (ii)

    \((x,y)\) satisfies the system of equations in (1),

where \(AC(J, \mathbb{R}\times\mathbb{R})\) is the space of absolutely continuous real-valued functions defined on J.

3 Main result

Throughout this section, let \(X= C(J, \mathbb{R})\) equipped with the supremum norm. Clearly it is a Banach space with respect to pointwise operations and the supremum norm.

Define scalar multiplication and a sum on \(X\times X\) as follows:

$$(x_{1}, y_{1}) + (x_{2}, y_{2}) = (x_{1} + x_{2}, y_{1} + y_{2}) $$

and

$$a(x, y) = (ax, ay), $$

for \(a \in\mathbb{R}\). Then \(X\times X\) is a vector space on \(\mathbb{R}\).

In the following lemma we introduce a certain Banach space which is used in our results.

Lemma 3

Let \(\widetilde{X}:= X\times X\). Define

$$\bigl\Vert (x,y)\bigr\Vert =\|x\|+\|y\|. $$

Then X̃ is a Banach space with respect to the above norm.

Proof

Clearly X̃ is a Banach space and \(\|\cdot\|\) is a norm on X̃. □

Now, we prove a coupled fixed point theorem which is a generalization of Lemma 2 of Dhage.

Theorem 1

Let S be a nonempty, closed, convex, and bounded subset of the Banach space X and \(\widetilde{S}=S \times S\). suppose that \(A:X\rightarrow X\) and \(B:S\rightarrow X\) are two operators such that

  1. (C1)

    there exists \(\varphi_{A} \in\Phi\) such that for all \(x,y \in X\), we have

    $$\|Ax-Ay\|\leq\sigma\varphi_{A}\bigl(\Vert x-y\Vert \bigr), $$

    for some constant \(\sigma>0\),

  2. (C2)

    B is completely continuous,

  3. (C3)

    \(x=Ax+By \Rightarrow x \in S\) for all \(y \in S\).

Then the operator \(T(x,y)=Ax+By\) has at least a coupled fixed point in S̃ whenever \(\sigma<1\).

Proof

It is easy to check that S̃ is a nonempty, closed, convex, and bounded subset of the Banach space X̃. Define \(\widetilde{A}: \widetilde{X} \rightarrow\widetilde{X}\), and \(\widetilde{B}: \widetilde{S}\rightarrow\widetilde{X}\) by

$$\begin{aligned}& \widetilde{A}(x,y)=(Ax,Ay), \\& \widetilde{B}(x,y)=(By,Bx). \end{aligned}$$

It is sufficient to prove \(\widetilde{A}(x,y)+\widetilde{B}(x,y)=(x,y)\) has at least one solution, because

$$\begin{aligned} \bigl(T(x,y), T(y,x)\bigr) =& (Ax+By, Ay+Bx) \\ =& (Ax, Ay)+(By, Bx) \\ =& \widetilde{A}(x,y)+\widetilde{B}(x,y)=(x,y), \end{aligned}$$

which implies that \(T(x,y)\) has at least one coupled fixed point. We claim that the operators à and B̃ satisfy all the conditions of Lemma 2 on the Banach space X̃. First, we show that à is Lipschitzian. By condition (C1), for every \(x=(x_{1},x_{2}),y=(y_{1},y_{2}) \in\widetilde{X}\) we have

$$\begin{aligned} \Vert \widetilde{A}x-\widetilde{A}y\Vert =& \bigl\Vert (Ax_{1},Ax_{2})-(Ay_{1},Ay_{2})\bigr\Vert \\ =&\bigl\Vert (Ax_{1}-Ay_{1}, Ax_{2}-Ay_{2}) \bigr\Vert \\ =& \Vert Ax_{1}-Ay_{1}\Vert +\Vert Ax_{2}-Ay_{2}\Vert \\ \leq& \sigma\bigl( \varphi\bigl(\Vert x_{1}-y_{1}\Vert \bigr)+ \varphi\bigl(\Vert x_{2}-y_{2}\Vert \bigr)\bigr) \\ < & \sigma\bigl(\Vert x_{1}-y_{1}\Vert + \Vert x_{2}-y_{2}\Vert \bigr) \\ =& \sigma\bigl\Vert (x_{1}-y_{1}, x_{2}-y_{2}) \bigr\Vert \\ =& \sigma \Vert x-y\Vert , \end{aligned}$$

which implies that à is a contraction with constant σ. Next, we show that B̃ is a compact and continuous operator on S̃.

Let \((x_{n})=(x_{1n},x_{2n})\) be a sequence in S̃ converging to a point \(x=(x_{1},x_{2})\in\widetilde{S}\), since B is continuous we have

$$\begin{aligned} \lim_{n \rightarrow\infty}\widetilde{B}x_{n} =&\Bigl(\lim _{n \rightarrow \infty} Bx_{2n},\lim_{n \rightarrow\infty} Bx_{1n}\Bigr) \\ =&(Bx_{2},Bx_{1})=\widetilde{B}(x_{1},x_{2})= \widetilde{B}x, \end{aligned}$$

so B̃ is continuous.

Let \(x=(x_{1},x_{2})\in \widetilde{S}\), we have

$$\begin{aligned} \bigl\Vert \widetilde{B}(x_{1},x_{2})\bigr\Vert =& \bigl\Vert (Bx_{1},Bx_{2})\bigr\Vert \\ =& \bigl(\Vert Bx_{1}\Vert +\Vert Bx_{2}\Vert \bigr) \\ \leq& 2 \Vert BS\Vert , \end{aligned}$$

for all \(x \in\widetilde{S}\), where \(\|BS\|=\sup\{\|Bx\|: x \in S\}\). This shows that B̃ is uniformly bounded on S̃.

Let \(\varepsilon>0\), since \(B(S)\) is an equi-continuous set in X, hence there exists \(\delta>0\) such that for \(t_{1},t_{2}\in J\), \(|t_{1}-t_{2}|<\delta\) implies that \(|Bx(t_{1})-Bx(t_{2})| \leq\varepsilon\) for all \(x \in S\). Then for any \(x=(x_{1},x_{2}) \in\widetilde{S}\), we have

$$\begin{aligned} \bigl\vert \widetilde{B}x(t_{1})-\widetilde{B}x(t_{2}) \bigr\vert =& \bigl\vert \bigl(Bx_{2}(t_{1}),Bx_{1}(t_{1}) \bigr)-\bigl(Bx_{2}(t_{2}),Bx_{1}(t_{2}) \bigr)\bigr\vert \\ =& \bigl\vert \bigl(Bx_{2}(t_{1})-Bx_{2}(t_{2}),Bx_{1}(t_{1})-Bx_{1}(t_{2}) \bigr)\bigr\vert \\ =& \sqrt{\bigl(Bx_{1}(t_{1})-Bx_{1}(t_{2}) \bigr)^{2}+ \bigl(Bx_{2}(t_{1})-Bx_{2}(t_{2}) \bigr)^{2}} \\ \leq& \sqrt{2} \varepsilon, \end{aligned}$$

so \(\widetilde{B}(\widetilde{S})\) is an equi-continuous set in X̃. Thus, \(\widetilde{B}(\widetilde{S})\) is compact by the Arzelà-Ascoli theorem. As a result, B̃ is a continuous and compact operator on S̃. So, B̃ is completely continuous on S̃.

Next, we show that hypothesis (iii) of Lemma 2 is satisfied. Let \(x=(x_{1},x_{2})\in\widetilde{X}\), \(y=(y_{1},y_{2})\in \widetilde{S}\) such that \(x=\widetilde{A}x + \widetilde{B}y\). Then by assumption (C3), we have

$$\begin{aligned} (x_{1},x_{2}) =& \widetilde{A}(x_{1},x_{2})+ \widetilde{B}(y_{1},y_{2}) \\ =& (Ax_{1},Ax_{2})+(By_{2},By_{1}) \\ =& (Ax_{1}+ By_{2},Ax_{2}+By_{1}), \end{aligned}$$

which implies that

$$\begin{aligned}& x_{1} = Ax_{1}+ By_{2}, \\& x_{2} = Ax_{2}+By_{1}. \end{aligned}$$

So, by assumption (C3), we have \(x_{1},x_{2} \in S\). Thus, \(x\in \widetilde{S}\). Then all conditions of Lemma 2 are satisfied and hence the operator equation \(\widetilde{A}x +\widetilde{B}x=x\) has at least one solution on S̃. Thus, \(T(x,y)\) has at least one coupled fixed point and the proof is completed. □

Now by applying Theorem 1, we study the existence of solution for the FHDEs system (1) under the following general assumptions.

  1. (H0)

    The function \(x \rightarrow x - f (t, x)\) is increasing in \(\mathbb {R}\) for all \(t \in J\).

  2. (H1)

    There exists a constant \(M\geq L>0\) such that

    $$\bigl\vert f\bigl(t,x(t)\bigr)-f\bigl(t,y(t)\bigr)\bigr\vert \leq \frac{L(\vert x(t)-y(t)\vert )}{2(M+\vert x(t)-y(t)\vert )}, $$

    for all \(t \in J\) and \(x, y \in\mathbb{R}\).

  3. (H2)

    Fix

    $$F_{0}=\max_{t\in J} \bigl\vert f(t,0)\bigr\vert . $$
  4. (H3)

    There exists a continuous function \(h \in C(J,\mathbb{R})\) such that

    $$g\bigl(t, x(t), y(t)\bigr)\leq h(t),\quad x, y \in\mathbb{R}, t\in J. $$

As a consequence of Lemma 1 we have the following lemma which is useful in the existence results.

Lemma 4

[23]

Assume that hypothesis (H0) holds, \(y \in C(J,\mathbb{R})\), \(0< p<1\), \(\alpha>0\), and \(f \in C(J \times\mathbb{R},\mathbb{R})\) with \(f(0,0)=0\). Then the unique solution of the initial value problem

$$ \textstyle\begin{cases} D^{p} [x(t)-f(t,x(t))] = y(t) ,\quad t \in J, \\ x (0) = 0 , \end{cases} $$
(2)

is

$$x(t)=f\bigl(t,x(t)\bigr)+\frac{1}{\Gamma(p)} \int_{0}^{t} \frac{y(s)}{(t-s)^{1-p}}\,ds,\quad t \in J. $$

Now we are going to prove the following existence theorem for the FHDEs of system (1).

Theorem 2

Assume that hypotheses (H1)-(H3) hold. Then the FHDEs of system (1) has a solution defined on J.

Proof

Set \(X=C(J,\mathbb{R})\) and a subset S of X defined by

$$S=\bigl\{ x \in X | \Vert x\Vert \leq N\bigr\} , $$

where \(N\geq L+F_{0}+\frac{T^{p}}{\Gamma(p+1)}\|h\|_{L^{1}}\).

Clearly S is a closed, convex, and bounded subset of the Banach space X. Now, we consider the system (1). Obviously, \(x(t)\) is a solution of the FHDEs system (1) if and only if \(x(t)\) satisfies the following system of integral equations:

$$ \textstyle\begin{cases} x(t)=f(t,x(t))+\frac{1}{\Gamma(p)}\int_{0}^{t} \frac{g(s,y(s),I^{\alpha}(y(s)))}{(t-s)^{1-p}} \,ds , \\ y(t)=f(t,y(t))+\frac{1}{\Gamma(p)}\int_{0}^{t} \frac{g(s,x(s),I^{\alpha}(x(s)))}{(t-s)^{1-p}} \,ds, \quad t \in J. \end{cases} $$
(3)

Define two operators \(A:X\rightarrow X\) and \(B:S\rightarrow X\) by

$$ \textstyle\begin{cases} Ax(t)=f(t,x(t)), \\ Bx(t)=\frac{1}{\Gamma(p)}\int_{0}^{t} (t-s)^{p-1} g(s,x(s),I^{\alpha}(x(s))) \,ds, \quad t \in J, \end{cases} $$

so, the system (3) is transformed into the system of operator equations as

$$ \textstyle\begin{cases} x(t)=Ax(t)+ By(t), \\ y(t)=Ay(t)+ Bx(t),\quad t\in J. \end{cases} $$

We shall show that the operators A and B satisfy all the conditions of Theorem 1.

Let \(x,y \in X\), by hypothesis (H1) we have

$$\bigl\vert Ax(t)-Ay(t)\bigr\vert = \bigl\vert f\bigl(t,x(t)\bigr)-f \bigl(t,y(t)\bigr)\bigr\vert \leq\frac{L(|x(t)-y(t)|)}{2(M+|x(t)-y(t)|)} \leq\frac{L(\|x-y\|)}{2(M+\|x-y\|)}, $$

for all \(t \in J\). Taking the supremum over t, we obtain

$$\|Ax-Ay\|\leq\frac{L(\|x-y\|)}{2(M+\|x-y\|)}. $$

This shows that A is a nonlinear contraction on X with a control function \(\frac{1}{2}\varphi\) where φ is defined by \(\varphi(r)=\frac {Lr}{M+r}\).

Next we show that B is compact and continuous operator on S.

Let \(\{x_{n}\}\) be a sequence in S converging to a point \(x \in S\). Then

$$\begin{aligned} \lim_{n\rightarrow\infty} Bx_{n}(t) =& \frac{1}{\Gamma(p)}\lim _{n\rightarrow\infty} \biggl( \int_{0}^{t} (t-s)^{p-1} g \bigl(s,x(s),I^{\alpha}\bigl(x(s)\bigr)\bigr) \,ds \biggr) \\ =& \frac{1}{\Gamma(p)} \int_{0}^{t} (t-s)^{p-1} \lim _{n\rightarrow\infty}g\bigl(s,x_{n}(s),I^{\alpha}\bigl(x_{n}(s)\bigr)\bigr)\,ds \\ =& \frac{1}{\Gamma(p)} \int_{0}^{t} (t-s)^{p-1} g \bigl(s,x(s),I^{\alpha}\bigl(x(s)\bigr)\bigr)\,ds \\ =&Bx(t), \end{aligned}$$

for all \(t \in J\), where the second equality holds by Lebesgue dominated convergence theorem. So B is a continuous function on S.

Let \(x \in S\), by assumption (H2), for \(t \in J\) we have

$$\begin{aligned} \bigl\vert Bx(t)\bigr\vert =& \frac{1}{\Gamma(p)}\biggl\vert \int_{0}^{t} (t-s)^{p-1} g \bigl(s,x(s),I^{\alpha}\bigl(x(s)\bigr)\bigr) \,ds \biggr\vert \\ \leq& \frac{1}{\Gamma(p)} \int_{0}^{t} (t-s)^{p-1} \bigl\vert g \bigl(s,x(s),I^{\alpha}\bigl(x(s)\bigr)\bigr)\bigr\vert \,ds \\ \leq& \frac{1}{\Gamma(p)} \int_{0}^{t} (t-s)^{p-1} h(s) \,ds \\ \leq& \frac{T^{p}}{\Gamma(p+1)} \|h\|_{L^{1}}. \end{aligned}$$

Taking the supremum over t, we obtain

$$ \|Bx\| \leq\frac{T^{p}}{\Gamma(p+1)}\|h\|_{L^{1}}, $$

for all \(x \in S\), so B is uniformly bounded on S. Now let \(t_{1},t_{2} \in J\), for any \(x\in S\) one has

$$\begin{aligned} \bigl\vert Bx(t_{1})-Bx(t_{2})\bigr\vert =& \frac{1}{\Gamma(p)}\biggl\vert \int_{0}^{t_{1}} (t_{1}-s)^{p-1} g \bigl(s,x(s),D^{\alpha}\bigl(x(s)\bigr)\bigr) \\ &{} - \int_{0}^{t_{2}} (t_{2}-s)^{p-1} g \bigl(s,x(s),D^{\alpha}\bigl(x(s)\bigr)\bigr) \,ds \biggr\vert \\ \leq& \frac{1}{\Gamma(p)} \biggl\vert \int_{0}^{t_{1}} (t_{1}-s)^{p-1} g \bigl(s,x(s),D^{\alpha}\bigl(x(s)\bigr)\bigr)\,ds \\ &{} - \int_{0}^{t_{1}} (t_{2}-s)^{p-1}g \bigl(s,x(s),D^{\alpha}\bigl(x(s)\bigr)\bigr) \,ds \biggr\vert \\ &{}+ \frac{1}{\Gamma(p)} \biggl\vert \int_{0}^{t_{1}} (t_{2}-s)^{p-1} g \bigl(s,x(s),D^{\alpha}\bigl(x(s)\bigr)\bigr)\,ds \\ &{} - \int_{0}^{t_{2}} (t_{2}-s)^{p-1}g \bigl(s,x(s),D^{\alpha}\bigl(x(s)\bigr)\bigr) \,ds \biggr\vert \\ \leq& \frac{\|h\|_{L^{1}}}{\Gamma(p)}\biggl( \biggl\vert \int_{0}^{t_{1}} (t_{1}-s)^{p-1}-(t_{2}-s)^{p-1} \,ds \biggr\vert + \biggl\vert \int_{t_{1}}^{t_{2}} (t_{2}-s)^{p-1} \,ds \biggr\vert \biggr) \\ \leq& \frac{\|h\|_{L^{1}}}{\Gamma(p+1)}\bigl( \bigl\vert t_{1}^{p}-t_{2}^{p} \bigr\vert +\bigl\vert (t_{2}-t_{1})^{p} \bigr\vert \bigr). \end{aligned}$$

Since \(t^{p}\) is uniformly continuous on J for \(0< p<1\), for any \(\varepsilon>0\) there exists \(\delta_{1}>0\) such that if \(|t_{1}-t_{2}|<\delta_{1}\) we have

$$\bigl\vert t_{1}^{p}-t_{2}^{p}\bigr\vert < \frac{\Gamma(p+1)}{2\|h\|_{L^{1}}}\varepsilon. $$

Let \(\delta=\min\{\delta_{1}, (\frac{\Gamma(p+1)}{2\|h\|_{L^{1}}}\varepsilon )^{\frac{1}{p}}\}\), if \(| t_{2}-t_{1} |<\delta\), we have

$$ \bigl\vert Bx(t_{1})-Bx(t_{2})\bigr\vert < \frac{\|h\|_{L^{1}}}{\Gamma(p+1)} \biggl( \frac{\Gamma(p+1)}{2\|h\|_{L^{1}}} \varepsilon+ \frac{\Gamma(p+1)}{2\|h\|_{L^{1}}} \varepsilon \biggr)=\varepsilon. $$

This implies that \(B(S)\) is equi-continuous. Thus, B is completely continuous on S. To prove hypothesis (C3) of Theorem 1, let \(x \in X\) and \(y \in S\) such that \(x=Ax+By\), by assumptions (H1) and (H2), we have

$$\begin{aligned} \bigl\vert x(t)\bigr\vert \leq& \bigl\vert Ax(t)\bigr\vert +\bigl\vert By(t)\bigr\vert \\ \leq& \bigl(\bigl\vert f\bigl(t,x(t)\bigr)-f(t,0)\bigr\vert +\bigl\vert f(t,0)\bigr\vert \bigr)+ \frac{1}{\Gamma(p)} \int_{0}^{t} (t-s)^{p-1} \bigl\vert g \bigl(s,y(s),I^{\alpha}\bigl(y(s)\bigr)\bigr)\bigr\vert \,ds \\ \leq& L+F_{0}+ \frac{1}{\Gamma(p)} \int_{0}^{t} (t-s)^{p-1} h(s) \,ds \\ \leq& L+F_{0}+\frac{T^{p}}{\Gamma(p+1)}\|h\|_{L^{1}}. \end{aligned}$$

Then by taking the supremum over t on J and by (H2) we conclude that

$$\|x\|\leq L+F_{0}+\frac{T^{p}}{\Gamma(p+1)}\|h\|_{L^{1}}\leq N, $$

which implies that \(x \in S\). So, the assumption (C3) of Theorem 1 has been proved. Therefore, all the conditions of Theorem 1 are satisfied, hence the operator \(T(x,y)=Ax+By\) has a coupled fixed point on S̃. As a result, the FHDE system (1) has a solution defined on J. □

4 Illustrative example

Example 1

We discuss the following system of fractional hybrid differential equations:

$$ \textstyle\begin{cases} D^{\frac{1}{2}} [x(t)-\frac{\sin(t) |x(t)|}{2 (2+|x(t)|)}] = \frac{t y(t)}{1+|y(t)|},\quad t \in J, \\ D^{\frac{1}{2}} [y(t)-\frac{\sin t |y(t)|}{2 (2+|y(t)|)}] = \frac{t x(t)}{1+|x(t)|} ,\quad t \in J=[0,\pi], \\ x (0) = 0, \qquad y (0) = 0. \end{cases} $$
(4)

Observe that this system of equations is a special case of the FHDEs of system (1) if we put

$$\begin{aligned}& f\bigl(t,x(t)\bigr)=\frac{\sin(t) |x(t)|}{2 (2+|x(t)|)}, \\& g \bigl(t,y(t),I^{\alpha}\bigl(y(t)\bigr)\bigr)=\frac{t y(t)}{1+|y(t)|}, \end{aligned}$$

for arbitrary \(x,y \in X\) and \(t \in J\) and we obtain

$$\begin{aligned} \bigl\vert f\bigl(t,x(t)\bigr)-f\bigl(t,y(t)\bigr)\bigr\vert \leq& \frac{1}{2}\biggl\{ \frac{|x(t)|}{2+|x(t)|}-\frac{|y(t)|}{2+|y(t)|}\biggr\} \\ \leq& \frac{1}{2}\biggl\{ \frac {|x(t)-y(t)|+|y(t)|}{2+|x(t)-y(t)|+|y(t)|}-\frac {|y(t)|}{2+|x(t)-y(t)|+|y(t)|}\biggr\} \\ \leq& \frac{1}{2}\biggl\{ \frac{|x(t)-y(t)|}{2+|x(t)-y(t)|+|y(t)|}\biggr\} \\ \leq& \frac{|x(t)-y(t)|}{2(2+|x(t)-y(t)|)} \end{aligned}$$

and \(g (t,y(t),I^{\alpha}(y(t)))\leq t\). It is easy to see that \(F_{0}=0\), \(L=1\), \(M=2\), \(T=\pi\), \(h(t)=t\). We conclude that \(L+F_{0}+\frac{T^{p}}{\Gamma(p+1)}\|h\|_{L^{1}}= 1+ {\pi}^{2} < 11\). Thus \(N \geq11\). It follows that conditions (H1)-(H3) are satisfied. Thus, by Theorem 2 we conclude that the problem (4) has a solution.