1 Introduction

Fractional differential equations recently attracted much attention in view of their extensive applications in engineering, physics, chemistry, biology, and other fields [13]. In particular, boundary value problems of fractional differential equations and inclusions subject to a variety of boundary conditions have been studied by many authors, for instance, see [411] and the references cited therein. Coupled systems of fractional-order differential equations also constitute an interesting area of investigation as such systems appear in the mathematical modeling of many phenomena like synchronization of chaotic systems [1214], anomalous diffusion [15], ecological effects [16], disease models [1719], etc. For some recent works on coupled systems of fractional-order differential equations, we refer the reader to the articles [2029].

Differential inclusions are found to be of great utility in studying dynamical systems and stochastic processes. For some recent results on boundary value problems for fractional differential inclusions, see [3033].

Recently, in [34], the authors studied a boundary value problem of coupled Caputo type fractional differential inclusions of the form:

$$ \textstyle\begin{cases} {}^{c}D^{\alpha } u(t) \in F(t, u(t), v(t)),\quad t\in [0,T], 1< \alpha \leq 2, \\ {}^{c}D^{\beta } v(t) \in G(t,u(t), v(t)),\quad t\in [0,T], 1< \beta \leq 2, \\ u(0)=\nu _{1} v(T),\qquad u'(0)=\nu _{2} v'(T), \\ v(0)=\mu _{1} u(T),\qquad v'(0)=\mu _{2} u'(T), \end{cases} $$

where \({}^{c}D^{\alpha }\), \({}^{c}D^{\beta }\) denote the Caputo fractional derivatives of orders α and β respectively, \(F,G : [0,T] \times \mathbb{R}\times \mathbb{R} \to {\mathcal{P}}(\mathbb{R})\) are given multi-valued maps, \(\mathcal{P}(\mathbb{R})\) is the family of all nonempty subsets of \(\mathbb{R}\), and \(\nu _{i},\mu _{i}, i=1,2\), are real constants with \(\nu _{i}\mu _{i}\ne 1\), \(i=1,2\). By applying standard fixed point theorems for multi-valued maps, some new existence results for the given problem are derived when the multi-valued maps involved in the given problem have convex as well as non-convex values.

In this work, motivated by [34], we consider the following systems of Caputo and Riemann–Liouville type mixed order coupled fractional differential equations and inclusions:

$$ \textstyle\begin{cases} {}^{c}D^{\alpha } x(t)=f(t, x(t), y(t)),\quad t\in [0,T],0< \alpha \leq 1 \\ {}^{\mathrm{RL}}D^{\beta } y(t)=g(t,x(t), y(t)),\quad t\in [0,T], 1< \beta \leq 2, \end{cases} $$
(1.1)

and

$$ \textstyle\begin{cases} {}^{c}D^{\alpha } x(t) \in F(t, x(t), y(t)),\quad t\in [0,T],0< \alpha \leq 1 \\ {}^{\mathrm{RL}}D^{\beta } y(t) \in G(t,x(t), y(t)),\quad t\in [0,T], 1< \beta \leq 2, \end{cases} $$
(1.2)

subject to the following coupled fractional boundary conditions:

$$ \textstyle\begin{cases} x(0)=\lambda {}^{c}D^{p}y(\eta ),\quad 0< p< 1, \\ y(0)=0,\qquad y(T)=\gamma I^{q} x(\xi ), \end{cases} $$
(1.3)

where \({}^{c}D^{\alpha }\), \({}^{c}D^{p}\) are the Caputo fractional derivatives of order α and p respectively, \({}^{\mathrm{RL}}D ^{\beta }\) is the Riemann–Liouville fractional derivative of order β, \(I^{q}\) is the Riemann–Liouville fractional integral of order q, \(f, g: [0,T]\times \mathbb{R}\times \mathbb{R}\to \mathbb{R}\), \(F,G : [0,T] \times \mathbb{R}\times \mathbb{R} \to {\mathcal{P}}(\mathbb{R})\) are given continuous functions, \(\mathcal{P}(\mathbb{R})\) is the family of all nonempty subsets of \(\mathbb{R}\), \(\eta , \xi \in (0,T)\), and \(\lambda , \gamma \in {\mathbb{R}}\).

Here we emphasize that the proposed single and multi-valued problems include:

  • fractional derivatives of different orders\(\alpha \in (0, 1]\)and\(\beta \in (1, 2]\);

  • the first and second equations in the given systems are respectively of Caputo and Riemann–Liouville types;

  • the boundary conditions are of nonlocal type and contain both fractional derivatives and integrals.

The objective of the present work is to establish existence criteria for solutions of problems (1.1)–(1.3) and (1.2)–(1.3). For single-valued system (1.1)–(1.3), we rely on the Leray–Schauder alternative and the Banach contraction mapping principle to obtain the existence and uniqueness results, which are presented in Sect. 3. Section 4 contains the existence results for convex and non-convex valued multi-valued maps F and G involved in multi-valued system (1.2)–(1.3), which are respectively derived with the aid of the nonlinear alternative for Kakutani maps and Covitz and Nadler’s fixed point theorem. The background material related to our work is outlined in Sect. 2. Here we remark that the tools of the fixed point theory employed in our analysis are standard, however their exposition to the problems at hand is new.

2 Preliminaries

Let us begin this section with some basic definitions of multi-valued maps [35, 36].

Let \(({\mathcal{X}}, \|\cdot \|)\) be a normed space and that \({\mathcal{P}}_{cl}({\mathcal{X}})=\{{\mathcal{Y}} \in {\mathcal{P}}( {\mathcal{X}}) : {\mathcal{Y}} \mbox{ is closed}\}\), \({\mathcal{P}} _{cp, c}({\mathcal{X}})=\{{\mathcal{Y}} \in {\mathcal{P}}({\mathcal{X}}) : {\mathcal{Y}} \mbox{ is compact and convex}\}\).

A multi-valued map \({\mathcal{G}} : {\mathcal{X}} \to {\mathcal{P}}( {\mathcal{X}})\) is

  1. (a)

    convex (closed) valued if \({\mathcal{G}}(x)\) is convex (closed) for all \(x \in {\mathcal{X}}\);

  2. (b)

    upper semi-continuous (u.s.c.) on \({\mathcal{X}}\) if, for each \(x_{0} \in {\mathcal{X}}\), the set \({\mathcal{G}}(x_{0})\) is a nonempty closed subset of \({\mathcal{X}}\) and if, for each open set N of \({\mathcal{X}}\) containing \({\mathcal{G}}(x_{0})\), there exists an open neighborhood \(\mathcal{N}_{0}\) of \(x_{0}\) such that \(G(\mathcal{N} _{0}) \subseteq N\);

  3. (c)

    lower semi-continuous (l.s.c.) if the set \(\{y \in X : {\mathcal{G}}(y) \cap B \ne \emptyset \}\) is open for any open set B in E;

  4. (d)

    completely continuous if \({\mathcal{G}}(\mathbb{B})\) is relatively compact for every \(\mathbb{B} \in {\mathcal{P}}_{b}({\mathcal{X}})=\{ {\mathcal{Y}} \in {\mathcal{P}}({\mathcal{X}}) : {\mathcal{Y}} \mbox{ is bounded}\}\).

A multi-valued map \({\mathcal{G}} : [a,b] \to {\mathcal{P}}_{cl}( \mathbb{R})\) is said to be measurable if, for every \(y \in \mathbb{R}\), the function \(t \longmapsto d(y,{\mathcal{G}}(t)) = \inf \{|y-z|: z \in {\mathcal{G}}(t)\}\) is measurable.

A multi-valued map \({\mathcal{G}} : [a,b] \times \mathbb{R}^{2} \to {\mathcal{P}}(\mathbb{R})\) is said to be Carathéodory if (i) \(t \longmapsto G(t,x, y)\) is measurable for each \(x, y\in \mathbb{R}\) and (ii) \((x, y) \longmapsto {\mathcal{G}}(t,x, y)\) is upper semicontinuous for almost all \(t\in [a, b]\).

Further a Carathéodory function \({\mathcal{G}}\) is called \(L^{1}\)-Carathéodory if (i) for each \(\rho > 0\), there exists \(\varphi _{\rho } \in L^{1}( [a,b],\mathbb{R}^{+})\) such that \(\|{\mathcal{G}} (t, x, y)\| = \sup \{|v| : v \in {\mathcal{G}} (t, x, y)\} \le \varphi _{\rho } (t) \) for all \(x,y\in {\mathbb{R}}\) with \(\|x\|, \|y\| \le \rho \) and for a.e. \(t \in [a,b]\).

Next, we outline some preliminary concepts of fractional calculus.

Definition 2.1

The fractional integral of order σ with the lower limit zero for a function ζ is defined as

$$ I^{\sigma } \zeta (t)= \frac{1}{\varGamma (\sigma )} \int _{0}^{t}\frac{ \zeta (s)}{(t-s)^{1-\sigma }}\,ds,\quad t>0, \sigma >0, $$

provided the right-hand side is point-wise defined on \([0,\infty )\), where \(\varGamma (\cdot )\) is the gamma function, which is defined by \(\varGamma (\sigma )=\int _{0}^{\infty }t^{\sigma -1}e^{-t}\,dt\).

Definition 2.2

The Riemann–Liouville fractional derivative of order \(\sigma >0\), \(n-1< \sigma <n\), \(n\in {\mathbb{N}}\), is defined as follows:

$$ D_{0+}^{\sigma }\zeta (t)=\frac{1}{\varGamma (n-\sigma )} \biggl( \frac{d}{dt} \biggr) ^{n} \int _{0}^{t} (t-s)^{n-\sigma -1}\zeta (s)\,ds, $$

where the function ζ has absolutely continuous derivative up to order \((n-1)\).

Definition 2.3

The Caputo derivative of order σ for a function \(\zeta :[0, \infty )\to {\mathbb{R}}\) can be written as

$$\begin{aligned} {}^{c}D^{\sigma }_{0+} \zeta (t)= D_{0+}^{\sigma } \Biggl(\zeta (t)- \sum _{k=0}^{n-1}\frac{t^{k}}{k!}\zeta ^{(k)}(0) \Biggr),\quad t>0, n-1< \sigma < n. \end{aligned}$$

In the rest of the paper, we use \({}^{c}D^{\sigma }\) instead of \({}^{c}D^{\sigma }_{0+}\) for the sake of convenience.

Remark 2.4

If \(\zeta \in C^{n}[0,\infty )\), then

$$ {}^{c}D^{\sigma }\zeta (t)= \frac{1}{\varGamma (n-\sigma )} \int _{0}^{t} \frac{ \zeta ^{(n)}(s)}{(t-s)^{\sigma +1-n}}\,ds = I^{n-\sigma }\zeta ^{(n)}(t), \quad t>0, n-1< \sigma < n. $$

The following auxiliary lemma, which concerns the linear variant of system (1.1), plays a key role in the sequel.

Lemma 2.5

Let\(\phi ,h\in C([0,T], {\mathbb{R}})\). Then the solution of the linear fractional differential system

$$ \textstyle\begin{cases} {}^{c}D^{\alpha } x(t) =\phi (t),\quad t\in [0,T], 1< \alpha \leq 2 \\ {}^{\mathrm{RL}}D^{\beta } y(t)=h(t),\quad t\in [0,T], 1< \beta \leq 2, \\ x(0)=\lambda D^{p}y(\eta ), \\ y(0)=0,\qquad y(T)=\gamma I^{q} x(\xi ), \end{cases} $$
(2.1)

is equivalent to the system of integral equations

$$ x(t) = I^{\alpha }\phi (t)+\frac{\lambda }{\varLambda } \biggl[- T^{\beta -1} I^{\beta -p} h(\eta )+\frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl(\gamma I^{q+\alpha }\phi (\xi )-I^{\beta }h(T) \bigr) \biggr] $$
(2.2)

and

$$ y(t) = I^{\beta }h(t)+\frac{t^{\beta -1}}{\varLambda } \biggl[I^{\beta }h(T)- \gamma I^{q+\alpha }\phi (\xi )-\lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}h(\eta ) \biggr], $$
(2.3)

where it is assumed that

$$ \varLambda :=T^{\beta -1}+\lambda \gamma \frac{\varGamma (\beta )\xi ^{q} \eta ^{\beta -p-1}}{\varGamma (1+q)\varGamma (\beta -p)}\ne 0. $$
(2.4)

Proof

Applying the Riemann–Liouville operators \(I^{\alpha }\) and \(I^{\beta }\) to the Caputo and Riemann–Liouville fractional differential equations respectively in (2.1) and using the composition laws of fractional order integral and differential operators [2], we obtain

$$ x(t)=I^{\alpha }\phi (t)+c_{0} \quad \mbox{and}\quad y(t)=I^{\beta }h(t)+c _{1}t^{\beta -1}+c_{2}t^{\beta -2}, $$
(2.5)

where \(c_{0}\), \(c_{1}\), \(c_{2}\) are arbitrary constants.

By the boundary conditions of (2.1) in (2.5), we get \(c_{2}=0\) and a system of algebraic equations in the unknown constants \(c_{0}\) and \(c_{1}\):

$$\begin{aligned}& c_{0}-\lambda \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} c_{1} = \lambda I^{\beta -p}h(\eta ), \\& \gamma \frac{\xi ^{q}}{\varGamma (1+q)}c_{0}-T^{\beta -1}c_{1} = I^{ \beta }h(T)-\gamma I^{q+\alpha }\phi (\xi ). \end{aligned}$$

Solving the above system, we get

$$ c_{0}=\frac{\lambda }{\varLambda } \biggl[- T^{\beta -1} I^{\beta -p} h( \eta )+ \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl( \gamma I^{q+\alpha }\phi (\xi )-I^{\beta }h(T) \bigr) \biggr] $$

and

$$ c_{1}=\frac{1}{\varLambda } \biggl[I^{\beta }h(T)-\gamma I^{q+\alpha } \phi (\xi )-\lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}h( \eta ) \biggr]. $$

Substituting the values of \(c_{0}\), \(c_{1}\), \(c_{2}\) in (2.5), we get solutions (2.2) and (2.3). We can prove the converse of the lemma by direct computation. The proof is completed. □

3 Main results

3.1 Single-valued system (1.1)–(1.3)

Let \(X=\{x(t)|x(t)\in C([0,T], {\mathbb{R}})\}\) be the Banach space endowed with the norm \(\|x\|=\sup \{|x(t)|, t\in [0,T]\}\). Also let the product space \((X\times X, \|(x,y)\|)\) be the Banach space equipped with norm \(\|(x,y)\|=\|x\|+\|y\|\).

We define an operator \({\mathcal{H}}: X\times X\to X\times X\) by

$$ {\mathcal{H}}(x,y) (t)= \begin{pmatrix} {\mathcal{H}}_{1}(x,y)(t) \\ {\mathcal{H}}_{2}(x,y)(t) \end{pmatrix}, $$
(3.1)

where

$$ {\mathcal{H}}_{1}(x,y) (t)= I^{\alpha }\widehat{f}(t)+ \frac{\lambda }{ \varLambda } \biggl[- T^{\beta -1} I^{\beta -p} \widehat{g}( \eta )+ \frac{ \varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl(\gamma I ^{q+\alpha }\widehat{f}(\xi )-I^{\beta }\widehat{g}(T) \bigr) \biggr] $$

and

$$ {\mathcal{H}}_{2}(x,y) (t)= I^{\beta }\widehat{g}(t)+ \frac{t^{\beta -1}}{ \varLambda } \biggl[I^{\beta }\widehat{g}(T)-\gamma I^{q+\alpha } \widehat{f}(\xi )-\lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{ \beta -p} \widehat{g}(\eta ) \biggr], $$

and \(\widehat{f}(t)=f(t,x(t), y(t))\), \(\widehat{g}(t)=g(t, x(t), y(t))\). For convenience, we set the notations:

$$\begin{aligned}& M_{1} = \frac{T^{\alpha }}{\varGamma (1+\alpha )}+\frac{1}{ \vert \varLambda \vert } \vert \lambda \vert \vert \gamma \vert \frac{\varGamma (\beta )}{\varGamma (\beta -p)}\frac{ \eta ^{\beta -p-1}\xi ^{q+\alpha }}{\varGamma (q+\alpha +1)}, \end{aligned}$$
(3.2)
$$\begin{aligned}& M_{2} = \frac{T^{\beta -1}\eta ^{\beta -p-1} \vert \lambda \vert }{ \vert \varLambda \vert } \biggl[\frac{T\varGamma (\beta )}{\varGamma (\beta -p)\varGamma (\beta +1)}+ \frac{ \eta }{\varGamma (1+\beta )} \biggr], \end{aligned}$$
(3.3)
$$\begin{aligned}& M_{3} = \frac{T^{\beta -1} \vert \gamma \vert \xi ^{q+\alpha }}{ \vert \varLambda \vert \varGamma (q+\alpha +1)}, \end{aligned}$$
(3.4)
$$\begin{aligned}& M_{4} = \frac{T^{\beta }}{\varGamma (1+\beta )} \biggl(1+\frac{T^{\beta -1}}{ \vert \varLambda \vert } \biggr)+\frac{T^{\beta -1}}{ \vert \varLambda \vert } \vert \lambda \vert \vert \gamma \vert \frac{ \xi ^{q}\eta ^{\beta -p}}{\varGamma (1+q)\varGamma (\beta -p+1)}. \end{aligned}$$
(3.5)

Our first existence result is based on the Leray–Schauder alternative [37, p. 4].

Theorem 3.1

Assume that:

(\(A_{1}\)):

\(f,g:[0,T]\times \mathbb{R}\times \mathbb{R}\rightarrow \mathbb{R}\)are continuous functions and that there exist real constants\(k_{i},\gamma _{i}\geq 0\) (\(i=0,1,2\)) with\(k_{0}>0\), \(\gamma _{0}>0 \)such that, \(\forall x_{i}\in \mathbb{R}\) (\(i=1,2\)),

$$\begin{aligned}& \bigl\vert f(t,x_{1},x_{2}) \bigr\vert \leq k_{0} + k_{1} \vert x_{1} \vert +k_{2} \vert x_{2} \vert , \\& \bigl\vert g(t,x_{1},x_{2}) \bigr\vert \leq \gamma _{0}+\gamma _{1} \vert x_{1} \vert + \gamma _{2} \vert x_{2} \vert . \end{aligned}$$

If

$$ (M_{1}+M_{3})k_{1}+(M_{2}+M_{4}) \gamma _{1}< 1 \quad \textit{and}\quad (M_{1}+M_{3})k_{2}+(M_{2}+M_{4}) \gamma _{2}< 1, $$
(3.6)

where\(M_{i}\), \(i=1,2,3,4\), are given by (3.2)(3.5), then system (1.1)(1.3) has at least one solution on\([0, T]\).

Proof

Firstly we show that the operator \({\mathcal{H}}:X \times X\to X\times X\) defined by (3.1) is completely continuous. Notice that continuity of the operator \({\mathcal{H}}\) follows from that of the functions f and g.

Let \(\varOmega \subset X\times X\) be bounded. Then there exist positive constants \(L_{1}\) and \(L_{2}\) such that \(|f(t, x(t), y(t))|\le L_{1}\), \(|g(t, x(t), y(t))|\le L_{2}\), \(\forall (x,y)\in \varOmega \). Then, for any \((x,y)\in \varOmega \), we have

$$\begin{aligned} \bigl\vert {\mathcal{H}}_{1}(x, y) (t) \bigr\vert \le & \frac{T^{\alpha }}{\varGamma (1+ \alpha )}L_{1}+\frac{ \vert \lambda \vert }{ \vert \varLambda \vert } \biggl[T^{\beta -1} \frac{ \eta ^{\beta -p}}{\varGamma (\beta -p+1)}L_{2} \\ &{}+ \vert \lambda \vert \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \biggl( \frac{T^{\beta }}{\varGamma (1+\beta )}L_{2}+ \vert \gamma \vert \frac{ \xi ^{q+\alpha }}{\varGamma (q+\alpha +1)}L_{1} \biggr) \biggr] \\ =& M_{1}L_{1}+M_{2}L_{2}, \end{aligned}$$

which implies that

$$ \bigl\Vert {\mathcal{H}}_{1}(x,y) \bigr\Vert \le M_{1}L_{1}+M_{2}L_{2}. $$

In a similar way, we can find that

$$ \bigl\Vert {\mathcal{H}}_{2}(x,y) \bigr\Vert \le M_{3}L_{1}+M_{4}L_{2}. $$

From the above inequalities we conclude that the operator \({\mathcal{H}}\) is uniformly bounded, since \(\|{\mathcal{H}}(x,y)\| \le (M_{1}+M_{3})L_{1}+(M_{2}+M_{4})L_{2}\).

Next, we show that \({\mathcal{H}}\) is equicontinuous. Let \(t_{1}, t _{2} \in [0,T]\) with \(t_{1}< t_{2}\). Then we have

$$\begin{aligned}& \bigl\vert {\mathcal{H}}_{1}\bigl(x(t_{2}),y(t_{2}) \bigr)-{\mathcal{H}}_{1}\bigl(x(t_{1}),y(t _{1})\bigr) \bigr\vert \\& \quad \le L_{1} \biggl\vert \frac{1}{\varGamma (\alpha )} \int _{0}^{t_{2}} (t_{2}-s)^{\alpha -1}\,ds- \frac{1}{ \varGamma (\alpha )} \int _{0}^{t_{1}} (t_{1}-s)^{\alpha -1}\,ds \biggr\vert \\& \quad \le \frac{L_{1}}{\varGamma (\alpha )} \biggl\{ \int _{0}^{t_{1}} \bigl[(t_{2}-s)^{ \alpha -1}-(t_{1}-s)^{\alpha -1} \bigr] \,ds+ \int _{t_{1}}^{t_{2}}(t_{2}-s)^{ \alpha -1} \,ds \biggr\} \\& \quad \le \frac{L_{1}}{\varGamma (\alpha +1)}\bigl[2(t_{2}-t_{1})^{\alpha }+ \bigl\vert t _{2}^{\alpha }-t_{1}^{\alpha } \bigr\vert \bigr]. \end{aligned}$$

Analogously, we can obtain

$$\begin{aligned}& \bigl\vert {\mathcal{H}}_{2}\bigl(x(t_{2}),y(t_{2}) \bigr)-{\mathcal{H}}_{2}\bigl(x(t_{1}),y(t _{1})\bigr) \bigr\vert \\& \quad \le L_{2} \biggl[\frac{T^{\beta }}{\varGamma (1+\beta )}+ \vert \lambda \vert \vert \gamma \vert \frac{\xi ^{q}\eta ^{\beta -p}}{\varGamma (1+q)\varGamma (\beta -p+1)} \biggr]\frac{t _{2}^{\beta -1}-t_{1}^{\beta -1}}{ \vert \varLambda \vert } \\& \qquad {}+L_{1}\frac{ \vert \gamma \vert \xi ^{q+\alpha }}{\varGamma (q+\alpha +1)}\frac{t _{2}^{\beta -1}-t_{1}^{\beta -1}}{ \vert \varLambda \vert }+ \frac{L_{2}}{\varGamma ( \beta +1)}\bigl[2(t_{2}-t_{1})^{\beta }+ \bigl\vert t_{2}^{\beta }-t_{1}^{\beta } \bigr\vert \bigr]. \end{aligned}$$

Thus the operator \({\mathcal{H}}(x,y)\) is equicontinuous. In view of the foregoing arguments, we deduce that the operator \({\mathcal{H}}(x,y)\) is completely continuous.

Finally, it will be verified that the set \({\mathcal{Z}}=\{(x,y) \in X\times X| (x,y)=\theta {\mathcal{H}}(x,y), 0\le \theta \le 1\}\) is bounded. Let \((x,y)\in {\mathcal{Z}}\) with \((x,y)=\theta {\mathcal{H}}(x,y)\). For any \(t\in [0,T]\), we have

$$ x(t)=\theta {\mathcal{H}}_{1}(x,y) (t),\qquad y(t)=\theta {\mathcal{H}} _{2}(x,y) (t). $$

Then

$$\begin{aligned} \bigl\vert x(t) \bigr\vert \le & M_{1}\bigl(k_{0}+k_{1} \vert x \vert +k_{2} \vert y \vert \bigr)+M_{2} \bigl(\gamma _{0}+\gamma _{1} \vert x \vert +\gamma _{2} \vert y \vert \bigr) \\ =&M_{1}k_{0}+M_{2}\gamma _{0}+(M_{1}k_{1}+M_{2}\gamma _{1}) \vert x \vert +(M_{1}k _{2}+M_{2} \gamma _{2}) \vert y \vert , \end{aligned}$$

and

$$\begin{aligned} \bigl\vert y(t) \bigr\vert \le & M_{3}\bigl(k_{0}+k_{1} \vert x \vert +k_{2} \vert y \vert \bigr)+M_{4} \bigl(\gamma _{0}+\gamma _{1} \vert x \vert +\gamma _{2} \vert y \vert \bigr) \\ =&M_{3}k_{0}+M_{4}\gamma _{0}+(M_{3}k_{1}+M_{4}\gamma _{1}) \vert x \vert +(M_{3}k _{2}+M_{4} \gamma _{2}) \vert y \vert . \end{aligned}$$

In consequence, we have

$$ \Vert x \Vert \le M_{1}k_{0}+M_{2} \gamma _{0}+(M_{1}k_{1}+M_{2} \gamma _{1}) \Vert x \Vert +(M _{1}k_{2}+M_{2} \gamma _{2}) \Vert y \Vert $$

and

$$ \Vert y \Vert \le M_{3}k_{0}+M_{4} \gamma _{0}+(M_{3}k_{1}+M_{4} \gamma _{1}) \Vert x \Vert +(M _{3}k_{2}+M_{4} \gamma _{2}) \Vert y \Vert , $$

which imply that

$$\begin{aligned} \Vert x \Vert + \Vert y \Vert \le & (M_{1}+M_{3})k_{0}+ (M_{2}+M_{4})\gamma _{0} +\bigl[(M _{1}+M_{3})k_{1}+(M_{2}+M_{4}) \gamma _{1}\bigr] \Vert x \Vert \\ &{}+\bigl[(M_{1}+M_{3})k_{2}+(M_{2}+M_{4}) \gamma _{2}\bigr] \Vert y \Vert . \end{aligned}$$

Thus we have

$$ \bigl\Vert (x,y) \bigr\Vert \le \frac{(M_{1}+M_{3})k_{0}+ (M_{2}+M_{4})\gamma _{0}}{M_{0}}, $$

where \(M_{0}=\min \{1-[(M_{1}+M_{3})k_{1}+(M_{2}+M_{4})\gamma _{1}], 1-[(M _{1}+M_{3})k_{2}+(M_{2}+M_{4})\gamma _{2}]\}\), which establishes that the set \({\mathcal{Z}}\) is bounded. Thus, by the Leray–Schauder alternative [37], the operator \({\mathcal{H}}\) has at least one fixed point. Hence system (1.1)–(1.3) has at least one solution. The proof is complete. □

The uniqueness of solutions for problem (1.1)–(1.3) is proved in the next theorem via Banach’s contraction mapping principle.

Theorem 3.2

Assume that:

(\(A_{2}\)):

\(f,g:[0,T]\times \mathbb{R}\times \mathbb{R}\rightarrow \mathbb{R}\)are continuous functions and that there exist positive constants\(\ell _{1}\)and\(\ell _{2}\)such that, for all\(t\in [0,T]\)and\(x_{i},y_{i}\in \mathbb{R}\), \(i=1,2\), we have

$$\begin{aligned}& \bigl\vert f(t,x_{1},x_{2})-f(t,y_{1},y_{2}) \bigr\vert \leq \ell _{1} \bigl( \vert x_{1}-y_{1} \vert + \vert x _{2}-y_{2} \vert \bigr), \\& \bigl\vert g(t,x_{1},x_{2})-g(t,y_{1},y_{2}) \bigr\vert \leq \ell _{2} \bigl( \vert x_{1}-y_{1} \vert + \vert x _{2}-y_{2} \vert \bigr). \end{aligned}$$

Then system (1.1)(1.3) has a unique solution on\([0,T]\), provided that

$$ (M_{1}+M_{3})\ell _{1}+(M_{2}+M_{4}) \ell _{2}< 1, $$

where\(M_{i}\), \(i= 1,2,3,4\), are given by (3.2)(3.5).

Proof

Put \(\sup_{t\in [0,T]}f(t,0,0)=N_{1}<\infty \), \(\sup_{t\in [0,T]}g(t,0,0)=N_{2}<\infty \) and choose a positive number r such that

$$ r>\frac{(M_{1}+M_{3})N_{1}+(M_{2}+M_{4})N_{2}}{1-(M_{1}+M_{3})\ell _{1}-(M_{2}+M_{4})\ell _{2}}. $$

Then we show that \({\mathcal{H}}B_{r}\subset B_{r}\), where \(B_{r}=\{(x,y) \in X\times X: \|(x,y)\|\le r\}\) and \({\mathcal{H}}\) is defined by (3.1).

By assumption (\(A_{2}\)), for \((u,v)\in B_{r}\), \(t\in [0,T]\), we have

$$\begin{aligned} \bigl\vert f\bigl(t,x(t),y(t)\bigr) \bigr\vert \leq & \bigl\vert f \bigl(t,x(t),y(t)\bigr)-f(t,0,0) \bigr\vert + \bigl\vert f(t,0,0) \bigr\vert \\ \leq & \ell _{1} \bigl( \bigl\vert x(t) \bigr\vert + \bigl\vert y(t) \bigr\vert \bigr)+N_{1} \\ \leq &\ell _{1} \bigl( \Vert x \Vert + \Vert y \Vert \bigr)+N_{1} \leq \ell _{1} r + N_{1} \end{aligned}$$

and

$$ \bigl\vert g\bigl(t,x(t),y(t)\bigr) \bigr\vert \leq \ell _{2} \bigl( \Vert x \Vert + \Vert y \Vert \bigr)+N_{2} \leq \ell _{2} r +N _{2}. $$

In consequence, we obtain

$$\begin{aligned}& \bigl\vert {\mathcal{H}}_{1}(x, y) (t) \bigr\vert \\& \quad \le \frac{T^{\alpha }}{\varGamma (1+\alpha )}( \ell _{1} r + N_{1})+ \frac{ \vert \lambda \vert }{ \vert \varLambda \vert } \biggl[T^{\beta -1}\frac{\eta ^{\beta -p}}{\varGamma (\beta -p+1)}( \ell _{2} r + N_{2}) \\& \qquad {}+ \vert \lambda \vert \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \biggl( \frac{T^{\beta }}{\varGamma (1+\beta )}( \ell _{2} r + N_{2})+ \vert \gamma \vert \frac{\xi ^{q+\alpha }}{\varGamma (q+\alpha +1)}( \ell _{1} r + N _{1}) \biggr) \biggr] \\& \quad = (\ell _{1}r+N_{1})M_{1}+(\ell _{2} r+N_{2})M_{2} \\& \quad = (M_{1}\ell _{1}+M_{2}\ell _{2})r+M_{1}N_{1}+M_{2}N_{2}, \end{aligned}$$

which implies that

$$ \bigl\Vert {\mathcal{H}}_{1}(x,y) \bigr\Vert \le (M_{1}\ell _{1}+M_{2}\ell _{2})r+M_{1}N _{1}+M_{2}N_{2}. $$

In the same way, we can find that

$$ \bigl\Vert {\mathcal{H}}_{2}(x,y) \bigr\Vert \le (M_{3}\ell _{1}+M_{4}\ell _{2})r+M_{3}N _{1}+M_{4}N_{2}. $$

From the above inequalities, it follows that

$$ \bigl\Vert {\mathcal{H}}(x,y) \bigr\Vert \le \bigl[(M_{1}+M_{3}) \ell _{1}+(M_{2}+M_{4})\ell _{2} \bigr]r+(M_{1}+M_{3})N_{1}+(M_{2}+M_{4})N_{2} \le r. $$

Next, for \((x_{2},y_{2}), (x_{1},y_{1})\in X\times X\) and for any \(t\in [0,T]\), we get

$$\begin{aligned}& \bigl\vert {\mathcal{H}}_{1}(x_{2},y_{2}) (t)-{\mathcal{H}}_{1}(x_{1},y_{1}) (t) \bigr\vert \\& \quad \le \frac{T^{\alpha }}{\varGamma (1+\alpha )}\ell _{1}\bigl( \Vert x_{2}-x_{1} \Vert + \Vert y_{2}-y_{1} \Vert \bigr) \\& \qquad {}+\frac{ \vert \lambda \vert }{ \vert \varLambda \vert } \biggl[T^{\beta -1}\frac{\eta ^{\beta -p}}{ \varGamma (\beta -p+1)}\ell _{2}\bigl( \Vert x_{2}-x_{1} \Vert + \Vert y_{2}-y_{1} \Vert \bigr) \\& \qquad {}+ \vert \lambda \vert \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \biggl( \frac{T^{\beta }}{\varGamma (1+\beta )}\ell _{2}\bigl( \Vert x_{2}-x_{1} \Vert + \Vert y _{2}-y_{1} \Vert \bigr) \\& \qquad {}+ \vert \gamma \vert \frac{\xi ^{q+\alpha }}{\varGamma (q+\alpha +1)}\ell _{1}\bigl( \Vert x _{2}-x_{1} \Vert + \Vert y_{2}-y_{1} \Vert \bigr) \biggr) \biggr] \\& \quad \le ( M_{1} \ell _{1}+ M_{2} \ell _{2}) \bigl( \Vert x_{2}-x_{1} \Vert + \Vert y_{2}-y_{1} \Vert \bigr), \end{aligned}$$

which leads to

$$ \bigl\Vert {\mathcal{H}}_{1}(x_{2},y_{2})-{ \mathcal{H}}_{1}(x_{1},y_{1}) \bigr\Vert \le ( M_{1} \ell _{1}+ M_{2} \ell _{2}) \bigl( \Vert x_{2}-x_{1} \Vert + \Vert y_{2}-y_{1} \Vert \bigr). $$
(3.7)

Similarly, one can obtain

$$ \bigl\Vert {\mathcal{H}}_{2}(x_{2},y_{2}) (t)-{\mathcal{H}}_{2}(x_{1},y_{1}) \bigr\Vert \le ( M_{3} \ell _{1}+ M_{4} \ell _{2}) \bigl( \Vert x_{2}-x_{1} \Vert + \Vert y_{2}-y_{1} \Vert \bigr). $$
(3.8)

From (3.7) and (3.8), we deduce that

$$ \bigl\Vert {\mathcal{H}}(x_{2},y_{2})-{ \mathcal{H}}(x_{1},y_{1}) \bigr\Vert \le \bigl[(M_{1}+M _{3})\ell _{1}+(M_{2}+M_{4}) \ell _{2}\bigr]\bigl( \Vert x_{2}-x_{1} \Vert + \Vert y_{2}-y_{1} \Vert \bigr). $$

Since \((M_{1}+M_{3})\ell _{1}+(M_{2}+M_{4})\ell _{2}<1\), therefore, \({\mathcal{H}}\) is a contraction. So, by Banach’s contraction mapping principle, the operator \({\mathcal{H}}\) has a unique fixed point, which corresponds to a unique solution of problem (1.1)–(1.3). This completes the proof. □

Example 3.3

Consider the following system of fractional boundary value problem:

$$ \textstyle\begin{cases} {}^{c}D^{1/2}x(t) = \frac{1}{4(t+2)^{2}} \frac{ \vert x(t) \vert }{1+ \vert x(t) \vert }+1+\frac{1}{t^{3}+32}\sin ^{2}y(t),\quad t\in [0,1], \\ {}^{\mathrm{RL}}D^{3/2}x(t) = \frac{1}{32 \pi }\sin (2 \pi x(t))+\frac{ \vert y(t) \vert }{16(1+ \vert y(t) \vert )}+\frac{1}{2},\quad t\in [0,1], \\ u(0)=\sqrt{3}D^{1/2}y (\frac{1}{3} ), \\ y(0)=0,\qquad y(1)= \sqrt{2}I^{1/2}x (\frac{1}{2} ). \end{cases} $$
(3.9)

Here, \(\alpha =1/2\), \(\lambda =\sqrt{3}\), \(p=1/2\), \(\eta =1/3\), \(\beta =3/2\), \(\gamma =\sqrt{2}\), \(q=1/2\), \(\xi =1/2\), and \(f(t,x,y)= \frac{1}{4(t+2)^{2}}\frac{|x|}{1+|x|}+1+\frac{1}{t^{3}+32}\sin ^{2}y\) and \(g(t,x,y)=\frac{1}{32\pi }\sin (2\pi x)+\frac{|y|}{16(1+|y|)}+ \frac{1}{2}\). Note that \(|f(t,x_{1},y_{1})-f(t,x_{2},y_{2})| \le \frac{1}{16} |x_{1}-x_{2}|+ \frac{1}{16}|y_{1}-y_{2}|\), \(|g(t,x_{1},y_{1})-g(t,x_{2},y_{2})| \le \frac{1}{16} |x_{1}-x_{2}|+ \frac{1}{16}|y_{1}-y_{2}|\). Using the given data in (3.2)–(3.5), it is found that \(M_{1} \approx 1.5256638\), \(M_{2} \approx 0.58161945\), \(M_{3} \approx 0.258819045\), \(M_{4} \approx 1.26605098\). Clearly \(\ell _{1}=1/16\), \(\ell _{2}=1/16\), and consequently \((M_{1}+M_{3})\ell _{1}+(M_{2}+M_{4})\ell _{2} \approx 0.22700958<1\).

Thus all the conditions of Theorem 3.2 are satisfied; consequently, its conclusion applies to problem (3.9).

3.2 Multi-valued system (1.2)–(1.3)

Definition 3.4

A function \((x,y)\in C^{1}([0,T], {\mathbb{R}})\times C^{2}([0,T], {\mathbb{R}})\) satisfying the coupled boundary conditions \(x(0)= \lambda {}^{c}D^{p}y(\eta )\), \(y(0)=0\), \(y(T)=\gamma I^{q} x(\xi )\) and for which there exist functions \(f,g\in L^{1}([0,T], {\mathbb{R}})\) such that \(f(t)\in F(t,x(t),y(t))\), \(g(t)\in G(t,x(t),y(t))\) a.e. on \(t\in [0,T]\) and

$$ x(t) = I^{\alpha }f(t)+\frac{1}{\varLambda } \biggl[- \lambda T^{\beta -1} I ^{\beta -p} g(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl( \gamma I^{q+\alpha }f(\xi )-I^{\beta }g(T) \bigr) \biggr], $$
(3.10)

and

$$ y(t) = I^{\beta }g(t)+\frac{t^{\beta -1}}{\varLambda } \biggl[I^{\beta }g(T)- \gamma I^{q+\alpha }f(\xi )-\lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}g(\eta ) \biggr] $$
(3.11)

is called a solution of coupled system (1.2)–(1.3).

For each \((x,y)\in X\times X\), the sets of selections of F, G are defined by

$$ S_{F,(x,y)}=\bigl\{ f\in L^{1}\bigl([0,T], {\mathbb{R}}\bigr): f(t)\in F\bigl(t,x(t), y(t)\bigr) \mbox{ for a.e. } t\in [0,T]\bigr\} $$

and

$$ S_{G,(x,y)}=\bigl\{ g\in L^{1}\bigl([0,T], {\mathbb{R}}\bigr): g(t)\in G\bigl(t,x(t), y(t)\bigr) \mbox{ for a.e. } t\in [0,T]\bigr\} . $$

In view of Lemma 2.5, we define the operators \(\mathcal{K}_{1}, \mathcal{K}_{2}: X\times X\to {\mathcal{P}}(X\times X)\) as follows:

$$ \begin{aligned}[b] {\mathcal{K}}_{1}(x,y)(t)&=\bigl\{ h_{1}\in X\times X: \mbox{there exist }\ f \in S_{F,(x,y)}, g\in S_{G,(x,y)}\mbox{ such that} \\ &\quad h_{1}(x,y)(t)=Q_{1}(x,y)(t), \forall t\in [0,T]\bigr\} \end{aligned} $$
(3.12)

and

$$ \begin{aligned}[b] {\mathcal{K}}_{2}(x,y)(t)&=\bigl\{ h_{2}\in X\times X: \mbox{there exists }\ f \in S_{F,(x,y)}, g\in S_{G,(x,y)} \mbox{ such that} \\ &\quad h_{2}(x,y)(t)=Q_{2}(x,y)(t), \forall t\in [0,T]\bigr\} , \end{aligned} $$
(3.13)

where

$$\begin{aligned}& Q_{1}(x,y) (t) \\& \quad = I^{\alpha }f(t)+\frac{1}{\varLambda } \biggl[-\lambda T^{\beta -1} I ^{\beta -p} g(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl( \gamma I^{q+\alpha }f(\xi )-I^{\beta }g(T) \bigr) \biggr] \end{aligned}$$

and

$$ Q_{2}(x,y) (t)= I^{\beta }g(t)+\frac{t^{\beta -1}}{\varLambda } \biggl[I^{ \beta }g(T)-\gamma I^{q+\alpha }f(\xi )-\lambda \gamma \frac{\xi ^{q}}{ \varGamma (1+q)}I^{\beta -p}g(\eta ) \biggr]. $$

Then we define an operator \(\mathcal{K}: X\times X\to {\mathcal{P}}(X \times X)\) by

$$ \mathcal{K}(x,y) (t)= \begin{pmatrix} {\mathcal{K}}_{1}(x,y)(t) \\ \mathcal{K}_{2}(x,y)(t) \end{pmatrix}, $$

where \(\mathcal{K}_{1}\) and \(\mathcal{K}_{2}\) are defined by (3.12) and (3.13).

3.2.1 The Carathéodory case

Our first result dealing with convex values F and G is proved via the Leray–Schauder nonlinear alternative for multi-valued maps [37].

Theorem 3.5

Suppose that the following conditions are satisfied:

(\(B_{1}\)):

\(F,G : [0,T] \times \mathbb{R}^{2} \to {\mathcal{P}}( \mathbb{R})\)are\(L^{1}\)-Carathéodory and have convex values;

(\(B_{2}\)):

There exist continuous nondecreasing functions\(\psi _{1}, \psi _{2},\phi _{1},\phi _{2} : [0,\infty ) \to (0,\infty )\)and functions\(p_{1},p_{2} \in C([0,T],\mathbb{R}_{+})\)such that

$$ \bigl\Vert F(t,x,y) \bigr\Vert _{\mathcal{P}}:=\sup \bigl\{ \vert f \vert : f \in F(t,x,y)\bigr\} \le p_{1}(t)\bigl[ \psi _{1}\bigl( \Vert x \Vert \bigr)+\phi _{1}\bigl( \Vert y \Vert \bigr)\bigr] $$

and

$$ \bigl\Vert G(t,x,y) \bigr\Vert _{\mathcal{P}}:=\sup \bigl\{ \vert g \vert : g \in G(t,x,y)\bigr\} \le p_{2}(t)\bigl[ \psi _{2}\bigl( \Vert x \Vert \bigr)+\phi _{2}\bigl( \Vert y \Vert \bigr)\bigr] $$

for each\((t,x,y) \in [0,T]\times \mathbb{R}^{2}\);

(\(B_{3}\)):

There exists a number\(N>0\)such that

$$ \frac{N}{(M_{1}+M_{3}) \Vert p_{1} \Vert (\psi _{1}(N)+\phi _{1}(N))+(M_{2}+M_{4}) \Vert p_{2} \Vert (\psi _{2}(N)+\phi _{2}(N))}> 1, $$

where\(M_{i}\) (\(i=1, 2, 3, 4\)) are given by (3.2)(3.5).

Then coupled system (1.2)(1.3) has at least one solution on\([0,T]\).

Proof

Consider the operators \(\mathcal{K}_{1}, \mathcal{K}_{2}: X\times X\to {\mathcal{P}}(X\times X)\) defined by (3.12) and (3.13). From (\(B_{1}\)), it follows that the sets \(S_{F,(x,y)}\) and \(S_{G,(x,y)}\) are nonempty for each \((x,y)\in X \times X\). Then, for \(f\in S_{F,(x,y)}\), \(g\in S_{G,(x,y)}\) for \((x,y)\in X\times X\), we have

$$ h_{1}(x,y) (t) = I^{\alpha }f(t)+\frac{1}{\varLambda } \biggl[-\lambda T^{ \beta -1} I^{\beta -p} g(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl(\gamma I^{q+\alpha }f(\xi )-I^{ \beta }g(T) \bigr) \biggr] $$

and

$$ h_{2}(x,y) (t) = I^{\beta }g(t)+\frac{t^{\beta -1}}{\varLambda } \biggl[I ^{\beta }g(T)-\gamma I^{q+\alpha }f(\xi )-\lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}g(\eta ) \biggr], $$

where \(h_{1}\in {\mathcal{K}}_{1}(x,y)\), \(h_{2}\in {\mathcal{K}}_{2}(x,y)\), and so \((h_{1},h_{2})\in {\mathcal{K}}(x,y)\).

It will be established in several steps that the operator \(\mathcal{K}\) satisfies the hypotheses of Leray–Schauder nonlinear alternative. First we show that \(\mathcal{K}(x,y)\) is convex valued. Let \((h_{i},\bar{h}_{i})\in (\mathcal{K}_{1}, \mathcal{K}_{2})\), \(i=1,2\). Then there exist \(f_{i}\in S_{F,(x,y)}\), \(g_{i} \in S_{G,(x,y)}\), \(i=1,2\), such that, for each \(t \in [0,T]\), we have

$$ h_{i}(t)= I^{\alpha }f(t)+\frac{1}{\varLambda } \biggl[- \lambda T^{\beta -1} I^{\beta -p} g(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl( \gamma I^{q+\alpha }f(\xi )-I^{\beta }g(T) \bigr) \biggr] $$

and

$$ \bar{h}_{i}(t) = I^{\beta }g(t)+\frac{t^{\beta -1}}{\varLambda } \biggl[I ^{\beta }g(T)-\gamma I^{q+\alpha }f(\xi )-\lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}g(\eta ) \biggr]. $$

Let \(0 \le \omega \le 1\). Then, for each \(t \in [0,T]\), we have

$$\begin{aligned}& \bigl[\omega h_{1}+(1-\omega )h_{2}\bigr](t) \\& \quad = I^{\alpha }\bigl[\omega f_{1}(s)+(1-\omega )f_{2}(s)\bigr](t)+\frac{1}{ \varLambda } \biggl[-\lambda T^{\beta -1} I^{\beta -p} \bigl[\omega g_{1}(s)+(1- \omega )g_{2}(s)\bigr](\eta ) \\& \qquad {}+\lambda \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl(I^{\beta } \bigl[\omega g_{1}(s)+(1-\omega )g_{2}(s)\bigr](T) \\& \qquad {}-\gamma I^{q+\alpha }\bigl[\omega f_{1}(s)+(1-\omega )f_{2}(s)\bigr](\xi ) \bigr) \biggr] \end{aligned}$$

and

$$\begin{aligned}& \bigl[\omega \bar{h}_{1}+(1-\omega )\bar{h}_{2} \bigr](t) \\& \quad = I^{\beta }\bigl[\omega g_{1}(s)+(1-\omega )g_{2}(s)\bigr](t)+\frac{t^{ \beta -1}}{\varLambda } \biggl[I^{\beta } \bigl[\omega g_{1}(s)+(1-\omega )g_{2}(s)\bigr](T) \\& \qquad {}-\gamma I^{q+\alpha }\bigl[\omega f_{1}(s)+(1-\omega )f_{2}(s)\bigr](\xi )- \lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p} \bigl[\omega g_{1}(s)+(1- \omega )g_{2}(s)\bigr](\eta ) \biggr]. \end{aligned}$$

We deduce that \(S_{F,(x,y)}\), \(S_{G,(x,y)}\) are convex valued, since F, G are convex valued. Obviously, \(\omega h_{1}+(1-\omega )h_{2} \in {\mathcal{K}}_{1}\), \(\omega \bar{h}_{1}+(1-\omega )\bar{h}_{2} \in {\mathcal{K}}_{2}\), and hence \(\omega (h_{1},\bar{h}_{1})+(1- \omega )(h_{2},\bar{h}_{2})\in {\mathcal{K}}\).

Now we show that \(\mathcal{K}\) maps bounded sets into bounded sets in \(X\times X\). For a positive number r, let \(B_{r} = \{(x,y) \in X \times X: \|(x,y)\| \le r \}\) be a bounded set in \(X\times X\). Then there exist \(f \in S_{F,(x,y)}\), \(g \in S_{G,(x,y)}\) such that

$$ h_{1}(x,y) (t) = I^{\alpha }f(t)+\frac{1}{\varLambda } \biggl[-\lambda T^{ \beta -1} I^{\beta -p} g(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl(\gamma I^{q+\alpha }f(\xi )-I^{ \beta }g(T) \bigr) \biggr] $$

and

$$ h_{2}(x,y) (t) = I^{\beta }g(t)+\frac{t^{\beta -1}}{\varLambda } \biggl[I ^{\beta }g(T)-\gamma I^{q+\alpha }f(\xi )-\lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}g(\eta ) \biggr]. $$

Then we have

$$\begin{aligned}& \bigl\vert h_{1}(x,y) (t) \bigr\vert \\& \quad \le I^{\alpha } \bigl\vert f(t) \bigr\vert +\frac{1}{ \vert \varLambda \vert } \biggl[ \vert \lambda \vert T^{ \beta -1} I^{\beta -p} \bigl\vert g(\eta ) \bigr\vert + \vert \lambda \vert \frac{\varGamma (\beta )}{ \varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl(I^{\beta } \bigl\vert g(T) \bigr\vert + \vert \gamma \vert I ^{q+\alpha } \bigl\vert f(\xi ) \bigr\vert \bigr) \biggr] \\& \quad \le \frac{T^{\alpha }}{\varGamma (1+\alpha )} \Vert p_{1} \Vert \bigl(\psi _{1}(r)+\phi _{1}(r)\bigr)+\frac{1}{ \vert \varLambda \vert } \biggl[ \vert \lambda \vert T^{\beta -1}\frac{ \eta ^{\beta -p}}{\varGamma (\beta -p+1)} \Vert p_{2} \Vert \bigl(\psi _{2}(r)+\phi _{2}(r) \bigr) \\& \qquad {}+ \vert \lambda \vert \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \biggl( \frac{T^{\beta }}{\varGamma (1+\beta )} \Vert p_{2} \Vert \bigl(\psi _{2}(r)+\phi _{2}(r)\bigr) \\& \qquad {}+ \vert \gamma \vert \frac{\xi ^{q+\alpha }}{\varGamma (q+\alpha +1)} \Vert p_{1} \Vert \bigl(\psi _{1}(r)+\phi _{1}(r)\bigr) \biggr) \biggr] \\& \quad = M_{1} \Vert p_{1} \Vert \bigl(\psi _{1}(r)+\phi _{1}(r)\bigr)+M_{2} \Vert p_{2} \Vert \bigl(\psi _{2}(r)+ \phi _{2}(r)\bigr) \end{aligned}$$

and

$$ \bigl\vert h_{2}(x,y) (t) \bigr\vert \le M_{3} \Vert p_{1} \Vert \bigl(\psi _{1}(r)+\phi _{1}(r)\bigr)+M_{4} \Vert p_{2} \Vert \bigl(\psi _{2}(r)+\phi _{2}(r)\bigr). $$

Thus,

$$ \bigl\Vert h_{1}(x,y) \bigr\Vert \le M_{1} \Vert p_{1} \Vert \bigl(\psi _{1}(r)+\phi _{1}(r) \bigr)+M_{2} \Vert p _{2} \Vert \bigl(\psi _{2}(r)+\phi _{2}(r)\bigr) $$

and

$$ \bigl\Vert h_{2}(x,y) \bigr\Vert \le M_{3} \Vert p_{1} \Vert \bigl(\psi _{1}(r)+\phi _{1}(r) \bigr)+M_{4} \Vert p _{2} \Vert \bigl(\psi _{2}(r)+\phi _{2}(r)\bigr). $$

Hence we obtain

$$\begin{aligned} \bigl\Vert (h_{1},h_{2}) \bigr\Vert =& \bigl\Vert h_{1}(x,y) \bigr\Vert + \bigl\Vert h_{2}(x,y) \bigr\Vert \\ \le &(M_{1}+M_{3}) \Vert p_{1} \Vert \bigl(\psi _{1}(r)+\phi _{1}(r)\bigr)+(M_{2}+M_{4}) \Vert p _{2} \Vert \bigl(\psi _{2}(r)+\phi _{2}(r)\bigr). \end{aligned}$$

Next, we show that \(\mathcal{K}\) is equicontinuous. Let \(t_{1}, t_{2} \in [0,T]\) with \(t_{1}< t_{2}\). Then there exist \(f \in S_{F,(x,y)}\), \(g \in S_{G,(x,y)}\) such that

$$ h_{1}(x,y) (t) = I^{\alpha }f(t)+\frac{1}{\varLambda } \biggl[-\lambda T^{ \beta -1} I^{\beta -p} g(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl(\gamma I^{q+\alpha }f(\xi )-I^{ \beta }g(T) \bigr) \biggr] $$

and

$$ h_{2}(x,y) (t) = I^{\beta }g(t)+\frac{t^{\beta -1}}{\varLambda } \biggl[I ^{\beta }g(T)-\gamma I^{q+\alpha }f(\xi )-\lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}g(\eta ) \biggr]. $$

Then we have

$$\begin{aligned}& \bigl\vert h_{1}(x,y) (t_{2})-h_{1}(x,y) (t_{1}) \bigr\vert \\& \quad \le \Vert p_{1} \Vert \bigl(\psi _{1}(r)+\phi _{1}(r)\bigr) \biggl\vert \frac{1}{\varGamma (\alpha )} \int _{0}^{t_{2}} (t_{2}-s)^{\alpha -1}\,ds- \frac{1}{ \varGamma (\alpha )} \int _{0}^{t_{1}} (t_{1}-s)^{\alpha -1}\,ds \biggr\vert \\& \quad \le \frac{ \Vert p_{1} \Vert (\psi _{1}(r)+\phi _{1}(r))}{\varGamma (\alpha )} \biggl\{ \int _{0}^{t_{1}} \bigl[(t_{2}-s)^{\alpha -1}-(t_{1}-s)^{\alpha -1} \bigr] \,ds+ \int _{t_{1}}^{t_{2}}(t_{2}-s)^{\alpha -1} \,ds \biggr\} \\& \quad \le \frac{ \Vert p_{1} \Vert (\psi _{1}(r)+\phi _{1}(r))}{\varGamma (\alpha +1)}\bigl[2(t _{2}-t_{1})^{\alpha }+ \bigl\vert t_{2}^{\alpha }-t_{1}^{\alpha } \bigr\vert \bigr]. \end{aligned}$$

Analogously, we can obtain

$$\begin{aligned}& \bigl\vert h_{2}(x,y) (t_{2})-h_{2}(x,y) (t_{1}) \bigr\vert \\ & \quad \le \Vert p_{2} \Vert \bigl(\psi _{2}(r)+\phi _{2}(r)\bigr) \biggl[\frac{T^{\beta }}{\varGamma (1+\beta )}+ \vert \lambda \vert \vert \gamma \vert \frac{\xi ^{q}\eta ^{\beta -p}}{\varGamma (1+q) \varGamma (\beta -p+1)} \biggr]\frac{t_{2}^{\beta -1}-t_{1}^{\beta -1}}{ \vert \varLambda \vert } \\ & \qquad {}+ \Vert p_{1} \Vert \bigl(\psi _{1}(r)+\phi _{1}(r)\bigr)\frac{ \vert \gamma \vert \xi ^{q+\alpha }}{ \varGamma (q+\alpha +1)}\frac{t_{2}^{\beta -1}-t_{1}^{\beta -1}}{ \vert \varLambda \vert } \\ & \qquad {}+\frac{ \Vert p_{2} \Vert (\psi _{2}(r)+\phi _{2}(r))}{\varGamma (\beta +1)}\bigl[2(t _{2}-t_{1})^{\beta }+ \bigl\vert t_{2}^{\beta }-t_{1}^{\beta } \bigr\vert \bigr]. \end{aligned}$$

Therefore, the operator \(\mathcal{K}(x,y)\) is equicontinuous, and thus, by the Ascoli–Arzelá theorem, the operator \(\mathcal{K}(x,y)\) is completely continuous. We know from [35, Proposition 1.2] that a completely continuous operator is upper semicontinuous if it has a closed graph. Thus we need to prove that \(\mathcal{K}\) has a closed graph. Let \((x_{n},y_{n})\to (x_{*},y_{*})\), \((h_{n},\bar{h}_{n}) \in {\mathcal{K}}(x_{n},y_{n})\) and \((h_{n},\bar{h}_{n})\to (h_{*}, \bar{h}_{*})\), then we need to show \((h_{*},\bar{h}_{*})\in {\mathcal{K}}(x _{*},y_{*})\). Observe that \((h_{n},\bar{h}_{n})\in {\mathcal{K}}(x _{n},y_{n})\) implies that there exist \(f_{n}\in S_{F,(x_{n},y_{n})}\) and \(g_{n}\in S_{G,(x_{n},y_{n})}\) such that

$$\begin{aligned}& h_{n}(x_{n},y_{n}) (t) \\ & \quad = I^{\alpha }f_{n}(t)+\frac{1}{\varLambda } \biggl[-\lambda T^{\beta -1} I ^{\beta -p} g_{n}(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma ( \beta -p)} \eta ^{\beta -p-1} \bigl(\gamma I^{q+\alpha }f_{n}( \xi )-I ^{\beta }g_{n}(T) \bigr) \biggr] \end{aligned}$$

and

$$ \bar{h}_{n}(x_{n},y_{n}) (t)= I^{\beta }g_{n}(t)+\frac{t^{\beta -1}}{ \varLambda } \biggl[I^{\beta }g_{n}(T)-\gamma I^{q+\alpha }f_{n}( \xi )- \lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}g_{n}( \eta ) \biggr]. $$

Let us consider the continuous linear operators \(\varPhi _{1}, \varPhi _{2}: L ^{1}([0,T], X\times X)\to C([0,T], X\times X)\) given by

$$ \varPhi _{1}(x,y) (t) = I^{\alpha }f(t)+\frac{1}{\varLambda } \biggl[-\lambda T ^{\beta -1} I^{\beta -p} g(\eta )+\lambda \frac{\varGamma (\beta )}{ \varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl(\gamma I^{q+\alpha }f( \xi )-I^{\beta }g(T) \bigr) \biggr] $$

and

$$ \varPhi _{2}(x,y) (t) = I^{\beta }g(t)+\frac{t^{\beta -1}}{\varLambda } \biggl[I ^{\beta }g(T)-\gamma I^{q+\alpha }f(\xi )-\lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}g(\eta ) \biggr]. $$

From [38] we know that \((\varPhi _{1},\varPhi _{2})\circ (S_{F},S _{G})\) is a closed graph operator. Further, we have \((h_{n},\bar{h} _{n}) \in (\varPhi _{1},\varPhi _{2})\circ (S_{F,(x_{n},y_{n})}, S_{G,(x_{n},y _{n})})\) for all n. Since \((x_{n},y_{n})\to (x_{*},y_{*})\), \((h_{n},\bar{h}_{n})\to (h_{*},\bar{h}_{*})\) it follows that \(f_{*}\in S_{F,(x,y)}\) and \(g_{*}\in S_{G,(x,y)}\) such that

$$\begin{aligned}& h_{*}(x_{*},y_{*}) (t) \\ & \quad = I^{\alpha }f_{*}(t)+\frac{1}{\varLambda } \biggl[-\lambda T^{\beta -1} I ^{\beta -p} g_{*}(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma ( \beta -p)} \eta ^{\beta -p-1} \bigl(\gamma I^{q+\alpha }f_{*}( \xi )-I ^{\beta }g_{*}(T) \bigr) \biggr] \end{aligned}$$

and

$$ \bar{h}_{*}(x_{*},y_{*}) (t)+I^{\beta }g_{*}(t)+\frac{t^{\beta -1}}{ \varLambda } \biggl[I^{\beta }g_{*}(T)-\gamma I^{q+\alpha }f_{*}( \xi )- \lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}g_{*}( \eta ) \biggr], $$

that is, \((h_{n},\bar{h}_{n})\in {\mathcal{K}}(x_{*},y_{*})\).

Finally, we establish the a priori bounds on solutions. Let \((x,y)\in \nu {\mathcal{K}}(x,y)\). Then there exist \(f\in S_{F,(x,y)}\) and \(g\in S_{G,(x,y)}\) such that

$$ x(t) = \nu I^{\alpha }f(t)+\nu \frac{1}{\varLambda } \biggl[-\lambda T^{ \beta -1} I^{\beta -p} g(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl(\gamma I^{q+\alpha }f(\xi )-I^{ \beta }g(T) \bigr) \biggr] $$

and

$$ y(t) = \nu I^{\beta }g(t)+\nu \frac{t^{\beta -1}}{\varLambda } \biggl[I^{ \beta }g(T)- \gamma I^{q+\alpha }f(\xi )-\lambda \gamma \frac{\xi ^{q}}{ \varGamma (1+q)}I^{\beta -p}g( \eta ) \biggr]. $$

For each \(t\in [0,T]\), we obtain

$$ \Vert x \Vert \le M_{1} \Vert p_{1} \Vert \bigl( \psi _{1}\bigl( \Vert x \Vert \bigr)+\phi _{1}\bigl( \Vert y \Vert \bigr)\bigr)+M_{2} \Vert p_{2} \Vert \bigl(\psi _{2}\bigl( \Vert x \Vert \bigr)+\phi _{2} \bigl( \Vert y \Vert \bigr)\bigr) $$

and

$$ \Vert y \Vert \le M_{3} \Vert p_{1} \Vert \bigl(\psi _{1}\bigl( \Vert x \Vert \bigr)+\phi _{1} \bigl( \Vert y \Vert \bigr)\bigr)+M_{4} \Vert p_{2} \Vert \bigl(\psi _{2}\bigl( \Vert x \Vert \bigr)+\phi _{2}\bigl( \Vert y \Vert \bigr)\bigr), $$

following the same arguments as in the second step.

Thus

$$\begin{aligned} \bigl\Vert (x,y) \bigr\Vert =& \Vert x \Vert + \Vert y \Vert \\ \le &(M_{1}+M_{3}) \Vert p_{1} \Vert \bigl(\psi _{1}\bigl( \Vert x \Vert \bigr)+\phi _{1} \bigl( \Vert y \Vert \bigr)\bigr) \\ &{}+(M_{2}+M_{4}) \Vert p_{2} \Vert \bigl(\psi _{2}\bigl( \Vert x \Vert \bigr)+\phi _{2} \bigl( \Vert y \Vert \bigr)\bigr), \end{aligned}$$

which implies that

$$ \frac{ \Vert (x,y) \Vert }{(M_{1}+M_{3}) \Vert p_{1} \Vert (\psi _{1}( \Vert x \Vert )+\phi _{1}( \Vert y \Vert ))+(M_{2}+M_{4}) \Vert p_{2} \Vert (\psi _{2}( \Vert x \Vert )+\phi _{2}( \Vert y \Vert ))}\le 1. $$

In view of (\(B_{3}\)), there exists N such that \(\|(x,y)\|\ne N\). Let us set

$$ U = \bigl\{ (x,y)\in X\times X: \bigl\Vert (x,y) \bigr\Vert < N \bigr\} . $$

Note that the operator \(\mathcal{K} :\overline{U} \to \mathcal{P}_{cp,cv}(X) \times {\mathcal{P}}_{cp,cv}(X)\) is completely continuous and upper semicontinuous. There is no \((x,y) \in \partial U\) such that \((x,y) \in \nu {\mathcal{K}}(x,y)\) for some \(\nu \in (0,1)\) by the choice of U. Hence, by the nonlinear alternative of Leray–Schauder type [37], we deduce that \(\mathcal{K}\) has a fixed point \((x,y) \in \overline{U}\), which is a solution of coupled system (1.2)–(1.3). This completes the proof. □

3.2.2 The Lipschitz case

This subsection is concerned with the case when the multi-valued maps in system (1.2) have non-convex values.

Let \((X,d)\) be a metric space induced from the normed space \((X; \|\cdot \|)\), and let \(H_{d} : {\mathcal{P}}(X) \times {\mathcal{P}}(X) \to \mathbb{R} \cup \{\infty \}\) be defined by \(H_{d}(U, V) = \max \{ \sup_{u \in U}d(u,V), \sup_{v \in V}d(U, v)\}\), where \(d(U,v) = \inf_{u\in U}d(u, v)\) and \(d(u,V) = \inf_{v\in V}d(u, v)\). Then \(({\mathcal{P}}_{b,cl}(X), H_{d})\) is a metric space and \(({\mathcal{P}}_{cl}(X), H_{d})\) is a generalized metric space (see [39]).

Definition 3.6

A multi-valued operator \(\mathcal{G} : X \to {\mathcal{P}}_{cl}(X)\) is called (i) γ-Lipschitz if and only if there exists \(\gamma > 0\) such that \(H_{d}(\mathcal{G}(a),\mathcal{G}(b)) \le \gamma d(a,b)\) for each \(a, b \in X\); and (ii) a contraction if and only if it is γ-Lipschitz with \(\gamma < 1\).

In the forthcoming result, we make use of the fixed point theorem for multi-valued maps due to Covitz and Nadler [40].

Theorem 3.7

If

(\(B_{3}\)):

\(F, G: [0,T] \times \mathbb{R}^{2} \to {\mathcal{P}} _{cp}(\mathbb{R})\)are such that\(F(\cdot ,x, y) : [0,T] \to {\mathcal{P}} _{cp}(\mathbb{R})\)and\(G(\cdot ,x, y) : [0,T] \to {\mathcal{P}}_{cp}( \mathbb{R})\)are measurable for each\(x,y \in \mathbb{R}\);

(\(B_{4}\)):
$$ H_{d}(F(t,x,y), F(t,\bar{x},\bar{y})\le m_{1}(t) \bigl( \vert x-\bar{x} \vert + \vert y- \bar{y} \vert \bigr) $$

and

$$ H_{d}(G(t,x,y), G(t,\bar{x},\bar{y})\le m_{2}(t) \bigl( \vert x-\bar{x} \vert + \vert y- \bar{y} \vert \bigr) $$

for almost all\(t \in [0,T]\)and\(x,y, \bar{x},\bar{y} \in \mathbb{R}\)with\(m_{1},m_{2} \in C([0,T], \mathbb{R}^{+})\)and\(d(0,F(t, 0,0)) \le m_{1}(t)\), \(d(0,G(t,0,0))\le m_{2}(t)\)for almost all\(t \in [0,T]\)

hold, then coupled system (1.2)(1.3) has at least one solution on\([0,T]\)provided that

$$ (M_{1}+M_{3}) \Vert m_{1} \Vert +(M_{2}+M_{4}) \Vert m_{2} \Vert < 1. $$
(3.14)

Proof

The sets \(S_{F,(x,y)}\) and \(S_{G,(x,y)}\) are nonempty for each \((x,y) \in X\times Y\) by assumption (\(B_{3}\)), so F and G have measurable selections (see Theorem III.6 in [41]). Now we show that the operator \(\mathcal{K}\) satisfies the assumptions of Covitz and Nadler’s fixed point theorem [40].

First we show that \(\mathcal{K}(x,y)\in {\mathcal{P}}_{cl}(X)\times {\mathcal{P}}_{cl}(X)\) for each \((x,y)\in X\times X\). Let \((h_{n}, \bar{h} _{n})\in {\mathcal{K}}(x_{n},y_{n})\) such that \((h_{n}, \bar{h}_{n})\to (h,\bar{h})\) in \(X\times X\). Then \((h,\bar{h})\in X \times X\) and there exist \(f_{n}\in S_{F,(x_{n},y_{n})}\) and \(g_{n}\in S_{G,(x_{n},y_{n})}\) such that

$$\begin{aligned}& h_{n}(x_{n},y_{n}) (t) \\& \quad = I^{\alpha }f_{n}(t)+\frac{1}{\varLambda } \biggl[- \lambda T^{\beta -1} I^{\beta -p} g_{n}(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma ( \beta -p)} \eta ^{\beta -p-1} \bigl(I^{\beta }g_{n}(T)- \gamma I^{q+ \alpha }f_{n}(\xi ) \bigr) \biggr] \end{aligned}$$

and

$$ \bar{h}_{n}(x_{n},y_{n}) (t) = I^{\beta }g_{n}(t)+\frac{t^{\beta -1}}{ \varLambda } \biggl[I^{\beta }g_{n}(T)-\gamma I^{q+\alpha }f_{n}( \xi )- \lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}g_{n}( \eta ) \biggr]. $$

Since F and G have compact values, we pass onto subsequences (denoted as sequences) to get that \(f_{n}\) and \(g_{n}\) converge to f and g in \(L^{1} ([0,T],\mathbb{R})\) respectively. Thus \(f \in S_{F,(x,y)}\) and \(g\in S_{G,(x,y)}\) for each \(t \in [0,T]\) and that

$$\begin{aligned} h_{n}(x_{n},y_{n}) (t) \to& h(x,y) (t) \\ =& I^{\alpha }f(t)+\frac{1}{\varLambda } \biggl[-\lambda T^{\beta -1} I ^{\beta -p} g(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl( \gamma I^{q+\alpha }f(\xi )-I^{\beta }g(T) \bigr) \biggr], \end{aligned}$$

and

$$\begin{aligned} \bar{h}_{n}(x_{n},y_{n}) (t) \to& \bar{h}(x,y) (t) \\ =& I^{\beta }g(t)+\frac{t^{\beta -1}}{\varLambda } \biggl[I^{\beta }g(T)- \gamma I^{q+\alpha }f(\xi )-\lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}g( \eta ) \biggr]. \end{aligned}$$

Hence \((h,\bar{h}) \in {\mathcal{K}}\), which implies that \(\mathcal{K}\) is closed.

Next we show that there exists \(\widehat{\theta }<1\) (defined by (3.14)) such that

$$ H_{d}\bigl(\mathcal{K}(x,y), \mathcal{K}(\bar{x},\bar{y})\bigr)\le \widehat{\theta }\bigl( \Vert x-\bar{x} \Vert + \Vert y-\bar{y} \Vert \bigr) \quad \mbox{for each } x, \bar{x}, y,\bar{y}\in X. $$

Let \((x, \bar{x}),(y,\bar{y})\in X\times X\) and \((h_{1},\bar{h_{1}}) \in {\mathcal{K}}(x,y)\). Then there exist \(f_{1} \in S_{F,(x,y)}\) and \(g_{1}\in S_{G,(x,y)}\) such that, for each \(t \in [0,T]\), we have

$$\begin{aligned}& h_{1}(x_{n},y_{n}) (t) \\& \quad = I^{\alpha }f_{1}(t)+\frac{1}{\varLambda } \biggl[-\lambda T^{\beta -1} I ^{\beta -p} g_{1}(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma ( \beta -p)} \eta ^{\beta -p-1} \bigl(I^{\beta }g_{1}(T)- \gamma I^{q+ \alpha }f_{1}(\xi ) \bigr) \biggr] \end{aligned}$$

and

$$ \bar{h}_{1}(x_{n},y_{n}) (t) = I^{\beta }g_{1}(t)+\frac{t^{\beta -1}}{ \varLambda } \biggl[I^{\beta }g_{1}(T)- \gamma I^{q+\alpha }f_{1}(\xi )- \lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}g_{1}(\eta ) \biggr]. $$

By (\(B_{4}\)), we have

$$ H_{d}\bigl(F(t,x,y), F(t,\bar{x},\bar{y})\bigr)\le m_{1}(t) \bigl( \bigl\vert x(t)-\bar{x}(t) \bigr\vert + \bigl\vert y(t)- \bar{y}(t) \bigr\vert \bigr) $$

and

$$ H_{d}\bigl(G(t,x,y), G(t,\bar{x},\bar{y})\bigr)\le m_{2}(t) \bigl( \bigl\vert x(t)-\bar{x}(t) \bigr\vert + \bigl\vert y(t)- \bar{y}(t) \bigr\vert \bigr). $$

So, there exist \(f \in F(t,x(t),y(t))\) and \(g\in G(t,x(t), y(t))\) such that

$$ \bigl\vert f_{1}(t)-w \bigr\vert \le m_{1}(t) \bigl( \bigl\vert x(t)-\bar{x}(t) \bigr\vert + \bigl\vert y(t)-\bar{y}(t) \bigr\vert \bigr) $$

and

$$ \bigl\vert g_{1}(t)-z \bigr\vert \le m_{2}(t) \bigl( \bigl\vert x(t)-\bar{x}(t) \bigr\vert + \bigl\vert y(t)-\bar{y}(t) \bigr\vert \bigr). $$

Define \(V_{1}, V_{2} : [0,T] \to \mathcal{P}(\mathbb{R})\) by

$$ V_{1}(t)=\bigl\{ f \in L^{1}\bigl([0,T],\mathbb{R}\bigr) : \bigl\vert f_{1}(t)-w \bigr\vert \le m_{1}(t) \bigl( \bigl\vert x(t)- \bar{x}(t) \bigr\vert + \bigl\vert y(t)-\bar{y}(t) \bigr\vert \bigr)\bigr\} $$

and

$$ V_{2}(t)=\bigl\{ g \in L^{1}\bigl([0,T],\mathbb{R}\bigr) : \bigl\vert g_{1}(t)-z \bigr\vert \le m_{2}(t) \bigl( \bigl\vert x(t)- \bar{x}(t) \bigr\vert + \bigl\vert y(t)-\bar{y}(t) \bigr\vert \bigr)\bigr\} . $$

Since the multi-valued operators \(V_{1}(t)\cap F(t,x(t), y(t))\) and \(V_{2}(t)\cap G(t,x(t), y(t))\) are measurable (Proposition III.4 in [41]), there exist functions \(f_{2}(t)\), \(g_{2}(t)\) which are a measurable selection for \(V_{1}\), \(V_{2}\) and \(f_{2}(t)\in F(t, x(t), y(t))\), \(g_{2}(t)\in G(t, x(t), y(t))\) such that, for a.e. \(t \in [0,T]\), we have

$$ \bigl\vert f_{1}(t)-f_{2}(t) \bigr\vert \le m_{1}(t) \bigl( \bigl\vert x(t)-\bar{x}(t) \bigr\vert + \bigl\vert y(t)-\bar{y}(t) \bigr\vert \bigr) $$

and

$$ \bigl\vert g_{1}(t)-g_{2}(t) \bigr\vert \le m_{g}(t) \bigl( \bigl\vert x(t)-\bar{x}(t) \bigr\vert + \bigl\vert y(t)-\bar{y}(t) \bigr\vert \bigr). $$

Let

$$\begin{aligned}& h_{2}(x_{n},y_{n}) (t) \\& \quad = I^{\alpha }f_{2}(t)+\frac{1}{\varLambda } \biggl[-\lambda T^{\beta -1} I ^{\beta -p} g_{2}(\eta )+\lambda \frac{\varGamma (\beta )}{\varGamma ( \beta -p)} \eta ^{\beta -p-1} \bigl(I^{\beta }g_{2}(T)- \gamma I^{q+ \alpha }f_{2}(\xi ) \bigr) \biggr] \end{aligned}$$

and

$$ \bar{h}_{2}(x_{n},y_{n}) (t) = I^{\beta }g_{2}(t)+\frac{t^{\beta -1}}{ \varLambda } \biggl[I^{\beta }g_{2}(T)- \gamma I^{q+\alpha }f_{2}(\xi )- \lambda \gamma \frac{\xi ^{q}}{\varGamma (1+q)}I^{\beta -p}g_{2}(\eta ) \biggr]. $$

Thus,

$$\begin{aligned}& \bigl\vert h_{1}(x,y) (t)-h_{2}(x,y) (t) \bigr\vert \\& \quad \le I^{\alpha } \bigl\vert f_{1}(s)-f_{2}(s) \bigr\vert (t)+\frac{1}{ \vert \varLambda \vert } \biggl[ \vert \lambda \vert T^{\beta -1} I^{\beta -p} \bigl\vert g_{1}(s)-g_{2}(s) \bigr\vert (\eta ) \\& \qquad {}+ \vert \lambda \vert \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl(I^{\beta } \bigl\vert g_{1}(s)-g_{2}(s) \bigr\vert (T)+ \vert \gamma \vert I^{q+\alpha } \bigl\vert f_{1}(s)-f _{2}(s) \bigr\vert (\xi ) \bigr) \biggr] \\& \quad \le I^{\alpha }m_{1}(s) \bigl( \bigl\vert x(s)-\bar{x}(s) \bigr\vert + \bigl\vert y(s)-\bar{y}(s) \bigr\vert \bigr) (t) \\& \qquad {}+\frac{1}{ \vert \varLambda \vert } \biggl[ \vert \lambda \vert T^{\beta -1} I^{\beta -p} m _{2}(s) \bigl( \bigl\vert x(s)-\bar{x}(s) \bigr\vert + \bigl\vert y(s)-\bar{y}(s) \bigr\vert \bigr) (\eta ) \\& \qquad {}+ \vert \lambda \vert \frac{\varGamma (\beta )}{\varGamma (\beta -p)} \eta ^{\beta -p-1} \bigl(I^{\beta }m_{2}(s) \bigl( \bigl\vert x(s)-\bar{x}(s) \bigr\vert + \bigl\vert y(s)-\bar{y}(s) \bigr\vert \bigr) (T) \\& \qquad {}+ \vert \gamma \vert I^{q+\alpha }m_{1}(s) \bigl( \bigl\vert x(s)-\bar{x}(s) \bigr\vert + \bigl\vert y(s)-\bar{y}(s) \bigr\vert \bigr) ( \xi ) \bigr) \biggr] \\& \quad \le M_{1} \Vert m_{1} \Vert \bigl( \Vert x- \bar{x} \Vert + \Vert y-\bar{y} \Vert \bigr)+M_{2} \Vert m_{2} \Vert \bigl( \Vert x- \bar{x} \Vert + \Vert y-\bar{y} \Vert \bigr). \end{aligned}$$

Hence

$$ \bigl\Vert h_{1}(x,y)-h_{2}(x,y) \bigr\Vert \le \bigl(M_{1} \Vert m_{1} \Vert +M_{2} \Vert m_{2} \Vert \bigr) \bigl( \Vert x- \bar{x} \Vert + \Vert y-\bar{y} \Vert \bigr). $$

In a similar manner, we can establish that

$$ \bigl\Vert \bar{h}_{1}(x,y)-\bar{h}_{2}(x,y) \bigr\Vert \le \bigl(M_{3} \Vert m_{1} \Vert +M_{4} \Vert m_{2} \Vert \bigr) \bigl( \Vert x- \bar{x} \Vert + \Vert y-\bar{y} \Vert \bigr). $$

Thus

$$ \bigl\Vert (h_{1},\bar{h}_{1}), (h_{2}, \bar{h}_{2}) \bigr\Vert \le \bigl[(M_{1}+M_{3}) \Vert m_{1} \Vert +(M_{2}+M_{4}) \Vert m_{2} \Vert \bigr]\bigl( \Vert x-\bar{x} \Vert + \Vert y- \bar{y} \Vert \bigr). $$

Analogously, interchanging the roles of \((x,y)\) and \((\bar{x},\bar{y})\), we can obtain

$$ H_{d}\bigl(T(x,y), T(\bar{x},\bar{y})\bigr) \le \bigl[(M_{1}+M_{3}) \Vert m_{1} \Vert +(M_{2}+M _{4}) \Vert m_{2} \Vert \bigr] \bigl( \Vert x-\bar{x} \Vert + \Vert y-\bar{y} \Vert \bigr). $$

Therefore \(\mathcal{K}\) is a contraction in view of assumption (3.14). Hence it follows by Covitz and Nadler’s fixed point theorem [40] that \(\mathcal{K}\) has a fixed point \((x,y)\), which is a solution of problem (1.2)–(1.3). This completes the proof. □

4 Conclusion

In the present research we studied the existence of solutions for coupled fractional differential equations and inclusions involving fractional derivatives of different orders and supplemented with nonlocal boundary conditions containing fractional derivative and integral. In the single-valued case we establish existence and uniqueness of solutions by applying the Leray–Schauder alternative and the Banach contraction mapping principle respectively. In the multi-valued case we proved existence results for both convex and non-convex multi-valued maps via the nonlinear alternative for Kakutani maps and Covitz and Nadler’s fixed point theorem. Examples illustrating the obtained results are also constructed.