1 Introduction

Fractional differential equations have arisen as an excellent instrument for the description of hereditary properties of various processes and applications in plenty of fields such as porous medium, network, electro-magnetics, aerodynamics, visco-elasticity, ecology, bioscience, medicine biology, and so on. In the monograph [1], readers can find many mathematical models of fractional differential equations and their applications. Therefore, the study of fractional differential equation has attracted much attention and research. In recent years, many scholars have begun researching some new type of fractional differential equations, which include the neutral terms, integro-differential terms, or coupled terms. There have been some papers dealing with these fractional differential equations (see [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35]). However, it is difficult and challenging to study this type of fractional differential equations since it involves the time delays and the abstract operators. So the purpose of this paper is to consider a class of coupled systems of neutral fractional integro-differential equations with infinite delay in a Banach space as follows:

$$ \textstyle\begin{cases} {}^{c}D_{t}^{\alpha }[x(t)-g_{1}(t,x_{t},y_{t})]=\mathscr{A}[x(t)-g_{1}(t,x _{t},y_{t})]+f_{1} (t,x_{t},y_{t},\int _{0}^{t}e_{1}(t,s,x_{s},y _{s})\,ds ), \\ {}^{c}D_{t}^{\beta }[y(t)-g_{2}(t,x_{t},y_{t})]=\mathscr{B}[y(t)-g_{2}(t,x _{t},y_{t})]+f_{2} (t,x_{t},y_{t},\int _{0}^{t}e_{2}(t,s,x_{s},y _{s})\,ds ), \\ x(t)=\phi _{1}(t)\in \mathcal{B}_{h}, y(t)=\phi _{2}(t)\in \mathcal{B}_{h},\quad t\in (-\infty ,0], \end{cases} $$
(1.1)

where \(0<\alpha \), \(\beta <1\), \(J=[0,b]\), \({}^{c}D_{t}^{\alpha }\) and \({}^{c}D_{t}^{\beta }\) denote the Caputo fractional derivatives. \(\mathscr{A}\), \(\mathscr{B}\) are the infinitesimal generators of a strongly continuous semigroup \(\{\mathcal{T}_{\alpha }(t),t\geq 0\}\) and \(\{\mathcal{T}_{\beta }(t),t\geq 0\}\) in a Banach space X, respectively. \(g_{i}: J\times \mathcal{B}_{h}^{2} \rightarrow X\), \(f_{i}: J\times \mathcal{B}_{h}^{2}\times X \rightarrow X\), \(e_{i}: J^{2}\times \mathcal{B}_{h}^{2} \rightarrow X\) (\(i=1,2\)) are three given continuous functions. We assume that the histories \(x_{t},y_{t}: (-\infty , 0] \rightarrow X\), \(x_{t}(s)=x(t+s)\), \(y_{t}(s)=x(t+s)\), \(s\leq 0\), belong to an abstract phase space \(\mathcal{B}_{h}\) characterized in Preliminaries. Two given initial functions \(\phi _{1}\), \(\phi _{2}\) also belong to an abstract phase space \(\mathcal{B}_{h}\).

In addition, the main inspiration also comes from the literature [10, 13]. In [10], the authors investigated the existence of a mild solution for the following neutral fractional functional integro-differential equation with infinite delay:

$$\begin{aligned} \textstyle\begin{cases} {}^{c}D^{q}_{t}[x(t)-g(t,x_{t})]=\mathscr{A}x(t)+f (t,x_{t}, \int _{0}^{t}h(t,s,x_{s})\,ds ), \quad t\in J=[0,b], \\ x(t)=\phi (t)\in \mathcal{B}_{h}, \quad t\in (-\infty , 0], \end{cases}\displaystyle \end{aligned}$$

where \({}^{c}D^{q}_{t}\) is the Caputo fractional derivative of order \(0< q<1\). \(\mathscr{A}\) is the infinitesimal generator of a strongly continuous semigroup of \(\{\mathcal{T}(t), t>0\}\) in a Banach space X. \(f: J\times \mathcal{B}_{h}\times X\), \(g: J\times \mathcal{B} _{h}\), \(h: J^{2}\times \mathcal{B}_{h}\) are given functions, where \(\mathcal{B}_{h}\) is a phase space. The histories \(x_{t}: (-\infty , 0] {\rightarrow }X\) defined by \(x_{t}(s)=x(t+s)\), \(s\leq 0\), belong to an abstract space \(\mathcal{B}_{h}\).

In [13], Zhang, Huang, and Liu researched the existence and uniqueness of a mild solution for the fractional equation with nonlocal conditions and infinite delay as follows:

$$\begin{aligned} \textstyle\begin{cases} {}^{c}D_{t}^{q}[x(t)-u(t,x_{t})]=\mathscr{A}[x(t)-u(t,x_{t})]+f (t,x _{t}, \int _{0}^{t}h(t,s,x_{s})\,ds ), \quad t\in J=[0,b], t\neq t_{k}, \\ \Delta x|_{t=t_{k}}=I_{k}(x(t_{k}^{-})), \quad k=1,2,\ldots ,n, \\ x(t)=\phi (t)\in \mathcal{B}_{h}, \quad t\in (-\infty , 0], \end{cases}\displaystyle \end{aligned}$$

where \({}^{c}D^{q}_{t}\) is the Caputo fractional derivative of order \(0< q<1\). The operator \(\mathscr{A}\) generates a strongly continuous semigroup of bounded liner operators \(\{\mathcal{T}(t), t>0\}\) in the Banach space X. And u, f are two given continuous functions, \(x(\cdot )\) belongs to the Banach space X, and \(x_{t}(\cdot )\) denotes \(x_{t}(\theta )=x(t+\theta )\), \(\theta \in (-\infty , 0]\). \(I_{k}: X {\rightarrow }X\), \(\Delta x|_{t=t_{k}}=x(t_{k}^{+})-x(t_{k}^{-})\) with \(x(t_{k}^{+})=\lim_{h{\rightarrow }0^{+}}x(t_{k}+h)\), \(x(t_{k}^{-})= \lim_{h{\rightarrow }0^{-}}x(t_{k}+h)\), \(k=1,2,\ldots ,n\), for \(0=t_{0}< t_{1}< t_{2}<\cdots <t_{n}<t_{n+1}=b\).

To the best our knowledge, there are rare papers dealing with the coupled system of neutral fractional integro-differential equations with infinite delay in a Banach space. Therefore, it is important and interesting to study the existence of a mild solution for system (1.1) by the fixed point theorem. Our results are based on the properties of the analytic semigroup and ideas and techniques.

The rest of this paper is organized as follows. In Sect. 2, we recall some definitions and lemmas including the Caputo fractional calculus, the Mittag-Leffler function, and the analytic semigroup. In Sect. 3, we shall prove the existence of solutions for system (1.1). In Sect. 4, an example is given to demonstrate the application of our main results. Finally, the conclusion is given to simply recall our studies and results obtained in Sect. 5.

2 Preliminaries

In this section, we present some primary components, including notations, definitions, lemmas, and so on, which are required in the whole paper.

First, we present the abstract phase space \(\mathcal{B}_{h}\), which has been used in [36]. Assume that \(h: (-\infty ,0]\rightarrow (0,+\infty )\) is a continuous function with \(l=\int _{-\infty }^{0}h(t)\,dt<+\infty \). For any \(c>0\), we define \(\mathcal{B} = \{\psi : [-c,0]\rightarrow X\text{ such that }\psi (t)\text{ is bounded and measurable}\}\) and equip with the norm \(\|\psi \|_{[-c, 0]} = \sup_{s\in [-c, 0]} |\psi (s)|\), \(\forall \psi \in \mathcal{B}\). Let us define

$$\begin{aligned} \mathcal{B}_{h}&= \biggl\{ \psi :(-\infty ,0]\rightarrow X \text{such that for any }a>0, \psi |_{[-a, 0]}\in \mathcal{B} \\ &\quad \text{and } \int _{-\infty }^{0}h(s) \Vert \psi \Vert _{[s,0]}\,ds < + \infty \biggr\} . \end{aligned}$$

If \(\mathcal{B}_{h}\) is equipped with the norm \(\|\psi \|_{ \mathcal{B}_{h}} = \int _{-\infty }^{0}h(s)\|\psi \|_{[s,0]}\,ds\), \(\forall \psi \in \mathcal{B}_{h}\), then it is obvious that \((\mathcal{B}_{h}, \|\cdot \|_{\mathcal{B}_{h}})\) is a Banach space. Define

$$\begin{aligned} \mathcal{B}_{h}^{\prime }=\bigl\{ w=(w_{1},w_{2})|w_{i}:(- \infty , b]\rightarrow X \text{ such that } w_{i}(0)= \phi _{i}(0)\in \mathcal{B}_{h},i=1,2\bigr\} \end{aligned}$$

equipped with a seminorm \(\|w\|_{\mathcal{B}_{h}^{\prime }} =\max \{ \|\phi _{1}\|_{\mathcal{B}_{h}},\|\phi _{2}\|_{\mathcal{B}_{h}},\|w_{1}\|_{b}, \|w_{2}\|_{b}\}\), where \(\|w_{i}\|_{b}=\sup_{s\in [0,b]} \{\|w_{i}(s) \|\}\), \(i=1,2\).

Obviously, \(\mathcal{B}_{h}^{\prime }\subset \mathbb{X}=X\times X\). For any \((x,y)\in \mathbb{X}\), define the seminorm \(\|(x,y)\|=\|(x,y)\|_{ \mathcal{B}'_{h}}\). Thus \(\mathcal{B}_{h}^{\prime }\) is a Banach space.

Lemma 2.1

([10, 36])

Assume \(w=(w_{1},w_{2})\in \mathcal{B}_{h}^{\prime }\), then for \(t \in J\), \(w_{it} \in \mathcal{B}_{h}\). Moreover, \(l|w_{i}(t)| \leq \|w _{it}\|_{\mathcal{B}_{h}} \leq \|\phi _{i}\|_{\mathcal{B}_{h}} + l \sup_{s\in [0, t]}|w_{i}(s)|\), where \(l = \int _{-\infty }^{0}h(t) < +\infty \), \(i=1,2\).

Definition 2.1

([37, 38])

The Riemann–Liouville fractional integral of order α of a continuous function \(f: (a,\infty )\rightarrow R\) is given by

$$\begin{aligned} I_{a^{+}}^{\alpha }f(t)=\frac{1}{\varGamma (\alpha )} \int _{a}^{t}(t-s)^{ \alpha -1}f(s)\,ds, \end{aligned}$$

provided that the right-hand side is pointwise defined on \((a,\infty )\).

Definition 2.2

([37, 38])

If \(f\in C^{n}((a,\infty ),R)\) and \(\alpha >0\), then the Caputo fractional derivative of order α is given by

$$\begin{aligned} {}^{c}D_{a^{+}}^{\alpha }f(t)=\frac{1}{\varGamma (n-\alpha )} \int _{a}^{t}\frac{f ^{(n)}(s)}{(t-s)^{\alpha -n+1}}\,ds, \end{aligned}$$

where \(n-1<\alpha \leq n\), provided that the right-hand side is pointwise defined on \((a,\infty )\).

Lemma 2.2

([37])

Assume that \(u\in C(a,b)\cap L(a,b)\) with a Caputo fractional derivative of order \(\alpha >0\) that belongs to \(C^{n}(a,b)\), then

$$\begin{aligned} I_{a^{+}}^{\alpha } {}^{c}D_{a^{+}}^{\alpha }u(t)=u(t)+c_{1}+c_{2}(t-a)+ \cdots +c_{n}(t-a)^{n-1} \end{aligned}$$

for some \(c_{i}\in R\), \(i=0,1,2,\ldots ,n-1\), where n is the smallest integer greater than or equal to α.

Definition 2.3

([37])

A two-parameter function of the Mittag-Leffler type is defined by the series expansion

$$\begin{aligned} E_{\alpha ,\beta }(z)=\sum_{k=0}^{\infty } \frac{z^{k}}{\varGamma (\alpha k+\beta )}=\frac{1}{2\pi i} \int _{C}\frac{\lambda ^{\alpha -\beta }e ^{\lambda }}{\lambda ^{\alpha }-z}\,d\lambda , \quad \alpha , \beta , z\in \mathbb{C}, \operatorname{Re}(\alpha )>0, \end{aligned}$$

where C is a contour which starts and ends at −∞ and encircles the disc \(|\lambda |\leq \sqrt{|z|}\) counter clockwise. \(\mathbb{C}\) is the complex field, \(\operatorname{Re}(z)\) stands for the real part of complex number z.

For short, \(E_{\alpha }(z)=E_{\alpha ,1}(z)\). It is an entire function which provides a simple generalization of the exponent function \(E_{1}(z)=e^{z}\) and the cosine function \(E_{2}(z^{2})=\cosh (z)\), \(E_{2}(-z^{2})=\cos (z)\), and plays a vital role in the theory of fractional differential equations. The most interesting properties of the Mittag-Leffler functions are associated with their Laplace integral

$$\begin{aligned} \int _{0}^{\infty }e^{-\lambda t}t^{\beta -1}E_{\alpha ,\beta } \bigl(\omega t^{\alpha }\bigr)\,dt=\frac{\lambda ^{\alpha -\beta }}{\lambda ^{\alpha }- \omega }, \quad \operatorname{Re}(\lambda )>\omega ^{\frac{1}{\alpha }}, \omega >0, \end{aligned}$$

and for more details, see [37].

Definition 2.4

([39])

A closed and linear operator \(\mathscr{A}\) is said to be sectorial if there are constants \(\omega \in \mathbb{R}\), \(\theta \in [\frac{ \pi }{2},\pi ]\), \(M>0\) such that the following two conditions are satisfied:

  1. (1)

    \(\rho (\mathscr{A})\subset S_{\omega ,\theta }=\{\lambda \in \mathbb{C}: \lambda \neq \omega , |\arg (\lambda -\omega )|< \theta \}\),

  2. (2)

    \(\|(\lambda I-\mathscr{A})^{-1}\| \leq \frac{\mathcal{M}}{| \lambda -\omega |}\), \(\lambda \in S_{\omega ,\theta }\).

Definition 2.5

([39])

Let \(\mathscr{A}\) be a closed and linear operator with the domain \(D(\mathscr{A})\) defined in a Banach space X. Let \(\rho ( \mathscr{A})\) be the resolvent set of \(\mathscr{A}\). \(\mathscr{A}\) is called the generator of an α-resolvent family if there exist \(\omega \geq 0\) and a strongly continuous function \(\mathcal{T}_{ \alpha }: \mathbb{R}_{+}{\rightarrow }\mathcal{L}(X)\), where \(\mathcal{L}(X)\) is a Banach space of all bounded linear operators from X into X and the corresponding norm is denoted by \(\|\cdot \|\) such that \(\{\lambda ^{\alpha }: \operatorname{Re}(\lambda )>\omega \}\subset \rho ( \mathscr{A})\) and

$$\begin{aligned} \bigl(\lambda ^{\alpha }I-\mathscr{A}\bigr)^{-1}x= \int _{0}^{\infty }e^{\lambda t} \mathcal{T}_{\alpha }(t)x\,dt, \quad \operatorname{Re}(\lambda )>\omega , x \in X, \end{aligned}$$

where \(\mathcal{T}_{\alpha }(t)\) is called the α-resolvent family generated by \(\mathscr{A}\).

Definition 2.6

([39])

Let \(\mathscr{A}\) be a closed and linear operator with the domain \(D(\mathscr{A})\) defined in a Banach space X and \(\alpha >0\). We say that \(\mathscr{A}\) is the generator of a solution operator if there exist \(\omega \geq 0\) and a strongly continuous function \(\mathcal{T} _{\alpha }: \mathbb{R}_{+}{\rightarrow }\mathcal{L}(X)\) such that \(\{\lambda ^{\alpha }: \operatorname{Re}(\lambda )>\omega \}\subset \rho (\mathscr{A})\) and

$$\begin{aligned} \lambda ^{\alpha -1}\bigl(\lambda ^{\alpha }I-\mathscr{A} \bigr)^{-1}x= \int _{0} ^{\infty }e^{\lambda t} \mathcal{T}_{\alpha }(t)x\,dt, \quad \operatorname{Re}(\lambda )>\omega , x \in X, \end{aligned}$$

where \(\mathcal{T}_{\alpha }(t)\) is called the solution operator generated by \(\mathscr{A}\).

The concept of the solution operator is closely related to the concept of a resolvent family. For more details on α-resolvent family and solution operators, we refer the reader to [38] and the references therein.

Lemma 2.3

([16])

If E is a real Banach space and \(F: E \rightarrow E\) is a contraction mapping, then F has a unique fixed point in E.

Lemma 2.4

(Krasnoselskii’s fixed point theorem [15])

Let B be a nonempty closed convex subset of a Banach space \((X, \|\cdot \|)\). Suppose that P and Q map B into X such that

  1. (i)

    \(Px + Qx \in B\);

  2. (ii)

    P is a contraction mapping;

  3. (iii)

    Q is continuous and compact.

Then there exists \(z \in B\) such that \(z = Pz + Qz\).

Lemma 2.5

If the functions \(g_{i}: J\times \mathcal{B}^{2}_{h}\rightarrow X\), \(f _{i}: J\times \mathcal{B}^{2}_{h}\times X\rightarrow X\), \(e_{i}: J^{2} \times \mathcal{B}^{2}_{h}\rightarrow X\) (\(i=1,2\)) satisfy the uniform Hölder condition with exponent \(\gamma \in (0,1]\), \(\mathscr{A}\) and \(\mathscr{B}\) are sectorial operators, then a pair of solutions \((x,y)\in \mathbb{X}=X\times X\) of system (1.1) is given by the following coupled system of integral equations:

$$\begin{aligned} \textstyle\begin{cases} x(t)= \mathcal{T}_{\alpha }(t)[\phi _{1}(0)-g_{1}(0,\phi _{1}(0),\phi _{2}(0))] +g_{1}(t,x_{t},y_{t}) \\ \hphantom{x(t)=}{}+\int _{0}^{t}\mathcal{T}_{\alpha }(t-s)f_{1} (s,x_{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau )\,ds, &t\in J, \\ y(t)= \mathcal{T}_{\beta }(t)[\phi _{2}(0)-g_{2}(0,\phi _{1}(0),\phi _{2}(0))] +g_{2}(t,x_{t},y_{t}) \\ \hphantom{y(t)=}{}+\int _{0}^{t}\mathcal{T}_{\beta }(t-s)f_{2} (s,x_{s},y_{s} \int _{0}^{s}e_{2}(s,\tau ,x_{\tau },y_{\tau })\,d\tau )\,ds, & t\in J, \end{cases}\displaystyle \end{aligned}$$
(2.1)

where

$$\begin{aligned}& \mathcal{T}_{\alpha }(t)=E_{\alpha ,1}\bigl(\mathscr{A}t^{\alpha } \bigr)=\frac{1}{2 \pi i} \int _{\hat{B}_{r}}e^{\lambda t}\frac{\lambda ^{\alpha -1}}{ \lambda ^{\alpha }-\mathscr{A}}\,d\lambda , \\& \mathcal{T}_{\beta }(t)=E_{\beta ,1}\bigl(\mathscr{B}t^{\alpha } \bigr)=\frac{1}{2 \pi i} \int _{\hat{B}_{r}}e^{\lambda t}\frac{\lambda ^{\beta -1}}{ \lambda ^{\beta }-\mathscr{B}}\,d\lambda , \end{aligned}$$

here \(\hat{B}_{r}\) denotes the Bromwich path; \(\mathcal{T}_{\alpha }(t)\) and \(\mathcal{T}_{\beta }(t)\) are the solution operators generated by \(\mathscr{A}\) and \(\mathscr{B}\), respectively.

Proof

Assume that \((x,y)\in \mathbb{X}=X \times X\) is a pair of solutions of system (1.1), then we have for \(t\in J\)

$$\begin{aligned} \textstyle\begin{cases} {}^{c}D_{t}^{\alpha }[x(t)-g_{1}(t,x_{t},y_{t})] =\mathscr{A}[x(t)-g _{1}(t,x_{t},y_{t})]+f_{1} (t,x_{t},y_{t} \int _{0}^{t}e_{1}(t,s,x _{s},y_{s})\,ds ), \\ {}^{c}D_{t}^{\beta }[y(t)-g_{2}(t,x_{t},y_{t})] =\mathscr{B}[y(t)-g _{2}(t,x_{t},y_{t})]+f_{2} (t,x_{t},y_{t} \int _{0}^{t}e_{2}(t,s,x _{s},y_{s})\,ds ). \end{cases}\displaystyle \end{aligned}$$
(2.2)

Applying Lemma 2.2, (2.2) gives

$$\begin{aligned} \textstyle\begin{cases} x(t)-g_{1}(t,x_{t},y_{t})= c_{1}+\frac{1}{\varGamma (\alpha )}\int _{0} ^{t}(t-s)^{\alpha -1}\mathscr{A}[x(s) -g_{1}(s,x_{s},y_{s})]\,ds \\ \hphantom{x(t)-g_{1}(t,x_{t},y_{t})=} {}+\frac{1}{\varGamma (\alpha )}\int _{0}^{t}(t-s)^{\alpha -1}f_{1} (s,x _{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau )\,ds, \\ y(t)-g_{2}(t,x_{t},y_{t})= c_{2}+\frac{1}{\varGamma (\beta )}\int _{0} ^{t}(t-s)^{\beta -1}\mathscr{B}[y(s) -g_{2}(s,x_{s},y_{s})]\,ds \\ \hphantom{y(t)-g_{2}(t,x_{t},y_{t})=}{}+\frac{1}{\varGamma (\beta )}\int _{0}^{t}(t-s)^{\beta -1}f_{2} (s,x _{s},y_{s}, \int _{0}^{s}e_{2}(s,\tau ,x_{\tau },y_{\tau })\,d\tau )\,ds. \end{cases}\displaystyle \end{aligned}$$
(2.3)

By using the initial conditions \(x(0)=\phi _{1}(0)\), \(y(0)=\phi _{2}(0)\), we get

$$\begin{aligned} \textstyle\begin{cases} c_{1}=\phi _{1}(0)-g_{1}(0,\phi _{1}(0),\phi _{2}(0)), \\ c_{2}=\phi _{2}(0)-g_{2}(0,\phi _{1}(0),\phi _{2}(0)). \end{cases}\displaystyle \end{aligned}$$

Therefore,

$$\begin{aligned} \textstyle\begin{cases} x(t)-g_{1}(t,x_{t},y_{t})\\ \quad = \phi _{1}(0)-g_{1}(0,\phi _{1}(0),\phi _{2}(0))+\frac{1}{ \varGamma (\alpha )}\int _{0}^{t}(t-s)^{\alpha -1}\mathscr{A}[x(s) \\ \qquad {} -g_{1}(s,x_{s},y_{s})]\,ds +\frac{1}{\varGamma (\alpha )}\int _{0}^{t}(t-s)^{ \alpha -1}f_{1} (s,x_{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{ \tau },y_{\tau })\,d\tau )\,ds, \\ y(t)-g_{2}(t,x_{t},y_{t})\\ \quad = \phi _{2}(0)-g_{2}(0,\phi _{1}(0),\phi _{2}(0))+\frac{1}{ \varGamma (\beta )}\int _{0}^{t}(t-s)^{\beta -1}\mathscr{B}[y(s) \\ \qquad {}-g_{2}(s,x_{s},y_{s})]\,ds +\frac{1}{\varGamma (\beta )}\int _{0}^{t}(t-s)^{ \beta -1}f_{2} (s,x_{s},y_{s}, \int _{0}^{s}e_{2}(s,\tau ,x_{\tau },y _{\tau })\,d\tau )\,ds. \end{cases}\displaystyle \end{aligned}$$
(2.4)

Taking the Laplace transformation of system (2.4) on both sides, we obtain

$$\begin{aligned} \textstyle\begin{cases} \mathscr{L}\{x(t)-g_{1}(t,x_{t},y_{t})\}\\ \quad = \frac{1}{\lambda }[\phi _{1}(0)-g _{1}(0,\phi _{1}(0),\phi _{2}(0))] \\ \qquad {} +\frac{1}{\lambda ^{\alpha }}\mathscr{A}\mathscr{L}\{x(t) -g_{1}(t,x _{t},y_{t})\} +\frac{1}{\lambda ^{\alpha }}\mathscr{L}\{f_{1} (t,x _{t},y_{t}, \int _{0}^{t}e_{1}(t,s,x_{s},y_{s})\,ds )\}, \\ \mathscr{L}\{y(t)-g_{2}(t,x_{t},y_{t})\}\\ \quad = \frac{1}{\lambda }[\phi _{2}(0)-g_{2}(0,\phi _{1}(0),\phi _{2}(0))] \\ \qquad {} +\frac{1}{\lambda ^{\beta }}\mathscr{B}\mathscr{L}\{y(t) -g_{2}(t,x _{t},y_{t})\} +\frac{1}{\lambda ^{\beta }}\mathscr{L}\{f_{2} (t,x _{t},y_{t}, \int _{0}^{t}e_{2}(t,s,x_{s},y_{s})\,ds )\}. \end{cases}\displaystyle \end{aligned}$$
(2.5)

Taking the inverse Laplace transformation on both sides of system (2.5), we have

$$\begin{aligned} \textstyle\begin{cases} x(t)= \mathcal{T}_{\alpha }(t)[\phi _{1}(0)-g_{1}(0,\phi _{1}(0),\phi _{2}(0))] +g_{1}(t,x_{t},y_{t}) \\ \hphantom{x(t)=}{}+\int _{0}^{t}\mathcal{T}_{\alpha }(t-s)f_{1} (s,x_{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau )\,ds, & t\in J, \\ y(t)= \mathcal{T}_{\beta }(t)[\phi _{2}(0)-g_{2}(0,\phi _{1}(0),\phi _{2}(0))] +g_{2}(t,x_{t},y_{t}) \\ \hphantom{y(t)=}{}+\int _{0}^{t}\mathcal{T}_{\beta }(t-s)f_{2} (s,x_{s},y_{s} \int _{0}^{s}e_{2}(s,\tau ,x_{\tau },y_{\tau })\,d\tau )\,ds, & t\in J. \end{cases}\displaystyle \end{aligned}$$
(2.6)

By the above computation, we obtain the mild solution defined by (2.1) satisfying (1.1). The proof is completed. □

Definition 2.7

A pair of continuous functions \((x,y): (-\infty ,+\infty ){\rightarrow }\mathbb{X}=X\times X\) is said to be a pair of mild solutions of system (1.1) if \(x(0)=\phi _{1}(0)\in \mathcal{B}_{h}\), \(y(0)=\phi _{2}(0)\in \mathcal{B}_{h}\), and \((x,y)\) satisfies the following coupled system (2.1) of integral equations.

3 Existence of mild solutions

In this section, we shall employ the fixed point theorems of the operator to prove the existence of a mild solution to system (1.1). According to Lemma 2.5, we define the operator \(F: \mathbb{X}\rightarrow \mathbb{X}\) by

$$\begin{aligned} F(x,y) (t)= \bigl(F_{1}(x,y) (t), F_{2}(x,y) (t) \bigr)^{T}, \quad \forall (x,y)\in \mathbb{X}, t\in (-\infty ,b], \end{aligned}$$
(3.1)

where

$$\begin{aligned}& F_{1}(x,y) (t)=\textstyle\begin{cases} \mathcal{T}_{\alpha }(t)[\phi _{1}(0)-g_{1}(0,\phi _{1}(0),\phi _{2}(0))] +g_{1}(t,x_{t},y_{t}) \\ \quad {}+\int _{0}^{t}\mathcal{T}_{\alpha }(t-s)f_{1} (s,x_{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau )\,ds, & t \in J, \\ \phi _{1}(t), & t\in (-\infty ,0], \end{cases}\displaystyle \end{aligned}$$
(3.2)
$$\begin{aligned}& F_{2}(x,y) (t)=\textstyle\begin{cases} \mathcal{T}_{\beta }(t)[\phi _{2}(0)-g_{2}(0,\phi _{1}(0),\phi _{2}(0))] +g_{2}(t,x_{t},y_{t}) \\ \quad {}+\int _{0}^{t}\mathcal{T}_{\beta }(t-s)f_{2} (s,x_{s},y_{s} \int _{0}^{s}e_{2}(s,\tau ,x_{\tau },y_{\tau })\,d\tau )\,ds, & t \in J, \\ \phi _{2}, & t\in (-\infty ,0]. \end{cases}\displaystyle \end{aligned}$$
(3.3)

So the existence of solutions for system (1.1) is equivalent to the existence of the fixed point for the operator F defined by (3.1)–(3.3). Next we shall prove our main results.

Theorem 3.1

Assume that conditions \((H_{1})\)\((H_{5})\) hold. Then system (1.1) has a pair of unique mild solutions \((x^{*},y^{*}) \in \mathbb{X}\).

\((H_{1})\) :

The strongly continuous semigroups of bounded linear operators \(\mathcal{T}_{\alpha }(t)\) and \(\mathcal{T}_{\beta }(t)\) generated by \(\mathscr{A}\) and \(\mathscr{B}\) are all compact, and there exist two positive constants \(N_{1}\) and \(N_{2}\) such that the modules of \(\mathcal{T}_{\alpha }(t)\) and \(\mathcal{T}_{\beta }(t)\) satisfy

$$\begin{aligned} \bigl\Vert \mathcal{T}_{\alpha }(t) \bigr\Vert \leq N_{1}, \qquad \bigl\Vert \mathcal{T}_{\beta }(t) \bigr\Vert \leq N_{2}, \quad \forall t\in J. \end{aligned}$$
\((H_{2})\) :

The functions \(g_{i}: J\times \mathcal{B}^{2}_{h} \rightarrow X\) (\(i=1,2\)) are the continuous functions, and there exist some positive constants \(a_{i1}\) and \(a_{i2}\) such that, for all \(t\in J\), \(x_{t}\), \(y_{t}\), \(\overline{x}_{t}\), \(\overline{y}_{t} \in \mathcal{B}_{h}\)

$$\begin{aligned} \bigl\Vert g_{i}(t,x_{t},y_{t})-g_{i}(t, \overline{x}_{t},\overline{y}_{t}) \bigr\Vert &\leq a_{i1} \Vert x_{t}-\overline{x}_{t} \Vert _{\mathcal{B}_{h}} +a_{i2} \Vert y _{t}- \overline{y}_{t} \Vert _{\mathcal{B}_{h}}. \end{aligned}$$
\((H_{3})\) :

The functions \(f_{i}: J\times \mathcal{B}^{2}_{h} \times X \rightarrow X\) (\(i=1,2\)) are the continuous functions, and there exist some positive constants \(b_{i1}\), \(b_{i2}\), and \(b_{i3}\) such that for all \(t\in J\), \(x_{t}\), \(y_{t}\), \(\overline{x} _{t}\), \(\overline{y}_{t}\in \mathcal{B}_{h}\), u, \(v\in X\)

$$\begin{aligned} \bigl\Vert f_{i}(t,x_{t},y_{t},u)-f_{i}(t, \overline{x}_{t},\overline{y}_{t},v) \bigr\Vert &\leq b_{i1} \Vert x_{t}-\overline{x}_{t} \Vert _{\mathcal{B}_{h}} +b_{i2} \Vert y _{t}- \overline{y}_{t} \Vert _{\mathcal{B}_{h}}+b_{i3} \Vert u-v \Vert . \end{aligned}$$
\((H_{4})\) :

The functions \(e_{i}: J^{2}\times \mathcal{B}^{2} _{h}\rightarrow X\) (\(i=1,2\)) are the continuous functions, and there exist some positive constants \(c_{i1}\) and \(c_{i2}\) such that for all \(t,s\in J\), \(x_{s}\), \(y_{s}\), \(\overline{x}_{s}\), \(\overline{y}_{s} \in \mathcal{B}_{h}\)

$$\begin{aligned} \bigl\Vert e_{i}(t,s,x_{s},y_{s})-e_{i}(t,s, \overline{x}_{s},\overline{y}_{s}) \bigr\Vert &\leq c_{i1} \Vert x_{s}-\overline{x}_{s} \Vert _{\mathcal{B}_{h}} +c_{i2} \Vert y _{s}- \overline{y}_{s} \Vert _{\mathcal{B}_{h}}. \end{aligned}$$
\((H_{5})\) :

\(l[a_{i1}+a_{i2}+bN_{i}(b_{i1}+b_{12})+b^{2}N_{i}b _{i3}(c_{i1}+c_{i2})]<1\), \(i=1,2\), where \(l=\int _{-\infty }^{0}h(s)\,ds\).

Proof

Now we use Lemma 2.3 to prove that \(F: \mathbb{X}{\rightarrow }\mathbb{X}\) defined by (3.1)–(3.3) has a unique fixed point. In fact, for all \(t\in (-\infty ,b]\), \((x,y)\), \((\overline{x}, \overline{y})\in \mathbb{X}\), when \(t\in (-\infty ,0]\), we have

$$\begin{aligned} F(x,y) (t)-F(\overline{x},\overline{y}) (t)=\bigl(\phi _{1}(t)-\phi _{1}(t), \phi _{2}(t)-\phi _{2}(t) \bigr)=(0,0), \end{aligned}$$

which implies that

$$\begin{aligned} \bigl\Vert F(x,y) (t)-F(\overline{x},\overline{y}) (t) \bigr\Vert =0. \end{aligned}$$
(3.4)

When \(t\in J=[0,b]\), it follows from \((H_{1})\)\((H_{4})\) that

$$\begin{aligned} & \bigl\Vert F_{1}(x,y) (t)-F_{1}( \overline{x},\overline{y}) (t) \bigr\Vert \\ &\quad = \biggl\Vert g _{1}(t,x_{t},y_{t})-g_{1}(t, \overline{x}_{t},\overline{y}_{t}) + \int _{0}^{t}\mathcal{T}_{\alpha }(t-s) \\ &\qquad {}\times \biggl[f_{1} \biggl(s,x_{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x _{\tau },y_{\tau })\,d\tau \biggr)\,ds-f_{1} \biggl(s,\overline{x}_{s}, \overline{y}_{s}, \int _{0}^{s}e_{1}(s,\tau , \overline{x}_{\tau }, \overline{y}_{\tau })\,d\tau \biggr)\,ds \biggr] \biggr\Vert \\ &\quad \leq \bigl\Vert g_{1}(t,x_{t},y_{t})-g_{1}(t, \overline{x}_{t},\overline{y} _{t}) \bigr\Vert + \int _{0}^{b} \bigl\vert \mathcal{T}_{\alpha }(t-s) \bigr\vert \biggl\| f_{1} \biggl(s,x _{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau \biggr)\,ds \\ &\qquad{}-f_{1} \biggl(s,\overline{x}_{s}, \overline{y}_{s}, \int _{0}^{s}e_{1}(s, \tau , \overline{x}_{\tau },\overline{y}_{\tau })\,d\tau \biggr)\,ds \biggr\| \\ &\quad \leq \bigl\Vert g_{1}(t,x_{t},y_{t})-g_{1}(t, \overline{x}_{t},\overline{y} _{t}) \bigr\Vert + \int _{0}^{b} \bigl\vert \mathcal{T}_{\alpha }(t-s) \bigr\vert \biggl\| f_{1} \biggl(s,x _{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau \biggr)\,ds \\ &\qquad{}-f_{1} \biggl(s,\overline{x}_{s}, \overline{y}_{s}, \int _{0}^{s}e_{1}(s, \tau , \overline{x}_{\tau },\overline{y}_{\tau })\,d\tau \biggr)\,ds \biggr\| \\ &\quad \leq a_{11} \Vert x_{t}- \overline{x}_{t} \Vert _{\mathcal{B}_{h}} +a_{12} \Vert y _{t}-\overline{y}_{t} \Vert _{\mathcal{B}_{h}} +N_{1} \int _{0}^{b} \biggl[b _{11} \Vert x_{s}-\overline{x}_{s} \Vert _{\mathcal{B}_{h}} +b_{12} \Vert y_{s}- \overline{y}_{s} \Vert _{\mathcal{B}_{h}} \\ &\qquad{}+b_{13} \biggl\Vert \int _{0}^{s} \bigl[e_{1}(s,\tau ,x_{\tau },y_{\tau }) -e_{1}(s,\tau , \overline{x}_{\tau },\overline{y}_{\tau }) \bigr]\,d\tau \biggr\Vert \biggr]\,ds \\ &\quad \leq a_{11} \Vert x_{t}- \overline{x}_{t} \Vert _{\mathcal{B}_{h}} +a_{12} \Vert y _{t}-\overline{y}_{t} \Vert _{\mathcal{B}_{h}} +N_{1} \int _{0}^{b} \biggl[b _{11} \Vert x_{s}-\overline{x}_{s} \Vert _{\mathcal{B}_{h}} +b_{12} \Vert y_{s}- \overline{y}_{s} \Vert _{\mathcal{B}_{h}} \\ &\qquad{}+b_{13} \int _{0}^{s} \bigl[c_{11} \Vert x_{\tau }-\overline{x}_{\tau } \Vert _{\mathcal{B}_{h}} +c_{12} \Vert y_{\tau }-\overline{y}_{\tau } \Vert _{ \mathcal{B}_{h}} \bigr]\,d\tau \biggr]\,ds \\ &\quad \leq la_{11} \Vert x-\overline{x} \Vert _{b} +la_{12} \Vert y-\overline{y} \Vert _{b} +N _{1} \int _{0}^{b} \biggl[lb_{11} \Vert x-\overline{x} \Vert _{b} +lb_{12} \Vert y- \overline{y} \Vert _{b} \\ &\qquad{}+b_{13} \int _{0}^{b} \bigl[lc_{11} \Vert x-\overline{x} \Vert _{b} +lc_{12} \Vert y- \overline{y} \Vert _{b} \bigr]\,d\tau \biggr]\,ds \\ &\quad = la_{11} \Vert x-\overline{x} \Vert _{b} +la_{12} \Vert y-\overline{y} \Vert _{b} +bN _{1} \bigl[lb_{11} \Vert x- \overline{x} \Vert _{b} +lb_{12} \Vert y-\overline{y} \Vert _{b} \\ &\qquad{}+bb_{13} \bigl[lc_{11} \Vert x- \overline{x} \Vert _{b} +lc_{12} \Vert y- \overline{y} \Vert _{b} \bigr] \bigr] \\ &\quad \leq l\bigl[a_{11}+a_{12}+bN_{1}(b_{11}+b_{12})+b^{2}N_{1}b_{13}(c_{11}+c _{12})\bigr] \bigl\Vert (x-\overline{x},y-\overline{y}) \bigr\Vert \\ &\quad < \bigl\Vert (x-\overline{x},y- \overline{y}) \bigr\Vert . \end{aligned}$$
(3.5)

Similarly, we have

$$\begin{aligned} & \bigl\Vert F_{2}(x,y) (t)-F_{2}( \overline{x},\overline{y}) (t) \bigr\Vert \\ &\quad \leq l\bigl[a_{21}+a _{22}+bN_{2}(b_{21}+b_{22})+b^{2}N_{2}b_{23}(c_{21}+c_{22}) \bigr] \bigl\Vert (x-\overline{x},y-\overline{y}) \bigr\Vert \\ &\quad{} < \bigl\Vert (x-\overline{x},y-\overline{y}) \bigr\Vert , \quad t\in J=[0,b]. \end{aligned}$$
(3.6)

In view of \((H_{5})\) and (3.4)–(3.6), we know that F defined by (3.1)–(3.3) is a contraction mapping. Then F has a unique fixed point \((x^{*},y^{*})\in \mathbb{X}\). Therefore, system (1.1) has a pair of unique mild solutions \((x^{*},y^{*})\in \mathbb{X}\). The proof of Theorem 3.1 is completed. □

Theorem 3.2

Assume that \((H_{1})\)\((H_{2})\) hold. If the following conditions \((H_{6})\)\((H_{8})\) hold, then system (1.1) has at least a pair of mild solutions.

\((H_{6})\) :

The functions \(f_{i}: J\times \mathcal{B}^{2}_{h} \times X \rightarrow X\) (\(i=1,2\)) are the continuous functions, and there exist some positive constants \(B_{i1}\), \(B_{i2}\), and \(B_{i3}\) such that for all \(t\in J\), \(x_{t}\), \(y_{t}\in \mathcal{B} _{h}\), \(u\in X\)

$$\begin{aligned} \bigl\Vert f_{i}(t,x_{t},y_{t},u) \bigr\Vert &\leq B_{i1} \Vert x_{t} \Vert _{\mathcal{B}_{h}} +B _{i2} \Vert y_{t} \Vert _{\mathcal{B}_{h}}+B_{i3} \Vert u \Vert . \end{aligned}$$
\((H_{7})\) :

The functions \(e_{i}: J^{2}\times \mathcal{B}^{2} _{h}\rightarrow X\) (\(i=1,2\)) are the continuous functions, and there exist some positive constants \(C_{i1}\) and \(C_{i2}\) such that for all \(t,s\in J\), \(x_{s}\), \(y_{s}\in \mathcal{B}_{h}\)

$$\begin{aligned} \bigl\Vert e_{i}(t,s,x_{s},y_{s}) \bigr\Vert &\leq C_{i1} \Vert x_{s} \Vert _{\mathcal{B}_{h}} +C _{i2} \Vert y_{s} \Vert _{\mathcal{B}_{h}}. \end{aligned}$$
\((H_{8})\) :

\(\varrho _{i}=N_{i}+(l+N_{i})(a_{i1}+a_{i2})+lbN_{i}(B _{i1}+B_{i2})+lb^{2}N_{i}B_{i3}(C_{i1}+C_{i2})<1\), \(i=1,2\), where \(l=\int _{-\infty }^{0}h(s)\,ds\).

Proof

Let \(r=\max \{\frac{(1+N_{1})\sup_{t\in J}\|g_{1}(t,0,0)\|}{1- \varrho _{1}}, \frac{(1+N_{2})\sup_{t\in J}\|g_{2}(t,0,0)\|}{1-\varrho _{2}} \}\), \(\overline{\varOmega }=\{(x,y)\in \mathbb{X}:\|(x,y)\| \leq r\}\), where \(\|(x,y)\|=\|(x,y)\|_{\mathcal{B}'_{h}}\) Clearly, Ω̅ is a nonempty closed convex subset of a Banach space \(\mathbb{X}\). Define the operator \(F: \mathbb{X}{\rightarrow } \mathbb{X}\) as (3.1)–(3.3). We split \(F=P+Q\) such that, for all \(t\in (-\infty ,b]\), \((x,y)\in \mathbb{X}\),

$$\begin{aligned}& P_{1}(x,y) (t)=\textstyle\begin{cases} \phi _{1}(t), & t\in (-\infty ,0], \\ \mathcal{T}_{\alpha }(t)[\phi _{1}(0)-g_{1}(0,\phi _{1}(0),\phi _{2}(0))] +g_{1}(t,x_{t},y_{t}), & t\in J, \end{cases}\displaystyle \end{aligned}$$
(3.7)
$$\begin{aligned}& P_{2}(x,y) (t)=\textstyle\begin{cases} \phi _{2}(t), & t\in (-\infty ,0], \\ \mathcal{T}_{\beta }(t)[\phi _{2}(0)-g_{2}(0,\phi _{1}(0),\phi _{2}(0))] +g_{2}(t,x_{t},y_{t}), & t\in J, \end{cases}\displaystyle \end{aligned}$$
(3.8)
$$\begin{aligned}& Q_{1}(x,y) (t)=\textstyle\begin{cases} 0, & t\in (-\infty ,0], \\ \int _{0}^{t}\mathcal{T}_{\alpha }(t-s)f_{1} (s,x_{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau )\,ds, & t \in J, \end{cases}\displaystyle \end{aligned}$$
(3.9)
$$\begin{aligned}& Q_{2}(x,y) (t)=\textstyle\begin{cases} 0, & t\in (-\infty ,0], \\ \int _{0}^{t}\mathcal{T}_{\beta }(t-s)f_{2} (s,x_{s},y_{s} \int _{0}^{s}e_{2}(s,\tau ,x_{\tau },y_{\tau })\,d\tau )\,ds, & t \in J. \end{cases}\displaystyle \end{aligned}$$
(3.10)

When \(t\in (-\infty ,0]\), for all \((x,y)\in \overline{\varOmega }\), we have

$$\begin{aligned} \bigl\Vert F(x,y) (t) \bigr\Vert = \bigl\Vert (\phi _{1},\phi _{2}) \bigr\Vert =\max \bigl\{ \Vert \phi _{1} \Vert _{ \mathcal{B}_{h}}, \Vert \phi _{2} \Vert _{\mathcal{B}_{h}}\bigr\} \leq \bigl\Vert (x,y) \bigr\Vert \leq r. \end{aligned}$$
(3.11)

When \(t\in J=[0,b]\), for all \((x,y)\in \overline{\varOmega }\), it follows from \((H_{1})\)\((H_{2})\) and \((H_{6})\)\((H_{8})\) that

$$\begin{aligned} &\bigl\Vert F_{1}(x,y) (t) \bigr\Vert \\ &\quad = \biggl\Vert \mathcal{T}_{\alpha }(t)\bigl[\phi _{1}(0)-g _{1}\bigl(0,\phi _{1}(0),\phi _{2}(0)\bigr) \bigr] +g_{1}(t,x_{t},y_{t}) \\ &\qquad{}+ \int _{0}^{t}\mathcal{T}_{\alpha }(t-s)f_{1} \biggl(s,x_{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau \biggr)\,ds \biggr\Vert \\ &\quad = \biggl\Vert \mathcal{T}_{\alpha }(t)\bigl[\phi _{1}(0)+g_{1}(0,0,0)-\bigl(g_{1}\bigl(0, \phi _{1}(0),\phi _{2}(0)\bigr)-g_{1}(0,0,0) \bigr)\bigr] \\ &\qquad{}+\bigl(g_{1}(t,x_{t},y_{t})-g_{1}(t,0,0) \bigr)+g_{1}(t,0,0)\\ &\qquad{} + \int _{0}^{t}\mathcal{T}_{\alpha }(t-s)f_{1} \biggl(s,x _{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau \biggr)\,ds \biggr\Vert \\ &\quad \leq \bigl\vert \mathcal{T}_{\alpha }(t) \bigr\vert \bigl[ \Vert \phi _{1} \Vert _{\mathcal{B}_{h}}+ \bigl\Vert g_{1}(0,0,0) \bigr\Vert +a_{11} \Vert \phi _{1} \Vert _{\mathcal{B}_{h}}+a_{12} \Vert \phi _{2} \Vert _{\mathcal{B}_{h}} \bigr]\\ &\qquad{} +\sup_{t\in J} \bigl\Vert g_{1}(t,0,0) \bigr\Vert +a_{11} \Vert x _{t} \Vert _{\mathcal{B}_{h}}+a_{12} \Vert y_{t} \Vert _{\mathcal{B}_{h}} \\ &\qquad{} + \int _{0}^{b} \bigl\vert \mathcal{T}_{ \alpha }(t-s) \bigr\vert \biggl[B_{11} \Vert x_{s} \Vert _{\mathcal{B}_{h}}+B_{12} \Vert y_{s} \Vert _{\mathcal{B}_{h}}+B_{13} \int _{0}^{b} \bigl\Vert e_{1}(s, \tau ,x_{\tau },y_{ \tau }) \bigr\Vert \,d\tau \biggr]\,ds \\ &\quad \leq N_{1} \Bigl[ \Vert \phi _{1} \Vert _{\mathcal{B}_{h}}+\sup_{t\in J} \bigl\Vert g_{1}(t,0,0) \bigr\Vert +a_{11} \Vert \phi _{1} \Vert _{\mathcal{B}_{h}}+a_{12} \Vert \phi _{2} \Vert _{ \mathcal{B}_{h}} \Bigr]\\ &\qquad{} +\sup_{t\in J} \bigl\Vert g_{1}(t,0,0) \bigr\Vert +a_{11} \Vert x_{t} \Vert _{\mathcal{B}_{h}}+a_{12} \Vert y_{t} \Vert _{\mathcal{B}_{h}} \\ &\qquad{}+bN_{1} \bigl[B_{11} \Vert x_{s} \Vert _{\mathcal{B}_{h}}+B_{12} \Vert y_{s} \Vert _{\mathcal{B}_{h}}+bB_{13} \bigl(C_{11} \Vert x_{\tau } \Vert _{\mathcal{B}_{h}}+C_{12} \Vert y_{\tau } \Vert _{\mathcal{B}_{h}}\bigr) \bigr] \\ &\quad \leq N_{1} \Bigl[ \Vert \phi _{1} \Vert _{\mathcal{B}_{h}}+\sup_{t\in J} \bigl\Vert g_{1}(t,0,0) \bigr\Vert +a_{11} \Vert \phi _{1} \Vert _{\mathcal{B}_{h}}+a_{12} \Vert \phi _{2} \Vert _{ \mathcal{B}_{h}} \Bigr]\\ &\qquad{} +\sup_{t\in J} \bigl\Vert g_{1}(t,0,0) \bigr\Vert +la_{11} \Vert x \Vert _{b}+la_{12} \Vert y \Vert _{b} +bN_{1} \bigl[lB_{11} \Vert x \Vert _{b}+lB_{12} \Vert y \Vert _{b} \\ &\qquad{}+bB _{13} \bigl(lC_{11} \Vert x \Vert _{b}+lC_{12} \Vert y \Vert _{b}\bigr) \bigr] \\ &\quad \leq \bigl[N_{1}+(l+N_{1}) (a_{11}+a_{12})+lbN_{1}(B_{11}+B_{12})+lb ^{2}N_{1}B_{13}(C_{11}+C_{12}) \bigr] \bigl\Vert (x,y) \bigr\Vert \\ &\qquad{}+(1+N_{1})\sup_{t\in J} \bigl\Vert g_{1}(t,0,0) \bigr\Vert =\varrho _{1} \bigl\Vert (x,y) \bigr\Vert +(1+N _{1})\sup_{t\in J} \bigl\Vert g_{1}(t,0,0) \bigr\Vert \\ &\quad\leq r. \end{aligned}$$
(3.12)

Similarly, we also get

$$\begin{aligned} & \bigl\Vert F_{2}(x,y) (t) \bigr\Vert \\ &\quad \leq \bigl[N_{2}+(l+N_{2}) (a_{21}+a_{22})+lbN_{2}(B _{21}+B_{22})+lb^{2}N_{2}B_{23}(C_{21}+C_{22}) \bigr] \bigl\Vert (x,y) \bigr\Vert \\ &\qquad{}+(1+N_{2})\sup_{t\in J} \bigl\Vert g_{2}(t,0,0) \bigr\Vert =\varrho _{2} \bigl\Vert (x,y) \bigr\Vert +(1+M _{2})\sup_{t\in J} \bigl\Vert g_{2}(t,0,0) \bigr\Vert \\ &\quad \leq r. \end{aligned}$$
(3.13)

(3.11)–(3.13) indicate that \(P(x,y)+Q(x,y)\in \overline{ \varOmega }\), that is, condition (i) of Lemma 2.4 holds. Similar to (3.11)–(3.13), we also prove that P and Q map Ω̅ into \(\overline{\varOmega }\subset {X}\), and Q is uniformly bounded. Next we only need to show that P is a contraction mapping, and Q is equicontinuous. Indeed, for all \((x,y)\), \((\overline{x},\overline{y})\in \overline{\varOmega }\), when \(t\in (-\infty ,0]\), \(P(x,y)(t)-P(\overline{x},\overline{y})(t)=(\phi _{1}(t)-\phi _{1}(t),\phi _{2}(t)-\phi _{2}(t))=(0,0)\), which leads to

$$\begin{aligned} \bigl\Vert P(x,y) (t)-P(\overline{x},\overline{y}) (t) \bigr\Vert =0. \end{aligned}$$
(3.14)

When \(t\in J=[0,b]\), from (3.7)–(3.8), \((H_{2})\), and \((H_{8})\), we have

$$\begin{aligned} & \bigl\Vert P_{i}(x,y) (t)-P_{i}( \overline{x},\overline{y}) (t) \bigr\Vert \\ &\quad = \bigl\Vert g_{i}(t,x _{t},y_{t})-g_{i}(t,\overline{x}_{t}, \overline{y}_{t}) \bigr\Vert \\ &\quad \leq a_{i1} \Vert x_{t}- \overline{x}_{t} \Vert _{\mathcal{B}_{h}}+a_{i2} \Vert y _{t}-\overline{y}_{t} \Vert _{\mathcal{B}_{h}} \leq la_{i1} \Vert x- \overline{x} \Vert _{b}+la_{i2} \Vert y-\overline{y} \Vert _{b} \\ &\quad \leq l(a_{i1}+a_{i2}) \bigl\Vert (x-\overline{x},y- \overline{y}) \bigr\Vert < \varrho _{i} \bigl\Vert (x- \overline{x},y-\overline{y}) \bigr\Vert < \bigl\Vert (x-\overline{x},y- \overline{y}) \bigr\Vert , \quad i=1,2. \end{aligned}$$
(3.15)

According to (3.14) and (3.15), we conclude that P is contract, namely condition (ii) of Lemma 2.4 holds.

Now we verify that Q is equicontinuous. For all \((x,y)\in \overline{ \varOmega }\), \(t_{1}, t_{2}\in (-\infty ,b]\) with \(t_{1}< t_{2}\). When \(-\infty < t_{1}< t_{2}\leq 0\), (3.9) and (3.10) give

$$\begin{aligned} \bigl\Vert \bigl(Q(x,y) (t_{2})-Q(x,y) (t_{1})\bigr) \bigr\Vert = \bigl\Vert (0,0) \bigr\Vert =0 \rightarrow 0, \quad \text{as } t_{2}\rightarrow t_{1}. \end{aligned}$$
(3.16)

When \(0\leq t_{1}< t_{2}\leq b\), from (3.9), (3.10), \((H_{1})\), \((H_{6})\), and \((H_{7})\), we have

$$\begin{aligned} & \bigl\Vert Q_{1}(x,y) (t_{2})-Q_{1}(x,y) (t_{1}) \bigr\Vert \\ &\quad = \biggl\Vert \int _{0}^{t_{2}} \mathcal{T}_{\alpha }(t_{2}-s)f_{1} \biggl(s,x_{s},y_{s}, \int _{0}^{s}e _{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau \biggr)\,ds \\ &\qquad{}- \int _{0}^{t_{1}}\mathcal{T}_{\alpha }(t_{1}-s)f_{1} \biggl(s,x_{s},y _{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau \biggr)\,ds \biggr\Vert \\ &\quad = \biggl\Vert \int _{0}^{t_{1}}\mathcal{T}_{\alpha }(t_{2}-s)f_{1} \biggl(s,x _{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau \biggr)\,ds \\ &\qquad{}+ \int _{t_{1}}^{t_{2}}\mathcal{T}_{\alpha }(t_{2}-s)f_{1} \biggl(s,x _{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau \biggr)\,ds \\ &\qquad{}- \int _{0}^{t_{1}}\mathcal{T}_{\alpha }(t_{1}-s)f_{1} \biggl(s,x_{s},y _{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau \biggr)\,ds \biggr\Vert \\ &\quad = \biggl\Vert \int _{0}^{t_{1}} \bigl[\mathcal{T}_{\alpha }(t_{2}-s)- \mathcal{T}_{\alpha }(t_{1}-s) \bigr]f_{1} \biggl(s,x_{s},y_{s}, \int _{0} ^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau \biggr)\,ds \\ &\qquad{}+ \int _{t_{1}}^{t_{2}}\mathcal{T}_{\alpha }(t_{2}-s)f_{1} \biggl(s,x _{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d\tau \biggr)\,ds \biggr\Vert \\ &\quad \leq \int _{0}^{b} \bigl\vert \mathcal{T}_{\alpha }(t_{2}-s)-\mathcal{T} _{\alpha }(t_{1}-s) \bigr\vert \biggl\Vert f_{1} \biggl(s,x_{s},y_{s}, \int _{0}^{s}e_{1}(s, \tau ,x_{\tau },y_{\tau })\,d\tau \biggr) \biggr\Vert \,ds \\ &\qquad{}+ \int _{t_{1}}^{t_{2}} \bigl\vert \mathcal{T}_{\alpha }(t_{2}-s) \bigr\vert \biggl\Vert f _{1} \biggl(s,x_{s},y_{s}, \int _{0}^{s}e_{1}(s,\tau ,x_{\tau },y_{\tau })\,d \tau \biggr) \biggr\Vert \,ds \\ &\quad \leq \bigl[lb(B_{11}+B_{12})+lb^{2}B_{13}(C_{11}+C_{12}) \bigr] \bigl\vert \mathcal{T}_{\alpha }(t_{2}-s)- \mathcal{T}_{\alpha }(t_{1}-s) \bigr\vert \\ &\qquad{}+ \bigl[lbN_{1}(B_{11}+B_{12})+lb^{2}N_{1}B_{13}(C_{11}+C_{12}) \bigr](t_{2}-t_{1}) \\ &\quad \rightarrow 0, \quad \text{as } t_{2}\rightarrow t_{1}. \end{aligned}$$
(3.17)

Similar to (3.17), we also get

$$\begin{aligned} &\bigl\Vert Q_{2}(x,y) (t_{2})-Q_{2}(x,y) (t_{1}) \bigr\Vert \\ &\quad \leq \bigl[lb(B_{21}+B_{22})+lb ^{2}B_{23}(C_{21}+C_{22}) \bigr] \bigl\vert \mathcal{T}_{\beta }(t_{2}-s)- \mathcal{T}_{\beta }(t_{1}-s) \bigr\vert \\ &\qquad{}+ \bigl[lbN_{2}(B_{21}+B_{22})+lb^{2}N_{2}B_{23}(C_{21}+C_{22}) \bigr](t_{2}-t_{1}) \\ &\quad \rightarrow 0, \quad \text{as } t_{2}\rightarrow t_{1}. \end{aligned}$$
(3.18)

When \(-\infty < t_{1}<0<t_{2}\leq b\), noticing that \(t_{2}\rightarrow t_{1}\) means that \(t_{2}\rightarrow 0^{+}\) and \(t_{1}\rightarrow 0^{-}\), we obtain

$$\begin{aligned} \bigl\Vert \bigl(Q(x,y) (t_{2})-Q(x,y) (t_{1})\bigr) \bigr\Vert \rightarrow \bigl\Vert (0,0) \bigr\Vert =0, \quad \text{as } t_{2}\rightarrow t_{1}. \end{aligned}$$
(3.19)

From (3.16)–(3.19), we know that Q is equicontinuous. Hence, by the Arzela–Ascoli theorem, we conclude that \(Q:\overline{ \varOmega }{\rightarrow }\overline{\varOmega }\) is completely continuous, that is, condition (iii) of Lemma 2.4 also holds. Thus all the conditions of Lemma 2.4 hold. According to Lemma 2.4, we know that system (1.1) has a pair of solutions \((x^{*},y^{*})\in \overline{\varOmega }\). The proof is completed. □

4 Illustrative example

Consider the following coupled system of neutral partial integro-differential equations with fractional derivative of the form

$$\begin{aligned} \textstyle\begin{cases} {}^{c}D_{t}^{\alpha }[u(t,x)-g_{1}(t,u_{t},v_{t})]\\ \quad =\frac{\partial ^{2}}{ \partial x^{2}}[u(t,x)-g_{1}(t,u_{t},v_{t})] \\ \qquad {} +f_{1} (t,u_{t},v_{t},\int _{0}^{t}e_{1}(t,s,u_{s},v_{s})\,ds ),\quad x \in [0,\pi ],t\in [0,1], \\ {}^{c}D_{t}^{\beta }[v(t,x)-g_{2}(t,u_{t},v_{t})]\\ \quad =\frac{\partial ^{2}}{ \partial x^{2}}[v(t,x)-g_{2}(t,u_{t},v_{t})] \\ \qquad {} +f_{2} (t,u_{t},v_{t},\int _{0}^{t}e_{2}(t,s,u_{s},v_{s})\,ds ),\quad x \in [0,\pi ],t\in [0,1], \\ u(t,x)=\phi _{1}(t,x), v(t,x)=\phi _{2}(t,x),\quad t\in (-\infty ,0], x \in [0,\pi ], \\ u(t,0)=u(t,\pi )=v(t,0)=v(t,\pi )=0,\quad t\geq 0, \end{cases}\displaystyle \end{aligned}$$
(4.1)

where \(0<\alpha \), \(\beta <1\), \(u_{t}=u(t+\theta ,x)\), \(v_{t}=v(t+ \theta ,x)\), \(\theta \leq 0\), \(\phi _{1}(t,x), \phi _{2}(t,x)\in \mathcal{B}_{h}\), \({}^{c}D_{t}^{\alpha }\) and \({}^{c}D_{t}^{\beta }\) are Caputo fractional derivatives.

Let \(X=L^{2}[0,\pi ]\) with the norm \(\|u\|_{L^{2}}=(\int _{0}^{\pi }|u(s)|^{2}\,ds)^{ \frac{1}{2}}\). \(\mathbb{X}=X\times X\) with the norm \(\|(u,v)\|=\max \{\|u\|_{L^{2}},\|v\|_{L^{2}}\}\).We choose \(h=e^{2s}\), \(s<0\), then \(l=\int _{-\infty }^{0}h(s)\,ds=\frac{1}{2}<\infty \) for \(t\leq 0\) and determine

$$\begin{aligned} \Vert \phi \Vert _{\mathfrak{B}_{h}}= \int _{-\infty }^{0}h(s) \sup_{\theta \in [s,0]} \bigl\Vert \phi (\theta ) \bigr\Vert _{L^{2}}\,ds. \end{aligned}$$

Define the operator \(\mathscr{A},\mathscr{B}: D\subset X{\rightarrow }X\) by \(\mathscr{A}w=\mathscr{B}w=w''\) with the domain D defined as

$$\begin{aligned} D=\bigl\{ w\in X: w, w' \text{ are absolutely continuous}, w''\in X, w(0)=w(\pi )=0\bigr\} . \end{aligned}$$

Then

$$\begin{aligned} \mathscr{A}w=\mathscr{B}w=\sum_{n=1}^{\infty }n^{2} \langle w,w_{n} \rangle w_{n}, \quad w\in D, \end{aligned}$$

in which \(w_{n}(x)=\sqrt{\frac{2}{\pi }}\sin (nx)\) (\(n=1,2,\ldots \)) is the orthogonal set of eigenvectors of \(\mathscr{A}\) and \(\mathscr{B}\). It has been long known that \(\mathscr{A}\) and \(\mathscr{B}\) are the infinitesimal generators of an analytic semigroup \(\{\mathcal{T}_{ \alpha }(t)\}_{t\geq 0}\) and \(\{\mathcal{T}_{\beta }(t)\}_{t\geq 0}\) in X and are provided by

$$\begin{aligned} \mathcal{T}_{\alpha }(t)w=\mathcal{T}_{\beta }(t)w=\sum _{n=1}^{\infty }e^{-n^{2}t}\langle w, w_{n}\rangle w_{n}, \quad \forall w\in X, t>0. \end{aligned}$$

Since the analytic semigroups \(\{\mathcal{T}_{\alpha }(t)\}_{t\geq 0}\) and \(\{\mathcal{T}_{\beta }(t)\}_{t\geq 0}\) are compact [37], there exist some constants \(N_{1},N_{2}>0\) such that \(\|\mathcal{T}_{\alpha }(t)\|_{\mathcal{L}(X)}\leq N_{1}\) and \(\|\mathcal{T}_{\beta }(t)\| _{\mathcal{L}(X)}\leq N_{2}\).

Take \(g_{i}(t,u_{t},v_{t})=\frac{\sin t(u_{t}^{2}+v_{t}^{2})}{100}\) (\(i=1,2\)), \(f_{i}(t,u_{t},v_{t},w)=\frac{u_{t}^{3}+\cos (v_{t})+w}{100}\) (\(i=1,2\)), \(e_{i}(t,s,u_{s},v_{s})=\frac{e^{t+s-2}(u_{t}+v_{t})}{100}\) (\(i=1,2\)). By the simple calculation, we have

$$\begin{aligned}& \bigl\Vert g_{i}(t,x_{t},y_{t})-g_{i}(t, \overline{x}_{t},\overline{y}_{t}) \bigr\Vert \leq \frac{\sqrt{\pi }}{50} \Vert x_{t}-\overline{x}_{t} \Vert _{\mathcal{B} _{h}} +\frac{\sqrt{\pi }}{50} \Vert y_{t}- \overline{y}_{t} \Vert _{ \mathcal{B}_{h}}, \\& \bigl\Vert f_{i}(t,x_{t},y_{t},w)-f_{i}(t, \overline{x}_{t},\overline{y}_{t}, \overline{w}) \bigr\Vert \leq \frac{3\sqrt{\pi }}{100} \Vert x_{t}-\overline{x} _{t} \Vert _{\mathcal{B}_{h}} +\frac{\sqrt{\pi }}{100} \Vert y_{t}- \overline{y}_{t} \Vert _{\mathcal{B}_{h}}+ \frac{\sqrt{\pi }}{100} \Vert w- \overline{w} \Vert , \\& \bigl\Vert e_{i}(t,s,x_{s},y_{s})-e_{i}(t,s, \overline{x}_{s},\overline{y}_{s}) \bigr\Vert \leq \frac{\sqrt{\pi }}{100} \Vert x_{s}-\overline{x}_{s} \Vert _{ \mathcal{B}_{h}} +\frac{\sqrt{\pi }}{100} \Vert y_{s}- \overline{y}_{s} \Vert _{\mathcal{B}_{h}}. \end{aligned}$$

From the above discussion, we obtain \(l=\frac{1}{2}\), \(b=1\), \(a_{11}=a_{21}= a_{12}=a_{22}=\frac{\sqrt{\pi }}{50}\), \(b_{11}=b _{21}=\frac{3\sqrt{\pi }}{100}\), \(b_{12}=b_{22}=b_{13}=b_{23}=c_{11}=c _{21}=c_{12}=c_{21}=\frac{\sqrt{\pi }}{100}\). Taking \(N_{i}= \frac{1}{5}\) (\(i=1,2\)), we derive

$$\begin{aligned} l\bigl[a_{i1}+a_{i2}+bN_{i}(b_{i1}+b_{i2})+b^{2}N_{i}b_{i3}(c_{i1}+c_{i2}) \bigr] \approx 0.0426< 1, \quad i=1,2. \end{aligned}$$

Thus, conditions \((H_{1})\)\((H_{5})\) hold. According to Theorem 3.1, we know that system (4.1) has a pair of unique mild solutions \((u^{*},v^{*})\in L^{2}[0,\pi ]\times L^{2}[0,\pi ]\). Similarly, we easily verify that conditions \((H_{6})\)\((H_{8})\) also hold. It follows from Theorem 3.2 that system (4.1) has at least a pair of mild solutions \((u^{*},v^{*})\in L^{2}[0,\pi ] \times L^{2}[0,\pi ]\).

5 Conclusions

The fractional differential equation has a significant role to play in some phenomena and processes of many fields such as physics, chemistry, aerodynamics, electrodynamics of a complex medium, polymer rheology, capacitor theory, electrical circuits, biology, control theory, fitting of experimental data, and so on. Therefore, the subject of fractional differential equations is gaining much importance and attention. In recent years, some scholars began researching the neutral fractional differential equation. These types of equations usually contain the fractional derivatives of time delay and the abstract operators. So it is difficult and challenging to study them. There are rare papers dealing with a coupled system of neutral fractional equations with time delays. Thereby, we study a class of coupled systems of neutral fractional differential integro-differential equations with infinite delay in a Banach space in this paper. By applying the Banach contraction principle and Krasnoselskii’s fixed point theorem, some new existence criteria of mild solutions are obtained.