1 Introduction

The subject of fractional calculus has gained considerable popularity and importance due to its frequent appearance in various fields of science and engineering such as control, porous media, electrochemistry, viscoelasticity, electromagnetic, etc. (see [1–4]). Recently, fractional differential equations have been of great interest due to the intensive development of theory of itself and its applications (see [5–10]). Moreover, the existence of solutions to some coupled systems of fractional differential equations have been studied by many authors (see [11–16]). For instance, Ahmad and Nieto (see [11]) considered a three-point boundary value problem for a coupled system of nonlinear fractional differential equations given by

$$ \left \{ \textstyle\begin{array}{l} D^{\alpha}u(t)=f(t,v(t),D^{p} v(t)),\quad t\in(0,1), \\ D^{\beta}v(t)=g(t,u(t),D^{q} u(t)), \quad t\in(0,1), \\ u(0)=0, \qquad u(1)=\gamma u(\eta), \qquad v(0)=0, \qquad v(1)=\gamma v(\eta), \end{array}\displaystyle \right . $$

where \(1<\alpha,\beta<2\), \(p,q,\gamma>0\), \(0<\eta<1\), \(\alpha-q,\beta-p\geq1\), \(\gamma\eta^{\alpha-1},\gamma\eta^{\beta-1}<1\), and \(D^{\alpha}\) is the standard Riemann-Liouville fractional derivative. Under certain growth conditions on f and g, an existence result was obtained by using the Schauder fixed point theorem. In addition, Bai and Fang (see [12]) discussed the existence of a positive solution to the singular coupled system of the form

$$ \left \{ \textstyle\begin{array}{l} D^{s} u=f(t,v), \quad 0< t< 1, \\ D^{p} v=g(t,u), \quad 0< t< 1, \end{array}\displaystyle \right . $$

where \(0< s,p<1\), \(D^{s}\) is the standard Riemann-Liouville fractional derivative, \(f,g:(0,1]\times[0,+\infty)\rightarrow[0,+\infty)\) are two given continuous functions, and \(\lim_{t\rightarrow0^{+}}f(t,\cdot)=\lim_{t\rightarrow0^{+}}g(t,\cdot )=+\infty\). A nonlinear alternative of Leray-Schauder type and the Krasnoselskii fixed point theorem in a cone were applied to establish the existence results on a positive solution.

The anti-periodic boundary value problems occur in the mathematical modeling of a variety of physical processes (see [17, 18]) and recently received considerable attention. For an example and details of the anti-periodic boundary value problems, see [19, 20] and the references therein.

The turbulent flow in a porous medium is a fundamental mechanics problem. For studying this type of problems, Leibenson (see [21]) introduced the p-Laplacian equation as follows:

$$ \bigl(\phi_{p} \bigl(x'(t)\bigr) \bigr)'=f\bigl(t,x(t),x'(t)\bigr), $$
(1.1)

where \(\phi_{p}(s)=|s|^{p-2}s\), \(p>1\). Obviously, \(\phi_{p}\) is invertible and its inverse operator is \(\phi_{q}\), where \(q>1\) is a constant such that \(1/p+1/q=1\). In the past few decades, many important results as regards (1.1) with certain boundary value conditions have been obtained. We refer the readers to [22–25] and the references cited therein. However, as far as we know, there are relatively few results on the anti-periodic boundary value problems (ABVPs for short) for coupled systems of the fractional p-Laplacian equations.

Motivated by the works mentioned previously, in this paper, we investigate the existence of solutions for ABVP for a coupled system of the fractional p-Laplacian equation of the form

$$ \left \{ \textstyle\begin{array}{l} D_{0^{+}}^{\beta}\phi_{p}(D_{0^{+}}^{\alpha}u(t))=f(t,v(t),D_{0^{+}}^{\gamma}v(t)),\quad t\in[0,1], \\ D_{0^{+}}^{\delta}\phi_{p}(D_{0^{+}}^{\gamma}v(t))=g(t,u(t),D_{0^{+}}^{\alpha}u(t)), \quad t\in[0,1], \\ u(0)=-u(1), \qquad D_{0^{+}}^{\alpha}u(0)=-D_{0^{+}}^{\alpha}u(1), \\ v(0)=-v(1), \qquad D_{0^{+}}^{\gamma}v(0)=-D_{0^{+}}^{\gamma}v(1), \end{array}\displaystyle \right . $$
(1.2)

where \(0<\alpha,\beta,\gamma,\delta\leq1\), \(D_{0^{+}}^{\alpha}\) is a Caputo fractional derivative of order α, and \(f,g:[0,1]\times\mathbb {R}^{2}\rightarrow\mathbb{R}\) are continuous. Note that the nonlinear operator \(D_{0^{+}}^{\beta}\phi_{p}(D_{0^{+}}^{\alpha})\) reduces to the linear operator \(D_{0^{+}}^{\beta}D_{0^{+}}^{\alpha}\) when \(p=2\) and the additive index law

$$ D_{0^{+}}^{\beta}D_{0^{+}}^{\alpha}u(t)=D_{0^{+}}^{\alpha+\beta}u(t) $$

holds under some reasonable constraints on the function u (see [26]).

The rest of this paper is organized as follows. Section 2 contains some necessary notations, definitions and lemmas. In Section 3, based on the Schaefer fixed point theorem, we establish one theorem on the existence of solutions for ABVP (1.2) (Theorem 3.1). Finally, in Section 4, an explicit example is given to illustrate the main result.

2 Preliminaries

For convenience of the readers, we present here some necessary basic knowledge and definitions as regards the fractional calculus theory, which can be found, for instance, in [27, 28].

Definition 2.1

The Riemann-Liouville fractional integral operator of order \(\alpha>0\) of a function \(u:(0,+\infty )\rightarrow\mathbb{R}\) is given by

$$ I_{0^{+}}^{\alpha}u(t)=\frac{1}{\Gamma(\alpha)}\int_{0}^{t} (t-s)^{\alpha-1}u(s)\, ds, $$

provided that the right side integral is pointwise defined on \((0,+\infty)\).

Definition 2.2

The Caputo fractional derivative of order \(\alpha>0\) of a continuous function \(u:(0,+\infty)\rightarrow \mathbb{R}\) is given by

$$\begin{aligned} D_{0^{+}}^{\alpha}u(t) =&I_{0^{+}}^{n-\alpha} \frac{d^{n}u(t)}{d t^{n}} \\ =&\frac{1}{\Gamma(n-\alpha)}\int_{0}^{t}(t-s)^{n-\alpha-1}u^{(n)}(s) \, ds, \end{aligned}$$

where n is the smallest integer greater than or equal to α, provided that the right side integral is pointwise defined on \((0,+\infty)\).

Lemma 2.1

(see [28])

Let \(\alpha>0\). Assume that \(u,D_{0^{+}}^{\alpha}u\in L([0,1],\mathbb{R})\). Then the following equality holds:

$$ I_{0^{+}}^{\alpha}D_{0^{+}}^{\alpha}u(t)=u(t)+c_{0}+c_{1}t+ \cdots+c_{n-1}t^{n-1}, $$

where \(c_{i}\in{\mathbb{R}}\), \(i=0,1,\ldots,n-1\), and n is the smallest integer greater than or equal to α.

Next, we will give the Schaefer fixed point theorem (see for example [25]), which will be used in this paper.

Lemma 2.2

Let X be a Banach space and \(T:X\rightarrow X\) is a completely continuous operator. If the set \(\Omega=\{u\in X|u=\lambda Tu,\lambda\in(0,1)\}\) is bounded, then T has at least one fixed point in X.

In this paper, we take \(Z=C([0,1],\mathbb{R})\) with the norm \(\|z\| _{0}=\max_{t\in[0,1]}|z(t)|\), \(X=\{u|u,D_{0^{+}}^{\alpha}u\in Z\}\) with the norm \(\|u\|_{X}=\max\{\|u\|_{0},\|D_{0^{+}}^{\alpha}u\|_{0}\}\), and \(Y=\{ v|v,D_{0^{+}}^{\gamma}v\in Z\}\) with the norm \(\|v\|_{Y}=\max\{\|v\|_{0},\| D_{0^{+}}^{\gamma}v\|_{0}\}\). For \((u,v)\in X\times Y\), let \(\|(u,v)\|_{X\times Y}=\max\{\|u\|_{X}, \|v\| _{Y}\}\). Obviously, \((X\times Y,\|\cdot\|_{X\times Y})\) is a Banach space.

3 Existence result

In this section, a theorem on the existence of solutions for ABVP (1.2) will be given under the nonlinear growth restrictions of f and g.

As a consequence of Lemma 2.1, we have the following result, which is useful in what follows.

Lemma 3.1

Given \((h_{1},h_{2})\in Z \times Z\), the unique solution of

$$ \left \{ \textstyle\begin{array}{l} D_{0^{+}}^{\beta}\phi_{p}(D_{0^{+}}^{\alpha}u(t))=h_{1}(t), \quad t\in[0,1], \\ D_{0^{+}}^{\delta}\phi_{p}(D_{0^{+}}^{\gamma}v(t))=h_{2}(t), \quad t\in[0,1], \\ u(0)=-u(1),\qquad D_{0^{+}}^{\alpha}u(0)=-D_{0^{+}}^{\alpha}u(1), \\ v(0)=-v(1),\qquad D_{0^{+}}^{\gamma}v(0)=-D_{0^{+}}^{\gamma}v(1) \end{array}\displaystyle \right . $$
(3.1)

is

$$\begin{aligned} \bigl(u(t),v(t)\bigr) =&\bigl(B_{1}(h_{1})+I_{0^{+}}^{\alpha}\phi_{q}\bigl(A_{1}(h_{1})+I_{0^{+}}^{\beta}h_{1}\bigr) (t), \\ & B_{2}(h_{2})+I_{0^{+}}^{\gamma}\phi_{q}\bigl(A_{2}(h_{2})+I_{0^{+}}^{\delta}h_{2}\bigr) (t)\bigr), \end{aligned}$$

where

$$\begin{aligned}& A_{1}(h_{1}) = -\frac{1}{2}I_{0^{+}}^{\beta}h_{1}(1) \\& \hphantom{A_{1}(h_{1})} = -\frac{1}{2\Gamma(\beta)} \int_{0}^{1}(1-s)^{\beta-1}h_{1}(s) \,ds, \\& B_{1}(h_{1}) = -\frac{1}{2}I_{0^{+}}^{\alpha}\phi_{q}\bigl(A_{1}(h_{1})+I_{0^{+}}^{\beta}h_{1}\bigr) (1) \\& \hphantom{B_{1}(h_{1})} = -\frac{1}{2\Gamma(\alpha)}\int_{0}^{1}(1-s)^{\alpha-1} \phi_{q} \biggl(A_{1}(h_{1}) \\& \hphantom{B_{1}(h_{1})={}}{}+\frac{1}{\Gamma(\beta)}\int_{0}^{s}(s- \tau)^{\beta-1}h_{1}(\tau )\, d\tau\biggr)\,ds, \\& A_{2}(h_{2}) = -\frac{1}{2}I_{0^{+}}^{\delta}h_{2}(1) \\& \hphantom{A_{2}(h_{2})} = -\frac{1}{2\Gamma(\delta)} \int_{0}^{1}(1-s)^{\delta-1}h_{2}(s) \,ds, \\& B_{2}(h_{2}) =-\frac{1}{2}I_{0^{+}}^{\gamma}\phi_{q}\bigl(A_{2}(h_{2})+I_{0^{+}}^{\delta}h_{2}\bigr) (1) \\& \hphantom{B_{2}(h_{2})}=-\frac{1}{2\Gamma(\gamma)}\int_{0}^{1}(1-s)^{\gamma-1} \phi_{q} \biggl(A_{2}(h_{2}) \\& \hphantom{B_{2}(h_{2}) ={}}{}+\frac{1}{\Gamma(\delta)}\int_{0}^{s}(s- \tau)^{\delta-1}h_{2}(\tau )\, d\tau\biggr)\,ds, \end{aligned}$$

and \(\phi_{q}\) is understood as the operator \(\phi_{q}:Z\rightarrow Z\) defined by \(\phi_{q}(z)(t)=\phi_{q}(z(t))\).

Proof

Assume that \((u,v)\) satisfies the equations of ABVP (3.1), then Lemma 2.1 implies that

$$ \phi_{p}\bigl(D_{0^{+}}^{\alpha}u(t) \bigr)=c_{1}+I_{0^{+}}^{\beta}h_{1}(t), \quad \forall c_{1}\in \mathbb{R}. $$

From the boundary value condition \(D_{0^{+}}^{\alpha}u(0)=-D_{0^{+}}^{\alpha}u(1)\), one has

$$ c_{1}=-\frac{1}{2}I_{0^{+}}^{\beta}h_{1}(1)=A_{1}(h_{1}). $$

Thus we have

$$ u(t)=c_{2}+I_{0^{+}}^{\alpha}\phi_{q} \bigl(A_{1}(h_{1})+I_{0^{+}}^{\beta}h_{1}\bigr) (t),\quad \forall c_{2}\in\mathbb{R}. $$

By the condition \(u(0)=-u(1)\), we get

$$ c_{2}=-\frac{1}{2}I_{0^{+}}^{\alpha}\phi_{q}\bigl(A_{1}(h_{1})+I_{0^{+}}^{\beta}h_{1}\bigr) (1)=B_{1}(h_{1}). $$

A similar proof can show that

$$ v(t)=c_{4}+I_{0^{+}}^{\gamma}\phi_{q} \bigl(c_{3}+I_{0^{+}}^{\delta}h_{2}\bigr) (t), $$

where

$$\begin{aligned}& c_{3}=-\frac{1}{2}I_{0^{+}}^{\delta}h_{2}(1)=A_{2}(h_{2}), \\& c_{4}=-\frac{1}{2}I_{0^{+}}^{\gamma}\phi_{q}\bigl(A_{2}(h_{2})+I_{0^{+}}^{\delta}h_{2}\bigr) (1)=B_{2}(h_{2}). \end{aligned}$$

The proof is complete. □

Define the operator \(\mathcal{T}:X\times Y\rightarrow X\times Y\) by

$$\begin{aligned} \mathcal{T}(u,v) (t) =&\bigl(B_{1}(N_{1}v)+I_{0^{+}}^{\alpha}\phi_{q}\bigl(A_{1}(N_{1}v)+I_{0^{+}}^{\beta}N_{1}v\bigr) (t), \\ &B_{2}(N_{2}u)+I_{0^{+}}^{\gamma}\phi_{q}\bigl(A_{2}(N_{2}u)+I_{0^{+}}^{\delta}N_{2}u\bigr) (t)\bigr) \\ :=&\bigl(T_{1}v(t),T_{2}u(t)\bigr), \quad \forall t \in[0,1], \end{aligned}$$

where

$$\begin{aligned}& T_{1}v(t)= -\frac{1}{2\Gamma(\alpha)}\int_{0}^{1}(1-s)^{\alpha-1} \phi_{q} \biggl(-\frac{1}{2\Gamma(\beta)} \\& \hphantom{T_{1}v(t)={}}{} \cdot\int_{0}^{1}(1- \tau)^{\beta-1}f\bigl(\tau,v(\tau),D_{0^{+}}^{\gamma}v(\tau) \bigr)\,d\tau \\& \hphantom{T_{1}v(t)={}}{} +\frac{1}{\Gamma(\beta)} \int_{0}^{s}(s- \tau)^{\beta-1}f\bigl(\tau,v(\tau),D_{0^{+}}^{\gamma}v(\tau) \bigr)\,d\tau \biggr)\,ds \\& \hphantom{T_{1}v(t)={}}{} +\frac{1}{\Gamma(\alpha)}\int_{0}^{t}(t-s)^{\alpha-1} \phi_{q} \biggl(-\frac{1}{2\Gamma(\beta)} \\& \hphantom{T_{1}v(t)={}}{} \cdot\int_{0}^{1}(1- \tau)^{\beta-1}f\bigl(\tau,v(\tau),D_{0^{+}}^{\gamma}v(\tau) \bigr)\,d\tau \\& \hphantom{T_{1}v(t)={}}{} +\frac{1}{\Gamma(\beta)} \int_{0}^{s}(s- \tau)^{\beta-1}f\bigl(\tau,v(\tau),D_{0^{+}}^{\gamma}v(\tau) \bigr)\,d\tau \biggr)\,ds,\quad \forall t\in[0,1], \\& T_{2}u(t) = -\frac{1}{2\Gamma(\gamma)}\int_{0}^{1}(1-s)^{\gamma-1} \phi_{q} \biggl(-\frac{1}{2\Gamma(\delta)} \\& \hphantom{T_{2}u(t) ={}}{} \cdot\int_{0}^{1}(1- \tau)^{\delta-1}g\bigl(\tau,u(\tau),D_{0^{+}}^{\alpha}u(\tau) \bigr)\,d\tau \\& \hphantom{T_{2}u(t) ={}}{} +\frac{1}{\Gamma(\delta)} \int_{0}^{s}(s- \tau)^{\delta-1}g\bigl(\tau,u(\tau),D_{0^{+}}^{\alpha}u(\tau) \bigr)\,d\tau \biggr)\,ds \\& \hphantom{T_{2}u(t) ={}}{} +\frac{1}{\Gamma(\gamma)}\int_{0}^{t}(t-s)^{\gamma-1} \phi_{q} \biggl(-\frac{1}{2\Gamma(\delta)} \\& \hphantom{T_{2}u(t) ={}}{}\cdot\int_{0}^{1}(1- \tau)^{\delta-1}g\bigl(\tau,u(\tau),D_{0^{+}}^{\alpha}u(\tau) \bigr)\,d\tau \\& \hphantom{T_{2}u(t) ={}}{} +\frac{1}{\Gamma(\delta)} \int_{0}^{s}(s- \tau)^{\delta-1}g\bigl(\tau,u(\tau),D_{0^{+}}^{\alpha}u(\tau) \bigr)\,d\tau \biggr)\,ds,\quad \forall t\in[0,1], \end{aligned}$$

and \(N_{1}:Y\rightarrow Z\), \(N_{2}:X\rightarrow Z\) are Nemytskii operators defined by

$$\begin{aligned}& N_{1}v(t)=f\bigl(t,v(t),D_{0^{+}}^{\gamma}v(t)\bigr), \quad \forall t\in[0,1], \\& N_{2}u(t)=g\bigl(t,u(t),D_{0^{+}}^{\alpha}u(t)\bigr), \quad \forall t\in[0,1]. \end{aligned}$$

Clearly, the fixed points of \(\mathcal{T}\) are the solutions of ABVP (1.2).

Our main result, based on the Schaefer fixed point theorem and Lemma 3.1, is stated as follows.

Theorem 3.1

Let \(f,g:[0,1]\times\mathbb{R}^{2}\rightarrow \mathbb{R}\) be continuous. Assume that

  1. (H)

    for \(\forall(u,v)\in\mathbb{R}^{2}\), \(t\in[0,1]\), there exist nonnegative functions \(a_{1},b_{1},c_{1},a_{2}, b_{2},c_{2}\in Z\) such that

    $$\begin{aligned}& \bigl\vert f(t,u,v)\bigr\vert \leq a_{1}(t)+b_{1}(t)|u|^{p-1}+c_{1}(t)|v|^{p-1}, \\& \bigl\vert g(t,u,v)\bigr\vert \leq a_{2}(t)+b_{2}(t)|u|^{p-1}+c_{2}(t)|v|^{p-1}. \end{aligned}$$

Then ABVP (1.2) has at least one solution, provided that

$$ L:=\frac{3\omega_{1}}{2\Gamma(\beta+1)}\cdot\frac{3\omega_{2}}{2\Gamma (\delta+1)} < 1, $$
(3.2)

where

$$\begin{aligned}& \omega_{1}= \frac{3^{p-1}\|b_{1}\|_{0}}{2^{p-1}(\Gamma(\gamma+1))^{p-1}}+\|c_{1}\|_{0}, \\& \omega_{2}= \frac{3^{p-1}\|b_{2}\|_{0}}{2^{p-1}(\Gamma(\alpha+1))^{p-1}}+\|c_{2}\|_{0}. \end{aligned}$$

Proof

The proof will be given in the following two steps.

Step 1: \(\mathcal{T}:X\times Y\rightarrow X\times Y\) is completely continuous.

By the definitions of \(T_{1}\) and \(T_{2}\), we obtain

$$\begin{aligned}& D_{0^{+}}^{\alpha}T_{1}v(t)=\phi_{q} \bigl(A_{1}(N_{1}v)+I_{0^{+}}^{\beta}N_{1}v\bigr) (t), \\& D_{0^{+}}^{\gamma}T_{2}u(t)=\phi_{q} \bigl(A_{2}(N_{2}u)+I_{0^{+}}^{\delta}N_{2}u\bigr) (t). \end{aligned}$$

Obviously, the operators \(T_{1}\), \(D_{0^{+}}^{\alpha}T_{1}\), \(T_{2}\), \(D_{0^{+}}^{\gamma}T_{2}\) are compositions of the continuous operators. So \(T_{1}\), \(D_{0^{+}}^{\alpha}T_{1}\), \(T_{2}\), \(D_{0^{+}}^{\gamma}T_{2}\) are continuous in Z. Hence, \(\mathcal{T}\) is a continuous operator in \(X \times Y\).

Let \(\Omega:=\Omega_{1}\times\Omega_{2}\subset X\times Y\) be an open bounded set, then \(T_{1}(\overline{\Omega_{2}})\), \(T_{2}(\overline{\Omega _{1}})\), and \(D_{0^{+}}^{\alpha}T_{1}(\overline{\Omega_{2}})\), \(D_{0^{+}}^{\gamma}T_{2}(\overline{\Omega_{1}})\) are bounded. Moreover, for \(\forall(u,v)\in \overline{\Omega}\), \(t\in[0,1]\), there exist constants \(L_{1},L_{2},L_{3}>0\) such that

$$\begin{aligned}& \bigl\vert A_{1}(N_{1}v)+I_{0^{+}}^{\beta}N_{1}v(t)\bigr\vert \leq L_{1}, \\& \bigl\vert A_{2}(N_{2}u)+I_{0^{+}}^{\delta}N_{2}u(t)\bigr\vert \leq L_{2}, \\& \max\bigl\{ \bigl\vert I_{0^{+}}^{\beta}N_{1}v(t)\bigr\vert ,\bigl\vert I_{0^{+}}^{\delta}N_{2}u(t)\bigr\vert \bigr\} \leq L_{3}. \end{aligned}$$

Thus, in view of the Arzelà-Ascoli theorem, we need only to prove that \(\mathcal{T}(\overline{\Omega})\subset X\times Y\) is equicontinuous.

For \(0\leq t_{1}< t_{2}\leq1\), \((u,v)\in\overline{\Omega}\), we have

$$\begin{aligned}& \bigl\vert T_{1}v(t_{2})-T_{1}v(t_{1}) \bigr\vert \\& \quad = \frac{1}{\Gamma(\alpha)} \biggl\vert \int_{0}^{t_{2}}(t_{2}-s)^{\alpha-1} \phi_{q}\bigl(A_{1}(N_{1}v)+I_{0^{+}}^{\beta}N_{1}v(s)\bigr)\,ds \\& \qquad {}-\int_{0}^{t_{1}}(t_{1}-s)^{\alpha-1} \phi_{q}\bigl(A_{1}(N_{1}v)+I_{0^{+}}^{\beta}N_{1}v(s)\bigr)\,ds\biggr\vert \\& \quad \leq \frac{L_{1}^{q-1}}{\Gamma(\alpha)} \biggl\{ \int_{0}^{t_{1}} \bigl[(t_{1}-s)^{\alpha-1}-(t_{2}-s)^{\alpha-1}\bigr] \,ds +\int_{t_{1}}^{t_{2}}(t_{2}-s)^{\alpha-1} \,ds \biggr\} \\& \quad = \frac{L_{1}^{q-1}}{\Gamma(\alpha+1)}\bigl[t_{1}^{\alpha}-t_{2}^{\alpha} +2(t_{2}-t_{1})^{\alpha}\bigr]. \end{aligned}$$

Similarly, one has

$$ \bigl\vert T_{2}u(t_{2})-T_{2}u(t_{1}) \bigr\vert \leq\frac{L_{2}^{q-1}}{\Gamma(\gamma+1)}\bigl[t_{1}^{\gamma}-t_{2}^{\gamma} +2(t_{2}-t_{1})^{\gamma}\bigr]. $$

Since \(t^{\alpha}\) is uniformly continuous in \([0,1]\), we see that \((T_{1}(\overline{\Omega_{2}}),T_{2}(\overline{\Omega_{1}}))\subset Z\times Z\) is equicontinuous. A similar proof can show that \((I_{0^{+}}^{\beta}N_{1}(\overline{\Omega_{2}}),I_{0^{+}}^{\delta}N_{2}(\overline{\Omega _{1}}))\subset Z\times Z\) is equicontinuous. This, together with the uniformly continuity of \(\phi_{q}(s)\) on \([-L_{3},L_{3}]\), shows that \((D_{0^{+}}^{\alpha}T_{1}(\overline{\Omega_{2}}), D_{0^{+}}^{\gamma }T_{2}(\overline{\Omega_{1}}))\subset Z\times Z\) is also equicontinuous. Thus, we find that \(\mathcal{T}:X\times Y\rightarrow X\times Y\) is compact.

Step 2: A priori bounds.

Set

$$ \Omega=\bigl\{ (u,v)\in X\times Y|(u,v)=\lambda^{q-1}\mathcal{T}(u,v), \lambda\in(0,1)\bigr\} . $$

Now it remains to show that the set Ω is bounded.

Since \(0<\alpha\leq1\), by Lemma 2.1, we have

$$ I_{0^{+}}^{\alpha}D_{0^{+}}^{\alpha}u(t)=u(t)+c_{0}. $$

So we get

$$ c_{0}=-u(0)=I_{0^{+}}^{\alpha}D_{0^{+}}^{\alpha}u(1)-u(1). $$

Hence, from the anti-periodic boundary value condition \(u(0)=-u(1)\), one has

$$ c_{0}=\frac{1}{2}I_{0+}^{\alpha}D_{0^{+}}^{\alpha}u(1). $$

Thus we obtain

$$ u(t) =-\frac{1}{2}I_{0+}^{\alpha}D_{0^{+}}^{\alpha}u(1)+I_{0+}^{\alpha}D_{0^{+}}^{\alpha}u(t), $$

which together with

$$\begin{aligned} \bigl\vert I_{0^{+}}^{\alpha}D_{0^{+}}^{\alpha}u(t)\bigr\vert =&\frac{1}{\Gamma(\alpha)}\biggl\vert \int_{0}^{t}(t-s)^{\alpha-1}D_{0^{+}}^{\alpha}u(s)\,ds\biggr\vert \\ \leq&\frac{1}{\Gamma(\alpha)}\bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0}\cdot\frac{1}{\alpha }t^{\alpha}\\ \leq&\frac{1}{\Gamma(\alpha+1)}\bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0}, \quad \forall t\in[0,1], \end{aligned}$$

yields

$$ \|u\|_{0}\leq\frac{3}{2\Gamma(\alpha+1)}\bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0}. $$
(3.3)

Similarly, we can get

$$ \|v\|_{0}\leq\frac{3}{2\Gamma(\gamma+1)}\bigl\Vert D_{0^{+}}^{\gamma}v\bigr\Vert _{0}. $$
(3.4)

For \((u,v)\in\Omega\), we have

$$\begin{aligned}& u(t)=\lambda^{q-1}\bigl(B_{1}(N_{1}v)+I_{0^{+}}^{\alpha}\phi _{q}\bigl(A_{1}(N_{1}v)+I_{0^{+}}^{\beta}N_{1}v\bigr) (t)\bigr), \\& v(t)=\lambda^{q-1}\bigl(B_{2}(N_{2}u)+I_{0^{+}}^{\gamma}\phi _{q}\bigl(A_{2}(N_{2}u)+I_{0^{+}}^{\delta}N_{2}u\bigr) (t)\bigr). \end{aligned}$$

Thus we get

$$\begin{aligned}& D_{0^{+}}^{\alpha}u(t)=\lambda^{q-1}\phi_{q} \bigl(A_{1}(N_{1}v)+I_{0^{+}}^{\beta}N_{1}v(t)\bigr), \\& D_{0^{+}}^{\gamma}v(t)=\lambda^{q-1}\phi_{q} \bigl(A_{2}(N_{2}u)+I_{0^{+}}^{\delta}N_{2}u(t)\bigr), \end{aligned}$$

which together with \(\phi_{q}(\lambda)=\lambda^{q-1}\) (\(\lambda\in(0,1)\)) yields

$$\begin{aligned}& \phi_{p}\bigl(D_{0^{+}}^{\alpha}u(t)\bigr) =\lambda \bigl(A_{1}(N_{1}v)+I_{0^{+}}^{\beta}N_{1}v(t)\bigr), \\& \phi_{p}\bigl(D_{0^{+}}^{\gamma}v(t)\bigr) =\lambda \bigl(A_{2}(N_{2}u)+I_{0^{+}}^{\delta}N_{2}u(t)\bigr). \end{aligned}$$

From the hypothesis (H), for \(\forall t\in[0,1]\), we get

$$\begin{aligned} \bigl\vert I_{0^{+}}^{\beta}N_{1}v(t)\bigr\vert \leq&\frac{1}{\Gamma(\beta)}\int_{0}^{t}(t-s)^{\beta -1} \bigl\vert f\bigl(s,v(s),D_{0^{+}}^{\gamma}v(s)\bigr)\bigr\vert \,ds \\ \leq&\frac{1}{\Gamma(\beta)}\bigl(\|a_{1}\|_{0}+ \|b_{1}\|_{0}\|v\|_{0}^{p-1} + \|c_{1}\|_{0}\bigl\Vert D_{0^{+}}^{\gamma}v \bigr\Vert _{0}^{p-1}\bigr)\cdot\frac{1}{\beta}t^{\beta}\\ \leq&\frac{1}{\Gamma(\beta+1)}\bigl(\|a_{1}\|_{0}+ \|b_{1}\|_{0}\|v\|_{0}^{p-1} + \|c_{1}\|_{0}\bigl\Vert D_{0^{+}}^{\gamma}v \bigr\Vert _{0}^{p-1}\bigr), \end{aligned}$$

which together with \(|\phi_{p}(D_{0^{+}}^{\alpha}u(t))|=|D_{0^{+}}^{\alpha}u(t)|^{p-1}\) yields

$$ \bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0}^{p-1} \leq\frac{3}{2\Gamma(\beta+1)}\bigl(\|a_{1} \|_{0}+\|b_{1}\|_{0}\|v\|_{0}^{p-1} +\|c_{1}\|_{0}\bigl\Vert D_{0^{+}}^{\gamma}v \bigr\Vert _{0}^{p-1}\bigr). $$
(3.5)

Repeating arguments similar to the above we can arrive at

$$ \bigl\Vert D_{0^{+}}^{\gamma}v\bigr\Vert _{0}^{p-1} \leq\frac{3}{2\Gamma(\delta+1)}\bigl(\|a_{2} \|_{0}+\|b_{2}\|_{0}\|u\|_{0}^{p-1} +\|c_{2}\|_{0}\bigl\Vert D_{0^{+}}^{\alpha}u \bigr\Vert _{0}^{p-1}\bigr). $$
(3.6)

From (3.3)-(3.6), we obtain

$$\begin{aligned}& \bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0}^{p-1} \leq \frac{3}{2\Gamma(\beta+1)}\biggl[\Vert a_{1}\Vert _{0} + \biggl(\Vert c_{1}\Vert _{0} \\& \hphantom{\bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0}^{p-1} \leq {}}{}+\frac{3^{p-1}\Vert b_{1}\Vert _{0}}{2^{p-1}(\Gamma(\gamma +1))^{p-1}}\biggr)\bigl\Vert D_{0^{+}}^{\gamma}v\bigr\Vert _{0}^{p-1}\biggr] \\& \hphantom{\bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0}^{p-1} } = \frac{3}{2\Gamma(\beta+1)}\bigl(\Vert a_{1}\Vert _{0} +\omega_{1}\bigl\Vert D_{0^{+}}^{\gamma}v\bigr\Vert _{0}^{p-1}\bigr), \\& \bigl\Vert D_{0^{+}}^{\gamma}v\bigr\Vert _{0}^{p-1} \leq \frac{3}{2\Gamma(\delta+1)}\biggl[\Vert a_{2}\Vert _{0} + \biggl(\Vert c_{2}\Vert _{0} \\& \hphantom{\bigl\Vert D_{0^{+}}^{\gamma}v\bigr\Vert _{0}^{p-1} \leq{}}{} +\frac{3^{p-1}\Vert b_{2}\Vert _{0}}{2^{p-1}(\Gamma(\alpha +1))^{p-1}}\biggr)\bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0}^{p-1}\biggr] \\& \hphantom{\bigl\Vert D_{0^{+}}^{\gamma}v\bigr\Vert _{0}^{p-1} } = \frac{3}{2\Gamma(\delta+1)}\bigl(\Vert a_{2}\Vert _{0} +\omega_{2}\bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0}^{p-1}\bigr). \end{aligned}$$

So we have

$$\begin{aligned}& \bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0}^{p-1} \leq\frac{3}{2\Gamma(\beta+1)} \biggl(\Vert a_{1}\Vert _{0} + \frac{3\omega_{1}}{2\Gamma(\delta+1)}\bigl(\Vert a_{2}\Vert _{0}+ \omega_{2}\bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0}^{p-1}\bigr) \biggr), \\& \bigl\Vert D_{0^{+}}^{\gamma}v\bigr\Vert _{0}^{p-1} \leq\frac{3}{2\Gamma(\delta+1)} \biggl(\Vert a_{2}\Vert _{0} + \frac{3\omega_{2}}{2\Gamma(\beta+1)}\bigl(\Vert a_{1}\Vert _{0}+ \omega_{1}\bigl\Vert D_{0^{+}}^{\gamma}v\bigr\Vert _{0}^{p-1}\bigr) \biggr). \end{aligned}$$

Hence, in view of (3.2), we can get

$$\begin{aligned}& \bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0} \leq \biggl(\frac{M_{1}}{1-L} \biggr)^{q-1}:=L_{11}, \end{aligned}$$
(3.7)
$$\begin{aligned}& \bigl\Vert D_{0^{+}}^{\gamma}v\bigr\Vert _{0} \leq \biggl(\frac{M_{2}}{1-L} \biggr)^{q-1}:=L_{21}, \end{aligned}$$
(3.8)

where

$$\begin{aligned}& M_{1}= \frac{3}{2\Gamma(\beta+1)} \biggl(\|a_{1}\|_{0} + \frac{3\omega_{1}}{2\Gamma(\delta+1)}\|a_{2}\|_{0} \biggr), \\& M_{2}= \frac{3}{2\Gamma(\delta+1)} \biggl(\|a_{2}\|_{0} + \frac{3\omega_{2}}{2\Gamma(\beta+1)}\|a_{1}\|_{0} \biggr). \end{aligned}$$

Thus, from (3.3) and (3.4), one has

$$\begin{aligned}& \|u\|_{0}\leq\frac{3L_{11}}{2\Gamma(\alpha+1)}:=L_{12}, \end{aligned}$$
(3.9)
$$\begin{aligned}& \|v\|_{0}\leq\frac{3L_{21}}{2\Gamma(\gamma+1)}:=L_{22}. \end{aligned}$$
(3.10)

Therefore, combining (3.7) and (3.9) with (3.8) and (3.10), we have

$$\begin{aligned} \bigl\Vert (u,v)\bigr\Vert _{X\times Y} =&\max\bigl\{ \|u \|_{0},\bigl\Vert D_{0^{+}}^{\alpha}u\bigr\Vert _{0},\|v\|_{0},\bigl\Vert D_{0^{+}}^{\gamma}v\bigr\Vert _{0}\bigr\} \\ \leq&\max\{L_{11},L_{12},L_{21},L_{22} \}. \end{aligned}$$

As a consequence of the Schaefer fixed point theorem, we deduce that \(\mathcal{T}\) has at least one fixed point which is the solution of ABVP (1.2). The proof is complete. □

4 An example

In this section, we will give an example to illustrate our main result.

Example 4.1

Consider the following ABVP for the coupled system of the fractional p-Laplacian equation:

$$ \left \{ \textstyle\begin{array}{l} D_{0^{+}}^{\frac{3}{4}}\phi_{3}(D_{0^{+}}^{\frac{1}{2}}u(t)) =-\frac{25}{3}+\frac{1}{10}v^{2}(t)+te^{-|D_{0^{+}}^{\frac{3}{4}}v(t)|},\quad t\in[0,1], \\ D_{0^{+}}^{\frac{1}{2}}\phi_{3}(D_{0^{+}}^{\frac{3}{4}}v(t)) =\cos t+\frac{1}{4}u^{2}(t)+t\cos{(D_{0^{+}}^{\frac{1}{2}}u(t))}, \quad t\in [0,1], \\ u(0)=-u(1), \qquad D_{0^{+}}^{\frac{1}{2}} u(0)=-D_{0^{+}}^{\frac{1}{2}} u(1), \\ v(0)=-v(1),\qquad D_{0^{+}}^{\frac{3}{4}} v(0)=-D_{0^{+}}^{\frac{3}{4}} v(1). \end{array}\displaystyle \right . $$
(4.1)

Corresponding to ABVP (1.2), we get \(p=3\), \(\alpha=1/2\), \(\beta =3/4\), \(\gamma=3/4\), \(\delta=1/2\), and

$$\begin{aligned}& f(t,u,v)=-\frac{25}{3}+\frac{1}{10}u^{2}+te^{-|v|}, \\& g(t,u,v)=\cos t+\frac{1}{4}u^{2}+t\cos v. \end{aligned}$$

Choose \(a_{1}(t)=10\), \(b_{1}(t)=1/10\), \(c_{1}(t)=0\), \(a_{2}(t)=2\), \(b_{2}(t)=1/4\), \(c_{2}(t)=0\). By a simple calculation, we obtain \(\|b_{1}\|_{0}=1/10\), \(\|c_{1}\|_{0}=0\), \(\|b_{2}\| _{0}=1/4\), \(\|c_{2}\|_{0}=0\), and

$$\begin{aligned}& \omega_{1}= \frac{3^{2}}{2^{2} (\Gamma(\frac{3}{4}+1) )^{2}}\times\frac {1}{10}+0\leq0.266374, \\& \omega_{2}= \frac{3^{2}}{2^{2} (\Gamma(\frac{1}{2}+1) )^{2}}\times\frac {1}{4}+0\leq0.716197, \\& L=\frac{3}{2}\frac{\omega_{1}}{\Gamma(\frac{3}{4}+1)} \frac{3}{2}\frac{\omega_{2}}{\Gamma(\frac{1}{2}+1)}< 1. \end{aligned}$$

Obviously, ABVP (4.1) satisfies all assumptions of Theorem 3.1. Hence, ABVP (4.1) has at least one solution.