1 Introduction

The purpose of this paper is to establish the existence and uniqueness of solutions for the following system of fractional differential equations with Riemann–Stieltjes integral boundary conditions (for short, FBVP):

$$ \textstyle\begin{cases} D^{\alpha}_{0+}u(t)+h(t)f(t,v(t))=0,\quad 0< t< 1, \\ D^{\alpha}_{0+}v(t)+h(t)g(t,u(t))=0,\quad 0< t< 1, \\ u(0)=u'(0)=0, \qquad u(1)=\int_{0}^{1}u(\tau)\,d\beta(\tau),\\ v(0)=v'(0)=0, \qquad v(1)=\int_{0}^{1}v(\tau)\,d\beta(\tau), \end{cases} $$
(1.1)

where \(2<\alpha\leq3\) is a real number, \(D^{\alpha}_{0+}\) is the standard Riemann–Liouville differentiation, β is right continuous on \([0,1)\), left continuous at \(t=1\), and nondecreasing on \([0,1]\) with \(\beta(0)=0\), \(\int_{0}^{1}u(\tau)\,d\beta(\tau)\) denotes the Riemann–Stieltjes integral of u with respect to β. Here the nonlinear terms \(f,g:[0,1]\times(-\infty,+\infty)\to(-\infty,+\infty)\) are continuous sign-changing functions and f, g may be unbounded from below, \(h:(0,1)\to[0,+\infty)\) with \(0<\int_{0}^{1}h(s)\,ds<+\infty\) is continuous and is allowed to be singular at \(t=0,1\).

Fractional differential equations play an important role in many engineering and scientific disciplines such as physics, chemistry, aerodynamics, electrodynamics of complex medium, polymer rheology, diffusive transport akin to diffusion, probability, electrical networks, etc. For details, see [13] and the references therein. By using a nonlinear alternative of Leray–Schauder theorem and Krasnoselskii’s fixed point theorem in a cone, Bai and Fang in [4] obtained the existence of positive solutions for the following singular coupled system of nonlinear fractional differential equations:

$$ \textstyle\begin{cases} D^{s} u= f(t,v),& 0< t< 1,\\ D^{p} v=g(t,u),& 0< t< 1, \end{cases} $$

where \(0 < s < 1\), \(0 < p < 1\), \(D^{s}\), \(D^{p}\) are two standard Riemann–Liouville fractional derivatives, \(f , g : (0, 1] \times [0,+ \infty)\to[0,+\infty)\) are two given continuous functions. Su [5] established sufficient conditions for the existence of solutions for the following coupled system of fractional differential equations with two-point boundary conditions:

$$ \textstyle\begin{cases} D^{\alpha}u(t)=f (t, v(t), D^{\mu}v(t)),\quad 0< t< 1,\\ D^{\beta}v(t)=g (t, u(t), D^{\nu}u(t)),\quad 0< t< 1, \\ u(0)=u(1)=v(0)= v(1)=0, \end{cases} $$

where \(1 <\alpha,\beta< 2\), \(\mu,\nu> 0\), \(\alpha-\nu\geq1\), \(\beta-\mu\geq1\), \(f , g: [0, 1] \times R\times R \to R\) are given functions, and D is the standard Riemann–Liouville fractional derivative. Ahmad and Nieto [6] extended the results of [5] to a three-point boundary value problem for the following coupled system of fractional differential equations:

$$ \textstyle\begin{cases} D^{\alpha}u(t)=f (t, v(t), D^{\mu}v(t)),\quad0< t< 1, \\ D^{\beta}v(t)=g (t, u(t), D^{\nu}u(t)),\quad0< t< 1, \\ u(0)=0,\qquad u(1)=\gamma u(\eta),\\ v(0)=0,\qquad v(1)=\gamma v(\eta), \end{cases} $$

where \(1 <\alpha,\beta< 2\), \(\mu,\nu,\gamma> 0\), \(\alpha-\nu\geq1\), \(\beta-\mu\geq1\), \(0<\eta<1\), \(\gamma\eta^{\alpha -1}<1\), \(\gamma\eta^{\beta-1}<1\) \(f , g: [0, 1] \times R\times R \to R\) are given functions, and D is the standard Riemann–Liouville fractional derivative. Yang [7] established sufficient conditions for the existence and nonexistence of positive solutions to boundary values problem for a coupled system of nonlinear fractional differential equations as follows:

$$ \textstyle\begin{cases} D^{\alpha}u(t)+a(t)f (t, v(t))=0,\quad0< t< 1, \\ D^{\beta}v(t)+b(t)g (t, u(t))=0,\quad0< t< 1, \\ u(0)=0,\qquad u(1)=\int_{0}^{1}\phi(t)u(t)\,dt,\\ v(0)=0, \qquad v(1)=\int_{0}^{1}\psi(t)v(t)\,dt, \end{cases} $$

where \(1 <\alpha,\beta\leq2\), \(a,b\in C((0,1),[0,+\infty))\), \(\phi, \psi\in L^{1}[0,1]\) are nonnegative and \(f,g\in C([0,1]\times[0,+\infty),[0,+\infty))\), and D is the standard Riemann–Liouville fractional derivative.

Inspired by the above papers and some known results on fractional differential equations with integral boundary conditions [830], this paper is to establish the existence and uniqueness of nontrivial solutions to FBVP (1.1) under the conditions that the nonlinear terms f, g of FBVP (1.1) are allowed to be sign-changing and unbounded from below. Finally, it is worth mentioning that the main technique used here is the topological degree theory, the theory of linear operators. As far as we know, there are few works that deal with system of fractional differential equations with Riemann–Stieltjes integral conditions where the nonlinear terms may be unbounded from below. The main results here are different from [432, 3537].

2 Preliminaries and lemmas

Let \(E=C[0,1]\) be a Banach space with the norm \(\|u\|=\max_{0\leq t\leq1}|u(t)|\) for \(u\in E\). Let \(P= \{u\in E\mid u(t)\geq0, t\in[0,1] \}\). Then P is a total cone in E, that is, \(E=\overline{P-P}\). Let \(P^{*}= \{g\in E^{*}\mid g(u)\geq0 \mbox{ for all }u\in P \}\). Then \(P^{*}\) is the dual cone of P. Let \(E^{*}\) denote the dual space of E, then by Riesz representation theorem, \(E^{*}\) is given by

$$\begin{aligned} E^{*}&= \bigl\{ \nu\mid\nu\mbox{ is right continuous on } [0,1) \mbox{ and is bounded variation on } \\ &\quad [0,1] \mbox{ with } \nu(0)=0 \bigr\} . \end{aligned} $$

Let \(E^{2}\) be equipped with the norm \(\|(u,v)\|_{1}=\|u\|+\|v\|\). Then \(E^{2}\) is also a real Banach space and \(P^{2}=P\times P\) is a cone in \(E^{2}\). Let \((u_{1},v_{1})\geq(u_{2},v_{2})\) denote \(u_{1}\geq u_{2}\), \(v_{1}\geq v_{2}\) for \((u_{1},v_{1}), (u_{2},v_{2})\in E^{2}\) and \(B_{r}=\{(u,v)\in E ^{2}\mid \|(u,v)\|_{1}< r\}\) for any \(r>0\).

Definition 2.1

The Riemann–Liouville fractional integral of order \(\alpha>0\) of a function \(y : (0,+\infty) \to \mathbb{R}\) is given by

$$I^{\alpha}_{0+}y(t)=\frac{1}{\Gamma(\alpha)} \int_{0}^{t} (t-s)^{\alpha-1}y(s)\,ds, $$

provided the right-hand side is defined on \((0,+\infty)\) pointwisely.

Definition 2.2

The Riemann–Liouville fractional derivative of order \(\alpha>0\) of a function \(y : (0,+\infty) \to\mathbb{R}\) is given by

$$D^{\alpha}_{0+}y(t)=\frac{1}{\Gamma(n-\alpha)} \biggl(\frac{d}{dt} \biggr)^{n} \int_{0}^{t} \frac{y(s)}{(t-s)^{\alpha-n+1}}\,ds, $$

where \(n=[\alpha]+1\), \([\alpha]\) denotes the integer part of the number α, provided that the right-hand side is defined on \((0,+\infty)\) pointwisely.

Lemma 2.1

Let \(\alpha> 0\). If we assume \(u \in C(0,1)\cap L(0,1)\), then the fractional differential equation

$$D^{\alpha}_{0+}u(t)=0 $$

has \(u(t)=c_{1}t^{\alpha-1}+c_{2}t^{\alpha-2}+\cdots+ c_{N}t^{\alpha-N}\), \(c_{i}\in\mathbb{R}\), \(i=1,2,\ldots, N\), as unique solutions, where N is the smallest integer greater than or equal to α.

Lemma 2.2

Assume that \(u \in C(0,1)\cap L(0,1)\) with a fractional derivative of order \(\alpha> 0\) that belongs to \(C(0,1)\cap L(0,1)\). Then

$$I^{\alpha}_{0+}D^{\alpha}_{0+}u(t)=u(t)+c_{1}t^{\alpha-1}+c_{2}t^{\alpha -2}+ \cdots+ c_{N}t^{\alpha-N} $$

for some \(c_{i}\in\mathbb{R}\), \(i=1,2,\ldots, N\), N is the smallest integer greater than or equal to α.

Lemma 2.3

([31])

Given \(y\in L(0,1)\) and \(2<\alpha\leq3\), then the unique solution of

$$ \textstyle\begin{cases} D^{\alpha}_{0+}u(t)+y(t)=0,\quad0< t< 1,\\ u(0)=u'(0)=0,\quad u(1)=0, \end{cases} $$

is \(u(t)=\int_{0}^{1} G_{0}(t,s)y(s)\,ds\), where

$$G_{0}(t,s)= \textstyle\begin{cases} \frac{[t(1-s)]^{\alpha-1}-(t-s)^{\alpha-1}}{\Gamma(\alpha)},&0\leq s\leq t\leq1,\\ \frac{[t(1-s)]^{\alpha-1}}{\Gamma(\alpha)},&0\leq t\leq s\leq1. \end{cases} $$

Lemma 2.4

([31])

The Green function \(G_{0}(t,s)\) has the following properties:

  1. (1)

    \(\Gamma(\alpha)k(t)q(s)\leq G_{0}(t,s)\leq(\alpha-1)q(s)\) for \(t,s\in[0,1]\),

  2. (2)

    \(\Gamma(\alpha)k(t)q(s)\leq G_{0}(t,s)\leq(\alpha-1)k(t)\) for \(t,s\in[0,1]\),

where

$$k(t)=\frac{t^{\alpha-1}(1-t)}{\Gamma(\alpha)},\qquad q(s)=\frac{s(1-s)^{\alpha-1}}{\Gamma(\alpha)}. $$

By Lemma 2.1, the unique solution of the problem

$$ \textstyle\begin{cases} D^{\alpha}_{0+}u(t)=0,\quad0< t< 1,\\ u(0)=u'(0)=0,\qquad u(1)=1, \end{cases} $$

is \(u(t)=t^{\alpha-1}\). Then it is easy to verify, as a consequence of Lemma 2.3, that FBVP (1.1) is equivalent to the system of perturbed integral equations

$$ \textstyle\begin{cases} u(t)=\int_{0}^{1}G_{0}(t,s)h(s)f(s,v(s))\,ds+t^{\alpha-1}\int_{0}^{1} u(\tau)\,d\beta (\tau),\\ v(t)=\int_{0}^{1}G_{0}(t,s)h(s)g(s,u(s))\,ds+t^{\alpha-1}\int_{0}^{1} v(\tau)\,d\beta (\tau). \end{cases} $$
(2.1)

Define \(\Gamma=\int_{0}^{1} t^{\alpha-1}\,d\beta(t)\), \(g_{\beta}(s)=\int_{0}^{1}G_{0}(t,s)\,d\beta(t)\). Then we have the following lemma.

Lemma 2.5

Given \(y(t)\in C(0,1)\cap L(0,1)\) and \(2<\alpha\leq3\), then

$$ \textstyle\begin{cases} D^{\alpha}_{0+}u(t)+y(t)=0,\quad0< t< 1,\\ u(0)=u'(0)=0,\qquad u(1)=\int_{0}^{1}u(\tau)\,d\beta(\tau), \end{cases} $$

has the unique solution

$$u(t)= \int_{0}^{1} G(t,s)y(s)\,ds, $$

where the Green function \(G(t,s)\) is given by

$$ G(t,s)=\frac{t^{\alpha-1}}{1-\Gamma}g_{\beta}(s)+G_{0}(t,s), \quad t,s\in[0,1]. $$
(2.2)

Proof

Multiply (2.1) by \(d\beta(t)\) on both sides and integrate over \([0,1]\) to obtain

$$ \begin{aligned} \int_{0}^{1}u(t)\,d\beta(t)&= \int_{0}^{1} \int_{0}^{1}G_{0}(t,s)y(s)\,ds\,d \beta(t)+ \int _{0}^{1}t^{\alpha-1} \int_{0}^{1}u(\tau)\,d\beta(\tau)\,d\beta(t) \\ &= \int_{0}^{1} \int_{0}^{1}G_{0}(t,s)y(s)\,ds\,d \beta(t)+ \int_{0}^{1}t^{\alpha-1}\,d\beta (t) \int_{0}^{1}u(\tau)\,d\beta(\tau). \end{aligned} $$

Consequently,

$$ \int_{0}^{1}u(t)\,d\beta(t)=\frac{1}{1-\Gamma} \int_{0}^{1} \int _{0}^{1}G_{0}(t,s)y(s)\,ds\,d \beta(t). $$

Replacing \(\int_{0}^{1} u(\tau)\,d\beta(\tau)\) in (2.1) with the above equality, we obtain

$$ \begin{aligned} u(t)&= \int_{0}^{1}G_{0}(t,s)y(s)\,ds+ \frac{t^{\alpha-1}}{1-\Gamma} \int_{0}^{1} \biggl( \int_{0}^{1} G_{0}(t,s)\,d\beta(t) \biggr)y(s)\,ds \\ &= \int_{0}^{1} \biggl(G_{0}(t,s)+ \frac{t^{\alpha-1}}{1-\Gamma}g_{\beta}(s) \biggr)y(s)\,ds \\ &= \int_{0}^{1} G(t,s)y(s)\,ds. \end{aligned} $$

Reversely, if \(u(t)=\int_{0}^{1}G(t,s)y(s)\,ds\), then \(u(0)=0\) and \(u(1)=\int_{0}^{1} u(\tau)\,d\beta(\tau)\) via (2.1). According to Definition 2.2, Lemma 2.3, and Lemma 2.4, \(D^{\alpha}_{0+}u(t)+y(t)=0\) holds. □

By Lemma 2.5, \((u,v)\in E^{2}\) is a solution of FBVP (1.1) if and only if

$$ \textstyle\begin{cases} u(t)=\int_{0}^{1}G(t,s)h(s)f(s,v(s))\,ds,\\ v(t)=\int_{0}^{1}G(t,s)h(s)g(s,u(s))\,ds. \end{cases} $$

Define

$$\begin{aligned}& (A_{1}v) (t)= \int_{0}^{1}G(t,s)h(s)f \bigl(s,v(s) \bigr)\,ds, \\& (A_{2}u) (t)= \int_{0}^{1}G(t,s)h(s)g \bigl(s,u(s) \bigr)\,ds, \\& A(u,v) (t)= \bigl((A_{1}v) (t),(A_{2}u) (t) \bigr). \end{aligned}$$

It is easy to show that \(A:E^{2}\to E^{2}\) is a completely continuous nonlinear operator, and if \((u,v)\in E^{2}\) is a fixed point of A, then \((u,v)\) is a solution of FBVP (1.1) by Lemma 2.5.

For any \(u\in E\), define \(K:E\to E\) as follows:

$$ (Ku) (t)= \int_{0}^{1}G(t,s)h(s) u(s) \,ds,\quad u\in E. $$
(2.3)

Then \(K:E\to E\) is a completely continuous linear operator and \(K(P)\subset P\) holds. Since \(h\in C(0,1)\cap L(0,1)\) with \(\int_{0}^{1} h(t)\,dt>0\), by [32], the spectral radius \(r(K)\) of K is positive. The Krein–Rutman theorem [33] asserts that there exist \(\phi\in P\setminus\{0\}\) and \(\omega\in P^{*}\setminus\{0\}\) corresponding to the number \(\lambda_{1}=1/r(K)\) relative to K such that

$$ \lambda_{1}K\phi=\phi $$
(2.4)

and

$$ \lambda_{1}K^{*}\omega=\omega,\quad\omega(1)=1. $$
(2.5)

Here \(K^{*}:E^{*}\to E^{*}\) is the dual operator of K given by

$$ \bigl(K^{*}v \bigr) (s)= \int_{0}^{s} \int_{0}^{1}G(t,\tau)h(\tau)\,dv(t)\,d\tau, \quad v \in E^{*}. $$

Now we testify that \(K^{*}:E^{*}\to E^{*}\) is the dual operator of K. In fact,

$$\begin{aligned} \bigl\langle K^{*}v(s), u(s)\bigr\rangle &= \int_{0}^{1} u(s)\,dK^{*}v(s)= \int_{0}^{1}u(s) \int_{0}^{1} G(t,s)h(s)\,dv(t)\,ds \\ &= \int_{0}^{1} \biggl( \int_{0}^{1}G(t,s)h(s)u(s)\,ds \biggr)\,dv(t) \\ &= \int_{0}^{1}(Ku) (t)\,dv(t)=\bigl\langle v(t),(Ku) (t)\bigr\rangle . \end{aligned} $$

So \(K^{*}:E^{*}\to E^{*}\) is the dual operator of K.

The continuity of G, the integrability of h, and the representation of \(K^{*}\) induce that \(\omega\in C^{1}[0, 1]\). Let \(e(t):= \omega'(t)\). Then \(e\in P \setminus\{0\}\), and (2.5) can be rewritten equivalently as

$$ r(K)e(s) = \int_{0}^{1}G(t,s)h(s)e(t)\,dt, \quad \int_{0}^{1} e(t)\,dt=1. $$
(2.6)

Lemma 2.6

Let \(0\leq\Gamma=\int_{0}^{1} t^{\alpha-1}\,d\beta(t)<1\) and \(g_{\beta}(s)=\int_{0}^{1}G_{0}(t,s)\,d\beta(t)\geq0\) for \(s\in[0,1]\), then there exists \(\delta>0\) such that \(P_{0}=\{u\in P\mid \int_{0}^{1}u(t)e(t)\,dt\ge\delta\|u\|\}\) is a subcone of P and \(K(P)\subset P_{0}\).

Proof

Let \(\delta=\int_{0}^{1} \frac{(1-t)t^{\alpha-1}}{\alpha-1}e(t)\,dt\). It is obvious that \(G(t,s)>0\) holds for \(t,s\in(0,1)\). By Lemma 2.4 and (2.2),

$$ G(t,s)=G_{0}(t,s)+\frac{t^{\alpha-1}}{1-\Gamma}g_{\beta}(s)\leq G_{0}(t,s)+\frac{1}{1-\Gamma}g_{\beta}(s)\leq \frac{s(1-s)^{\alpha-1}}{\Gamma(\alpha-1)}+\frac{g_{\beta}(s)}{1-\Gamma} $$

and

$$ \begin{aligned} G(t,s)&=G_{0}(t,s)+\frac{t^{\alpha-1}}{1-\Gamma}g_{\beta}(s)\geq \frac{(1-t)s[t(1-s)]^{\alpha-1}}{(\alpha-1)\Gamma(\alpha-1)}+\frac {(1-t)t^{\alpha-1}g_{\beta}(s)}{(\alpha-1)(1-\Gamma)}\\ & =\frac{(1-t)t^{\alpha-1}}{\alpha-1} \biggl( \frac{s(1-s)^{\alpha-1}}{\Gamma (\alpha-1)}+\frac{g_{\beta}(s)}{1-\Gamma} \biggr). \end{aligned} $$

For any \(u\in P\),

$$ \max_{t\in [0,1]} \bigl\vert (Ku) (t) \bigr\vert =\max _{t\in[0,1]} \biggl\vert \int_{0}^{1}G(t,s)h(s)u(s)\,ds \biggr\vert \leq \int_{0}^{1} \biggl(\frac{s(1-s)^{\alpha-1}}{\Gamma(\alpha-1)}+ \frac{g_{\beta}(s)}{1-\Gamma} \biggr)h(s)u(s)\,ds. $$

Then we have

$$ \begin{aligned} \int_{0}^{1} (Ku) (t)e(t)\,dt&= \int_{0}^{1} \int_{0}^{1}G(t,s)h(s)u(s)\,ds e(t)\,dt \\ &\geq \int_{0}^{1} \int_{0}^{1}\frac{(1-t)t^{\alpha-1}}{\alpha-1} \biggl( \frac {s(1-s)^{\alpha-1}}{\Gamma(\alpha-1)}+\frac{g_{\beta}(s)}{1-\Gamma} \biggr)h(s)u(s)\,ds e(t)\,dt \\ &= \int_{0}^{1}\frac{(1-t)t^{\alpha-1}}{\alpha-1}e(t)\,dt\cdot \int_{0}^{1} \biggl(\frac{s(1-s)^{\alpha-1}}{\Gamma(\alpha-1)}+ \frac{g_{\beta}(s)}{1-\Gamma} \biggr)h(s)u(s)\,ds \\ &\geq \int_{0}^{1}\frac{(1-t)t^{\alpha-1}}{\alpha-1}e(t)\,dt\cdot \Vert Ku \Vert \\ &=\delta \Vert Ku \Vert . \end{aligned} $$

That means \(K(P)\subset P_{0}\). □

Lemma 2.7

([34])

Let E be a real Banach space and \(\Omega\subset E \) be a bounded open set with \(0\in\Omega\). Suppose that \(A: \bar{\Omega}\to E\) is a completely continuous operator. (1) If there is \(y_{0}\in E\) with \(y_{0} \neq0\) such that \(u\neq Au + \mu y_{0}\), for all \(u\in\partial\Omega\) and \(\mu\geq0\), then \(\operatorname{deg}(I-A,\Omega,0) = 0\). (2) If \(Au\neq\mu u\) for all \(u\in \partial\Omega\) and \(\mu\geq1\), then \(\operatorname{deg}(I-A,\Omega,0) = 1 \). Here deg stands for the Leray–Schauder topological degree in E.

Lemma 2.8

Assume that the following assumptions are satisfied:

(C1):

There exist \(\phi\in P\setminus\{0\}\), \(\omega\in P^{*}\setminus\{0\}\) such that (2.4), (2.5) hold and K maps P into \(P_{0}\).

(C2):

There exists a continuous operator \(H:E\to P \) such that

$$\lim_{ \Vert u \Vert + \Vert v \Vert \to+\infty}\frac{ \Vert Hu \Vert + \Vert Hv \Vert }{ \Vert u \Vert + \Vert v \Vert }=0. $$
(C3):

There exist two bounded continuous operators \(F,G:E\to E\) and \(u_{0},v_{0}\in E\) such that \((Fv+v_{0}+Hv, Gu+u_{0}+Hu)\in P^{2}\) for all \((u, v)\in E^{2}\).

(C4):

There exist \(m_{0},n_{0}\in E\) and \(\zeta>0\) such that \((KFv,KGu)\geq(\lambda_{1}(1+\zeta)Kv-KHv-m_{0}, \lambda_{1}(1+\zeta)Ku-KHu-n_{0})\) for all \((u, v)\in E^{2}\).

Let \(A_{1}=KF\), \(A_{2}=KG\), \(A(u,v)(t)=((A_{1}v)(t), (A_{2}u)(t))\), then there exists \(R>0\) such that

$$\operatorname{deg}(I-A,B_{R},0)=0, $$

where \(B_{r}= \{(u, v)\in E^{2}\mid\|(u,v)\|_{1}< r \}\) for any \(r>0\).

Proof

Choose a constant \(l_{0}=(\delta\lambda_{1})^{-1}(1+\zeta^{-1})+\|K\|>0\). By (C2), for \(0<\varepsilon_{0}<l_{0}^{-1}\), there exists \(R_{1}>0\) such that \(\|u\|+\|v\|>R_{1}\) implies

$$ \Vert Hu \Vert + \Vert Hv \Vert < \varepsilon_{0} \bigl( \Vert u \Vert + \Vert v \Vert \bigr). $$
(2.7)

Now we shall show

$$ (u,v)\neq A(u,v)+\mu(\phi,\phi)\quad\mbox{for any } (u,v)\in \partial B_{R} \mbox{ and } \mu\geq0, $$
(2.8)

provided that R is sufficiently large.

In fact, if (2.8) is not true, then there exist \((u_{1},v_{1})\in \partial B_{R}\) and \(\mu_{1}\geq0\) satisfying

$$ (u,v)=A(u,v)+\mu(\phi,\phi), $$
(2.9)

that is,

$$ (u_{1},v_{1})=(KFv_{1}+ \mu_{1} \phi, KGu_{1}+\mu_{1} \phi). $$
(2.10)

Since \(\phi\in P\setminus\{0\}\), \(e(t)\in P\setminus\{0\}\), \(\int_{0}^{1}\phi(t)e(t)\,dt>0\). Multiply (2.10) by \(e(t)\) on both sides and integrate respectively on \([0,1]\). Then by (C4), (2.6), we get

$$ \begin{aligned} \int_{0}^{1} u_{1}(t)e(t)\,dt&= \int_{0}^{1}(KFv_{1}) (t)e(t)\,dt+ \mu_{1} \int_{0}^{1}\phi (t)e(t)\,dt \\ &\geq \lambda_{1} (1+\zeta) \int_{0}^{1} \int_{0}^{1}G(t,s)h(s)v_{1}(s)dse(t)\,dt\\ &\quad {}- \int _{0}^{1}(KHv_{1}) (t)e(t)\,dt- \int_{0}^{1}m_{0}(t)e(t)\,dt \\ &=\lambda_{1}(1+\zeta) \int_{0}^{1} \int_{0}^{1}G(t,s)h(s)v_{1}(s)e(t)\,ds \,dt\\ &\quad {}- \int _{0}^{1} \int_{0}^{1}G(t,s)h(s) (Hv_{1}) (s)e(t) \,ds\,dt- \int_{0}^{1}m_{0}(t)e(t)\,dt \\ &=\lambda_{1}(1+\zeta) \int_{0}^{1} \biggl[ \int_{0}^{1}G(t,s)h(s)e(t)\,dt \biggr]v_{1}(s) \,ds\\ &\quad {}- \int_{0}^{1} \biggl[ \int_{0}^{1}G(t,s)h(s)e(t)\,dt \biggr](Hv_{1}) (s)\,ds- \int _{0}^{1}m_{0}(t)e(t)\,dt \\ &=\lambda_{1}(1+\zeta)r(K) \int_{0}^{1}e(s)v_{1}(s)\,ds\\ &\quad {}-r(K) \int _{0}^{1}(Hv_{1}) (s)e(s)\,ds- \int_{0}^{1}m_{0}(t)e(t)\,dt \\ &=(1+\zeta) \int_{0}^{1}v_{1}(t)e(t)\,dt\\ &\quad {}-r(K) \int_{0}^{1}(Hv_{1}) (t)e(t) \,dt- \int_{0}^{1}m_{0}(t)e(t)\,dt, \end{aligned} $$

and

$$ \begin{aligned} \int_{0}^{1}v_{1}(t)e(t)\,dt&= \int_{0}^{1}(KGu_{1}) (t)e(t)\,dt+ \mu_{1} \int _{0}^{1}\phi(t)e(t)\,dt \\ &\geq(1+\zeta) \int_{0}^{1}u_{1}(t)e(t)\,dt-r(K) \int_{0}^{1}(Hu_{1}) (t)e(t)\,dt- \int _{0}^{1}n_{0}(t)e(t)\,dt. \end{aligned} $$

According to the two above inequalities, we obtain

$$ \begin{aligned} \int_{0}^{1} \bigl[u_{1}(t)+v_{1}(t) \bigr]e(t)\,dt&\geq (1+\zeta) \int_{0}^{1} \bigl[u_{1}(t)+v_{1}(t) \bigr]e(t)\,dt\\ &\quad {}-r(K) \int_{0}^{1} \bigl[(Hu_{1}) (t)+(Hv_{1}) (t) \bigr]e(t)\,dt\\ &\quad {} - \int_{0}^{1} \bigl[m_{0}(t)+n_{0}(t) \bigr]e(t)\,dt. \end{aligned} $$

Then we derive

$$ \begin{aligned}[b] \int_{0}^{1} \bigl[u_{1}(t)+v_{1}(t) \bigr]e(t)\,dt&\leq \zeta^{-1} \biggl[ r(K) \int_{0}^{1} \bigl[(Hu_{1}) (t)+(Hv_{1}) (t) \bigr]e(t)\,dt\\ &\quad {} + \int_{0}^{1} \bigl[m_{0}(t)+n_{0}(t) \bigr]e(t)\,dt \biggr]. \end{aligned} $$
(2.11)

By computation, we obtain

$$ \begin{aligned} &\int_{0}^{1}(KHu_{1}) (t)e(t)\,dt=r(K) \int_{0}^{1}(Hu_{1}) (t)e(t)\,dt,\\ &\int_{0}^{1}(KHv_{1}) (t)e(t)\,dt=r(K) \int_{0}^{1}(Hv_{1}) (t)e(t)\,dt. \end{aligned} $$
(2.12)

By (2.6), (2.7), (2.11), and (2.12), we get

$$ \begin{aligned}[b] & \int _{0}^{1} \bigl[u_{1}(t)+v_{1}(t)+(KHv_{1}) (t)+(KHu_{1}) (t)+(Ku_{0}) (t)+(Kv_{0}) (t) \bigr]e(t)\,dt \\ &\quad \leq\zeta^{-1} \biggl[r(K) \int_{0}^{1} \bigl[(Hv_{1}) (t)+(Hu_{1}) (t) \bigr]e(t)\,dt + \int_{0}^{1} \bigl[m_{0}(t)+n_{0}(t) \bigr]e(t)\,dt \biggr] \\ &\qquad {}+r(K) \int_{0}^{1}(Hu_{1}) (t)e(t)\,dt+r(K) \int_{0}^{1}(Hv_{1}) (t)e(t)\,dt\\ &\qquad {} + \int_{0}^{1}(Ku_{0}) (t)e(t)\,dt+ \int_{0}^{1}(Ku_{0}) (t)e(t)\,dt \\ &\quad =\zeta^{-1}(1+\zeta)r(K) \int_{0}^{1} \bigl[(Hu_{1}) (t)+(Hv_{1}) (t) \bigr]e(t)\,dt\\ &\qquad {} +\zeta^{-1} \int_{0}^{1} \bigl[m_{0}(t)+n_{0}(t) \bigr]e(t)\,dt + \int_{0}^{1} \bigl[(Ku_{0}) (t)+(Kv_{0}) (t) \bigr]e(t)\,dt \\ &\quad \leq\zeta^{-1}(1+\zeta)r(K) \bigl( \Vert Hu \Vert + \Vert Hv \Vert \bigr)+\zeta^{-1} \int _{0}^{1} \bigl[m_{0}(t)+n_{0}(t) \bigr]e(t)\,dt\\ &\qquad {} + \int_{0}^{1} \bigl[(Ku_{0}) (t)+(Kv_{0}) (t) \bigr]e(t)\,dt \\ &\quad \leq\zeta^{-1}(1+\zeta)r(K)\varepsilon_{0} \bigl( \Vert u \Vert + \Vert v \Vert \bigr)+l_{1}, \end{aligned} $$
(2.13)

where \(l_{1}=\zeta^{-1}\int_{0}^{1}[m_{0}(t)+n_{0}(t)]e(t)\,dt +\int_{0}^{1}[(Ku_{0})(t)+(Kv_{0})(t)]e(t)\,dt\) is a constant.

(C3) shows \((Fv_{1}+v_{0}+Hv_{1}, Gu_{1}+u_{0}+Hu_{1})\in P^{2}\) and (C1) implies \(\mu_{1}\phi=\mu_{1}\lambda_{1}K\phi\in P_{0}\). Then (C1), (2.10), and Lemma 2.6 tell us that

$$\begin{aligned}& u_{1}+KHv_{1}+Kv_{0}=KFv_{1}+ \mu_{1}\phi+KHv_{1}+Kv_{0} =K(Fv_{1}+Hv_{1}+v_{0})+ \mu_{1}\phi\in P_{0}, \\& v_{1}+KHu_{1}+Ku_{0}=KGu_{1}+ \mu_{1}\phi+KHu_{1}+Ku_{0} =K(Gv_{1}+Hu_{1}+u_{0})+ \mu_{1}\phi\in P_{0}. \end{aligned}$$

The definition of \(P_{0}\) yields

$$ \begin{aligned} & \int_{0}^{1}(u_{1}+KHv_{1}+Kv_{0}) (t)e(t)\,dt\geq \delta \Vert u_{1}+KHv_{1}+Kv_{0} \Vert \geq \delta \Vert u_{1} \Vert -\delta \Vert KHv_{1} \Vert -\delta \Vert Kv_{0} \Vert , \\ & \int_{0}^{1}(v_{1}+KHu_{1}+Ku_{0}) (t)e(t)\,dt\geq\delta \Vert v_{1}+KHu_{1}+Ku_{0} \Vert \geq \delta \Vert v_{1} \Vert -\delta \Vert KHu_{1} \Vert -\delta \Vert Ku_{0} \Vert , \end{aligned} $$

where δ is given in Lemma 2.6. By adding the above two inequalities, we obtain

$$ \begin{aligned}[b] &\int_{0}^{1}(u_{1}+v_{1}+KHu_{1}+KHv_{1}+Ku_{0}+Kv_{0}) (t)e(t)\,dt\\ &\quad \geq \delta \bigl( \Vert v_{1} \Vert + \Vert u \Vert \bigr)-\delta \bigl( \Vert KHu_{1} \Vert + \Vert KHv_{1} \Vert \bigr)-\delta \bigl( \Vert Ku_{0} \Vert + \Vert Kv_{0} \Vert \bigr). \end{aligned} $$
(2.14)

It follows from (2.7), (2.13), and (2.14) that

$$ \begin{aligned}[b] \Vert u_{1} \Vert + \Vert v_{1} \Vert &\leq\delta^{-1} \int_{0}^{1}(u_{1}+v_{1}+KHu_{1}+KHv_{1}+Ku_{0}+Kv_{0}) (t)e(t)\,dt\\ &\quad {} + \Vert KHu_{1} \Vert + \Vert KHv_{1} \Vert + \Vert Ku_{0} \Vert + \Vert Kv_{0} \Vert \\ &\leq \varepsilon_{0}(\delta\lambda_{1})^{-1} \bigl(1+\zeta^{-1} \bigr) \bigl( \Vert u_{1} \Vert + \Vert v_{1} \Vert \bigr)+l_{1}\delta^{-1}\\ &\quad {} + \varepsilon_{0} \Vert K \Vert \cdot \bigl( \Vert u_{1} \Vert + \Vert v_{1} \Vert \bigr)+ \Vert Ku_{0} \Vert + \Vert Kv_{0} \Vert \\ &=\varepsilon_{0} l_{0} \bigl( \Vert u_{1} \Vert + \Vert v_{1} \Vert \bigr)+l_{2}, \end{aligned} $$
(2.15)

where \(l_{2}=l_{1}\delta^{-1}+\|Ku_{0}\|+\|Kv_{0}\|\) is a constant.

Since \(0<\varepsilon_{0} l_{0}<1\), then (2.15) deduces that (2.8) holds provided that R is sufficiently large such that \(R>\max\{l_{2}/(1-\varepsilon_{0} l_{0} ),R_{1}\}\). By (2.15) and Lemma 2.7, we have

$$ \operatorname{deg}(I-A,B_{R},0)=0. $$

 □

3 Existence

Theorem 3.1

Assume that the following conditions are satisfied:

(A1):

\(f,g:[0,1]\times\mathbb{R}\to\mathbb{R}\) are continuous.

(A2):

There exist nonnegative functions \(b_{i}(t),c_{i}(t)\in C[0,1]\) with \(c_{i}(t)\not\equiv0\) and two continuous even functions \(B_{i}:\mathbb{R}\to\mathbb{R}^{+}\) such that

$$ \begin{aligned} &f(t,x)\geq-b_{1}(t)-c_{1}(t)B_{1}(x) \quad \textit{for all } x\in\mathbb{R}, \\ &g(t,y)\geq-b_{2}(t)-c_{2}(t)B_{2}(y) \quad \textit{for all } y\in\mathbb{R}. \end{aligned} $$

Moreover, \(B_{i}\) is nondecreasing on \(\mathbb{R}^{+}\) and satisfies \(\lim_{x\to+\infty}\frac{B_{i}(x)}{x}=0\), (\(i=1,2\)).

(A3):

\(\liminf_{x\to+\infty}\frac{f(t,x)}{x}>\lambda_{1}\), \(\liminf_{x\to+\infty}\frac{g(t,y)}{y}>\lambda_{1}\), uniformly on \(t\in[0,1]\).

(A4):

\(\limsup_{x\to 0} \vert \frac{f(t,x)}{x} \vert <\lambda_{1}\), \(\limsup_{x\to 0} \vert \frac{g(t,y)}{y} \vert <\lambda_{1}\), uniformly on \(t\in[0,1]\).

Here \(\lambda_{1}=1/r(K)\) is a number and the operator K is defined by (2.3).

Then FBVP (1.1) has at least one nontrivial solution.

Proof

We first show that all conditions in Lemma 2.8 are satisfied. By Lemma 2.6, condition (C1) of Lemma 2.8 is satisfied. \((T_{i}u)(t)=B_{i}(u(t))\) (\(i=1,2\)) for any \(u\in E\). Obviously \(T_{1},T_{2}:E\to P\) are continuous operators. By (A2), for any \(\varepsilon>0\), there is \(L>0\) such that when \(z>L\), \(B_{i}(z)<\varepsilon z\) holds. Thus, for \(w\in E \) with \(\|w\|>L\), \(B_{i}(\|w\|)<\varepsilon\|w\|\) holds. The fact that \(B_{i}\) is nondecreasing on \(\mathbb{R}^{+}\) yields \((T_{i}w)(t)\leq T_{i}(\|w\|)\) for any \(w\in P\), \(t\in[0,1]\). Since \(B_{i}:\mathbb{R} \to\mathbb{R}^{+}\) is an even function, \(\|T_{i}w\|\leq T_{i}(\|w\|)\) holds for \(w\in E\). Therefore,

$$ \Vert T_{i}w \Vert \leq T_{i} \bigl( \Vert w \Vert \bigr)< \varepsilon \Vert w \Vert , \quad\forall w\in E \mbox{ with } \Vert w \Vert >L, $$

that is, \(\lim_{\|w\| \to+\infty}\frac{\|T_{i}w\|}{\|w\|}=0\).

Define \((Hw)(t)=\max\{C_{1}(T_{1}w)(t),C_{2}(T_{2}w)(t)\}\) for any \(w\in E\), \(t\in[0,1]\), where \(C_{i}=\max_{t\in[0,1]}c_{i}(t)\), \(i=1,2\). By \(\lim_{\|w\| \to+\infty}\frac{\|T_{i}w\|}{\|w\|}=0\), \(\lim_{\|u\|+\|v\| \to +\infty}\frac{\|T_{i}u\|+\|T_{i}v\|}{\|u\|+\|v\|}=0\) holds. Therefore \(\lim_{\|u\|+\|v\| \to+\infty}\frac{\|Hu\|+\|Hv\|}{\|u\|+\|v\|}=0\) holds. Then we obtain that H satisfies condition (C2) in Lemma 2.8.

Take \(v_{0}(t)\equiv b_{1}=\max_{t\in[0,1]}b_{1}(t)>0\), \(u_{0}(t)\equiv b_{2}=\max_{t\in[0,1]}b_{2}(t)>0\), and \((Fv)(t)=f(t,v(t))\), \((Gu)(t)=g(t,u(t))\) for \(t\in[0,1]\), \((u,v)\in E^{2}\), then it follows from (A1) that

$$ (Fv+v_{0}+Hv, Gu+u_{0}+Hu)\in P^{2}\quad \mbox{for all }(u,v)\in E^{2}, $$

which shows that condition (C3) in Lemma 2.8 holds.

By (A3), there exist \(\varepsilon_{1}>0\) and a sufficiently large number \(L_{1}>0\) such that

$$ f(t,x)\geq\lambda_{1}(1+\varepsilon_{1})x,\qquad g(t,y)\geq \lambda_{1}(1+\varepsilon_{1})y,\quad \forall x,y\geq L_{1}. $$
(3.1)

Combining (3.1) with (A2), the above constants \(b_{1}\), \(b_{2}\) satisfy

$$\begin{aligned}& f(t,x)\geq\lambda_{1}(1+\varepsilon_{1})x-b_{1}-c_{1}B_{1}(x),\\& g(t,y)\geq\lambda_{1}(1+\varepsilon_{1})y-b_{2}-c_{2}B_{1}(y) \quad\mbox{for all }x,y\in\mathbb{R}, \end{aligned}$$

and so

$$ (Fv,Gu)\geq \bigl(\lambda_{1}(1+\varepsilon_{1})v-b_{1}-Hv, \lambda_{1}(1+\varepsilon_{1})u-b_{2}-Hu \bigr) \quad \mbox{for all }(u,v)\in E^{2}. $$
(3.2)

Since K is a positive linear operator, from (3.2), we have

$$ \begin{aligned} \bigl((KFv) (t),(KGu) (t) \bigr)&\geq \bigl(\lambda_{1}(1+ \varepsilon_{1}) (Kv ) (t)-Kb_{1}- (KHv ) (t),\\ &\quad \lambda_{1}(1+\varepsilon_{1}) (Ku ) (t)-Kb_{2}- (KHu ) (t) \bigr)\quad \forall t\in[0,1],(u,v)\in E^{2}. \end{aligned} $$

Let \(m_{0}(t)=(Kb_{1})(t)\), \(n_{0}(t)=(Kb_{2})(t)\). Then condition (C4) in Lemma 2.8 is satisfied.

According to Lemma 2.8, we derive that there exists a sufficiently large number \(R>0\) such that

$$ \operatorname{deg}(I-A,B_{R},0)=0. $$
(3.3)

From (A4), it follows that there exist \(0<\varepsilon_{2}<1\) and \(0< r< R\) such that

$$\begin{aligned}& \bigl\vert f(t,x) \bigr\vert \leq(1-\varepsilon_{2}) \lambda_{1} \vert x \vert ,\qquad \bigl\vert g(t,y) \bigr\vert \leq(1- \varepsilon_{2})\lambda_{1} \vert y \vert ,\\& \quad \forall t\in[0,1], x,y\in\mathbb{R} \mbox{ with } \vert x \vert \leq r, \vert y \vert \leq r. \end{aligned}$$

Thus

$$ \begin{aligned}[b] &\bigl\vert (A_{1}u) (t) \bigr\vert \leq(1- \varepsilon_{2})\lambda_{1} \bigl(K \vert v \vert \bigr) (t),\qquad (A_{2}v) (t)|\leq(1-\varepsilon_{2}) \lambda_{1} \bigl(K \vert u \vert \bigr) (t),\\ &\quad \forall t \in[0,1], u,v\in E \mbox{ with } \Vert u \Vert \leq r, \Vert v \Vert \leq r. \end{aligned} $$
(3.4)

Next we will prove that

$$ (u,v)\neq\mu A(u,v)\quad\mbox{ for all }(u,v)\in\partial B_{r} \mbox{ and }\mu\in[0,1]. $$
(3.5)

If there exist \((u_{1},v_{1})\in\partial B_{r}\) and \(\mu_{1}\in[0,1]\) such that \((u_{1},v_{1})\neq\mu A(u_{1},v_{1})\), that is,

$$ \begin{aligned}&u_{1}(t)=(A_{1}v_{1}) (t)=\mu_{1} \int_{0}^{1} G(t,s)h(s)f \bigl(s,v_{1}(s) \bigr)\,ds, \\ &v_{1}(t)=(A_{2}u_{1}) (t)=\mu_{1} \int_{0}^{1} G(t,s)h(s)g \bigl(s,u_{1}(s) \bigr)\,ds. \end{aligned} $$

Let \(z(t)=|u_{1}(t)|+|v_{1}(t)|\). Then \(z\in P\) and by (3.4),

$$ \begin{aligned} z(t)&\leq(1-\varepsilon_{2})\lambda_{1} \bigl[ \bigl(K \vert u_{1} \vert \bigr) (t)+ \bigl(K \vert v_{1} \vert \bigr) (t) \bigr] \\ &=(1-\varepsilon_{2})\lambda_{1} \bigl(K \bigl( \vert u_{1} \vert + \vert v_{1} \vert \bigr) \bigr) (t)= (1-\varepsilon_{2})\lambda_{1}(Kz) (t). \end{aligned} $$

The nth iteration of this inequality shows that \(z(t)\leq(1-\varepsilon_{2})^{n}\lambda_{1}^{n} (K^{n}z)(t)\) (\(n=1,2,\ldots\)), so \(\|z\|\leq (1-\varepsilon_{2})^{n}\lambda_{1}^{n}\|K^{n}\|\cdot\|z\|\), that is, \(1\leq (1-\varepsilon_{2})^{n}\lambda_{1}^{n}\|K^{n}\|\). This yields \(1-\varepsilon_{2}=(1-\varepsilon_{2})\lambda_{1}r(K)=(1-\varepsilon_{2})\lambda _{1}\lim_{n\to \infty}\sqrt[n]{\|K^{n}\|}\geq1\), which is a contradictory inequality. Hence, (3.5) holds.

It follows from (3.5) and Lemma 2.7 that

$$ \operatorname{deg}(I-A,B_{r},0)=1. $$
(3.6)

By (3.3), (3.6), and the additivity of Leray–Schauder degree, we obtain

$$ \operatorname{deg}(I-A,B_{R}\setminus \overline{B}_{r},0)= \operatorname{deg}(I-A,B_{R},0)-\operatorname{deg}(I-A,B_{r},0)=-1. $$

So A has at least one fixed point on \(B_{R}\setminus\overline{B}_{r}\), namely FBVP (1.1) has at least one nontrivial solution. □

4 Uniqueness

Theorem 4.1

Assume that (A1)–(A4) are satisfied. Moreover, the following conditions are satisfied:

(A5):

\(0<\int_{0}^{1} g_{\beta}(s)h(s)\,ds<+\infty\) and there exists a constant \(k<[\int_{0}^{1}(\frac{s(1-s)^{\alpha-1}}{\Gamma(\alpha-1)}+\frac{g_{\beta}(s)}{1-\Gamma} )h(s)\,ds]^{-1}\) such that

$$\bigl\vert f(t,x)-f(t,y) \bigr\vert \leq k \vert x-y \vert ,\qquad \bigl\vert g(t,x)-g(t,y) \bigr\vert \leq k \vert x-y \vert \quad \textit{for any } x,y\in\mathbb{R}. $$

Then FBVP (1.1) has a unique solution.

Proof

It follows from \(|f(t,x)-f(t,y)|\leq k|x-y|\), \(|g(t,x)-g(t,y)|\leq k|x-y|\) for any \(x,y\in\mathbb{R}\) that (A1) holds. Then by Theorem 3.1, FBVP (1.1) has at least one nontrivial solution. Suppose that FBVP (1.1) has two different solutions \((u_{1}(t),v_{1}(t))\) and \((u_{2}(t),v_{2}(t))\). By Lemma 2.6, \(G(t,s)\leq \frac{s(1-s)^{\alpha-1}}{\Gamma(\alpha-1)}+\frac{g_{\beta}(s)}{1-\Gamma}\). Then from (A5) it follows that

$$\begin{aligned}& \begin{aligned} \Vert u_{1}-u_{2} \Vert & =\max _{t\in[0,1]} \bigl\vert (A_{1}v_{1}) (t)-(A_{1}v_{2}) (t) \bigr\vert \\ &\leq \max _{t\in[0,1]} \int_{0}^{1}G(t,s)h(s) \bigl\vert f \bigl(s,v_{1}(s) \bigr)-f \bigl(s,v_{2}(s) \bigr) \bigr\vert \,ds \\ &\leq k \Vert v_{1}-v_{2} \Vert \max _{t\in[0,1]} \int_{0}^{1}G(t,s)h(s)\,ds \\ &\leq k \Vert v_{1}-v_{2} \Vert \int_{0}^{1} \biggl(\frac{s(1-s)^{\alpha-1}}{\Gamma(\alpha -1)}+ \frac{g_{\beta}(s)}{1-\Gamma} \biggr)h(s)\,ds \\ &< \Vert v_{1}-v_{2} \Vert , \end{aligned} \\& \begin{aligned} \Vert v_{1}-v_{2} \Vert & =\max _{t\in[0,1]} \bigl\vert (A_{2}u_{1}) (t)-(A_{2}u_{2}) (t) \bigr\vert \\ &\leq \max _{t\in[0,1]} \int_{0}^{1}G(t,s)h(s) \bigl\vert f \bigl(s,u_{1}(s) \bigr)-f \bigl(s,u_{2}(s) \bigr) \bigr\vert \,ds \\ &\leq k \Vert u_{1}-u_{2} \Vert \max _{t\in[0,1]} \int_{0}^{1}G(t,s)h(s)\,ds \\ &\leq k \Vert u_{1}-u_{2} \Vert \int_{0}^{1} \biggl(\frac{s(1-s)^{\alpha-1}}{\Gamma(\alpha -1)}+ \frac{g_{\beta}(s)}{1-\Gamma} \biggr)h(s)\,ds \\ &< \Vert u_{1}-u_{2} \Vert . \end{aligned} \end{aligned}$$

By adding the above two inequalities, we obtain \(\|v_{1}-v_{2}\|+\|u_{1}-u_{2}\|<\|v_{1}-v_{2}\|+\|u_{1}-u_{2}\|\), which is a contradictory inequality. Therefore \((u_{1}(t),v_{1}(t))=(u_{2}(t),v_{2}(t))\) and FBVP (1.1) has a unique solution. □

5 Examples

Example 5.1

Consider FBVP (1.1) with

$$\beta(t)= \textstyle\begin{cases} 0,&[0,\frac{1}{3}),\\ \frac{1}{8},&[\frac{1}{3},\frac{2}{3}),\\ \frac{1}{2},&[\frac{2}{3},1], \end{cases} $$

\(h(t)=\frac{1}{\sqrt{t(1-t)}}\) and

$$\begin{aligned}& f(t,x)= \textstyle\begin{cases} \sum_{i=1}^{n}(-1)^{i}a_{i}-(1+t^{2}) \vert x \vert ^{\frac{1}{3}}\ln( \vert x \vert +1)+(1+t^{2})\ln2,& x\in(-\infty,-1),\\ \sum_{i=1}^{n}a_{i}x^{i}, &x\in[-1,+\infty), \end{cases}\displaystyle \\& g(t,y)= \textstyle\begin{cases} \sum_{i=1}^{n}(-1)^{i}a_{i}-(1+t^{4}) \vert y \vert ^{\frac{1}{3}}\ln( \vert y \vert ^{\frac {1}{3}}+1)+(1+t^{4})\ln2,& y\in(-\infty,-1),\\ \sum_{i=1}^{n}a_{i}y^{i}, &y\in[-1,+\infty), \end{cases}\displaystyle \end{aligned}$$

where \(0< a_{1}<\lambda_{1}\), \(a_{n}>0\). Obviously, \(\Gamma=\int_{0}^{1}t^{\alpha-1}\,d\beta(t)=\frac{1}{8}(\frac{1}{3})^{\alpha-1} +\frac{3}{8}(\frac{2}{3})^{\alpha-1}<\frac{7}{24}<1\). Then h is singular at \(t=0,1\) and f, g are unbounded from below. Take \(c_{1}(t)=1+t^{2}\), \(c_{2}(t)=1+t^{4}\), \(b_{1}(t)=\sum_{i=1}^{n}a_{i}+(1+t^{2})\ln2\), \(b_{2}(t)=\sum_{i=1}^{n}a_{i}+(1+t^{4})\ln2\), \(B_{1}(x)=|x|^{\frac{1}{3}}\ln(|x|+1)\), \(B_{2}(x)=|x|^{\frac{1}{3}}\ln(|x|^{\frac{1}{3}}+1)\). Then all the conditions in Theorem 3.1 are satisfied. Therefore, FBVP (1.1) with the above \(\beta(t)\), \(h(t)\), \(f(t,x)\), \(g(t,y)\) has at least one nontrivial solution.

Example 5.2

Consider FBVP (1.1) with

$$\begin{aligned}& \beta(t)= \textstyle\begin{cases} 0,&[0,\frac{1}{3}),\\ \frac{1}{3},&[\frac{1}{3},1], \end{cases}\displaystyle \\& f(t,x)=g(t,x)= \textstyle\begin{cases} -a_{1}-(1+t^{2})\ln( \vert x \vert +1)+(1+t^{2})\ln2,& x\in(-\infty,-1),\\ a_{1}x, &x\in[-1,1),\\ a_{2}+a_{2}\ln(x+1)+a_{1}-a_{2}-a_{2}\ln2, &x\in[1,+\infty). \end{cases}\displaystyle \end{aligned}$$

\(h(t)=\frac{\Gamma(\alpha-1)}{2(5a_{2}+2)\sqrt{t}(1-t)^{\alpha-1}}\), where \(0< a_{1}<\lambda_{1}<a_{2}\).

Take \(c_{i}(t)=1+t^{2}\), \(b_{i}(t)=a_{1}+(1+t^{2})\ln2\), \(B_{i}(x)=\ln(|x|+1)\), \(i=1,2\). Then (A2) is satisfied. The choice of \(a_{1}\), \(a_{2}\) guarantees that (A3) and (A4) are satisfied. By some simple computation, we obtain that \(\Gamma=\int_{0}^{1}t^{\alpha-1}\,d\beta(t) =\frac{1}{3^{\alpha}}<1\), \(|f(t,x)-f(t,y)|\leq (a_{1}+\frac{3}{2}a_{2}+1)|x-y|\), \(|g(t,x)-g(t,y)|\leq (a_{1}+\frac{3}{2}a_{2}+1)|x-y|\) for any \(x,y\in\mathbb{R}\) and \(\int_{0}^{1}(\frac{s(1-s)^{\alpha-1}}{\Gamma(\alpha-1)}+\frac{g_{\beta}(s)}{1-\Gamma} )h(s)\,ds<\frac{4}{3(5a_{2}+2)}\). Hence (A5) holds. So FBVP (1.1) with the above \(\beta(t)\), \(h(t)\), \(f(t,x)\), \(g(t,y)\) has a unique solution.