1 Introduction

Spatial propagation dynamics of parabolic type systems has been widely investigated in the literature. In the past decades, some important results were established for monotone semiflows; see [1–6] and a survey paper by Zhao [7]. In particular, there are some important thresholds that have been widely and intensively studied, and one is the minimal wave speed of traveling wave solutions, which plays an important role modeling biological processes and chemical kinetic [8, 9]. Here, the minimal wave speed implies the existence (nonexistence) of a desired traveling wave solution if the wave speed is not less (is less) than the threshold.

It is well known that energy transfer is one basic law in nature and one typical model on the topic is the predator–prey system, and the spatial distribution of individuals is also important to understand the evolutionary process [10–13]. Since the work of Dunbar [14–16], much attention has been paid to traveling wave solutions of reaction–diffusion systems with predator–prey nonlinearities to model the transmission of energy. However, the dynamics of predator–prey systems is a very field of research since they do not generate monotone semiflows, and there are many open problems on the minimal wave speed of traveling wave solutions.

In this paper, we shall investigate the following nonmonotone system:

$$ \textstyle\begin{cases} \frac{\partial u_{1}(x,t)}{\partial t}=d_{1}[J_{1}*u_{1}](x,t)+r_{1}u _{1}(x,t)F_{1}(u_{1},u_{2})(x,t), \\ \frac{\partial u_{2}(x,t)}{\partial t}=d_{2}[J_{2}*u_{2}](x,t)+r _{2}u_{2}(x,t)F_{2}(u_{1},u_{2})(x,t), \end{cases} $$
(1.1)

in which \(x\in \mathbb{R}\), \(t>0\), \((u_{1},u_{2})\in \mathbb{R}^{2}\), \(r_{1}\), \(r_{2}\), \(d_{1}\) and \(d_{2}\) are positive constants, \(F_{1}\) and \(F_{2}\) are defined by

$$\begin{aligned}& \begin{aligned} F_{1}(u_{1},u_{2}) (x,t) ={}&1-a_{1}u_{1}(x,t) \\ &{}-b_{1} \int_{-\tau }^{0}u_{1}(x,t+s)\,d\eta_{11}(s)-c_{1} \int_{-\tau } ^{0}u_{2}(x,t+s)\,d\eta_{12}(s), \end{aligned} \\& \begin{aligned} F_{2}(u_{1},u_{2}) (x,t) ={}&1-a_{2}u_{2}(x,t) \\ &{}-b_{2} \int_{-\tau }^{0}u_{2}(x,t+s)\,d\eta_{22}(s)+c_{2} \int_{-\tau } ^{0}u_{1}(x,t+s)\,d\eta_{21}(s), \end{aligned} \end{aligned}$$

hereafter, \(a_{1}> 0\), \(a_{2}> 0\), \(b_{1}\ge 0\), \(b_{2} \ge 0\), \(c_{1}\ge 0\), \(c _{2}\ge 0\), \(\tau > 0\) are constants such that

$${\eta } _{ij}(s) \text{ is nondecreasing on }[-\tau,0]\quad \text{ and }\quad {\eta } _{ij}(0)-{\eta } _{ij}(-\tau)=1, \quad i,j=1,2. $$

Moreover, \([J_{1}*u_{1}](x,t)\) and \([J_{2}*u_{2}](x,t)\) formulate the spatial dispersal of individuals (see Bates [17], Fife [18] and Hopf [19] for the backgrounds and applications of dispersal models) and are illustrated by

$$\begin{aligned}& {}[ J_{1}\ast u_{1}](x,t) = \int_{\mathbb{R}}J_{1}(x-y)\bigl[u_{1}(y,t)-u _{1}(x,t)\bigr]\,dy, \\& {}[ J_{2}\ast u_{2}](x,t) = \int_{\mathbb{R}}J_{2}(x-y)\bigl[u_{2}(y,t)-u _{2}(x,t)\bigr]\,dy, \end{aligned}$$

where \(J_{1}\), \(J_{2}\) are probability kernel functions formulating the random dispersal of individuals and satisfy the following assumptions:

  1. (J1)

    \(J_{i}\) is nonnegative and continuous for each \(i=1,2\);

  2. (J2)

    for any \(\lambda \in \mathbb{R}\), \(\int_{\mathbb{R}}J_{i}(y)e ^{\lambda y}\,dy < \infty\), \(i=1,2\);

  3. (J3)

    \(\int_{\mathbb{R}}J_{i}(y)\,dy=1\), \(J_{i}(y)=J_{i}(-y)\), \(y \in \mathbb{R}\), \(i=1,2\).

Clearly, (1.1) is a predator–prey system and does not generate monotone semiflows. In Yu and Yuan [20], Zhang et al. [21], if \(a_{1}=a_{2}=0\) with small delay or \(b_{1}=b_{2}=0\), the authors obtained a threshold. If the wave speed is larger than the threshold, they proved the existence of traveling wave solutions, which formulates that both the predator and the prey invade a new habitat. But the question remains open of the existence or nonexistence of traveling wave solution if the wave speed is not larger than the threshold. Our main purpose of this paper is to answer the question.

The rest of this paper is organized as follows. In Sect. 2, we recall some known results. Section 3 is concerned with the existence of nonconstant traveling wave solutions. In Sect. 4, the asymptotic behavior and nonexistence of traveling wave solutions are presented. Finally, we give a discussion of the methods and results in this paper.

2 Preliminaries

In this part, we shall give some preliminaries. Since \(a_{1}>0\), \(a_{2}>0\) are positive constants, we assume that \(a_{1}=a_{2}=1\) due to the scaling recipe. Let

$$\bigl(u_{1}(x,t), u_{2}(x,t)\bigr)=\bigl( \phi_{1} (\xi),\phi_{2} (\xi)\bigr),\quad \xi = x+ct, $$

be a traveling wave solution of (1.1). Then \((\phi_{1} (\xi),\phi _{2} (\xi))\) and c satisfy

$$ \textstyle\begin{cases} d_{1}[J_{1}\ast \phi_{1}](\xi)-c\phi_{1}^{\prime }(\xi)+r_{1}\phi_{1}( \xi)F_{1}(\phi_{1},\phi_{2})(\xi)=0,&\xi \in \mathbb{R}, \\ d_{2}[J_{2}\ast \phi_{2}](\xi)-c\phi_{2}^{\prime }(\xi)+r_{2}\phi_{2}( \xi)F_{2}(\phi_{1},\phi_{2})(\xi)=0,&\xi \in \mathbb{R}, \end{cases} $$
(2.1)

with

$$\begin{aligned}& {}[ J_{1}\ast \phi_{1}](\xi)= \int_{\mathbb{R}}J_{1}(y)\phi _{1}(\xi -y)\,dy- \phi_{1}(\xi), \\& {}[ J_{2}\ast \phi_{21}](\xi)= \int_{\mathbb{R}}J_{2}(y)\phi_{2}( \xi -y)\,dy- \phi_{2}(\xi), \end{aligned}$$

and

$$\begin{aligned}& \begin{aligned} F_{1}(\phi_{1},\phi_{2}) (\xi) ={}&1- \phi_{1}(\xi) \\ &{}-b_{1} \int_{-\tau }^{0}\phi_{1}(\xi +cs)\,d\eta_{11}(s)-c_{1} \int_{- \tau }^{0}\phi_{2}(\xi +cs)\,d\eta_{12}(s), \end{aligned} \\& \begin{aligned} F_{2}(\phi_{1},\phi_{2}) (\xi) ={}&1- \phi_{2}(\xi) \\ &{}-b_{2} \int_{-\tau }^{0}\phi_{2}(\xi +cs)\,d\eta_{22}(s)+c_{2} \int_{- \tau }^{0}\phi_{1}(\xi +cs)\,d\eta_{21}(s). \end{aligned} \end{aligned}$$

Similar to [20, 22], we shall focus on the positive \((\phi_{1},\phi_{2})\) satisfying

$$ \lim_{\xi \to -\infty }\phi_{i} (\xi)=0,\qquad \lim _{\xi \to \infty }\phi _{i} (\xi)=k_{i},\quad i=1,2, $$
(2.2)

where \((k_{1},k_{2})\) is the unique spatial homogeneous steady state of (1.1) and

$$k_{1}=\frac{1+b_{2}-c_{1}}{(1+b_{1})(1+b_{2})+c_{1}c_{2}},\qquad k_{2}=\frac{1+b _{1}+c_{2}}{(1+b_{1})(1+b_{2})+c_{1}c_{2}} $$

provided that

$$ 1+b_{2}>c_{1}. $$
(2.3)

When the scalar equation is concerned, Jin and Zhao [23] studied a periodic equation with dispersal. Their results remain true for the following equation with constant coefficients:

$$ \textstyle\begin{cases} \frac{\partial u(x,t)}{\partial t}=d[J*u](x,t)+r u(x,t) [ 1-u(x,t) ] , \\ u(x,0)=\chi (x),\quad x\in \mathbb{R}, \end{cases} $$
(2.4)

where J satisfies (J1)–(J3), \(d >0\) and \(r>0\) are constants, and the initial value \(\chi (x)\) is uniformly continuous and bounded. By [23], Theorem 2.3, we have the following comparison principle of (2.4).

Lemma 2.1

Assume that \(0\le \chi (x)\le 1\). Then (2.4) admits a solution for all \(x\in \mathbb{R}\), \(t>0\). If \(w(x,0)\) is uniformly continuous and bounded, and \(w(x,t)\) satisfies

$$\textstyle\begin{cases} \frac{\partial w(x,t)}{\partial t}\ge (\le)d[J*w](x,t)+r w(x,t) [ 1-w(x,t) ] ,\quad x\in \mathbb{R},t>0, \\ w(x,0)\ge (\le)\chi (x),\quad x\in \mathbb{R}, \end{cases} $$

then

$$w(x,t)\ge (\le) u(x,t),\quad x\in \mathbb{R}, t>0. $$

For \(\lambda >0\), define

$$c'=\inf_{\lambda >0}\frac{d [ \int_{\mathbb{R}}J(y)e^{\lambda y}\,dy-1 ] +r}{\lambda }. $$

Then \(c'>0\) holds. Moreover, it also admits the following property [23].

Lemma 2.2

Assume that \(\chi (x)>0\). Then, for any \(c< c'\), we have

$$\liminf_{t\to \infty }\inf_{\vert x\vert < ct}u(x,t)=\limsup _{t\to \infty } \sup_{\vert x\vert < ct}u(x,t)=1. $$

If \(\chi (x)\) has nonempty compact support, then

$$\lim_{t\to \infty }\sup_{\vert x\vert >ct}u(x,t)=0,\quad c>c'. $$

For \(\lambda >0\), \(c>0\), we further define \(c^{\ast } =\max \{c_{1}^{ \ast },c_{2}^{\ast }\}\) with

$$\begin{aligned}& c_{1}^{\ast }=\inf_{\lambda >0} \frac{d_{1} [ \int_{\mathbb{R}}J _{1}(y)e^{\lambda y}\,dy-1 ] +r_{1}}{\lambda }, \\& c_{2}^{\ast } =\inf_{\lambda >0} \frac{d_{2} [ \int_{\mathbb{R}}J _{2}(y)e^{\lambda y}\,dy-1 ] +r_{2}}{\lambda }, \end{aligned}$$

and

$$\begin{aligned}& \Theta_{1}(\lambda,c) =d_{1} \biggl[ \int_{\mathbb{R}}J_{1}(y)e^{ \lambda y}\,dy-1 \biggr] -c \lambda +r_{1}, \\& \Theta_{2}(\lambda,c) =d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y)e^{ \lambda y}\,dy-1 \biggr] -c \lambda +r_{2}. \end{aligned}$$

By the convexity, we have the following conclusion.

Lemma 2.3

Assume that \(c^{\ast }\), \(\Theta_{1}(\lambda,c)\), \(\Theta_{2}(\lambda,c)\) are defined as the above.

  1. (1)

    \(c_{i}^{\ast }>0\) holds and \(\Theta_{i}(\lambda,c)=0\) has two distinct positive roots \(\lambda_{i}^{c}<\lambda_{i+2}^{c}\) for any \(c>c^{\ast }\) and each \(i=1,2\). Moreover, for each \(i=1,2\), and \(c>c_{i}^{\ast }\), if \(\lambda_{i}\in (\lambda_{i}^{c},\lambda_{i+2} ^{c})\), then \(\Theta_{i}(\lambda_{i},c)<0\).

  2. (2)

    If \(c\in (0,c_{i}^{\ast })\), then \(\Theta_{i}(\lambda,c)>0\) for any \(\lambda >0\) and \(i=1,2\).

  3. (3)

    If \(c=c_{i}^{\ast }\), then \(\Theta_{i}(\lambda,c^{\ast }) \geq 0\) for any \(\lambda >0\) and \(\Theta_{i}(\lambda,c^{\ast })=0\) has a unique positive root \(\lambda_{i}^{{\ast }}\), where \(i=1,2\).

For convenience, we use the following notation:

$$\begin{aligned}& \begin{aligned} H_{1}(\phi_{1},\psi_{1},\phi_{2}) (\xi) ={}&1-\phi_{1}(\xi) \\ &{}-b_{1} \int_{-\tau }^{0}\psi_{1}(\xi +cs)\,d\eta_{11}(s)-c_{1} \int_{- \tau }^{0}\phi_{2}(\xi +cs)\,d\eta_{12}(s), \end{aligned} \\& \begin{aligned} H_{2}(\phi_{1},\psi_{1},\phi_{2}) ( \xi) ={}&1-\phi_{2}(\xi) \\ &{}-b_{2} \int_{-\tau }^{0}\psi_{1}(\xi +cs)\,d\eta_{22}(s)+c_{2} \int_{- \tau }^{0}\phi_{1}(\xi +cs)\,d\eta_{21}(s), \end{aligned} \end{aligned}$$

for any positive bounded continuous functions \(\phi_{1}(\xi)\), \(\psi _{1}(\xi)\), \(\phi_{2}(\xi)\), \(\xi \in \mathbb{R}\).

Similar to Pan [24], Theorem 3.2, we can prove the following conclusions.

Lemma 2.4

Assume that \(\underline{\phi}_{1}(\xi)\), \(\overline{\phi}_{1}(\xi)\), \(\underline{\phi}_{2}(\xi)\), \(\overline{\phi}_{2}(\xi)\) are continuous functions satisfying

  1. (A1)

    \(0\le \underline{\phi}_{1}(\xi)\le \overline{\phi}_{1}( \xi)\le 1\), \(0\le \underline{\phi}_{2}(\xi)\le \overline{\phi}_{2}( \xi)\le 1+c_{2}\), \(\xi \in \mathbb{R}\);

  2. (A2)

    there exists a set E containing finite points of \(\mathbb{R}\) such that they are differentiable and their derivatives are bounded if \(\xi \in \mathbb{R}\backslash E\);

  3. (A3)

    they satisfies the following inequalities:

    $$\begin{aligned}& d_{1}[J_{1}\ast \overline{\phi}_{1}](\xi)-c\overline{\phi}_{1}^{ \prime }(\xi)+r_{1} \overline{\phi}_{1}(\xi)H_{1}(\overline{\phi} _{1},\underline{\phi}_{1},\underline{\phi}_{2}) (\xi)\leq 0, \end{aligned}$$
    (2.5)
    $$\begin{aligned}& d_{1}[J_{1}\ast \underline{\phi}_{1}](\xi)-c\underline{\phi}_{1} ^{\prime }(\xi)+r_{1} \underline{\phi}_{1}(\xi)H_{1}(\underline{ \phi }_{1},\overline{\phi}_{1},\overline{\phi}_{2}) (\xi) \geq 0, \end{aligned}$$
    (2.6)
    $$\begin{aligned}& d_{2}[J_{2}\ast \overline{\phi}_{2}](\xi)-c \overline{\phi}_{2}^{ \prime }(\xi)+r_{2}\overline{\phi}_{2}(\xi)H_{2}(\overline{\phi} _{1}, \underline{\phi}_{2},\overline{\phi}_{2}) (\xi) \leq 0, \end{aligned}$$
    (2.7)
    $$\begin{aligned}& d_{2}[J_{2}\ast \underline{\phi}_{2}](\xi)-c \underline{\phi}_{2} ^{\prime }(\xi)+r_{2}\underline{ \phi }_{2}(\xi)H_{2}(\underline{ \phi }_{1}, \overline{\phi}_{2},\underline{\phi}_{2}) (\xi) \geq 0, \end{aligned}$$
    (2.8)

    for \(\xi \in \mathbb{R}\backslash E\).

Then (2.1) has a positive solution \((\phi_{1}(\xi),\phi_{2}( \xi))\) such that

$$\underline{\phi}_{1}(\xi)\leq \phi_{1}(\xi)\leq \overline{\phi} _{1}(\xi),\qquad \underline{\phi}_{2}(\xi)\leq \phi_{2}(\xi)\leq \overline{ \phi }_{2}(\xi),\quad \xi \in \mathbb{R}. $$

Remark 2.5

Here, \((\overline{\phi}_{1}(\xi), \overline{\phi}_{2}(\xi))\), \((\underline{\phi}_{1}(\xi), \underline{\phi}_{2}(\xi))\) are a pair of generalized upper and lower solutions of (2.1). Therefore, the existence of traveling wave solutions is deduced to the existence of generalized upper and lower solutions, of which the recipe has been earlier utilized in delayed reaction–diffusion systems by Ma [25] and Wu and Zou [26] for quasimonotone systems, and by Huang and Zou [27] for predator–prey systems. When the dispersal models are involved, we also refer to [20, 21, 28–31].

3 Existence of traveling wave solutions

In this section, we shall present the existence of traveling wave solutions for any \(c\ge c^{*}\). When the wave speed is large, there exists a positive traveling wave solution.

Theorem 3.1

If \(c>c^{*}\), then (2.1) has a positive solution \((\phi_{1}( \xi),\phi_{2}(\xi))\) such that

$$ 0< \phi_{1}(\xi)< 1,\qquad 0< \phi_{2}(\xi)< 1+c_{2},\quad \xi \in \mathbb{R} $$
(3.1)

and

$$\lim_{\xi \rightarrow -\infty }\bigl(\phi_{1}(\xi), \phi_{2}(\xi)\bigr)=(0,0),\qquad \lim_{\xi \rightarrow -\infty }\bigl( \phi_{1}(\xi)e^{-\lambda_{1}^{c} \xi }, \phi_{2}(\xi)e^{-\lambda_{2}^{c}\xi }\bigr)=(1,1). $$

Proof

We shall prove it by Lemma 2.4, and first construct generalized upper and lower solutions. For convenience, we denote \(\lambda_{i} ^{c}\) by \(\lambda_{i}\) for simplicity, and we prove the result for any fixed \(c>c^{*}\).

Define continuous functions

$$\underline{\phi}_{1}(\xi)=\max \bigl\{ e^{\lambda_{1}\xi }-qe^{\eta \lambda_{1}\xi },0 \bigr\} ,\qquad \underline{\phi}_{2}(\xi)=\max \bigl\{ e^{\lambda _{2}\xi }-qe^{\eta \lambda_{2}\xi },0 \bigr\} $$

and

$$\overline{\phi}_{1}(\xi)=\min \bigl\{ e^{\lambda_{1}\xi },1\bigr\} ,\qquad \overline{ \phi }_{2}(\xi)=\min \bigl\{ e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2} \xi },1+c_{2} \bigr\} , $$

where

$$\eta \in \biggl( 1,\min \biggl\{ \frac{\lambda_{3}}{\lambda_{1}},\frac{ \lambda_{4}}{\lambda_{2}}, \frac{\lambda_{1}+\lambda_{2}}{\lambda_{1}},\frac{ \lambda_{1}+\lambda_{2}}{\lambda_{2}} \biggr\} \biggr) $$

and \(p>1\), \(q>1\) are constants, of which the definitions will be clarified later. We now show these functions satisfy (2.5)–(2.8) if they are differentiable.

If \(\overline{\phi}_{1}(\xi)=1< e^{\lambda_{1}\xi }\), then \(H_{1}(\overline{ \phi }_{1},\underline{\phi}_{1},\underline{\phi}_{2})(\xi)\leq 0 \) such that (2.5) is clear. Otherwise, \(\overline{\phi}_{1}( \xi)=e^{\lambda_{1}\xi }<1\) implies that

$$\begin{aligned}& d_{1}[J_{1}\ast \overline{\phi}_{1}](\xi)-c\overline{\phi}_{1} ^{\prime }(\xi)+r_{1} \overline{\phi}_{1}(\xi)H_{1}(\overline{ \phi }_{1},\underline{\phi}_{1},\underline{\phi}_{2}) (\xi) \\& \quad \leq d_{1}[J_{1}\ast \overline{\phi}_{1}](\xi)-c\overline{\phi} _{1}^{\prime }(\xi)+r_{1}\overline{\phi}_{1}(\xi) \\& \quad =d_{1} \biggl[ \int_{\mathbb{R}}J_{1}(y)\overline{\phi}_{1}( \xi -y)\,dy-e ^{\lambda_{1}\xi } \biggr] -c\lambda_{1}e^{\lambda_{1}\xi }+r_{1}e^{ \lambda_{1}\xi } \\& \quad \leq d_{1} \biggl[ \int_{\mathbb{R}}J_{1}(y)e^{\lambda_{1}(\xi -y)}\,dy-e ^{\lambda_{1}\xi } \biggr] -c\lambda_{1}e^{\lambda_{1}\xi }+r_{1}e^{ \lambda_{1}\xi } \\& \quad =e^{\lambda_{1}\xi } \biggl\{ d_{1} \biggl[ \int_{\mathbb{R}}J_{1}(y)e ^{\lambda_{1}y}\,dy-1 \biggr] -c \lambda_{1}+r_{1} \biggr\} \\& \quad =0, \end{aligned}$$

which implies what we wanted.

If \(\overline{\phi}_{2}(\xi)=1+c_{2}< e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }\), then \(H_{2}(\overline{\phi}_{1},\underline{\phi} _{2},\overline{\phi}_{2})(\xi)\leq 0\) such that (2.7) is clear. Otherwise, \(\overline{\phi}_{2}(\xi)=e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }<1+c_{2}\) such that

$$\begin{aligned}& r_{2}\overline{\phi}_{2}(\xi)H_{2}(\overline{ \phi }_{1},\underline{ \phi }_{2},\overline{\phi}_{2}) (\xi) \\& \quad =r_{2}\overline{\phi}_{2}(\xi) \biggl[ 1-\overline{\phi}_{2}( \xi)-b_{2} \int_{-\tau }^{0}\underline{\phi}_{2}(\xi +cs)\,d\eta_{22}(s)+c _{2} \int_{-\tau }^{0}\overline{\phi}_{1}(\xi +cs)\,d\eta_{21}(s) \biggr] \\& \quad \leq r_{2}\overline{\phi}_{2}(\xi) \biggl[ 1+c_{2} \int_{-\tau }^{0}\overline{ \phi }_{1}(\xi +cs)\,d\eta_{21}(s) \biggr] \\& \quad \leq r_{2}\overline{\phi}_{2}(\xi) \bigl[ 1+c_{2}e^{\lambda_{1} \xi } \bigr] \\& \quad =r_{2}\bigl[e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }\bigr] \bigl[ 1+c_{2}e ^{\lambda_{1}\xi } \bigr] \\& \quad =r_{2}\bigl[e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }\bigr]+r_{2}c_{2}e ^{\lambda_{1}\xi }\bigl[e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }\bigr] \end{aligned}$$

and

$$\begin{aligned}& d_{2}[J_{2}\ast \overline{\phi}_{2}](\xi)-c \overline{\phi}_{2} ^{\prime }(\xi)+r_{2}\overline{\phi}_{2}(\xi)H_{2}(\overline{ \phi }_{1}, \underline{\phi}_{2},\overline{\phi}_{2}) (\xi) \\& \quad =d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y)\overline{\phi}_{2}(\xi -y)\,dy- \bigl( e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi } \bigr) \biggr] \\& \qquad {}-c \bigl( \lambda_{2}e^{\lambda_{2}\xi }+p\eta \lambda_{2}e^{\eta \lambda_{2}\xi } \bigr) +r_{2}\overline{\phi}_{2}(\xi)H_{2}( \overline{ \phi }_{1},\underline{\phi}_{2},\overline{\phi}_{2}) (\xi) \\& \quad \leq d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y)\bigl[e^{\lambda_{2}(\xi -y)}+pe ^{\eta \lambda_{2}(\xi -y)}\bigr]\,dy- \bigl( e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi } \bigr) \biggr] \\& \qquad {}-c \bigl( \lambda_{2}e^{\lambda_{2}\xi }+p\eta \lambda_{2}e^{\eta \lambda_{2}\xi } \bigr) +r_{2}\overline{\phi}_{2}(\xi)H_{2}( \overline{ \phi }_{1},\underline{\phi}_{2},\overline{\phi}_{2}) (\xi) \\& \quad \leq d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y)\bigl[e^{\lambda_{2}(\xi -y)}+pe ^{\eta \lambda_{2}(\xi -y)}\bigr]\,dy- \bigl( e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi } \bigr) \biggr] \\& \qquad {}-c \bigl( \lambda_{2}e^{\lambda_{2}\xi }+p\eta \lambda_{2}e^{\eta \lambda_{2}\xi } \bigr) +r_{2}\bigl[e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2} \xi } \bigr]+r_{2}c_{2}e^{\lambda_{1}\xi }\bigl[e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi } \bigr] \\& \quad =p \biggl\{ d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y)e^{\eta \lambda_{2}( \xi -y)}\,dy-e^{\eta \lambda_{2}\xi } \biggr] -c\eta \lambda_{2}e^{ \eta \lambda_{2}\xi }+r_{2}e^{\eta \lambda_{2}\xi } \biggr\} \\& \qquad {}+r_{2}c_{2}e^{\lambda_{1}\xi }\bigl[e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2} \xi } \bigr] \\& \quad =p\Theta_{2}(\eta \lambda_{2},c)e^{\eta \lambda_{2}\xi }+r_{2}c_{2}e ^{\lambda_{1}\xi }\bigl[e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }\bigr] \\& \quad =e^{\eta \lambda_{2}\xi } \bigl[ p\Theta_{2}(\eta \lambda_{2},c)/2+r _{2}c_{2}e^{(\lambda_{1}+\lambda_{2}-\eta \lambda_{2})\xi } \bigr] +pe ^{\eta \lambda_{2}\xi } \bigl[ \Theta_{2}(\eta \lambda_{2},c)/2+r_{2}c _{2}e^{\lambda_{1}\xi } \bigr] . \end{aligned}$$

Note that

$$\eta \lambda_{2}\xi < \ln \frac{1+c_{2}}{p}, $$

then there exists \(p_{1}>1+c_{2}\) such that \(p= p_{1}\) leads to

$$p\Theta_{2}(\eta \lambda_{2},c)/2+r_{2}c_{2}e^{(\lambda_{1}+\lambda_{2}- \eta \lambda_{2})\xi }< 0,\qquad \Theta_{2}(\eta \lambda_{2},c)/2+r_{2}c_{2}e ^{\lambda_{1}\xi }< 0 $$

since \(\lambda_{1}+\lambda_{2}-\eta \lambda_{2}>0\), \(\xi <0\) and \(\Theta_{2}(\eta \lambda_{2},c)<0\) is a constant.

When \(\underline{\phi}_{1}(\xi)=0>e^{\lambda_{1}\xi }-qe^{\eta \lambda_{1}\xi }\), then \(H_{1}(\underline{\phi}_{1},\overline{\phi} _{1},\overline{\phi}_{2})(\xi)=0\) such that (2.6) is clear. Otherwise, \(\underline{\phi}_{1}(\xi)=e^{\lambda_{1}\xi }-qe^{ \eta \lambda_{1}\xi }>0\). Firstly, let \(q>q_{1}>1\) such that \(e^{\lambda_{1}\xi }-q_{1}e^{\eta \lambda_{1}\xi }>0\) implies \(\xi <0\) and

$$\overline{\phi}_{2}(\xi)< 2e^{\lambda_{2}\xi }, $$

which is admissible once p is fixed. Therefore, the monotonicity and \(q>q_{1}\) indicate

$$\begin{aligned}& r_{1}\underline{\phi}_{1}(\xi)H_{1}( \phi_{1},\psi_{1},\phi_{2}) ( \xi) \\& \quad =r_{1}\underline{\phi}_{1}(\xi) \biggl[ 1- \underline{\phi}_{1}( \xi)-b_{1} \int_{-\tau }^{0}\overline{\phi}_{1}(\xi +cs)\,d\eta_{11}(s)-c _{1} \int_{-\tau }^{0}\overline{\phi}_{2}(\xi +cs)\,d\eta_{12}(s) \biggr] \\& \quad \geq r_{1}\underline{\phi}_{1}(\xi)-r_{1}\underline{\phi}_{1} ^{2}(\xi)-r_{1}b_{1}\underline{\phi}_{1}(\xi) \overline{\phi}_{1}( \xi)-2r_{1}c_{1}e^{\lambda_{2}\xi } \underline{\phi}_{1}(\xi) \\& \quad \geq r_{1}\underline{\phi}_{1}(\xi)-r_{1}(1+b_{1})e^{2\lambda_{1} \xi }-2r_{1}c_{1}e^{(\lambda_{1}+\lambda_{2})\xi } \\& \quad =r_{1}e^{\lambda_{1}\xi }-r_{1}q_{1}e^{\eta \lambda_{1}\xi }-r_{1}(1+b _{1})e^{2\lambda_{1}\xi }-2r_{1}c_{1}e^{(\lambda_{1}+\lambda_{2}) \xi }. \end{aligned}$$

By what we have done, (2.6) is true once

$$\begin{aligned}& d_{1}[J_{1}\ast \underline{\phi}_{1}](\xi)-c\underline{\phi}_{1} ^{\prime }(\xi)+r_{1}e^{\lambda_{1}\xi }-r_{1}q_{1}e^{\eta \lambda_{1} \xi }-r_{1}(1+b_{1})e^{2\lambda_{1}\xi }-2r_{1}c_{1}e^{(\lambda_{1}+ \lambda_{2})\xi } \\& \quad \geq d_{1} \biggl[ \int_{\mathbb{R}}J_{1}(y) \bigl( e^{\lambda_{1}( \xi -y)}-qe^{\eta \lambda_{1}(\xi -y)} \bigr)\,dy- \bigl( e^{\lambda _{1}\xi }-qe^{\eta \lambda_{1}\xi } \bigr) \biggr] \\& \qquad {}- \bigl( c\lambda_{1}e^{\lambda_{1}\xi }-cq\eta \lambda_{1}e^{\eta \lambda_{1}\xi } \bigr) +r_{1}e^{\lambda_{1}\xi }-r_{1}qe^{\eta \lambda_{1}\xi } \\& \qquad {}-r_{1}(1+b_{1})e^{2\lambda_{1}\xi }-2r_{1}c_{1}e^{(\lambda_{1}+ \lambda_{2})\xi } \\& \quad =-qe^{\eta \lambda_{1}\xi } \biggl\{ d_{1} \biggl[ \int_{\mathbb{R}}J _{1}(y)e^{\eta \lambda_{1}y}\,dy-1 \biggr] -c \eta \lambda_{1}+r_{1} \biggr\} \\& \qquad {}-r_{1}(1+b_{1})e^{2\lambda_{1}\xi }-2r_{1}c_{1}e^{(\lambda_{1}+ \lambda_{2})\xi } \\& \quad =-q\Theta_{1}(\eta \lambda_{1},c)e^{\eta \lambda_{1}\xi }-r_{1}(1+b _{1})e^{2\lambda_{1}\xi }-2r_{1}c_{1}e^{(\lambda_{1}+\lambda_{2}) \xi } \\& \quad \geq 0. \end{aligned}$$
(3.2)

Let

$$q>-\frac{r_{1}(1+b_{1})+2r_{1}c_{1}}{\Theta_{1}(\eta \lambda_{1},c)}+q _{1}:=q_{2}, $$

then (3.2) holds since \(\xi <0\) and

$$e^{\eta \lambda_{1}\xi }>e^{2\lambda_{1}\xi }>0,\qquad e^{\eta \lambda_{1} \xi }>e^{(\lambda_{1}+\lambda_{2})\xi }>0. $$

The verification of (2.6) is finished.

We now consider (2.8), which is clear if \(\underline{\phi}_{2}( \xi)=0>e^{\lambda_{2}\xi }-qe^{\eta \lambda_{2}\xi }\). If \(\underline{ \phi }_{2}(\xi)=e^{\lambda_{2}\xi }-qe^{\eta \lambda_{2}\xi }>0\), we first select \(q_{3}\geq q_{2}\) implies

$$\overline{\phi}_{2}(\xi)< 2e^{\lambda_{2}\xi } $$

for any \(q\geq q_{3}\), which is admissible for fixed \(p=p_{1}\). Then

$$\begin{aligned}& r_{2}\underline{\phi}_{2}(\xi)H_{2}( \underline{\phi}_{1},\overline{ \phi }_{2},\underline{\phi}_{2}) (\xi) \\& \quad =r_{2}\underline{\phi}_{2}(\xi) \biggl[ 1-\underline{ \phi }_{2}( \xi)-b_{2} \int_{-\tau }^{0}\overline{\phi}_{2}(\xi +cs)\,d\eta_{22}(s)+c _{2} \int_{-\tau }^{0}\underline{\phi}_{1}(\xi +cs)\,d\eta_{21}(s) \biggr] \\& \quad \geq r_{2}\underline{\phi}_{2}(\xi) \biggl[ 1- \underline{\phi}_{2}( \xi)-b_{2} \int_{-\tau }^{0}\overline{\phi}_{2}(\xi +cs)\,d\eta_{22}(s) \biggr] \\& \quad \geq r_{2}\underline{\phi}_{2}(\xi) \bigl[ 1-e^{\lambda_{2}\xi }-2b _{2}e^{\lambda_{2}\xi } \bigr] \\& \quad =r_{2}\underline{\phi}_{2}(\xi)-r_{2} ( 1+2b_{2} ) \underline{ \phi }_{2}(\xi)e^{\lambda_{2}\xi } \\& \quad \geq r_{2} \bigl( e^{\lambda_{2}\xi }-qe^{\eta \lambda_{2}\xi } \bigr) -r_{2} ( 1+2b_{2} ) e^{2\lambda_{2}\xi }. \end{aligned}$$

Therefore, if

$$q>q_{3}-\frac{r_{2} ( 1+2b_{2} ) }{\Theta_{2}(\eta \lambda _{2},c)}:=q_{4}, $$

then (2.8) holds since

$$\begin{aligned}& d_{2}[J_{2}\ast \underline{\phi}_{2}](\xi)-c \underline{\phi}_{2} ^{\prime }(\xi)+r_{2}\underline{ \phi }_{2}(\xi)H_{2}(\underline{ \phi }_{1}, \overline{\phi}_{2},\underline{\phi}_{2}) (\xi) \\& \quad \geq d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y)\bigl[e^{\lambda_{2}(\xi -y)}-qe ^{\eta \lambda_{2}(\xi -y)}\bigr]\,dy- \bigl( e^{\lambda_{2}\xi }-qe^{\eta \lambda_{2}\xi } \bigr) \biggr] \\& \qquad {}-c \bigl( \lambda_{2}e^{\lambda_{2}\xi }-q\eta \lambda_{2}e^{\eta \lambda_{2}\xi } \bigr) +r_{2} \bigl( e^{\lambda_{2}\xi }-qe^{\eta \lambda_{2}\xi } \bigr) -r_{2}(1+2b_{2})e^{2\lambda_{2}\xi } \\& \quad =-qe^{\eta \lambda_{2}\xi } \biggl\{ d_{2} \biggl[ \int_{\mathbb{R}}J _{2}(y)e^{\eta \lambda_{2}y}\,dy-1 \biggr] -c \eta \lambda_{2}+r_{2} \biggr\} -r_{2}(1+2b_{2})e^{2\lambda_{2}\xi } \\& \quad =-q\Theta_{2}(\eta \lambda_{2},c) e^{\eta \lambda_{2}\xi }-r_{2}(1+2b _{2})e^{2\lambda_{2}\xi } \\& \quad \geq 0,\quad \xi < 0. \end{aligned}$$

Summarizing what we have done, it suffices to verify that (3.1) is true. We now show \(\phi_{1}(\xi)>0\), \(\xi \in \mathbb{R}\). If \(\phi_{1}(\xi_{0})=0\), then it arrives the minimal and so \(\phi_{1}'( \xi_{0})=0\), which further implies that

$$\int_{\mathbb{R}}J_{1}(y){\phi}_{1}( \xi_{0} -y)\,dy=0. $$

Therefore, \(\phi_{1}(\xi)=0\) on an interval. Repeating the process, we see that \(\phi_{1}(\xi)=0\), \(\xi \in \mathbb{R}\). A contradiction occurs since \(\underline{\phi}_{1}(\xi)>0\) if −ξ is large. Similarly, we can verify (3.1). The proof is complete. □

Theorem 3.2

Assume that \(c^{\ast }=c_{1}^{\ast }>c_{2}^{\ast }\). Further suppose that \(k_{1}(y)\) admits compact support. Then (2.1) with \(c=c^{*}\) has a positive solution \((\phi_{1}(\xi),\phi_{2}(\xi))\) such that

$$0< \phi_{1}(\xi)< 1,0< \phi_{2}(\xi)< 1+c_{2},\xi \in \mathbb{R}, \quad \lim_{\xi \rightarrow -\infty }\bigl(\phi_{1}(\xi), \phi_{2}(\xi)\bigr)=(0,0) $$

and

$$\phi_{1}(\xi)\sim \mathcal{O}\bigl(-\xi e^{\lambda_{1}^{\ast }\xi } \bigr),\qquad \phi_{2}(\xi)\sim \mathcal{O} \bigl(e^{\lambda_{2}\xi }\bigr),\quad \xi \rightarrow -\infty. $$

Proof

By Lemma 2.3, \(\Theta_{1}(\lambda,c^{\ast })\) arrives at its minimum when \(\lambda =\lambda_{1}^{\ast }\), and so

$$d_{1} \int_{\mathbb{R}}J_{1}(y)ye^{\lambda_{1}^{\ast }y}\,dy=c^{\ast }. $$

Let \(S>0\) be a constant such that \(k_{1}(y)=0\), \(\vert y\vert >S\). Moreover, let \(\eta >1\) such that

$$\lambda_{1}^{\ast }/2+\lambda_{2}-\eta \lambda_{2}>0, \qquad \Theta_{2}\bigl( \eta \lambda_{2},c^{\ast } \bigr)< 0. $$

Consider the continuous function \(-L\xi e^{\lambda_{1}^{\ast }\xi }\), \(\xi <0\), where \(L>0\) is a constant. Clearly, if \(L>1\) is large, then

$$ \max_{\xi < 0} \bigl\{ -L\xi e^{\lambda_{1}^{\ast }\xi } \bigr\} >1,\qquad \xi_{2}- \xi_{1}>2S+c^{\ast }\tau, $$
(3.3)

where \(\xi_{2}\), \(\xi_{1}\) with \(\xi_{2}-\xi_{1}>0\) are two roots of \(-L\xi e^{\lambda_{1}^{\ast }\xi }=1\). Moreover, let \(q>L\) be a constant clarified later, then there exists \(\xi_{3}=-q^{2}/L^{2}<-1\) such that

$$( -L\xi -q\sqrt{-\xi } ) e^{\lambda_{1}^{\ast }\xi }>0, \quad \xi < \xi_{3}. $$

By the above constants, define the continuous functions

$$\begin{aligned}& \underline{\phi}_{1}(\xi)=\textstyle\begin{cases} ( -L\xi -q\sqrt{-\xi } ) e^{\lambda_{1}^{\ast }\xi },& \xi < \xi_{3}, \\ 0,& \xi \geq \xi _{3}, \end{cases}\displaystyle \\& \overline{\phi}_{1}(\xi)=\textstyle\begin{cases} -L\xi e^{\lambda_{1}^{\ast }\xi },&\xi < \xi_{1}, \\ 1,& \xi \geq \xi_{1}, \end{cases}\displaystyle \end{aligned}$$

and

$$\underline{\phi}_{2}(\xi)=\max \bigl\{ e^{\lambda_{2}\xi }-qe^{\eta \lambda_{2}\xi },0 \bigr\} ,\qquad \overline{\phi}_{2}(\xi)=\min \bigl\{ e^{\lambda_{2} \xi }+pe^{\eta \lambda_{2}\xi },1+c_{2} \bigr\} , $$

where \(p>1\), \(q>1\) are constants, of which the definition will be further illustrated later. We now show these functions satisfy (2.5)–(2.8) if they are differentiable.

If \(\overline{\phi}_{1}(\xi)=1\), then \(H_{1}(\overline{\phi}_{1},\underline{ \phi }_{1},\underline{\phi}_{2})(\xi)\leq 0\) such that (2.5) is clear. Otherwise, \(\overline{\phi}_{1}(\xi)=-L\xi e^{\lambda_{1} ^{\ast }\xi }<1\) implies that

$$r_{1}\overline{\phi}_{1}(\xi)H_{1}( \overline{\phi}_{1},\underline{ \phi }_{1},\underline{ \phi }_{2}) (\xi)\leq r_{1}\overline{\phi}_{1}( \xi)=-r_{1}L\xi e^{\lambda_{1}^{\ast }\xi }, \quad \xi < \xi_{1}, $$

and (3.3) indicates that

$$\begin{aligned}& d_{1}[J_{1}\ast \overline{\phi}_{1}](\xi)-c^{\ast }\overline{ \phi }_{1}^{\prime }(\xi)+r_{1}\overline{\phi}_{1}(\xi)H_{1}( \overline{ \phi }_{1},\underline{\phi}_{1},\underline{ \phi }_{2}) (\xi) \\& \quad \leq d_{1}[J_{1}\ast \overline{\phi}_{1}](\xi)-c^{\ast }\overline{ \phi }_{1}^{\prime }( \xi)+r_{1}\overline{\phi}_{1}(\xi) \\& \quad \leq -d_{1}L \biggl[ \int_{\mathbb{R}}J_{1}(y) (\xi -y)e^{\lambda_{1} ^{\ast }(\xi -y)}\,dy-\xi e^{\lambda_{1}^{\ast }\xi } \biggr] \\& \qquad {}+c^{\ast }Le^{\lambda_{1}^{\ast }\xi }+c^{\ast }\lambda_{1}^{\ast }L \xi e^{\lambda_{1}^{\ast }\xi }-r_{1}L\xi e^{\lambda_{1}^{\ast } \xi } \\& \quad =-d_{1}L \biggl[ \xi \int_{\mathbb{R}}J_{1}(y)e^{\lambda_{1}^{\ast }( \xi -y)}\,dy-\xi e^{\lambda_{1}^{\ast }\xi }- \int_{\mathbb{R}}J_{1}(y)ye ^{\lambda_{1}^{\ast }(\xi -y)}\,dy \biggr] \\& \qquad {}+c^{\ast }Le^{\lambda_{1}^{\ast }\xi }+c^{\ast }\lambda_{1}^{\ast }L \xi e^{\lambda_{1}^{\ast }\xi }-r_{1}L\xi e^{\lambda_{1}^{\ast } \xi } \\& \quad =-L\xi e^{\lambda_{1}^{\ast }\xi } \biggl\{ d_{1} \biggl[ \int_{\mathbb{R}}J_{1}(y)e^{\lambda_{1}^{\ast }y}\,dy-1 \biggr] -c^{ \ast }\lambda_{1}^{\ast }+r_{1} \biggr\} \\& \qquad {}+d_{1}Le^{\lambda_{1}^{\ast }\xi } \biggl[ \int_{\mathbb{R}}J_{1}(y)ye ^{-\lambda_{1}^{\ast }y}\,dy \biggr] +c^{\ast }Le^{\lambda_{1}^{\ast } \xi } \\& \quad =0, \end{aligned}$$

which implies what we wanted.

If \(\overline{\phi}_{2}(\xi)=1+c_{2}< e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }\), then \(H_{2}(\overline{\phi}_{1},\underline{\phi} _{2},\overline{\phi}_{2})(\xi)\leq 0\) such that (2.7) is clear. Otherwise, let \(p_{2}>0\) such that \(\overline{\phi}_{2}(\xi)=e^{ \lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }<1+c_{2}\) with \(p\geq p_{2}\) implies that

$$\overline{\phi}_{1}(\xi)< e^{\lambda_{1}^{\ast }\xi /2}, $$

which is evident by simple limit analysis. Thus, the monotonicity implies

$$\begin{aligned}& r_{2}\overline{\phi}_{2}(\xi)H_{2}(\overline{ \phi }_{1},\underline{ \phi }_{2},\overline{\phi}_{2}) (\xi) \\& \quad =r_{2}\overline{\phi}_{2}(\xi) \biggl[ 1-\overline{\phi}_{2}( \xi)-b_{2} \int_{-\tau }^{0}\underline{\phi}_{2}\bigl( \xi +c^{\ast }s\bigr)\,d\eta_{22}(s)+c_{2} \int_{-\tau }^{0}\overline{\phi}_{1}\bigl( \xi +c^{ \ast }s\bigr)\,d\eta_{21}(s) \biggr] \\& \quad \leq r_{2}\overline{\phi}_{2}(\xi) \biggl[ 1+c_{2} \int_{-\tau }^{0}\overline{ \phi }_{1}\bigl( \xi +c^{\ast }s\bigr)\,d\eta_{21}(s) \biggr] \\& \quad \leq r_{2}\overline{\phi}_{2}(\xi) \bigl[ 1+c_{2}e^{\lambda_{1} ^{\ast }\xi /2} \bigr] \\& \quad =r_{2}\bigl[e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }\bigr] \bigl[ 1+c_{2}e ^{\lambda_{1}^{\ast }\xi /2} \bigr] \\& \quad =r_{2}\bigl[e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }\bigr]+r_{2}c_{2}e ^{\lambda_{1}^{\ast }\xi /2}\bigl[e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2} \xi }\bigr] \end{aligned}$$

and

$$\begin{aligned}& d_{2}[J_{2}\ast \overline{\phi}_{2}](\xi)-c^{\ast }\overline{ \phi }_{2}^{\prime }(\xi)+r_{2}\overline{\phi}_{2}(\xi)H_{2}(\overline{ \phi }_{1},\underline{\phi}_{2},\overline{\phi}_{2}) (\xi) \\& \quad =d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y)\overline{\phi}_{2}(\xi -y)\,dy- \bigl( e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi } \bigr) \biggr] \\& \qquad {}-c^{\ast } \bigl( \lambda_{2}e^{\lambda_{2}\xi }+p\eta \lambda_{2}e ^{\eta \lambda_{2}\xi } \bigr) +r_{2}\overline{\phi}_{2}(\xi)H_{2}(\overline{ \phi }_{1}, \underline{\phi}_{2},\overline{\phi}_{2}) (\xi) \\& \quad \leq d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y)\bigl[e^{\lambda_{2}(\xi -y)}+pe ^{\eta \lambda_{2}(\xi -y)}\bigr]\,dy- \bigl( e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi } \bigr) \biggr] \\& \qquad {}-c^{\ast } \bigl( \lambda_{2}e^{\lambda_{2}\xi }+p\eta \lambda_{2}e ^{\eta \lambda_{2}\xi } \bigr) +r_{2}\overline{\phi}_{2}(\xi)H_{2}(\overline{ \phi }_{1}, \underline{\phi}_{2},\overline{\phi}_{2}) (\xi) \\& \quad \leq d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y)\bigl[e^{\lambda_{2}(\xi -y)}+pe ^{\eta \lambda_{2}(\xi -y)}\bigr]\,dy- \bigl( e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi } \bigr) \biggr] \\& \qquad {}-c^{\ast } \bigl( \lambda_{2}e^{\lambda_{2}\xi }+p\eta \lambda_{2}e ^{\eta \lambda_{2}\xi } \bigr) +r_{2} \bigl[e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }\bigr]+r_{2}c_{2}^{\ast }e^{\lambda_{1}^{\ast }\xi /2} \bigl[e ^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }\bigr] \\& \quad =pe^{\eta \lambda_{2}\xi } \biggl\{ d_{2} \biggl[ \int_{\mathbb{R}}J _{2}(y)e^{\eta \lambda_{2}y}\,dy-1 \biggr] -c^{\ast }\eta \lambda_{2}+r _{2} \biggr\} +r_{2}c_{2}e^{\lambda_{1}^{\ast }\xi /2}\bigl[e^{\lambda_{2} \xi }+pe^{\eta \lambda_{2}\xi } \bigr] \\& \quad =p\Theta_{2}\bigl(\eta \lambda_{2},c^{\ast } \bigr)e^{\eta \lambda_{2}\xi }+r _{2}c_{2}e^{\lambda_{1}^{\ast }\xi /2} \bigl[e^{\lambda_{2}\xi }+pe^{\eta \lambda_{2}\xi }\bigr] \\& \quad =e^{\eta \lambda_{2}\xi } \bigl[ p\Theta_{2}\bigl(\eta \lambda_{2},c^{ \ast } \bigr)/2+r_{2}c_{2}e^{(\lambda_{1}^{\ast }/2+\lambda_{2}-\eta \lambda _{2})\xi } \bigr] \\& \qquad {}+pe^{\eta \lambda_{2}\xi } \bigl[ \Theta_{2}\bigl(\eta \lambda_{2},c^{ \ast } \bigr)/2+r_{2}c_{2}e^{\lambda_{1}^{\ast }\xi /2} \bigr] . \end{aligned}$$

Note that

$$\eta \lambda_{2}\xi < \ln \frac{1+c_{2}}{p}, $$

then there exists \(p_{3}>p_{2}+1+c_{2}\) such that \(p\ge p_{3}\) leads to

$$\begin{aligned}& p\Theta_{2}\bigl(\eta \lambda_{2},c^{\ast } \bigr)/2+r_{2}c_{2}e^{(\lambda_{1}/2+ \lambda_{2}-\eta \lambda_{2})\xi }< 0, \\& \Theta_{2} \bigl(\eta \lambda_{2},c ^{\ast }\bigr)/2+r_{2}c_{2}e^{\lambda_{1}^{\ast }\xi /2}< 0 \end{aligned}$$

since \(\lambda_{1}^{\ast }/2+\lambda_{2}-\eta \lambda_{2}>0\), \(\xi <0\) and \(\Theta_{2}(\eta \lambda_{2},c^{\ast })<0\) is a constant. Now, we fix it by \(p=p_{3}\).

When \(\underline{\phi}_{1}(\xi)=0\) with \(\xi <\xi_{3}\), then \(H_{1}(\underline{\phi}_{1},\overline{\phi}_{1},\overline{\phi} _{2})(\xi)=0\) such that (2.6) is clear. Otherwise, if \(\xi \ge \xi_{3}\), then \(\underline{\phi}_{1}(\xi)= ( -L\xi -q\sqrt{- \xi } ) e^{\lambda_{1}^{\ast }\xi }>0\). Firstly, let \(q>q_{1}>1\) such that \(-L\xi -q\sqrt{-\xi }>0\) implies \(\xi <0\) and

$$\overline{\phi}_{2}(\xi)< 2e^{\lambda_{2}\xi },\qquad \underline{\phi}_{1}( \xi)\leq \overline{\phi}_{1}(\xi)< e^{\theta \lambda_{1}^{\ast } \xi } $$

for some \(\theta \in {}[ \frac{2}{3},1)\) with \(\theta \lambda_{1} ^{\ast }+\lambda_{2}>\lambda_{1}^{\ast }\), which is admissible once p is fixed. Therefore, \(q>q_{1}\) indicates

$$\begin{aligned}& r_{1}\underline{\phi}_{1}(\xi)H_{1}( \underline{\phi}_{1},\overline{ \phi }_{1},\overline{\phi}_{2}) (\xi) \\& \quad =r_{1}\underline{\phi}_{1}(\xi) \biggl[ 1- \underline{\phi}_{1}( \xi)-b_{1} \int_{-\tau }^{0}\overline{\phi}_{1}\bigl( \xi +c^{\ast }s\bigr)\,d\eta_{11}(s)-c_{1} \int_{-\tau }^{0}\overline{\phi}_{2}\bigl(\xi +c^{ \ast }s\bigr)\,d\eta_{12}(s) \biggr] \\& \quad \geq r_{1}\underline{\phi}_{1}(\xi)-r_{1}\underline{\phi}_{1} ^{2}(\xi)-r_{1}b_{1}\underline{\phi}_{1}(\xi) \overline{\phi}_{1}( \xi)-2r_{1}c_{1}e^{\lambda_{2}\xi } \underline{\phi}_{1}(\xi) \\& \quad \geq r_{1}\underline{\phi}_{1}(\xi)-r_{1}(1+b_{1})e^{2\theta \lambda_{1}^{\ast }\xi }-2r_{1}c_{1}e^{(\theta \lambda_{1}^{\ast }+ \lambda_{2})\xi } \\& \quad =r_{1} ( -L\xi -q\sqrt{-\xi } ) e^{\lambda_{1}^{\ast } \xi }-r_{1}(1+b_{1})e^{2\theta \lambda_{1}^{\ast }\xi }-2r_{1}c_{1}e ^{(\theta \lambda_{1}^{\ast }+\lambda_{2})\xi }. \end{aligned}$$

Moreover, (3.3) leads to

$$\begin{aligned}& d_{1}[J_{1}\ast \underline{\phi}_{1}](\xi)-c^{\ast }\underline{ \phi }_{1}^{\prime }(\xi) \\& \quad =d_{1} \biggl[ \int_{\mathbb{R}}J_{1}(y)\underline{\phi}_{1}(\xi -y)\,dy-\underline{ \phi }_{1}(\xi) \biggr] -c^{\ast }\underline{\phi}_{1}^{\prime }( \xi) \\& \quad \geq d_{1} \biggl\{ \int_{\mathbb{R}}J_{1}(y) \bigl[ \bigl( -L(\xi -y)-q \sqrt{-( \xi -y)} \bigr) e^{\lambda_{1}^{\ast }(\xi -y)} \bigr]\,dy \\& \qquad {} - ( -L\xi -q\sqrt{-\xi } ) e^{\lambda_{1}^{\ast } \xi } \biggr\} -c^{\ast } \bigl[ ( -L\xi -q\sqrt{-\xi } ) e ^{\lambda_{1}^{\ast }\xi } \bigr] ^{\prime } \\& \quad =d_{1}e^{\lambda_{1}^{\ast }\xi } \biggl[ \int_{\mathbb{R}}J_{1}(y) \bigl[ \bigl( -L(\xi -y) \bigr) e^{-\lambda_{1}^{\ast }y} \bigr]\,dy+L \xi \biggr] \\& \qquad {}-qd_{1}e^{\lambda_{1}^{\ast }\xi } \biggl[ \int_{\mathbb{R}}J_{1}(y)\sqrt{-( \xi -y)}e^{-\lambda_{1}^{\ast }y}\,dy- \sqrt{-\xi } \biggr] \\& \qquad {}+c^{\ast }L \bigl( 1+\lambda_{1}^{\ast }\xi \bigr) e^{\lambda_{1} ^{\ast }\xi }+c^{\ast }q \biggl( \lambda_{1}^{\ast } \sqrt{-\xi }-\frac{1}{2\sqrt{- \xi }} \biggr) e^{\lambda_{1}^{\ast }\xi }. \end{aligned}$$

By what we have done, (2.6) is true if

$$\begin{aligned}& d_{1}[J_{1}\ast \underline{\phi}_{1}](\xi)-c^{\ast }\underline{ \phi }_{1}^{\prime }(\xi)+r_{1}\underline{\phi}_{1}(\xi)H_{1}( \underline{ \phi }_{1},\overline{\phi}_{1},\overline{\phi}_{2}) (\xi) \\& \quad \geq d_{1}e^{\lambda_{1}^{\ast }\xi } \biggl[ \int_{\mathbb{R}}J_{1}(y) \bigl[ \bigl( -L(\xi -y) \bigr) e^{-\lambda_{1}^{\ast }y} \bigr]\,dy+L \xi \biggr] \\& \qquad {}-qd_{1}e^{\lambda_{1}^{\ast }\xi } \biggl[ \int_{\mathbb{R}}J_{1}(y)\sqrt{-( \xi -y)}e^{-\lambda_{1}^{\ast }y}\,dy- \sqrt{-\xi } \biggr] \\& \qquad {}+c^{\ast }L \bigl( 1+\lambda_{1}^{\ast }\xi \bigr) e^{\lambda_{1} ^{\ast }\xi }+c^{\ast }q \biggl( \lambda_{1}^{\ast } \sqrt{-\xi }-\frac{1}{2\sqrt{- \xi }} \biggr) e^{\lambda_{1}^{\ast }\xi } \\& \qquad {}+r_{1} ( -L\xi -q\sqrt{-\xi } ) e^{\lambda_{1}^{\ast } \xi }-r_{1}(1+b_{1})e^{2\theta \lambda_{1}^{\ast }\xi }-2r_{1}c_{1}e ^{(\theta \lambda_{1}^{\ast }+\lambda_{2})\xi } \\& \quad =-L\xi e^{\lambda_{1}^{\ast }\xi } \biggl\{ d_{1} \biggl[ \int_{\mathbb{R}}J_{1}(y)e^{-\lambda_{1}^{\ast }y}\,dy-1 \biggr] -c^{ \ast }\lambda_{1}^{\ast }+r_{1} \biggr\} \\& \qquad {}+d_{1}Le^{\lambda_{1}^{\ast }\xi } \biggl[ \int_{\mathbb{R}}J_{1}(y)ye ^{-\lambda_{1}^{\ast }y}\,dy+c^{\ast } \biggr] -qd_{1}e^{\lambda_{1}^{ \ast }\xi } \biggl[ \int_{\mathbb{R}}J_{1}(y)\sqrt{-(\xi -y)}e^{- \lambda_{1}^{\ast }y}\,dy- \sqrt{-\xi } \biggr] \\& \qquad {}+c^{\ast }q \biggl( \lambda_{1}^{\ast }\sqrt{-\xi }- \frac{1}{2\sqrt{- \xi }} \biggr) e^{\lambda_{1}^{\ast }\xi }-r_{1}q\sqrt{-\xi }e^{ \lambda_{1}^{\ast }\xi }-r_{1}(1+b_{1})e^{2\theta \lambda_{1}^{\ast } \xi }-2r_{1}c_{1}e^{(\theta \lambda_{1}^{\ast }+\lambda_{2})\xi } \\& \quad =-qd_{1}e^{\lambda_{1}^{\ast }\xi } \biggl[ \int_{\mathbb{R}}J_{1}(y)\sqrt{-( \xi -y)}e^{-\lambda_{1}^{\ast }y}\,dy- \sqrt{-\xi } \biggr] +c^{\ast }q \biggl( \lambda_{1}^{\ast } \sqrt{-\xi }-\frac{1}{2\sqrt{-\xi }} \biggr) e ^{\lambda_{1}^{\ast }\xi } \\& \qquad {}-r_{1}q\sqrt{-\xi }e^{\lambda_{1}^{\ast }\xi }-r_{1}(1+b_{1})e ^{2\theta \lambda_{1}^{\ast }\xi }-2r_{1}c_{1}e^{(\theta \lambda_{1} ^{\ast }+\lambda_{2})\xi } \\& \quad =e^{\lambda_{1}^{\ast }\xi } \biggl\{ -qd_{1} \biggl[ \int_{\mathbb{R}}J _{1}(y)\sqrt{-(\xi -y)}e^{-\lambda_{1}^{\ast }y}\,dy- \sqrt{-\xi } \biggr] +c^{\ast }q \biggl( \lambda_{1}^{\ast } \sqrt{-\xi }-\frac{1}{2\sqrt{- \xi }} \biggr) \\& \qquad {} -r_{1}q\sqrt{-\xi }-r_{1}(1+b_{1})e^{(2\theta -1)\lambda _{1}^{\ast }\xi }-2r_{1}c_{1}e^{(\theta \lambda_{1}^{\ast }+\lambda _{2}-\lambda_{1}^{\ast })\xi } \biggr\} \\& \quad \geq 0 \end{aligned}$$

or

$$\begin{aligned}& q \biggl\{ -d_{1} \biggl[ \int_{\mathbb{R}}J_{1}(y)\sqrt{-(\xi -y)}e ^{-\lambda_{1}^{\ast }y}\,dy-\sqrt{-\xi } \biggr] +c^{\ast } \biggl( \lambda_{1}^{\ast }\sqrt{-\xi }-\frac{1}{2\sqrt{-\xi }} \biggr) -r _{1}\sqrt{-\xi } \biggr\} \\& \quad \geq r_{1}(1+b_{1})e^{(2\theta -1)\lambda_{1}^{\ast }\xi }+2r_{1}c _{1}e^{(\theta \lambda_{1}^{\ast }+\lambda_{2}-\lambda_{1}^{\ast }) \xi }. \end{aligned}$$

We first analyze the left of the above inequality

$$\begin{aligned}& -d_{1} \biggl[ \int_{\mathbb{R}}J_{1}(y)\sqrt{-(\xi -y)}e^{-\lambda_{1} ^{\ast }y}\,dy- \sqrt{-\xi } \biggr] +c^{\ast } \biggl( \lambda_{1}^{ \ast } \sqrt{-\xi }-\frac{1}{2\sqrt{-\xi }} \biggr) -r_{1}\sqrt{- \xi } \\& \quad =-d_{1} \biggl\{ \int_{\mathbb{R}}J_{1}(y) \bigl[ \sqrt{-\xi }+\sqrt{-( \xi -y)}-\sqrt{-\xi } \bigr] e^{-\lambda_{1}^{\ast }y}\,dy-\sqrt{- \xi } \biggr\} \\& \qquad {}+c^{\ast } \biggl( \lambda_{1}^{\ast }\sqrt{-\xi }- \frac{1}{2\sqrt{- \xi }} \biggr) -r_{1}\sqrt{-\xi } \\& \quad =-d_{1} \biggl\{ \int_{\mathbb{R}}J_{1}(y) \bigl[ \sqrt{-(\xi -y)}-\sqrt{- \xi } \bigr] e^{-\lambda_{1}^{\ast }y}\,dy \biggr\} -c^{\ast }\frac{1}{2\sqrt{- \xi }} \\& \quad =d_{1} \biggl\{ \int_{\mathbb{R}}J_{1}(y) \bigl[ \sqrt{-\xi }-\sqrt{-( \xi -y)} \bigr] e^{-\lambda_{1}^{\ast }y}\,dy \biggr\} -\frac{d_{1}}{2\sqrt{- \xi }} \int_{\mathbb{R}}J_{1}(y)ye^{\lambda_{1}^{\ast }y}\,dy \\& \quad =d_{1} \int_{\mathbb{R}}J_{1}(y) \biggl[ \frac{y}{2\sqrt{-\xi }}+ \sqrt{- \xi }-\sqrt{-(\xi -y)} \biggr] e^{-\lambda_{1}^{\ast }y}\,dy \\& \quad =d_{1} \int_{\mathbb{R}}J_{1}(y) \biggl[ \frac{y}{2\sqrt{-\xi }}- \frac{y}{\sqrt{- \xi }+\sqrt{-(\xi -y)}} \biggr] e^{-\lambda_{1}^{\ast }y}\,dy \\& \quad =d_{1} \int_{\mathbb{R}}J_{1}(y) \biggl[ \frac{y}{2\sqrt{-\xi }}- \frac{y}{\sqrt{- \xi }+\sqrt{-(\xi -y)}} \biggr] e^{-\lambda_{1}^{\ast }y}\,dy \\& \quad =d_{1} \int_{\mathbb{R}}J_{1}(y) \biggl[ \frac{y [ \sqrt{-( \xi -y)}-\sqrt{-\xi } ] }{2\sqrt{-\xi } [ \sqrt{- \xi }+\sqrt{-(\xi -y)} ] } \biggr] e^{-\lambda_{1}^{\ast }y}\,dy \\& \quad =d_{1} \int_{\mathbb{R}}J_{1}(y) \biggl[ \frac{y^{2}}{2\sqrt{-\xi } [ \sqrt{-\xi }+\sqrt{-(\xi -y)} ] ^{2}} \biggr] e^{- \lambda_{1}^{\ast }y}\,dy \\& \quad \geq d_{1} \int_{\mathbb{R}}J_{1}(y) \biggl[ \frac{y^{2}}{2\sqrt{-( \xi -S)} [ \sqrt{-(\xi -S)}+\sqrt{-(\xi -S)} ] ^{2}} \biggr] e^{-\lambda_{1}^{\ast }y}\,dy \\& \quad =\frac{d_{1}}{8 [ -(\xi -S) ] ^{3/2}} \int_{\mathbb{R}}J _{1}(y)y^{2}e^{-\lambda_{1}^{\ast }y}\,dy. \end{aligned}$$

Let

$$q\geq \frac{\max_{\xi < 0} \{ 8 [ -(\xi -S) ] ^{3/2} [ r_{1}(1+b_{1})e^{(2\theta -1)\lambda_{1}^{\ast }\xi }+2r_{1}c_{1}e ^{(\theta \lambda_{1}+\lambda_{2}-\lambda_{1}^{\ast })\xi } ] \} }{d_{1}\int_{\mathbb{R}}J_{1}(y)y^{2}e^{-\lambda_{1}^{\ast }y}\,dy}+q _{1}:=q_{2}, $$

then (3.2) holds since \(\xi <0\) and

$$(2\theta -1)\lambda_{1}^{\ast }>0,\qquad \theta \lambda_{1}^{\ast }+ \lambda_{2}- \lambda_{1}^{\ast }>0. $$

The verification of (2.7) is finished.

We now consider (2.8), which is clear if \(\underline{\phi}_{2}( \xi)=0>e^{\lambda_{2}\xi }-qe^{\eta \lambda_{2}\xi }\). If \(\underline{ \phi }_{2}(\xi)=e^{\lambda_{2}\xi }-qe^{\eta \lambda_{2}\xi }>0\), we first select \(q_{3}\geq q_{2}\) such that \(\underline{\phi}_{2}( \xi)>0\) implies

$$\overline{\phi}_{2}(\xi)< 2e^{\lambda_{2}\xi } $$

for any \(q\geq q_{3}\), which is admissible for fixed \(p=p_{1}\). Then

$$\begin{aligned}& r_{2}\underline{\phi}_{2}(\xi)H_{2}( \underline{\phi}_{1},\overline{ \phi }_{2},\underline{\phi}_{2}) (\xi) \\& \quad =r_{2}\underline{\phi}_{2}(\xi) \biggl[ 1-\underline{ \phi }_{2}( \xi)-b_{2} \int_{-\tau }^{0}\overline{\phi}_{2}\bigl(\xi +c^{\ast }s\bigr)\,d\eta_{22}(s)+c_{2} \int_{-\tau }^{0}\underline{\phi}_{1}\bigl( \xi +c^{ \ast }s\bigr)\,d\eta_{21}(s) \biggr] \\& \quad \geq r_{2}\underline{\phi}_{2}(\xi) \biggl[ 1- \underline{\phi}_{2}( \xi)-b_{2} \int_{-\tau }^{0}\overline{\phi}_{2}\bigl(\xi +c^{\ast }s\bigr)\,d\eta_{22}(s) \biggr] \\& \quad \geq r_{2}\underline{\phi}_{2}(\xi) \bigl[ 1-e^{\lambda_{2}\xi }-2b _{2}e^{\lambda_{2}\xi } \bigr] \\& \quad =r_{2}\underline{\phi}_{2}(\xi)-r_{2} \underline{\phi}_{2}(\xi) \bigl[ e^{\lambda_{2}\xi }+2b_{2}e^{\lambda_{2}\xi } \bigr] \\& \quad \geq r_{2} \bigl( e^{\lambda_{2}\xi }-qe^{\eta \lambda_{2}\xi } \bigr) -r_{2}e^{\lambda_{2}\xi } \bigl[ e^{\lambda_{2}\xi }+2b_{2}e^{\lambda_{2} \xi } \bigr] . \end{aligned}$$

Therefore, if

$$q>q_{3}-\frac{r_{2} ( 1+2b_{2} ) }{\Theta_{2}(\eta \lambda _{2},c^{*})}:=q_{4}, $$

then (2.8) holds since

$$\begin{aligned}& d_{2}[J_{2}\ast \underline{\phi}_{2}](\xi)-c^{\ast }\underline{ \phi }_{2}^{\prime }(\xi)+r_{2}\underline{\phi}_{2}(\xi)H_{2}(\underline{ \phi }_{1},\overline{\phi}_{2},\underline{\phi}_{2}) (\xi) \\& \quad \geq d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y)\bigl[e^{\lambda_{2}(\xi -y)}-qe ^{\eta \lambda_{2}(\xi -y)}\bigr]\,dy- \bigl( e^{\lambda_{2}\xi }-qe^{\eta \lambda_{2}\xi } \bigr) \biggr] \\& \qquad {}-c^{\ast } \bigl( \lambda_{2}e^{\lambda_{2}\xi }-q\eta \lambda_{2}e ^{\eta \lambda_{2}\xi } \bigr) +r_{2} \bigl( e^{\lambda_{2}\xi }-qe ^{\eta \lambda_{2}\xi } \bigr) -r_{2}e^{\lambda_{2}\xi } \bigl[ e^{ \lambda_{2}\xi }+2b_{2}e^{\lambda_{2}\xi } \bigr] \\& \quad =-qe^{\eta \lambda_{2}\xi } \biggl\{ d_{2} \biggl[ \int_{\mathbb{R}}J _{2}(y)e^{\eta \lambda_{2}y}\,dy-1 \biggr] -c^{\ast }\eta \lambda_{2}+r _{2} \biggr\} -r_{2}e^{\lambda_{2}\xi } \bigl[ e^{\lambda_{2}\xi }+2b _{2}e^{\lambda_{2}\xi } \bigr] \\& \quad \geq 0,\quad \xi < 0. \end{aligned}$$

By Lemma 2.4 and a discussion similar to (3.1), we complete the proof. □

Theorem 3.3

If \(c^{\ast }=c_{2}^{\ast }>c_{1}^{\ast }\). Further suppose that \(k_{2}(y)\) admits compact support. Then (2.1) with \(c=c^{\ast }\) has a positive solution \((\phi_{1}(\xi),\phi_{2}(\xi))\) such that

$$0< \phi_{1}(\xi)< 1,0< \phi_{2}(\xi)< 1+c_{2},\xi \in \mathbb{R},\quad \lim_{\xi \rightarrow -\infty }\bigl(\phi_{1}(\xi), \phi_{2}(\xi)\bigr)=(0,0), $$

and

$$\phi_{1}(\xi)\sim \mathcal{O}\bigl(e^{\lambda_{1}\xi }\bigr),\qquad \phi_{2}(\xi) \sim \mathcal{O}\bigl(-\xi e^{\lambda_{2}^{\ast }\xi }\bigr),\quad \xi \rightarrow - \infty. $$

Proof

Under the assumption, we see that

$$d_{2} \int_{\mathbb{R}}J_{2}(y)ye^{\lambda_{2}^{\ast }y}\,dy=c^{\ast } $$

by Lemma 2.3. Let \(S>0\) be a constant such that \(k_{2}(y)=0\), \(\vert y\vert >S\). Select a constant \(\eta >1\) such that

$$\lambda_{2}^{\ast }/2+\lambda_{1}-\eta \lambda_{1}>0, \qquad \Theta_{1}\bigl( \eta \lambda_{1},c^{\ast } \bigr)< 0. $$

Let \(L>1\) be large enough such that

$$-L\xi e^{\lambda_{2}^{\ast }\xi }=1+c_{2} $$

has two real roots \(\xi_{5}< \xi_{6}\) and \(\xi_{6}-\xi_{5}>2S\).

We now define

$$\underline{\phi}_{1}(\xi)=\max \bigl\{ e^{\lambda_{1}\xi }-qe^{\eta \lambda_{1}\xi },0 \bigr\} ,\qquad \overline{\phi}_{1}(\xi)=\min \bigl\{ e^{\lambda_{1} \xi },1\bigr\} $$

and

$$\begin{aligned}& \underline{\phi}_{2}(\xi)=\textstyle\begin{cases} ( -L\xi -q\sqrt{-\xi } ) e^{\lambda_{2}^{\ast }\xi },& \xi < \xi_{3}, \\ 0, &\xi \geq \xi_{3}, \end{cases}\displaystyle \\& \overline{\phi}_{2}(\xi)= \textstyle\begin{cases} ( -L\xi +p\sqrt{-\xi } ) e^{\lambda_{2}^{\ast }\xi },& \xi < \xi_{4}, \\ 1+c_{2},& \xi \geq \xi_{4}, \end{cases}\displaystyle \end{aligned}$$

where \(\xi_{3}=L^{2}/q^{2}\) and \(\xi_{4}<\xi_{5}\) such that \(\overline{\phi}_{2}(\xi)\) is continuous.

For \(\overline{\phi}_{1}(\xi)\), the verification is similar to that in Theorem 3.1 and we omit it here. If \(\overline{\phi}_{2}( \xi)=1+c_{2}\), then \(H_{2}(\overline{\phi}_{1},\underline{\phi} _{2},\overline{\phi}_{2})(\xi)\leq 0\) such that (2.7) is clear. Otherwise, let \(p_{2}>0\) such that

$$\overline{\phi}_{2}(\xi)\ge \underline{\phi}_{2} (\xi),\quad \xi \in \mathbb{R}. $$

Thus,

$$\begin{aligned}& r_{2}\overline{\phi}_{2}(\xi)H_{2}(\overline{ \phi }_{1},\underline{ \phi }_{2},\overline{\phi}_{2}) (\xi) \\& \quad =r_{2}\overline{\phi}_{2}(\xi) \biggl[ 1-\overline{\phi}_{2}( \xi)-b_{2} \int_{-\tau }^{0}\underline{\phi}_{2}\bigl( \xi +c^{\ast }s\bigr)\,d\eta_{22}(s)+c_{2} \int_{-\tau }^{0}\overline{\phi}_{1}\bigl( \xi +c^{ \ast }s\bigr)\,d\eta_{21}(s) \biggr] \\& \quad \leq r_{2}\overline{\phi}_{2}(\xi) \biggl[ 1+c_{2} \int_{-\tau }^{0}\overline{ \phi }_{1}\bigl( \xi +c^{\ast }s\bigr)\,d\eta_{21}(s) \biggr] \\& \quad \leq r_{2}\overline{\phi}_{2}(\xi) \bigl[ 1+c_{2}e^{\lambda_{1} \xi } \bigr] \\& \quad =r_{2}e^{\lambda_{2}^{\ast }\xi } ( -L\xi +p\sqrt{-\xi } ) \bigl[ 1+c_{2}e^{\lambda_{1}\xi } \bigr] \end{aligned}$$

and

$$\begin{aligned}& d_{2}[J_{2}\ast \overline{\phi}_{2}](\xi)-c^{\ast }\overline{ \phi }_{2}^{\prime }(\xi)+r_{2}\overline{\phi}_{2}(\xi)H_{2}(\overline{ \phi }_{1},\underline{\phi}_{2},\overline{\phi}_{2}) (\xi) \\& \quad \leq d_{2} \biggl\{ \int_{\mathbb{R}}J_{2}(y) \bigl[ \bigl( -L(\xi -y)+p\sqrt{-( \xi -y)} \bigr) e^{\lambda_{2}^{\ast }(\xi -y)} \bigr]\,dy \\& \qquad {} - \bigl[ ( -L\xi +p\sqrt{-\xi } ) e^{\lambda_{2} ^{\ast }\xi } \bigr] \biggr\} \\& \qquad {}-c^{\ast }\lambda_{2}^{\ast } ( -L\xi +p\sqrt{-\xi } ) e ^{\lambda_{2}^{\ast }\xi }-c^{\ast } \biggl( -L-\frac{p}{2\sqrt{- \xi }} \biggr) e^{\lambda_{2}^{\ast }\xi } \\& \qquad {}+r_{2}e^{\lambda_{2}^{\ast }\xi } ( -L\xi +p\sqrt{-\xi } ) +r_{2}c_{2}e^{\lambda_{2}^{\ast }\xi }e^{\lambda_{1}\xi } ( -L \xi +p\sqrt{-\xi } ) \\& \quad =-L\xi e^{\lambda_{2}^{\ast }\xi } \biggl\{ d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y)e^{-\lambda_{2}^{\ast }y}\,dy-1 \biggr] -c^{ \ast }\lambda_{2}^{\ast }+r_{2} \biggr\} \\& \qquad {}+d_{2}L \biggl[ \int_{\mathbb{R}}J_{2}(y)ye^{-\lambda_{2}^{\ast }y}\,dy+c ^{\ast } \biggr] \\& \qquad {}+d_{2}pe^{\lambda_{2}^{\ast }\xi } \biggl\{ \biggl[ \int_{\mathbb{R}}J _{2}(y)\sqrt{-(\xi -y)}e^{-\lambda_{2}^{\ast }y}\,dy- \sqrt{-\xi } \biggr] -c^{\ast }\lambda_{2}^{\ast }\sqrt{-\xi }+r\sqrt{-\xi } \biggr\} \\& \qquad {}+\frac{c^{\ast }p}{2\sqrt{-\xi }}e^{\lambda_{2}^{\ast }\xi }+r _{2}c_{2}e^{\lambda_{2}^{\ast }\xi }e^{\lambda_{1}\xi } ( -L \xi +p\sqrt{-\xi } ) \\& \quad =d_{2}pe^{\lambda_{2}^{\ast }\xi } \biggl\{ \biggl[ \int_{\mathbb{R}}J _{2}(y)\sqrt{-(\xi -y)}e^{-\lambda_{2}^{\ast }y}\,dy- \sqrt{-\xi } \biggr] -c^{\ast }\lambda_{2}^{\ast }\sqrt{-\xi }+r\sqrt{-\xi } \biggr\} \\& \qquad {}+\frac{c^{\ast }p}{2\sqrt{-\xi }}e^{\lambda_{2}^{\ast }\xi }+r _{2}c_{2}e^{\lambda_{2}^{\ast }\xi }e^{\lambda_{1}\xi } ( -L \xi +p\sqrt{-\xi } ) \\& \quad =d_{2}pe^{\lambda_{2}^{\ast }\xi } \int_{\mathbb{R}}J_{2}(y) \bigl[ \sqrt{-( \xi -y)}-\sqrt{- \xi } \bigr] e^{-\lambda_{2}^{\ast }y}\,dy \\& \qquad {}+\frac{c^{\ast }p}{2\sqrt{-\xi }}e^{\lambda_{2}^{\ast }\xi }+r _{2}c_{2}e^{\lambda_{2}^{\ast }\xi }e^{\lambda_{1}\xi } ( -L \xi +p\sqrt{-\xi } ) \\& \quad =d_{2}pe^{\lambda_{2}^{\ast }\xi } \int_{\mathbb{R}}J_{2}(y) \biggl[ \frac{y}{\sqrt{-( \xi -y)}+\sqrt{-\xi }} \biggr] e^{-\lambda_{2}^{\ast }y}\,dy \\& \qquad {}+\frac{c^{\ast }p}{2\sqrt{-\xi }}e^{\lambda_{2}^{\ast }\xi }+r _{2}c_{2}e^{\lambda_{2}^{\ast }\xi }e^{\lambda_{1}\xi } ( -L \xi +p\sqrt{-\xi } ) \\& \quad =d_{2}pe^{\lambda_{2}^{\ast }\xi } \int_{\mathbb{R}}J_{2}(y) \biggl[ \frac{y}{\sqrt{-( \xi -y)}+\sqrt{-\xi }}- \frac{y}{2\sqrt{-\xi }} \biggr] e^{-\lambda_{2} ^{\ast }y}\,dy \\& \qquad {}+r_{2}c_{2}e^{\lambda_{2}^{\ast }\xi }e^{\lambda_{1}\xi } ( -L \xi +p \sqrt{-\xi } ) \\& \quad =d_{2}pe^{\lambda_{2}^{\ast }\xi } \int_{\mathbb{R}}J_{2}(y)\frac{y [ \sqrt{-\xi }-\sqrt{-(\xi -y)} ] }{2\sqrt{-\xi } [ \sqrt{-(\xi -y)}+\sqrt{-\xi } ] }e^{-\lambda_{2}^{ \ast }y}\,dy \\& \qquad {}+r_{2}c_{2}e^{\lambda_{2}^{\ast }\xi }e^{\lambda_{1}\xi } ( -L \xi +p \sqrt{-\xi } ) \\& \quad =d_{2}pe^{\lambda_{2}^{\ast }\xi } \int_{\mathbb{R}}J_{2}(y)\frac{-y ^{2}}{2\sqrt{-\xi } [ \sqrt{-(\xi -y)}+\sqrt{-\xi } ] ^{2}}e^{-\lambda_{2}^{\ast }y}\,dy \\& \qquad {}+r_{2}c_{2}e^{\lambda_{2}^{\ast }\xi }e^{\lambda_{1}\xi } ( -L \xi +p \sqrt{-\xi } ) \\& \quad \leq \frac{-d_{2}pe^{\lambda_{2}^{\ast }\xi }}{8 ( \vert \xi \vert +S ) ^{\frac{3}{2}}} \int_{\mathbb{R}}J_{2}(y)y^{2}e^{-\lambda_{2}^{\ast }y}\,dy+r _{2}c_{2}e^{\lambda_{2}^{\ast }\xi }e^{\lambda_{1}\xi } ( -L \xi +p\sqrt{- \xi } ) \\& \quad \leq 0 \end{aligned}$$

if

$$p\geq \frac{\max_{\xi < 0} \{ 8r_{2}c_{2} ( \vert \xi \vert +S ) ^{\frac{3}{2}}e^{\lambda_{1}\xi } ( -L\xi +p\sqrt{-\xi } ) \} }{d_{2}\int_{\mathbb{R}}J_{2}(y)y^{2}e^{-\lambda_{2}^{\ast }y}\,dy}. $$

When \(\underline{\phi}_{1}(\xi)=0\) with \(\xi <\xi_{3}\), then \(H_{1}(\underline{\phi}_{1},\overline{\phi}_{1},\overline{\phi} _{2})(\xi)=0\) such that (2.6) is clear. Otherwise, if \(\xi \geq \xi_{3}\), then \(\underline{\phi}_{1}(\xi)=e^{\lambda_{1} \xi }-qe^{\eta \lambda_{1}\xi }>0\). Firstly, let \(q>q_{1}>1\) such that \(e^{\lambda_{1}\xi }-qe^{\eta \lambda_{1}\xi }>0\) implies \(\xi <0\) and

$$\overline{\phi}_{2}(\xi)< 2e^{\lambda_{2}^{\ast }\xi },\qquad \underline{ \phi }_{1}(\xi)\leq \overline{\phi}_{1}(\xi)\le e^{\lambda_{1} \xi }, $$

which is admissible once p is fixed. Therefore, \(q>q_{1}\) indicates

$$\begin{aligned}& r_{1}\underline{\phi}_{1}(\xi)H_{1}( \underline{\phi}_{1},\overline{ \phi }_{1},\overline{\phi}_{2}) (\xi) \\& \quad =r_{1}\underline{\phi}_{1}(\xi) \biggl[ 1- \underline{\phi}_{1}( \xi)-b_{1} \int_{-\tau }^{0}\overline{\phi}_{1}\bigl( \xi +c^{\ast }s\bigr)\,d\eta_{11}(s)-c_{1} \int_{-\tau }^{0}\overline{\phi}_{2}\bigl(\xi +c^{ \ast }s\bigr)\,d\eta_{12}(s) \biggr] \\& \quad \geq r_{1}\underline{\phi}_{1}(\xi)-r_{1}\underline{\phi}_{1} ^{2}(\xi)-r_{1}b_{1}\underline{\phi}_{1}(\xi) \overline{\phi}_{1}( \xi)-2r_{1}c_{1}e^{\lambda_{2}^{\ast }\xi } \underline{\phi}_{1}( \xi) \\& \quad \geq r_{1}\underline{\phi}_{1}(\xi)-r_{1}(1+b_{1})e^{2\lambda_{1} \xi }-2r_{1}c_{1}e^{(\lambda_{1}+\lambda_{2}^{\ast })\xi } \\& \quad =r_{1} \bigl( e^{\lambda_{1}\xi }-qe^{\eta \lambda_{1}\xi } \bigr) -r _{1}(1+b_{1})e^{2\lambda_{1}\xi }-2r_{1}c_{1}e^{(\lambda_{1}+\lambda_{2} ^{\ast })\xi }. \end{aligned}$$

Moreover, (3.3) leads to

$$\begin{aligned}& d_{1}[J_{1}\ast \underline{\phi}_{1}](\xi)-c^{\ast }\underline{ \phi }_{1}^{\prime }(\xi)+r_{1}\underline{\phi}_{1}(\xi)H_{1}( \underline{ \phi }_{1},\overline{\phi}_{1},\overline{\phi}_{2}) (\xi) \\& \quad \geq d_{1} \biggl[ \int_{\mathbb{R}}J_{1}(y)\underline{\phi}_{1}( \xi -y)\,dy-\underline{\phi}_{1}(\xi) \biggr] -c^{\ast }\underline{ \phi }_{1}^{\prime }(\xi) \\& \qquad {}+r_{1} \bigl( e^{\lambda_{1}\xi }-qe^{\eta \lambda_{1}\xi } \bigr) -r _{1}(1+b_{1})e^{2\lambda_{1}\xi }-2r_{1}c_{1}e^{(\lambda_{1}+\lambda_{2} ^{\ast })\xi } \\& \quad \geq d_{1} \biggl[ \int_{\mathbb{R}}J_{1}(y) \bigl[ \bigl( e^{\lambda_{1}( \xi -y)}-qe^{\eta \lambda_{1}(\xi -y)} \bigr) \bigr]\,dy- \bigl( e^{ \lambda_{1}\xi }-qe^{\eta \lambda_{1}\xi } \bigr) \biggr] \\& \qquad {}-c^{\ast } \bigl( \lambda_{1}e^{\lambda_{1}\xi }-q\eta \lambda_{1}e ^{\eta \lambda_{1}\xi } \bigr) +r_{1} \bigl( e^{\lambda_{1}\xi }-qe ^{\eta \lambda_{1}\xi } \bigr) \\& \qquad {} -r_{1}(1+b_{1})e^{2\lambda_{1}\xi }-2r_{1}c_{1}e^{(\lambda_{1}+ \lambda_{2}^{\ast })\xi } \\& \quad =-q\Theta_{1} \bigl( \eta \lambda_{1},c^{\ast } \bigr) e^{\eta \lambda_{1}\xi }-r_{1}(1+b_{1})e^{2\lambda_{1}\xi }-2r_{1}c_{1}e^{( \lambda_{1}+\lambda_{2}^{\ast })\xi } \\& \quad \geq 0 \end{aligned}$$

provided that

$$q>\frac{r_{1}(1+b_{1})-2r_{1}c_{1}}{-\Theta_{1} ( \eta \lambda_{1},c ^{\ast } ) }+q_{1}:=q_{2}. $$

Let \(q_{3}\ge q_{2}\) such that \(q>q_{3}\) indicates

$$\underline{\phi}_{2}(\xi) < \overline{\phi}_{2}(\xi),\quad \xi \in \mathbb{R}, $$

and \(q>q_{3}\), \(( -L\xi -q\sqrt{-\xi } ) >0\), imply

$$( -L\xi +q\sqrt{-\xi } ) e^{\lambda_{2}^{\ast }\xi }< e ^{2\lambda_{2}^{\ast }\xi /3} $$

and so

$$\begin{aligned}& r_{2}\underline{\phi}_{2}(\xi)H_{2}( \underline{\phi}_{1},\overline{ \phi }_{2},\underline{\phi}_{2}) (\xi) \\& \quad =r_{2}\underline{\phi}_{2}(\xi) \biggl[ 1-\underline{ \phi }_{2}( \xi)-b_{2} \int_{-\tau }^{0}\overline{\phi}_{2}\bigl(\xi +c^{\ast }s\bigr)\,d\eta_{22}(s)+c_{2} \int_{-\tau }^{0}\underline{\phi}_{1}\bigl( \xi +c^{ \ast }s\bigr)\,d\eta_{21}(s) \biggr] \\& \quad \geq r_{2}\underline{\phi}_{2}(\xi) \biggl[ 1- \underline{\phi}_{2}( \xi)-b_{2} \int_{-\tau }^{0}\overline{\phi}_{2}\bigl(\xi +c^{\ast }s\bigr)\,d\eta_{22}(s) \biggr] \\& \quad \geq r_{2}\underline{\phi}_{2}(\xi) \bigl[ 1-(1+b_{2})e^{2\lambda_{2} ^{\ast }\xi /3} \bigr] \\& \quad =r_{2} ( -L\xi -q\sqrt{-\xi } ) e^{\lambda_{2}^{\ast } \xi }-r_{2}(1+b_{2})e^{4\lambda_{2}^{\ast }\xi /3}. \end{aligned}$$

By direct calculations, we see

$$\begin{aligned}& d_{2}[J_{2}\ast \underline{\phi}_{2}](\xi)-c^{\ast }\underline{ \phi }_{2}^{\prime }(\xi)+r_{2}\underline{\phi}_{2}(\xi)H_{2}(\underline{ \phi }_{1},\overline{\phi}_{2},\underline{\phi}_{2}) (\xi) \\& \quad \geq d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y) \bigl[ \bigl( -L(\xi -y)-q\sqrt{-( \xi -y)} \bigr) e^{\lambda_{2}^{\ast }(\xi -y)} \bigr]\,dy- ( -L \xi -q\sqrt{-\xi } ) e^{\lambda_{2}^{\ast }\xi } \biggr] \\& \qquad {}+c^{\ast }L \bigl( 1+\lambda_{2}^{\ast }\xi \bigr) e^{\lambda_{2} ^{\ast }\xi }+c^{\ast }q \biggl( \lambda_{2}^{\ast } \sqrt{-\xi }-\frac{1}{2\sqrt{- \xi }} \biggr) e^{\lambda_{2}^{\ast }\xi } \\& \qquad {}+r_{2} ( -L\xi -q\sqrt{-\xi } ) e^{\lambda_{2}^{\ast } \xi }-r_{2}(1+b_{2})e^{4\lambda_{2}^{\ast }\xi /3} \\& \quad =d_{2} \biggl[ \int_{\mathbb{R}}J_{2}(y) \bigl( -L(\xi -y) \bigr) e ^{\lambda_{2}^{\ast }(\xi -y)}\,dy+L\xi e^{\lambda_{2}^{\ast }\xi } \biggr] +c^{\ast }L \bigl( 1+ \lambda_{2}^{\ast }\xi \bigr) e^{\lambda_{2}^{ \ast }\xi }-r_{2}L\xi e^{\lambda_{2}^{\ast }\xi } \\& \qquad {}-q\sqrt{-\xi }d_{2} \int_{\mathbb{R}}J_{2}(y)e^{\lambda_{2}^{ \ast }(\xi -y)}\,dy+q\sqrt{-\xi }e^{\lambda_{2}^{\ast }\xi } \\& \qquad {}+c^{\ast }q\lambda_{2}^{\ast }\sqrt{-\xi }e^{\lambda_{2}^{\ast } \xi }+r_{2} ( -L\xi -q\sqrt{-\xi } ) e^{\lambda_{2}^{ \ast }\xi } \\& \qquad {}+d_{2}q \int_{\mathbb{R}}J_{2}(y) \bigl[ \sqrt{-\xi }-\sqrt{-( \xi -y)} \bigr] e^{\lambda_{2}^{\ast }(\xi -y)}\,dy-\frac{c^{\ast }q}{2\sqrt{- \xi }}e^{\lambda_{2}^{\ast }\xi }-r_{2}(1+b_{2})e^{4\lambda_{2}^{ \ast }\xi /3} \\& \quad =d_{2}q \int_{\mathbb{R}}J_{2}(y) \bigl[ \sqrt{-\xi }-\sqrt{-( \xi -y)} \bigr] e^{\lambda_{2}^{\ast }(\xi -y)}\,dy-\frac{c^{\ast }q}{2\sqrt{- \xi }}e^{\lambda_{2}^{\ast }\xi }-r_{2}(1+b_{2})e^{4\lambda_{2}^{ \ast }\xi /3} \\& \quad =d_{2}q \int_{\mathbb{R}}J_{2}(y) \biggl[ \frac{-y}{\sqrt{-\xi }-\sqrt{-( \xi -y)}} \biggr] e^{\lambda_{2}^{\ast }(\xi -y)}\,dy-\frac{c^{\ast }q}{2\sqrt{- \xi }}e^{\lambda_{2}^{\ast }\xi }-r_{2}(1+b_{2})e^{4\lambda_{2}^{ \ast }\xi /3} \\& \quad =d_{2}q \int_{\mathbb{R}}J_{2}(y) \biggl[ \frac{y}{2\sqrt{-\xi }}- \frac{y}{\sqrt{- \xi }+\sqrt{-(\xi -y)}} \biggr] e^{\lambda_{2}^{\ast }(\xi -y)}\,dy-r _{2}(1+b_{2})e^{4\lambda_{2}^{\ast }\xi /3} \\& \quad =d_{2}q \int_{\mathbb{R}}J_{2}(y) \biggl[ \frac{y^{2}}{2\sqrt{- \xi } [ \sqrt{-\xi }+\sqrt{-(\xi -y)} ] } \biggr] e^{ \lambda_{2}^{\ast }(\xi -y)}\,dy-r_{2}(1+b_{2})e^{4\lambda_{2}^{\ast } \xi /3} \\& \quad =d_{2}q \int_{\mathbb{R}}J_{2}(y)\frac{y^{2}e^{\lambda_{2}^{\ast }( \xi -y)}}{2\sqrt{-\xi } [ \sqrt{-\xi }+\sqrt{-(\xi -y)} ] ^{2}}\,dy-r_{2}(1+b_{2})e^{4\lambda_{2}^{\ast }\xi /3} \\& \quad =e^{\lambda_{2}^{\ast }\xi } \biggl\{ d_{2}q \int_{\mathbb{R}}J_{2}(y)\frac{y ^{2}e^{\lambda_{2}^{\ast }y}}{2\sqrt{-\xi } [ \sqrt{-\xi }+\sqrt{-( \xi -y)} ] ^{2}}\,dy-r_{2}(1+b_{2})e^{\lambda_{2}^{\ast }\xi /3} \biggr\} \\& \quad \geq e^{\lambda_{2}^{\ast }\xi } \biggl\{ d_{2}q \int_{\mathbb{R}}J _{2}(y)\frac{y^{2}e^{\lambda_{2}^{\ast }y}}{8(\vert \xi \vert +S)^{3/2}}\,dy-r _{2}(1+b_{2})e^{\lambda_{2}^{\ast }\xi /3} \biggr\} \\& \quad \geq 0 \end{aligned}$$

if

$$q\ge \sup_{\xi < 0}\frac{8e^{\lambda_{2}^{\ast }\xi /3}(S+\vert \xi \vert )^{3/2}r _{2}(1+b_{2})}{d_{2}\int_{\mathbb{R}}J_{2}(y)y^{2}e^{\lambda_{2}^{ \ast }y}\,dy}+q_{4}:=q_{5}. $$

Fix \(q=q_{5}\), we complete the proof by Lemma 2.4 and a discussion similar to (3.1). □

Theorem 3.4

Assume that \(c_{1}^{*}=c_{2}^{*}\). Further suppose that \(k_{1}\), \(k_{2}\) have compact supports. Then (2.1) with \(c=c^{\ast }\) has a positive solution \((\phi_{1}(\xi),\phi_{2}(\xi))\) such that

$$0< \phi_{1}(\xi)< 1,0< \phi_{2}(\xi)< 1+c_{2},\xi \in \mathbb{R}, \quad \lim_{\xi \rightarrow -\infty }\bigl(\phi_{1}(\xi), \phi_{2}(\xi)\bigr)=(0,0) $$

and

$$\phi_{1}(\xi)\sim \mathcal{O}\bigl(-\xi e^{\lambda_{1}^{\ast }\xi }\bigr),\qquad \phi _{2}(\xi)\sim \mathcal{O}\bigl(-\xi e^{\lambda_{2}^{\ast }\xi }\bigr),\quad \xi \rightarrow -\infty. $$

Proof

Using the notation in Theorems 3.2–3.3, we define

$$\begin{aligned}& \underline{\phi}_{1}(\xi)= \textstyle\begin{cases} ( -L\xi -q\sqrt{-\xi } ) e^{\lambda_{1}^{\ast }\xi },& \xi < \xi_{1}, \\ 0, &\xi \geq \xi _{1}, \end{cases}\displaystyle \\& \overline{\phi}_{1}(\xi)= \textstyle\begin{cases} -L\xi e^{\lambda_{1}^{\ast }\xi },&\xi < \xi_{2}, \\ 1, &\xi \geq \xi_{2}, \end{cases}\displaystyle \end{aligned}$$

and

$$\begin{aligned}& \underline{\phi}_{2}(\xi)= \textstyle\begin{cases} ( -L\xi -q\sqrt{-\xi } ) e^{\lambda_{2}^{\ast }\xi }, &\xi < \xi_{3}, \\ 0,& \xi \geq \xi_{3}, \end{cases}\displaystyle \\& \overline{\phi}_{2}(\xi)=\textstyle\begin{cases} ( -L\xi +p\sqrt{-\xi } ) e^{\lambda_{2}^{\ast }\xi },& \xi < \xi_{4}, \\ 1+c_{2},& \xi \geq \xi_{4}, \end{cases}\displaystyle \end{aligned}$$

where \(p,q>1\) are large enough, \(\xi_{1}\), \(\xi_{2}\), \(\xi_{3}\), \(\xi_{4}\) are similar to above. Then we can obtain a pair of upper and lower solutions. Since the verification is similar to those in Theorems 3.2–3.3, we omit it here. □

4 Asymptotic behavior and nonexistence of traveling wave solutions

In the previous section, we obtain the existence of nonconstant traveling wave solutions of (1.1). In this part, we shall first consider the behavior if \(\xi \to \infty \) by the idea of contracting rectangle [32] in Lin and Ruan [33]. For \(s\in [0,1]\), define the continuous functions

$$a_{1}(s)=sk_{1},\qquad b_{1}(s)=sk_{1}+(1-s) (1+\epsilon) $$

and

$$a_{2}(s)=(1-s)+sk_{2},\qquad b_{2}(s)=sk_{2}+(1-s) (1+c_{2}) (1+\epsilon) $$

with \(\epsilon \in (0,1)\) such that

$$1-b_{1}(1+\epsilon)-c_{1}(1+c_{2}) (1+ \epsilon)>0,\qquad 1-b_{2}(1+c_{2}) (1+ \epsilon)>0. $$

Then they satisfy

  1. (C1)

    \(1-a_{1}(s)-b_{1}b_{1}(s)-c_{1}b_{2}(s) > 0\),

  2. (C2)

    \(1-a_{2}(s)-b_{2}b_{2}(s)+c_{2}a_{1}(s) > 0\),

  3. (C3)

    \(1-b_{1}(s)-b_{1}a_{1}(s)-c_{1}a_{2}(s) < 0\),

  4. (C4)

    \(1-b_{2}(s)-b_{2}a_{2}(s)+c_{2}b_{1}(s) < 0\),

for any \(s\in (0,1)\), we now verify them [34]. In (C1), we have

$$\begin{aligned}& 1-a_{1}(s)-b_{1}b_{1}(s)-c_{1}b_{2}(s) \\& \quad =1-sk_{1}-b_{1}\bigl[sk_{1}+(1-s) (1+ \epsilon)\bigr] \\& \qquad {}-c_{1}\bigl[sk_{2}+(1-s) (1+c_{2}) (1+\epsilon)\bigr] \\& \quad =(1-s)\bigl[1-b_{1}(1+\epsilon)-c_{1}(1+c_{2}) (1+\epsilon)\bigr] \\& \quad >0. \end{aligned}$$

(C2) is true since

$$\begin{aligned}& 1-a_{2}(s)-b_{2}b_{2}(s)+c_{2}a_{1}(s) \\& \quad =1-sk_{2}-b_{2}\bigl[sk_{2}+(1-s) (1+c_{2}) (1+\epsilon)\bigr]+c_{2}sk_{1} \\& \quad =(1-s)\bigl[1-b_{2}(1+c_{2}) (1+\epsilon)\bigr] \\& \quad >0. \end{aligned}$$

On (C3), we have

$$\begin{aligned}& 1-b_{1}(s)-b_{1}a_{1}(s)-c_{1}a_{2}(s) \\& \quad =1-\bigl[sk_{1}+(1-s) (1+\epsilon)\bigr]-b_{1}sk_{1}-c_{1} \bigl[(1-s)+sk_{2}\bigr] \\& \quad < 1-\bigl[sk_{1}+(1-s)\bigr]-b_{1}sk_{1}-c_{1} \bigl[(1-s)+sk_{2}\bigr] \\& \quad =-c_{1}(1-s) \\& \quad \leq 0. \end{aligned}$$

Finally, (C4) is true since

$$\begin{aligned}& 1-b_{2}(s)-b_{2}a_{2}(s)+c_{2}b_{1}(s) \\& \quad =1-\bigl[sk_{2}+(1-s) (1+c_{2}) (1+\epsilon)\bigr] \\& \qquad {}-b_{2}\bigl[(1-s)+sk_{2}\bigr]+c_{2} \bigl[sk_{1}+(1-s) (1+\epsilon)\bigr] \\& \quad < 1-\bigl[sk_{2}+(1-s) (1+c_{2})\bigr] \\& \qquad {}-b_{2}\bigl[(1-s)+sk_{2}\bigr]+c_{2} \bigl[sk_{1}+(1-s)\bigr] \\& \quad =(1-s)\bigl[1-(1+c_{2})-b_{2}+c_{2}\bigr] \\& \quad =-b_{2}(1-s) \\& \quad \leq 0. \end{aligned}$$

Remark 4.1

In Pan [34], we proved the stability of positive steady state by (C1)–(C4) of the corresponding kinetic system. Moreover, Faria [35] gave some sharp conditions on the general Lotka–Volterra systems with delays.

Theorem 4.2

Assume that \(c\ge c^{*}\). Further suppose that \((\phi_{1}(\xi),\phi _{2}(\xi))\) is a solution of (2.1) and satisfies

$$ 0< \phi_{1}(\xi)< 1,0< \phi_{2}(\xi)< 1+c_{2},\xi \in \mathbb{R}, \quad \lim_{\xi \rightarrow -\infty }\bigl( \phi_{1}(\xi),\phi_{2}(\xi)\bigr)=(0,0). $$
(4.1)

If

$$ b_{1} +c_{1}(1+c_{2}) < 1,\qquad b_{2}(1+c_{2})< 1, $$
(4.2)

then

$$ \lim_{\xi \to \infty }\phi_{i} (\xi)=k_{i},\quad i=1,2. $$
(4.3)

Proof

We first verify that

$$\liminf_{\xi \to \infty }\phi_{i} (\xi)>0,\quad i=1,2. $$

By (4.1), we see that

$$\begin{aligned}& d_{1}[J_{1}\ast \phi_{1}](\xi)-c \phi_{1}^{\prime }(\xi)+r_{1}\phi_{1}( \xi)F_{1}(\phi_{1},\phi_{2}) (\xi) \\& \quad \geq d_{1}[J_{1}\ast \phi_{1}](\xi)-c \phi_{1}^{\prime }(\xi)+r _{1}\phi_{1}( \xi)\bigl[1-b_{1}-c_{1}(1+c_{2})- \phi_{1}(\xi)\bigr] \end{aligned}$$

for any \(\xi \in \mathbb{R}\). Then \(u_{1}(x,t)=\phi_{1}(x+ct)\) satisfies

$$\textstyle\begin{cases} \frac{\partial u_{1}(x,t)}{\partial t}\geq d_{1}[J_{1}\ast u_{1}](x,t)+r _{1}u_{1}(x,t)[1-b_{1}-c_{1}(1+c_{2})-u_{1}(x,t)], \\ u_{1}(x,0)=\phi_{1}(x), \end{cases} $$

for \(x\in \mathbb{R}\), \(t>0\). By Lemmas 2.1 and 2.2, we have

$$\liminf_{t\rightarrow \infty }u_{1}(0,t)\geq 1-b_{1}-c_{1}(1+c_{2})>0 $$

and so

$$\liminf_{\xi \rightarrow \infty }\phi_{1}(\xi)\geq 1-b_{1}-c_{1}(1+c _{2}) >0 $$

by the definition of traveling wave solutions.

Similarly, we have

$$\textstyle\begin{cases} \frac{\partial u_{2}(x,t)}{\partial t}\geq d_{2}[J_{2}\ast u_{2}](x,t)+r _{2}u_{2}(x,t) [ 1-b_{2}(1+c_{2})-a_{2}u_{2}(x,t) ] , \\ u_{2}(x,0)=\phi_{2}(x), \end{cases} $$

for \(x\in \mathbb{R}\), \(t>0\). Then Lemmas 2.1 and 2.2 imply that

$$\liminf_{t\rightarrow \infty }u_{2}(0,t)\geq 1-b_{2}(1+c_{2})>0 $$

and so

$$\liminf_{\xi \rightarrow \infty }\phi_{2}(\xi)\geq 1-b_{2}(1+c_{2})>0. $$

Define

$$\begin{aligned}& \liminf_{\xi \rightarrow \infty }\phi_{1}(\xi)=\phi_{1}^{-},\qquad \liminf_{\xi \rightarrow \infty }\phi_{2}(\xi)=\phi_{2}^{-}, \\& \limsup_{\xi \rightarrow \infty }\phi_{1}(\xi) = \phi_{1}^{+},\qquad \limsup_{\xi \rightarrow \infty } \phi_{2}(\xi)=\phi_{2}^{+}. \end{aligned}$$

Then there exists \(s^{\prime }\in (0,1]\) such that

$$\begin{aligned}& a_{1}\bigl(s^{\prime }\bigr) \leq \phi_{1}^{-} \leq \phi_{1}^{+}\leq b_{1}\bigl(s ^{\prime }\bigr), \\& a_{2}\bigl(s^{\prime }\bigr) \leq \phi_{2}^{-} \leq \phi_{2}^{+}\leq b_{2}\bigl(s ^{\prime } \bigr). \end{aligned}$$

Define \(s=\sup s^{\prime }\). If \(s=1\), then the result is true. Otherwise, \(s<1\) and at least one of the following is true:

$$a_{1}(s)=\phi_{1}^{-},\qquad \phi_{1}^{+}=b_{1}(s),\qquad a_{2}(s)= \phi_{2}^{-},\qquad \phi_{2} ^{+}=b_{2}(s). $$

If \(a_{1}(s)=\phi_{1}^{-}\), then there exists \(\{\xi_{m}\}_{m=1}^{ \infty }\) such that

$$\lim_{m\to \infty } \xi_{m}=\infty, \qquad \lim _{m\to \infty } \phi_{1}(\xi _{m})=a_{1}(s) $$

and

$$\liminf_{m\to \infty } \bigl[d_{1}[J_{1}* \phi_{1}](\xi_{m})-c\phi_{1}'(\xi _{m})\bigr]\ge 0. $$

By (C1), we see that

$$\begin{aligned}& \liminf_{m\rightarrow \infty }\biggl[1-\phi_{1}( \xi_{m})-b_{1} \int_{- \tau }^{0}\phi_{1}( \xi_{m}+cs)\,d\eta_{11}(s)-c_{1} \int_{-\tau }^{0}\phi_{2}( \xi_{m}+cs)\,d\eta_{12}(s)\biggr] \\& \quad \geq 1-a_{1}(s)-b_{1}b_{1}(s)-c_{1}b_{2}(s) \\& \quad >0, \end{aligned}$$

which implies a contradiction by the definition of \(\phi_{1}(\xi)\), \(\phi_{2}(\xi)\).

By a similar discussion of

$$\phi_{1}^{+}=b_{1}(s),\qquad a_{2}(s)= \phi_{2}^{-},\qquad \phi_{2}^{+}=b_{2}(s), $$

we complete the proof. □

We now present the nonexistence of (2.1) with (2.2) if \(c< c^{*}\).

Theorem 4.3

If \(c< c^{*}\), then there is not a positive solution of (2.1) with (2.2).

Proof

Were the statement false, then, for some \(c'\in (0,c^{*})\), there is a positive solution \((\phi_{1}(\xi),\phi_{2}(\xi))\) of (2.1) with (2.2). Firstly, it is easy to confirm that

$$0< \phi_{1}(\xi)< 1,\qquad 0< \phi_{2}(\xi)< 1+c_{2},\quad \xi \in \mathbb{R}. $$

If \(c^{*}=c_{1}^{*}\), then there exists \(\epsilon >0\) such that

$$\inf_{\lambda >0}\frac{d_{1} [ \int_{\mathbb{R}}J_{1}(y)e^{\lambda y}\,dy-1 ] +r_{1}(1-\epsilon)}{\lambda }>c^{\prime }. $$

Let \(\xi^{\prime }\in \mathbb{R}\) such that

$$\sup_{x\leq \xi^{\prime }} \biggl[ b_{1} \int_{-\tau }^{0}\phi_{1}\bigl(x+c ^{\prime }s\bigr)\,d\eta_{11}(s)+c_{1} \int_{-\tau }^{0}\phi_{2}\bigl(x+c^{\prime }s \bigr)\,d\eta_{12}(s) \biggr] =\epsilon, $$

then

$$d_{1}[J_{1}\ast \phi_{1}](\xi)-c \phi_{1}^{\prime }(\xi)+r_{1}\phi_{1}( \xi)\bigl[1-\epsilon -\phi_{1}(\xi)\bigr]\geq 0,\quad \xi \leq \xi^{\prime }. $$

Define \(\inf_{x\geq \xi^{\prime }}\phi_{1}(\xi)=\underline{\underline{ \phi_{1}}}\), then \(\underline{\underline{\phi_{1}}}>0\) by the positivity and limit behavior. Let \(M\geq 1\) such that

$$M-1\geq \frac{b_{1}+c_{1}(1+c_{2})}{\underline{\underline{\phi_{1}}}}, $$

then

$$d_{1}[J_{1}\ast \phi_{1}](\xi)-c \phi_{1}^{\prime }(\xi)+r_{1}\phi_{1}( \xi)\bigl[1-\epsilon -M\phi_{1}(\xi)\bigr]\geq 0,\quad \xi \in \mathbb{R}. $$

Therefore, \(\phi_{1}(\xi)=\phi_{1}(x+c^{\prime }t)=u_{1}(x,t)\) satisfies

$$\textstyle\begin{cases} \frac{\partial u_{1}(x,t)}{\partial t}\geq d_{1}[J_{1}\ast u_{1}](x,t)+r _{1}u_{1}(x,t)[1-\epsilon -M u_{1}(x,t)],\quad x\in \mathbb{R},t>0, \\ u_{1}(x,0)=\phi_{1}(x),\quad x\in \mathbb{R}. \end{cases} $$

By Lemma 2.1, we see that if

$$-2x= \biggl[ \inf_{\lambda >0}\frac{d_{1} [ \int_{\mathbb{R}}J_{1}(y)e ^{\lambda y}\,dy-1 ] +r_{1}(1-\epsilon)}{\lambda }+c^{\prime } \biggr] t, $$

then

$$\liminf_{t\rightarrow \infty }u_{1}(x,t)\geq \frac{1-\epsilon }{M}>0, $$

which also implies that \(x+c^{\prime }t\rightarrow -\infty \), \(t\rightarrow \infty \) and

$$\lim_{\xi \to -\infty } \phi_{1}(\xi)=\limsup _{t\rightarrow \infty }u _{1}(x,t)=0, $$

a contradiction occurs.

Similarly, we can prove the result if \(c^{*}=c^{*}_{2}\). The proof is complete. □

5 Conclusion and discussion

In this paper, we firstly show the existence and nonexistence of traveling wave solutions for all positive wave speed, and thus obtain the minimal wave speed. In [20, 21], the authors studied the existence of traveling wave solutions when \(c>c^{*}\), and the traveling wave solutions decay exponentially. In this paper, if \(c=c^{*}\), these traveling wave solutions do not decay exponentially, the asymptotic behavior coincides with the conclusions in [36, 37] when \(b_{1}=b_{2}=c_{1}=c_{2}\). That is, for the minimal wave speed, the corresponding traveling wave solutions may have different properties. Moreover, there are also some results on the minimal wave speed of nonmonotone coupled systems with time delay, which was proved by constructing upper and lower solutions, part of recent results can be found in Fu [38], Lin [39] and Yang and Li [40].

In mathematical biology, the spreading speed is also an important threshold [41]. For monotone systems, see Liang and Zhao [3], Lui [4, 42], Weinberger [5], Weinberger et al. [6]. Recently, Pan [43] estimated the invasion speed of the predator in a predator–prey system, which equals the minimal invasion wave speed in Lin [44]. It is a challenging question to estimate the spreading speeds of (1.1), of which the corresponding undelayed system with classical Laplacian diffusion were studied by Lin [45], Pan [46], Wang and Zhang [47], Wang and Zhao [48].