1 Introduction

In population dynamics, predator–prey systems have been widely studied due to their importance as well as plentiful dynamical behaviors. When the system involves only one predator and one prey, there have been many results for both ordinary differential systems and reaction–diffusion equations. For the topic, we refer to Cantrell and Cosner [4], Fife [13], Ghergu and Radulescu [17], Murray [35, 36], Pao [39], Smith [42], Smoller [43], Ye et al. [50], Zhang and Feng [52], Zhao [54].

When the spatiotemporal dynamics is concerned, since the pioneer work in [14, 25], much attention has been paid to the traveling wave solutions of parabolic equations; we refer to Volpert et al. [44] for some earlier results and a survey paper by Zhao [55] for some recent conclusions. If a system is of predator–prey type with two species, then several methods, including phase analysis, shooting methods, Conley index and fixed point theorem, have been applied to establish the existence of traveling wave solutions; we refer to some important results by Dunbar [9–11], Gardner and Smoller [16], Gardner and Jones [15], Chen et al. [7], Huang et al. [19], Huang and Zou [21], Huang [23], Hsu et al. [18], Li and Li [26], Lin et al. [30], Lin [28, 29], Lin et al. [32], Pan [37], Wang et al. [45], Wang et al. [47], Zhang et al. [51].

In fact, due to the diversity and complexity of ecosystems, the study of the interaction among multi-species has more practical significance. However, when there are more than 2 species, studying the ecosystem becomes more difficult. For example, it is difficult to study the existence of traveling wave solutions by phase analysis. Moreover, even for ordinary differential systems, the investigation of dynamical behavior becomes more difficult due to the deficiency of general Poincare–Bendixson theorem in \(\mathbb{R}^{n}\) with \(n\ge3\). Of course, there are also some conclusions about the dynamics of predator–prey systems with multiple species. For example, Caristi et al. [6] investigated the coexistence of a predator and two species on bounded habitat with Neumann boundary condition, Du and Xu [8], Shang et al. [41], Zhang et al. [53] proved the existence of traveling wave solutions to a reaction–diffusion system with multiple species.

Recently, Huang and Lin [24] considered the following three species reaction-diffusion system (see Cantrell and Cosner [4]):

$$ \textstyle\begin{cases} \frac{\partial u_{1}(x,t)}{\partial t} =d_{1}\Delta u_{1}(x,t)+ u_{1}(x,t)[b_{1}-u_{1}(x,t)-b_{12}u_{2}(x,t)-b_{13}u_{3}(x,t)],\\ \frac{\partial u_{2}(x,t)}{\partial t} =d_{2}\Delta u_{2}(x,t)+ u_{2}(x,t)[b_{2}-b_{21}u_{1}(x,t)-u_{2}(x,t)-b_{23}u_{3}(x,t)],\\ \frac{\partial u_{3}(x,t)}{\partial t} =d_{3}\Delta u_{3}(x,t)+ u_{3}(x,t)[b_{3}+b_{31}u_{1}(x,t)+b_{32}u_{2}(x,t)-u_{3}(x,t)], \end{cases} $$
(1.1)

where all parameters are positive, \(u_{1}(x,t), u_{2}(x,t)\) denote the densities of two competing prey species located at \(x\in\varOmega \subseteq\mathbb{R}^{n}\) at time t, \(u_{3}\) denotes the density of predator feeding on species 1 and species 2 located at \(x\in\varOmega \subseteq\mathbb{R}^{n}\) at time t. The authors obtained the minimal wave speed of non-negative traveling wave solutions connecting trivial equilibrium with positive equilibrium, in which the limit behavior is verified by the abstract results in Lin and Ruan [31].

Because all the parameters are positive in [24], this implies that the predator could survive without two preys in the system. In this paper, we suppose that all the parameters, except \(b_{3}<0\), are positive and the predator only feeds on species 1 and 2. Under the conditions of (1.1), there are several different dynamical problems in the literature. For example, \(u_{3}\) invades the habitat in which \(u_{1},u_{2}\) coexist, \(u_{1},u_{3}\) invade the habitat of \(u_{2}\). In population dynamics, these problems often imply different thresholds in (1.1), and the corresponding control problem is also very important [40].

The purpose of this paper is to discuss the situation when prey \(u_{2}\) and predator \(u_{3}\) invade the habitat of prey \(u_{1}\). We study the global stability of the positive equilibrium with the help of contracting rectangles [42] when the domain is bounded, then the existence as well as noexistence of traveling wave solutions when \(x\in\mathbb{R}\). More precisely, we first establish the existence of a nontrivial traveling wave solution by the generalized upper and lower solution. To verify the limit behavior of traveling wave solutions, we use the results in global stability as well as the asymptotic spreading of Fisher equation [1]. Finally, the nonexistence of traveling wave solutions is obtained by constructing auxiliary equations as well as using the theory of asymptotic spreading.

It should be noted that Lin and Ruan [31] applied the idea of contracting rectangles to verify the limit behavior of a traveling wave solution. But in [31] they needed strictly contracting rectangles, a condition which is stronger than the general stability conditions in [42]. In this paper, using the basic idea in [31] and adding necessary discussion, we prove the limit behavior of traveling wave solutions by general contracting rectangles. We believe the technique can be applied to more models.

The rest of this paper is organized as follows. In Sect. 2, we will give some preliminaries. If the domain is bounded, the global stability of the positive equilibrium is proved in Sect. 3. In Sect. 4, by using Schauder’s fixed point theorem and constructing upper and lower solutions, we obtain the existence of non-negative traveling wave solutions. Then the asymptotic behavior of traveling wave solutions is also studied in Sect. 5. In Sect. 6, the nonexistence of traveling wave solutions is established by using the relevant conclusions of the asymptotic speed of spreading.

2 Preliminaries

We first introduce some notations. For \(u=(u_{1},u_{2},u_{3}), v=(v_{1},v_{2},v_{3})\in\mathbb{R}^{3}\), we write \(u\geq v\) provided \(u_{i}\geq v_{i}\) for \(i=1,2,3\). Let X be the following functional space:

$$X=\bigl\{ u:\mathbb{R} \to\mathbb{R}^{3} \text{ is bounded and uniformly continuous}\bigr\} , $$

which is a Banach space equipped with the standard supremum norm. If \(a,b\in\mathbb{R}^{3}\) with \(a\leq b\), then \(X_{[a,b]}\) is defined by

$$X_{[a,b]}=\bigl\{ u \in X:a\leq u(x)\leq b,x\in\mathbb{R}\bigr\} . $$

If \(u(x)=(u_{1}(x),u_{2}(x),u_{3}(x))\) and \(v(x)=(v_{1}(x),v_{2}(x),v_{3}(x))\in X\), then \(u(x)\geq v(x)\) implies that \(u(x)\geq v(x)\) for all \(x\in\mathbb{R}\); \(u(x)>v(x)\) is interpreted as \(u(x)\geq v(x)\) but \(u(x)>v(x)\) for some \(x\in\mathbb{R}\).

By rescaling, (1.1) is equivalent to the following system:

$$ \textstyle\begin{cases} \frac{\partial u_{1}(x,t)}{\partial t} =d_{1}\Delta u_{1}(x,t)+r_{1} u_{1}(x,t)[1-u_{1}(x,t)-a_{12}u_{2}(x,t)-a_{13}u_{3}(x,t)],\\ \frac{\partial u_{2}(x,t)}{\partial t} =d_{2}\Delta u_{2}(x,t)+r_{2} u_{2}(x,t)[1-a_{21}u_{1}(x,t)-u_{2}(x,t)-a_{23}u_{3}(x,t)],\\ \frac{\partial u_{3}(x,t)}{\partial t} =d_{3}\Delta u_{3}(x,t)+r_{3} u_{3}(x,t)[-1+a_{31}u_{1}(x,t)+a_{32}u_{2}(x,t)-u_{3}(x,t)], \end{cases} $$
(2.1)

where all the parameters are positive. Clearly, the corresponding kinetic system is

$$ \textstyle\begin{cases} \frac{d u_{1}(t)}{d t} =r_{1} u_{1}(t)[1-u_{1}(t)-a_{12}u_{2}(t)-a_{13}u_{3}(t)],\\ \frac{d u_{1}(t)}{d t} =r_{2} u_{2}(t)[1-a_{21}u_{1}(t)-u_{2}(t)-a_{23}u_{3}(t)],\\ \frac{d u_{1}(t)}{d t} =r_{3} u_{3}(t)[-1+a_{31}u_{1}(t)+a_{32}u_{2}(t)-u_{3}(t)]. \end{cases} $$
(2.2)

Assume that

$$ \textstyle\begin{cases} a_{12}+a_{13}(a_{31}+a_{32}-1)< 1,\\ a_{21}+a_{23}(a_{31}+a_{32}-1)< 1,\\ a_{32}[1-a_{21}-a_{23}(a_{31}+a_{32}-1)]+a_{31}[1-a_{12}-a_{13}(a_{31}+a_{32}-1)]>1, \end{cases} $$
(2.3)

then (2.2) has an equilibrium point \(E=(1,0,0)\) and a unique positive equilibrium point \(K=(k_{1},k_{2},k_{3})\) defined by

$$\begin{aligned} &k_{1}=\frac{a_{12}a_{23}+a_{13}a_{32}-a_{23}a_{32}+a_{12} -a_{13}-1}{a_{12}a_{23}a_{31}+a_{13}a_{32}a_{21}+a_{12}a_{21}-a_{13}a_{31}-a_{23}a_{32}-1}, \\ &k_{2}=\frac{a_{21}a_{13}+a_{23}a_{31}-a_{31}a_{13}+a_{21} -a_{23}-1}{a_{12}a_{23}a_{31}+a_{13}a_{32}a_{21}+a_{12}a_{21}-a_{13}a_{31}-a_{23}a_{32}-1}, \\ &k_{3}=\frac{a_{32}a_{21}+a_{31}a_{12}-a_{12}a_{21}-a_{31} -a_{32}+1}{a_{12}a_{23}a_{31}+a_{13}a_{32}a_{21}+a_{12}a_{21}-a_{13}a_{31}-a_{23}a_{32}-1}. \end{aligned}$$

In fact, the existence and uniqueness of K are not evident if (2.3) holds. But this will be clear from the stability analysis in Sect. 3. More precisely, we could obtain the existence of equilibria by the limit behavior in Sect. 3; then for any positive equilibria, we shall prove the global stability which implies the uniqueness of the positive steady state. Furthermore, (2.2) also has several semi-trivial steady states, which do not affect our following discussion.

For convenience, we introduce the definition of contracting rectangle (see Smith [42]). Consider the following initial value problem:

$$ \textstyle\begin{cases} x'(t)=f(x(t)),\\ x(0)=\phi, \end{cases} $$
(2.4)

where \(x(t)=(x_{1}(t),x_{2}(t),\dots,x_{n}(t))\), \(f(x(t))= (f_{1}(x(t)),f_{2}(x(t)),\dots ,f_{n}(x(t)))\), \(\phi=(\phi_{1},\phi_{2},\dots, \phi_{n})\) and there exists \(e\in\mathbb{R}^{n}\) such that \(f_{i}(e)=0\).

Denote a one-parameter family of order intervals \(\varSigma (s)=[a(s),b(s)]\), \(0\leq s\leq1\) such that for \(0\leq s_{1}\leq s_{2} \leq1\),

$$a(0)\leq a(s_{1})\leq a(s_{2})\leq a(1)=e=b(1)\leq b(s_{2})\leq b(s_{1}) \leq b(0), $$

where

$$a(s)=\bigl(a_{1}(s),a_{2}(s),\dots,a_{n}(s) \bigr),\qquad b(s)=\bigl(b_{1}(s),b_{2}(s),\dots,b_{n}(s) \bigr), $$

and \(a_{i}(s)\), \(b_{i}(s)\), \(i=1,2,\dots,n\), are continuous of \(s\in[0,1]\).

Definition 2.1

\(\varSigma(s)\) is said to be a contracting rectangle of (2.4) if for any \(s\in[0,1)\) and \(\phi=(\phi_{1},\phi_{2},\dots,\phi_{n})\in\varSigma (s)\), we have

  1. (1)

    \(f_{i}(\phi)\ge0\) whenever \(\phi\in\varSigma(s)\) and \(\phi _{i}=a_{i}(s)\) while \(f_{i}(\phi)\le0\) whenever \(\phi\in\varSigma(s)\) and \(\phi _{i}=b_{i}(s)\);

  2. (2)

    for each s, at least one of above 2n inequalities is strict.

Using the contracting rectangle, we have the following stability result (see Smith [42, Theorem 5.2.5]).

Lemma 2.2

Assume that \(\varSigma(s)\) is a contracting rectangle of (2.4). If \(\phi\in\varSigma(0)\), then e is globally stable.

We also present some results for the Fisher equation. Assume that \(D,R,M\) are positive constants, \(Z(x)>0\) is a bounded and continuous function with nonempty support. Consider the initial value problem associated to the Fisher equation

$$ \textstyle\begin{cases} \frac{\partial Z(x,t)}{\partial t}=D\Delta Z(x,t)+RZ(x,t)[1-MZ(x,t)],\\ Z(x,0) =Z(x) ,\quad x\in\mathbb{R}. \end{cases} $$
(2.5)

From Fife [13], Ye et al. [50] and Aronson and Weinberger [1], (2.5) has the following properties.

Lemma 2.3

  1. (i)

    Equation (2.5) admits a unique solution \(Z(x,t)\geq0\) which is twice differentiable in \(x\in\mathbb{R}\) and differentiable in \(t>0\).

  2. (ii)

    Assume that \(\overline{Z}(x,t)\), \(\underline{Z}(x,t)\) are continuous and bounded for \(x\in\mathbb{R}\), \(t\geq0\), twice differentiable in \(x\in\mathbb{R}\) and differentiable in \(t>0\). If they satisfy

    $$ \textstyle\begin{cases} \frac{\partial\overline{Z}(x,t)}{\partial t} \geq D\Delta\overline {Z}(x,t)+R\overline{Z}(x,t)[1-M\overline{Z}(x,t)],\\ \overline{Z}(x,0) \geq Z(x),\quad x\in\mathbb{R} \end{cases} $$

    and

    $$ \textstyle\begin{cases} \frac{\partial\underline{Z}(x,t)}{\partial t} \leq D\Delta\underline {Z}(x,t)+R\underline{Z}(x,t)[1-M\underline{Z}(x,t)],\\ \underline{Z}(x,0) \leq Z(x) ,\quad x \in\mathbb{R}, \end{cases} $$

    then \(\overline{Z}(x,t)\geq Z(x,t)\geq\underline{Z}(x,t)\), where \(Z(x,t)\) is a solution to (2.5).

  3. (iii)

    If \(Z(x,t)\) satisfies (2.5), and \(Z(x)\) admits nonempty support, then

    $$\liminf _{t \to\infty}\inf _{|x|< (2\sqrt{DR}-\epsilon)t}Z(x,t) =\limsup _{t \to\infty}\sup _{|x|< (2\sqrt{DR}-\epsilon)t}Z(x,t) = \frac{1}{M} $$

    for any \(\epsilon\in(0,2{\sqrt{DR}})\).

3 Stability of the positive equilibrium

In this section, we shall establish global stability of the positive equilibrium (2.1) on a smooth bounded domain Ω with Neumann boundary condition by using the contracting rectangles, throughout which (2.3) holds. We first consider (2.2) if

$$u_{1}(0)>0,\qquad u_{2}(0)>0,\qquad u_{3}(0)>0. $$

By the quasipositivity, we see that

$$u_{1}(t)>0,\qquad u_{2}(t)>0,\qquad u_{3}(t)>0 $$

for all \(t>0\). It is evident that this model is defined for all \(t\in (0,\infty)\).

Due to positivity, we see that

$$\frac{d u_{1}(t)}{d t} \le r_{1} u_{1}(t)\bigl[1-u_{1}(t) \bigr] $$

and so

$$\limsup_{t\to\infty} u_{1}(t) \le1. $$

Similarly, we have

$$\limsup_{t\to\infty} u_{2}(t) \le1, $$

which further indicates that

$$\limsup_{t\to\infty} u_{3}(t) \le a_{31}+a_{32}-1. $$

Returning to the first equation of (2.2), we obtain

$$\liminf_{t\to\infty} u_{1}(t) \ge 1-a_{12}-a_{13}(a_{31}+a_{32}-1)=: \underline{u}_{1}. $$

Similarly, we have

$$\liminf_{t\to\infty} u_{2}(t) \ge 1-a_{21}-a_{23}(a_{31}+a_{32}-1)=: \underline{u}_{2} $$

and

$$\begin{aligned} \liminf_{t\to\infty} u_{3}(t) \ge& a_{31}\bigl[1-a_{12}-a_{13}(a_{31}+a_{32}-1) \bigr]\\ &{}+a_{32}\bigl[1-a_{21}-a_{23}(a_{31}+a_{32}-1) \bigr]-1 \\ =& a_{31} \underline{u}_{1} + a_{32} \underline{u}_{2} -1=: \underline{u}_{3}. \end{aligned}$$

Repeating the process, we further have

$$\begin{aligned} &\limsup_{t\to\infty} u_{1}(t) \le1-a_{12} \underline {u}_{2}-a_{13}\underline{u}_{3} < 1, \\ &\limsup_{t\to\infty} u_{2}(t) \le1-a_{21} \underline {u}_{2}-a_{23}\underline{u}_{3} < 1 \end{aligned}$$

and

$$0< \limsup_{t\to\infty} u_{3}(t) < a_{31}+a_{32}-1 . $$

By the theory of dynamical systems [52], we see that (2.2) has at least one positive steady state \(K=(k_{1},k_{2},k_{2})\) with

$$0< \underline{u}_{i} < k_{i} < 1,\quad i=1,2, $$

and

$$0< \underline{u}_{3} < k_{3}< a_{31}+a_{32}-1. $$

If K is globally stable, then K is the unique positive steady state, which implies the conclusions in Sect. 2. We now prove the global stability by contracting rectangles.

Lemma 3.1

Assume that (2.3) holds. Denote \(\varSigma(s)=[a(s),b(s)]\) with \(a(s)=(a_{1}(s),a_{2}(s), a_{3}(s))\), \(b(s)=(b_{1}(s),b_{2}(s),b_{3}(s))\) and

$$\begin{aligned} &a_{1}(s)=sk_{1}+(1-s)\underline{u}_{1},\qquad b_{1}(s)=sk_{1}+(1-s), \\ &a_{2}(s)=sk_{2}+(1-s)\underline{u}_{2},\qquad b_{2}(s)=sk_{2}+(1-s), \\ &a_{3}(s)=sk_{3}+(1-s)\underline{u}_{3},\qquad b_{3}(s)=sk_{3}+(1-s)(a_{31}+a_{32}-1). \end{aligned}$$

Then \(\varSigma(s)=[a(s),b(s)],s\in(0,1)\) is a contracting rectangle of (2.2). Moreover, \((k_{1},k_{2},k_{3})\) is globally stable.

Proof

We now verify the definition of a contracting rectangle.

  1. (1)

    (i) If \(u_{1}=sk_{1}+(1-s)\underline{u}_{1}\), then

    $$u_{2}\leq sk_{2}+(1-s),\qquad u_{3} \leq sk_{3}+(1-s) (a_{31}+a_{32}-1) $$

    so that

    $$\begin{aligned} &1-u_{1}-a_{12}u_{2}-a_{13}u_{3} \\ &\quad \geq 1-\bigl[sk_{1}+(1-s)\underline {u}_{1} \bigr]-a_{12}\bigl[sk_{2}+(1-s)\bigr]-a_{13} \bigl[sk_{3}+(1-s) (a_{31}+a_{32}-1)\bigr] \\ &\quad = (1-s)-(1-s)\bigl\{ 1-\bigl[a_{12}+a_{13}(a_{31}+a_{32}-1) \bigr]+\bigl[a_{12}+a_{13}(a_{31}+a_{32}-1) \bigr]\bigr\} \\ &\quad = 0. \end{aligned}$$

    Therefore

    $$f_{1}(u_{1},u_{2},u_{3})|_{u_{1}=sk_{1}+(1-s)\underline{u}_{1}} \geq0. $$

    (ii) If \(u_{1}=sk_{1}+(1-s)\), then

    $$u_{2}\geq sk_{2}+(1-s)\underline{u}_{2},\qquad u_{3} \geq sk_{3}+(1-s)\underline{u}_{3} $$

    so that

    $$\begin{aligned} & 1-u_{1}-a_{12}u_{2}-a_{13}u_{3} \\ & \quad \leq 1-\bigl[sk_{1}+(1-s)\bigr]-a_{12} \bigl[sk_{2}+(1-s)\underline {u}_{2}\bigr]-a_{13} \bigl[sk_{3}+(1-s)\underline{u}_{3}\bigr] \\ &\quad = -(1-s) (a_{12}\underline{u}_{2}+a_{13} \underline{u}_{3}) \\ &\quad < 0. \end{aligned}$$

    Therefore

    $$f_{1}(u_{1},u_{2},u_{3})|_{u_{1}=sk_{1}+(1-s)}< 0. $$
  2. (2)

    (i) If \(u_{2}=sk_{2}+(1-s)\underline{u}_{2}\), then

    $$u_{1}\leq sk_{1}+(1-s),\qquad u_{3} \leq sk_{3}+(1-s) (a_{31}+a_{32}-1) $$

    so that

    $$1-a_{21}u_{1}-u_{2}-a_{23}u_{3} \geq0 $$

    and

    $$f_{2}(u_{1},u_{2},u_{3})|_{u_{2}=sk_{2}+(1-s)\underline{u}_{2}} \geq0. $$

    (ii) If \(u_{2}=sk_{2}+(1-s)\), then

    $$u_{1}\geq sk_{1}+(1-s)\underline{u}_{1},\qquad u_{3} \geq sk_{3}+(1-s)\underline{u}_{3} $$

    so that

    $$1-a_{21}u_{1}-u_{2}-a_{23}u_{3} < 0 $$

    and

    $$f_{2}(u_{1},u_{2},u_{3})|_{u_{2}=sk_{2}+(1-s)}< 0. $$
  3. (3)

    (i) If \(u_{3}=sk_{3}+(1-s)\underline{u}_{3}\), then

    $$u_{1}\geq sk_{1}+(1-s)\underline{u}_{1},\qquad u_{2} \geq sk_{2}+(1-s)\underline{u}_{2} $$

    so that

    $$-1+a_{31}u_{1}+a_{32}u_{2}-u_{3} \geq 0 $$

    and

    $$f_{3}(u_{1},u_{2},u_{3})|_{u_{3}=sk_{3}+(1-s)\underline{u}_{3}} \geq0. $$

    (ii) If \(u_{3}=sk_{3}+(1-s)(a_{31}+a_{32}-1)\), then

    $$u_{1}\leq sk_{1}+(1-s),\qquad u_{2}\leq sk_{2}+(1-s) $$

    so that

    $$-1+a_{31}u_{1}+a_{32}u_{2}-u_{3} \leq0 $$

    and

    $$f_{3}(u_{1},u_{2},u_{3})|_{u_{3}=sk_{3}+(1-s)(a_{31}+a_{32}-1)} \leq0. $$

According to the definition of a contracting rectangle, \(\varSigma (s)=[a(s),b(s)]\) is a contracting rectangle of (2.2). Then \((k_{1},k_{2},k_{3})\) is globally stable by Lemma 2.2. The proof is complete. □

The stability result further implies the following properties.

Theorem 3.2

Assume that Ω is a bounded domain with smooth boundary ∂Ω. Consider the following initial boundary value problem:

$$ \textstyle\begin{cases} \frac{\partial u_{1}(x,t)}{\partial t}=d_{1}\Delta u_{1}(x,t)+r_{1}u_{1}(x,t)[1-u_{1}(x,t)-a_{12}u_{2}(x,t)-a_{13}u_{3}(x,t)],\\\quad x \in\varOmega, t>0, \\ \frac{\partial u_{2}(x,t)}{\partial t}=d_{2}\Delta u_{2}(x,t)+r_{2}u_{2}(x,t)[1-a_{21}u_{1}(x,t)-u_{2}(x,t)-a_{23}u_{3}(x,t)],\\\quad x \in\varOmega, t>0, \\ \frac{\partial u_{3}(x,t)}{\partial t}=d_{3}\Delta u_{3}(x,t)+r_{3}u_{3}(x,t)[-1+a_{31}u_{1}(x,t)+a_{32}u_{2}(x,t)-u_{3}(x,t)],\\\quad x\in \varOmega, t>0, \\ \frac{\partial u_{1}(x,t)}{\partial\mathbf{n}}=\frac{\partial u_{2}(x,t)}{\partial\mathbf{n}}=\frac{\partial u_{3}(x,t)}{\partial\mathbf {n}}=0,\quad x\in \partial\varOmega, t>0, \\ u_{i}(x,0)=\phi_{i}(x)>0,\quad i=1,2,3, x\in\varOmega,\end{cases} $$
(3.1)

where n is the outward unit normal vector of ∂Ω, \(\phi_{i}(x),i=1,2,3\), are continuous and bounded. If (2.3) holds, then

$$\lim_{t\to\infty} u_{i}(\cdot, t)=k_{i},\quad i=1,2,3. $$

Since the boundary condition is of Neumann type, the proof is similar to that of ODEs with nice properties of Laplacian operator, and we omit the proof here. We end this section by making the following remark.

Remark 3.3

The dynamics of this model has been investigated by other methods, for instance, Cantrell et al. [5] studied it under Dirichlet conditions. We now give the proof in order to study the traveling wave solutions in Sect. 5.

4 Existence of the traveling wave solutions

We now investigate the existence of traveling wave solutions and first give the following definition.

Definition 4.1

A traveling wave solution of (2.1) is a special solution taking the form \(u(x,t)=\varPhi(x+ct)\in C^{2}(\mathbb{R},\mathbb{R}^{3})\) with

$$u(x,t)=\bigl(u_{1}(x,t),u_{2}(x,t),u_{3}(x,t)\bigr), \qquad \varPhi(\xi)=\bigl(\phi_{1}(\xi),\phi _{2}(\xi),\phi_{3}(\xi)\bigr),\quad \xi=x+ct, $$

in which Φ is the wave profile that propagates through the one-dimension spatial domain \(\mathbb{R}\) at the constant wave speed \(c > 0\).

By definition, \(\varPhi(\xi)=(\phi_{1}(\xi),\phi_{2}(\xi),\phi_{3}(\xi))\) must satisfy

$$ \textstyle\begin{cases} d_{1}\phi''_{1}(\xi)-c\phi'_{1}(\xi)+r_{1}\phi_{1}(\xi)[1-\phi_{1}(\xi)-a_{12}\phi _{2}(\xi)-a_{13}\phi_{3}(\xi)]=0,\\ d_{2}\phi''_{2}(\xi)-c\phi'_{2}(\xi)+r_{2}\phi_{2}(\xi)[1-a_{21}\phi_{1}(\xi)-\phi _{2}(\xi)-a_{23}\phi_{3}(\xi)]=0,\\ d_{3}\phi''_{3}(\xi)-c\phi'_{3}(\xi)+r_{3}\phi_{3}(\xi)[-1+a_{31}\phi_{1}(\xi )+a_{32}\phi_{2}(\xi)-\phi_{3}(\xi)]=0. \end{cases} $$
(4.1)

Since we shall discuss the dynamical behavior that \(u_{2}\) and \(u_{3}\) invade the habitat of \(u_{1}\), \(\varPhi(\xi)\) will satisfy the following asymptotic boundary conditions:

$$ \begin{aligned} &\lim_{\xi\to-\infty}\bigl(\phi_{1}(\xi), \phi_{2}(\xi),\phi_{3}(\xi)\bigr) =(1,0,0), \\ &\lim_{\xi\to+\infty}\bigl(\phi_{1}(\xi),\phi_{2}( \xi),\phi_{3}(\xi)\bigr) =(k_{1},k_{2},k_{3}). \end{aligned} $$
(4.2)

In population dynamics, positive solutions of (4.1)–(4.2) describe the following biological process: at any fixed location \(x\in \mathbb{R}\), there was only one prey a long time ago (\(t \to-\infty\) such that \(x+ct\to-\infty\)), and the predator and two preys will coexist after a long-term species interaction (\(t\to+\infty\) such that \(x+ct\to+\infty\)).

Letting \(\varPsi(\xi)=(\varphi_{1},\varphi_{2},\varphi_{3})(\xi)=(1-\phi_{1},\phi _{2},\phi_{3})(\xi)\), we obtain

$$ \textstyle\begin{cases} d_{1}\varphi''_{1}(\xi)-c\varphi'_{1}(\xi) +r_{1}[1-\varphi_{1}(\xi)][a_{12}\varphi_{2}(\xi)+a_{13}\varphi_{3}(\xi) -\varphi_{1}(\xi)]=0,\\\quad \xi\in\mathbb{R},\\ d_{2}\varphi''_{2}(\xi)-c\varphi'_{2}(\xi) +r_{2}\varphi_{2}(\xi)[1-a_{21}+a_{21}\varphi_{1}(\xi)-\varphi_{2}(\xi) -a_{23}\varphi_{3}(\xi)]=0,\\\quad \xi\in\mathbb{R},\\ d_{3}\varphi''_{3}(\xi)-c\varphi'_{3}(\xi) +r_{3}\varphi_{3}(\xi)[a_{31}-1+a_{32}\varphi_{2}(\xi)- a_{31}\varphi_{1}(\xi)-\varphi_{3}(\xi)]=0,\\\quad \xi\in\mathbb{R}. \end{cases} $$
(4.3)

Due to (4.2), we have

$$ \begin{aligned} &\lim_{\xi\to-\infty}\bigl(\varphi_{1}(\xi), \varphi_{2}(\xi),\varphi_{3}(\xi )\bigr)=(0,0,0), \\ &\lim_{\xi\to+\infty}\bigl(\varphi_{1}(\xi), \varphi_{2}(\xi),\varphi_{3}(\xi )\bigr)=(1-k_{1},k_{2},k_{3}). \end{aligned} $$
(4.4)

Define a constant

$$ \beta\geq r_{2}\bigl[a_{21}+a_{23}(a_{31}+a_{32}-1) \bigr]+r_{3}(a_{31}+a_{32}) $$
(4.5)

such that

$$\beta\varphi_{1}+r_{1}[1-\varphi_{1}] [a_{12}\varphi_{2} +a_{13}\varphi_{3}- \varphi_{1}] $$

is nondecreasing with respect to \(0\leq\varphi_{1}\leq1\), \(0\leq\varphi _{2}\leq1\), \(0\leq\varphi_{3}\leq a_{31}+a_{32}-1\),

$$\beta\varphi_{2}+r_{2}\varphi_{2}[1-a_{21}+a_{21} \varphi_{1} -\varphi_{2}-a_{23} \varphi_{3}] $$

is nondecreasing with respect to \(0\leq\varphi_{1}\leq1\), \(0\leq\varphi _{2}\leq1\), while it is nonincreasing with respect to \(0\leq\varphi _{3}\leq a_{31}+a_{32}-1\), and

$$\beta\varphi_{3}+r_{3}\varphi_{3}[a_{31}-1+a_{32} \varphi_{2} -a_{31}\varphi_{1}- \varphi_{3}] $$

is nonincreasing with respect to \(0\leq\varphi_{1}\leq1\), while it is nondecreasing with respect to \(0\leq\varphi_{2}\leq1\), \(0\leq\varphi _{3}\leq a_{31}+a_{32}-1\).

For \(\varPsi(\xi)=(\varphi_{1},\varphi_{2},\varphi_{3})(\xi)\in X_{[0,M]}\) with \(M=(1,1,a_{31}+a_{32}-1)\), denote

$$ \textstyle\begin{cases} F_{1}(\varphi_{1},\varphi_{2},\varphi_{3})(\xi)=\beta\varphi_{1}(\xi) +r_{1}[1-\varphi_{1}(\xi)][a_{12}\varphi_{2}(\xi)+a_{13}\varphi_{3}(\xi) -\varphi_{1}(\xi)],\\\quad \xi\in\mathbb{R},\\ F_{2}(\varphi_{1},\varphi_{2},\varphi_{3})(\xi)=\beta\varphi_{2}(\xi) +r_{2}\varphi_{2}(\xi)[1-a_{21}+a_{21}\varphi_{1}(\xi)-\varphi_{2}(\xi) -a_{23}\varphi_{3}(\xi)],\\\quad \xi\in\mathbb{R},\\ F_{3}(\varphi_{1},\varphi_{2},\varphi_{3})(\xi)=\beta\varphi_{3}(\xi) +r_{3}\varphi_{3}(\xi)[a_{31}-1+a_{32}\varphi_{2}(\xi)- a_{31}\varphi_{1}(\xi)-\varphi_{3}(\xi)],\\\quad \xi\in\mathbb{R}. \end{cases} $$

Then (4.3) can be rewritten as

$$ d_{i}\varphi''_{i}( \xi)-c\varphi'_{i}(\xi)-\beta\varphi_{i}(\xi) +F_{i}(\varPsi) (\xi)=0,\quad i=1,2,3. $$
(4.6)

Define constants

$$\lambda_{i1}(c)=\frac{c-\sqrt{c^{2}+4\beta d_{i}}}{2d_{i}},\qquad \lambda _{i2}(c)= \frac{c+\sqrt{c^{2}+4\beta d_{i}}}{2d_{i}},\quad i=1,2,3. $$

Then \(\beta>0\) implies \(\lambda_{i1}<0<\lambda_{i2}\) and

$$d_{i}\lambda^{2}_{i1}-c\lambda_{i1}- \beta=0,\qquad d_{i}\lambda^{2}_{i2}-c\lambda _{i2}-\beta=0,\quad i=1,2,3. $$

For \(\varPsi(\xi)=(\varphi_{1},\varphi_{2},\varphi_{3})(\xi)\in X_{[0,M]}\), define an operator \(P=(P_{1},P_{2},P_{3}):X_{[\mathbf{0},\mathbf{M}]} \to X\) (see Wu and Zou [49]) by

$$ P_{i}(\varPsi) (\xi)=\frac{1}{d_{i}(\lambda_{i2} -\lambda_{i1})} \biggl[ \int_{-\infty} ^{\xi} e^{\lambda_{i1}(\xi-s)}+ \int_{\xi}^{+\infty}e^{\lambda_{i2}(\xi-s)} \biggr] F_{i}(\varPsi) (s)\,ds, $$
(4.7)

where \(i=1,2,3\), \(\xi\in\mathbb{R}\). Then a fixed point of operator P is a solution of (4.3) or (4.6). On the other hand, a solution of (4.3) or (4.6) is a fixed point of operator P (see Huang [22]).

In the following, we will establish the existence of a nontrivial positive solution of (4.3) by combining Schauder’s fixed point theorem with the method of upper and lower solutions (for quasimonotone systems, we refer to [20, 34, 46, 49]). We now introduce the definition of upper and lower solutions of (4.3).

Definition 4.2

\(\overline{\varPsi}(\xi)=(\overline{\varphi}_{1},\overline{\varphi}_{2}, \overline{\varphi}_{3})(\xi)\), \(\underline{\varPsi}(\xi)=(\underline{\varphi}_{1},\underline{\varphi}_{2}, \underline{\varphi}_{3})(\xi)\in X_{[0,M]}\) are a pair of upper and lower solutions of (4.3), if \(\overline{\varPsi}'',\overline{\varPsi}', \underline{\varPsi}'',\underline{\varPsi}'\) are bounded and continuous for each \(\xi\in\mathbb{R}\setminus\mathbb{T}\) with \(\mathbb{T}=\{ T_{1},T_{2},\dots,T_{m}\}\) and they satisfy

$$ \textstyle\begin{cases} d_{1}\overline{\varphi}''_{1}(\xi) -c\overline{\varphi}'_{1}(\xi) +r_{1}[1-\overline{\varphi}_{1}(\xi)][a_{12}\overline{\varphi}_{2}(\xi) +a_{13}\overline{\varphi}_{3}(\xi)-\overline{\varphi}_{1}(\xi)]\leq0,\\ d_{2}\overline{\varphi}''_{2}(\xi) -c\overline{\varphi}'_{2}(\xi) +r_{2}\overline{\varphi}_{2}(\xi)[1-a_{21}+ a_{21}\overline{\varphi}_{1}(\xi)-\overline{\varphi}_{2}(\xi) -a_{23}\underline{\varphi}_{3}(\xi)]\leq0,\\ d_{3}\overline{\varphi}''_{3}(\xi)-c\overline{\varphi}'_{3}(\xi) +r_{3}\overline{\varphi}_{3}(\xi)[a_{31}-1 +a_{32}\overline{\varphi}_{2}(\xi)- a_{31}\underline{\varphi}_{1}(\xi) -\overline{\varphi}_{3}(\xi)]\leq0,\\ d_{1}\underline{\varphi}''_{1}(\xi) -c\underline{\varphi}'_{1}(\xi) +r_{1}[1-\underline{\varphi}_{1}(\xi)] [a_{12}\underline{\varphi}_{2}(\xi) +a_{13}\underline{\varphi}_{3}(\xi) -\underline{\varphi}_{1}(\xi)]\geq0,\\ d_{2}\underline{\varphi}''_{2}(\xi)-c\underline{\varphi}'_{2}(\xi) +r_{2}\underline{\varphi}_{2}(\xi)[1-a_{21} +a_{21}\underline{\varphi}_{1}(\xi)-\underline{\varphi}_{2}(\xi) -a_{23}\overline{\varphi}_{3}(\xi)]\geq0,\\ d_{3}\underline{\varphi}''_{3}(\xi)-c\underline{\varphi}'_{3}(\xi) +r_{3}\underline{\varphi}_{3}(\xi)[a_{31}-1 +a_{32}\underline{\varphi}_{2}(\xi) -a_{31}\overline{\varphi}_{1}(\xi) -\underline{\varphi}_{3}(\xi)]\geq0. \end{cases} $$
(4.8)

Lemma 4.3

Assume that (4.3) has a pair of upper and lower solutions satisfying

  1. (1)

    \(\underline{\varPsi}(\xi)\leq\overline{\varPsi}(\xi)\), \(\xi\in \mathbb{R}\);

  2. (2)

    \(\underline{\varPsi}'(\xi_{-})\leq\overline{\varPsi}'(\xi_{+})\), \(\overline{\varPsi}'(\xi_{+})\leq\overline{\varPsi}'(\xi_{-})\), \(\xi\in\mathbb {T}\), herein

    $$\underline{\varPsi}'(\xi_{\pm})=\lim _{t \to\xi_{\pm}}\underline{ \varPsi }'(t),\qquad \overline{\varPsi}'(\xi_{\pm})= \lim _{t \to\xi_{\pm}}\overline {\varPsi}'(t). $$

    Then (4.3) has a positive solution \(\varPsi(\xi)\) such that \(\underline{\varPsi}(\xi)\leq\varPsi(\xi)\leq\overline{\varPsi}(\xi)\).

Proof

We prove this lemma by Schauder’s fixed point theorem. Since a similar result has been proved in several earlier papers mentioned above [31], we only give the scheme.

Define

$$\begin{aligned} &B_{\mu}\bigl(\mathbb{R},\mathbb{R}^{3}\bigr)=\Bigl\{ u\in X: \sup _{\xi\in\mathbb {R}}\bigl\{ \bigl\Vert \mathbf{u}(\xi) \bigr\Vert e^{-\mu \vert \xi \vert }\bigr\} < \infty\Bigr\} , \\ &\bigl\vert \mathbf{u}(\xi) \bigr\vert _{\mu}=\sup _{\xi\in\mathbb{R}} \bigl\{ \bigl\Vert \mathbf {u}(\xi) \bigr\Vert e^{-\mu \vert \xi \vert }\bigr\} , \end{aligned}$$

where

$$ \mu\in\bigl(0,\min\{-\lambda_{11},-\lambda_{21},- \lambda_{31}\}\bigr), $$

then \((B_{\mu}(\mathbb{R},\mathbb{R}^{3}),|\cdot|_{\mu})\) is a Banach space. Let

$$ \varLambda=\bigl\{ \varPsi(\xi)\in X_{[\mathbf{0},\mathbf{M}]}: \underline{\varPsi}(\xi)\leq\varPsi(\xi) \leq\overline{\varPsi}(\xi)\bigr\} . $$

Obviously, Λ is nonempty and convex. It is also closed and bounded with respect to the decay norm \(|\cdot|_{\mu}\).

We now verify that \(P:\varLambda\to\varLambda\). For \(\varPsi(\xi)=(\varphi_{1},\varphi_{2},\varphi_{3})(\xi)\in\varLambda\) and each fixed \(\xi\in\mathbb{R}\), the definition of operator P and the choice of β imply that it suffices to prove that

$$ \textstyle\begin{cases} \underline{\varphi}_{1}(\xi)\leq P_{1}(\underline{\varphi}_{1},\underline {\varphi}_{2}, \underline{\varphi}_{3})(\xi)\leq P_{1}(\overline{\varphi}_{1},\overline {\varphi}_{2}, \overline{\varphi}_{3})(\xi) \leq\overline{\varphi}_{1}(\xi),\\ \underline{\varphi}_{2}(\xi)\leq P_{2}(\underline{\varphi}_{1},\underline {\varphi}_{2}, \overline{\varphi}_{3})(\xi)\leq P_{2}(\overline{\varphi}_{1},\overline{\varphi}_{2}, \underline{\varphi}_{3})(\xi) \leq\overline{\varphi}_{2}(\xi),\\ \underline{\varphi}_{3}(\xi)\leq P_{3}(\overline{\varphi}_{1},\underline {\varphi}_{2}, \underline{\varphi}_{3})(\xi)\leq P_{3}(\underline{\varphi}_{1},\overline{\varphi}_{2}, \overline{\varphi}_{3})(\xi) \leq\overline{\varphi}_{3}(\xi). \end{cases} $$
(4.9)

Without loss of generality, we assume that \(T_{1}< T_{2}<\cdots<T_{m}\) and denote \(T_{0}=-\infty\), \(T_{m+1}=+\infty\). If \(\xi\in\mathbb {R}\backslash\mathbb{T}\), namely, \(\xi\in(T_{k},T_{k+1})\) with some \(k\in \{0,1,\dots,m\}\), then

$$\begin{aligned} & P_{1}(\underline{\varphi}_{1},\underline{ \varphi}_{2},\underline{\varphi }_{3}) (\xi) \\ &\quad = \frac{1}{d_{1}(\lambda_{12}-\lambda_{11})} \biggl[ \int^{\xi}_{-\infty }e^{\lambda_{11}(\xi-s)} + \int^{+\infty}_{\xi}e^{\lambda_{12}(\xi-s)} \biggr] F_{1}(\underline{\varphi}_{1},\underline{ \varphi}_{2},\underline{\varphi }_{3}) (s)\,ds \\ &\quad = \frac{1}{d_{1}(\lambda_{12}-\lambda_{11})} \biggl[ \int^{\xi}_{-\infty }e^{\lambda_{11}(\xi-s)} + \int^{+\infty}_{\xi}e^{\lambda_{12}(\xi-s)} \biggr] \bigl[\beta \underline{\varphi}_{1}(s)+c\underline{\varphi }'_{1}(s)-d_{1} \underline{\varphi}''_{1}(s) \bigr]\,ds \\ &\quad \geq \underline{\varphi}_{1}(\xi)+\frac{1}{\lambda_{12}-\lambda_{11}} \Biggl[\sum _{j=1}^{k}e^{\lambda_{11}(\xi-T_{j})} \bigl(\underline{ \varphi}'_{1}(T_{j+})-\underline{\varphi }'_{1}(T_{j-}) \bigr) \\ &\qquad {}+ \sum _{j=k+1}^{m}e^{\lambda_{12}(\xi-T_{j})} \bigl( \underline {\varphi}'_{1}(T_{j+}) -\underline{ \varphi}'_{1}(T_{j-}) \bigr) \Biggr] \\ &\quad \geq \underline{\varphi}_{1}(\xi). \end{aligned}$$

Because ξ was arbitrary and due to the continuity, we have \(P_{1}(\underline{\varphi}_{1},\underline{\varphi}_{2},\underline{\varphi }_{3})(\xi)\geq\underline{\varphi}_{1}(\xi)\) in \(\mathbb{R}\). In a similar way, we can verity the remainder of (4.9).

Note that the compactness in [24] is independent of the monotonicity, then \(P:\varLambda\to\varLambda\) is compact in the sense of the decay norm \(|\cdot|_{\mu}\) by a discussion similar to that in [24].

By Schauder’s fixed point theorem, there exists \(\varPsi(\xi)=(\varphi _{1},\varphi_{2},\varphi_{3})(\xi)\in\varLambda\) which is a positive solution of (4.3) satisfying \(\underline{\varPsi}(\xi)\leq\varPsi(\xi)\leq \overline{\varPsi}(\xi)\). The proof is complete. □

Next, we construct the upper and lower solutions of (4.3), and we assume that

$$ a_{31}>1. $$
(4.10)

For any fixed

$$c>\max\bigl\{ 2\sqrt{d_{2}r_{2} (1-a_{21}) }, 2 \sqrt{d_{3}r_{3} (a_{31}-1)}\bigr\} :=c^{*}, $$

we define positive constants \(\gamma_{21}< \gamma_{22},\gamma_{31}<\gamma_{32}\) such that

$$\begin{aligned} &d_{2}\gamma^{2}_{21}-c\gamma_{21}+r_{2}(1-a_{21})=d_{2} \gamma^{2}_{22}-c\gamma _{22}+r_{2}(1-a_{21})=0, \\ &d_{3}\gamma^{2}_{31}-c\gamma_{31}+r_{3}(a_{31}-1)=d_{3} \gamma^{2}_{32}-c\gamma _{32}+r_{3}(a_{31}-1)=0. \end{aligned}$$

Further choose \(\epsilon>0\) such that

$$\gamma_{21}+\epsilon< \min\{2\gamma_{21}, \gamma_{22},\gamma_{21}+\gamma _{31}\}, $$

and

$$\gamma_{31}+\epsilon< \min\{2\gamma_{31}, \gamma_{32}, \gamma_{21}+\gamma _{31}\}. $$

Let

$$\gamma_{11}=\min \{\gamma_{21},\gamma_{31} \}. $$

We now assume that

$$ d_{1}\gamma_{11}^{2} -c \gamma_{11} < 0. $$
(4.11)

Define \(\varGamma=(\gamma_{21}, \gamma_{22}) \cap(\gamma_{31},\gamma_{32})\), then Γ is nonempty if c is large enough or other parameters satisfy suitable conditions. In particular, when Γ is nonempty, we further assume that there exists \(\gamma\in\varGamma\) such that

$$ d_{1}\gamma^{2} -c\gamma< 0,\quad \gamma< \gamma_{21}+\gamma_{31}. $$
(4.12)

Remark 4.4

Assume that all the parameters in (2.1) are fixed. Then there exists \(c'\ge c^{*}\) such that (4.11)–(4.12) hold.

For any given \(c>c^{*}\), we now fix these constants and define continuous functions as follows:

$$\begin{aligned} &\overline{\varphi}_{1}(\xi)=\min\bigl\{ 1,e^{\gamma_{11}\xi}+pe^{\gamma\xi} \bigr\} ,\qquad \underline{\varphi}_{1}(\xi)=0, \\ &\overline{\varphi}_{2}(\xi)=\min\bigl\{ 1,e^{\gamma_{21}\xi}+pe^{\gamma\xi} \bigr\} ,\qquad \underline{\varphi}_{2}(\xi)=\max\bigl\{ 0,e^{\gamma_{21}\xi}-qe^{(\gamma _{21}+\epsilon)\xi} \bigr\} , \\ &\overline{\varphi}_{3}(\xi)=\min\bigl\{ a_{31}+a_{32}-1,e^{\gamma_{31}\xi }+pa_{32}e^{\gamma\xi} \bigr\} ,\qquad \underline{\varphi}_{3}(\xi)=\max\bigl\{ 0,e^{\gamma_{31}\xi}-qe^{(\gamma _{31}+\epsilon)\xi} \bigr\} , \end{aligned}$$

in which \(p>1\), \(q>1\) will be clarified in the following lemma.

Lemma 4.5

Assume that \(c>c^{*}\). Further suppose that Γ is nonempty such that (4.11)–(4.12) hold. Then there exist \(p, q\) such that \(\overline{\varPsi}(\xi)=(\overline{\varphi}_{1}(\xi ),\overline{\varphi}_{2}(\xi),\overline{\varphi}_{3}(\xi))\) and \(\underline {\varPsi}(\xi)=(\underline{\varphi}_{1}(\xi), \underline{\varphi}_{2}(\xi ),\underline{\varphi}_{3}(\xi))\) are a pair of upper and lower solutions of (4.3).

Proof

It suffices to verify (4.8) one by one.

  1. (1)

    (i) If \(\overline{\varphi}_{1}(\xi)=1< e^{\gamma_{11}\xi }+pe^{\gamma\xi}\), then

    $$d_{1}\overline{\varphi}''_{1}( \xi)-c\overline{\varphi}'_{1}(\xi) +r_{1} \bigl[1-\overline{\varphi}_{1}(\xi)\bigr] \bigl[a_{12} \overline{\varphi}_{2}(\xi)+ a_{13}\overline{ \varphi}_{3}(\xi)-\overline{\varphi}_{1}(\xi)\bigr]=0. $$

    (ii) If \(\overline{\varphi}_{1}(\xi)=e^{\gamma_{11}\xi}+pe^{\gamma\xi }<1\), then \(\xi<0\) and

    $$\overline{\varphi}_{2}(\xi)\leq e^{\gamma_{21}\xi}+pe^{\gamma\xi },\qquad \overline{\varphi}_{3}(\xi)\leq e^{\gamma_{31}\xi}+pa_{32}e^{\gamma _{31}\xi} $$

    so that

    $$\begin{aligned} & a_{12}\overline{\varphi}_{2}(\xi) +a_{13} \overline{\varphi}_{3}(\xi)-\overline{\varphi}_{1}(\xi) \\ &\quad \leq a_{12}\bigl(e^{\gamma_{21}\xi}+pe^{\gamma\xi}\bigr) +a_{13}\bigl(e^{\gamma_{31}\xi}+pa_{32}e^{\gamma\xi}\bigr)- \bigl(e^{\gamma_{11}\xi }+pe^{\gamma\xi}\bigr) \\ &\quad \leq (a_{12}+a_{13}-1)e^{\gamma_{11}\xi}+(a_{12}+a_{13}a_{32}-1)pe^{\gamma\xi } \\ &\quad < 0 \end{aligned}$$

    because \(\gamma_{11}= \min\{\gamma_{21},\gamma_{31}\}\), \(a_{12}+a_{13}a_{32}<1\) and \(a_{12}+a_{13}<1\). Therefore, we have

    $$\begin{aligned} & d_{1}\overline{\varphi}''_{1}( \xi)-c\overline{\varphi}'_{1}(\xi) +r_{1} \bigl[1-\overline{\varphi}_{1}(\xi)\bigr] \bigl[a_{12} \overline{\varphi}_{2}(\xi) +a_{13}\overline{ \varphi}_{3}(\xi)-\overline{\varphi}_{1}(\xi)\bigr] \\ &\quad \leq d_{1}\overline{\varphi}''_{1}( \xi)-c\overline{\varphi}'_{1}(\xi) \\ &\quad =\bigl(d_{1}\gamma^{2}_{11}-c \gamma_{11}\bigr)e^{\gamma_{11}\xi}+\bigl(d_{1}\gamma ^{2}-c\gamma\bigr)pe^{\gamma\xi} \\ &\quad < 0. \end{aligned}$$
  2. (2)

    (i) If \(\overline{\varphi}_{2}(\xi)=1< e^{\gamma_{21}\xi }+pe^{\gamma\xi}\), then \(\overline{\varphi}_{1}(\xi)\leq1\) so that

    $$\begin{aligned} & d_{2}\overline{\varphi}''_{2}( \xi)-c\overline{\varphi}'_{2}(\xi) +r_{2} \overline{\varphi}_{2}(\xi)\bigl[1-a_{21}+a_{21} \overline{\varphi}_{1}(\xi )-\overline{\varphi}_{2}(\xi) -a_{23}\underline{\varphi}_{3}(\xi)\bigr] \\ & \quad \leq d_{2}\overline{\varphi}''_{2}( \xi)-c\overline{\varphi}'_{2}(\xi) +r_{2} \overline{\varphi}_{2}(\xi)\bigl[1-a_{21}+a_{21} \overline{\varphi}_{1}(\xi )-\overline{\varphi}_{2}(\xi) \bigr] \\ &\quad \leq r_{2}(1-a_{21}+a_{21}-1) \\ &\quad = 0. \end{aligned}$$

    (ii) If \(\overline{\varphi}_{2}(\xi)=e^{\gamma_{21}\xi}+pe^{\gamma\xi}<1\), then

    $$\begin{aligned} & d_{2}\overline{\varphi}''_{2}( \xi)-c\overline{\varphi}'_{2}(\xi) +r_{2} \overline{\varphi}_{2}(\xi)\bigl[1-a_{21}+a_{21} \overline{\varphi}_{1}(\xi)- \overline{\varphi}_{2}( \xi)-a_{23}\underline{\varphi}_{3}(\xi)\bigr] \\ &\quad \leq d_{2}\overline{\varphi}''_{2}( \xi)-c\overline{\varphi}'_{2}(\xi) +r_{2} \overline{\varphi}_{2}(\xi)\bigl[1-a_{21}+a_{21} \overline{\varphi}_{1}(\xi )-\overline{\varphi}_{2}(\xi) \bigr] \\ &\quad \leq d_{2}\bigl(\gamma^{2}_{21}e^{\gamma_{21}\xi}+p \gamma^{2}e^{\gamma\xi}\bigr) -c\bigl(\gamma_{21}e^{\gamma_{21}\xi}+ p\gamma e^{\gamma\xi}\bigr) \\ &\qquad {}+r_{2}\bigl(e^{\gamma_{21}\xi} +pe^{\gamma\xi}\bigr) \bigl[1-a_{21}+a_{21}\bigl(e^{\gamma_{11}\xi}+p e^{\gamma\xi } \bigr)-\bigl(e^{\gamma_{21}\xi}+p e^{\gamma\xi}\bigr)\bigr]. \end{aligned}$$

    If \(\gamma_{11}=\gamma_{21}\le\gamma_{31}\), then

    $$a_{21}\bigl(e^{\gamma_{11}\xi}+p e^{\gamma\xi}\bigr)\le \bigl(e^{\gamma_{21}\xi}+p e^{\gamma\xi}\bigr) $$

    so that

    $$\begin{aligned} & d_{2}\overline{\varphi}''_{2}( \xi)-c\overline{\varphi}'_{2}(\xi) +r_{2} \overline{\varphi}_{2}(\xi)\bigl[1-a_{21}+a_{21} \overline{\varphi}_{1}(\xi)- \overline{\varphi}_{2}( \xi)-a_{23}\underline{\varphi}_{3}(\xi)\bigr] \\ &\quad \leq d_{2}\bigl(\gamma^{2}_{21}e^{\gamma_{21}\xi}+p \gamma^{2}e^{\gamma\xi}\bigr) -c\bigl(\gamma_{21}e^{\gamma_{21}\xi}+ p\gamma e^{\gamma\xi}\bigr)+r_{2}\bigl(e^{\gamma _{21}\xi} +pe^{\gamma\xi}\bigr)[1-a_{21}] \\ &\quad \leq 0 \end{aligned}$$

    by the definitions of \(\gamma_{21}\) and γ. Otherwise, \(\gamma_{11}=\gamma_{31}<\gamma_{21}\) so that

    $$\gamma< \gamma_{11}+\gamma_{21} $$

    and

    $$\begin{aligned} & d_{2}\overline{\varphi}''_{2}( \xi)-c\overline{\varphi}'_{2}(\xi) +r_{2} \overline{\varphi}_{2}(\xi)\bigl[1-a_{21}+a_{21} \overline{\varphi}_{1}(\xi)- \overline{\varphi}_{2}( \xi)-a_{23}\underline{\varphi}_{3}(\xi)\bigr] \\ &\quad \leq d_{2}\bigl(\gamma^{2}_{21}e^{\gamma_{21}\xi}+p \gamma^{2}e^{\gamma\xi}\bigr) -c\bigl(\gamma_{21}e^{\gamma_{21}\xi}+ p\gamma e^{\gamma\xi}\bigr)\\ &\qquad {}+r_{2}\bigl(e^{\gamma _{21}\xi} +pe^{\gamma\xi}\bigr)\bigl[1-a_{21}+a_{21}e^{\gamma_{11}\xi} \bigr] \\ &\quad \leq \bigl[d_{2}\gamma^{2}-c\gamma+r_{2}(1-a_{21}) \bigr]pe^{\gamma\xi}+r_{2}a_{21}e^{(\gamma _{11}+\gamma_{21})\xi}+pr_{2}a_{21}e^{(\gamma_{11}+\gamma)\xi} \\ &\quad \leq 0 \end{aligned}$$

    provided that

    $$ \bigl[d_{2}\gamma^{2}-c\gamma+r_{2}(1-a_{21}) \bigr]pe^{\gamma\xi} +2r_{2}a_{21}e^{(\gamma_{11}+\gamma_{21})\xi}\leq0 $$
    (4.13)

    and

    $$ \bigl[d_{2}\gamma^{2}-c\gamma+r_{2}(1-a_{21}) \bigr] +2r_{2}a_{21}e^{\gamma_{11}\xi}\leq0. $$
    (4.14)

    Clearly, (4.13) is true if

    $$p>1-\frac{2r_{2}a_{21}}{d_{2}\gamma^{2}-c\gamma+r_{2}(1-a_{21})}:=p_{1} (>1) $$

    and (4.14) is true if

    $$p> \biggl(\frac{2r_{2}a_{21}}{-(d_{2}\gamma^{2}-c\gamma+r_{2}(1-a_{21}))} \biggr)^{\frac{\gamma}{\gamma_{11}}}+1:=p_{2}>1. $$
  3. (3)

    (i) If \(\overline{\varphi}_{31}(\xi )=a_{31}+a_{32}-1< e^{\gamma_{31}\xi}+pa_{32}e^{\gamma\xi}\), then \(\overline{\varphi}_{2}(\xi)\leq1\) so that

    $$\begin{aligned} & d_{3}\overline{\varphi}''_{3}( \xi)-c\overline{\varphi}'_{3}(\xi)+r_{3}\overline {\varphi}_{3}(\xi)\bigl[a_{31}-1 +a_{32}\overline{ \varphi}_{2}(\xi)-a_{31}\underline{\varphi}_{1}(\xi )-\overline{\varphi}_{3}(\xi)\bigr] \\ &\quad \leq d_{3}\overline{\varphi}''_{3}( \xi)-c\overline{\varphi}'_{3}(\xi)+r_{3}\overline {\varphi}_{3}(\xi)\bigl[a_{31}-1 +a_{32}-\overline{ \varphi}_{3}(\xi)\bigr] \\ &\quad = 0. \end{aligned}$$

    (ii) If \(\overline{\varphi}_{3}(\xi)=e^{\gamma_{31}\xi}+pa_{32}e^{\gamma \xi}< a_{31}+a_{32}-1\), then \(\overline{\varphi}_{2}(\xi)\leq e^{\gamma_{21}\xi}+pe^{\gamma\xi}\) so that

    $$\begin{aligned} & d_{3}\overline{\varphi}''_{3}( \xi)-c\overline{\varphi}'_{3}(\xi) +r_{3} \overline{\varphi}_{3}(\xi)\bigl[a_{31}-1+a_{32} \overline{\varphi}_{2}(\xi)- a_{31}\underline{ \varphi}_{1}(\xi)-\overline{\varphi}_{3}(\xi)\bigr] \\ &\quad \leq d_{3}\overline{\varphi}''_{3}( \xi)-c\overline{\varphi}'_{3}(\xi) +r_{3} \overline{\varphi}_{3}(\xi)\bigl[a_{31}-1+a_{32} \overline{\varphi}_{2}(\xi )-\overline{\varphi}_{3}(\xi)\bigr] \\ &\quad \leq \bigl[d_{3}\gamma^{2}_{3}-c \gamma_{31}+r_{3}(a_{31}-1)\bigr]e^{\gamma_{31}\xi} + \bigl[d_{3}\gamma^{2}-c\gamma+r_{3}(a_{31}-1) \bigr]pa_{32}e^{\gamma\xi} \\ &\qquad {}+ r_{3}\bigl(e^{\gamma_{31}\xi}+pa_{32}e^{\gamma\xi} \bigr)\bigl[a_{32}\bigl(e^{\gamma _{21}\xi}+pe^{\gamma\xi}\bigr)- \bigl(e^{\gamma_{31}\xi}+pa_{32}e^{\gamma\xi}\bigr)\bigr] \\ &\quad \leq \bigl[d_{3}\gamma^{2}-c\gamma+r_{3}(a_{31}-1) \bigr]pa_{32}e^{\gamma\xi }+r_{3}a_{32}e^{(\gamma_{21}+\gamma_{31})\xi} +r_{3}{a_{32}}^{2}pe^{(\gamma_{21}+\gamma)\xi} \\ &\quad \leq 0 \end{aligned}$$

    provided that

    $$ \bigl[d_{3}\gamma^{2}-c\gamma+r_{3}(a_{31}-1) \bigr]pe^{\gamma\xi}+2r_{3}e^{(\gamma _{21}+\gamma_{31})\xi}\leq0 $$
    (4.15)

    and

    $$ \bigl[d_{3}\gamma^{2}-c\gamma+r_{3}(a_{31}-1) \bigr]+2r_{3}a_{32}e^{\gamma_{21}\xi}\leq0. $$
    (4.16)

    For (4.15), since \(\gamma< \gamma_{21}+\gamma_{31}\), we have

    $$\begin{aligned} & \bigl[d_{3}\gamma^{2}-c\gamma+r_{3}(a_{31}-1) \bigr]pe^{\gamma\xi}+2r_{3}e^{(\gamma _{21}+\gamma_{31})\xi} \\ &\quad\leq e^{\gamma\xi}\bigl\{ \bigl[d_{3}\gamma^{2}-c \gamma+r_{3}(a_{31}-1)\bigr]p+2r_{3}\bigr\} , \end{aligned}$$

    which is true if

    $$p\geq\frac{-2r_{3}}{d_{3}\gamma^{2}-c\gamma+r_{3}(a_{31}-1)}+1:=p_{3}. $$

    Since \(e^{\gamma_{31}\xi}+pa_{32}e^{\gamma\xi}< a_{31}+a_{32}-1\), then

    $$pe^{\gamma\xi}< \frac{a_{31}+a_{32}-1}{a_{32}} $$

    so that

    $$\xi< \frac{1}{\gamma}\ln\frac{a_{31}+a_{32}-1}{pa_{32}} $$

    and

    $$e^{\gamma_{21}\xi}< \biggl( \frac{a_{31}+a_{32}-1}{pa_{32}} \biggr)^{\frac{\gamma_{21}}{\gamma}}. $$

    Clearly, (4.16) holds if

    $$\biggl( \frac{a_{31}+a_{32}-1}{pa_{32}} \biggr)^{\frac{\gamma _{21}}{\gamma}} < \frac{d_{3}\gamma^{2}-c\gamma+r_{3}(a_{31}-1)}{-2r_{3}a_{32}}, $$

    which is true provided that

    $$p> \frac{a_{31}+a_{32}-1}{a_{32}} \biggl[ \biggl(\frac{d_{3}\gamma^{2}-c\gamma +r_{3}(a_{31}-1)}{-2r_{3}a_{32}} \biggr)^{\frac{\gamma}{\gamma_{21}+\gamma _{31}}} +1 \biggr]:=p_{4}. $$
  4. (4)

    If \(\underline{\varphi}_{1}(\xi)=0\), then

    $$\begin{aligned} & d_{1}\underline{\varphi}''_{1}( \xi)-c\underline{\varphi}'_{1}(\xi) +r_{1} \bigl[1-\underline{\varphi}_{1}(\xi)\bigr] \bigl[a_{12} \underline{\varphi}_{2}(\xi) +a_{13}\underline{ \varphi}_{3}(\xi)-\underline{\varphi}_{1}(\xi)\bigr] \\ &\quad = r_{1}\bigl[a_{12}\underline{\varphi}_{2}( \xi)+a_{13}\underline{\varphi}_{3}(\xi)\bigr] \\ &\quad \geq 0. \end{aligned}$$
  5. (5)

    (i) If \(\underline{\varphi}_{2}(\xi)=0>e^{\gamma_{21}\xi }-qe^{(\gamma_{21}+\epsilon)\xi}\), then

    $$d_{2}\underline{\varphi}''_{2}( \xi)-c\underline{\varphi}'_{2}(\xi) +r_{2} \underline{\varphi}_{2}(\xi)\bigl[1-a_{21}+ a_{21} \underline{\varphi}_{1}(\xi)-\underline{\varphi}_{2}(\xi )-a_{23}\overline{\varphi}_{3}(\xi)\bigr]=0. $$

    (ii) If \(\underline{\varphi}_{2}(\xi)=e^{\gamma_{21}\xi}-qe^{(\gamma _{21}+\epsilon)\xi}>0\), then

    $$\underline{\varphi}_{2}(\xi)< e^{\gamma_{21}\xi},\qquad\overline{\varphi }_{3}(\xi)\leq e^{\gamma_{31}\xi}+pa_{32}e^{\gamma\xi} $$

    so that

    $$\begin{aligned} & d_{2}\underline{\varphi}''_{2}( \xi)-c\underline{\varphi}'_{2}(\xi) +r_{2} \underline{\varphi}_{2}(\xi)\bigl[1-a_{21}+a_{21} \underline{\varphi}_{1}(\xi )-\underline{\varphi}_{2}(\xi) -a_{23}\overline{\varphi}_{3}(\xi)\bigr] \\ &\quad \geq d_{2}\underline{\varphi}''_{2}( \xi)-c\underline{\varphi}'_{2}(\xi) +r_{2} \underline{\varphi}_{2}(\xi)\bigl[1-a_{21}-\underline{ \varphi}_{2}(\xi )-a_{23}\overline{\varphi}_{3}(\xi) \bigr] \\ &\quad \geq d_{2}\bigl[\gamma^{2}_{2}e^{\gamma_{21}\xi}-q( \gamma_{21}+\epsilon)^{2}e^{(\gamma _{21}+\epsilon)\xi}\bigr] -c\bigl[ \gamma_{21}e^{\gamma_{21}\xi}-q(\gamma_{21}+ \epsilon)e^{(\gamma _{21}+\epsilon)\xi}\bigr] \\ &\qquad {}+r_{2}\bigl(e^{\gamma_{21}\xi}-qe^{(\gamma_{21}+\epsilon)\xi }\bigr) \bigl[1-a_{21}-e^{\gamma_{21}\xi}-a_{23}\bigl(e^{\gamma_{31}\xi} +pa_{32}e^{\gamma\xi}\bigr)\bigr] \\ &\quad \geq -\bigl[d_{2}(\gamma_{21}+\epsilon)^{2}-c( \gamma_{21}+\epsilon )+r_{2}(1-a_{21}) \bigr]qe^{(\gamma_{21}+\epsilon)\xi} \\ &\qquad {} -r_{2}a_{23}\bigl[e^{(\gamma_{21}+\gamma_{31})\xi}+pa_{32}e^{(\gamma+\gamma _{21})\xi} \bigr]-r_{2}e^{2\gamma_{21} \xi} \\ &\quad \geq {-}\bigl[d_{2}(\gamma_{21}+\epsilon)^{2}-c( \gamma_{21}+\epsilon )+r_{2}(1-a_{21}) \bigr]qe^{(\gamma_{21}+\epsilon)\xi} \\ &\qquad {}-r_{2}e^{(\gamma_{21}+\epsilon)\xi}\bigl[1+a_{23}(1+a_{32}p) \bigr]. \end{aligned}$$

    Let

    $$q\geq q_{1}=1+\frac{-r_{2}[1+a_{23}(1+a_{32}p)]}{d_{2}(\gamma_{21}+\epsilon )^{2}-c(\gamma_{21}+\epsilon)+r_{2}(1-a_{21})}, $$

    then

    $$d_{2}\underline{\varphi}''_{2}( \xi)-c\underline{\varphi}'_{2}(\xi) +r_{2} \underline{\varphi}_{2}(\xi)\bigl[1-a_{21} +a_{21} \underline{\varphi}_{1}(\xi)-\underline{\varphi}_{2}(\xi )-a_{23}\overline{\varphi}_{3}(\xi)\bigr]\geq0. $$
  6. (6)

    (i) If \(\underline{\varphi}_{3}(\xi)=0>e^{\gamma_{31}\xi}-qe^{(\gamma _{31}+\epsilon)\xi}\), then

    $$d_{3}\underline{\varphi}''_{3}( \xi)-c\underline{\varphi}'_{3}(\xi) +r_{3} \underline{\varphi}_{3}(\xi)\bigl[a_{31}-1 +a_{32} \underline{\varphi}_{2}(\xi)-a_{31}\overline{ \varphi}_{1}(\xi )-\underline{\varphi}_{3}(\xi)\bigr]=0. $$

    (ii) If \(\underline{\varphi}_{3}(\xi)=e^{\gamma_{31}\xi}-qe^{(\gamma _{31}+\epsilon)\xi}>0\), then

    $$\overline{\varphi}_{1}(\xi)\leq e^{\gamma_{11}\xi}+pe^{\gamma\xi},\qquad \underline{\varphi}_{3}(\xi)< e^{\gamma_{31}\xi} $$

    so that

    $$\begin{aligned} & d_{3}\underline{\varphi}''_{3}( \xi)-c\underline{\varphi}'_{3}(\xi) +r_{3} \underline{\varphi}_{3}(\xi)\bigl[a_{31}-1+ a_{32} \underline{\varphi}_{2}(\xi)-a_{31}\overline{ \varphi}_{1}(\xi )-\underline{\varphi}_{3}(\xi)\bigr] \\ &\quad \geq d_{3}\underline{\varphi}''_{3}( \xi)-c\underline{\varphi}'_{3}(\xi) +r_{3} \underline{\varphi}_{3}(\xi)\bigl[a_{31}-1-a_{31} \overline{\varphi}_{1}(\xi )-\underline{\varphi}_{3}(\xi) \bigr] \\ &\quad \geq d_{3}\bigl[\gamma_{31}^{2}e^{\gamma_{31}\xi}-q( \gamma_{31}+\epsilon)^{2}e^{(\gamma _{31}+\epsilon)\xi}\bigr] -c\bigl[ \gamma_{31}e^{\gamma_{31}\xi}-q(\gamma_{31}+ \epsilon)e^{(\gamma _{31}+\epsilon)\xi}\bigr] \\ &\qquad {}+ r_{3}\bigl(e^{\gamma_{31}\xi}-qe^{(\gamma_{31}+\epsilon)\xi }\bigr) \bigl[a_{31}-1-a_{31}\bigl(e^{\gamma_{11}\xi}+ pe^{\gamma\xi} \bigr)-e^{\gamma_{31}\xi}\bigr] \\ &\quad \geq -\bigl[d_{3}(\gamma_{31}+\epsilon)^{2}-c( \gamma_{31}+\epsilon )+r_{3}(a_{31}-1) \bigr]qe^{(\gamma_{31}+\epsilon)\xi} \\ &\qquad {} -r_{3}a_{31}\bigl[e^{(\gamma_{31}+\gamma_{11})\xi}+pe^{(\gamma_{31}+\gamma )\xi} \bigr]-r_{3}e^{2\gamma_{31}\xi} \\ &\quad \geq -\bigl[d_{3}(\gamma_{31}+\epsilon)^{2}-c( \gamma_{31}+\epsilon )+r_{3}(a_{31}-1) \bigr]qe^{(\gamma_{31}+\epsilon)\xi} \\ &\qquad {}-r_{3}e^{(\gamma_{31}+\epsilon)\xi}\bigl[1+a_{31}(1+p) \bigr]. \end{aligned}$$

    Let

    $$q\geq q_{2}=1+\frac{-r_{3}[1+a_{31}(1+p)]}{d_{3}(\gamma_{31}+\epsilon )^{2}-c(\gamma_{31}+\epsilon) +r_{3}(a_{31}-1)}, $$

    then

    $$d_{3}\underline{\varphi}''_{3}( \xi)-c\underline{\varphi}'_{3}(\xi) +r_{3} \underline{\varphi}_{3}(\xi)\bigl[a_{31}-1+a_{32} \underline{\varphi}_{2}(\xi) -a_{31}\overline{ \varphi}_{1}(\xi)-\underline{\varphi}_{3}(\xi)\bigr]\geq0. $$

By what we have done, we first fix \(p=p_{1}+p_{2}+p_{3}+p_{4}\), then let \(q=q_{1}+q_{2}\), which completes the proof. □

Summarizing the above, we have the following conclusions.

Theorem 4.6

Assume that (4.10) holds. If \(c>c^{*}\) is such that (4.11)–(4.12) are true, then (4.3) has a nonconstant positive solution.

About the traveling wave solution, we also give the following remark.

Remark 4.7

By direct calculations in P, we see that \(\varphi_{i}'(\xi),\varphi _{i}''(\xi), i=1,2,3\), are uniformly bounded.

5 Asymptotic behavior of traveling wave solutions

In this section, we consider the asymptotic behavior of traveling wave solutions of (4.3) by using the idea of contracting rectangles.

Theorem 5.1

Assume that (2.3) holds. If \(\varPsi(\xi) = (\varphi_{1}(\xi),\varphi _{2}(\xi),\varphi_{3}(\xi))\) is a positive solution of (4.3) given in Theorem 4.6, then (4.4) is true.

Proof

According to Theorem 4.6, we have \(\lim _{\xi\to-\infty}\varPsi(\xi)=\mathbf{0}\). Now we verify

$$\lim_{\xi\to+\infty}\bigl(\varphi_{1}(\xi), \varphi_{2}(\xi),\varphi_{3}(\xi )\bigr)=(1-k_{1},k_{2},k_{3}), $$

which is equivalent to

$$ \lim _{\xi\to+\infty}\bigl(\phi_{1}(\xi),\phi_{2}(\xi), \phi_{3}(\xi)\bigr)=(k_{1},k_{2},k_{3}). $$
(5.1)

By Theorem 4.6, \(u_{1}(x,t)=\phi_{1}(\xi)\) satisfies

$$ \textstyle\begin{cases} \frac{\partial u_{1}(x,t)}{\partial t} = d_{1}\Delta u_{1}(x,t)+r_{1}u_{1}(x,t)[1-u_{1}(x,t)-a_{12}u_{2}(x,t)-a_{13}u_{3}(x,t)],\\ u_{1}(x,0)= \phi_{1}(x)>0, \end{cases} $$

and so

$$ \textstyle\begin{cases} \frac{\partial u_{1}(x,t)}{\partial t} \ge d_{1}\Delta u_{1}(x,t)+r_{1} u_{1}(x,t)[\underline{u}_{1}-u_{1}(x,t)],\\ u_{1}(x,0)= \phi_{1}(x)>0 \end{cases} $$

for all \(x\in\mathbb{R},t>0\). Then Lemma 2.3 indicates that

$$\liminf_{t\to\infty}u_{1}(0,t)\ge\underline{u}_{1}>0, $$

which implies that

$$\liminf_{\xi\to\infty}\phi_{1}(\xi) \ge\underline{u}_{1} >0. $$

Similarly, we can verify that

$$\liminf _{\xi\to+\infty}\phi_{2}(\xi)\geq\underline{u}_{2}>0 , \qquad \limsup _{\xi\to+\infty}\phi_{3}(\xi)\le a_{32}+a_{31}-1. $$

We now verify that

$$\limsup_{\xi\to\infty} \phi_{1}(\xi) < 1. $$

By what we have done, we only need to show that \(\limsup_{\xi\to\infty} \phi_{1}(\xi) =1\) is impossible. If \(\limsup_{\xi\to\infty} \phi_{1}(\xi) =1\), then there exists \(\{\xi_{m}\}_{m=1}^{\infty}\) such that

$$\xi_{m} \to\infty, \quad\phi_{1}(\xi_{m}) \to1, \quad m\to\infty, $$

and Remark 4.7 indicates that

$$\lim_{m\to\infty}\bigl[d_{1}\phi''_{1}( \xi_{m})-c\phi'_{1}(\xi_{m})\bigr] \le0. $$

Moreover, if m is large, then \(2\phi_{2}(\xi_{m}) \ge\underline{u}_{2}\) and

$$r_{1}\phi_{1}(\xi_{m})\bigl[1- \phi_{1}(\xi_{m})-a_{12}\phi_{2}( \xi_{m})-a_{13}\phi_{3}(\xi _{m})\bigr] < 0 $$

and so

$$d_{1}\phi''_{1}( \xi_{m})-c\phi'_{1}(\xi_{m})+r_{1} \phi_{1}(\xi_{m})\bigl[1-\phi_{1}(\xi _{m})-a_{12}\phi_{2}(\xi_{m})-a_{13} \phi_{3}(\xi_{m})\bigr]< 0, $$

which indicates a contradiction.

By a similar discussion, we can prove that

$$a_{i}(0)< \liminf_{\xi\to\infty}\phi_{i}(\xi) \le \limsup_{\xi\to\infty }\phi_{i}(\xi) < b_{i} (0), \quad i=1,2,3, $$

where \(a_{i}(0),b_{i}(0)\) are defined by Lemma 3.1.

If (5.1) does not hold, then there exists some \(s_{0}\in(0,1)\) such that

$$ a_{i}(s_{0})\le \liminf_{\xi\to\infty} \phi_{i}(\xi) \le\limsup_{\xi\to \infty}\phi_{i}( \xi) \le b_{i} (s_{0}),\quad i=1,2,3, $$
(5.2)

and at least one equality holds.

If \(b_{1}(s_{0})= \limsup_{\xi\to\infty}\phi_{1}(\xi)\), then there exists \(\{\xi_{m}\}_{m=1}^{\infty}\) such that

$$\xi_{m} \to\infty,\quad\phi_{1}(\xi_{m}) \to b_{1}(s_{0}),\quad m\to\infty, $$

and

$$\limsup_{m\to\infty}\bigl[d_{1}\phi''_{1}( \xi_{m})-c\phi'_{1}(\xi_{m})\bigr] \le0. $$

Moreover, we have

$$\begin{aligned} & \limsup_{m\to\infty}\bigl\{ r_{1}\phi_{1}( \xi_{m})\bigl[1-\phi_{1}(\xi_{m})-a_{12} \phi _{2}(\xi_{m})-a_{13}\phi_{3}( \xi_{m})\bigr] \bigr\} \\ &\quad < -r_{1}b_{1}(s_{0}) (1-s_{0}) (a_{12}\underline{u}_{2}+a_{13} \underline{u}_{3})< 0 \end{aligned}$$

by Lemma 3.1, which indicates that

$$d_{1}\phi''_{1}( \xi_{m})-c\phi'_{1}(\xi_{m})+r_{1} \phi_{1}(\xi_{m})\bigl[1-\phi_{1}(\xi _{m})-a_{12}\phi_{2}(\xi_{m})-a_{13} \phi_{3}(\xi_{m})\bigr]< 0 $$

if m is large enough. Thus, \(b_{1}(s_{0})> \limsup_{\xi\to\infty}\phi _{1}(\xi)\).

If \(\limsup_{\xi\to\infty}\phi_{3}(\xi) = b_{3} (s_{0})\), then there exists \(\{\xi_{m}\}_{m=1}^{\infty}\) such that

$$\xi_{m} \to\infty,\quad\phi_{3}(\xi_{m}) \to b_{3}(s_{0}),\quad m\to\infty, $$

and

$$\limsup_{m\to\infty}\bigl[d_{1}\phi''_{1}( \xi_{m})-c\phi'_{1}(\xi_{m})\bigr] \le0. $$

Moreover, by Sect. 3 and \(b_{1}(s_{0})> \limsup_{\xi\to\infty}\phi _{1}(\xi)\), we have

$$\begin{aligned} & \limsup_{m\to\infty}\bigl\{ r_{3}\phi_{3}(\xi) \bigl[-1+a_{31}\phi_{1}(\xi)+a_{32}\phi _{2}(\xi)-\phi_{3}(\xi)\bigr] \bigr\} \\ &\quad < r_{3}b_{3}(s_{0})\bigl[-1+a_{31}b_{1}(s_{0})+a_{32}b_{2}(s_{0})-b_{3}(s_{0}) \bigr]=0, \end{aligned}$$

and so

$$d_{3}\phi''_{3}( \xi_{m})-c\phi'_{3}(\xi_{m})+r_{3} \phi_{3}(\xi_{m})\bigl[-1+a_{31}\phi_{1}( \xi _{m})+a_{32}\phi_{2}(\xi_{m})- \phi_{3}(\xi_{m})\bigr]< 0 $$

if m is large enough. Thus, \(\limsup_{\xi\to\infty}\phi_{3}(\xi) < b_{3} (s_{0})\). It should be noted that in Sect. 3, we cannot obtain a strict inequality for \(b_{3}(s)\), but we can obtain strict a inequality in the above inequality since \(b_{1}(s_{0})> \limsup_{\xi\to\infty}\phi _{1}(\xi)\).

By similar discussions, we find that every inequality in (5.2) is strict. A contradiction occurs, which implies (5.1). The proof is complete. □

Remark 5.2

Different from Lin and Ruan [31], we did not use a strictly contracting rectangle.

6 Minimal wave speed

In this section, we shall prove that (4.3)–(4.4) has no positive solution if \(c< c^{*}\), which implies that \(c^{*}\) is the minimal wave speed. The method is similar to that in Lin [29] and Lin and Ruan [31].

Theorem 6.1

If \(c< c^{*}\), then (4.3)–(4.4) has no positive solution.

Proof

If the statement is false, then there exists some \(c'< c^{*}\) such that (4.3)–(4.4) with \(c=c'\) has a positive solution \(\varPsi(\xi)=(\varphi_{1}(\xi),\varphi_{2}(\xi),\varphi_{3}(\xi))\) which satisfies (4.4). We now discuss two cases: \(c^{*}=2\sqrt{d_{2}r_{2}(1-a_{21})}\) and \(c^{*}=2\sqrt {d_{3}r_{3}(a_{31}-1)}\).

If \(c^{*}=2\sqrt{d_{2}r_{2}(1-a_{21})}\), then there exists \(\epsilon_{1} >0\) such that

$$2\sqrt{d_{2}r_{2}(1-a_{21}-2a_{23} \epsilon_{1})} >c'. $$

By the asymptotic boundary condition \(\lim _{\xi\to-\infty}\varphi _{3}(\xi)=0\) and the strict positivity of solution to (4.3), there exists \(\xi_{1}\in\mathbb{R}\) such that

$$a_{23} \varphi_{3}(\xi)< a_{23} \epsilon_{1},\quad\xi< \xi_{1}. $$

When \(\xi\ge\xi_{1}\), the positivity and limit behavior of \(\varphi_{2}(\xi)\) indicate that

$$\inf_{\xi\ge\xi_{1}}\varphi_{2}(\xi)>0. $$

Let

$$L=\frac{a_{23} (a_{31}+a_{32}-1)}{\inf_{\xi\ge\xi_{1}}\varphi_{2}(\xi)}, $$

then \(a_{23}\varphi_{3}(\xi) \le L \varphi_{2}(\xi) \) and

$$1-a_{21}+a_{21}\varphi_{1}(\xi)- \varphi_{2}(\xi)-a_{23}\varphi_{3}(\xi )>1-a_{21}-(L+1)\varphi_{2}(\xi) $$

for \(\xi\ge\xi_{1}\).

Therefore, we have

$$ \textstyle\begin{cases} \frac{\partial w_{2}(x,t)}{\partial t} \geq d_{1}\Delta w_{2}(x,t)+r_{2}w_{2}(x,t)(1-a_{21}-a_{23}\epsilon _{1}-(L+1)w_{2}(x,t)),\\ w_{2}(x,0)= \varphi_{2}(x) >0 \end{cases} $$

for \(x\in\mathbb{R}, t>0\).

Namely, \(\varphi_{2}(x+c't)\) is the upper solution of

$$ \textstyle\begin{cases} \frac{\partial w_{2}(x,t)}{\partial t} = d_{2}\Delta w_{2}(x,t)+r_{2}w_{2}(x,t)(1-a_{21}-a_{23}\epsilon _{1}-(L+1)w_{2}(x,t)),\\ w_{2}(x,0)= \varphi_{2}(x). \end{cases} $$

By the theory of asymptotic spreading (Lemma 2.3), we see that

$$\liminf_{t\to\infty}\inf_{|x|\le c_{1}t}w_{2}(x,t) \ge\frac {1-a_{21}-a_{23}\epsilon_{1}}{L+1}>0 $$

with \(c_{1}=2\sqrt{d_{2}r_{2}(1-a_{21}-2a_{23}\epsilon_{1})}\). Letting \(-x=c_{1}t\), one gets

$$\liminf_{t\to\infty}w_{2}(-c_{1}t,t)\ge \frac{1-a_{21}-a_{23}\epsilon_{1}}{L+1}>0. $$

At the same time, we have

$$\xi=x+c't=\bigl(c'-c_{1}\bigr)t\to-\infty, \quad t\to\infty $$

so that

$$\lim_{\xi\to-\infty}\varphi_{2} (\xi)=\lim _{t\to\infty}w_{2}(-c_{1}t,t)=0, $$

which gives a contradiction.

Similarly, we can obtain a contradiction if \(c^{*}=2\sqrt{d_{3}r_{3}(a_{31}-1)}\).

Thus, for any \(c< c^{*}\), (4.3)–(4.4) has no positive solution. The proof is complete. □

7 Conclusion and discussion

For a parabolic system, if there exists a constant \(c_{0}\) such that \(c\ge c_{0}\) (\(c> c_{0}\)) implies that the system has a desired traveling wave solution while \(c< c_{0}\) (\(c\le c_{0}\)) implies the nonexistence of desired traveling wave solution, then \(c_{0}\) is the so-called minimal wave speed. In population dynamics, the minimal wave speed is an important threshold [36].

In this paper, we obtain the existence of traveling wave solutions if \(c>c^{*}\) and nonexistence of traveling wave solutions if \(c< c^{*}\). In a weaker sense, we formulate the minimal wave speed. However, we cannot directly confirm the existence or nonexistence of traveling wave solutions with \(c=c^{*}\) by the method in this paper.

Besides the minimal wave speed, spreading speed is also an important threshold, which may equal to the minimal wave speed of traveling wave solutions. For monotone systems, some important results have been established, see Fang et al. [12], Liang and Zhao [27], Lui [33], Weinberger et al. [48]. For predator–prey systems of two species, Lin [29] and Pan [38] proved a similar result, also see Bianca et al. [2, 3]. However, for the predator–prey system with three species, the question of estimating the asymptotic spreading of each species remains open.