1 Introduction

In the past decades, neutral stochastic functional differential equations (NSFDEs) have been widely discussed by many researchers because of potential applications in control theory, mechanics, engineering, economics, etc. A lot of interesting properties of the solutions for the system such as existence, uniqueness and stability have been obtained (see, e.g., [3–10] and the references therein). There exist impulsive effects in many areas, such as physics, economics, mechanics and engineering, which are changed abruptly at certain moments of time. Therefore, NSFDEs with impulses have been examined. For more details, we refer the reader to [11–14] and the references therein. Moreover, the attracting and invariant sets are also interesting topics for the stochastic system (see, e.g., [15, 16]). In particular, the authors in [17] discussed the p-attracting and p-invariant sets for NSFDEs with impulses. Long et al. [2] investigated global attracting sets of NSFDEs with impulses by establishing impulsive integral inequalities. Since then Li and Xu [18] obtained the attracting and quasi-invariant sets of the mild solution of NSFDEs. Wang and Li [19] made further efforts on attracting and quasi-invariant sets of the mild solution of impulsive NSFDEs with infinite delays by establishing impulsive integral inequalities.

Due to the wide application of fractional Brownian motion (fBm) in hydrology, economics, telecommunications and medicine, much interesting work has been carried out on stochastic differential equations driven by fBm (see, e.g., [20–26] and the references therein). More precisely, Boufoussi and Hajji [27] obtained existence and uniqueness of the mild solution for NSFDEs driven by fBm in a Hilbert space and discussed the exponential stability in the mean square sense. Tien [28] established existence, uniqueness and asymptotic behavior of impulsive NSFDEs driven by fBm with finite and infinite delays. Recently, Arthi et al. [29] considered a class of neutral stochastic integro-differential equations with impulses driven by fBm. With Lipschitz conditions and semigroup properties, they proved the existence and uniqueness of the mild solution. Furthermore, they discussed the exponential stability under some sufficient conditions. Based on the new impulsive integral inequalities, Li [1] obtained the global attracting and quasi-invariant sets of impulsive NSFDEs driven by fBm with Hurst parameter \(H>\frac{1}{2}\).

In this paper, we consider the neutral stochastic integro-differential equations with impulses driven by fBm in [29] as the following form:

$$ \textstyle\begin{cases} \mathrm{d} [x(t) - g (t, x_{t}, \int_{0}^{t}a_{1} (t, s, x_{s} )\, \mathrm{d}s ) ] \\ \quad = [Ax(t) + f (t, x_{t}, \int_{0}^{t}a_{2} (t, s, x_{s} )\, \mathrm{d}s ) ]\, \mathrm{d}t \\ \qquad {}+ \tilde{F}(t)\, \mathrm{d}B_{Q}^{H}(t),\quad t\ge0, t \neq t_{k}, k = 1, 2, \ldots, \\ \bigtriangleup x(t_{k}) = I_{k} (x(t_{k}^{-}) ),\quad k = 1, 2, \ldots, \\ x_{0}(t) = \varphi\in\mathcal{P}\mathcal{C}_{\mathcal{F}_{0}}^{b},\quad -r\leq t \leq0, \end{cases} $$
(1.1)

where A is the infinitesimal generator of an analytic semigroup \(\{T(t)\}_{t\geq0}\) in Hilbert space \(\mathbb{H}\), \(B_{Q}^{H}\) is a fBm with Hurst parameter H, \(g, f: \mathbb{R}^{+} \times\mathcal {P}\mathcal{C}\times\mathbb{H} \rightarrow\mathbb{H}\), \(a_{1}, a_{2}: \mathbb{R}^{+} \times\mathbb{R}^{+} \times\mathcal{P}\mathcal {C}\rightarrow\mathbb{H}\), \(\tilde{F}: \mathbb{R}^{+}\rightarrow \mathcal{L}_{2}^{0}(\mathbb{K}, \mathbb{H})\). The impulsive moment \(t_{k}\) (\(k=1, 2,\ldots\)) satisfies \(0< t_{1}< t_{2}<\cdots<t_{k}<\cdots\) and \(\lim_{k\rightarrow+\infty} t_{k}=+\infty\), \(I_{k}: \mathbb{H} \rightarrow\mathbb{H}\), \(\bigtriangleup x(t) = x(t^{+}) - x(t^{-})\), where \(x(t^{+})\) and \(x(t^{-})\) denote the right and left limit of x at t, respectively. \(\mathcal{P}\mathcal{C}\) = \(\mathcal {P}\mathcal{C} ([-r, 0], \mathbb{H} )\) = {\(\phi: [-r, 0]\rightarrow\mathbb{H}\), \(\phi(t)\) is continuous everywhere except a finite number of points t̃ at which \(\phi({\tilde{t}}^{-})\), \(\phi(\tilde{t}^{+})\) exist and \(\phi(\tilde{t}^{-})=\phi(t)\)}. For \(\phi\in\mathcal{P}\mathcal{C}\), \(\|\phi\|_{\mathcal{P}\mathcal {C}} = \sup_{s\in[-r, 0]}\|\phi(s)\| < + \infty\). For any continuous function x and \(t\in[0, b]\), \(x_{t}(\theta) = x(t +\theta)\), \(-r\leq\theta\leq0\). \(\mathcal{P}\mathcal{C}_{\mathcal {F}_{t}}^{b}([-r,0], \mathbb{H})\) denotes the family of all \(\mathcal {F}_{t}\)-measurable, \(\mathcal{P}\mathcal{C} ([-r, 0], \mathbb{H} )\)-value random variables ϕ with the norm \(\|\phi\|^{p} = \sup_{s\in[-r, 0]}\mathbb{E}\|\phi(s)\|_{\mathbb{H}}^{p} < + \infty\) for \(p>0\).

To the best of our knowledge, there is no result on the attracting and quasi-invariant sets of the mild solution for system (1.1). To close the gap, we aim to derive the attracting and quasi-invariant sets for (1.1) by establishing two impulsive integral inequalities which improve the results in [1] and [2], respectively. Moreover, exponential stability of the mild solution is established with sufficient conditions.

The paper is organized as follows. In Section 2, some basic notions, preliminaries and assumptions are provided. Section 3 is devoted to studying the attracting and quasi-invariant sets for neutral stochastic integro-differential equations with impulses driven by fBm. By a product, the globally mean square exponential stability of the mild solution is derived.

2 Notations and preliminaries

In this section, we begin with some notations and preliminary results with respect to fBm. Let \((\Omega, \mathcal{F}, \{\mathcal{F}_{t}\}_{t \in J}, \mathbb{P} )\) be a complete probability space satisfying the usual conditions. \(\mathbb{E}(\cdot)\) denotes the mathematical expectation with respect to \(\mathbb{P}\). A one-dimensional fBm with Hurst parameter \(H\in(0, 1)\) is a centered Gaussian process \(\beta^{H} = \beta^{H}(t)\) with the covariance function

$$R(t, s) = \mathbb{E} \bigl[\beta^{H}(t)\beta^{H}(s) \bigr] = \frac {1}{2} \bigl(|t|^{2H} + |s|^{2H} - |t-s|^{2H} \bigr). $$

In this paper, we consider \(H > \frac{1}{2}\) and \(\beta^{H}(t)\) has the following representation:

$$\beta^{H}(t) = \int_{0}^{t}K(t, s)\, \mathrm{d}\beta(s), $$

where \(\beta(s)\) is a standard Brownian motion and the kernel \(K(t, s)\) is given by

$$K(t, s) = c_{H}s^{\frac{1}{2}-H} \int_{s}^{t}(u-s)^{H-\frac{3}{2}}u^{H-\frac{1}{2}}\, \mathrm{d}u, \quad t \geq s, $$

where \(c_{H}\) is a nonnegative constant with respect to H.

For the deterministic function \(\varphi\in L^{2} ([0, b] )\), the fractional Wiener integral of φ with respect to \(\beta^{H}\) is defined by

$$\int_{0}^{b}\varphi(s)\, \mathrm{d} \beta^{H}(s) = \int_{0}^{b}K^{*}_{H}\varphi(s)\, \mathrm{d} \beta(s), $$

where \(K^{*}_{H}\varphi(s) = \int_{s}^{t}\varphi(t)\frac{\partial K(t, s)}{\partial t}\, \mathrm{d}s\).

Let \((\mathbb{K}, \|\cdot\|_{\mathbb{K}} )\) and \((\mathbb{H}, \|\cdot\|_{\mathbb{H}} )\) be two real separable Hilbert spaces with their vector norms and inner products, respectively. \(\mathcal {L} (\mathbb{H}, \mathbb{K} )\) is the set of all linear bounded operators from \(\mathbb{H}\) to \(\mathbb{K}\) equipped with the norm \(\|\cdot\|\). For the sake of convenience, we use the same notation \(\|\cdot\|\) to denote the norms in \(\mathbb{K}\), \(\mathbb{H}\), \(\mathcal{L}(\mathbb{H}, \mathbb{K})\). \(e_{n}\) (\(n=1, 2, \ldots\)) denotes a complete orthonormal basis in \(\mathbb{H}\) and \(Q\in\mathcal{L} (\mathbb{H},\mathbb{H} )\) is an operator defined by \(Qe_{n} = \lambda_{n}e_{n}\) with finite trace \(\operatorname{tr}Q = \sum_{n=1}^{+\infty}\lambda_{n} < +\infty\), where \(\lambda_{n}\) (\(n=1, 2, \ldots\)) is a nonnegative real number. We define the infinite-dimensional fBm on \(\mathbb{H}\) with covariance Q as

$$B^{H}_{Q}(t)= \sum_{n=1}^{\infty} \beta^{H}_{n}(t)Q^{\frac{1}{2}}e_{n} = \sum _{n=1}^{\infty}\sqrt{\lambda_{n}}e_{n} \beta^{H}_{n}(t), $$

where \(\beta^{H}_{n}(t)\) is real, independent fBm. The process is an \(\mathbb{H}\)-valued Q-fBm, starts from 0, has zero mean and covariance

$$\mathbb{E}\bigl\langle B^{H}_{Q}(t), x\bigr\rangle \bigl\langle B^{H}_{Q}(t), y\bigr\rangle = R(t, s)\bigl\langle Q(x), y\bigr\rangle , \quad \text{for all }x, y\in\mathbb{K}, \text{and }t, s \ge0. $$

In the following parts, we introduce the Wiener integral with respect to the Q-fBm. Let \(\mathcal{L}_{2}^{0} = \mathcal {L}_{2}^{0}(\mathbb{H}, \mathbb{K})\) denote the space of all Q-Hilbert-Schmidt operators \(\psi: \mathbb{H}\rightarrow\mathbb{K}\) equipped with the norm

$$\Vert \psi \Vert _{\mathcal {L}_{2}^{0}} = \sum_{n=1}^{\infty} \Vert \sqrt{\lambda_{n}}\psi e_{n} \Vert ^{2}< \infty $$

and the inner product \(\langle\varphi, \psi\rangle_{\mathcal{L}_{2}^{0}} = \sum_{n=1}^{\infty}\langle\varphi e_{n}, \psi e_{n}\rangle\) for \(\varphi, \psi\in\mathcal{L}_{2}^{0}\).

Now, we give the definition of the fractional Wiener integral of the function \(\psi: [0, b]\rightarrow \mathcal{L}_{2}^{0}\) with respect to Q-fBm as follows:

$$\int_{0}^{t}\psi(s)\, \mathrm{d}B^{H}_{Q}(s) = \sum_{n=1}^{\infty}\int_{0}^{t}\psi (s)Q^{\frac{1}{2}}e_{n}\, \mathrm{d}\beta_{n}^{H}(s)=\sum_{n=1}^{\infty}\int _{0}^{t} \bigl(K^{*}_{H}\bigl(\psi Q^{\frac{1}{2}}e_{n}\bigr) \bigr) (s)\, \mathrm{d} \beta_{n}(s), $$

where \(\beta_{n}\) is the standard Brownian motion with respect to \(\beta_{n}^{H}\).

In what follows, we need the property of the stochastic integral from [30] to prove our main results.

Lemma 2.1

If \(\psi: [0, b]\rightarrow\mathcal {L}_{2}^{0}\) satisfies \(\int_{0}^{b} \|\psi(s) \|^{2}_{\mathcal {L}_{2}^{0}}\, \mathrm{d}s < \infty\), then the integral \(\int_{0}^{t}\psi(s)\, \mathrm{d}B^{H}_{Q}(s)\) is well defined as an \(\mathbb{H}\)-valued random variable and we have

$$\mathbb{E} \biggl\Vert \int_{0}^{t}\psi(s)\,\mathrm{d}B^{H}_{Q}(s) \biggr\Vert ^{2}\le c_{H}t^{2H-1} \int_{0}^{t} \bigl\Vert \psi(s) \bigr\Vert ^{2}_{\mathcal{L}_{2}^{0}}\,\mathrm{d}s. $$

We assume that \(T(t)\), \(t\geq0\) is a uniformly bounded and analytic semigroup. \(A: D(A)\rightarrow\mathbb{K}\) is the infinitesimal generator of \(T(t)\) on \(\mathbb{K}\). We also assume that there exist a constant \(M\geq1\) and \(\mu\in\mathbb{R}\) such that \(\|T(t)\| \leq Me^{\mu t}\), for every \(t\geq0\), \(0\in\sigma(A)\), where \(\sigma(A)\) is the resolvent set of A. Then it is possible to define the fractional power \((-A)^{\alpha}\) for \(0<\alpha\leq1\) as a closed linear operator on its domain \(D(-A)^{\alpha}\). Moreover, the subspace \(D(-A)^{\alpha}\) is dense in \(\mathbb{K}\) and the equality \(\|\rho\|_{\alpha}= \|(-A)^{\alpha}\rho\|\) defines a norm in \(D(-A)^{\alpha}\). Let \(\mathbb{K}_{\alpha}\) symbolize the space \(D(-A)^{\alpha}\) endowed with the norm \(\|\cdot\| _{\alpha}\).

Lemma 2.2

(see [31])

Assume that the above conditions hold. Then:

  1. (1)

    if \(0<\alpha\leq1\), then \(\mathbb{K}_{\alpha}\) is a Banach space,

  2. (2)

    if \(0<\beta\leq\alpha\), then the injection \(\mathbb{K}_{\alpha}\hookrightarrow\mathbb{K}_{\beta}\) is continuous,

  3. (3)

    there exists \(M_{\alpha}>0\), for any \(0<\alpha\leq1\), such that

    $$\bigl\Vert \bigl(-A^{\alpha}\bigr)T(t) \bigr\Vert \leq \frac{M_{\alpha}}{t_{\alpha}}e^{-\mu t},\quad t > 0, \mu> 0. $$

Throughout this paper, we assume that there exists at least one solution for (1.1), which is denoted by \(x(t)\) or \(x_{t}(0, \varphi)\).

Definition 2.3

The zero solution (or trivial solution) of (1.1) is said to be p-exponentially (\(p\geq2\)) stable if there exist positive constants λ and \(M > 1\), for any initial value \(\varphi\in\mathcal{P}\mathcal{C}_{\mathcal {F}_{0}}^{b}([-r, 0], \mathbb{H})\), such that

$$\mathbb{E} \bigl\Vert x(t) \bigr\Vert ^{p} \leq M \Vert \varphi \Vert ^{p}_{L^{p}}e^{-\lambda t},\quad t\geq0. $$

Definition 2.4

The set \(\mathcal{S}\subset\mathbb{H}\) is called a quasi-invariant set of (1.1) if there exist positive constants k and l such that, for any initial value \(\varphi\in\mathcal{P}\mathcal {C}_{\mathcal{F}_{0}}^{b} ([-r, 0], \mathbb{H} )\), the solution \(kx_{t}(0, \varphi) + l \in\mathcal{S}\), \(t \geq0\).

Definition 2.5

The set \(\mathcal{S}\subset\mathbb{H}\) is called a global attracting set of (1.1) if, for any initial value \(\varphi \in\mathcal{P}\mathcal{C}_{\mathcal{F}_{0}}^{b} ([-r, 0], \mathbb{H} )\), the solution \(x_{t}(0, \varphi)\) satisfies

$$\operatorname{dist} \bigl(x_{t}(0, \varphi), \mathcal{S} \bigr) \rightarrow0,\quad \text{as } t \rightarrow\infty, $$

where

$$\operatorname{dist}(\varphi, \mathcal{S}) = \inf_{\psi\in\mathcal{S}}\rho \bigl(\varphi(s), \psi(s) \bigr),\quad \mbox{for } \varphi\in\mathcal {P} \mathcal{C}_{\mathcal {F}_{0}}^{b} \bigl([-r, 0], \mathbb{H} \bigr) $$

and \(\rho(\cdot, \cdot)\) is any distance in \(\mathcal{P}\mathcal {C}_{\mathcal{F}_{0}}^{b} ([-r, 0], \mathbb{H} )\).

Definition 2.6

An \(\mathbb{H}\)-valued random process \(x(t)\), \(t \in[-r,\infty]\) is called a mild solution of (1.1) if:

  1. (i)

    \(x(t) \in\mathcal{P}\mathcal{C} ([-r, \infty], L^{2}(\Omega, \mathbb{H}) )\),

  2. (ii)

    for \(t\in[-r, 0]\), \(x(t) = \phi(t)\),

  3. (iii)

    for \(t\ge0\), \(x(t)\) satisfies the following integral equation:

    $$\begin{aligned} x(t) =& T(t) \bigl[\phi(0)- g(0, \phi, 0) \bigr] + g \biggl(t, x_{t}, \int_{0}^{t}a_{1} (t, s, x_{s} ) \,\mathrm{d}s \biggr) \\ &{} + \int_{0}^{t} AT(t-s) g \biggl(s, x_{s}, \int_{0}^{s}a_{1} (s, \tau, x_{\tau})\,\mathrm{d}\tau \biggr)\,\mathrm{d}s \\ &{} + \int_{0}^{t} T(t-s) f \biggl(s, x_{s}, \int_{0}^{s}a_{2} (s, \tau, x_{\tau})\,\mathrm{d}\tau \biggr)\,\mathrm{d}s \\ &{} + \sum_{0< t_{i}< t}T(t-t_{i})I_{i} \bigl(x\bigl(t_{i}^{-}\bigr) \bigr) + \int_{0}^{t} T(t-s)\tilde{F}(s)\, \mathrm{d}B_{Q}^{H}(s)\quad \mbox{a.s.} \end{aligned}$$
    (2.1)

Lemma 2.7

(see [32])

For any \(x\in\mathbb{R}_{+}^{n}\) and \(p>0\), we have

$$|x|^{p} = n^{(\frac{p}{2}-1)\vee0}\sum_{i=1}^{n}x_{i}^{p}, \qquad \Biggl(\sum_{i=1}^{n}x_{i} \Biggr)^{p} = n^{(\frac{p}{2}-1)\vee0}\sum_{i=1}^{n}x_{i}^{p}. $$

In order to obtain our main results, we always make the following assumptions:

  1. (H1)

    There exist positive constants M and μ such that the strongly continuous semigroup \(\|T(t)\|\leq Me^{-\mu t}\).

  2. (H2)

    There exist positive constants \(L_{f}\) and \(M_{f}\), for any \(\psi_{i}, \varphi_{i}\in\mathcal{P}\mathcal{C}\), \(i = 1, 2\), such that

    $$\bigl\Vert f(t,\psi_{1}, \varphi_{1}) - f(t, \psi_{2}, \varphi_{2}) \bigr\Vert \leq L_{f} \bigl[ \Vert \psi_{1} - \psi_{2} \Vert + \Vert \varphi_{1} - \varphi_{2} \Vert \bigr] $$

    and \(\|f(t, 0, 0) \| \leq M_{f}\).

  3. (H3)

    There exist constants \(\frac{1}{2}<\beta<1\) and positive constants \(L_{g}\), \(M_{g}\), for any \(\psi_{i}, \varphi_{i}\in\mathcal{P}\mathcal{C}\), \(i = 1, 2\), such that

    $$\bigl\Vert (-A)^{\beta}\bigl[g(t,\psi_{1}, \varphi_{1}) - g(t, \psi_{2}, \varphi_{2})\bigr] \bigr\Vert \leq L_{g}\bigl[ \Vert \psi_{1} - \psi_{2} \Vert + \Vert \varphi_{1} - \varphi_{2} \Vert \bigr] $$

    and \(\|(-A)^{\beta}g(t, 0, 0) \| \leq M_{g}\).

  4. (H4)

    There exist positive constants \(L_{1}\) and \(L_{2}\), for any \(\psi\in\mathcal{P}\mathcal {C}\), where

    $$\biggl\Vert \int_{0}^{t}a_{1}(t, s, \psi)\,\mathrm{d}s \biggr\Vert \leq L_{1} \Vert \psi \Vert ,\qquad \biggl\Vert \int_{0}^{t}a_{2}(t, s, \psi)\,\mathrm{d}s \biggr\Vert \leq L_{2} \Vert \psi \Vert . $$
  5. (H5)

    The functions \(I_{j}\) (\(j = 1, 2, \ldots\)) satisfy the following condition: there exist \(d_{j}> 0\) (\(j = 1, 2, \ldots\)), for any \(\psi, \varphi\in\mathcal{P}\mathcal{C}\), such that

    $$\bigl\Vert I_{j}(\psi) - I_{j}(\varphi) \bigr\Vert \leq d_{j} \Vert \psi- \varphi \Vert $$

    and \(\|I_{j}(0) \| = 0\).

  6. (H6)

    The function \(\tilde{F}: [0, +\infty)\rightarrow \mathcal{L}_{Q}^{0}(\mathbb{H}, \mathbb{K})\) satisfies

    $$\int_{0}^{+\infty}e^{\mu s} \bigl\Vert \tilde{F}(s) \bigr\Vert ^{p}_{\mathcal {L}_{Q}^{0}}\,\mathrm{d}s < +\infty. $$
  7. (H7)

    There exist positive constants δ, \(\Lambda_{\lambda}\) and \(\delta^{*}\) satisfying the following inequalities:

    $$\begin{aligned}& \delta \geq \max \bigl\{ \Vert \phi \Vert ^{p}, \delta^{*} \bigr\} , \\& \Lambda_{\lambda}= 12^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}\bigl[L_{g}(1+L_{1}) \bigr]^{p}e^{\lambda r} \\& \hphantom{\Lambda_{\lambda}={}}{}+ \bigl\{ 12^{p-1}M_{1-\beta}^{p} \mu^{p-p\beta-\frac{p}{q}}\bigl[\Gamma (1+q\beta-q)\bigr]^{\frac{p}{q}} \bigl[L_{g}(1+L_{2})\bigr]^{p} \\& \hphantom{\Lambda_{\lambda}={}}{}+12^{p-1}M^{p}\bigl[L_{f}(1+L_{2}) \bigr]^{p}\mu^{1-p}\bigr\} \frac{ e^{\lambda r}}{\mu- \lambda} + \sum _{j=1}^{\infty}b_{j} < 1, \\& \delta^{*} = \biggl\{ 12^{p-1}M^{p} \bigl[ \Vert \phi \Vert ^{p} + 2^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p} \bigl(L_{g}^{p} \Vert \phi \Vert ^{p} + M_{g}^{p} \bigr) \bigr] \\& \hphantom{\delta^{*} ={}}{}+ C_{p} \biggl[M^{2}c_{H}H(2H-1) \sup_{t\ge0}\bigl\{ t^{2H-1}e^{-\epsilon t}\bigr\} \int_{0}^{+\infty} e^{\mu s} \bigl\Vert \tilde{F}(s) \bigr\Vert ^{2}_{\mathcal{L}_{Q}^{0}}\,\mathrm{d}s \biggr]^{p/2}\biggr\} \\& \hphantom{\delta^{*} ={}}{}\times(1 - \Lambda_{\lambda})^{-1}, \end{aligned}$$

    where \(\lambda\in(0, \mu)\), \(p\geq2\), \(q >1\), \(\frac{1}{p} + \frac{1}{q} =1\) and \(\epsilon>0\) satisfies \((\mu-\epsilon)p\ge2\mu\).

Remark 2.8

It is easy to derive the notion that system (1.1) has a unique mild solution under the above assumptions (H1)-(H6).

3 Main results

In this section, we propose some integral inequalities which are useful in our following calculus.

Lemma 3.1

For any positive constants \(\mu_{2}\) and \(\mu_{3}\), we assume that there exist some positive constants η, \(\eta_{i}\) (\(i = 2, 3\)) and \(b_{j}\) (\(j = 1, 2, \ldots\)) and the function \(x(t)\) is in \(\mathcal {P}\mathcal{C} ([-r, \infty), \mathbb{R}^{+} )\) such that

$$ x(t)\leq\eta_{2} \Vert x_{t} \Vert + \eta_{3} \int_{0}^{t}e^{-\mu_{2}(t-s)} \Vert x_{s} \Vert \,\mathrm{d}s + \sum_{0 < t_{j} < t }b_{j}e^{-\mu_{3}(t - t_{j})} \bigl\Vert x\bigl(t_{j}^{-}\bigr) \bigr\Vert + \eta,\quad t \ge0. $$
(3.1)

If

$$ \sigma: = \eta_{2} + \frac{\eta_{3}}{\mu_{2}} + \sum _{j=1}^{\infty}b_{j} < 1, $$
(3.2)

then

$$ x(t) \leq(1 - \sigma)^{-1}\eta,\quad t \in[-r, \infty), $$
(3.3)

provided

$$ x(t) \leq(1 - \sigma)^{-1}\eta,\quad t \in[-r, 0]. $$
(3.4)

Proof

If (3.3) is not true, we can affirm that there exists a \(t_{1} \ge0\) such that

$$ x(t_{1}) \geq(1 - \sigma)^{-1}\eta, \qquad x(t) < (1 - \sigma)^{-1}\eta,\quad t\in[-r, t_{1}). $$
(3.5)

From (3.1) and (3.5), we have

$$\begin{aligned} x(t_{1}) \le& \eta_{2}(1 - \sigma)^{-1}\eta+ \eta_{3} \int_{0}^{t_{1}}e^{-\mu_{2}(t_{1}-s)}(1 - \sigma)^{-1}\eta\,\mathrm{d}s \\ &{} + \sum_{0 < t_{j} < t_{1} }b_{j}e^{-\mu_{3}(t - t_{j})}(1 - \sigma)^{-1}\eta+ \eta \\ \le& \Biggl[\eta_{2} + \frac{\eta_{3}}{\mu_{2}} + \sum _{j=1}^{\infty}b_{j} \Biggr](1 - \sigma)^{-1}\eta+ \eta \\ = & (1 - \sigma)^{-1}\eta. \end{aligned}$$

Hence, it contradicts the first inequality of (3.5), so the proof is complete. □

Remark 3.2

If \(\mu_{1}=\mu_{2}=\mu\), then Lemma 3.1 becomes Lemma 3.2 in [1].

Lemma 3.3

For any positive constants \(\mu_{1}\), \(\mu_{2}\) and \(\mu_{3} \), we assume that there exist some positive constants η, \(\eta_{i}\) (\(i = 1, 2, 3\)) and \(b_{j}\) (\(j = 1, 2, \ldots\)) and the function \(x(t)\) is in \(\mathcal{P}\mathcal{C} ([-r, \infty), \mathbb{R}^{+} )\) such that

$$ x(t)\leq \textstyle\begin{cases} \eta_{1}e^{-\mu_{1}t} + \eta_{2}\|x_{t}\| + \eta_{3}\int_{0}^{t}e^{-\mu_{2}(t-s)}\|x_{s}\|\,\mathrm{d}s \\ \quad {}+ \sum_{0 < t_{j} < t }b_{j}e^{-\mu_{3}(t - t_{j})} \|x(t_{j}^{-}) \| + \eta,& t \ge0, \\ \phi(t), &t \in[-r, 0]. \end{cases} $$
(3.6)

If

$$ \sigma: = \eta_{2} + \frac{\eta_{3}}{\mu_{2}} + \sum _{j=1}^{\infty} b_{j} < 1, $$
(3.7)

then we have

$$ x(t) \leq\delta e^{-\lambda t} + (1 - \sigma)^{-1} \eta,\quad t \in[-r, \infty), $$
(3.8)

where \(\lambda\in(0, \lambda^{*})\), \(\lambda^{*} = \min\{\mu_{1}, \mu_{2}\}\) and the following inequalities are satisfied:

$$ \delta\geq\max \biggl\{ \|\phi \|, \frac{\eta_{1}}{1 - \sigma_{\lambda}} \biggr\} , \quad \sigma_{\lambda}=\eta_{2}e^{\lambda r} + \frac{\eta_{3} e^{\lambda r}}{\mu_{2} - \lambda} + \sum_{j=1}^{\infty}b_{j} < 1. $$
(3.9)

Proof

By (3.7), we verify that there exist constants \(\lambda\in(0, \lambda^{*})\) and δ such that (3.9) can be defined well. Firstly, we claim that, for any \(\tilde{\delta} \geq\delta\),

$$ x(t) \le\tilde{\delta}e^{-\lambda t} + (1-\sigma)^{-1} \eta= y(t),\quad \text{for } t\in[-r, b]. $$
(3.10)

It is clear that (3.10) holds for any \(t\in[-r, 0]\). By the contradiction method, if (3.10) is not true for \(t \in [0, b]\), we can find a \(\tilde{t}\in[0, b]\) such that

$$ x(\tilde{t}) \geq y(\tilde{t}),\qquad x(t) < y(t), \quad \text{for } t\in[-r, \tilde{t}). $$
(3.11)

Next, we will give some contradictions by these conditions.

By (3.6), (3.10) and (3.11), we have

$$\begin{aligned} x(\tilde{t}) \le& \eta_{1}e^{-\mu_{1}\tilde{t}} + \eta_{2} \Vert x_{\tilde{t}} \Vert + \eta_{3} \int_{0}^{\tilde{t}}e^{-\mu_{2}(\tilde{t}-s)} \biggl[\tilde { \delta}e^{-\lambda(s +\tau)} + \frac{\eta}{1-\sigma} \biggr]\,\mathrm{d}s \\ &{}+ \sum_{0 < t_{j} < \tilde{t}}b_{j}e^{-\mu_{1}(\tilde{t} - t_{j})} \bigl\Vert x\bigl(t_{j}^{-}\bigr) \bigr\Vert + \eta \\ \le& \eta_{1}e^{-\mu_{1}\tilde{t}} + \Biggl[\eta_{2}e^{\lambda r} + \frac{\eta_{3}e^{\lambda r}}{\mu_{2}-\lambda} + \sum_{j = 1 }^{\infty}b_{j} \Biggr]\tilde{\delta}e^{-\lambda\tilde{t}} \\ &{}+ \Biggl[\eta_{2} + \frac{\eta_{3}}{\mu_{2}} + \sum _{j = 1 }^{\infty}b_{j} \Biggr]\frac{\eta}{1 - \sigma} + \eta. \end{aligned}$$

Following from (3.7) and (3.9), we have

$$\begin{aligned} x(\tilde{t}) \le& \eta_{1}e^{-\lambda\tilde{t}} + \sigma_{\lambda}\tilde{\delta } e^{-\lambda\tilde{t}} + \sigma\frac{\eta}{1 - \sigma} + \eta \\ \le& \tilde{\delta}e^{-\lambda\tilde{t}} + \frac{\eta}{1 - \sigma}, \end{aligned}$$

which contradicts the first inequality of (3.11), so the proof is complete. □

Remark 3.4

If \(\mu_{1}=\mu_{2}=\mu_{3}=\mu\), we see that Lemma 3.1 in [2] is a special case of Lemma 3.3.

Corollary 3.5

If the assumptions of Lemma  3.3 hold and \(\eta= 0\), then all solutions of inequality (3.6) are converged to zero in the p-exponential sense.

Theorem 3.6

Assume that the conditions (H1)-(H7) are satisfied. Then

$$\mathcal{S}_{1} = \bigl\{ \varphi\in\mathcal {P}\mathcal{C}_{\mathcal{F}_{0}}^{b} \bigl([-r, 0], \mathbb{H}\bigr)\mid \Vert \varphi \Vert ^{p}_{L^{p}} \leq(1 - \Lambda)^{-1}J \bigr\} $$

is a global attracting set of the mild solution of (1.1) and

$$\mathcal{S}_{2} = \bigl\{ \varphi\in\mathcal{P}\mathcal{C}_{\mathcal {F}_{0}}^{b} \bigl([-r, 0], \mathbb{H}\bigr)\mid \Vert \varphi \Vert ^{p}_{L^{p}} \leq\varpi \bigr\} $$

is a quasi-invariant set of the mild solution of (1.1) if the following relations hold:

$$\begin{aligned} \begin{aligned}[b] \Lambda ={} & 12^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}\bigl[L_{g}(1+L_{1}) \bigr]^{p} \\ &{}+ 12^{p-1}M_{1-\beta}^{p} \mu^{p-p\beta-\frac{p}{q}-1}\bigl(\Gamma (1+q\beta-q)\bigr)^{\frac{p}{q}} \bigl[L_{g}(1+L_{2})\bigr]^{p} \\ &{}+ 12^{p-1}M^{p}\bigl[L_{f}(1+L_{2}) \bigr]^{p}\mu^{-p} +6^{p-1}M^{p} \Biggl( \sum_{j=1}^{\infty}d_{j} \Biggr)^{p} < 1, \end{aligned} \end{aligned}$$
(3.12)

where

$$\begin{aligned} \begin{aligned} &J = 12^{p-1}M^{p} \bigl[ \mathbb{E} \bigl\Vert \phi(0) \bigr\Vert ^{p} + 2^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}\bigl(L_{g}^{p} \mathbb{E} \Vert \phi \Vert ^{p} + M_{g}^{p} \bigr) \bigr] \\ &\hphantom{J ={}}{}+6^{p-1}C_{p} \biggl[M^{2}c_{H}H(2H-1) \sup_{t\ge0}\bigl\{ t^{2H-1}e^{-\epsilon t}\bigr\} \int_{0}^{+\infty} e^{\mu s} \bigl\Vert \tilde{F}(s) \bigr\Vert ^{2}_{\mathcal{L}_{Q}^{0}}\,\mathrm{d}s \biggr]^{p/2} \\ &\hphantom{J ={}} {} + 12^{p-1}M_{1-\beta}^{p}M_{g}^{p} \mu^{p-p\beta-\frac{p}{q}-1}\bigl(\Gamma (1+q\beta-q)\bigr)^{\frac{p}{q}} \\ &\hphantom{J ={}} {}+ 12^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}M_{g}^{p}+ 12^{p-1}M^{p}M_{f}^{p} \mu^{-p}, \\ &\varpi = 12^{p-1}M^{p} \bigl[ \Vert \phi \Vert ^{p} + 2^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}\bigl(L_{g}^{p} \Vert \phi \Vert ^{p} + M_{g}^{p}\bigr) \bigr] \\ &\hphantom{\varpi = {}}{}+ 6^{p-1}C_{p} \biggl[M^{2}c_{H}H(2H-1) \sup_{t\ge0}\bigl\{ t^{2H-1}e^{-\epsilon t}\bigr\} \int_{0}^{+\infty} e^{\mu s} \bigl\Vert \tilde{F}(s) \bigr\Vert ^{2}_{\mathcal{L}_{Q}^{0}}\,\mathrm{d}s \biggr]^{p/2} \\ &\hphantom{\varpi = {}}{}+ 12^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}M_{g}^{p}+ 12^{p-1}M^{p}M_{f}^{p} \mu^{-p} \\ &\hphantom{\varpi = {}}{}+12^{p-1}M_{1-\beta}^{p}M_{g}^{p} \mu^{p-p\beta-\frac{p}{q}-1} \bigl[\Gamma (1+q\beta-q) \bigr]^{\frac{p}{q}} , \end{aligned} \end{aligned}$$
(3.13)

and \(\frac{1}{p} + \frac{1}{q} = 1\), \(p\geq2\), \(q>1\).

Proof

By (2.1), we have

$$\begin{aligned} \mathbb{E} \bigl\Vert x(t) \bigr\Vert ^{p} =& \mathbb{E} \biggl\Vert T(t)\bigl[\phi(0)- g (0, \phi, 0 )\bigr] + g \biggl(t, x_{t}, \int_{0}^{t}a_{1} (t, s, x_{s}) \,\mathrm{d}s \biggr) \\ &{} + \int_{0}^{t} AT(t-s) g \biggl(s, x_{s}, \int_{0}^{s}a_{1}(s, \tau, x_{\tau})\,\mathrm{d}\tau \biggr)\,\mathrm{d}s \\ &{}+ \int_{0}^{t} T(t-s) f \biggl(s, x_{s}, \int_{0}^{s}a_{1}(s, \tau, x_{\tau})\,\mathrm{d}\tau \biggr)\,\mathrm{d}s \\ &{} + \sum_{0< t_{i}< t}T(t-t_{i})I_{i} \bigl(x\bigl(t_{i}^{-}\bigr)\bigr) + \int_{0}^{t} T(t-s) \tilde{F}(s)\, \mathrm{d}B_{Q}^{H}(s) \biggr\Vert ^{p}. \end{aligned}$$
(3.14)

From Lemma 2.7, we have

$$\begin{aligned} \mathbb{E} \bigl\Vert x(t) \bigr\Vert ^{p} \leq& 6^{p-1}\mathbb{E} \bigl\Vert T(t)\bigl[\phi(0)- g(0, \phi, 0)\bigr] \bigr\Vert ^{p} + 6^{p-1}\mathbb{E} \biggl\Vert g\biggl(t, x_{t}, \int_{0}^{t}a_{1}(t, s, x_{s})\, \mathrm{d}s\biggr) \biggr\Vert ^{p} \\ &{} + 6^{p-1}\mathbb{E} \biggl\Vert \int_{0}^{t} AT(t-s) g \biggl(s, x_{s}, \int_{0}^{s}a_{1}(s, \tau, x_{\tau})\,\mathrm{d}\tau \biggr)\,\mathrm{d}s \biggr\Vert ^{p} \\ &{}+ 6^{p-1}\mathbb{E} \biggl\Vert \int_{0}^{t} T(t-s) f \biggl(s, x_{s}, \int_{0}^{s}a_{1}(s, \tau, x_{\tau})\,\mathrm{d}\tau \biggr)\,\mathrm{d}s \biggr\Vert ^{p} \\ &{} + 6^{p-1}\mathbb{E} \biggl\Vert \sum _{0< t_{i}< t}T(t-t_{i})I_{i}\bigl(x \bigl(t_{i}^{-}\bigr) \bigr) \biggr\Vert ^{p} + 6^{p-1}\mathbb{E} \biggl\Vert \int_{0}^{t} T(t-s) \tilde{F}(s)\, \mathrm{d}B_{Q}^{H}(s) \biggr\Vert ^{p} \\ : = & 6^{p-1} \sum_{i=1}^{6}G_{i}(t). \end{aligned}$$
(3.15)

From (H1) and (H3), we have

$$\begin{aligned} G_{1}(t) = & \mathbb{E} \bigl\Vert T(t)\bigl[ \phi(0)- g(0, \phi, 0)\bigr] \bigr\Vert ^{p} \\ \leq&2^{p-1}M^{p}e^{-p\mu t} \bigl\{ \mathbb{E} \bigl\Vert \phi(0) \bigr\Vert ^{p} + \mathbb{E} \bigl\Vert (-A)^{-\beta}(-A)^{\beta}g(0, \phi, 0) \bigr\Vert ^{p} \bigr\} \\ \leq& 2^{p-1}M^{p}e^{-\mu t} \bigl\{ \mathbb{E} \bigl\Vert \phi(0) \bigr\Vert ^{p} + \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}\mathbb{E} \bigl\Vert (-A)^{\beta}g(0, \phi, 0) \bigr\Vert ^{p} \bigr\} \\ \leq& 2^{p-1}M^{p}e^{-\mu t} \bigl\{ \mathbb{E} \bigl\Vert \phi(0) \bigr\Vert ^{p} + \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}\mathbb{E} \bigl(L_{g} \Vert \phi \Vert + M_{g} \bigr)^{p} \bigr\} \\ \leq& 2^{p-1}M^{p}e^{-\mu t} \bigl\{ \mathbb{E} \bigl\Vert \phi(0) \bigr\Vert ^{p} + 2^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p} \bigl(L_{g}^{p} \mathbb{E} \Vert \phi \Vert ^{p} + M_{g}^{p} \bigr) \bigr\} \\ = & E_{1}e^{-\mu t}, \end{aligned}$$
(3.16)

where \(E_{1}: = 2^{p-1}M^{p} \{\mathbb{E} \|\phi(0) \|^{p} + 2^{p-1} \|(-A)^{-\beta} \|^{p} (L_{g}^{p}\mathbb{E} \|\phi \|^{p} + M_{g}^{p} ) \}\).

From (H3) and (H4), we derive

$$\begin{aligned} G_{2}(t) = &\mathbb{E} \biggl\Vert g\biggl(t, x_{t}, \int_{0}^{t}a_{1}(t, s, x_{s})\, \mathrm{d}s\biggr) \biggr\Vert ^{p} = \mathbb{E} \biggl\Vert (-A)^{-\beta}(-A)^{\beta}g\biggl(t, x_{t}, \int_{0}^{t}a_{1}(t, s, x_{s})\, \mathrm{d}s\biggr) \biggr\Vert ^{p} \\ \leq& \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}\mathbb{E} \bigl[L_{g}\bigl( \Vert x _{s} \Vert + L_{1} \Vert x _{s} \Vert \bigr)+ M_{g} \bigr]^{p} \\ \leq &2^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}\bigl[L_{g}(1+L_{1})\bigr]^{p} \mathbb {E} \Vert x _{s} \Vert ^{p} + 2^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}M_{g}^{p}. \end{aligned}$$
(3.17)

By (H1), (H3) and the Hölder inequality, we have

$$\begin{aligned} G_{3}(t) = &\mathbb{E} \biggl\Vert \int_{0}^{t} AT(t-s) g \biggl(s, x_{s}, \int_{0}^{s}a_{1}(s, \tau, x_{\tau})\,\mathrm{d}\tau \biggr)\,\mathrm{d}s \biggr\Vert ^{p} \\ = &\mathbb{E} \biggl\Vert \int_{0}^{t} (-A)^{1-\beta}T(t-s) (-A)^{\beta}g \biggl(s, x_{s}, \int_{0}^{s}a_{1}(s, \tau, x_{\tau})\,\mathrm{d}\tau \biggr)\,\mathrm{d}s \biggr\Vert ^{p} \\ \leq&\mathbb{E} \biggl[ \int_{0}^{t} \biggl\Vert (-A)^{1-\beta}T(t-s) (-A)^{\beta}g \biggl(s, x_{s}, \int_{0}^{s}a_{1}(s, \tau, x_{\tau})\,\mathrm{d}\tau \biggr) \biggr\Vert \,\mathrm{d}s \biggr]^{p} \\ \leq& \mathbb{E} \biggl[ \int_{0}^{t} \frac{M_{1-\beta}}{(t-s)^{1-\beta}}e^{-\mu(t-s)} \bigl[L_{g}(1+L_{2}) \Vert x_{s} \Vert + M_{g} \bigr] \,\mathrm{d}s \biggr]^{p} \\ \leq& M_{1-\beta}^{p} \biggl[ \int_{0}^{t}\bigl[(t-s)^{\beta-1}e^{-\mu (t-s)} \bigr]^{q}\,\mathrm{d}s \biggr]^{\frac{p}{q}}\mathbb{E} \int_{0}^{t}e^{-\mu (t-s)} \bigl[L_{g}(1+L_{2}) \Vert x_{s} \Vert + M_{g} \bigr]^{p}\, \mathrm{d}s \\ \leq& 2^{p-1}M_{1-\beta}^{p} \bigl[\mu^{q-q\beta-1} \Gamma(1+q\beta -q) \bigr]^{\frac{p}{q}} \int_{0}^{t} \bigl[e^{-\mu(t-s)} \bigl[L_{g}(1+L_{2})\bigr]^{p}\mathbb{E} \Vert x_{s} \Vert ^{p}+M_{g}^{p} \bigr]\, \mathrm{d}s \\ \leq& 2^{p-1}M_{1-\beta}^{p}\mu^{p-p\beta-\frac{p}{q}} \bigl[ \Gamma (1+q\beta-q) \bigr]^{\frac{p}{q}}\bigl[L_{g}(1+L_{2}) \bigr]^{p} \int _{0}^{t}e^{-\mu(t-s)}\mathbb{E} \Vert x_{s} \Vert ^{p}\,\mathrm{d}s \\ &{} + 2^{p-1}M_{1-\beta}^{p}M_{g}^{p} \mu^{p-p\beta-\frac{p}{q}-1}\bigl[\Gamma (1+q\beta-q)\bigr]^{\frac{p}{q}}, \end{aligned}$$
(3.18)

where \(\frac{1}{p} + \frac{1}{q} = 1\), \(p\geq2\), \(q>1\).

By (H1), (H2) and (H4), we obtain

$$\begin{aligned} G_{4}(t) = &\mathbb{E} \biggl\Vert \int_{0}^{t} T(t-s) f\biggl(s, x_{s}, \int_{0}^{s}a_{2}(s, \tau, x_{\tau})\,\mathrm{d}\tau\biggr)\,\mathrm{d}s \biggr\Vert ^{p} \\ \leq& M^{p}\mathbb{E} \biggl\{ \int_{0}^{t}e^{-\mu(t-s)} \bigl[L_{f}(1+L_{2}) \Vert x_{s} \Vert + M_{f} \bigr]\,\mathrm{d}s \biggr\} ^{p} \\ \leq& 2^{p-1}M^{p}\bigl[L_{f}(1+L_{2}) \bigr]^{p}\mathbb{E} \biggl[ \int _{0}^{t}e^{-\mu(t-s)} \Vert x_{s} \Vert \,\mathrm{d}s \biggr]^{p}+ 2^{p-1}M^{p} \biggl[ \int_{0}^{t}e^{-\mu(t-s)} M_{f}\, \mathrm{d}s \biggr]^{p} \\ \leq& 2^{p-1}M^{p}\bigl[L_{f}(1+L_{2}) \bigr]^{p} \biggl[ \int_{0}^{t}e^{-\mu (t-s)}\,\mathrm{d}s \biggr]^{p-1} \int_{0}^{t}e^{-\mu(t-s)}\mathbb{E} \Vert x_{s} \Vert ^{p}\,\mathrm{d}s + 2^{p-1}M^{p}M_{f}^{p} \mu^{-p} \\ \leq &2^{p-1}M^{p} \bigl[L_{f}(1+L_{2}) \bigr]^{p}\mu^{1-p} \int_{0}^{t}e^{-\mu (t-s)}\mathbb{E} \Vert x_{s} \Vert ^{p}\,\mathrm{d}s + 2^{p-1}M^{p}M_{f}^{p} \mu^{-p}. \end{aligned}$$
(3.19)

By (H1), (H5) and the Hölder inequality, we deduce

$$\begin{aligned} G_{5}(t) = & \mathbb{E} \biggl\Vert \sum _{0< t_{i}< t}T(t-t_{i})I_{i}\bigl(x \bigl(t_{i}^{-}\bigr)\bigr) \biggr\Vert ^{p} \leq M^{p}\mathbb{E} \biggl\{ \sum_{0< t_{j}< t}e^{-\mu(t-t_{j})}d_{j} \bigl\Vert x\bigl(t_{j}^{-}\bigr) \bigr\Vert \biggr\} ^{p} \\ \leq& M^{p}\Biggl\{ \sum_{j=1}^{m}d_{j} \Biggr\} ^{p-1}\sum_{t_{j}< t}d_{j}e^{-p\mu (t-t_{j})} \mathbb{E} \bigl\Vert x\bigl(t_{j}^{-}\bigr) \bigr\Vert ^{p} \\ \leq& M^{p}\Biggl\{ \sum_{j=1}^{m}d_{j} \Biggr\} ^{p-1}\sum_{t_{j}< t}d_{j}e^{-\mu (t-t_{j})} \mathbb{E} \bigl\Vert x\bigl(t_{j}^{-}\bigr) \bigr\Vert ^{p}. \end{aligned}$$
(3.20)

By (H1), (H6), Lemma 2.1 and the Kahane-Khintchine inequality, there exists a constant \(C_{p}\) such that

$$ G_{6}(t) = \mathbb{E} \biggl\Vert \int_{0}^{t} T(t-s) \tilde{F}(s)\, \mathrm{d}B_{Q}^{H}(s) \biggr\Vert ^{p} \leq C_{p} \biggl[\mathbb{E} \biggl\Vert \int_{0}^{t} T(t-s) \tilde{F}(s)\, \mathrm{d}B_{Q}^{H}(s) \biggr\Vert ^{2} \biggr]^{p/2}. $$

Choosing a suitable \(\epsilon>0\) small enough such that \((\mu-\epsilon)p\ge2\mu\), we derive

$$\begin{aligned} \mathbb{E} \biggl\Vert \int_{0}^{t} T(t-s) \tilde{F}(s)\, \mathrm{d}B_{Q}^{H}(s) \biggr\Vert ^{2} \le& M^{2}c_{H}H(2H-1)t^{2H-1} \int_{0}^{t}e^{-2\mu(t-s)} \bigl\Vert \tilde{F}(s) \bigr\Vert ^{2}_{\mathcal {L}_{Q}^{0}}\,\mathrm{d}s \\ \leq &M^{2}c_{H}H(2H-1)t^{2H-1} \int_{0}^{t}e^{-\mu(t-s)} \bigl\Vert \tilde{F}(s) \bigr\Vert ^{2}_{\mathcal {L}_{Q}^{0}}\,\mathrm{d}s \\ \leq&e^{-(\mu-\epsilon)t}M^{2}c_{H}H(2H-1)t^{2H-1}e^{-\epsilon t} \int_{0}^{+\infty} e^{\mu s} \bigl\Vert \tilde{F}(s) \bigr\Vert ^{2}_{\mathcal {L}_{Q}^{0}}\,\mathrm{d}s. \end{aligned}$$

Then, by (H6), we assume

$$ E_{6}=C_{p} \biggl[M^{2}c_{H}H(2H-1)\sup _{t\ge0}\bigl\{ t^{2H-1}e^{-\epsilon t}\bigr\} \int_{0}^{+\infty} e^{\mu s} \bigl\Vert \tilde{F}(s) \bigr\Vert ^{2}_{\mathcal{L}_{Q}^{0}}\,\mathrm{d}s \biggr]^{p/2},\quad \mbox{for all }t\ge0. $$

Therefore

$$ G_{6}(t)\leq E_{6}e^{-\mu t}. $$
(3.21)

From (3.15)-(3.21), we have

$$\begin{aligned} \mathbb{E} \bigl\Vert x(t) \bigr\Vert ^{p} \leq& 6^{p-1}E_{1}e^{-\mu t} + 12^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}\bigl[L_{g}(1+L_{1}) \bigr]^{p}\mathbb {E} \Vert x _{s} \Vert ^{p} + 12^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}M_{g}^{p} \\ &{}+ 12^{p-1}M_{1-\beta}^{p}\mu^{p-p\beta-\frac{p}{q}}\bigl[ \Gamma (1+q\beta-q)\bigr]^{\frac{p}{q}}\bigl[L_{g}(1+L_{2}) \bigr]^{p} \int _{0}^{t}e^{-\mu(t-s)}\mathbb{E} \Vert x_{s} \Vert ^{p}\,\mathrm{d}s \\ &{}+ 12^{p-1}M_{1-\beta}^{p}M_{g}^{p} \mu^{p-p\beta-\frac{p}{q}-1}\bigl[\Gamma (1+q\beta-q)\bigr]^{\frac{p}{q}} \\ &{}+ 12^{p-1}M^{p}\bigl[L_{f}(1+L_{2}) \bigr]^{p}\mu^{1-p} \int_{0}^{t}e^{-\mu (t-s)}\mathbb{E} \Vert x_{s} \Vert ^{p}\,\mathrm{d}s + 12^{p-1}M^{p}M_{f}^{p} \mu^{-p} \\ &{}+ 6^{p-1}M^{p}\Biggl(\sum_{j=1}^{\infty}d_{j} \Biggr)^{p-1}\sum_{t_{j}< t}d_{j}e^{-\mu(t-t_{j})} \mathbb{E} \bigl\Vert x\bigl(t_{j}^{-}\bigr) \bigr\Vert ^{p} + 6^{p-1}E_{6}e^{-\mu t} \\ =&6^{p-1}[E_{1} + E_{6}]e^{-\mu t} + 12^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p} \bigl[L_{g}(1+L_{1})\bigr]^{p}\mathbb {E} \Vert x _{s} \Vert ^{p} \\ &{} + 12^{p-1}M_{1-\beta}^{p}\mu^{p-p\beta-\frac{p}{q}}\bigl[ \Gamma (1+q\beta-q)\bigr]^{\frac{p}{q}}\bigl[L_{g}(1+L_{2}) \bigr]^{p} \int _{0}^{t}e^{-\mu(t-s)}\mathbb{E} \Vert x_{s} \Vert ^{p}\,\mathrm{d}s \\ &{}+12^{p-1}M^{p}\bigl[L_{f}(1+L_{2}) \bigr]^{p}\mu^{1-p} \int _{0}^{t}e^{-\mu(t-s)}\mathbb{E} \Vert x_{s} \Vert ^{p}\,\mathrm{d}s \\ &{} +6^{p-1}M^{p}\Biggl(\sum_{j=1}^{\infty}d_{j} \Biggr)^{p-1}\sum_{t_{j}< t}d_{j}e^{-\mu(t-t_{j})} \mathbb{E} \bigl\Vert x\bigl(t_{j}^{-}\bigr) \bigr\Vert ^{p} + 12^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}M_{g}^{p} \\ &{} +12^{p-1}M_{1-\beta}^{p}M_{g}^{p} \mu^{p-p\beta-\frac{p}{q}-1}\bigl[\Gamma (1+q\beta-q)\bigr]^{\frac{p}{q}} + 12^{p-1}M^{p}M_{f}^{p}\mu^{-p}. \end{aligned}$$
(3.22)

By Lemma 3.3 and (H7), there exist positive constants δ and J such that

$$ \mathbb{E} \bigl\Vert x(t) \bigr\Vert ^{p} \leq \delta e^{-\lambda t} + (1 - \Lambda )^{-1}J, \quad t \in[-r, \infty), $$
(3.23)

where \(\lambda\in(0,\mu)\).

Therefore, \(\mathcal{S}_{1}\) is an attracting set of the mild solution of (1.1).

When \(\varphi\in\mathcal{S}_{2}\), we deduce, from (3.22),

$$\begin{aligned} \mathbb{E} \bigl\Vert x(t) \bigr\Vert ^{p} \leq& 6^{p-1}M^{p} \bigl[\mathbb{E} \bigl\Vert \phi(0) \bigr\Vert ^{p} + 2^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}\bigl(L_{g}^{p} \Vert \phi \Vert ^{p} + M_{g}^{p}\bigr) \bigr] \\ &{}+ 6^{p-1}C_{p} \biggl[M^{2}c_{H}H(2H-1) \sup_{t\ge0}\bigl\{ t^{2H-1}e^{-\epsilon t}\bigr\} \int_{0}^{+\infty} e^{\mu s} \bigl\Vert \tilde{F}(s) \bigr\Vert ^{2}_{\mathcal{L}_{Q}^{0}}\,\mathrm{d}s \biggr]^{p/2} \\ &{}+ 12^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}L_{g}^{p}(1+L_{1})^{p} \mathbb{E} \Vert x _{s} \Vert ^{p} \\ &{} + 12^{p-1}M_{1-\beta}^{p}\mu^{p-p\beta-\frac{p}{q}}\bigl[ \Gamma (1+q\beta-q)\bigr]^{\frac{p}{q}}\bigl[L_{g}(1+L_{2}) \bigr]^{p} \int _{0}^{t}e^{-\mu(t-s)}\mathbb{E} \Vert x_{s} \Vert ^{p}\,\mathrm{d}s \\ &{}+12^{p-1}M^{p}\bigl[L_{f}(1+L_{2}) \bigr]^{p}\mu^{1-p} \int _{0}^{t}e^{-\mu(t-s)}\mathbb{E} \Vert x_{s} \Vert ^{p}\,\mathrm{d}s \\ &{} +6^{p-1}M^{p}\Biggl[\sum_{j=1}^{\infty}d_{j} \Biggr]^{p-1}\sum_{t_{j}< t}d_{j}e^{-\mu(t-t_{j})} \mathbb{E} \bigl\Vert x\bigl(t_{j}^{-}\bigr) \bigr\Vert ^{p} + 12^{p-1} \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p}M_{g}^{p} \\ &{}+12^{p-1}M_{1-\beta}^{p}M_{g}^{p} \mu^{p-p\beta-\frac{p}{q}-1} \bigl(\Gamma(1+q\beta-q)\bigr)^{\frac{p}{q}} + 12^{p-1}M^{p}M_{f}^{p}\mu^{-p}. \end{aligned}$$
(3.24)

By Lemma 3.1 and (3.24), we have

$$ \mathbb{E} \bigl\Vert x(t) \bigr\Vert ^{p} \leq (1 - \Lambda)^{-1} \varpi. $$
(3.25)

Therefore, \(\mathcal{S}_{2}\) is a quasi-invariant set of the mild solution of (1.1). □

Corollary 3.7

Assume that the assumptions (H1)-(H7) hold and \(M_{f} = M_{g} = 0\) are satisfied. Then the trivial solution of (1.1) is globally mean square exponentially stable if the following inequality holds:

$$\begin{aligned}& 6 \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{p} \bigl[L_{g}(1+L_{1})\bigr]^{2} + 6M_{1-\beta}^{2}\bigl[L_{g}(1+L_{2}) \bigr]^{2}\mu^{-2\beta-1}\Gamma(2\beta -1) \\& \quad {}+ 6 M^{2}\bigl[L_{f}(1+L_{2}) \bigr]^{2}\mu^{-2} + 6 M^{2} \Biggl(\sum _{j=1}^{\infty}d_{j} \Biggr)^{2} < 1. \end{aligned}$$
(3.26)

Proof

From (H1)-(H7) and (3.15), we have

$$\begin{aligned} \mathbb{E} \bigl\Vert x(t) \bigr\Vert ^{2} \leq& 12 M^{2} \bigl[\mathbb{E} \bigl\Vert \varphi(0) \bigr\Vert ^{2} + \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{2}L_{g}^{2}\mathbb{E} \Vert \varphi \Vert ^{2} \bigr]e^{-\mu t} \\ &{}+ 6 \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{2}\bigl[L_{g}(1+L_{1}) \bigr]^{2}\mathbb {E} \Vert x_{s} \Vert ^{2} \\ &{} +6 M^{2}\bigl[L_{f}(1+L_{2}) \bigr]^{2}\mu^{-1} \int_{0}^{t}e^{-\mu(t-s)}\mathbb {E} \Vert x_{s} \Vert ^{2}\,\mathrm{d}s \\ &{}+6 M^{2}\Biggl(\sum _{j=1}^{\infty}d_{j}\Biggr)\sum _{t_{j}< t}d_{j}e^{-\mu (t-t_{j})}\mathbb{E} \bigl\Vert x\bigl(t_{j}^{-}\bigr) \bigr\Vert ^{2} \\ &{}+ 6M_{1-\beta}^{2}\bigl[L_{g}(1+L_{2}) \bigr]^{2}\mu^{-2\beta}\Gamma(2\beta -1) \int_{0}^{t}e^{-\mu(t-s)}\mathbb{E} \Vert x_{s} \Vert ^{2}\,\mathrm{d}s \\ &{}+6C_{p}M^{2}c_{H}t^{2H-1} \int_{0}^{+\infty}e^{\mu s} \bigl\Vert \tilde{F}(s) \bigr\Vert ^{2}_{\mathcal{L}_{Q}^{0}}\,\mathrm{d}se^{-\mu t}. \end{aligned}$$

Let \(\lambda\in(0,\mu)\). Then we derive

$$\begin{aligned} \mathbb{E} \bigl\Vert x(t) \bigr\Vert ^{2} \leq& 12 M^{2}\bigl\{ \mathbb{E} \bigl\Vert \varphi(0) \bigr\Vert ^{2} + \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{2}L_{g}^{2}\mathbb{E} \Vert \varphi \Vert ^{2}\bigr\} e^{-\lambda t} \\ &{}+ 6 \bigl\Vert (-A)^{-\beta} \bigr\Vert ^{2}\bigl[L_{g}(1+L_{1}) \bigr]^{2}\mathbb {E} \Vert x_{s} \Vert ^{2} \\ &{} + \bigl\{ 6M_{1-\beta}^{2}\bigl[L_{g}(1+L_{2}) \bigr]^{2}\mu^{-2\beta}\Gamma (2\beta-1) \\ &{}+6 M^{2} \bigl[L_{f}(1+L_{2}) \bigr]^{2}\mu^{-1} \bigr\} \int_{0}^{t}e^{-\lambda(t-s)}\mathbb{E} \Vert x_{s} \Vert ^{2}\,\mathrm{d}s \\ &{} +6 M^{2}\Biggl(\sum_{j=1}^{\infty}d_{j} \Biggr)\sum_{t_{j}< t}d_{j}e^{-\lambda (t-t_{j})} \mathbb{E} \bigl\Vert x\bigl(t_{j}^{-}\bigr) \bigr\Vert ^{2} \\ &{}+6C_{p}M^{2}c_{H}\sup_{t\ge0} \bigl\{ t^{2H-1}e^{-(\mu-\lambda) t}\bigr\} \int_{0}^{+\infty}e^{\mu s} \bigl\Vert \tilde{F}(s) \bigr\Vert ^{2}_{\mathcal{L}_{Q}^{0}}\,\mathrm{d}se^{-\lambda t}. \end{aligned}$$
(3.27)

From (3.26) and Corollary 3.5 we conclude that the trivial solution of (1.1) is globally mean square exponentially stable. □

4 Conclusions

In this paper, we introduced two new impulsive integral inequalities with respect to neutral stochastic integro-differential equations with impulses driven by fBm. We studied the attracting and quasi-invariant sets of the system by use of the impulsive integral inequalities. Furthermore, exponential stability of the mild solution is established under sufficient conditions.