1 Introduction

In recent years, the late-time cosmic acceleration has been confirmed by several observational pieces of evidence [1–6]. This important discovery leads to one of the great puzzles in cosmology, and various plausible models have been developed to unravel the nature of such a late-time speed-up over the last decade. There have been many attempts at building up reasonable and consistent models by modifying the standard cosmology, which can be roughly categorized into two major directions: one is to introduce a dominant dark energy component in the Universe (see, e.g., Ref. [7]), while the other is to modify Einstein’s general relativity at large scales (see, e.g., Refs. [8–10]).

An intriguing brane-world scenario proposed by Dvali, Gabadadze, and Porrati (DGP) provides a new mechanism with an induced gravity (IG) term, i.e., a four-dimensional (4D) Ricci scalar, included in the brane action [11]. The IG term is expected to arise as a quantum correction due to the matter field on the brane [12], and it makes possible to reproduce the correct 4D Newtonian gravity at short distances even if the bulk is a five-dimensional (5D) Minkowski space-time with an infinite size [11]. The promising feature of the DGP model is that, when generalized to a Friedmann–Lemaître–Robertson–Walker brane with ordinary matter on it, one of its solutions, called the self-accelerating branch, will become asymptotically de Sitter in the far future, giving rise to a late-time accelerating phase without needing to introduce additional substances on the brane that violates the strong energy condition [13, 14].

Despite this advantage, it was pointed out later on that the self-accelerating branch is plagued with a ghost instability [15–20]. The spin-2 perturbations in this branch, viewed as an effective 4D massive gravity theory on a de Sitter background, are composed of a tower of infinite Kaluza–Klein (KK) massive gravitons. Then the mass of the lowest mode m is within the range \(0<m^2<2H^2\) if the brane tension is positive, where H is the Hubble parameter, and thus there will be a spin-2 ghost excitation in its helicity-0 component [21].Footnote 1 On the other hand, if the brane tension is negative, the lowest mass is larger than the critical scale, i.e., \(2H^2<m^2\), but the spin-0 perturbation, associated with the brane-bending mode, becomes a ghost instead [17]. In the specific case without brane tension, the lowest mass is equal to the critical scale. Even in this marginal case a detailed analysis shows the existence of a ghost from the mixing between the spin-0 sector and the helicity-0 part of the spin-2 sector [18]. Furthermore, the appearance of ghosts in the DGP self-accelerating branch cannot be eliminated even by invoking a second brane in the bulk with a stabilization mechanism [24]. For more discussions on DGP ghosts, please see Ref. [20] and the references therein. Nonlinear instabilities of the model have also been discussed in Refs. [25–27].

In this paper, we will investigate the possibility of avoiding the ghost in a generalized DGP model. A natural generalization to the DGP gravitational action, based on its higher-dimensional nature, is by adding the Gauss–Bonnet (GB) term to the original 5D bulk action [28–48]. This modification then yields the most general field equation for the bulk metric with its derivatives only up to the second order [28]. Moreover, this GB term keeps the bulk theory ghost-free and arises as the leading-order correction to the low-energy effective action of the heterotic string theory [29, 30], and furthermore, it plays an essential role in the Chern–Simons gauge theory of gravity [31–33].

This approach has been first discussed in Ref. [34]. Modes that have the potential to be ghosts are (i) the helicity-0 excitation of the lightest KK graviton and (ii) the brane-bending mode. In Ref. [34], considering a model with a minus sign in front of the bulk Einstein–Hilbert term, the authors succeeded in eliminating these modes. However, considering only an Einstein–Hilbert term in the 5D action even with a minus sign generates ghost modes for the graviton in the bulk. The ghosts can be eliminated by introducing a GB term with a negative prefactor and by taking the anti-de Sitter (AdS\(_5\)) vacuum called the GB vacuum. It is a kind of a ghost condensation of spin-2 particles. Therefore, their model is ghost-free both in the five-dimensional theory and in the four-dimensional effective theory.

The purpose of this paper is, in contrast, confirming if the straightforward extension of DGP model with the GB term still bothers the ghost mode. As discussed in Ref. [34], we expect that the discrete light mode of the KK tower of the graviton and the brane-bending mode to appear in the four-dimensional effective action. Even though these two modes have the potential to be a ghost, their existence does not necessarily imply a ghost excitation. The massive graviton is a ghost excitation only if its mass squared is less than \(2H^2\), while the brane-bending mode becomes a ghost only if the kinetic term has a wrong sign. In the original DGP model, the ghost conditions have been carefully checked [15–20], and one of the above mentioned condition is always satisfied, i.e. one of the ghost modes appears. This means that the flip of the sign in the kinetic term of the brane-bending mode happens when the squared value of the mass of the lightest KK graviton is \(2H^2\). Nevertheless, its reason is still mysterious [24] and there is no reason why in the extended models the same happens. Moreover, we may expect that the GB terms give a large correction to the self-accelerating branch. With GB corrections, there are three branches of solutions; two branches appearing in the original DGP (the normal branch and the self-accelerating branch) and one additional branch called the GB branch. The transition into the GB branch appears at the high energy region of the self-accelerating branch, and thus it is natural to expect that the self-accelerating branch is largely modified. Therefore, a detailed analysis is needed to confirm the existence of the ghost. Then we study herein the linear perturbations around the background given by (2.8) and (2.9) with \(\epsilon =+1\) which includes as a particular case the self-accelerating branch, and carefully examine whether or not the ghost excitations appearing in the DGP model could be possibly evaded in this framework. We show that even by including the GB term into the bulk, the ghost excitations are still present in this model.

The outline of the paper is as follows. In Sect. 2, we consider the generalized DGP model with the GB term as well as a cosmological constant in the bulk. We then review the background solutions of this system. In Sect. 3, we study the linear perturbations over an AdS\(_5\) bulk with a de Sitter brane within the model introduced in Sect. 2. In Sect. 4, we analyze the effective action for these perturbations, from which we examine the existence of the ghosts in this model. Finally, we present our summary in Sect. 5.

2 The model

We consider a generalized DGP model with a GB term and a cosmological constant included in the bulk action. The gravitational action for this system is given by [35–43]

$$\begin{aligned} S=&\,\frac{1}{2\kappa _5^2}\int _M\mathrm{d}^5x\sqrt{-\,^{(5)\!}g}[\mathcal {R}-2\Lambda _5+\alpha (\mathcal {R}^2-4\mathcal {R}_{ab} \mathcal {R}^{ab}+\nonumber \\&\mathcal {R}_{abcd}\mathcal {R}^{abcd})]-\frac{1}{\kappa _5^2}\int _{\Sigma _{\pm }}\mathrm{d}^4x\sqrt{-g}[K+2\alpha (J-\nonumber \\&2G^{\mu \nu }K_{\mu \nu })]+\int _{\Sigma }\mathrm{d}^4x\sqrt{-g}\left[ \frac{\gamma }{2\kappa ^2_4}R-\lambda +\fancyscript{L}_m\right] , \end{aligned}$$
(2.1)

where the 5D manifold M is split into two regions by a brane hypersurface \(\Sigma \), and the two sides of the brane are denoted by \(\Sigma _{\pm }\). The Latin indices \(a,b,c,\ldots ,\) run from 0 to 4, while the Greek indices \(\mu ,\nu ,\ldots ,\) run from 0 to 3. \(^{(5)\!}g_{ab}\) is the five-dimensional metric, and \(g_{ab}=\,^{(5)\!}g_{ab}-n_an_b\) is the induced metric on the brane, with \(n^a\) being the unit normal vector to the brane; \(\mathcal {R}\), R, \(\kappa _5^2\), \(\Lambda _5\,(<0)\), \(\lambda \), and \(\fancyscript{L}_m\) are the 5D Ricci scalar, the 4D Ricci scalar of the induced metric, the bulk gravitational constant, the bulk cosmological constant, the brane tension, and the matter Lagrangian on the brane, respectively. The GB parameter is denoted by \(\alpha \) (\(\ge \)0), which has the dimension of length square, and the strength of the IG term is characterized by a dimensionless parameter \(\gamma \). Moreover, the second term in Eq. (2.1) corresponds to the generalized York–Gibbons–Hawking surface term [49–52], where \(K_{\mu \nu }\) is the extrinsic curvature, \(G_{\mu \nu }\) the Einstein tensor of the induced metric, and J the trace of

$$\begin{aligned} J_{\mu \nu }&=\frac{1}{3}(2KK_{\mu \sigma }K^{\sigma }\,_{\nu }+K_{\rho \sigma }K^{\rho \sigma }K_{\mu \nu }\nonumber \\&\quad -2K_{\mu \rho }K^{\rho \sigma } K_{\sigma \nu }-K^2K_{\mu \nu }). \end{aligned}$$
(2.2)

The 5D field equation, obtained by varying the bulk action in Eq. (2.1), is given by [52–56]

$$\begin{aligned} \mathcal {G}_{ab}+\Lambda _5 \,^{(5)\!}g_{ab}-\frac{\alpha }{2} \mathcal {H}_{ab}=0, \end{aligned}$$
(2.3)

where \(\mathcal {G}_{ab}\) is the 5D Einstein tensor and the quadratic curvature correction \(\mathcal {H}_{ab}\) reads

$$\begin{aligned} \mathcal {H}_{ab}&=(\mathcal {R}^2-4\mathcal {R}_{cd}\mathcal {R}^{cd}+\mathcal {R}_{cdef} \mathcal {R}^{cdef})\,^{(5)\!}g_{ab}\nonumber \\&\quad -4(\mathcal {R}\mathcal {R}_{ab}-2\mathcal {R}_{ac}\mathcal {R}_{b}\,^{c}-2\mathcal {R}_{acbd}\mathcal {R}^{cd}\nonumber \\&\quad +\mathcal {R}_{acde}\mathcal {R}_b\,^{cde}) \end{aligned}$$
(2.4)

With the \(\mathbb {Z}_2\) symmetry assumed across the brane, the junction condition imposed at the brane is then given by [52–56],

$$\begin{aligned}&K_{\mu \nu }+2\alpha \left[ 3J_{\mu \nu }-\frac{2}{3}Jg_{\mu \nu }\!+\!\left( 2P_{\mu \rho \sigma \nu }\!-\!\frac{2}{3}g_{\mu \nu }G_{\rho \sigma }\right) K^{\rho \sigma }\right] \nonumber \\&\quad =r_c\left[ -\kappa _4^2\left( T_{\mu \nu }-\frac{1}{3}Tg_{\mu \nu }+\frac{1}{3}\lambda g_{\mu \nu }\right) \right. \nonumber \\&\quad \quad +\left. \gamma \left( G_{\mu \nu }-\frac{1}{3}G g_{\mu \nu }\right) \right] , \end{aligned}$$
(2.5)

where the crossover scale \(r_c\) is defined by \(r_c\equiv \kappa _5^2/2\kappa _4^2\), \(T_{\mu \nu }\) is the energy-momentum tensor of the matter content on the brane, and

$$\begin{aligned} P_{\mu \nu \rho \sigma }=R_{\mu \nu \rho \sigma }+2R_{\nu [\rho }g_{\sigma ]\mu }-2R_{\mu [\rho }g_{\sigma ]\nu }+Rg_{\mu [\rho }g_{\sigma ]\nu }. \end{aligned}$$
(2.6)

Here, we consider a background solution corresponding to a de Sitter brane with a vanishing energy-momentum tensor of matter, i.e., \(T_{\mu \nu }=0\), and the de Sitter brane is embedded in a bulk corresponding to two symmetric pieces of an AdS\(_5\) space-time glued through the brane. In this configuration, the bulk field equation (2.3) for the AdS\(_5\) bulk implies the relation \(\Lambda _5=-6\mu ^2+12\alpha \mu ^4\), where \(\mu \) is the energy scale associated with the AdS\(_5\) length \(\ell \equiv 1{/}\mu \) and has the following solutions:

$$\begin{aligned} \mu ^2=\frac{1}{4\alpha }\left[ 1\pm \sqrt{1+\frac{4}{3}\alpha \Lambda _5}\right] . \end{aligned}$$
(2.7)

However, it has been proved that the bulk solution with the \(+\) sign in Eq. (2.7) is perturbatively unstable [57–60]; therefore, we will focus on the \(-\) branch from now on, and the energy scale \(\mu ^2\) is then bounded asFootnote 2 \(0<\mu ^2<1/4\alpha \) accordingly. Moreover, this background can be described by the bulk metric

$$\begin{aligned} \mathrm{d}s^2=\mathrm{d}y^2+n^2(y)\gamma _{\mu \nu }\mathrm{d}x^\mu \mathrm{d}x^\nu , \end{aligned}$$
(2.8)

where the brane is located at \(y=0\), \(\gamma _{\mu \nu }\) is the 4D de Sitter metric with its scalar curvature \(R[\gamma _{\mu \nu }]=12H^2\), and the warp factor n(y) is given by

$$\begin{aligned} n(y)=\frac{H}{\mu } \sinh {[\mu (y_*+\epsilon |y|)]}, \end{aligned}$$
(2.9)

with

$$\begin{aligned} y_*=\frac{1}{\mu }\mathrm {arcsinh}{\left( \frac{\mu }{H}\right) }, \end{aligned}$$
(2.10)

where the warp factor has two possible branches with \(\epsilon =\pm 1\) and is normalized at the position of the brane as \(n(0)=1\).

From the junction condition (2.5), we derive the generalized Friedmann equation on the brane using the bulk metric (2.8),

$$\begin{aligned} \left[ 1\!+\!\frac{8}{3}\alpha \left( H^2\!-\!\frac{\mu ^2}{2}\right) \right] \sqrt{H^2+\mu ^2}=-\epsilon \,r_c\left( \frac{\kappa ^2_4}{3}\lambda \!-\!\gamma H^2\right) . \end{aligned}$$
(2.11)

The relation between \(\lambda \) and \(\sqrt{H^2+\mu ^2}\) is shown in Fig. 1 and the exact solutions of Eq. (2.11) have been analyzed in Refs. [35–43]. In general, there are three branches of solutions in Eq. (2.11), among which the “self-accelerating branch” with \(\epsilon =+1\) includes the DGP self-accelerating solution in the absence of the GB term and for a vanishing bulk cosmological constant [13, 14], while the “normal branch” with \(\epsilon =-1\), when switching off both the GB and the IG effects, will recover the Randall–Sundrum single brane model. We have an additional branch called the GB branch. The transition to the GB branch appears at the high energy regime of the self-accelerating branch (see Fig. 1). The ghost problem is known to arise on the DGP self-accelerating branch, i.e., \(\epsilon =+1\) with \(\alpha \rightarrow 0\) in Eq. (2.11). We will therefore restrict our analysis to the solution with \(\epsilon =+1\) and \(\alpha >0\), seeking to see if the GB term can in some way alleviate the ghost problem of the DGP self-accelerating model.

Fig. 1
figure 1

The relation between \(\lambda \) and \(\sqrt{H^2+\mu ^2}\) for \(\epsilon =+1\) in Eq. (2.11): the solid line corresponds to the self-accelerating branch which is connected with the other two branches at one of the edges. The upper dotted line is called GB branch, while the lower branch corresponds to the normal branch with \(\epsilon =+1\)

Full size image

3 Perturbed equations

We now study the linear perturbations on the self-accelerating branch following the methodology used in Ref. [24] (see also Ref. [61]). To derive the perturbed bulk field equation, we start with the perturbed metric

$$\begin{aligned} \mathrm{d}s^2=\mathrm{d}y^2+(n^2\gamma _{\mu \nu }+h^{TT}_{\mu \nu })\mathrm{d}x^\mu \mathrm{d}x^\nu , \end{aligned}$$
(3.1)

with the transverse-traceless (TT) gauge-fixing conditions,

$$\begin{aligned} \nabla ^{\mu }h^{TT}_{\mu \nu }=0, \quad h^{TT\mu }_{\,\,\,\mu }=0, \end{aligned}$$
(3.2)

where \(\nabla _{\mu }\) is the covariant derivative associated with the background induced metric \(g_{\mu \nu }=n^2\gamma _{\mu \nu }\). Using this TT perturbed metric (3.1), we obtain the perturbed bulk field equation from Eq. (2.3),

$$\begin{aligned} n^2h^{TT\prime \prime }_{\mu \nu }-2H^2h^{TT}_{\mu \nu }&-4n^2\mu ^2h^{TT}_{\mu \nu } =-(\square -2H^2)h^{TT}_{\mu \nu }, \end{aligned}$$
(3.3)

where the d’Alembertian is defined by \(\square =\gamma ^{\mu \nu }\nabla _{\mu }\nabla _{\nu }\), and the prime denotes a partial derivative with respect to y. However, if we choose the TT gauge for the metric perturbations as in Eq. (3.1), the brane position cannot be fixed at \(y=0\) but will in general reside at \(y=\xi (x^{\mu })\) deviating slightly from the unperturbed position.

For the calculation of the perturbed junction condition, it is more convenient to introduce the Gaussian normal coordinate adapted to the brane hypersurface,

$$\begin{aligned} \mathrm{d}s^2=\mathrm{d}\bar{y}^2+(n^2\gamma _{\mu \nu }+\bar{h}_{\mu \nu })\mathrm{d}\bar{x}^\mu \mathrm{d}\bar{x}^\nu , \end{aligned}$$
(3.4)

where the brane is now chosen to be fixed at \(\bar{y}=0\). The perturbed junction condition imposed at the brane is then obtained by using this perturbed metric in Eq. (2.5),

$$\begin{aligned}&(1-4\alpha \mu ^2)\left( \bar{h}_{\mu \nu }^{\prime }-2\sqrt{H^2+\mu ^2}\,\,\bar{h}_{\mu \nu }\right) \nonumber \\&\quad =-\kappa ^2_5\left( T_{\mu \nu }-\frac{1}{3}\gamma _{\mu \nu }T\right) \nonumber \\&\qquad \quad +2\left( \gamma r_c-4\alpha \sqrt{H^2+\mu ^2}\right) \left( X_{\mu \nu }-\frac{1}{3}\gamma _{\mu \nu }X\right) , \end{aligned}$$
(3.5)

where the energy-momentum tensor \(T_{\mu \nu }\) is assumed to be of first order, and

$$\begin{aligned} X_{\mu \nu }&=-\frac{1}{2}(\square \bar{h}_{\mu \nu }-\nabla _\mu \nabla _\rho \bar{h}^\rho _{\,\nu }-\nabla _\nu \nabla _\rho \bar{h}^\rho _{\,\mu } +\nabla _\mu \nabla _\nu \bar{h})\nonumber \\&\qquad +\frac{1}{2}\gamma _{\mu \nu }(\square \bar{h}-\nabla _\rho \nabla _\sigma \bar{h}^{\rho \sigma })+H^2\left( \bar{h}_{\mu \nu }+\frac{1}{2}\gamma _{\mu \nu }\bar{h}\right) . \end{aligned}$$
(3.6)

Now we have two perturbed equations expressed under two different gauge-fixing conditions. The gauge transformation between the metric written in the TT coordinate (3.1) and that in the Gaussian normal coordinate (3.4) can be carried out by the following coordinate transformations:

$$\begin{aligned}&y-\bar{y}=\hat{\xi }^y(x^\rho ),\end{aligned}$$
(3.7)
$$\begin{aligned}&x^\mu -\bar{x}^\mu =\frac{\sqrt{H^2+n^2\mu ^2}}{nH^2}\gamma ^{\mu \sigma }\partial _\sigma \hat{\xi }^y(x^\rho )+\hat{\xi }^\mu (x^\rho ). \end{aligned}$$
(3.8)

Therefore, the transformation of the metric perturbations between these two coordinates is obtained as follows:

$$\begin{aligned} \bar{h}_{\mu \nu }=&\,h^{TT}_{\mu \nu }+\frac{2\,n\sqrt{H^2+n^2\mu ^2}}{H^2}\,\nabla _\mu \nabla _\nu \,\hat{\xi }^y\nonumber \\&+2\,n\sqrt{H^2+n^2\mu ^2}\,\gamma _{\mu \nu }\hat{\xi }^y+2\nabla _{(\mu }\hat{\xi }_{\nu )}. \end{aligned}$$
(3.9)

Furthermore, we can fix the function \(\hat{\xi }^{\mu }\) such that the combination of the second and the last term in Eq. (3.9) vanishes at the position of the brane. As a result, after fixing the function \(\hat{\xi }^{\mu }\) in this way, we have

$$\begin{aligned} \bar{h}_{\mu \nu }=&\,h^{TT}_{\mu \nu }\!+\!\frac{2\,n}{H^2}\left( \sqrt{H^2+n^2\mu ^2}-n\sqrt{H^2+\mu ^2}\right) \nabla _\mu \nabla _\nu \,\hat{\xi }^y\nonumber \\&+2\,n\sqrt{H^2+n^2\mu ^2}\,\,\gamma _{\mu \nu }\hat{\xi }^y. \end{aligned}$$
(3.10)

Substituting the transformation (3.10) into the perturbed junction condition (3.5), we can decompose the perturbed junction condition imposed at \(y=0\) into the traceless part,

$$\begin{aligned}&(1-4\alpha \mu ^2)\left( h^{TT\prime }_{\mu \nu }-2\sqrt{H^2+\mu ^2}\,h^{TT}_{\mu \nu }\right) \nonumber \\&\quad +\left( \gamma r_c-4\alpha \sqrt{H^2+\mu ^2}\right) \nonumber \\&\quad \times \,(\square -2H^2)h^{TT}_{\mu \nu }=-\kappa ^2_5\,\Sigma _{\mu \nu }, \end{aligned}$$
(3.11)

where

$$\begin{aligned} \Sigma _{\mu \nu }=&\left( T_{\mu \nu }-\frac{1}{4}\gamma _{\mu \nu } T\right) -\frac{2}{\kappa ^2_5}\bigg [1+4\alpha (\mu ^2+2H^2)\nonumber \\&-2\gamma r_c\sqrt{H^2+\mu ^2}\bigg ]\left( \nabla _\mu \nabla _\nu -\frac{1}{4}\gamma _{\mu \nu }\square \right) \hat{\xi }^y, \end{aligned}$$
(3.12)

as well as the trace part,

$$\begin{aligned}&(\square +4H^2)\hat{\xi }^y \nonumber \\&\quad =\frac{\kappa ^2_5}{6\left[ 2\gamma r_c\sqrt{H^2+\mu ^2}-1-4\alpha (\mu ^2+2H^2)\right] }T. \end{aligned}$$
(3.13)

\(\hat{\xi }^y\) shows the position of the brane under TT gauge-fixing condition, and thus this equation represents the dynamics of the brane-bending mode. Moreover, it can be shown that the traceless source \(\Sigma _{\mu \nu }\) also satisfies the transverse condition, \(\nabla ^{\mu }\Sigma _{\mu \nu }=0\), by using Eq. (3.13) and the following identity holding for any 4D scalar F:

$$\begin{aligned} \nabla ^{\mu }\left( \nabla _{\mu }\nabla _{\nu }-\frac{1}{4}\gamma _{\mu \nu }\square \right) F=\frac{3}{4}n^{-2}\nabla _{\nu } (\square +4H^2) F. \end{aligned}$$
(3.14)

We now proceed to derive the equations for the KK spin-2 modes and the brane-bending mode from the viewpoint of the 4D effective theory. The complete field equation for \(h^{TT}\) can be obtained by combining its bulk part (3.3) and the junction condition imposed on it (3.11),

$$\begin{aligned}&\left( \hat{L}+\frac{\square -2H^2}{n^2}\right) h^{TT}_{\mu \nu }=-\frac{2}{1-4\alpha \mu ^2} \nonumber \\&\quad \times \, \left[ \kappa ^2_5\,\Sigma _{\mu \nu }\!+\!\left( \gamma r_c -4\alpha \sqrt{H^2+\mu ^2}\right) (\square -2H^2)h^{TT}_{\mu \nu }\right] \delta (y), \end{aligned}$$
(3.15)

where the operator \(\hat{L}\) is defined as

$$\begin{aligned} \hat{L}\equiv \frac{1}{n^2}\partial _yn^4\partial _y\frac{1}{n^2}. \end{aligned}$$
(3.16)

Equation (3.15) can be further separated into an eigenvalue problem through the KK decomposition of the metric perturbations \(h^{TT}\),

$$\begin{aligned} h^{TT}_{\mu \nu }=\int \mathrm{d}m\,h_{\mu \nu }^m(x^{\mu })\,u_m(y), \end{aligned}$$
(3.17)

where “\(\int \)” denotes a summation over the discrete modes and an integration over the continuous modes, and \(u_m\) is the eigenfunction of the eigenvalue equation

$$\begin{aligned} -\hat{L}\,u_m\!=\!\frac{m^2}{n^2}\left[ 1\!+\!\frac{2}{1-4\alpha \mu ^2}\left( \gamma r_c-4\alpha \times \sqrt{H^2+\mu ^2}\right) \delta (y)\right] u_m, \end{aligned}$$
(3.18)

where m is the mass eigenvalue. Then the field equation (3.15), in terms of the KK decomposition (3.17), reduces to the simpler form,

$$\begin{aligned} \int&\mathrm{d}m\,\frac{1}{n^2}\left\{ \left[ 1+\frac{2}{1-4\alpha \mu ^2}\left( \gamma r_c-4\alpha \sqrt{H^2+\mu ^2}\right) \delta (y)\right] \right. \nonumber \\&\left. \times (\square -2H^2-m^2)h^m_{\mu \nu }\,u_m\right\} =-\frac{2\,\kappa ^2_5}{1-4\alpha \mu ^2}\Sigma _{\mu \nu }\delta (y). \end{aligned}$$
(3.19)

In addition, it can be shown that the eigenmodes \(u_m\) in Eq. (3.18) are mutually orthogonal with respect to the scalar product

$$\begin{aligned} (u_{\tilde{m}},u_m)&\equiv \int ^{\infty }_{-\infty }\frac{\mathrm{d}y}{n^2}\bigg \{\left[ 1 -4\alpha \mu ^2\right. \nonumber \\&+\left. 2\left( \gamma r_c -4\alpha \sqrt{H^2+\mu ^2}\right) \delta (y)\right] u_{\tilde{m}}u_m\bigg \}. \end{aligned}$$
(3.20)

This definition of the scalar product (3.20) will always give rise to a positive number in the self-accelerating branch [see Eqs. (2.7) and (2.11)]. Therefore, we will use it here to normalize each eigenmode; i.e., the eigenmodes \(u_m\) satisfy the condition \((u_{\tilde{m}},u_m)=\delta (\tilde{m},m)\), where the delta function \(\delta (\tilde{m},m)\) is a Kronecker delta for the discrete modes and a Dirac delta function for a continuous spectrum. Notice that the scalar product defined in Eq. (3.20) is the same as that given in Ref. [43] for the self-accelerating branch, since the eigenmodes defined in Ref. [43], \(\mathcal {E}_{m}\), correspond to those defined here through \(n^2\mathcal {E}_{m}=u_m\).

Given the scalar product constructed in Eq. (3.20), we can further simplify Eq. (3.19) by operating \(\int ^{\infty }_{-\infty }\mathrm{d}y \,\tilde{u}_m\) on both sides of Eq. (3.19) and using the orthonormality of the eigenmodes. The resulting equation is then given by

$$\begin{aligned} (\square -2H^2-m^2)h^m_{\mu \nu }=-2\kappa ^2_5\,\Sigma _{\mu \nu }\,u_m(0). \end{aligned}$$
(3.21)

From Eqs. (3.17) and (3.21), the solution for the metric perturbation \(h^{TT}_{\mu \nu }\) can be written as

$$\begin{aligned} h^{TT}_{\mu \nu }=-2\kappa ^2_5\int \mathrm{d}m\,\frac{u_m(0)u_m(y)}{\square -2H^2-m^2}\Sigma _{\mu \nu }. \end{aligned}$$
(3.22)

Notice that \(h^{TT}_{\mu \nu }\) is also sourced by the scalar mode \(\hat{\xi }^y\) [see Eqs. (3.12) and (3.13)]. To obtain the full induced metric perturbations on the brane, we substitute Eqs. (3.13) and (3.22) back into Eq. (3.10) and evaluate the metric perturbation \(\bar{h}_{\mu \nu }\) at \(y=0\). In addition, we note that the gauge in the form of the Gaussian normal coordinate (3.4) is not yet completely fixed, so we can further fix this gauge freedom by eliminating the term proportional to \(\nabla _{\mu }\nabla _{\nu }\,\hat{\xi }^{y}\). Therefore, after neglecting the term that can be erased by the gauge fixing, the induced metric perturbations on the brane, \(\bar{h}^b_{\mu \nu }(x^\mu )\equiv \bar{h}_{\mu \nu }(x^\mu ,0)\), is given by

$$\begin{aligned} \bar{h}^b_{\mu \nu }&(x^\mu )=-2\kappa _5^2\int \mathrm{d}m \,u^2_m(0)\bigg \{\frac{1}{\square -2H^2-m^2}\nonumber \\&\left[ \left( T_{\mu \nu }-\frac{1}{4}\gamma _{\mu \nu }T\right) +\frac{1}{3(m^2-2H^2)}\left( \nabla _{\mu } \nabla _{\nu }-\frac{1}{4}\gamma _{\mu \nu }\square \right) T\right] \nonumber \\&+\frac{1}{12(m^2-2H^2)}\gamma _{\mu \nu }T\bigg \}+\frac{\kappa ^2_5}{6}\gamma _{\mu \nu }\nonumber \\&\times \bigg [\frac{2\sqrt{H^2+\mu ^2}}{2\gamma r_c\sqrt{H^2+\mu ^2}-1-4\alpha (\mu ^2+2H^2)}\nonumber \\&+4H^2\left( \int \mathrm{d}m\frac{u^2_m(0)}{(m^2-2H^2)}\right) \bigg ]\frac{1}{\square +4H^2}T, \end{aligned}$$
(3.23)

where we have already made use of the identity

$$\begin{aligned}&\frac{1}{\square -2H^2-m^2}\left( \nabla _{\mu }\nabla _{\nu }-\frac{1}{4}\gamma _{\mu \nu }\,\square \right) F \nonumber \\&\quad =\left( \nabla _{\mu }\nabla _{\nu } \right. \,\,\left. -\frac{1}{4}\gamma _{\mu \nu }\,\square \right) \frac{1}{\square +6H^2-m^2}\,F, \end{aligned}$$
(3.24)

holding for an arbitrary 4D scalar F. Furthermore, we have also applied the operator decomposition as follows:

$$\begin{aligned}&\frac{1}{(\square +6H^2-m^2)(\square +4H^2)}\,F \nonumber \\&\quad =\frac{1}{m^2-2H^2}\left( \frac{1}{\square +6H^2-m^2}-\frac{1}{\square +4H^2}\right) F, \end{aligned}$$
(3.25)

as long as \(m^2\ne 2H^2\). As can be seen from the resulting solution in Eq. (3.23), the physical degrees of freedom contained in the induced metric perturbations on the brane are effectively composed not only of a KK tower of massive spin-2 gravitons [cf. Eq. (A8)] but of a spin-0 excitation associated with the brane-bending mode, corresponding to the second term of Eq. (3.23). In addition, we highlight that, if the trace of the energy-momentum tensor is nonzero, \(T\ne 0\), there is no pathology appearing in this model when \(m^2=2H^2\), in contrast with the Fierz–Pauli model for the spin-2 field (see Appendix A). In the Fierz–Pauli model, if the graviton mass square reaches the critical scale, \(2H^2\), the constraint for the trace of the field, Eq. (A6), implies that the trace of the energy-momentum tensor is zero. However, the seemingly divergent expression in Eq. (3.23) when \(m^2=2H^2\) is simply because we have decomposed the result into the spin-2 and the spin-0 sector through Eq. (3.25), which does not hold if \(m^2=2H^2\). In this sense, the spin-2 and the spin-0 sector are degenerate at this critical scale. Moreover, we can also check that there are no divergent expressions in Eqs. (3.13) and (3.22) when \(m^2=2H^2\) before arriving at the final result (3.23).

4 Effective action and existence of ghosts

To check explicitly the existence of ghosts in this model, we need to construct the 4D effective action responsible for all the degrees of freedom included in the solution (3.23). As we mentioned in the previous section, the induced metric perturbations (3.23) in general consist of a massive KK tower of spin-2 modes as well as a spin-0 mode, which are denoted here by \(\bar{h}^{m}_{\mu \nu }\) and s, respectively. Therefore, the induced metric perturbations \(\bar{h}^b_{\mu \nu }\) can be expressed as

$$\begin{aligned} \bar{h}^b_{\mu \nu }=\int \mathrm{d}m\,\bar{h}^m_{\mu \nu }+\frac{1}{4}\gamma _{\mu \nu }s. \end{aligned}$$
(4.1)

In terms of these notations, the lowest-order action for the coupling of the matter trapped on the brane to the induced gravitons is then written as

$$\begin{aligned} S_{\text {m}}= & {} \frac{1}{2}\int \mathrm{d}^4x\sqrt{-\gamma }\,\bar{h}^{b}_{\mu \nu }\,T^{\mu \nu } \nonumber \\= & {} \frac{1}{2}\int \mathrm{d}m\int \mathrm{d}^4x\sqrt{-\gamma }\,\bar{h}^{m}_{\mu \nu }\,T^{\mu \nu } +\frac{1}{8}\int \mathrm{d}x^4\sqrt{-\gamma }\,s\,T.\nonumber \\ \end{aligned}$$
(4.2)

Now we continue to deduce the kinetic part of the effective action, and we will deal with the spin-2 and the spin-0 sector separately for convenience. For the spin-2 modes alone, we temporarily consider the traceless source here for simplicity, i.e., \(T=0\), and so are the spin-2 fields \(\bar{h}^m_{\mu \nu }\) (see Appendix A for the general case without this restriction). Then the kinetic part of the effective action for these TT spin-2 fields \(\bar{h}^m_{\mu \nu }\) takes the form

$$\begin{aligned} S_{h}=\int \mathrm{d}m\,\alpha _m\int \mathrm{d}^4x\sqrt{-\gamma }\,\bar{h}^{m\mu \nu }(\square -2H^2-m^2)\bar{h}^{m}_{\mu \nu }, \end{aligned}$$
(4.3)

where the undetermined coefficients \(\alpha _m\) can be fixed by comparing the equation of motion derived from the action \(S_h+S_{\text {m}}\) with the one given in Eq. (3.23) as we consider only the spin-2 part with \(T=0\). As a result, we have

$$\begin{aligned} \alpha _m=\frac{1}{8\kappa ^2_5u_m^2(0)}. \end{aligned}$$
(4.4)

On the other hand, for the spin-0 mode alone, its kinetic part of the effective action is given by

$$\begin{aligned} S_{s}=\beta _s\int \mathrm{d}^4x\sqrt{-\gamma }\,s(\square +4H^2)s, \end{aligned}$$
(4.5)

where the coefficient \(\beta _s\) can be fixed as well using the same method, i.e., matching the field equation derived from the action \(S_s+S_{\text {m}}\) (only taking into account the second term of \(S_{\text {m}}\) here) with only the spin-0 part of Eq. (3.23). Then we end up with the result

$$\begin{aligned} \beta _s&=-\frac{3}{64\kappa _5^2}\bigg [\frac{\sqrt{H^2+\mu ^2}}{2\gamma r_c\sqrt{H^2+\mu ^2}-1-4\alpha (\mu ^2+2H^2)}\nonumber \\&\quad +2H^2\left( \int \mathrm{d}m\frac{u^2_m(0)}{(m^2-2H^2)}\right) \bigg ]^{-1}. \end{aligned}$$
(4.6)

For the massive KK modes in this framework, it can be shown that the zero mode with \(m=0\) satisfying the eigenvalue Eq. (3.18) is not normalizable with respect to the scalar product (3.20). Therefore, as in the DGP model, there is no physically admissible zero mode in the self-accelerating branch here. However, it has been shown that, as mentioned in the Introduction, the helicity-0 component of the spin-2 field becomes a ghost if its mass is in the range \(0<m^2<2H^2\) [21]. Thus, if the lightest massive KK mode in this model has a mass in this forbidden range, this system will contain a spin-2 ghost. On the contrary, if the mass of the lightest KK mode is higher than the critical scale, i.e., \(2H^2<m^2\), the spin-2 perturbations become healthy; however, whether the spin-0 mode is healthy in this case has yet to be checked.

We now turn to the spin-0 mode in the case where the mass of the lightest KK mode satisfies \(2H^2<m^2\). The coefficient of the action for the spin-0 mode, \(\beta _s\), is explicitly given in Eq. (4.6), in which the second term inside the bracket is always positive in this case. Besides, it is seen from Fig. 1 that the self-accelerating branch of this model is restricted to the parameter space

$$\begin{aligned}&\frac{\gamma r_c-\sqrt{\gamma ^2r_c^2-8\alpha (1-4\alpha \mu ^2)}}{8\alpha }<\sqrt{H^2+\mu ^2} \nonumber \\&\quad <\frac{\gamma r_c+\sqrt{\gamma ^2r_c^2-8\alpha (1-4\alpha \mu ^2)}}{8\alpha }, \end{aligned}$$
(4.7)

where the upper and lower bounds come from the transition to the GB branch and the normal branch but with \(\epsilon =+1\), respectively. Out of the range (4.7) with \(\epsilon =+1\), since the Hubble parameter is a decreasing function of the energy density, it cannot describe our Universe [41–43]. Within the region of the above parameter space, the first term in the bracket of Eq. (4.6) turns out to be a positive value. As a result, the coefficient \(\beta _s\) is always negative as long as the mass of the lowest mode is larger than the critical scale, i.e., \(2H^2<m^2\), indicating the presence of a spin-0 ghost. Consequently, similarly to the DGP model, there always exists a ghost in the self-accelerating branch of this model, which implies that the ghost instability present in the DGP model still cannot be removed by invoking a GB term in the bulk action.

The surprising fact is that the sign flip of the first term in the bracket of Eq. (4.6) always happens with the transition of branches, that is, they are mysteriously related. The lower bound of the inequality (4.7) has been already found in the original DGP model, while the upper bound the inequality (4.7) is shown here for the first time. Then, although the GB branch (with \(\epsilon =+1\)) is a theoretical object and can never describe our Universe, it is interesting that this branch can be ghost-free. Indeed, the GB branch despite having both modes (helicity-0 of the spin-2 sector and the brane-bending mode) can be ghost-free, unlike the self-accelerating branch where either the brane bending or the helicity-0 of the spin-2 is a ghost. The normal branch has a similar behavior to that of the GB branch. In summary, it might be possible to construct a viable cosmological model where despite the existence of the brane bending and the helicity-0 modes, there is no ghost. This may imply the possibility of a ghost-free interesting solution for \(\epsilon =+1\) with a nontrivial modification of the DGP model.

5 Summary

In this paper, we looked into a generalized DGP brane-world scenario with a GB term as well as a cosmological constant both incorporated in the bulk action. To check whether this framework can possibly provide a way out of the DGP ghost instability, we have studied the linear perturbations around a de Sitter self-accelerating brane embedded in an AdS\(_5\) bulk space-time. Having the linear perturbations analyzed in this system, we end up with the effective induced metric perturbations on the brane, Eq. (3.23), the physical degrees of freedom in which, as long as none of the KK modes has a critical mass, \(m^2\ne 2H^2\), can be effectively described in terms of the massive KK tower of the spin-2 gravitons as well as the spin-0 excitation associated with the brane-bending mode. Moreover, in contrast with the Fierz–Pauli model for the spin-2 field, gravity in this system can couple to matter with nonzero trace of the energy-momentum tensor when \(m^2=2H^2\), at which the spin-2 and the spin-0 perturbations are degenerate. Therefore, one can no longer divide the degrees of freedom into the spin-2 and the spin-0 sector at this critical scale.

It has been shown that the massive spin-2 field contains a ghost excitation in its helicity-0 component if the mass is in the range \(0<m^2<2H^2\) [21]. Thus, from the massive gravity theory viewpoint, if the mass of the lightest KK mode here is within this forbidden range, there will be a spin-2 ghost excitation present in this system. On the other hand, provided that the mass of the lightest mode becomes larger than the critical scale, \(2H^2<m^2\), although the spin-2 sector becomes healthy in this case, the spin-0 mode is shown to be a ghost instead, similarly to the DGP model. For the specific case where the lightest mass is equal to the critical scale, \(m^2=2H^2\), whether or not a ghost exists in this model cannot be verified rigorously through the method we used here. Presumably, this model still contains a ghost in this marginal case as happens in the DGP model [18]. However, this specific fine-tuning condition, i.e., \(m^2=2H^2\), is easily broken provided that we consider physical matter fields on the brane, in which case the Hubble parameter in general varies with time. As a result, the DGP ghost instability at the level of linear perturbations still cannot be eliminated by invoking the GB term in the bulk action.

Our result shows that in the self-accelerating branch the ghost mode always appears, while in the other branches (with \(\epsilon =+1\)) we have the ghost-free parameter range, although these branches cannot describe our Universe. This is because the sign of the kinetic term of the brane-bending mode not only depends on the mass of the lightest KK graviton but also seems to be related to the branches. We can see it from Eq. (4.6). The sign of the first term in the bracket of Eq. (4.6) is positive for the self accelerating branch and negative for the other two branches. Therefore, the brane-bending mode has both informations of the branch and the value of the lightest KK mass. This relation might be important for the future understanding of the origin of the ghost. Finally, could other generalizations of the DGP model along the line of Refs. [62, 63] appease the ghost problem? We leave this question to a future work.