1 Introduction

Since Kaluza and Klein proposed the idea that our fundamental spacetime may be more than four dimensions in 1920s [1, 2], extra dimension theories have received much attention from theoretical physicists. In Kaluza–Klein (KK) theory, the extra dimension is compactified as a circle with the radius of Planck length \(l_{Pl} \sim 10^{-35}\)m, and this strategy of compactifying extra dimensions dominates higher-dimensional unified physics [3]. Inspired by the string theory, the braneworld scenario provides another way to hide the extra dimensions. In this scenario, our universe is a brane embedded in a higher-dimensional bulk. All the Standard Model particles are trapped on the brane and only the graviton can propagate into the extra dimensions.

In 1999, Randall and Sundrum proposed a well-known braneworld model [4], which is now dubbed RS1 model. In this model, the bulk is a slice of a five-dimensional anti-de Sitter (AdS) spacetime, and there is a brane at each boundary of the slice. In order to maintain a flat four-dimensional spacetime on the brane, the fine-tuning condition requires that the tension of the ultraviolet (UV) brane be positive while the tension of the infrared (IR) brane be negative. Our universe lives on the IR brane, so owing to the exponentially warped extra dimension, a fundamental electroweak scale of the order of Planck scale (\(M_{Pl}\sim 10^{16}\)TeV) is redshifted to the observed effective electroweak scale (\(\sim \)246GeV). The RS1 model provides a natural way to solve the gauge hierarchy problem without introducing a new large hierarchy, so it draws lots of attention in the last two decades, see, e.g., Refs. [5,6,7,8,9,10,11,12,13,14] for introduction and recent works.

The RS1 model is based on the 5D general relativity (GR), where the 5D graviton is massless and possesses 5 degrees of freedom (DOF), therefore, there is a massless 4D graviton, a massless scalar radion, and a tower of massive 4D graviton in the effective 4D theory on the brane [4]. In the braneworld scenario, the effective 4D theory on the brane is related to the DOF of the high-dimensional gravitational theory. So it is natural to ask what is the difference between the 4D effective theories on the brane if the higher-dimensional graviton possesses different numbers of DOF. One simple exploration is to replace GR with the massive gravity theory, where the higher-dimensional massive graviton possesses more numbers of DOF.

Massive gravity theory is a generalization of GR by endowing the graviton with a nonzero mass. The first pioneering work was done by Fierz and Pauli in 1939 by adding non-derivative interaction terms, known as the Fierz–Pauli mass term, into the linearized level of GR [15]. However, the theory does not reduce to GR in the massless limit, so it suffers from a discontinuity problem, known as van Dam-Veltman-Zakharov discontinuity [16,17,18]. This problem can be alleviated by generalizing the linear Fierz–Pauli mass term to nonlinear terms [19]. However, the nonlinear terms will generate a higher derivative term and lead to the Boulware-Deser ghost instability [20]. A breakthrough was achieved by de Rham, Gabadadze, and Tolley about a decade ago [21, 22]. They constructed a nonlinear massive gravity theory with the field equation at most second order in time derivatives by getting rid of ghosts for all nonlinear self-interactions of the longitudinal component in the decoupling limit [23, 24]. This massive gravity theory is now dubbed dRGT gravity. However, it is found that the dRGT gravity may suffer from other pathologies. In Ref. [25], Deser and Waldron argued that the theory admits second order superluminal shock wave solutions, and the acausal characteristic can arise for any choice of background. A detailed and correct analysis of characteristic equations was performed by Izumi and Ong [26], and they showed that the theory does admit a well-posed Cauchy problem. Such acausality problem were further discussed in Refs. [27,28,29]. The problem of lacking the stable and realistic Friedmann-Lemaître-Robertson-Walker cosmological solutions was discussed in Refs. [30,31,32,33,34].

On the other hand, although the Lorentz invariance is widely regarded as a fundamental symmetry of nature, there are still some hints that the Lorentz invariance may be broken at high energies in quantum gravity [35, 36]. Especially, the aforementioned standard problems of Lorentz-invariant massive gravity theories may be avoided by switching to the Lorentz-violating massive gravity theory, see Refs. [37,38,39,40,41] for examples. A simple example of Lorentz-violating massive gravities was proposed by Lin in Ref. [42], where the theory is built with the Einstein–Hilbert action plus 3 canonical massless scalar fields. The spatial condensation of the scalar fields breaks spatial diffeomorphisms, which gives rise to 3 Goldstone excitations. After “eating” the 3 Goldstone bosons in the unitary gauge, the graviton gets weight and possesses 5 well-behaved DOF. It was found that the theory could remove the infrared divergence of the inflationary loop diagram owing to the graviton mass [42]. For a recent review on massive gravity theories see Refs. [43,44,45] and the references therein.

The higher-dimensional scenario has a close relation with massive gravities, as it naturally incorporates a tower of non-pathological massive spin-2 gravitons in the 4D graviton spectrum. Nevertheless, they are usually not a purely massive gravity theories due to the presence of a massless graviton associated with the unbroken 4D diffeomorphisms. A more interesting theory is the Dvali-Gabadadze-Porrati model, whose graviton looks like a resonance and effectively acquires a soft mass [46, 47]. In Ref. [48], the authors considered the RS1 model with a localized Fierz–Pauli mass term for the graviton on the IR brane, which reproduces GR for observers localized on the UV brane at distances smaller than the IR scale but realizes a massive gravity at longer distance with a ghost radion. The embedding of a 4D dRGT massive gravity into a 5D dRGT massive gravity by resorting to the single-brane RS2 model was studied by Gabadadze in Ref. [49], where the value of strong scale \(\varLambda _3\) can be significantly increased. More relevant works see Refs. [50,51,52,53] for examples.

In this work, we are interested in generating the massive gravity with spatial condensation into a 5D spacetime and studying the braneworld scenario in the massive gravity. The background spacetime in the spatial condensation scenario exhibits invariance under translation and 3D rotations. In order to achieve a Minkowski flat brane model, we require that the background spacetime is invariant under the 4D Poincaré transformation in the higher dimensional generalization. The condensation of 4 background scalars spontaneously breaks the 4D diffeomorphisms, and generates 4 Goldstone excitations associated with the broken symmetries. Consequently, by “eating” the 4 Goldstone excitations in the unitary gauge, the 5D massless spin-2 graviton with 5 DOF becomes massive and possesses 9 DOF on the spectrum.

The layout of the paper is as follows: In Sect. 2, the toy model is built in the 5D massive theory. In Sect. 3, the full linear perturbations have been considered. In Sect. 4, the mass spectra of KK particles have been calculated and the resolution of the gauge hierarchy problem has been discussed as an application of the model. In Sect. 5, the Goldberger-Wise (GW) mechanism has been used to stabilize the radius of the extra dimension. Finally, brief conclusions are presented. Throughout the paper, the small Latin letters \(a, b, \cdots =0,1,2,3,\) label the group indices of the internal metric of scalar fields, and the capital Latin letters \(A, B, \cdots =0,1,2,3,5,\) and Greek letters \(\mu ,\nu , \cdots =0,1,2,3,\) label the 5D and 4D spacetime indices, respectively.

2 Model building

We start from a 5D extension of the Lorentz-violating massive gravity [42], and the action is given by a 5D Einstein-Hilbert term plus 4 canonical massless scalar fields, i.e.,

$$\begin{aligned} S= & {} M^3\int {d^5x}\sqrt{-g}\left[ \frac{R}{2}-\frac{1}{2}m^2g^{MN} \partial _M\phi ^a\partial _N\phi ^b\eta _{ab} -\varLambda \right] \nonumber \\{} & {} -\int d^4 x \sqrt{-g_{\textrm{I}}} V_{\textrm{I}}-\int d^4 x \sqrt{-g_{\textrm{II}}} V_{\textrm{II}}, \end{aligned}$$
(1)

where M is the 5D fundamental gravity scale, \(\varLambda \) is the 5D cosmological constant, \(V_{I}\) and \(V_{II}\) represent the brane tensions. The parameter m is proportional to the mass of the 5D graviton, and this relationship will become evident in Sect. 4. Here, the internal metric \(\eta _{ab}\) for the scalar fields is selected to be the Minkowski metric. This choice guarantees the preservation of 4D Poincaré symmetry, allowing for the attainment of a Minkowski flat brane model. However, it is evident that the utilization of the Minkowski metric results in one of the bulk scalars, namely \(\phi ^0\), having a kinetic term with the wrong sign. Consequently, this gives rise to a ghost scalar mode in the linear perturbation theory, which will be clearly observed in Sect. 3.

The background solution spontaneously breaks the 5D Lorentz invariance when the scalar fields are fixed to their vacuum values

$$\begin{aligned} \langle \phi ^a\rangle =\delta _\mu ^a x^\mu , \end{aligned}$$
(2)

with \(x^\mu \) the coordinates of 4D spacetime. Thus, the condensation spontaneously generates a preferred 4D frame.

In order to investigate a RS1 model-like braneworld solution, the metric ansatz preserving the 4D Poincaré invariance reads

$$\begin{aligned} ds^{2}_{5}=a^2(y){\eta }_{\mu \nu }dx^{\mu }dx^{\nu }+dy^2, \end{aligned}$$
(3)

where a(y) is the warp factor and \(y \in [-y_\pi , y_\pi ]\) denotes an \(S^1/Z_2\) orbifold extra dimension. The UV brane with tension \(V_{\textrm{I}}\) locates at \(y=0\) and the IR brane with tension \(V_{\textrm{II}}\) locates at \(y=y_\pi \).

The field equations are obtained by varying the action (1) with respect to the metric \(g^{MN}\),

$$\begin{aligned} G_{MN}= & {} m^2 \Big (\partial _M \phi ^a \partial _N \phi ^a -\frac{1}{2} g_{MN} \partial ^K \phi ^a \partial _K \phi ^a\Big ) -\varLambda g_{MN}\nonumber \\{} & {} -\frac{\sqrt{-g_{\textrm{I}}} }{\sqrt{-g}}\frac{V_{\textrm{I}}}{M^3 } g_{\textrm{I}\mu \nu } \delta ^{\mu }_M \delta ^{\nu }_N \delta \left( y\right) \nonumber \\{} & {} -\frac{\sqrt{-g_{\textrm{II}}} }{ \sqrt{-g}} \frac{V_{\textrm{II}}}{M^3}g_{\textrm{II}\mu \nu } \delta ^{\mu }_M \delta ^{\nu }_N \delta \left( y-y_\pi \right) . \end{aligned}$$
(4)

Further, with the ansatzes of braneworld metric and scalar field condensation, the field equations are written explicitly as

$$\begin{aligned} 3\left( H'+2H^2 \right)= & {} -\frac{m^2}{a^2}-\varLambda -\frac{ V_{\textrm{I}} \delta \left( y\right) + V_{\textrm{II}} \delta \left( y-y_{\textrm{b}}\right) }{M^3},\nonumber \\ \end{aligned}$$
(5)
$$\begin{aligned} 6H^2= & {} -\frac{2m^2}{a^2}-\varLambda , \end{aligned}$$
(6)

where \(H\equiv a'/a\) and the prime denotes the derivative with respect to the extra dimension coordinate y.

The field equations simply give the following solution

$$\begin{aligned} a(y)=e^{-k|y|}+\epsilon ^2e^{k|y|}, \end{aligned}$$
(7)

where \(\epsilon \equiv \frac{m}{2\sqrt{3}k}\), \(k^2\equiv -\varLambda /6\), and |y| represents the absolute value of y in order to be consistent with the \(Z_2\) symmetry. Since the minimum value of warp factor is \(2\epsilon \), the 5D graviton has to be light enough, i.e., \(\epsilon <a(y_\pi )\sim 10^{-16}\), to generate enough redshift for the fundamental electroweak scale. As illustrated in Fig. 1, the warp factor is different from that of RS model only far from the center for a tiny graviton mass, \(m/k\ll 1\).

Fig. 1
figure 1

The profiles of the warp factor a(y) with respect to different mass parameters

Full size image

After matching the delta functions in (5), we have the fine-tuning conditions,

$$\begin{aligned} V_I= & {} 6 k M^3 \left[ 1-\frac{2 \epsilon ^2}{1+\epsilon ^2}\right] , \end{aligned}$$
(8)
$$\begin{aligned} V_{II}= & {} 6 k M^3 \left[ 1-\frac{2}{1+\epsilon ^2 e^{2 k y_\pi }}\right] . \end{aligned}$$
(9)

In the case of \(e^{-k y_\pi }>\epsilon \), the exponential decay term of warp factor (7) is dominant in the whole bulk. It is clear that \(V_I>0\) and \(V_{II}<0\) in this case, namely, there is a positive tension brane located at the origin and a negative tension brane at \(y_\pi \). This is the model we will focus on in the rest of the paper. Especially, in the limit \(\epsilon e^{k y_\pi }\ll 1\), the warp factor (7) approaches the exponential decay form of RS model. Correspondingly, the fine-turning conditions reduce to the case of RS model as well,

$$\begin{aligned} V_I\approx -V_{II}\approx 6 k M^3, ~~~ \varLambda =-6k^2. \end{aligned}$$
(10)

Besides, there is another interesting case that both brane tensions are positive in case of \(e^{-k|y_\pi |}<\epsilon <1\). It would generate a new brane configuration distinct from the RS1 model. Especially, the scenario that our universe is confined on the positive tension brane may potentially provide a possible model with better properties [9, 54, 55]. This model will be studied in detail in our another work.

3 Linear perturbations

In order to investigate the stability of the model, we consider the full linear perturbations against the background. The perturbed metric is written as

$$\begin{aligned} ds^2=\left( g_{MN}+h_{MN}\right) dx^Mdx^N, \end{aligned}$$
(11)

where \(h_{MN}\) represents the perturbations against the background metric \(g_{MN}\) given in Eq. (3). Due to the 4D Lorentz symmetry of our background spacetime, it is convenient to decompose the perturbed metric into the scalar, transverse vector and transverse-traceless tensor modes, and to rewrite it by

$$\begin{aligned} h_{55}&= -2\xi , \end{aligned}$$
(12)
$$\begin{aligned} h_{\mu 5}&= -a \left( S_{\mu }+\partial _\mu \beta \right) , \end{aligned}$$
(13)
$$\begin{aligned} h_{\mu \nu }&=a^2\left[ D_{\mu \nu }+2\eta _{\mu \nu }\psi +\frac{1}{2} \left( \partial _{\mu }F_{\nu }+\partial _{\nu }F_{\mu } \right) +2\partial _\mu \partial _\nu E \right] , \end{aligned}$$
(14)

where the vector modes satisfy the transverse condition \(\partial ^\mu S_{\mu }=\partial ^\mu F_{\mu }=0\), and the tensor mode satisfies the transverse-traceless (TT) condition \(\partial ^\mu D_{\mu \nu }=0\).

Correspondingly, the perturbed scalar fields are

$$\begin{aligned} \phi ^a=x^a+\pi ^a, \end{aligned}$$
(15)

where \(\pi ^a=\delta ^a_\mu \pi ^\mu \) is the Goldstone excitation of the condensation. In order to maintain the scalar condensation (2) under the general coordinate transformation \(x^M \rightarrow x^M+\epsilon ^M\), the Goldstone excitation has to transform opposite to the 4D coordinates simultaneously, i.e., \(\pi ^\mu \rightarrow \pi ^\mu -\epsilon ^\mu \). This Stückelberg trick non-linearly restores the general covariance of the theory. As a result, it behaves like a vector field and can be decomposed as \(\pi ^\mu =\eta ^{\mu \nu }\left( \partial _\nu \varphi +A_\nu \right) \), with \(\varphi \) a scalar field and \(A_\mu \) a transverse vector field.

Consequently, there are 1 TT tensor, 3 transverse vector and 5 scalar modes in total. By decomposing \(\epsilon _\mu =a^2(\epsilon ^{V}_{\mu }+\partial _\mu \epsilon ^s)\) with \(\epsilon ^{V}_{\mu }\) a transverse vector and \(\epsilon ^s\) a scalar, these 9 perturbed modes transform as follows under the general coordinate transformation,

$$\begin{aligned} D_{\mu \nu }&\rightarrow D_{\mu \nu }, \quad S_{\mu } \rightarrow S_{\mu } + a \epsilon ^{V}_{\mu }{}' ,\quad F_{\mu } \rightarrow F_{\mu }-2 \epsilon ^{V}_{\mu } ,\nonumber \\ A_{\mu }&\rightarrow A_{\mu }-{\epsilon }^{V}_{\mu }, \quad \psi \rightarrow \psi -H\epsilon _5, \quad E \rightarrow E-\epsilon ^s, \nonumber \\ \beta&\rightarrow \beta +a {\epsilon }^s{}'+\frac{\epsilon _5}{a}, \quad \xi \rightarrow \xi +\epsilon '_5, \quad \varphi \rightarrow \varphi -{\epsilon }^s. \end{aligned}$$
(16)

Naively, we have 5 gauge freedoms to eliminate one vector and two scalar modes by fixing \({\epsilon }^{V}_\mu \), \({\epsilon }^s\) and \(\epsilon _5\). A commonly used gauge choice is the so-called unitary gauge, in which the Goldstone excitations \(\pi ^a\) of scalar fields are closed, i.e., \(A_\mu =\varphi =0\). Moreover, we have another gauge freedom \(\epsilon _5\) to set \(\psi =0\). Then from the fact that \(Z_\mu -\pi _\mu \), with \(Z_\mu \equiv a^2\left( F_\mu /2+\partial _\mu E \right) \), is a gauge invariant quantity, one observes that the 4 Goldstone excitations \(\pi ^\mu \) are “eaten” by the 5D graviton in the unitary gauge. Consequently, the 5D massless spin-2 graviton with 5 DOF gets weight and becomes massive, with 9 DOF on the spectrum. After gauge fixing, we have 1 TT tensor, 2 transverse vector and 3 scalar modes left. However, not all modes among them are physical, since some of them can be eliminated by some constraint equations, which can be easily extracted by utilizing the Arnowitt–Deser–Misner decomposition.

The Arnowitt–Deser–Misner decomposition of the 5-dimensional spacetime metric reads [56]

$$\begin{aligned} ds^2=N^2dy^2+\gamma _{\mu \nu } \left( dx^{\mu }+N^{\mu }dy\right) \left( dx^{\nu }+N^{\nu }dy \right) , \end{aligned}$$
(17)

where N is the lapse, \(N^\mu \) the shift vector, and \(\gamma _{\mu \nu }\) the induced metric on the slices. Correspondingly, the bulk action of (1) is rewritten as

$$\begin{aligned} S= & {} \frac{M^3}{2}\int d^5x\sqrt{-\gamma }\Big [N \left( R^{(4)}-2 \varLambda -m^2 \gamma ^{\mu \nu } \partial _\mu \phi ^a \partial _\nu \phi ^a \right) \nonumber \\{} & {} -m^2 N^{-1}\left( \phi ^{a}{}'-N^{\mu }\partial _{\mu }\phi ^a\right) \left( \phi ^{a}{}'-N^{\nu }\partial _{\nu }\phi ^a\right) \nonumber \\{} & {} -N^{-1}\left( E_{\mu \nu } E^{\mu \nu }-E^2\right) \Big ], \end{aligned}$$
(18)

where \(E_{\mu \nu }=\frac{1}{2} \left( \gamma '_{\mu \nu }-\nabla _{\nu }N_{\mu }-\nabla _{\mu }N_{\nu }\right) \). The constraint equations are yielded by varying with respect to the lapse function N and shift vector \(N_\mu \), i.e.,

$$\begin{aligned}{} & {} R^{(4)}-2 \varLambda -m^2 \gamma ^{\mu \nu } \partial _\mu \phi ^a \partial _\nu \phi ^a +N^{-2}\big (E_{\mu \nu } E^{\mu \nu }-E^2\big )\nonumber \\{} & {} \quad +m^2N^{-2}\left( \phi ^{a}{}'-N^{\mu } \partial _{\mu }\phi ^a\right) \left( \phi ^{a}{}' -N^{\nu }\partial _{\nu }\phi ^a\right) = 0, \end{aligned}$$
(19)
$$\begin{aligned}{} & {} \nabla _{\nu }\left( \frac{E^{\mu }_{\nu }-\delta ^{\mu }_{\nu }E}{N}\right) -\frac{m^2}{N}\partial _{\mu }\phi ^a\left( \phi ^a{}'-N^{\alpha } \partial _\alpha \phi ^a\right) =0. \end{aligned}$$
(20)

Further, by considering small perturbations of the metric functions, expressed as \(N=\bar{N}+\delta N\), \(N^\mu =\bar{N}^\mu +\delta N^\mu \), and \(\gamma _{\mu \nu }=\bar{\gamma } _{\mu \nu }+\delta \gamma _{\mu \nu }\), where \(\bar{N}\), \(\bar{N}^\mu \), and \(\bar{\gamma }_{\mu \nu }\) represent the background quantities with \(\bar{N}=1\), \(\bar{N}^\mu =0\), and \(\bar{\gamma }_{\mu \nu }=a^2(y)\eta _{\mu \nu }\), respectively. The perturbations \(\delta N\), \(\delta N^\mu \), and \(\delta \gamma _{\mu \nu }\) can be explicitly extracted from Eqs. (12), (13), and (14). Consequently, one can obtain a constraint equation for vector modes and two constraint equations for scalar modes. Thus, one vector mode and two scalar modes can be worked out algebraically, leaving us one vector and one scalar modes eventually.

After scalar-vector-tensor decomposition and expanding the action (1) to the quadratic order of perturbations, the TT tensor, transverse vector and scalar modes are decoupled with each other, so they can be treated separately.

3.1 Tensor mode

By including only the tensor perturbation in the perturbed metric and dropping the boundary terms, the quadratic action for the tensor perturbation reads

$$\begin{aligned} S^{(2)}_T= & {} \frac{M^3}{8}\int d^4xdya^2\nonumber \\{} & {} \times \left[ -{a^{2}} D_{ \alpha \beta }' D{}'^{\alpha \beta } -{\partial _\lambda D_{\alpha \beta }}\partial ^{\lambda } D ^{\alpha \beta } 2 m^2 D_{\alpha \beta } D^{\alpha \beta } \right] ,\nonumber \\ \end{aligned}$$
(21)

where the indices are raised and lowered by the 4D Minkowski metric \(\eta _{\mu \nu }\). Further, after a coordinate transformation \(dy=adz\) that turns the background metric into a conformal form and a rescaling \(\tilde{D }={D }/{2}\), a canonical form is obtained as

$$\begin{aligned} S^{(2)}_T= & {} \frac{M^3}{2}\int {d^4x dz} a^3\nonumber \\{} & {} \times \left[ -\dot{\tilde{D }}_{\alpha \beta }\dot{\tilde{D }}^{\alpha \beta } -\partial _\lambda \tilde{D}_{\alpha \beta } \partial ^{\lambda } \tilde{D} ^{\alpha \beta } 2 m^2\tilde{D}_{\alpha \beta } \tilde{D}^{\alpha \beta } \right] , \end{aligned}$$
(22)

where the dot denotes the derivative with respect to the extra dimension coordinate z. The tensor mode is free from the ghost instability due to the correct sign of kinetic term. Moreover, the action is expressed more concisely in the momentum space, where the d’Alembert operator \(\partial ^{\alpha }\partial _{\alpha } \) is replaced by \( -p^2\equiv -p^\alpha p_\alpha \), with \(p^\alpha \) the four-momentum of tensor mode. Then, it yields

$$\begin{aligned} S^{(2)}_T=\frac{M^3}{2}\int d^4 {p} dz a^3 \left[ -\dot{\tilde{D }}_{\alpha \beta } \dot{\tilde{D }}^{\alpha \beta }-\left( {{p}^2} +{2 m^2}\right) \tilde{D}_{\alpha \beta } \tilde{D}^{\alpha \beta } \right] .\nonumber \\ \end{aligned}$$
(23)

By variation with respect to \(\tilde{D }^{\alpha \beta }\), we have the equation of motion

$$\begin{aligned} \ddot{\tilde{D} }_{\alpha \beta }+3H\dot{\tilde{D} }_{\alpha \beta } =({p}^2+2m^2){\tilde{D}_{\alpha \beta } }. \end{aligned}$$
(24)

Note that here and in what follows, \(H \equiv \dot{a}/a\) refers to the extra dimension coordinate z. After redefining \(\tilde{D}_{\alpha \beta }=a^{-\frac{3}{2}} {\mathcal {D}}_{\alpha \beta }\), a Schrödinger-like equation is obtained,

$$\begin{aligned} -\ddot{{\mathcal {D}} }_{\alpha \beta } +\left( \frac{3}{2} \dot{H}+\frac{9}{4}H^2\right) {{\mathcal {D}} }_{\alpha \beta } =M_T^2{{\mathcal {D}}}_{\alpha \beta } , \end{aligned}$$
(25)

where \(M_T^2 \equiv -{p}^2-2m^2\). The Hamiltonian can be further factorized into a supersymmetric quantum mechanics form, with \(H_T=A_T^\dag A_T=\left( \partial _z+\frac{3}{2}H\right) \left( -\partial _z+\frac{3}{2}H\right) \). Then, with the boundary condition \(\partial _z \tilde{D}_{ \alpha \beta } |_{z=0,z_b} =0\), the self-adjoint Hamiltonian gives non-negative eigenvalues [57], i.e., \(M_T^2\ge 0\). Consequently, with a positive 5D graviton mass m, the four-momentum of tensor mode \({p}^2=-M_T^2-2m^2<0\), i.e., the tensor excitations are all time-like particles. Thus, the model is also irrelevant to tachyonic instability. If m is set to zero, \(M_T^2=-{p}^2\ge 0\) gives us the well-known result that there is a massless graviton and a tower of massive gravitons in RS1 model.

3.2 Vector modes

By including only the vector perturbations in the perturbed metric and dropping the boundary terms, the quadratic action for vector perturbations reads

$$\begin{aligned} S_{V}^{(2)}= & {} \frac{M^3}{16}\int d^4xdy a^2 \Big [-a^2\partial _\beta F_\alpha {}'\partial ^\beta F^\alpha {}' -2m^2\partial _\beta F_\alpha \partial ^\beta F^\alpha \nonumber \\{} & {} -8a^2m^2 A_\alpha {}' A^\alpha {}'-8m^2 \partial _\beta A_\alpha \partial ^\beta A^\alpha -4\partial _{\beta }S_{\alpha } \partial ^\beta S^{\alpha } \nonumber \\{} & {} -8m^2S_\alpha S^\alpha +8m^2 \partial _\beta F_\alpha \partial ^\beta A^\alpha -4a\partial _\beta F_\alpha {}' \partial ^\beta S^\alpha \nonumber \\{} & {} -16 a m^2 S_\alpha A^\alpha {}'\Big ]. \end{aligned}$$
(26)

The constraint equation can be obtained by counting the first order perturbations of Eq. (20), or simplify by varying the above quadratic action with respect to \(S^\alpha \), i.e.,

$$\begin{aligned} 2\partial _{\beta }{\partial ^\beta S_{\alpha }}-4 m^2 S_{\alpha }+a \partial _{\beta }{\partial ^\beta F_{\alpha }'}-4am^2A_\alpha '=0. \end{aligned}$$
(27)

Then, \(S^{\alpha }\) can be worked out in momentum space, namely,

$$\begin{aligned} S_{\alpha }=-\frac{a {p}^2 F_{\alpha }' +4 a m^2 A_\alpha '}{2 ({p}^2+2 m^2)}. \end{aligned}$$
(28)

Working in the unitary gauge, \(A_\alpha \) is gauged away. Then, substituting the relation into the action (26) yields

$$\begin{aligned} S_V^{(2)}= M^3\int d^4 {p} dy \left[ -\frac{a^4 {p}^2 m^2 F_{\alpha }{}' F^{\alpha }{}'}{8 \left( {p}^2+2 m^2\right) }-\frac{a^2 {p}^2 m^2}{8} F_{\alpha } F^{\alpha } \right] . \end{aligned}$$
(29)

After performing a coordinate transformation into the z coordinate, a canonical normalized form can be obtained by redefining the vector perturbation as

$$\begin{aligned} \tilde{F}_{\alpha }=\sqrt{\frac{{p}^2 m^2}{{p}^2+2 m^2}} \frac{F_{\alpha }}{2}, \end{aligned}$$
(30)

yielding,

$$\begin{aligned} S^{(2)}_V=\frac{M^3}{2}\int d^4 {p} dz a^3 \left[ -\dot{\tilde{F }}_{\alpha } \dot{\tilde{F }}^{\alpha } -\left( {{p}^2}+{2 m^2}\right) \tilde{F}_{\alpha } \tilde{F}^{\alpha }\right] . \end{aligned}$$
(31)

The correct sign of the kinetic term ensures that the vector perturbation is free from the ghost instability. Then the equation of motion reads

$$\begin{aligned} \ddot{\tilde{F} }_{\alpha }+3H\dot{\tilde{F} }_{\alpha } =({p}^2+2m^2){\tilde{F}_{\alpha } }. \end{aligned}$$
(32)

A Schrödinger-like equation is given by redefining \(\tilde{F}_{\alpha }\rightarrow a^{-\frac{3}{2}} {\mathcal {F}}_{\alpha }\),

$$\begin{aligned} -\ddot{{\mathcal {F}}}_\alpha +\left( \frac{3}{2}\dot{H} +\frac{9}{4}H^2\right) {\mathcal {F}}_\alpha =M_V^2{\mathcal {F}}_\alpha , \end{aligned}$$
(33)

where \(M_V^2 \equiv -{p}^2-2m^2\). The Hamiltonian can also be factorized into a supersymmetric quantum mechanics form, with \(H_V=A_V^\dag A_V=\left( \partial _z+\frac{3}{2}H\right) \left( -\partial _z+\frac{3}{2}H\right) \). With the boundary condition \(\partial _z \tilde{F }_{\alpha } |_{z=0,z_b} =0\), the eigenvalues are non-negative \(M_V^2\ge 0\), i.e., \({p}^2 \le -2m^2\). Thus, the vector excitations are also time-like particles and irrelevant to tachyonic instabilities.

For the cases of \({p}^2 <-2m^2\), the formula (28) and the redefinition (30) are valid. However, for the case of \(M_V^2=-{p}_0^2-2m^2=0\), the formula (28) is problematic, and the constraint equation (27) leads to

$$\begin{aligned} F_\alpha '=2A_\alpha '. \end{aligned}$$
(34)

This implies that \(F_\alpha -2A_\alpha \equiv f_\alpha (x)\) is a purely 4D field. Especially, the field \(f_\alpha (x)\) is a gauge invariant quantity under the general coordinate transformation by the observation from (16). Then, the quadratic action (26) reduces to

$$\begin{aligned} S^{(2)}_{V0}= & {} -\frac{M^3}{8}\int {d^4 p}dz a^3 m^2 k_0^2 \left( F_{\alpha }-2 A_{\alpha }\right) \left( F^{\alpha } -2 A^{\alpha }\right) \nonumber \\= & {} -\frac{M^3}{8}\int {d^4 p}dz a^3 m^2 k_0^2 f_\alpha f^\alpha . \end{aligned}$$
(35)

The equation of motion reads \({p}_0^2f_\alpha =0\). Since \({p}_0^2=-2m^2\), it leads to \(f_\alpha =0\). This implies that the lightest vector mode does not exist in the mass spectrum. This is curial to recover the mass spectrum of RS1 model in the massless limit \(m=0\), where no massless vector mode exists due to the lack of continuous isometries of the higher dimension in the presence of 3-branes [4]. Moreover, all the massive vector modes are gauge dependent in the massless limit \(m=0\), therefore, they can be gauged away by gauge fixing.

3.3 Scalar modes

By including only the scalar perturbations in the perturbed metric and dropping the boundary terms, the quadratic action for scalar perturbations reads

$$\begin{aligned} S^{(2)}_{S}= & {} M^3\int d^4xdy a^2\nonumber \\{} & {} \times \Bigg (6 a^2 H^2 \xi ^2+4 m^2 \psi ^2 - 4 m^2 \xi \psi + 6 a^2 \psi '{}^2\nonumber \\{} & {} +12 a^2 H \xi \psi ' +3\partial _\alpha \psi \partial ^\alpha \psi - \frac{m^2}{2} \partial _\alpha \beta \partial ^\alpha \beta +3\xi \partial _\alpha \partial ^\alpha \psi \nonumber \\{} & {} +3 aH\xi \partial _\alpha \partial ^\alpha \beta + m^2 \xi \partial _\alpha \partial ^\alpha \varphi -m^2 \xi \partial _\alpha \partial ^\alpha E \nonumber \\{} & {} - 2m^2 \psi \partial _\alpha \partial ^\alpha \varphi +2m^2\psi \partial _\alpha \partial ^\alpha E +3 a^2 H \xi \partial _\alpha \partial ^\alpha E' \nonumber \\{} & {} +3 a\beta \partial _\alpha \partial ^\alpha \psi ' + m^2 a \beta \partial _\alpha \partial ^\alpha \varphi ' -\frac{m^2}{2} a^2 \partial _\alpha \varphi ' \partial ^\alpha \varphi '\nonumber \\{} & {} +3a^2 \psi '\partial _\alpha \partial ^\alpha E' -\frac{m^2}{2} \partial _\alpha \partial ^\alpha E \partial _\lambda \partial ^\lambda E \nonumber \\{} & {} -\frac{m^2 }{2} \partial _\alpha \partial ^\alpha \varphi \partial _\lambda \partial ^\lambda \varphi +m^2 \partial _\alpha \partial ^\alpha \varphi \partial _\lambda \partial ^\lambda E \Big ). \end{aligned}$$
(36)

From the first order perturbation of Eqs. (19) and (20) or simply varying the above action respect to the modes \(\beta \) and \(\xi \) respectively, the constraint equations are obtained as

$$\begin{aligned}{} & {} 12 a^2 H^2 \xi -4m^2\psi +12 a^2 H \psi '+3 \partial _\alpha \partial ^\alpha \psi +3aH\partial _{\alpha } \partial ^{\alpha }\beta \nonumber \\{} & {} \quad +m^2 \partial _{\alpha }\partial ^{\alpha }\varphi -m^2\partial _{\alpha }\partial ^{\alpha }E +3a^2H \partial _{\alpha }\partial ^{\alpha }E'=0, \end{aligned}$$
(37)
$$\begin{aligned}{} & {} m^2\partial _\alpha \beta + 3aH\partial _{\alpha }\xi + 3a \partial _\alpha \psi ' +m^2 a \partial _\alpha \varphi ' =0. \end{aligned}$$
(38)

After closing the scalar perturbations \(\varphi \) and \(\psi \) in the unitary gauge, we only have to take the remaining perturbations \(\xi \), \(\beta \) and E into account in the quadratic action, i.e.,

$$\begin{aligned} S^{(2)}_{S}= & {} M^3\int d^4xdy a^2\Bigg (6 a^2 H^2 \xi ^2 -\frac{m^2}{2}\partial _\alpha \beta \partial ^{\alpha }\beta \nonumber \\{} & {} +3 a H \xi \partial _{\alpha }\partial ^{\alpha }\beta -m^2\xi \partial _{\alpha }\partial ^{\alpha }E+3a^2H\xi \partial _{\alpha }\partial ^{\alpha }E'\nonumber \\{} & {} -\frac{m^2}{2}\partial _{\alpha }\partial ^{\alpha } E\partial _{\lambda }\partial ^{\lambda }E\Bigg ). \end{aligned}$$
(39)

Correspondingly, the constraint equations are rewritten as

$$\begin{aligned}&12 a^2 H^2 \xi +3aH\partial _{\alpha }\partial ^{\alpha } \beta -m^2\partial _{\alpha }\partial ^{\alpha }E +3a^2H \partial _{\alpha }\partial ^{\alpha }E'=0, \end{aligned}$$
(40)
$$\begin{aligned}&m^2\partial _\alpha \beta +3aH\partial _{\alpha }\xi =0. \end{aligned}$$
(41)

In momentum space, \(\beta \) and \(\xi \) can be worked out from the constraint equations as

$$\begin{aligned} \beta= & {} -\frac{3 aH}{m^2}\xi , \end{aligned}$$
(42)
$$\begin{aligned} \xi= & {} \frac{{p}^2m^2(3 a^2 HE'- m^2 E)}{3 a^2 H^2 \left( 3 {p}^2+4 m^2\right) }. \end{aligned}$$
(43)

Thus, the perturbations \(\beta \) and \(\xi \) can be eliminated by substituting these relations into the action (39), then a quadratic action for scalar perturbation E is achieved as

$$\begin{aligned} S^{(2)}_{S}\supset M^3 \int d^4{p} dy \frac{3a^2{p}^4 m^2}{6 {p}^2+8 m^2} \left[ -a^2E' E' -\left( {p}^2+2 m^2\right) E^2\right] . \end{aligned}$$
(44)

Further, after redefining the scalar perturbation as

$$\begin{aligned} \tilde{E}= \sqrt{\frac{-3 {p}^4 m^2}{3 {p}^2+4 m^2}}E, \end{aligned}$$
(45)

the above action can be rewritten as a canonical normalized form in the conformal coordinate z, i.e.,

$$\begin{aligned} S^{(2)}_{S}\supset \frac{M^3}{2}\int d^4 {p} dz a^3 \left[ \dot{\tilde{E} }\dot{\tilde{E}} +\left( {{p}^2}+{2 m^2}\right) \tilde{E}^2\right] . \end{aligned}$$
(46)

It is clear that there is an overall wrong-sign in the Lagrangian, which implies that the scalar mode \(\tilde{E}\) is a ghost field. However, the equation of motion is not affected by the overall sign of the Lagrangian, so it has a similar form to the tensor and vector modes, given by

$$\begin{aligned} \ddot{\tilde{E} }+3H\dot{\tilde{E} }=({p}^2+2m^2){\tilde{E} }. \end{aligned}$$
(47)

By redefining \(\tilde{E}=a^{-\frac{3}{2}}{\varepsilon }\), it can be rewritten as a Schrödinger-like equation,

$$\begin{aligned} -\ddot{{\varepsilon }}+\left( \frac{9 }{4}H^2+\frac{3 }{2} \dot{H}\right) {\varepsilon } = M_S^2{\varepsilon }, \end{aligned}$$
(48)

where \(M_S^2\equiv -{p}^2-2m^2\). The Hamiltonian can be factorized as a supersymmetric quantum mechanics form as well, \(H_S=A_S^\dag A_S=\left( \partial _z+\frac{3}{2}H\right) \left( -\partial _z +\frac{3}{2}H\right) \), so with the boundary condition \(\partial _z \tilde{E} |_{z=0,z_b} =0\), the eigenvalues are non-negative \(M_S^2\ge 0\), i.e., \({p}^2\le -2m^2\). Thus, the redefinition (45) is well-defined. In this case, all the KK excitations of the scalar mode are time-like particles.

In RS1 model, the presence of IR brane abruptly ends AdS space, so it spontaneously breaks the conformal invariance of AdS bulk in the IR. The massless radion is just the Goldstone boson associated with the broken dilatation invariance [12, 58]. For the current model, the scalar curvature reads \(R=-20k^2+\frac{4\,m^2e^{2k y}}{1+\epsilon ^2 e^{2ky}}\) in the bulk. Thus, the presence of scalar condensation induces a slight deformation in the bulk geometry, causing it to deviate from a pure AdS space. Consequently, the radion acquires a tiny mass due to this explicit symmetry breaking. In the massless limit \(m=0\), the quadratic action (36) will reduce to that of RS model, with only a massless radion in the mass spectrum [11, 59, 60].

4 Mass spectra and hierarchy resolution

For the lightest tensor and scalar modes corresponding to \(M_{T,S}=0\), their wave functions can be easily solved from Eqs. (25) and (48). After returning to the coordinate space, and utilizing the KK decompositions, \({\mathcal {D}}_{\alpha \beta }(x,z) =d_{\alpha \beta }(x)\varPsi (z)\) and \(\varepsilon (x,z) =e(x)\varPsi (z)\), a Schrödinger-like equation

$$\begin{aligned} -\ddot{\varPsi } +\left( \frac{3}{2} \dot{H}+\frac{9}{4}H^2\right) \varPsi =M_{T,S}^2\varPsi , \end{aligned}$$
(49)

with \(M_{T,S}^2=m_{T,S}^2-2m^2\), can be achieved from Eqs. (25) and (48). Here, \(m_{T,S}\) is the effective mass of KK particles in 4D point of view, satisfying the 4D Klein–Gordon equations \(\Box ^{(4)}d_{\mu \nu }=m_T^2 d_{\mu \nu }\) and \(\Box ^{(4)}e(x)=m_S^2 e(x)\).

The wave function of ground states is easily achieved by setting \(M_{T,S}=0\) or \(m_{T,S}=\sqrt{2}m\) in above Schrödinger-like equation, i.e.,

$$\begin{aligned} \varPsi ^{(0)}(z)=N_0a(z)^{\frac{3}{2}}, \end{aligned}$$
(50)

where \(N_0\) is the normalization factor. The boundary conditions \(\partial _z \tilde{D}_{ \alpha \beta } |_{z=0,z_b} =0\) and \(\partial _z \tilde{E} |_{z=0,z_b} =0\) lead to \(\left. \left( \varPsi -\frac{3}{2}H \right) \right| _{z=0,z_b}=0\). Obviously, the wave function \(\varPsi ^{(0)}(z)\) satisfies the boundary condition. All KK particles are massive in this model, which is a significant difference from the RS1 model, where the lightest KK particles are massless spin-2 graviton and spin-0 radion. Especially, from the redefinitions \(\tilde{D}_{\alpha \beta } =a^{-\frac{3}{2}}{\mathcal {D}}_{\alpha \beta }\) and \(\tilde{E}=a^{-\frac{3}{2}}{\varepsilon }\), their canonical normalized field configurations are given by \(\tilde{D}^{(0)}_{\alpha \beta }=a^{-\frac{3}{2}} {\mathcal {D}}^{(0)}_{\alpha \beta }=d^{(0)}_{\alpha \beta }(x)\) and \(\tilde{E}^{(0)}=a^{-\frac{3}{2}} {\varepsilon }^{(0)} =e^{(0)}(x)\). Therefore, the lightest graviton and radion propagate only on the brane.

However, the effective mass of 4D graviton is severely constrained by the gravitational experiments [45]. For instance, the detection of gravitational waves has provided a bound of the graviton mass, yielding \(m_g\le 4.7\times 10^{-23}\)eV [61]. Thus, the parameter m must be tinier than the experimental constraints, i.e., \(m<3.3\times 10^{-23}\)eV. Since the lightweight radion has the same mass as the lightest graviton, exchanging such nearly massless scalar particle would cause a fifth force and violate experimental observations. Therefore, similar to the RS1 model, the radion must gain weight to meet the experimental expectations, which is realized through GW mechanism [11] in the next section.

With the normalization condition \(\int ^{z_b}_{-z_b}\varPsi _0^2dz=1\), the normalization factor is worked out as

$$\begin{aligned} N_0^{-2}=\frac{1}{k}\left[ 1-e^{-2 k y_\pi }+4\epsilon ^2 k y_\pi -\epsilon ^4(1-e^{2 k y_\pi })\right] . \end{aligned}$$
(51)

By shutting down the 5D graviton mass m, one recovers the result of RS1 model. However, if we remove the visible brane, i.e., \(y_\pi \rightarrow \infty \), the quasi-massless graviton is no longer normalizable. This means that the effective 4D gravity theory cannot be recovered on the brane. Therefore, the RS2-like non-compact single brane model [62] is not a physically available one in current massive gravity.

The braneworld scenario provides a natural way to solve the gauge hierarchy problem, which is a crucial motivation of the well-known ADD model [63] and RS1 model [4]. In our toy model, the low-energy effective theory is obtained by including the nearly-massless gravitons, i.e.,

$$\begin{aligned} ds^2= & {} a^2(y)\bar{g}_{\mu \nu }(x)dx^\mu dx^\nu +dy^2\nonumber \\= & {} a^2(y)\left[ \eta _{\mu \nu }+\gamma _{\mu \nu }(x)\right] dx^\mu dx^\nu +dy^2, \end{aligned}$$
(52)

then the 4D effective gravitational mass scale \(M_{\textrm{eff}}\) is read from the curvature term in the action (1),

$$\begin{aligned} M_{\textrm{eff}}^2=M^3\int ^{y_\pi }_{-y_\pi } a^2 dy={N_0^{-2}}{M^3}. \end{aligned}$$
(53)

On the other hand, in order to produce a large hierarchy between the Planck scale and the electroweak scale, our Universe should be confined to the IR brane located at \(y_\pi \). Then, the Higgs field action on the brane is non-canonically normalized,

$$\begin{aligned} S_{\mathcal {H}}\supset \int d^4x \sqrt{| g_{\textrm{II}}|} \left[ -{g}_{\textrm{II}}^{\mu \nu }D_{\mu }{\mathcal {H}}^{\dag } D_{\nu }{\mathcal {H}}-\lambda ({\mathcal {H}}^{\dag } {\mathcal {H}}-v_0^2)^2 \right] , \end{aligned}$$
(54)

where \(g_{\textrm{II}\mu \nu }=a(y_\pi )\tilde{g}_{\mu \nu }\) is the induced metric on the brane at \(y_\pi \), and \(v_0\) the fundamental Higgs vacuum expectation value (VEV). Writting the warp factor explicitly, one has

$$\begin{aligned} S_{\mathcal {H}}&\supset \int d^4x \sqrt{|\tilde{g}|} \left[ -a(y_\pi )^{2}\tilde{g}^{\mu \nu }D_{\mu }{\mathcal {H}}^{\dag } D_{\nu }{\mathcal {H}}\right. \nonumber \\&\quad \left. -a(y_\pi )^4\lambda ({\mathcal {H}}^{\dag } {\mathcal {H}}-v_0^2)^2\right] , \end{aligned}$$
(55)

With a field renormalization, \({\mathcal {H}} \rightarrow \tilde{{\mathcal {H}}}/a(y_\pi )\), the effective action of Higgs on the brane is

$$\begin{aligned} S_{\mathcal {H}}\supset \int { d^4x \sqrt{|\tilde{g}|} \left[ -\tilde{g}^{\mu \nu }D_{\mu }\tilde{{\mathcal {H}}}^{\dag } D_{\nu }{\mathcal {H}}-\lambda (\tilde{{\mathcal {H}}}^{\dag } {\mathcal {H}}-v_{\textrm{eff}}^2)^2 \right] }, \end{aligned}$$
(56)

where the effective Higgs VEV, \(v_{\textrm{eff}}=a(y_\pi ) v_0\), sets the electroweak scale on the brane.

Therefore, in order to solve the gauge hierarchy problem, the fundamental parameters M, k, \(v_0\) are simply chosen to be of the same order as the Planck scale, \(M_{\textrm{Pl}}\). Then, the warp factor has to provide enough redshift to recover a TeV scale of Higgs VEV on the brane, namely, \(a(y_\pi )\sim 10^{-16}\). In the condition \(e^{-k |y_\pi |}>\epsilon \) and \(m<3.3\times 10^{-23}\)eV, it leads to \(e^{-ky_\pi } \approx a(y_\pi ) -\frac{\epsilon ^2}{a(y_\pi )} \sim 10^{-16}\). Thus, it requires \(y_\pi \approx 37/k\).

Since the mass splitting scale of massive KK modes is inversely proportional to the conformal size of extra dimension \(z_b\), i.e., \(\varDelta m_T \propto 1/z_b\). From the coordinate transformation \(dy=adz\), one has

$$\begin{aligned} z_\pi =\frac{1}{k \epsilon } \left[ \text {arctan} \left( \epsilon e^{k y_\pi }\right) -\text {arctan} \left( \epsilon \right) \right] , \end{aligned}$$
(57)

where the integral constant has been chosen so that \(z(y=0)=0\). Since \(e^{-ky_\pi } \sim 10^{-16}\), \(m<3.3\times 10^{-23}\)eV and \(k\sim M_{\textrm{Pl}}\), one has \(\epsilon e^{k y_\pi } \ll 1\). Thus, the formula can be rewritten approximately as \(z_\pi \approx \frac{e^{ky_\pi }}{k}\left( 1+{\mathcal {O}}(\epsilon ^2)\right) \). So the mass splitting scale is similar to the RS1 model, i.e., \(\varDelta m_T \sim k e^{-ky_\pi } \sim {\mathcal {O}}(\text {TeV})\).

5 Radius stabilization

In RS1 model, the position of IR brane is put by hand, namely, the radius is not dynamically fixed. It leads to the presence of a massless radion in the effective theory, which corresponds to the fluctuations of the radius of compact extra dimensions. However, the massless radion is phenomenologically unacceptable since it would contribute to Newton’s law and cause a fifth force. It is well-known that the GW mechanism [11] can be introduced to stabilize the size of the extra dimension and to increase the mass of radion. In current model, the position of IR brane is put by hand as well, we can also utilize the GW mechanism here to stabilize the brane position and increase the radion mass.

By adding a bulk scalar field \(\varPhi \) into the model, which has interaction with the two branes, the action is given by

$$\begin{aligned} S_\varPhi= & {} \frac{1}{2}\int d^4x\int ^\pi _{-\pi }d\theta \sqrt{-g} \left( -g^{MN}\partial _M\varPhi \partial _N\varPhi -m_\varPhi ^2\varPhi ^2 \right) \nonumber \\{} & {} -\int {}d^4x\sqrt{-g_{\textrm{I}}}\lambda _{\textrm{I}} (\varPhi ^2-v_{\textrm{I}}^2)^2\nonumber \\{} & {} -\int {}d^4x\sqrt{-g_{\textrm{II}}}\lambda _{\textrm{II}} (\varPhi ^2-v_{\textrm{II}}^2)^2, \end{aligned}$$
(58)

where \(g_{MN}\) is the background metric (3) with the radius \(r_c\) of compactified extra dimension given by \(y=r_c \theta \), \(\lambda _{\mathrm{I/II}}\) is the coupling parameters, and \(V_{\mathrm{I/II}}=\lambda _{\mathrm{I/II}} v_{\mathrm{I/II}}^4\).

The equation of motion of the scalar filed \(\varPhi \) is achieved by varying the action (58) with respect to \(\varPhi \),

$$\begin{aligned}{} & {} \frac{1}{r_c}{\partial _\theta }\left( a^4 {\partial _\theta \varPhi }\right) -a^4 m_{\varPhi }^2\varPhi -{4 a^4 \lambda _{\textrm{I}} \varPhi \left( \varPhi ^2-v_{\textrm{I}}^2\right) }\delta (\theta )\nonumber \\{} & {} \quad -4 a^4\lambda _{\textrm{II}} \left( \varPhi ^2-v_{\textrm{II}}^2\right) \varPhi \delta (\theta -\pi ) =0. \end{aligned}$$
(59)

Away from the two branes at \(\theta =0,\pi \), the general solution of this equation reads

$$\begin{aligned} \varPhi (\theta )= & {} e^{2 \sigma } \left[ A {}_2\text {F}_1 \left( 2,\nu +2,\nu +1, -\epsilon ^2e^{2 \sigma } \right) e^{ \nu \sigma }\right. \nonumber \\{} & {} \left. +B {}_2\text {F}_1\left( 2,2-\nu ,1-\nu ,-\epsilon ^2 e^{2 \sigma } \right) e^{-\nu \sigma }\right] , \end{aligned}$$
(60)

where \(\sigma (\theta )=k r_c \theta \), \(\nu =\sqrt{4+m_\varPhi ^2/k^2}\), \({}_2\text {F}_1\) is the hypergeometric function, A and B are integration constants. Especially, under the condition \(\epsilon e^{\sigma } \ll 1\), up to the first order of correction, the solution can be approximately written as

$$\begin{aligned} \varPhi (\theta )\simeq & {} e^{2 \sigma } \left[ A \left( 1-2 \epsilon ^2\frac{\nu +2}{\nu +1}e^{2 \sigma } \right) e^{ \nu \sigma }\right. \nonumber \\{} & {} \left. +B \left( 1-2\epsilon ^2\frac{\nu -2}{\nu -1} e^{2 \sigma }\right) e^{ -\nu \sigma }\right] . \end{aligned}$$
(61)

If we close the mass of 5D graviton, i.e., \(\epsilon =0\), the solution reduces to the one in GR [11]. The boundary conditions on the branes can be obtained by inserting the approximate solution into the equation of motion (59) and matching the delta functions, given by

$$\begin{aligned}{} & {} k \left[ (\nu +2) \left( 1+2\epsilon ^2\frac{\nu -2}{\nu +1}\right) A -(\nu -2 ) \left( 1+2\epsilon ^2\frac{\nu +2}{\nu -1}\right) B\right] \nonumber \\{} & {} \quad -2 \lambda _{\textrm{I}} \left( 1+4 \epsilon ^2\right) \varPhi (0)\left( \varPhi (0)^2 -v_{\textrm{I}}^2\right) =0, \end{aligned}$$
(62)
$$\begin{aligned}{} & {} k e^{2 \sigma (\pi ) } \left[ (\nu +2) e^{\nu \sigma (\pi )} \left( 1+2\epsilon ^2\frac{\nu -2}{\nu +1} e^{2 \sigma (\pi )}\right) A \right. \nonumber \\{} & {} \quad \left. -(\nu -2) e^{ - \nu \sigma (\pi ) } \left( 1+2\epsilon ^2\frac{\nu +2}{\nu -1} e^{2 \sigma (\pi ) }\right) B \right] \nonumber \\{} & {} \quad +2 \lambda _v \left( 1+4 \epsilon ^2 e^{2 \sigma (\pi )}\right) \varPhi (\pi ) \left( \varPhi (\pi )^2 -v_v^2\right) =0. \end{aligned}$$
(63)

The integration constants A and B can be determined from these boundary conditions.

However, rather than solving the above equations directly, we employ the trick of Ref. [11] to simplify the calculation. By substituting the approximate solution back into the action and integrating over \(\theta \), it yields the effective potential of compactified radius \(r_c\),

$$\begin{aligned} V_\varPhi (r_c)\approx & {} k A^2 (\nu +2) e^{2 \nu k r_c \pi } \left( 1-\frac{8 \epsilon ^2 e^{2 \pi k r_c}}{\nu +1}\right) \nonumber \\{} & {} + e^{-4 k r_c \phi }\lambda _{\textrm{II}} \left( 1+4 \epsilon ^2 e^{2 \pi k r_c}\right) \left( \varPhi (\pi )^2 -v_{\textrm{II}}^2\right) ^2\nonumber \\{} & {} +\lambda _{\textrm{I}} \left( \varPhi (0)^2 -v_{\textrm{I}}^2\right) ^2, \end{aligned}$$
(64)

where the limit of \(e^{k r_c \pi }\gg 1\) has been used in the calculation. Assuming that the interaction parameters \(\lambda _{\textrm{I}}\) and \(\lambda _{\textrm{II}}\) are very large [11], this effective potential implies the solution \(\varPhi (0)=v_{\textrm{I}}\) and \(\varPhi (\pi )=v_{\textrm{II}}\). Then, the integration constants A and B can be solved approximately, yielding

$$\begin{aligned} A\approx & {} v_{\textrm{II}} e^{-(\nu +2) \sigma (\pi ) } \left[ 1-2\epsilon ^2\frac{\nu +2}{\nu +1} e^{2 \sigma (\pi ) } \right] \nonumber \\{} & {} -v_{\textrm{I}} e^{-2 \nu \sigma (\pi ) } \left[ 1+4\epsilon ^2\frac{ \nu }{\nu ^2-1} e^{2 \sigma (\pi ) }\right] ,\end{aligned}$$
(65)
$$\begin{aligned} B\approx & {} v_{\textrm{I}} \left[ 1+2\epsilon ^2\frac{\nu +2}{\nu +1} \left( e^{\nu \sigma (\pi ) }-e^{2 \sigma (\pi ) }\right) \right] \nonumber \\{} & {} -v_{\textrm{II}} e^{-(\nu +2) \sigma (\pi ) } \left[ 1+2\epsilon ^2 \frac{\nu +2}{\nu +1} e^{\nu \sigma (\pi ) }\right] . \end{aligned}$$
(66)

Further, assuming that the mass of the scalar filed is a small quantity, i.e., \({m_\varPhi }/{k}\ll 1\), one has \(\nu =\sqrt{4+\frac{m_\varPhi ^2}{k^2}}\simeq 2+\delta \), where \(\delta =\frac{m_\varPhi ^2}{4 k^2}\) is a small quantity. With this approximation, the effective potential can be rewritten as

$$\begin{aligned} V_\varPhi (r_c)\approx & {} (4+\delta ) k e^{-4 k r_c \pi } \left( 1-\frac{8 \epsilon ^2e^{2 k r_c \pi } }{\delta +3}\right) \nonumber \\{} & {} \times \left[ \left( 1+\frac{4 (\delta +2) \epsilon ^2 e^{2 k r_c \pi } }{\delta ^2+4 \delta +3 }\right) e^{-\delta k r_c \pi } v_{\textrm{I}} \right. \nonumber \\{} & {} -\left. \left( 1-\frac{2 (\delta +4) \epsilon ^2 e^{2k r_c \pi }}{\delta +3}\right) v_{\textrm{II}} \right] ^2. \end{aligned}$$
(67)

As a result, the effective potential has a minimum at

$$\begin{aligned} r_c \approx \frac{1}{\pi k\delta }\ln \left( \frac{v_h}{v_v}\right) -\frac{4\epsilon ^2}{3\pi k} \left( \frac{v_h}{v_v}\right) ^{\frac{2}{\delta }} \left( 1+\frac{v_v}{v_h}\right) . \end{aligned}$$
(68)

The first leading term is just the result of GW mechanism in GR [11], and the second term proportional to \(\epsilon ^2\) is a tiny correction stemming from the 5D graviton mass. The mechanism provides a dynamical way to stabilize the compactified radius of the extra dimension without introducing another large hierarchy. For instance, if \(m_\varPhi /k=0.2\) and \({v_h}/{v_v}=1.45\), one obtains \(ky_\pi =k r_c\pi \approx 37\) to generate a proper hierarchy. After the radius stabilization, the radion acquires a mass roughly \({\mathcal {O}}(\delta )\)TeV [64], which is still smaller than the TeV scale.

6 Conclusions

In this work, we generalized the RS1 model in a 5D extension of the Lorentz-violating massive gravity. It is found that the theory supports two distinct brane configurations. The configuration possessing both positive and negative tension branes is similar to the RS1 model, while the other possessing only two positive tension branes is distinct from RS1 model. The gauge hierarchy problem was also solved for the RS1-like brane configuration as an application. In order to analyze the mass spectra and stability, we considered full linear perturbations against the background metric. It is found that the tensor and vector modes are free from the ghost and tachyonic instabilities, however, the scalar mode is a ghost filed.

Fig. 2
figure 2

The mass spectra of RS1 model and current model, where the blue lines refer to the tensor mode, red lines to vector mode, and green lines to ghost scalar mode

Full size image

As shown in Fig. 2, all the KK particles are massive. The tensor and scalar modes have similar mass spectra which start from \(\sqrt{2}m\) with a mass splitting of TeV scale, nevertheless, the lightest vector mode does not exist in the mass spectrum. Furthermore, the ground states of tensor and scalar modes propagate only along the brane. The graviton mass \(\sqrt{2}m\) has to be tiny enough to fit the experimental constraints. Consequently, the low-energy effective theory on the brane is a massive gravity plus a ghost scalar field. Although the ghost scalar filed would lead to quantum instability, it is interesting to note that the gravity coupled to a real ghost scalar field with a negative sign for the kinetic and potential terms can support non-singular, static solutions for topologically non-trivial wormhole-like geometry [65, 66].

Further, by imposing the GW mechanism to stabilize the size of the extra dimension, the light weight ghost radion would get weight to be the order of \({\mathcal {O}}(\delta )\) TeV. After the radius stabilization, the 4D effective theory on the brane includes a nearly massless graviton plus three towers of very massive spin-2, spin-1 and ghost spin-0 particles.

In order to achieve a Minkowski flat brane model, the 4D Poincaré invariance is required to be preserved on the brane. Therefore, the internal metric of the scalar fields is chosen to be the Minkowski metric in the action (1). However, the non-positive definite Minkowski metric leads to the bulk scalar \(\phi ^0\) having a kinetic term with the wrong sign. Since there is no further constraint to eliminate this ghost scalar mode, it results in the presence of the ghost in the linear theory. In order to get rid of the ghost scalar mode, we can choose the internal metric to be \(\delta _{ij}\) with \(i,j=1,2,3\), which is positive definite and removes the problematic mode \(\phi ^0\). However, in this scenario, the scalar condensation \(\langle \phi ^a\rangle =\delta _i^a x^i\) breaks the 4D Lorentz symmetry and preserves the SO(3) symmetry on the brane. As a result, we cannot achieve a Minkowski flat brane model anymore. Instead, the brane needs to be bent, for example, de Sitter brane. The attempt to establish such a stable bent brane model is left for our another work.