1 Introduction

The fundamental idea of a brane world [1,2,3,4,5,6] is that the visible universe is localized on a 3-brane which is embedded in a higher-dimensional bulk. A renowned realization is the Randall–Sundrum (RS) brane world model [5]. There, the five-dimensional geometry is a slice of \(AdS_5\) due to the negative cosmological constant in the bulk. In such a geometry, the massless graviton is trapped on the brane. As a consequence, the four-dimensional gravity can be recovered even when the extra dimension is infinitely large. This subverts the conventional wisdom that Newton’s law means only four non-compact dimensions.

An interesting question is that whether this holds for modified gravity theories. This is the motivation of this paper. In this paper, we consider an RS-like brane world model in modified gravity theory. We use a specific source field instead of the cosmological constant in the original RS model to construct a warped geometry. This actually leads to a domain wall configuration [7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24]. Note that there is no gravity in the original domain wall model considered by Rubakov and Shaposhnikov [25]. Usually, in this work one considers a canonical scalar field for simplicity. There is also work where a noncanonical scalar field (or K-field) was considered [26,27,28,29,30,31], which would lead to some interesting background solutions and Kaluza–Klein graviton structures (gravity resonances). The models with multiple scalar fields can be found in Refs. [32,33,34]. Recently, an attempt to get a domain wall with an interacting vector field was made in [35]. There, the vector field coupled to gravity nonminimally, resulting a normalizable gravity zero mode. What is more interesting, it was shown that it is possible to get a domain wall even without both source field and cosmological constant, namely pure geometrical brane [36,37,38,39]. These models are based on the modifications of the geometry sector of the Einstein equation, such as f(R) gravity. Inspired by this work, we expect to get a domain wall (thick brane) model in modified gravity theory.

In this paper, we consider the f(R, T) gravity theories [40]. This is a special type of modified gravity theory in the sense that it introduces an arbitrary coupling between gravity and source field. As stated in [40], the energy-momentum tensor may not be conserved, which implies that the massive particles would not follow the geodesics. In addition, this gravity model has major differences in cosmology and gravitational collapse. Some more work on cosmology with this gravity theory can be found in Refs. [41,42,43,44,45,46,47,48,49]. The thick brane world model of this theory was considered in Ref. [50], and some solutions and the stability of tensor mode were touched.

We will not consider a general f(R, T) in this work. Instead, we will consider a special class of this theory, namely, \(f(R,T)=R+F(T)\). We have particular interests in this theory because it can be regarded as a class of the general K-field theory with \(L(X,\phi )=K(X,\phi )-V(\phi )\) [51, 52], which can drive inflation with general initial data. Here X denotes the kinetic term of the scalar field \(\phi \). In Refs. [29, 30], the special K-field with \(L(X,\phi )=K(X)-V(\phi )\) was used to build domain walls, and some interesting results were obtained. It is straightforward to see that the theory we will consider is of the \(L(X,\phi )=K(X,\phi )-V(\phi )\) type. To see whether the four-dimensional gravity can be recovered, we would like to investigate the full linear gravitational perturbations. The effective four-dimensional gravity can be recovered only when the massless tensor mode is localized on the brane, and the vector and scalar modes are not localized, since the normalizable massless scalar mode or vector mode would lead to a “fifth force”. Except the recovering of four-dimensional gravity, another purpose is to study how the brane world model is modified by replacing R with \(R+F(T)\), at both the background and the perturbation levels. As is well known, the evolutions of perturbations are related to the background configuration, so it is necessary to get the background solutions at first.

We review the f(R, T) gravity theory and its equation of motion in Sect. 2, and try to get the background solutions of the brane model with \(f(R,T)=R+F(T)\). In Sect. 3, we investigate the full linear perturbations with the scalar-vector-tensor decomposition. Then we analyze the behaviors of these perturbation modes, and conclude whether this model gives a viable four-dimensional gravity. At last, we give the conclusions and summary.

2 Background solutions of the model

Let us start with the action and the field equations of the f(R, T) theories of gravity. In five-dimensional spacetime, the action takes the form

$$\begin{aligned} S=\frac{1}{2\kappa ^2}\int \mathrm{d}^5 x\sqrt{-g}f(R,T)+ \int \mathrm{d}^5 x\sqrt{-g}L_\mathrm{m}, \end{aligned}$$
(1)

where f(R, T) is an arbitrary smooth function of the Ricci scalar R and of the trace of the stress-energy tensor T, and \(2\kappa ^2=M_{*}^{-3}\) with \(M_{*}\) the five-dimensional fundamental scale. The stress-energy tensor is defined by

$$\begin{aligned} T_{MN}=-\frac{2}{\sqrt{-g}}\frac{\delta (\sqrt{-g}L_\mathrm{m})}{\delta g^{MN}}. \end{aligned}$$
(2)

As discussed in Ref. [40], this formalism of the action allows one to consider a wide class of theories. In this paper, we consider the choice of \(f(R,T)=R+F(T)\), which removes the higher-derivative terms of the field equations. The gravitational field equation can be obtained by varying the action with respect to the metric \(g^{MN}\), and the result is

$$\begin{aligned} R_{MN}-\frac{1}{2}\left( R+F(T)\right) g_{MN}=\kappa ^2 T_{MN} + \Theta _{MN}, \end{aligned}$$
(3)

where \(\Theta _{MN}=-\frac{\delta F(T)}{\delta g^{MN}}\). In this work, we consider a canonical scalar field for simplicity, that is,

$$\begin{aligned} L_\mathrm{m}=X-V(\phi )=-\frac{1}{2}\partial _{M}\phi \partial ^{M}\phi -V(\phi ), \end{aligned}$$
(4)

for which the energy-momentum tensor and its trace are given by

$$\begin{aligned} T_{MN}= & {} {\partial _M\phi } {\partial _N\phi }+ g_{MN} L_\mathrm{m}, \end{aligned}$$
(5)
$$\begin{aligned} T= & {} -\frac{3}{2}g^{MN} {\partial _M\phi } {\partial _N\phi }-5V =3X-5V. \end{aligned}$$
(6)

The corresponding equation of motion of the scalar field is

$$\begin{aligned} \frac{3}{2\kappa ^2}\nabla _{M}\left( F_T\nabla ^{M}\phi \right) +\nabla _{M}\nabla ^{M}\phi - \left( \frac{5F_T}{2\kappa ^2}+1\right) \frac{\partial V(\phi )}{\partial \phi }=0, \end{aligned}$$
(7)

here \(F_T\) is the derivative of F(T) with respect to T. In order to construct a thick brane world model, we use the metric ansatz

$$\begin{aligned} \mathrm{d}s^2\equiv g_{MN}\mathrm{d}x^{M}\mathrm{d}x^{N}=e^{2A(y)}\eta _{\mu \nu }\mathrm{d}x^{\mu }\mathrm{d}x^{\nu }+\mathrm{d}y^2. \end{aligned}$$
(8)

Here \(e^{A(y)}\) is the warp factor, and \(x^M=(x^\mu ,y)\). With this metric, the field equations (3) and (7) can be expressed in the following specific formalism:

$$\begin{aligned}&-6A'^2 - 3A''+\frac{1}{2}F(T)= \kappa ^2 \left( \frac{1}{2}{\phi '}^2+V(\phi )\right) ,\end{aligned}$$
(9)
$$\begin{aligned}&-6A'^2 +\frac{1}{2}F(T)= \kappa ^2 \left( -\frac{1}{2}{\phi '}^2+V(\phi )\right) -\frac{3}{2}F_{T}{\phi '}^2,\nonumber \\ \end{aligned}$$
(10)
$$\begin{aligned}&\left( \kappa ^2+\frac{3}{2}F_T\right) \phi '' + 4\left( \kappa ^2+\frac{3}{2}F_T\right) A'\phi ' +\frac{3}{2}F_{T}'\phi '\nonumber \\&\quad =\left( \kappa ^2+\frac{5}{2}F_T\right) V_{\phi }, \end{aligned}$$
(11)

where prime represents the derivative with respect to the extra dimension coordinate y. Now the system consists of Eqs. (9), (10), and (11). There are actually two independent equations because the covariant divergence of the Einstein tensor is zero (the covariant divergence of the energy-momentum tensor does not vanish). We have to solve this system which contains four indeterminate functions, and this implies that we are allowed to impose two constraints on this system. In the previous work [50], a series of solutions including the sine-Gordon type were obtained by using the first-order formalism equations. The first-order equations were derived by introducing a superpotential, and the equations were solved by giving a specific superpotential. To get more types of solutions except those found in Ref. [50], we do not follow this approach in this paper.

Fig. 1
figure 1

The shapes of the warp factor \(e^{A(y)}\) and scalar field \(\phi (y)\) with respect to ky and the scalar potential \(V_1(\phi )\) with respect to \(\phi \) for the first brane solution (14), (15). Note that the scalar potential is opening up (bottom left) for positive \(\upsilon _1^2 \kappa ^2-45\), and opening down (bottom right) for negative \(\upsilon _1^2 \kappa ^2-45\)

Full size image

As the first example, we consider the simplest case with \(F(T)=\alpha T\), for which the effective matter Lagrangian is

$$\begin{aligned} L_{\mathrm{eff}} = \frac{1}{{2\kappa ^2}}F(T)+L_\mathrm{m} = \left( 1+\frac{3\alpha }{{2\kappa ^2}}\right) X-\left( 1+\frac{5\alpha }{{2\kappa ^2}}\right) V. \end{aligned}$$
(12)

Using Eqs. (9) and (10), we get a largely simplified equation

$$\begin{aligned} -3A''=\left( \kappa ^2+\frac{3}{2}F_T \right) \phi '^2. \end{aligned}$$
(13)

Now the solutions can be obtained by solving Eqs. (11) and (13). We consider a kink scalar field solution, namely \(\phi (y)=\upsilon _1 \text {tanh}(ky)\). This supports the solution of the system as follows:

$$\begin{aligned} A(y)= & {} \text {sech}^2(k y)-4 \log \left( \cosh (k y)\right) -1,\end{aligned}$$
(14)
$$\begin{aligned} V_{1}(\phi )= & {} \frac{9 k^2}{2 \upsilon _1^4 \left( \upsilon _1^2 \kappa ^2-45\right) } \left( 78 \upsilon _1^4 \phi ^2-51 \upsilon _1^2 \phi ^4+8 \phi ^6\right) , \nonumber \\ \end{aligned}$$
(15)

with \(\alpha = \frac{12}{\upsilon _1^2}-\frac{2}{3}\kappa ^2\). Note that we have chosen appropriate parameters to make \(A(0)=0\). We show the plots of \(e^{A(y)}\), \(\phi (y)\), and \(V(\phi )\) in Fig. 1. Clearly, the scalar potential has a \(\phi ^6\) profile, and it opens up and down for positive and negative \(\upsilon _1^2 \kappa ^2-45\), respectively. The scalar field approaches a constant at infinity, which corresponds to the local maxima of the scalar potential for positive \(\upsilon _1^2 \kappa ^2-45\). This seems contrary to our common sense. However, it should be noted that the source part of the action and thus are modified. To see this clearly, we investigate the effective Lagrangian (12). For constant \(F_T\) considered here, we have

$$\begin{aligned} L_{\text {eff}}= & {} \frac{18}{c_{0}^{2}\kappa ^2}\left( -\frac{1}{2}(\partial \phi )^2 - V_{\text {eff}}(\phi )\right) ,\end{aligned}$$
(16)
$$\begin{aligned} V_{\text {eff}}(\phi )= & {} -\frac{\upsilon _1^2 \kappa ^2-45}{27}V(\phi ) \nonumber \\= & {} -\frac{9 k^2}{2 \upsilon _1^4} \left( 78 \upsilon _1^4 \phi ^2-51 \upsilon _1^2 \phi ^4+8 \phi ^6\right) . \end{aligned}$$
(17)

From this point of view, the infinity of the extra dimension corresponds to the minimum of the effective scalar potential \(V_{\text {eff}}(\phi )\) regardless of the sign of \({c_{0}^{2}\kappa ^2} -45\), which is consistent with our conventional wisdom. However, the original scalar potential \(V(\phi )\) in (6) does not need to follow this due to the inclusion of the F(T) term. For general F(T), there will be some more differences. Now we can see that the two solutions given by Fig. 1c, d are permissible. This is a new feature, different from the standard case.

The second example is for a more general power of T, i.e., \(F(T)=\alpha T^n\) with n a positive integer. The effective Lagrangian is

$$\begin{aligned} L_{\mathrm{eff}} = \frac{\alpha }{{2\kappa ^2}}(3X-5V)^n+X-V. \end{aligned}$$
(18)

One of the solution is given by

$$\begin{aligned} A(y)= & {} \log (\text {sech} (k y)),\end{aligned}$$
(19)
$$\begin{aligned} \phi (y)= & {} \upsilon _2 \arctan \left( \tanh \left( {k y}/{2}\right) \right) ,\end{aligned}$$
(20)
$$\begin{aligned} V_{2}(\phi )= & {} \frac{27 k^2}{4 \kappa ^2}\left( \frac{17n-14}{14-9n}-\cos \left( {4}\phi /{\upsilon _2} \right) \right) , \end{aligned}$$
(21)

with \(\upsilon _2=\frac{6\sqrt{5}}{\kappa }\). It can easily be checked that T is a constant here. Note that for \(n=1\) the action reduces to the first model, but this solution is different from the first one given in (14) and (15) since the scalar potential here is the sine-Gordon one. Similar to the solution in the first example, the infinity of the extra dimension is at the maxima of the scalar potential.

Fig. 2
figure 2

The plots of the potentials \(V_{2,3}(\phi )\) in the second and third solutions (21) and (24). The infinity of the extra dimension corresponds to \(\pm \phi _0=\pm \frac{\pi \upsilon _2}{4}\) (left) and \(\pm \upsilon _3\) (right)

Full size image

For nonconstant \(F_T\), it is much more difficult to get the solution. We give the solution here without the expression of F(T):

$$\begin{aligned} A (y)= & {} \log (\text {sech} (k y)),\end{aligned}$$
(22)
$$\begin{aligned} \phi (y)= & {} \upsilon _3 \tanh (k y),\end{aligned}$$
(23)
$$\begin{aligned} V_{3} (\phi )= & {} \frac{9 k^2 }{\upsilon _3^2 \kappa ^2} \left[ \phi ^2\!-\!\frac{15}{2 \kappa ^2} \log \left( 2 \kappa ^2 \left( \phi ^2\!-\!\upsilon _3^2\right) \!+\!15\right) \right] , ~~~~~\end{aligned}$$
(24)
$$\begin{aligned} F(y)= & {} k^2 \left( \kappa ^2 \upsilon _3^2 \text {sech}^4(k y)-36 \text {sech}^2(k y)+30\right) \nonumber \\&-\frac{135 k^2 }{\kappa ^2 \upsilon _3^2} \log \left( 15-2 \kappa ^2 \upsilon _3^2 \text {sech}^2(k y)\right) . \end{aligned}$$
(25)

We require

$$\begin{aligned} \upsilon _3^2<\frac{15}{2 \kappa ^2} \end{aligned}$$
(26)

to make the log term in (25) real. The plots of the potentials \(V_{2}\) and \(V_{3}\) are given in Fig. 2. It can be checked that all the three solutions above give asymptotically \(AdS_5\) bulk geometry.

3 Full Linear Perturbations

In this section, we discuss the full linear perturbations of this brane world model. From now on, we consider the case of an arbitrary F(T). In the following the calculation is done in the conformally flat coordinate system. The physical and conformal extra dimension coordinates are related by the equation \(\mathrm{d}y=e^{A(z)}\mathrm{d}z\), which together with (8) gives

$$\begin{aligned} \mathrm{d}s^2=e^{2A(z)}\left( \eta _{\mu \nu }\mathrm{d}x^\mu \mathrm{d}x^\nu + \mathrm{d}z^2\right) . \end{aligned}$$
(27)

Now we introduce the perturbation of this metric. For the background metric \(g_{MN}\), the perturbed metric is

$$\begin{aligned} \tilde{g}_{MN}=g_{MN}+\delta g_{MN}. \end{aligned}$$
(28)

Here \(\delta g_{MN}\) is the metric perturbation tensor. To linear order, the metric perturbation can be decomposed into scalar, vector, and tensor modes, or in the following specific formalism:

$$\begin{aligned}&\delta g_{MN}= e^{2A(z)}\nonumber \\&\quad \times \left( \begin{array}{cc} 2h_{\mu \nu } + \partial _{\mu }\xi _{\nu } +\partial _{\nu }\xi _{\mu } + 2\eta _{\mu \nu }\psi + 2\partial _{\mu }\partial _{\nu }\Phi &{} ~\zeta _{\mu } + \partial _{\mu }\varphi \\ \zeta _{\nu } + \partial _{\nu }\varphi &{} ~2\chi \\ \end{array} \right) .\nonumber \\ \end{aligned}$$
(29)

The tensor \(h_{\mu \nu }\), which is relevant to the gravitational waves, satisfies the transverse and traceless (TT) condition

$$\begin{aligned} \partial ^{\mu }h_{\mu \nu }=0,\quad \eta ^{\mu \nu }h_{\mu \nu }=0. \end{aligned}$$
(30)

The \(\xi _{\nu }\) and \(\zeta _{\mu }\) are transverse vector modes, i.e.

$$\begin{aligned} \partial ^{\mu }\xi _{\mu }=0,\quad \partial ^{\mu }\zeta _{\mu }=0. \end{aligned}$$
(31)

The remaining variables \(\psi \), \(\Phi \), \(\varphi \), and \(\chi \) represent the scalar degrees of freedom. Clearly, different kinds of modes decouple in the action. As we will see below, this is crucial for our analysis of perturbation modes.

Since we are interested in the behaviors of these modes, it is necessary to get their field equations. The perturbed field equations can be obtained by replacing the background metric in (3) by the perturbed metric (28). If this were done, we would get the field equations that contain the lowest order (zero order), the linear order, and the higher order terms of the metric perturbation. The lowest order parts are just the background Eqs. (9) and (10). We do not explore the higher order equations since this subject is beyond the scope of the present work. So we concentrate on the linear order field equations. Note that the decomposition (29) makes different kinds of modes decouple, therefore we are allowed to divide a linear order equation into three equations (the scalar, vector, and tensor equations). One can also get the field equations of the metric perturbation modes by writing down the quadratic order action, and then varying this action with respect to various perturbation modes, respectively. We adopt the former path in this work.

The metric perturbation tensor (29) contains seven variables, or 15 degrees of freedom totally. Nevertheless, not all of them are independent because of the gauge invariance. Let us consider the infinitesimal coordinate transformation

$$\begin{aligned} x^{A}\rightarrow \tilde{x}^{A}=x^{A} + \epsilon ^{A}. \end{aligned}$$
(32)

Under this transformation, the metric perturbation tensor \(\delta g_{MM}\) transforms as

$$\begin{aligned} \delta g_{MN}\rightarrow \delta \tilde{g}_{MN}=\delta g_{MN} + 2\nabla _{(M}\epsilon _{N)}. \end{aligned}$$
(33)

It is easy to check that the TT part \(h_{\mu \nu }\) is gauge invariant. Although the non-TT part is not gauge invariant, it is possible to construct some gauge invariant variables [12,13,14] by using combinations of the variables given in (29). We are allowed to choose a suitable gauge to eliminate the redundant degrees of freedom (5, in five dimensions). Usually, it would be convenient in the longitudinal gauge, i.e. \(\xi _{\mu }=0\) and \(\Phi =0=\varphi \). In this gauge, the metric perturbation tensor becomes

$$\begin{aligned} \delta g_{MN}=e^{2A(z)} \left( \begin{array}{cc} 2h_{\mu \nu }+2\eta _{\mu \nu }\psi &{} \zeta _{\mu } \\ \zeta _{\nu } &{} 2\chi \\ \end{array} \right) . \end{aligned}$$
(34)

Using this metric perturbation, we investigate the perturbed field equation (3):

$$\begin{aligned} \delta R_{MN}-\frac{1}{2}\delta \left( F(T)g_{MN}\right) =\kappa ^2 \delta T_{MN} +\frac{3}{2}\delta \left( F_{T}\partial _{M}\phi \partial _{N}\phi \right) . \end{aligned}$$
(35)

To linear order, this equation can be decomposed into \(\mu \nu \), \(\mu 5\), and 55 components.

The explicit forms of the \(\mu \nu \) components are

$$\begin{aligned}&\Big [\Box +\partial _{z}^{2}+3\partial _{z}A \partial _{z} +6\left( \partial _{z}A\right) ^2 +2\partial _{z}^{2}A +e^{2A}F \Big ] h_{\mu \nu }\nonumber \\&\quad -2\kappa ^2 \left( \frac{1}{2}(\partial _{z}\phi )^2+ e^{2A} V\right) h_{\mu \nu }=0, \end{aligned}$$
(36)
$$\begin{aligned} \partial _{\mu }\partial _{z}\zeta _{\nu } +\partial _{\nu }\partial _{z}\zeta _{\mu } +3\partial _{z}A\left( \partial _{\mu }\zeta _{\nu } +\partial _{\nu }\zeta _{\mu }\right) =0, \end{aligned}$$
(37)
$$\begin{aligned}&2\eta _{\mu \nu }\Box \psi -2\partial _{\mu }\partial _{\nu }\psi +3\eta _{\mu \nu }\partial _{z}^{2}\psi +9\eta _{\mu \nu }\partial _{z}A\partial _{z}\psi \nonumber \\&\quad +6\eta _{\mu \nu }\left( \left( \partial _{z}A\right) ^2 +\partial _{z}^{2}A\right) \psi -e^{2A}F(T)\eta _{\mu \nu }\psi \nonumber \\&\quad +\eta _{\mu \nu }\kappa ^2\left( (\partial _{z}\phi )^2+ 2e^{2A} V(\phi )\right) \psi +\eta _{\mu \nu }\Box \chi -\partial _{\mu }\partial _{\nu }\chi \nonumber \\&\quad -6\eta _{\mu \nu }\left( \left( \partial _{z}A\right) ^2 +\partial _{z}^{2}A\right) \chi -3\eta _{\mu \nu }\partial _{z}A\partial _{z}\chi \nonumber \\&\quad +\eta _{\mu \nu }\left[ \Big (\frac{3}{2}F_T +\kappa ^2\Big )\partial _{z}\phi \partial _{z} +\Big (\frac{5}{2}F_T +\kappa ^2\Big )e^{2A}{V_{\phi }}\right] \delta \phi \nonumber \\&\quad -\eta _{\mu \nu }\Big (\frac{3}{2}F_T +\kappa ^2\Big )\left( \partial _{z}\phi \right) ^2\chi =0, \end{aligned}$$
(38)

where \(\Box =\eta ^{\alpha \beta }\partial _\alpha \partial _\beta \) is the four-dimensional d’Alembert operator, and \(\delta \phi \) is the perturbation of the scalar field. Equation (36) is the equation of motion of the tensor perturbation mode. We can eliminate the first derivative term from (36) to obtain

$$\begin{aligned} \Box \tilde{h}_{\mu \nu } +\partial _{z}^{2}\tilde{h}_{\mu \nu } -\left( \frac{3}{2}\partial _{z}^{2}A +\frac{9}{4}(\partial _{z}A)^{2}\right) \tilde{h}_{\mu \nu }=0 \end{aligned}$$
(39)

by defining \(\tilde{h}_{\mu \nu }=e^{\frac{3}{2}A}h_{\mu \nu }\). This redefinition is actually equivalent to canonically normalize the kinetic term of the tensor mode. This is the equation of motion of the tensor mode, and we will analyze it in next section. The remaining two Eqs. (37) and (38) lead to

$$\begin{aligned} \partial _{z}\partial _{(\mu }\zeta _{\nu )} +3\partial _{z}A\partial _{(\mu }\zeta _{\nu )}= & {} 0,\end{aligned}$$
(40)
$$\begin{aligned} \chi +2\psi= & {} 0. \end{aligned}$$
(41)

As can be seen, Eq. (41) is just an algebraic equation of the two scalar modes. Usually, one cannot get similar relations in gravity theories with higher-derivative terms of the metric, for instance, the metric formalism f(R) theories of gravity [53,54,55]. Equation (40) is the field equation of the vector mode, and also we will analyze it in the next section.

The \(\mu 5\) components can be divided into two parts,

$$\begin{aligned}&\kappa ^2\left( \frac{1}{2}(\partial _{z}\phi )^2+e^{2A} V\right) \zeta _{\mu }-\frac{1}{2}\Box \zeta _{\mu } -\frac{1}{2}F(T)e^{2A}\zeta _{\mu } \nonumber \\&\quad +3\left( \left( \partial _{z}A\right) ^2+\partial _{z}^{2}A\right) \zeta _{\mu }=0, \end{aligned}$$
(42)
$$\begin{aligned}&3\partial _{z}A\partial _{\mu }\chi -3\partial _{\mu }\partial _{z}\psi -\left( \frac{3}{2}F_{T}+\kappa ^2\right) \partial _{z}\phi \partial _{\mu }\delta \phi =0. \end{aligned}$$
(43)

Combining Eqs. (9) and (42), we get

$$\begin{aligned} \Box \zeta _{\mu }=0. \end{aligned}$$
(44)

This implies that the vector mode is massless. By substituting Eq. (41) into (43), we get the solution of the perturbation of the scalar field

$$\begin{aligned} \delta \phi =-\frac{6(\partial _z \psi + 2\psi \partial _z A )}{(2\kappa ^2 + 3 F_T)\partial _z \phi }. \end{aligned}$$
(45)

It should be pointed out that it is impossible to get similar results in gravity theories with higher-derivative terms of source fields, for example, f(R) theories in the Palatini formalism [24, 54]. The solution (45) is crucial to the simplification of the 55 component of Eq. (35), which reads

$$\begin{aligned}&\partial _{z}\phi \left( \kappa ^2+\frac{3}{2}F_{T} -\frac{9(\partial _{z}\phi )^2}{2e^{2A}}F_{TT}\right) \partial _{z}\delta \phi + 3\Box \psi \\&\qquad -e^{2A} {V_{\phi }}\left( \kappa ^2 \!+\!\frac{5}{2}F_{T} \!+\! \frac{15(\partial _{z}\phi )^2}{2e^{2A}} F_{TT} \right) \delta \phi \!+\!12\partial _z A \partial _z\psi \\&\quad = \left( 2\kappa ^2 e^{2A} V-e^{2A}F \!-\! \frac{3}{2}F_{T}\left( \partial _{z}\phi \right) ^2 \!-\! \frac{9(\partial _{z}\phi )^4}{2e^{2A}}F_{TT}\right) \chi . \end{aligned}$$

Now it is clear that we can reduce the number of scalar perturbation modes of this system to be 1. Again using the background equations (9)–(11) and some manipulations, we get an equation which involves only one scalar mode \(\psi \):

$$\begin{aligned}&\bigg \{\Box + B(z)\big ( \partial _z^2 + \partial _z \left[ \text {ln}\left( \frac{e^{3A}}{G(z)\partial _z\phi }\right) \right] \partial _z \nonumber \\&\quad + 4\partial _z^2 A- 2\partial _z A \partial _z \text {ln}\left[ G(z)\partial _z \phi \right] \big )\bigg \}\psi =0, \end{aligned}$$
(46)

where \(B(z)=1-\frac{9F_{TT}(\partial _z\phi )^3}{2G(z)e^{2A(z)}}\) and \(G(z)=\left( \kappa ^2 + \frac{3}{2}F_T\right) \partial _z \phi \). This is the field equation of the scalar mode of the metric perturbation (29). Until now we have successfully obtained the field equations of various modes of the metric perturbation (29).

4 Localization of perturbation modes

In this section let us discuss the behaviors of the tensor, vector, and scalar modes. This requires the analysis of Eqs. (39), (40), and (46).

4.1 Tensor mode

Equation (39) is a Schrödinger-like equation of the tensor mode. Clearly, it is the same as that of general relativity. Therefore it has the same mass spectrum. To get a better understanding of the effective four-dimensional gravity, we make a decomposition

$$\begin{aligned} \tilde{h}_{\mu \nu }(x^{\mu },z)=\hat{h}_{\mu \nu }(x^{\mu })\Psi (z), \end{aligned}$$
(47)

and then following from (39) we get

$$\begin{aligned} \left( \partial _z + \frac{3}{2}\partial _zA\right) \left( -\partial _z + \frac{3}{2}\partial _zA\right) \Psi (z) =m^2\Psi (z). \end{aligned}$$
(48)

Note that this is consistent with \(\Box \hat{h}_{\mu \nu }=m^2 \hat{h}_{\mu \nu }\). The above equation gives the mass spectrum of the Kaluza–Klein (KK) modes of gravity, and it obviously avoids the tachyon instability. The zero mode corresponding to the solution with \(m=0\) is given by

$$\begin{aligned} \Psi _0 (z)\propto e^{3A(z)/2}. \end{aligned}$$
(49)

The recovering of the effective four-dimensional gravity requires the normalization of the zero mode:

$$\begin{aligned} \int |\Psi _0 (z)|^2 \mathrm{d}z< \infty . \end{aligned}$$
(50)

This is equivalent to having a finite four-dimensional Planck mass, if we define \(M_{\text {Pl}}^2=M_* ^3\int |\Psi _0 (z)|^2 \mathrm{d}z\), i.e.

$$\begin{aligned} S\supset M_* ^3\int |\Psi _0 (z)|^2 \mathrm{d}z \int \partial \hat{h}\partial \hat{h}. \end{aligned}$$
(51)

For our solution (14), this condition can surely be satisfied, so the zero mode can be localized.

In addition to the zero mode, Eq. (48) allows for a continuous mass spectrum for massive states. If the normalization condition cannot be satisfied for the massive state \(\Psi _\mathrm{m}\), then this massive state would be plane wave at infinity. In other words, the corresponding massive KK graviton cannot be localized on the brane. For our case, all the massive gravitons cannot be localized.

4.2 Vector mode

The field equations of the vector mode correspond to Eqs. (40) and (44). They are the same as in general relativity, so we do not investigate them in detail. It can be concluded straightforwardly that, if the tensor zero mode can be localized, then the vector mode cannot be localized on the brane.

4.3 Scalar mode

We now turn to the scalar mode equation (46). This equation has significant difference from that of general relativity [12,13,14]. To get a Schrödinger-like formalism equation (or the equation of the canonically normalized field), we first perform a coordinate transformation

$$\begin{aligned} \mathrm{d}z\equiv \sqrt{B(z)}\mathrm{d}r. \end{aligned}$$
(52)

Note that we have a constraint on the function F(T) to make \(B(z)>0\). If this is satisfied then the Eq. (46) can be written as

$$\begin{aligned} \Box \psi + \partial _r^2 \psi + K(r)\partial _r\psi + J(r)\psi =0, \end{aligned}$$
(53)

where

$$\begin{aligned} K(r)= & {} \partial _r \text {ln}\left( \frac{e^{3A(r)}}{G(r)\partial _r \phi }\right) ,\end{aligned}$$
(54)
$$\begin{aligned} J(r)= & {} 4\partial _r^2 A - 2\partial _r A \partial _r\text {ln}[G(r)\partial _r \phi ]. \end{aligned}$$
(55)

Clearly, since Eq. (53) contains a term with single derivative on \(\psi \), the scalar mode \(\psi \) is surely not canonically normalized in the perturbed quadratic action. This trouble can be solved by redefining the scalar mode as \(\tilde{\psi }(r)=\text {e}^{\int \frac{1}{2}K(r)\mathrm{d}r}\psi (r)\). In terms of \(\tilde{\psi }(r)\), Eq. (53) turns out to be

$$\begin{aligned} \Box \tilde{\psi } + \partial _r^2 \tilde{\psi } + \left[ J(r)-\frac{1}{2}\partial _r K-\frac{1}{4}K^2\right] \tilde{\psi }=0. \end{aligned}$$
(56)

The scalar mode can also be separated as \(\tilde{\psi }(x^\mu ,r)=\hat{\psi }(x^\mu )\Phi (r)\) with \(\Box \hat{\psi }(x^\mu )=m^2 \hat{\psi }(x^\mu )\). With this decomposition, we finally get a Schrödinger-like equation of the canonically normalized scalar mode \(\Phi (r)\):

$$\begin{aligned} - \partial _r^2 \Phi + V(r)\Phi =m^2 \Phi , \end{aligned}$$
(57)

with \(V(r)=\frac{1}{2}\partial _r K + \frac{1}{4}K^2 - J(r)\). It can be shown that this equation can be factorized as

$$\begin{aligned} \left( \partial _r + \frac{\partial _r I(r)}{I(r)}\right) \left( -\partial _r + \frac{\partial _r I(r)}{I(r)}\right) \Phi =m^2 \Phi , \end{aligned}$$
(58)

with \(I(r)=\partial _r A [e^{3A}B(r)G(r)\partial _r \phi ]^{-1/2}\). To get a real I(r) we require \(\kappa ^2 + \frac{3}{2}F_T > 0\). This formalism of equation ensures that no scalar mode with \(m^2 <0\) exists.

Furthermore, the zero mode (massless mode) solution is

$$\begin{aligned} \Phi _0 \propto I(r). \end{aligned}$$
(59)

As a comparison, we recall the scalar zero mode solution in the standard case, which corresponds to \(B(r)=1\) and \(G(r)=\kappa ^2\). It is more convenient to analyze the normalization condition in the physical coordinate y. Using the background equation (13), we have

$$\begin{aligned} \int |\Phi _0(r)|^2 \mathrm{d}r\propto & {} \int \frac{(\partial _z A)^2~\mathrm{d}z}{3e^{3A} \left[ (\partial _z A)^2 - \partial _z^2 A \right] \!- \!9F_{TT} (\partial _z \phi )^4 /2} \nonumber \\\propto & {} \int \frac{(\partial _y A)^2~\mathrm{d}y}{3e^{4A} \partial _y ^2 A - 9 e^{3A} F_{TT} (\partial _y \phi )^4 /2}. \end{aligned}$$
(60)

It has a significant difference with that of general relativity coupled to a canonical scalar field, in which \(F_{TT}=0\).

For our three background solutions given in Sect. 2, the corresponding \(F_{T}\) and \(F_{T T}\) are

$$\begin{aligned} F^{(1)}_{T}= & {} \alpha =\frac{12}{\upsilon _1^2}-\frac{2}{3}\kappa ^2,\end{aligned}$$
(61)
$$\begin{aligned} F^{(2)}_{T}= & {} -\frac{28}{45} \kappa ^2 ,\end{aligned}$$
(62)
$$\begin{aligned} F^{(3)}_{T}= & {} \frac{2 }{\upsilon _3^2} \cosh ^2(k y)-\frac{2}{3}\kappa ^2, \end{aligned}$$
(63)

and

$$\begin{aligned} F^{(1)}_{T T}= & {} 0,\end{aligned}$$
(64)
$$\begin{aligned} F^{(2)}_{T T}= & {} -\frac{14(n-1) (9 n-14) \kappa ^4 }{6075 n k^2}\quad (n\ge 1),\end{aligned}$$
(65)
$$\begin{aligned} F^{(3)}_{T T}= & {} \frac{\cosh ^6(k y) \left( 4 \upsilon _3^2 \kappa ^2-30 \cosh ^2(k y)\right) }{\upsilon _3^4 k^2 \left( 6 \upsilon _3^2 \kappa ^2-135 \cosh ^2(k y)\right) }, \end{aligned}$$
(66)

respectively. It is easy to verify that all the solutions given in Sect. 2 satisfy the two conditions, \(B=1-\frac{9F_{TT}(\partial _y\phi )^2}{2(\kappa ^2+\frac{3}{2}F_T)}>0\) and \(\kappa ^2+\frac{3}{2}F_T >0\). Substituting them to Eq. (60), we can see that the scalar zero mode cannot be normalized for all the three solutions. So our background solutions are stable, and the scalar mode cannot be localized.

It is worth noting here that the first model with \(F(T)=\alpha T\) is equivalent to general relativity coupled to a canonical scalar field at background and perturbation levels regardless of whether T is a constant or not. For the second model, namely, \(F=\alpha T^n\) with T a constant, even though the action is equivalent to general relativity coupled to a canonical scalar field, they are not equivalent (except \(n=1\)) at perturbation level. The third model is completely different from the standard case at both of background and perturbation levels.

5 Conclusions

To summarize, we investigated the thick brane world model in f(R, T) theories of gravity. The domain wall configuration was constructed by introducing a scalar field in the non-compact bulk. The background solution was obtained by giving a kink scalar field. All of the background quantities are smooth, and so there is no singularity in this asymptotically \(AdS_5\) space. In thick brane world models constructed with general relativity coupled to a canonical scalar field, the scalar potential can be \(\phi ^4\) type and sine-Gordon type etc., and these solutions share a common characteristic that the vacuum is at the minimum. However, in our model this does not need to be the case. This is a significant new feature that different from the standard case.

Besides, we studied the full linear perturbations of this model, including tensor, vector, and scalar modes. Among these modes, the tensor and vector modes are the same as that of general relativity coupled to a canonical scalar field, and only the scalar mode is modified due to the F(T) term (except for the special case of \(F(T)=\alpha T\)). We found that, to linear order, the scalar, vector, and tensor modes are stable and no tachyon state exists. Furthermore, we showed that only the tensor zero mode (four-dimensional massless graviton) can be localized on the brane, hence we obtained the viable four-dimensional gravity. These conclusions hold if the two constraints on the action, namely \(B(z)>0\) and \(\kappa ^2 + \frac{3}{2}F_T > 0\), are satisfied. We argue that these constraints are significant for building viable models from this class of theories.