Polarizations of gravitational waves in Horndeski theory

Abstract

We analyze the polarization content of gravitational waves in Horndeski theory. Besides the familiar plus and cross polarizations in Einstein’s General Relativity, there is one more polarization state which is the mixture of the transverse breathing and longitudinal polarizations. The additional mode is excited by the massive scalar field. In the massless limit, the longitudinal polarization disappears, while the breathing one persists. The upper bound on the graviton mass severely constrains the amplitude of the longitudinal polarization, which makes its detection highly unlikely by the ground-based or space-borne interferometers in the near future. However, pulsar timing arrays might be able to detect the polarization excited by the massive scalar field. Since additional polarization states appear in alternative theories of gravity, the measurement of the polarizations of gravitational waves can be used to probe the nature of gravity. In addition to the plus and cross states, the detection of the breathing polarization means that gravitation is mediated by massless spin 2 and spin 0 fields, and the detection of both the breathing and longitudinal states means that gravitation is propagated by the massless spin 2 and massive spin 0 fields.

1 Introduction

The detection of gravitational waves by the Laser Interferometer Gravitational-Wave Observatory (LIGO) Scientific Collaboration and VIRGO Collaboration further supports Einstein’s General Relativity (GR) and provides a new tool to study gravitational physics [1,2,3,4,5,6]. In order to confirm gravitational waves predicted by GR, it is necessary to determine the polarizations of gravitational waves. This can be done by the network of ground-based Advanced LIGO (aLIGO) and VIRGO, the future space-borne Laser Interferometer Space Antenna (LISA) [7] and TianQin [8], and pulsar timing arrays (e.g., the International Pulsar Timing Array and the European Pulsar Timing Array [9, 10]). In fact, in the recent GW170814 [4], the Advanced VIRGO detector joined the two aLIGO detectors, so they were able to test the polarization content of gravitational waves for the first time. The result showed that the pure tensor polarizations were favored against pure vector and pure scalar polarizations [4, 11]. Additionally, GW170817 is the first observation of a binary neutron star inspiral, and its electromagnetic counterpart, GRB 170817A, was later observed by the Fermi Gamma-ray Burst Monitor and the International Gamma-Ray Astrophysics Laboratory [5, 12, 13]. The new era of multi-messenger astrophysics comes.

In GR, the gravitational wave propagates at the speed of light and it has two polarization states, the plus and cross modes. In alternative metric theories of gravity, there may exist up to six polarizations, so the detection of the polarizations of gravitational waves can be used to distinguish different theories of gravity and probe the nature of gravity [14, 15]. For null plane gravitational waves, the six polarizations are classified by the little group E(2) of the Lorentz group with the help of the six independent Newman-Penrose (NP) variables $\varPsi _2$, $\varPsi _3$, $\varPsi _4$ and $\varPhi _{22}$ [16,17,18]. In particular, the complex variable $\varPsi _4$ denotes the familiar plus and cross modes in GR, the variable $\varPhi _{22}$ denotes the transverse breathing polarization, the complex variable $\varPsi _3$ corresponds to the vector-x and vector-y modes, and the variable $\varPsi _2$ corresponds to the longitudinal mode. Under the E(2) transformation, all other modes can be generated from $\varPsi _2$, so if $\varPsi _2\ne 0$, then we may see all six modes in some coordinates. The E(2) classification tells us the general polarization states, but it fails to tell us the correspondence between the polarizations and gravitational theories. In Brans-Dicke theory [19], in addition to the plus and cross modes $\varPsi _4$ of the massless gravitons, there exists another breathing mode $\varPhi _{22}$ due to the massless Brans-Dicke scalar field [17].

Brans-Dicke theory is a simple extension to GR. In Brans-Dicke theory, gravitational interaction is mediated by both the metric tensor and the Brans-Dicke scalar field, and the Brans-Dicke field plays the role of Newton’s gravitational constant. In more general scalar-tensor theories of gravity, the scalar field $\phi$ has self-interaction and it is usually massive. In 1974, Horndeski found the most general scalar-tensor theory of gravity whose action has higher derivatives of $g_{\mu \nu }$ and $\phi$, but the equations of motion are at most the second order [20]. Even though there are higher derivative terms, there is no Ostrogradsky instability [21], so there are three physical degrees of freedom in Horndeski theory, and we expect that there is an extra polarization state in addition to the plus and cross modes. If the scalar field is massless, then the additional polarization state should be the breathing mode $\varPhi _{22}$.

When the interaction between the quantized matter fields and the classical gravitational field is considered, the quadratic terms $R_{\mu \nu \alpha \beta } R^{\mu \nu \alpha \beta }$ and $R^2$ are needed as counterterms to remove the singularities in the energy-momentum tensor [22]. Although the quadratic gravitational theory is renormalizable [23], the theory has ghost due to the presence of higher derivatives [23, 24]. However, the general nonlinear f(R) gravity [25] is free of ghost and it is equivalent to a scalar-tensor theory of gravity [26, 27]. The effective mass squared of the equivalent scalar field is $f'(0)/3f''(0)$, and the massive scalar field excites both the longitudinal and transverse breathing modes [28,29,30]. The polarizations of gravitational waves in f(R) gravity were previously discussed in [31,32,33,34,35,36,37]. The authors in [33, 34] calculated the NP variables and found that $\varPsi _2$, $\varPsi _4$ and $\varPhi _{22}$ are nonvanishing. They then claimed that there are at least four polarization states in f(R) gravity. Recently, it was pointed out that the direct application of the framework of Eardley et. al. (ELLWW framework) [17, 18] derived for the null plane gravitational waves to the massive field is not correct, and the polarization state of the equivalent massive field in f(R) gravity is the mixture of the longitudinal and the transverse breathing modes [37]. Furthermore, the longitudinal polarization is independent of the effective mass of the equivalent scalar field, so it cannot be used to understand how the polarization reduces to the transverse breathing mode in the massless limit.

Since the polarizations of gravitational waves in alternative theories of gravity are not known in general, so we need to study it [38,39,40]. In this paper, the focus is on the polarizations of gravitational waves in the most general scalar-tensor theory, Horndeski theory. It is assumed that matter minimally couples to the metric, so that test particles follow geodesics. The gravitational wave solutions are obtained from the linearized equations of motion around the flat spacetime background, and the geodesic deviation equations are used to reveal the polarizations of the massive scalar field. The analysis shows that in Horndeski theory, the massive scalar field excites both breathing and longitudinal polarizations. The effect of the longitudinal polarization on the geodesic deviation depends on the mass and it is much smaller than that of the transverse polarization in the aLIGO and LISA frequency bands. In the massless limit, the longitudinal mode disappears, while the breathing mode persists.

The paper is organized as follows. Section 2 briefly reviews ELLWW framework for classifying the polarizations of null gravitational waves. In Sect. 3, the linearized equations of motion and the plane gravitational wave solution are obtained. In Sect. 3.1, the polarization states of gravitational waves in Horndeski theory are discussed by examining the geodesic deviation equations, and Sect. 3.2 discusses the failure of ELLWW framework for the massive Horndeski theory. Section 4 discusses the possible experimental tests of the extra polarizations. In particular, Sect. 4.1 mainly calculates the interferometer response functions for aLIGO, and Section 4.2 determines the cross-correlation functions for the longitudinal and transverse breathing polarizations. Finally, this work is briefly summarized in Sect. 5. In this work, we use the natural units and the speed of light in vacuum $c=1$.

2 Review of ELLWW Framework

Eardley et. al. devised a model-independent framework [17, 18] to classify the null gravitational waves in a generic metric theory of gravity based on the Newman-Penrose formalism [16]. The quasiorthonormal, null tetrad basis $E^\mu _a=(k^\mu ,l^\mu ,$ $m^\mu ,\bar{m}^\mu )$ is chosen to be

$$\begin{aligned} k^\mu= & {} \frac{1}{\sqrt{2}}(1,0,0,1), \end{aligned}$$

(1)

$$\begin{aligned} l^\mu= & {} \frac{1}{\sqrt{2}}(1,0,0,-1), \end{aligned}$$

(2)

$$\begin{aligned} m^\mu= & {} \frac{1}{\sqrt{2}}(0,1,i,0), \end{aligned}$$

(3)

$$\begin{aligned} \bar{m}^\mu= & {} \frac{1}{\sqrt{2}}(0,1,-i,0), \end{aligned}$$

(4)

where bar means the complex conjugation. They satisfy the relation $-k^\mu l_\mu =m^\mu \bar{m}_\mu =1$ and all other inner products vanish. Since the null gravitational wave propagates in the $+z$ direction, the Riemann tensor is a function of the retarded time $u=t-z$, which implies that $R_{abcd,p}=0$, where (a, b, c, d) range over $(k,l,m,\bar{m})$ and $(p,q,r,\cdots )$ range over ($k,m,\bar{m}$). The linearized Bianchi identity and the symmetry properties of $R_{abcd}$ imply that

$$\begin{aligned} R_{ab[pq;l]}=R_{ab[pq,l]} =\frac{1}{3}R_{abpq,l}=0. \end{aligned}$$

(5)

So $R_{abpq}$ is a trivial, nonwavelike constant, and one can set $R_{abpq}=R_{pqab}=0$. One could also verify that $R_{kk}=R_{km}=R_{k\bar{m}}=R_{mm}=R_{m\bar{m}}=R_{\bar{m}\bar{m}}=0$. The nonvanishing components of the Riemann tensor are $R_{plql}$. Since $R_{plql}$ is symmetric in exchanging p and q, there are six independent nonvanishing components and they can be expressed in terms of the following four NP variables,

$$\begin{aligned} \begin{aligned} \varPsi _2=&-\frac{1}{6}R_{klkl},\\ \varPsi _3=&-\frac{1}{2}R_{kl\bar{m}l},\\ \varPsi _4=&-R_{\bar{m}l\bar{m}l},\\ \varPhi _{22}=&-R_{ml\bar{m}l}, \end{aligned} \end{aligned}$$

(6)

and the remaining nonzero NP variables are $\varPhi _{11}=3\varPsi _2/2$, $\varPhi _{12}=\bar{\varPhi }_{21}=\bar{\varPsi }_3$ and $\varLambda =\varPsi _2/2$.¹ $\varPsi _2$ and $\varPhi _{22}$ are real while $\varPsi _3$ and $\varPsi _4$ are complex. So there are exactly six real independent variables, and they can be used to replace the six components of the electric part of the Riemann tensor.

These four NP variables $\{\varPsi _2,\varPsi _3,\varPsi _4,\varPhi _{22}\}$ can be classified according to their transformation properties under the group E(2), the little group of the Lorentz group for massless particles. E(2) consists of local transformations preserving the null vector $k^\mu$, i.e., under E(2) transformation,

$$\begin{aligned} \begin{aligned} \varPsi '_2=&\,\,\varPsi _2,\\ \varPsi '_3=&\,\,e^{-i\vartheta }(\varPsi _3+3\bar{\rho }\varPsi _2),\\ \varPsi '_4=&\,\,e^{-i2\vartheta }(\varPsi _4+4\bar{\rho }\varPsi _3+6\bar{\rho }^2\varPsi _2),\\ \varPhi '_{22}=\,\,&\varPhi _{22}+2\rho \varPsi _3+2\bar{\rho }\bar{\varPsi }_3+6\rho \bar{\rho }\varPsi _2, \end{aligned} \end{aligned}$$

(7)

where $\vartheta \in [0,2\pi )$ represents a rotation around the z direction and the complex number $\rho$ denotes a translation in the Euclidean 2-plane. From these transformations, one finds out that only $\varPsi _2$ is invariant and the amplitudes of the four NP variables are not observer-independent. However, the absence (zero amplitude) of some of the four NP variables is observer-independent. Based on this, six classes are defined as follows [17].

Class II$_6$:

$\varPsi _2\ne 0$. All observers measure the same nonzero amplitude of the $\varPsi _2$ mode, but the presence or absence of all other modes is observer-dependent.

Class III$_5$:

$\varPsi _2=0$, $\varPsi _3\ne 0$. All observers measure the absence of the $\varPsi _2$ mode and the presence of the $\varPsi _3$ mode, but the presence or absence of $\varPsi _4$ and $\varPhi _{22}$ is observer-dependent.

Class N$_3$:

$\varPsi _2=\varPsi _3=0,\,\varPsi _4\ne 0\ne \varPhi _{22}$. The presence or absence of all modes is observer-independent.

Class N$_2$:

$\varPsi _2=\varPsi _3=\varPhi _{22}=0,\,\varPsi _4\ne 0$. The presence or absence of all modes is observer-independent.

Class O$_1$:

$\varPsi _2=\varPsi _3=\varPsi _4=0,\,\varPhi _{22}\ne 0$. The presence or absence of all modes is observer-independent.

Class O$_0$:

$\varPsi _2=\varPsi _3=\varPsi _4=\varPhi _{22}=0$. No wave is observed.

Apparently, Class II$_6$ is the most general one. If $\varPsi _2\ne 0$, then we may see all six modes in some coordinates. By setting successive variables $\{\varPsi _2,\varPsi _3,\varPsi _4,\varPhi _{22}\}$ to zero, one obtains less and less general classes, and eventually, Class O$_0$ which is trivial and represents no wave. These NP variables can also be grouped according to their helicities. By setting $\rho =0$ in Eq. (7), one sees that $\varPsi _2$ and $\varPhi _{22}$ have helicity 0, $\varPsi _3$ has helicity 1 and $\varPsi _4$ has helicity 2.

In order to establish the relation between $\{\varPsi _2,\varPsi _3,\varPsi _4,\varPhi _{22}\}$ and the polarizations of the gravitational wave, one needs to examine the geodesic deviation equation in the Cartesian coordinates [17],

$$\begin{aligned} \ddot{x}^j=\frac{d^2x^j}{dt^2}=-R_{tjtk}x^k, \end{aligned}$$

(8)

where $x^j$ represents the deviation vector between two nearby geodesics and $j,\,k=1,2,3$. The so-called electric part $R_{tjtk}$ of the Riemann tensor is given by the following matrix,

$$\begin{aligned} R_{tjtk}=\left( \begin{array}{ccc} -\frac{1}{2}(\mathfrak {R}\varPsi _4+\varPhi _{22}) &{} \frac{1}{2}\mathfrak {I}\varPsi _4 &{} -2\mathfrak {R}\varPsi _3 \\ \frac{1}{2}\mathfrak {I}\varPsi _4 &{} \frac{1}{2}(\mathfrak {R}\varPsi _4-\varPhi _{22}) &{} 2\mathfrak {I}\varPsi _3 \\ -2\mathfrak {R}\varPsi _3 &{} 2\mathfrak {I}\varPsi _3 &{} -6\varPsi _2 \end{array} \right) , \end{aligned}$$

(9)

where $\mathfrak {R}$ and $\mathfrak {I}$ stand for the real and imaginary parts. So these NP variables can be used to label the polarizations of null gravitational waves. More specifically, $\mathfrak {R}\varPsi _4$ and $\mathfrak {I}\varPsi _4$ represent the plus and the cross polarizations, respectively; $\varPhi _{22}$ represents the transverse breathing polarization, and $\varPsi _2$ represents the longitudinal polarization; finally, $\mathfrak {R}\varPsi _3$ and $\mathfrak {I}\varPsi _3$ represent vector-x and vector-y polarizations, respectively. Figure 1 in Ref. [17] displays how these polarizations deform a sphere of test particles, which will not be reproduced here. In terms of $R_{tjtk}$, the plus mode is specified by $\hat{P}_+=-R_{txtx}+R_{tyty}$, the cross mode is specified by $\hat{P}_\times =R_{txty}$, the transverse breathing mode is represented by $\hat{P}_b=R_{txtx}+R_{tyty}$, the vector-x mode is represented by $\hat{P}_{xz}=R_{txtz}$, the vector-y mode is represented by $\hat{P}_{yz}=R_{tytz}$, and the longitudinal mode is represented by $\hat{P}_l=R_{tztz}$. For null gravitational waves, the four NP variables $\{\varPsi _2,\varPsi _3,\varPsi _4,\varPhi _{22}\}$ with six independent components are related with the six electric components of Riemann tensor, and they provide the six independent polarizations $\{\hat{P}_+,\hat{P}_\times ,\hat{P}_b,\hat{P}_{xz},\hat{P}_{yz},\hat{P}_l\}$. According to the E(2) classification, the longitudinal mode which corresponds to nonzero $\varPsi _2$ belongs to the most general class $\Pi _6$. The presence of the longitudinal mode means that all six polarizations are present in some coordinate systems. Although we obtain the six polarizations according to the E(2) classification for general metric theory of gravity, the polarizations of gravitational waves in alternative theories of gravity are unknown in general, so we need to discuss them case by case.

It is now ready to apply this framework to discuss some specific metric theories of gravity. For example, GR predicts the existence of the plus and the cross polarizations, and it can be easily checked that only $\varPsi _4\ne 0$. For Brans-Dicke theory [19], there is one more polarization, the transverse breathing polarization, excited by the massless scalar field, so in addition to $\varPsi _4\ne 0$, $\varPhi _{22}$ is also nonzero [17]. In particular, $\varPsi _2=0$, so the longitudinal polarization is absent in Brans-Dicke theory. So for Brans-Dicke theory,

$$\begin{aligned} R_{tjtk}^\mathrm {BD}=\left( \begin{array}{ccc} -\frac{1}{2}(\mathfrak {R}\varPsi _4+\varPhi _{22}) &{} \frac{1}{2}\mathfrak {I}\varPsi _4 &{} 0 \\ \frac{1}{2}\mathfrak {I}\varPsi _4 &{} \frac{1}{2}(\mathfrak {R}\varPsi _4-\varPhi _{22}) &{} 0 \\ 0 &{} 0 &{} 0 \end{array} \right) . \end{aligned}$$

(10)

In the next section, the plane gravitational wave solution to the linearized equation of motion will be obtained for Horndeski theory, and the polarization content will be determined. It will show that because of the massive scalar field, the electric part of the Riemann tensor takes a different form from Eq. (10). Its components are expressed in terms of a different set of NP variables.

3 Gravitational wave polarizations in Horndeski theory

In this section, the polarization content of the plane gravitational wave in Horndeski theory will be determined. The action of Horndeski theory is [20],

$$\begin{aligned} S=\int d^4x\sqrt{-g}(L_2+L_3+L_4+L_5), \end{aligned}$$

(11)

where

$$\begin{aligned}&L_2=K(\phi ,X),\quad L_3=-G_3(\phi ,X)\Box \phi , \\&L_4=G_4(\phi ,X)R+G_{4,X}\left[ (\Box \phi )^2-(\nabla _\mu \nabla _\nu \phi )(\nabla ^\mu \nabla ^\nu \phi )\right] , \\&L_5=G_5(\phi ,X)G_{\mu \nu }\nabla ^\mu \nabla ^\nu \phi -\frac{1}{6}G_{5,X}[(\Box \phi )^3\\&\qquad \quad -3(\Box \phi )(\nabla _\mu \nabla _\nu \phi )(\nabla ^\mu \nabla ^\nu \phi )\\&\qquad \quad +2(\nabla ^\mu \nabla _\alpha \phi )(\nabla ^\alpha \nabla _\beta \phi )(\nabla ^\beta \nabla _\mu \phi )]. \end{aligned}$$

Here, $X=-\nabla _\mu \phi \nabla ^\mu \phi /2$, $\Box \phi =\nabla _\mu \nabla ^\mu \phi$, the functions K, $G_3$, $G_4$ and $G_5$ are arbitrary functions of $\phi$ and X, and $G_{j,X}(\phi ,$ $X)=\partial G_j(\phi ,X)/\partial X$ with $j=4,5$. Horndeski theory reduces to several interesting theories as its subclasses by suitable choices of these functions. For example, one obtains GR with $K=G_3=G_5=0$ and $G_4=1/(16\pi G)$. For Brans-Dicke theory, $G_3=G_5=0$, $K=2\omega X/\phi$ and $G_4=\phi$. And finally, to reproduce f(R) gravity, one can set $G_3=G_5=0$, $K=f(\phi )-\phi f'(\phi )$ and $G_4=f'(\phi )$ with $f'(\phi )=d f(\phi )/d\phi$.

The variational principle gives rise to the equations of motion. The full set of equations can be found in Ref. [41], for instance. For the purpose of this work, one expands the metric tensor field $g_{\mu \nu }$ and $\phi$ in the following way,

$$\begin{aligned} g_{\mu \nu }= & {} \eta _{\mu \nu }+h_{\mu \nu }, \end{aligned}$$

(12)

$$\begin{aligned} \phi= & {} \phi _0+\upvarphi , \end{aligned}$$

(13)

where $\phi _0$ is a constant. The equations of motion are expanded up to the linear order in $h_{\mu \nu }$ and $\upvarphi$,

$$\begin{aligned}&-\frac{1}{2}K(0)+ G_{4}(0)G_{\mu \nu }^{(1)}-G_{4,\phi }(0)(\upvarphi _{\mu \nu }-\eta _{\mu \nu }\Box \upvarphi ) =0, \end{aligned}$$

(14)

$$\begin{aligned}&K_{,\phi }(0)+[K_{,X}(0)-2G_{3,\phi }(0)]\Box \upvarphi +K_{,\phi \phi }(0)\upvarphi \nonumber \\&\quad +G_{4,\phi }(0)R^{(1)}= 0, \end{aligned}$$

(15)

where $\Box =\partial ^\mu \partial _\mu$ from now on, $G_4(0)=G_4(\phi _0,0)$, $K(0)=K(\phi _0,0)$, and $G_{\mu \nu }^{(1)}$ and $R^{(1)}$ are the linearized Einstein tensor and Ricci scalar, respectively. In addition, the symbol $K_{,X}(0)$ means the value of $K_{,X}=\partial K/\partial X$ at $\phi =\phi _0$ and $X=X_0=0$, and the remaining symbols can be understood similarly.

In order to obtain the gravitational wave solutions around the flat background, one requires that $g_{\mu \nu }=\eta _{\mu \nu }$ and $\phi =\phi _0$ be the solution to Eqs. (14) and (15), which implies that

$$\begin{aligned} K(0)=0,\quad K_{,\phi }(0)=0. \end{aligned}$$

(16)

Substituting this result into Eqs. (14) and (15), and after some algebraic manipulations, one gets,

$$\begin{aligned}&(\Box -m^2)\upvarphi = 0, \end{aligned}$$

(17)

$$\begin{aligned}&G_{\mu \nu }^{(1)}-\frac{G_{4,\phi }(0)}{G_4(0)}(\partial _\mu \partial _\nu \upvarphi -\eta _{\mu \nu }\Box \upvarphi )=0, \end{aligned}$$

(18)

where $G_4(0)\ne 0$ and the mass squared of the scalar field is

$$\begin{aligned} m^2=-\frac{K_{,\phi \phi }(0)}{K_{,X}(0)-2G_{3,\phi }(0)+3G_{4,\phi }^2(0)/G_{4}(0)}. \end{aligned}$$

(19)

For Brans-Dicke theory, $K_{,\phi \phi }(0)=0$, so $m=0$. For the f(R) theory, one gets $m^2=f'(0)/3f''(0)$, which agrees with the previous work [37].

One can decouple Eq. (18) from Eq. (17) by reexpressing them in terms of the auxiliary tensor field $\tilde{h}_{\mu \nu }$ defined by,

$$\begin{aligned} \tilde{h}_{\mu \nu }=h_{\mu \nu }-\frac{1}{2}\eta _{\mu \nu }\eta ^{\alpha \beta }h_{\alpha \beta }-\eta _{\mu \nu }\sigma \upvarphi , \end{aligned}$$

(20)

with $\sigma =G_{4,\phi }(0)/G_{4}(0)$. The original metric tensor perturbation can be obtained by inverting the above relation, i.e.,

$$\begin{aligned} h_{\mu \nu }=\tilde{h}_{\mu \nu }-\frac{1}{2}\eta _{\mu \nu }\eta ^{\alpha \beta }\tilde{h}_{\alpha \beta }-\eta _{\mu \nu }\sigma \upvarphi . \end{aligned}$$

(21)

Therefore, Eq. (18) becomes

$$\begin{aligned} \partial _\rho \partial _{(\mu }\tilde{h}^\rho _{\nu )}-\frac{1}{2}\Box \tilde{h}_{\mu \nu }-\frac{1}{2}\eta _{\mu \nu }\partial _\rho \partial _\sigma \tilde{h}^{\rho \sigma } =0, \end{aligned}$$

(22)

The equations of motion remain invariant under the gauge transformation,

$$\begin{aligned} \upvarphi ' = \upvarphi ,\quad \tilde{h}'_{\mu \nu } = \tilde{h}_{\mu \nu }-\partial _\mu \xi _\nu -\partial _\nu \xi _\mu +\eta _{\mu \nu }\partial _\rho \xi ^\rho , \end{aligned}$$

(23)

with $x'^\mu =x^\mu +\xi ^\mu$. Therefore, one can choose the transverse traceless gauge $\partial _\nu \tilde{h}^{\mu \nu }=0$, $\eta ^{\mu \nu }\tilde{h}_{\mu \nu }=0$ by using the freedom of coordinate transformation, and the linearized Eqs. (17) and (18) become the wave equations

$$\begin{aligned} (\Box -m^2)\upvarphi= & {} 0, \end{aligned}$$

(24)

$$\begin{aligned} \Box \tilde{h}_{\mu \nu }= & {} 0. \end{aligned}$$

(25)

By the analogue of $\bar{h}_{\mu \nu }=h_{\mu \nu }-\eta _{\mu \nu }\eta ^{\rho \sigma }h_{\rho \sigma }$ in GR, it is easy to see that the field $\tilde{h}_{\mu \nu }$ represents the familiar massless graviton and it has two polarization states: the plus and cross polarizations. The scalar field $\upvarphi$ is massive in general, and it decouples from the massless tensor field $\tilde{h}_{\mu \nu }$. Suppose the massless and the massive modes both propagate in the $+z$ direction with the wave vectors,

$$\begin{aligned} k^\mu =(\varOmega ,0,0,\varOmega ),\quad q^\mu =(q_t,0,0,q_z), \end{aligned}$$

(26)

respectively, where the dispersion relation for the scalar field is $q_t^2-q_z^2=m^2$. The propagation speed of the massive scalar field is $v=\sqrt{q_t^2-m^2}/q_t$. The plane wave solutions to Eqs. (24) and (25) take the following form

$$\begin{aligned} \tilde{h}_{\mu \nu }= & {} e_{\mu \nu }e^{-ik\cdot x}, \end{aligned}$$

(27)

$$\begin{aligned} \upvarphi= & {} \upvarphi _0 e^{-iq\cdot x}, \end{aligned}$$

(28)

where $\upvarphi _0$ and $e_{\mu \nu }$ are the amplitudes of the waves with $k^\nu e_{\nu \mu }=0$ and $\eta ^{\mu \nu }e_{\mu \nu }=0$.

3.1 Polarizations

The polarizations of gravitational wave can be extracted by studying the relative acceleration of two nearby test particles moving in the field of the gravitational wave. One assumes that the matter fields minimally couple with the metric tensor $g_{\mu \nu }$, while there are no direct interactions between the matter fields and the scalar field $\phi$. Therefore, freely falling test particles follow geodesics, and the relative acceleration is given by the linearized geodesic deviation equations Eq. (8). The polarizations of gravitational wave can be understood by placing a sphere of test particles in the spacetime, and studying how this sphere deforms. With the solutions (27) and (28), one calculates the electric part $R_{tjtk}$ for the plane wave solution (27) and (28). Written as a $3\times 3$ matrix, it is given by

$$\begin{aligned} \begin{aligned}&R_{tjtk}=\\&\left( \begin{array}{ccc} -\frac{1}{2}q_t^2\sigma \upvarphi +\frac{1}{2}\varOmega ^2\tilde{h}_{xx} &{} \frac{1}{2}\varOmega ^2\tilde{h}_{xy} &{} 0 \\ \frac{1}{2}\varOmega ^2\tilde{h}_{xy} &{} -\frac{1}{2}q_t^2\sigma \upvarphi -\frac{1}{2}\varOmega ^2\tilde{h}_{xx} &{} 0 \\ 0 &{} 0 &{} -\frac{1}{2}m^2\sigma \upvarphi \end{array}\right) . \end{aligned} \end{aligned}$$

(29)

From this expression, one can easily recognize the familiar plus and cross polarizations by setting $\upvarphi =0$, which leads to a symmetric, traceless matrix with nonvanishing components $R_{txtx}=-R_{tyty}=\varOmega ^2\tilde{h}_{xx}/2$ and $R_{txty}=R_{tytx}=\varOmega ^2\tilde{h}_{xy}/2$. Indeed, $\tilde{h}_{\mu \nu }$ generates the plus and the cross polarizations.

To study the polarizations caused by the scalar field, one sets $\tilde{h}_{\mu \nu }=0$. If the scalar field is massless ($m=0$) as in Brans-Dicke theory, one finds out that $R_{tztz}=0$, and the scalar field excites only the transverse breathing polarization with $R_{txtx}=R_{tyty}$. If the scalar field is massive, one can perform a Lorentz boost such that $q_z=0$, i.e., one works in the rest frame of the scalar field. In the rest frame, from Eq. (29), we find that $q_t^2=m^2$ and $R_{txtx}=R_{tyty}=R_{tztz}\ne 0$. the geodesic deviation equations are,

$$\begin{aligned} \ddot{x}^j=\frac{1}{2}m^2\sigma \upvarphi x^j,\, j=1,2,3. \end{aligned}$$

(30)

Suppose the initial deviation vector between two geodesics is $x^j_{0}=(x_0,y_0,z_0)$, and one integrates the above equations twice to obtain the changes in the deviation vector,

$$\begin{aligned} \delta x^j\approx -\frac{1}{2}\sigma \upvarphi x^j_{0}. \end{aligned}$$

(31)

From the above equation, one finds that if $\sigma$ is independent of the mass m, so are $\delta x^j$. However, for f(R) gravity, $m^2\sigma =1/3$, so $\delta x^j$ is proportional to $1/m^2$ [37]. Eq. (31) implies that the sphere of test particles will oscillate isotropically in all directions, so the massive scalar field excites the longitudinal mode in addition to the breathing mode. Note that in the rest frame of a massive field (such as $\upvarphi$) one cannot take the massless limit, as there is no rest frame for a massless field with the speed of light as its propagation speed. However, the massless limit can be taken in Eq. (29) if the rest frame condition $q_t=m$ is not imposed a prior. In the massless limit, $R_{tztz}=0$ and $R_{txtx}=R_{tyty}\ne 0$.

However, in the actual observation, it is highly unlikely that the test particles, such the mirrors in aLIGO/VIRGO, are at the rest frame of the (massive) scalar gravitational wave. So one should also study how the scalar gravitational wave deforms the sphere when $q_z\ne 0$. In this case, the deviation vector is given by

$$\begin{aligned} \delta x\approx & {} -\frac{1}{2}\sigma \upvarphi x_0, \end{aligned}$$

(32)

$$\begin{aligned} \delta y\approx & {} -\frac{1}{2}\sigma \upvarphi y_0, \end{aligned}$$

(33)

$$\begin{aligned} \delta z\approx & {} -\frac{1}{2}\frac{m^2}{q_t^2}\sigma \upvarphi z_0. \end{aligned}$$

(34)

When the gravitational wave passes by, this sphere deforms as shown in Figs. 1 and 2 for the massive and massless scalars, respectively. In each figure, Panel (a) shows how the sphere deforms in the 3 dimensional space in which the semitransparent ellipsoids are the results of deforming the spheres, and Panels (b) and (c) show the top view and the side view with the dashed ellipses representing the results of the deformation of the circles, respectively. Because of the cylindrical symmetry around the z axis, the xOy cross section is not displayed in each figure. In both figures, one finds out that Panels (b) behave the same, as Eqs. (32) and (33) are independent of the mass. So Panels (b) represent the transverse breathing polarization, as the circle (the intersection of the original sphere with the xOy plane) deforms to a circle of a different size. From Panel (c) in Fig. 1, one finds out that the z coordinates of the test particles change (except the test particles on xOy plane), and the change in z is smaller than that in x (or y). This panel represents the longitudinal polarization. In contrast, Panel (c) in Fig. 2 shows that the z coordinates of the test particles remain the same, so in the massless case, there does not exist the longitudinal polarization. Note that in the above discussion, we distinguish the breathing polarization [described by Eqs. (32) and (33)] from the longitudinal one [described by Eq. (34)] in the massive case, but since they are excited by the same field $\upvarphi$, they represent a single degree of freedom. We thus call the polarization state excited by the scalar a mix polarization of the breathing and the longitudinal polarizations.

Fig. 1

The transverse breathing and the longitudinal polarizations excited by the massive scalar field

Fig. 2

The transverse breathing polarization excited by the massless scalar field

In summary, the polarizations of gravitational waves in Horndeski theory include the plus and cross polarizations induced by the spin 2 field $\tilde{h}_{\mu \nu }$. The scalar field excites both the transverse breathing and longitudinal polarizations if it is massive. However, if it is massless, the scalar field excites merely the transverse breathing polarization. In terms of the basis introduced for the massless fields in [17], there are three polarizations: the plus state $\hat{P}_+$, the cross state $\hat{P}_\times$ and the mix state of $\hat{P}_b$ and $\hat{P}_l$. In the massless limit, the mix state reduces to the pure state $\hat{P}_b$. Note that the vector modes $\hat{P}_{xz}$ and $\hat{P}_{yz}$ are absent in Horndeski theory, this seems to be in conflict with the E(2) classification because the longitudinal mode $\hat{P}_l$ is present, so we need to discuss the application of E(2) classification.

3.2 Newman-Penrose variables

Now we calculate the NP variables with the plane wave solution (27) and (28), we get

$$\begin{aligned} \begin{aligned} \varPsi _2=&\,\,\frac{1}{12}(R_{txtx}+R_{tyty}-2R_{tztz}\\&+2R_{xyxy}-R_{xzxz}-R_{yzyz})+\frac{1}{2}iR_{tzxy}\\ =&\,\,0, \end{aligned} \end{aligned}$$

(35)

and the nonvanishing ones are

$$\begin{aligned} \varPsi _4= & {} -\varOmega ^2(\tilde{h}_{xx}-i\tilde{h}_{xy}), \end{aligned}$$

(36)

$$\begin{aligned} \varPhi _{22}= & {} \frac{(q_t+\sqrt{q_t^2-m^2})^2}{4}\sigma \upvarphi , \end{aligned}$$

(37)

$$\begin{aligned} \varPhi _{00}= & {} \frac{(q_t-\sqrt{q_t^2-m^2})^2}{(q_t+\sqrt{q_t^2-m^2})^2}\varPhi _{22}, \end{aligned}$$

(38)

$$\begin{aligned} \varPhi _{11}= & {} \varLambda =\frac{m^2}{2(q_t+\sqrt{q_t^2-m^2})^2}\varPhi _{22}. \end{aligned}$$

(39)

Note that $\varPsi _2=-R_{tztz}/6$ for null gravitational waves only, in general case we should use Eq. (35). The naive application of the ELLWW framework to the massive case gives the conclusion that Horndeski gravity has the plus, cross and transverse breathing polarizations, because $\varPsi _2=\varPsi _3=0$. Moreover, $\varPhi _{00}$ is absent and both $\varPhi _{11}$ and $\varLambda$ are proportional to $\varPsi _2$ in the ELLWW framework, which is in contradiction with Eqs. (38) and  (39). These indicate the failure of this framework for the massive Horndeski theory. In particular, the absence of $\varPsi _2$ means that there would be no longitudinal polarization if the ELLWW framework were correct. However, the previous discussion in Sect. 3.1 clearly shows the existence of the longitudinal polarization. Therefore, the ELLWW framework cannot be applied to a theory which predicts the existence of massive gravitational waves. The massive scalar field excites both the breathing and longitudinal polarizations.

In fact, $R_{tjtk}$ can be rewritten in terms of NP variables as a matrix displayed below,

$$\begin{aligned} R_{tjtk}= \left( \begin{array}{ccc} \Upsilon -\frac{1}{2}\mathfrak {R}\varPsi _4 &{} \frac{1}{2}\mathfrak {I}\varPsi _4 &{} 0 \\ \frac{1}{2}\mathfrak {I}\varPsi _4 &{} \Upsilon +\frac{1}{2}\mathfrak {R}\varPsi _4 &{} 0 \\ 0 &{} 0 &{} -2(\varLambda +\varPhi _{11}) \end{array}\right) , \end{aligned}$$

(40)

with $\Upsilon =-2\varLambda -\frac{\varPhi _{00}+\varPhi _{22}}{2}$. It is rather different from Eq. (10). From the above expression, it is clear that some linear combinations of $\varPhi _{00}$, $\varPhi _{11}$, $\varPhi _{22}$ and $\varLambda$ correspond to the transverse breathing and the longitudinal polarizations. In particular, $\Upsilon =-2\varLambda -(\varPhi _{00}+\varPhi _{22})/2$ represents the transverse breathing polarization, while $-2(\varLambda +\varPhi _{11})$ represents the longitudinal polarization. The plus and cross polarizations are still represented by $\mathfrak {R}\varPsi _4$ and $\mathfrak {I}\varPsi _4$, respectively. In the massless limit, $\varPhi _{00}=\varPhi _{11}=\varLambda =0$ according to Eqs. (38) and (39), so Eq. (40) takes the same form as Eq. (10) since the ELLWW framework applies in this case. Note that when the scalar field is massive, Eq. (40) actually takes the same form as Eq. (29) by setting $q_z=0$ and $q_t=m$ in Eqs. (36), (37), (38) and (39).

These discussion tells us that the detection of polarizations probes the nature of gravity. If only the plus and cross modes are detected, then gravitation is mediated by massless spin 2 field and GR is confirmed. The detection of the breathing mode in addition to the plus and cross modes means that gravitation is mediated by massless spin 2 and spin 0 fields. If the breathing, plus, cross and longitudinal modes are detected, then gravitation is mediated by massless spin 2 and massive spin 0 fields. For the discussion on the detection of polarizations, please see Refs. [42, 43].

4 Experimental Tests

4.1 Interferometers

In this subsection, the response functions of the interferometers will be computed for the transverse breathing and longitudinal polarizations following the ideas in Refs. [28, 44]. The detection of GW170104 placed an upper bound on the mass $m_g$ of the graviton [3],

$$\begin{aligned} m_g<m_b=7.7\times 10^{-23}\,\mathrm {eV}/c^2. \end{aligned}$$

(41)

In obtaining this bound,the assumed dispersion relation $E^2=p^2c^2+m_g^2$, leads to dephasing $\delta \varPsi (m_g)$ of the waves relative to the phase evolution in GR, which is given by the template of the waveform predicted by GR. The constraint on $\delta \varPsi (m_g)$ from the observations was used to derive the bound on $m_g$. Since there are only the plus and cross polarizations in GR, this bound might not be simply applied to the scalar mode. Whether Eq. (41) can be applied to the scalar graviton is beyond the scope of the present work. Here we assume that the upper bound $m_b$ is applicable to the scalar field and study how the upper bound affects the detector responses to the scalar mode. This bound potentially places severe constraint on the effect of the longitudinal polarization on the geodesic deviation in the massive case in the high frequency band, while the transverse breathing mode can be much stronger as long as the amplitude of scalar field $\upvarphi$ is large enough. Therefore, although there exits the longitudinal mode in the massive scalar-tensor theory, the detection of its effect is likely very difficult in the high frequency band. However, in f(R) gravity, the displacement in the longitudinal direction is independent of the mass and the displacements in the transverse directions become larger for smaller mass, so this mix mode can place strong constraints on f(R) gravity.

In the interferometer, photons emanate from the beam splitter, are bounced back by the mirror and received by the beam splitter again. The round-trip propagation time when the gravitational wave is present is different from that when the gravitational wave is absent. To simplify the calculation of the response functions, the beam splitter is placed at the origin of the coordinate system. Then the change in the round-trip propagation time comes from two effects: the change in the relative distance between the beam splitter and the mirror due to the geodesic deviation, and the distributed gravitational redshift suffered by the photon in the field of the gravitational wave [44].

First, consider the response function for the longitudinal polarization. To this end, project the light in the z direction and place the mirror at $z=L$. The response function is thus given by,

$$\begin{aligned} \begin{aligned} Y_l(f)=&\frac{m^2}{16 \pi ^3 f^3} \Big \{4 \pi f e^{ i2 \pi f L (v+1)}\\&-\frac{e^{4 i \pi f L}}{\left( v-1\right) ^3L} \Big [e^{i2 \pi f L \left( v-1\right) }[2 \pi f L \left( v-1\right) \\&+i2 ]+2 \pi f L \left( v-1\right) -i2 \Big ]\\&+\frac{1}{\left( v+1\right) {}^3L}\Big [2 \pi f L \left( v+1\right) +e^{i2 \pi f L \left( v+1\right) } \\&\times \left[ 2 \pi f L \left( v+1\right) +2 i\right] -i2 \Big ]\Big \}, \end{aligned} \end{aligned}$$

(42)

where $f=q_t/(2\pi )$ is the frequency of gravitational waves and the propagation speed v varies with the frequency. To obtain the response function for the transverse mode, project the light in the x direction and place the mirror at $x=L$. The response function is,

$$\begin{aligned} Y_b(f)=e^{i2 \pi f L}+\frac{e^{i2 \pi f L} [2 \pi f L-\sin (2 \pi f L)]}{2 \pi f L}, \end{aligned}$$

(43)

which is independent of the mass m of the scalar field.

Figure 3 shows the absolute values of the longitudinal and transverse response functions for aLIGO ($L=4$ km) to a scalar gravitational wave with the mass $m=m_b$. Comparing the response functions shows that the transverse response is much larger than that of the longitudinal mode in high frequencies, so the detection of the longitudinal mode becomes very difficult in the high frequency band.

Fig. 3

The absolute values of the longitudinal and transverse response functions $|Y_l(f)|$ and $|Y_b(f)|$ as functions of f for aLIGO ($L=4$ km) to a scalar gravitational wave with the mass $m=m_b$ (dashed red curve). The solid curve denotes $|Y_b(f)|$ and the dashed curve denotes $|Y_l(f)|$

One can also understand the difficulty to use the interferometers to detect the longitudinal polarizations in the following way. Table 1 lists the magnitudes of $|\ddot{x}^j/x^j_0|$ normalized with $\sigma \upvarphi$ for the longitudinal and the transverse modes assuming the graviton mass is also $m_b$ at three different frequencies: 100 Hz, $10^{-3}$ Hz, and $10^{-7}$ Hz. The first frequency lies in the frequency band of the ground-based detector like aLIGO and VIRGO, the second frequency lies in the frequency band of space-borne observatory like LISA, and the last frequency lies in the sensitive band of pulsar timing arrays [45].

Table 1 The dependence of the magnitudes of $|\ddot{x}^j/x^j_0|$ for the longitudinal and transverse modes on the frequencies of gravitational waves in units of Hz$^2\sigma \upvarphi$ assuming the scalar mass is $m_b$

Table 1 shows that the effect of the longitudinal mode on the geodesic deviation is smaller than that of the transverse mode by 19 to 9 orders of magnitude at higher frequencies. But at the lower frequencies, i.e., at $10^{-7}$ Hz, the two modes have similar amplitudes. Therefore, aLIGO/VIRGO and LISA might find it difficult to detect the longitudinal mode, but pulsar time arrays should be able to detect the mix polarization state with both the longitudinal and transverse modes. The results are consistent with those for the massive graviton in a specific bimetric theory [46].

4.2 Pulsar timing arrays

A pulsar is a rotating neutron star or a white dwarf with a very strong magnetic field. It emits a beam of the electromagnetic radiation. When the beam points towards the Earth, the radiation can be observed, which explains the pulsed appearance of the radiation. Millisecond pulsars can be used as stable clocks [47]. When there is no gravitational wave, one can observe the pulses at a steady rate. The presence of the gravitational wave will alter this rate, because it will affect the propagation time of the radiation. This will lead to a change in the time-of-arrival (TOA), called time residual R(t). Time residuals caused by the gravitational wave will be correlated between pulsars, and the cross-correlation function is $C(\theta )=\langle R_a(t)R_b(t)\rangle$, where $\theta$ is the angular separation of pulsars a and b, and the brackets $\langle \,\rangle$ imply the ensemble average over the stochastic background. This enables the detection of gravitational waves and the probe of the polarizations.

The effects of the gravitational wave in GR on the time residuals were first considered in Refs. [48,49,50]. Hellings and Downs [51] proposed a method to detect the effects by using the cross-correlation of the time derivative of the time residuals between pulsars, while Jenet et. al. [52] directly worked with the time residuals instead of the time derivative. The later work was generalized to massless gravitational waves in alternative metric theories of gravity in Ref. [53], and further to massive gravitational waves in Refs. [54, 55]. More works have been done, for example, Refs. [56,57,58,59] and references therein.

In their treatments, it is assumed that all the polarization modes have the same mass, either zero or not. If all polarizations propagate in the $+z$ direction at the speed of light, there will be three linearly independent Killing vector fields, $\chi _1^\mu =\delta ^\mu _1,\,\chi _2^\mu =\delta _2^\mu$, and $\chi _3^\mu =\delta ^\mu _0+\delta ^\mu _3$. Using the conservation of $p_\mu \chi ^\mu _j$ ($j=1,2,3$) for photon’s 4-velocity $p^\mu$ satisfying $p^\nu \nabla _\nu p^\mu =0$, one obtains the change in the locally observed frequency of the radiation and integrates to obtain the time residual R(t) [48, 60, 61]. One could also directly integrate the time component of the photon geodesic equation to obtain the change in the frequency [54, 62]. The later method can be applied to massive case. In the present work, a different method will be used by simply calculating the 4-velocities of the photon and observers on the Earth and the pulsar. Since different polarizations propagate at different speeds, there are not enough linearly independent Killing vector fields. The first method cannot be used any way.

Fig. 4

The gravitational wave is propagating in the direction of $\hat{q}_z$, and the photon is traveling in $-\hat{n}$ direction at the leading order. $\hat{l}$ is perpendicular to $\hat{q}_z$ and in the same plane determined by $\hat{q}_z$ and $\hat{n}$. The angle between $\hat{n}$ and $\hat{l}$ is $\beta$

In order to calculate the time residual R(t) caused by the gravitational wave solution (27) and (28), one sets up a coordinate system shown in Fig. 4, so that when there is no gravitational wave, the Earth is at the origin, and the distant pulsar is assumed to be stationary in the coordinate system and one can always orient the coordinate system such that the pulsar is located at $\mathbf {x}_p=(L\cos \beta ,0,L\sin \beta )$. $\hat{q}_z$ is the unit vector pointing to the direction of the gravitational wave, $\hat{n}$ is the unit vector connecting the Earth to the pulsar, and $\hat{l}=\hat{q}_z\wedge (\hat{n}\wedge \hat{q}_z)/\cos \beta =[\hat{n}-\hat{q}_z(\hat{n}\cdot \hat{q}_z)]/\cos \beta$ is the unit vector parallel to the y axis. In the leading order, i.e., in the absence of gravitational waves, the 4-velocity of the photon is $\underline{u}^\mu =\gamma _0(1,-\cos \beta ,0,-\sin \beta )$ with $\gamma _0=\mathrm {d}t/\mathrm {d}\lambda$ a constant and $\lambda$ an arbitrary affine parameter. The perturbed photon 4-velocity is $u^\mu =\underline{u}^\mu +v^\mu$, and since $g_{\mu \nu }u^\mu u^\nu =0$, one obtains

$$\begin{aligned} v^0=\frac{\gamma _0}{2}\tilde{h}_{11}\cos ^2\beta -v_1\cos \beta -v_3\sin \beta . \end{aligned}$$

(44)

Note that one chooses the gauge such that $e_{11}=-e_{22}$, $e_{12}=e_{21}$ are the only nonvanishing amplitudes for $\tilde{h}_{\mu \nu }$, which can always be made. The photon geodesic equation is

$$\begin{aligned} \begin{aligned} 0=&\frac{\mathrm {d}u^\mu }{\mathrm {d}\lambda }+\varGamma ^\mu {}_{\rho \sigma }u^\rho u^\sigma \\ \approx&\gamma _0\frac{\mathrm {d}u^\mu }{\mathrm {d}t}+\varGamma ^\mu {}_{\rho \sigma }\underline{u}^\rho \underline{u}^\sigma . \end{aligned} \end{aligned}$$

(45)

The calculation shows that

$$\begin{aligned} v^0= & {} \gamma _0\Big \{\sigma \upvarphi _0\cos [(q_t+q_z\sin \beta )t-q_z(L+t_e)\sin \beta ]\nonumber \\&-\frac{e_{11}}{2}(1-\sin \beta )\cos [\varOmega (1+\sin \beta )t\nonumber \\&-\varOmega (L+t_e)\sin \beta ]\Big \}, \end{aligned}$$

(46)

$$\begin{aligned} v_1= & {} \gamma _0\{-\sigma \upvarphi _0\cos \beta \cos [(q_t+q_z\sin \beta )t-q_z(L+t_e)\sin \beta ]\nonumber \\&+e_{11}\cos \beta \cos [\varOmega (1+\sin \beta )t-\varOmega (L+t_e)\sin \beta ]\}, \end{aligned}$$

(47)

$$\begin{aligned} v_2= & {} \gamma _0e_{12}\cos \beta \cos \varOmega [(1+\sin \beta )t-(L+t_e)\sin \beta ], \end{aligned}$$

(48)

$$\begin{aligned} v_3= & {} \gamma _0\Big \{-\sigma \upvarphi _0\sin \beta \cos [(q_t+q_z\sin \beta )t-q_z(L+t_e)\sin \beta ]\nonumber \\&-\frac{e_{11}}{2}(1-\sin \beta )\cos [\varOmega (1+\sin \beta )t\nonumber \\&-\varOmega (L+t_e)\sin \beta ]\Big \}, \end{aligned}$$

(49)

where $t_e$ is the time when the photon is emitted from the pulsar. Eq. (46) is consistent with Eq. (44).

The 4-velocities of the Earth and the pulsar also change due to the gravitational wave. Take the 4-velocity of the Earth for instance. Suppose when the gravitational wave is present, its 4-velocity is given by $u_e^\mu =u_e^0(1,\mathbf {v}_e)$. The normalization of $u^\mu _e$ implies that

$$\begin{aligned} u^0_e=1+\frac{1}{2}h_{00}+O(v_e^2). \end{aligned}$$

(50)

The geodesic equation for the Earth is,

$$\begin{aligned} \begin{aligned} 0=&\frac{\mathrm {d}^2x^\mu }{\mathrm {d}\tau ^2}+\varGamma ^\mu {}_{\rho \nu }\frac{\mathrm {d}x^\rho }{\mathrm {d}\tau }\frac{\mathrm {d}x^\nu }{\mathrm {d}\tau }\\ \approx&(u^0_e)^2\left( \frac{\mathrm {d}^2x^\mu }{\mathrm {d}t^2}+\varGamma ^\mu {}_{00}\right) +u^0_e\frac{\mathrm {d}u^0_e}{\mathrm {d}t}\frac{\mathrm {d}x^\mu }{\mathrm {d}t}, \end{aligned} \end{aligned}$$

(51)

where $\tau$ is the proper time and $x^\mu$ are the coordinates of the Earth. One sets $x=y=0$, which is consistent with Eq. (51). Then, one obtains the following solution

$$\begin{aligned} v^3_e\approx -\frac{q_z}{2q_t}\sigma \upvarphi _0\cos q_t t. \end{aligned}$$

(52)

In addition, with Eq. (50),

$$\begin{aligned} u^0_e=1+\frac{1}{2}\sigma \upvarphi _0\cos q_t t. \end{aligned}$$

(53)

So the 4-velocity of the Earth is approximately

$$\begin{aligned} u^\mu _e=\left( 1+\frac{1}{2}\sigma \upvarphi _0\cos q_t t,0,0,-\frac{q_z}{2q_t}\sigma \upvarphi _0\cos q_t t\right) . \end{aligned}$$

(54)

Similarly, one can obtain the 4-velocity of the pulsar, which is

$$\begin{aligned} \begin{aligned} u^\mu _p=&\left( 1+\frac{1}{2}\sigma \upvarphi _0\cos (q_t t-q_zL\sin \beta ),0,\right. \\&\left. 0,-\frac{q_z}{2q_t}\sigma \upvarphi _0\cos (q_t t-q_zL\sin \beta )\right) , \end{aligned} \end{aligned}$$

(55)

up to linear order. The form of $u^\mu _p$ can be understood, realizing that $\chi _1^\mu$ and $\chi _2^\mu$ are still the Killing vector fields.

Therefore, the frequency of the photon measured by an observer comoving with the pulsar is

$$\begin{aligned} \begin{aligned} f_e=&-u_\mu u^\mu _p\\ =&\,\,\gamma _0\Big [1+\frac{q_t-q_z\sin \beta }{2q_t}\sigma \upvarphi _0\cos (q_t t_e-q_zL\sin \beta )\\&-\frac{e_{11}}{2}(1-\sin \beta )\cos \varOmega (t_e-L\sin \beta )\Big ], \end{aligned} \end{aligned}$$

(56)

and the frequency measured by another observer on the Earth is

$$\begin{aligned} \begin{aligned} f_r=&-u_\mu u^\mu _e\\ =&\gamma _0\Big [1+\frac{q_t-q_z\sin \beta }{2q_t}\sigma \upvarphi _0\cos q_t(t_e+L)\\&-\frac{e_{11}}{2}(1-\sin \beta )\cos \varOmega (t_e+L)\Big ]. \end{aligned} \end{aligned}$$

(57)

The frequency shift is thus given by

$$\begin{aligned} \begin{aligned} \frac{f_e-f_r}{f_r}=&\,\,\frac{q_t-q_z\hat{q}_z\cdot \hat{n}}{2q_t}\sigma \left[ \upvarphi (t-L,L\hat{n})-\upvarphi (t,0)\right] \\&-\frac{\hat{n}^j\hat{n}^k}{2(1+\hat{q}_z\cdot \hat{n})}\left[ \tilde{h}_{jk}(t-L,L\hat{n})-\tilde{h}_{jk}(t,0)\right] , \end{aligned} \end{aligned}$$

(58)

where $t=t_e+L$ is the time when the photon arrives at the Earth at the leading order. This expression (58) can be easily generalized to a coordinate system with an arbitrary orientation and at rest relative to the original frame. It can be checked that in the massless limit, the change in the ratio of frequencies agrees with Eq. (2) in Ref. [53]. The contribution of the massive scalar field also agrees with Eq. (2) in Ref. [55] which was derived provided all polarizations have the same mass.

Eq. (58) gives the frequency shift caused by a monochromatic gravitational wave. Now, consider the contribution of a stochastic gravitational wave background which consists of monochromatic gravitational waves,

$$\begin{aligned}&\tilde{h}_{jk}(t,\mathbf {x})\nonumber \\&\quad =\sum _{P=+,\times }\int _{-\infty }^{\infty }\frac{\mathrm {d}\varOmega }{2\pi }\int \mathrm {d}^2\hat{k}\Big \{\tilde{h}^P(\varOmega ,\hat{k})\epsilon ^P_{jk}\exp [i\varOmega (t-\hat{k}\cdot \mathbf {x})]\Big \}, \end{aligned}$$

(59)

$$\begin{aligned}&\upvarphi (t,\mathbf {x})\nonumber \\&\quad =\int _{-\infty }^{\infty }\frac{\mathrm {d}q_t}{2\pi }\int \mathrm {d}^2\hat{q}_z\Big \{\upvarphi _0(q_t,\hat{q}_z)\exp [i(q_tt-q_z\hat{q}_z\cdot \mathbf {x})]\Big \}, \end{aligned}$$

(60)

where $\tilde{h}^P(\varOmega ,\hat{k})$ is the amplitude of the gravitational wave propagating in the direction $\hat{k}$ at the angular frequency $\varOmega$ with the polarization $P=+$ or $\times$, $\epsilon ^P_{jk}$ is the polarization tensor for the polarization P, and $\upvarphi _0(q_t,\hat{q}_z)$ is the amplitude for the scalar gravitational wave propagating in the direction $\hat{q}_z$ at the angular frequency $q_t$. Let $\hat{i},\,\hat{j},\,\hat{k}$ form a right-handed triad frame so that $\hat{i}\times \hat{j}=\hat{k}$, then $\epsilon ^P_{jk}$ is given by

$$\begin{aligned} \epsilon ^+_{jk}=\hat{i}_j\hat{i}_k-\hat{j}_j\hat{j}_k,\quad \epsilon ^\times _{jk}=\hat{i}_j\hat{j}_k+\hat{j}_j\hat{i}_k. \end{aligned}$$

(61)

Assuming the gravitational wave background is isotropic, stationary and independently polarized, one defines the characteristic strains $\tilde{h}^P_c$ and $\upvarphi _c$ in the following way,

$$\begin{aligned}&\langle \tilde{h}^{P*}(\varOmega ,\hat{k})\tilde{h}^{P'}(\varOmega ',\hat{k}')\rangle \nonumber \\&\quad =\delta (\varOmega -\varOmega ')\delta (\hat{k}-\hat{k}')\delta ^{PP'}\frac{\pi |\tilde{h}_c^P|^2(\varOmega )}{4\varOmega }, \end{aligned}$$

(62)

$$\begin{aligned}&\langle \upvarphi _0^*(q_t,\hat{q}_z)\upvarphi _0(q'_t,\hat{q}'_z)\rangle =\delta (q_t-q'_t)\delta (\hat{q}_z-\hat{q}'_z)\frac{|\upvarphi _c|^2(q_t)}{q_t},\nonumber \\ \end{aligned}$$

(63)

where the star $*$ indicates the complex conjugation.

Since the gravitational wave with the plus or cross polarization behaves exactly the same way as in GR which can be found in Ref. [53], the focus will be on the contribution of the scalar field in the following discussion. The total time residual in TOA caused by the stochastic gravitational wave background is

$$\begin{aligned} R(T)=\int _{-\infty }^{\infty }\frac{\mathrm {d}q_t}{2\pi }\int \mathrm {d}^2\hat{q}_z\int _{0}^{T}\mathrm {d}t\frac{f_e-f_r}{f_r}, \end{aligned}$$

(64)

where the argument T is the total observation time. Substituting Eq. (58) (ignoring the second line) in, one obtains

$$\begin{aligned} \begin{aligned} R(T)=&\int _{-\infty }^{\infty }\frac{\mathrm {d}q_t}{2\pi }\int \mathrm {d}^2\hat{q}_z\frac{q_t-q_z\hat{q}_z\cdot \hat{n}}{i2q_t^2}\upvarphi _0(q_t,\hat{q}_z)(e^{iq_tT}-1)\\&\times [1-e^{-i(q_t+q_z\hat{q}_z\cdot n)L}]. \end{aligned} \end{aligned}$$

(65)

Therefore, one can now consider the correlation between two pulsars a and b which are located at positions $\mathbf {x}_a=L_1\hat{n}_1$ and $\mathbf {x}_b=L_2\hat{n}_2$, respectively. The angular separation is $\theta =\arccos (\hat{n}_1\cdot \hat{n}_2)$. The cross-correlation function is thus given by

$$\begin{aligned} \begin{aligned} C(\theta )=&\,\,\langle R_a(T)R_b(T)\rangle \\ =&\,\,\int _{m}^{\infty }\mathrm {d}q_t\int \mathrm {d}^2\hat{q}_z\frac{(q_t-q_z\hat{q}_z\cdot \hat{n}_1)(q_t-q_z\hat{q}_z\cdot \hat{n}_2)}{4\pi ^2 q_t^5}\\&\times |\upvarphi _c|^2(q_t)\mathscr {P}, \end{aligned} \end{aligned}$$

(66)

where

$$\begin{aligned} \mathscr {P}=1-\cos \varDelta _1-\cos \varDelta _2+\cos (\varDelta _1-\varDelta _2), \end{aligned}$$

(67)

and $\varDelta _j=(q_t+q_z\hat{q}_z\cdot \hat{n}_j)L_j$ with $j=1,2$. In obtaining this result, one uses Eq. (63), and takes the real part. In addition, T drops out, since the ensemble average also implies the average over the time [53].

Since the gravitational wave background is assumed to be isotropic, one can calculate $C(\theta )$ by setting

$$\begin{aligned} \hat{n}_1= & {} (0,0,1), \end{aligned}$$

(68)

$$\begin{aligned} \hat{n}_2= & {} (\sin \theta ,0,\cos \theta ). \end{aligned}$$

(69)

Also, let $\hat{q}_z=(\sin \theta _g\cos \phi _g,\sin \theta _g\sin \phi _g,\cos \theta _g)$, so

$$\begin{aligned} \varDelta _1= & {} (q_t+q_z\cos \theta _g)L_1, \end{aligned}$$

(70)

$$\begin{aligned} \varDelta _2= & {} [q_t+q_z(\sin \theta _g\cos \phi _g\sin \theta +\cos \theta _g\cos \theta )]L_2, \end{aligned}$$

(71)

and

$$\begin{aligned} \begin{aligned} C(\theta )=&\int _{m}^{\infty }\mathrm {d}q_t\int \mathrm {d}\theta _g\mathrm {d}\phi _g\sin \theta _g\frac{|\upvarphi _c|^2(q_t)}{4\pi ^2q_t^5}(q_t-q_z\cos \theta _g)\\&[q_t-q_z(\sin \theta _g\cos \phi _g\sin \theta +\cos \theta _g\cos \theta )]\mathscr {P}. \end{aligned} \end{aligned}$$

(72)

In the observation, pulsars are far away enough, so that $q_tL_j\gg 1$. This implies that one can approximate $\mathscr {P}\approx 1$ when $\theta \ne 0$, as the phases in cosines in the definition (67) of $\mathscr {P}$ oscillate fast enough. The integration can be partially done, yielding

$$\begin{aligned} C(\theta )\approx \int _{m}^{\infty }\mathrm {d}q_t\frac{|\upvarphi _c|^2(q_t)}{\pi q_t^3}\left( 1+\frac{q_z^2}{3q_t^2}\cos \theta \right) . \end{aligned}$$

(73)

But for $\theta =0$, one cannot simply set cosines in $\mathscr {P}$ to 1. In this case, one actually considers the auto-correlation function, so $\hat{n}_1=\hat{n}_2$ and $L_1=L_2=L$. The auto-correlation function is thus given by

$$\begin{aligned} \begin{aligned} C(0)=&2\int _{m}^{\infty }\mathrm {d}q_t\frac{|\upvarphi _c|^2(q_t)}{\pi q_t^3}\left[ 1+\frac{q_z^2}{3q_t^2}+\frac{2}{q_t^2L^2}\cos q_tL\cos q_zL\right. \\&\left. +\left( \frac{2}{q_tL}-\frac{1}{q_zL}+\frac{2}{q_t^2q_zL^3}-\frac{q_z}{q_t^2L}\right) \cos q_tL\sin q_zL\right. \\&\left. -\frac{2}{q_tq_zL^2}\sin q_tL\sin q_zL\right] . \end{aligned} \end{aligned}$$

(74)

Note that the observation time T sets a natural cutoff for the frequency, i.e., $q_t\ge 2\pi /T$. So the lower integration limits in Eqs. (73) and (74) should be replaced by $\text {Min}\{m,2\pi /T\}$.

Usually, one assumes that $\upvarphi _c(q_t)$ takes a form of $\upvarphi _c(q_t)\propto (q_t/q_t^c)^\alpha$ with $q_t^c$ some characteristic angular frequency. $\alpha$ is called the power-law index, and usually, $\alpha =0,\,-2/3$ or $-1$ [53, 63]. One can numerically integrate Eqs. (73) and (74) to obtain the so-called normalized correlation function $\zeta (\theta )=C(\theta )/C(0)$. In the integration, suppose the observation time $T=5$ years and the mass of the scalar field is $m_b=7.7\times 10^{-23}$ eV$/c^2$ [3]. The distance L takes a large enough value so that $q_tL\gg 1$ is satisfied. This gives rise to the right panel in Fig. 5, where the power-law index $\alpha$ takes different values. Together shown are the normalized correlation functions for the plus and cross polarizations (labeled by “GR”), and for the transverse breathing polarization (labeled by “Breathing”) in the left panel. The normalized correlation function for the transverse breathing polarization can be obtained simply by setting the mass $m=0$. It is clear that $\zeta (\theta )$ behaves very differently for $\tilde{h}_{\mu \nu }$ and $\upvarphi$, so it is possible to determine whether there are polarizations induced by the scalar field using pulsar timing arrays. Note that since L is large enough, $\zeta (\theta )$ barely changes with L.

Fig. 5

The normalized cross-correlation functions $\zeta (\theta )=C(\theta )/C(0)$. The upper panel shows the cross-correlations when the scalar field is massless, i.e., when there is no longitudinal polarization. The solid curve is for familiar GR polarizations (i.e., the plus or cross ones), and the dashed curve for the transverse breathing polarization. The lower panel shows the cross-correlations induced together by the transverse breathing and longitudinal polarizations when the mass of the scalar field is taken to be $m_b=7.7\times 10^{-23}\,\mathrm {eV}/c^2$. The calculation was done assuming $T=5$ yrs

One could also vary the mass of the scalar field. The result is shown in Fig. 6. This figure displays $\zeta (\theta )$ for six different multiples of $m_b$, including $m=0$ labeled by “Breathing”. The power-law index $\alpha$ is chosen to be 0. It shows that at large angles, $\zeta (\theta )$ is very sensitive to smaller masses with $m_b/5\lesssim m\lesssim m_b$, but for $m\gtrsim m_b$ or $m\lesssim m_b/5$, $\zeta (\theta )$ almost remains the same. Therefore, the measurement of the correlation can be used to constrain the mass of the scalar field.

Fig. 6

$\zeta (\theta )$ as a function of scalar mass m at $\alpha =0$. The calculations were also done for a 5-year observation

In Ref. [55], Lee also analyzed the time residual of TOA caused by massive gravitational waves and calculated the cross-correlation functions. His results (the right two panels in his Fig. 1) differ from those on the right panel in Fig. 5, because in his treatment, the longitudinal and the transverse polarizations were assumed to be independent. In Horndeski theory, however, it is not allowed to calculate the cross-correlation function separately for the longitudinal and the transverse polarizations, as they are both excited by the same field $\upvarphi$ and the polarization state is a single mode.

5 Conclusion

This work analyzes the gravitational wave polarizations in the most general scalar-tensor theory of gravity, Horndeski theory. It reveals that there are three independent polarization modes: the mixture state of the transverse breathing $\hat{P}_b=R_{txtx}+R_{tyty}$ and longitudinal $\hat{P}_l=R_{tztz}$ polarizations for the massive scalar field, and the usual plus $\hat{P}_+=-R_{txtx}+R_{tyty}$ and cross $\hat{P}_\times =R_{txty}$ polarizations for the massless gravitons. These results are consistent with the three propagating degrees of freedom in Horndeski theory. Since the propagation speed of the massive gravitational wave depends on the frequency and is smaller than the speed of light, the massive mode will arrive at the detector later than the massless gravitons. In addition to the difference of the propagation speed, the presence of both the longitudinal and breathing states without the vector-x and vector-y states are also the distinct signature of massive scalar degree of freedom for graviton.

Using the NP variables, we find that $\varPsi _2=0$. For null gravitational waves, this means that the longitudinal mode does not exist. However, our results show that the longitudinal mode exists in the massive case even though $\varPsi _2=0$. We also find that the NP variable $\varPhi _{00}\ne 0$, and $\varPhi _{00}$, $\varPhi _{11}$ and $\varLambda$ are all proportional to $\varPhi _{22}$. These results are in conflict with those for massless gravitational waves in [17], so the results further support the conclusion that the classification of the six polarizations for null gravitational waves derived from the little group E(2) of the Lorentz group is not applicable to the massive case. Although the longitudinal mode exists for the massive scalar field, it is difficult to be detected in the high frequency band because of the suppression by the extremely small graviton mass upper bound. Compared with aLIGO/VIRGO and LISA, pulsar timing arrays might be the primary tool to detect both the transverse breathing and longitudinal modes due to the massive scalar field. In the massless case, the longitudinal mode disappears and the mix state reduces to the pure transverse breathing mode. Brans-Dicke theory and f(R) gravity are subclasses of Horndeski theory, so the general results obtained can be applied to those theories. Despite the fact that in f(R) gravity, the longitudinal mode does not depend on the mass of graviton, its magnitude is much smaller than the transverse one in the high frequency band, which makes its detection unlikely by the network of aLIGO/VIGO and LISA, too. It is interesting that the transverse mode becomes stronger for smaller graviton mass in the f(R) gravity, so the detection of the mixture state can place strong constraint on f(R) gravity.

For null gravitational waves, the presence of the longitudinal mode means that all six polarizations can be detected in some coordinate systems. For Horndeski theory, we find that the vector modes are absent even though the longitudinal mode is present. Since the massive scalar field excites both the breathing and longitudinal polarizations, while the massless scalar field excites the breathing mode only, the detection of polarizations can be used to understand the nature of gravity. If only the plus and cross modes are detected, then gravitation is mediated by massless spin 2 field and GR is confirmed. The detection of the breathing mode in addition to the plus and cross modes means that gravitation is mediated by massless spin 2 and spin 0 fields. If the breathing, plus, cross and longitudinal modes are detected, then gravitation is mediated by massless spin 2 and massive spin 0 fields.