1 Introduction

Soon after the birth of General Relativity (GR), several alternative theories of gravity were proposed. The discovery of the accelerated expansion of the Universe [1, 2] revives the pursuit of these alternatives because the extra fields might account for the dark energy. Since Sep 14th, 2015, LIGO/Virgo collaborations have detected ten gravitational wave (GW) events [3,4,5,6,7,8,9]. This opens a new era of probing the nature of gravity in the highly dynamical, strong-field regime. Due to the extra fields, alternatives to GR generally predict that there are extra GW polarizations in addition to the plus and cross ones in GR. So the detection of the polarization content is very essential to test whether GR is the theory of gravity. In GW170814, the polarization content of GWs was measured for the first time, and the pure tensor polarizations were favored against pure vector and pure scalar polarizations [6]. Similar results were reached in the recent analysis on GW170817 [10]. More interferometers are needed to finally pin down the polarization content. Other detection methods might also determine the polarizations of GWs such as pulsar timing arrays (PTAs) [11,12,13,14].

Alternative metric theories of gravity may not only introduce extra GW polarizations, but also violate the strong equivalence principle (SEP) [15].Footnote 1 The violation of strong equivalence principle (vSEP) is due to the extra degrees of freedom, which indirectly interact with the matter fields via the metric tensor. This indirect interaction modifies the self-gravitating energy of the objects and leads to vSEP [17]. The self-gravitating objects no longer move along geodesics, even if there is only gravity acting on them, and the relative acceleration between the nearby objects does not follow the geodesic deviation equation. In the usual approach, one assumes that the test particles, such as the mirrors in the aLIGO, move along geodesics, so their relative acceleration is given by the geodesic deviation equation. Since the polarization content of GWs is determined by examining the relative acceleration, the departure from the geodesic motion might effectively result in different polarization contents, which can be detected by PTAs. Thus, the main topic of this work is to investigate the effects of vSEP on the polarization content of GWs and the observation of PTAs.

To be more specific, the focus is on the vSEP in the scalar-tensor theory, which is the simplest alternative metric theory of gravity. The scalar-tensor theory contains one scalar field \(\phi \) besides the metric tensor field \(g_{\mu \nu }\) to mediate the gravitational interaction. Because of the trivial transformation of the scalar field under the diffeomorphism, there are a plethora of scalar-tensor theories, such as Brans–Dicke theory [18], Einstein-dilaton-Gauss–Bonnet gravity (EdGB) [19] and f(R) gravity [20,21,22]. In 1974, Horndeski constructed the most general scalar-tensor theory [23]. Its action contains higher derivatives of \(\phi \) and \(g_{\mu \nu }\), but still gives rise to at most the second order differential field equations. So the Ostrogradsky instability is absent in this theory [24]. In fact, Horndeski theory includes previously mentioned theories as its subclasses. In this work, the vSEP in Horndeski theory will be studied.

Among the effects of vSEP, Nordtvedt effect is well-known for a long time [25, 26], and happens in the near zone of the source of the gravitational field. It leads to observable effects. For example, the Moon’s orbit around the Earth will be polarized when they are moving in the gravitational field generated by the Sun [27, 28]. The polarization of the Moon’s orbit has been constrained by the lunar laser ranging experiments [29], which gave the Nordtvedt parameter [30]

$$\begin{aligned} \eta _\text {N}=(0.6\pm 5.2)\times 10^{-4}, \end{aligned}$$
(1)

which measures vSEP in the following way,

$$\begin{aligned} \frac{m_g}{m_i}=1+\eta _\text {N}\varepsilon _\text {grav.}+O(\varepsilon _\text {grav.}^2), \end{aligned}$$
(2)

with \(m_g\) and \(m_i\) the gravitational and the inertial masses, and \(\varepsilon _\text {grav.}\) the ratio of the gravitational binding energy to the inertial energy. A similar polarization of the orbit of the millisecond pulsar-white dwarf (MSP-WD) system also happens due to the gravitational field of the Milky Way [31, 32]. In contrast with the Moon and the Earth, pulsars have large gravitational binding energies, so the observation of the orbit polarization of MSP-WD systems set constraints on vSEP in the strong field regime, which was discussed in Ref. [32]. The observation of a triple pulsar PSR J0337\(+\)1715 was used to set \(\varDelta =(-1.09\pm 0.74)\times 10^{-6}\) [33]. The vSEP also leads to the dipole gravitational radiation, and the variation of Newton’s constant G [34]. The dipole gravitational radiation for Horndeski theory has been studied in Ref. [35], and constraints on this theory were obtained. The pulsar timing observation of the binary system J1713+0747 has leads to \(\dot{G}/G=(-0.1\pm 0.9)\times 10^{-12}\text { yr}^{-1}\) and \(|\varDelta |<0.002\) [36].

As discussed above, none of the previous limits on vSEP was obtained directly using the GW. So probing vSEP by measuring the GW polarizations provides a novel way to test GR in the high speed and dynamical regime. It will become clear that although the vSEP will effectively enhance the longitudinal polarization, it is still very difficult for aLIGO to detect the effects of the longitudinal polarization, as the vSEP by the mirror is extremely weak. In contrast, neutron stars are compact objects with non-negligible self-gravitating energies. The vSEP by neutron stars is strong enough that the stochastic GW background will affect their motions, which is reflected in the cross-correlation function for PTAs [37,38,39,40]. By measuring the cross-correlation function, it is probably easier to detect the presence of vSEP. For this purpose, one only has to observe the change in the arriving time of radial pulses from neutron stars without requiring the neutron stars be in binary systems.

This work is organized as follows. Section 2 reviews the derivation of the geodesic deviation equation, and a generalized deviation equation for accelerated particles is discussed in Sect. 3. Section 4 derives the motion of a self-gravitating object in presence of GWs in Horndeski theory. The polarization content of GWs in Horndeski theory is revisited by taking the vSEP into account in Sect. 5. The generalized deviation equation is computed to reveal the polarization content of GWs. Section 6 calculates the cross-correlation function for PTAs due to GWs. Finally, Sect. 7 briefly summarizes this work. Penrose’s abstract index notation is used [41]. The units is chosen such that the speed of light \(c=1\) in vacuum.

2 Geodesic deviation equation

This section serves to review the idea to derive the geodesic deviation equation following Ref. [42]. In the next section, the derivation will be generalized to accelerated objects straightforwardly.

Let \(\gamma _s(t)\) represent a geodesic congruence, in which each geodesic is parameterized by t and labeled by s. Define the following tangent vector fields,

$$\begin{aligned} T^a=\left( \frac{\partial }{\partial t}\right) ^a,\quad S^a=\left( \frac{\partial }{\partial s}\right) ^a. \end{aligned}$$
(3)

\(S^a\) is called the deviation vector. Their commutator vanishes,

$$\begin{aligned} T^b\nabla _b S^a=S^b\nabla _bT^a. \end{aligned}$$
(4)

With a suitable parametrization, one requires that \(T^b\nabla _b T^a=0\) so that t is an affine parameter. Note that it is not necessary to set \(T^aT_a=-1\) for the following discussion. Whenever desired, one can always reparameterize to normalize it. It is now ready to derive the geodesic deviation equation,

$$\begin{aligned} A^a_\text {rel}=T^c\nabla _c(T^b\nabla _bS^a)=-R_{cbd}{}^aT^cS^bT^d, \end{aligned}$$
(5)

using Eq. (4). For details of derivation, please refer to Ref. [42].

The deviation vector \(S^a\) is not unique. A new parametrization of the geodesics,

$$\begin{aligned} t\rightarrow t'=\alpha (s)t+\beta (s), \end{aligned}$$
(6)

results in the change in \(S^a\) by a multiple of \(T^a\),

$$\begin{aligned} T'^a=\frac{T^a}{\alpha (s)},\quad S'^a=S^a+\frac{\mathrm {d}}{\mathrm {d}s'}\left( \frac{t'-\beta (s')}{\alpha (s')}\right) T^a. \end{aligned}$$
(7)

Therefore, there is a gauge freedom in choosing the deviation vector field \(S^a\). This gauge freedom will be used frequently below to simplify the analysis.

Firstly, there is a parametrization such that \(T_aT^a\) is a constant along the coordinate lines of the constant t, i.e., the integral curves of \(S^a\). In fact, one knows that,

$$\begin{aligned} S^b\nabla _b(T_aT^a)=2T_aS^b\nabla _bT^a=2T_aT^b\nabla _bS^a, \end{aligned}$$
(8)

and under the reparameterization (6), one gets

$$\begin{aligned} S'^b\nabla _b(T'_aT'^a)\!=\!\frac{2}{\alpha (s)^2}T_aT^b\nabla _b\left[ S^a\!+\!\frac{\partial }{\partial s}\left( \frac{t'-\beta (s)}{\alpha (s)}\right) T^a\right] ,\nonumber \\ \end{aligned}$$
(9)

so it is always possible to choose a parametrization to achieve that \(S'^b\nabla _b(T'_aT'^a)=0\). Physically, this means that all geodesics are parameterized by the “same” affine parameter \(t'\). Secondly, under the above parametrization, the inner product \(T^aS_a\) can be made constant along the geodesics,

$$\begin{aligned} \begin{aligned} T^b\nabla _b(T^aS_a)&=T_aT^b\nabla _bS^a\\&=T_aS^b\nabla _bT^a\\&=\frac{1}{2}S^b\nabla _b(T^aT_a)\\&=0. \end{aligned} \end{aligned}$$
(10)

An initial choice of \(T^aS_a=0\) will be preserved along the t coordinate line, so that \(S^a\) is always a spatial vector field for an observer with 4-velocity \(u^a=T^a/\sqrt{-T_bT^b}\) along its trajectory.

From the derivation, one should be aware that the geodesic deviation equation (5) is independent of the gauge choices made above, which only serves to make sure \(S^a\) is always a spatial vector relative to an observer with \(u^a\). In this way, there is no deviation in the time coordinate, that is, no time dilatation. This is because one concerns the change in the spatial distance between two nearby particles measured by either one of them.

3 Non-geodesic deviation equation

When particles are accelerated, they are not moving on geodesics. This happens when there are forces acting on these particles. This also happens for self-gravitating particles in the modified gravity theories, such as the scalar–tensor theory. Suppose a bunch of particles are accelerated and therefore, their velocities satisfy the following relations,

$$\begin{aligned} T^b\nabla _bT^a=A^a, \end{aligned}$$
(11)

with \(A^a \) the 4-acceleration and not proportional to \(T^a\). In the following, \(T^a\) is assumed to be some arbitrary timelike vector field which is not necessarily the 4-velocity of some particle. In this general discussion, the only assumption is that \(T^a\) satisfies Eq. (11). Now, the non-geodesic deviation equation can be derived similarly,

$$\begin{aligned} A^a_\mathrm {rel} =-R_{cbd}{}^aT^cS^bT^d+S^b\nabla _bA^a. \end{aligned}$$
(12)

Again, the derivation of this result does not reply on the gauge fixing made similarly in the previous section or the one to be discussed below. Compared with Eq. (5), there is one extra term, which is due to the fact that the trajectories are no longer geodesics. This equation and a more general one were derived in Ref. [43] using the definitions of curvature and torsion. The authors did not discuss the suitable gauge for extracting physical results which will be presented below.

If \(T^aA_a\ne 0\), one can reparameterize the integral curves of \(T^a\) to make it vanish. Indeed, a reparameterization \(t\rightarrow t'=\kappa (t)\) leads to

$$\begin{aligned} \begin{aligned} A'^a&=T'^b\nabla _bT'^a\\&=\frac{A^a }{\dot{\kappa }^2}-\frac{\ddot{\kappa }}{\dot{\kappa }^3}T^a, \end{aligned} \end{aligned}$$
(13)

where dot denotes the derivative with respect to t. So one can always find a new parametrization which annihilates \(T'^aA'_a\), that is,

$$\begin{aligned} \kappa (t)=\alpha \int \exp \left( \frac{A^a T_a}{T^bT_b}t\right) \mathrm {d}t+\beta , \end{aligned}$$
(14)

with \(\alpha ,\,\beta \) integration constants. From now on, \(T^aA_a=0\) is assumed which implies that

$$\begin{aligned} T^b\nabla _b(T^aT_a)=0. \end{aligned}$$
(15)

So although t may not be the proper time \(\tau \), it is a linear function of \(\tau \). A further reparameterization \(t'=\alpha 't+\beta '\) does not change the above relation.

Now, pick a congruence of these trajectories \(\sigma _s(t)\). So as in the previous section, \(\sigma _s(t)\)’s also lie on a 2-dimensional surface \(\Sigma \) parameterized by (ts). There also exists the similar gauge freedom to that discussed in Sect. 2, except that \(A^a \) depends on the gauge choice. For example, a reparametrization \(t\rightarrow t'=\alpha (s)t+\beta (s)\) results in changes in \(S^a\) (given by Eq. (7)) and \(A^a \), i.e., \(A^a \rightarrow A^a /\alpha ^2(s)\).

With this gauge freedom, one also chooses a suitable gauge such that \(T^aS_a\) remains constant along each trajectory. In fact, it can be shown that

$$\begin{aligned} T^b\nabla _b(T^aS_a)=S_aA^a +\frac{1}{2}S^b\nabla _b(T^aT_a). \end{aligned}$$
(16)

One requires that \(T^aS_a=0\) along the integral curves of \(T^a\), i.e., \(T^b\nabla _b(T^aS_a)=0\). This implies that

$$\begin{aligned} S^b\nabla _b(T^aT_a)=-2S_aA^a . \end{aligned}$$
(17)

This expression means that if the trajectory \(\sigma _0(t)\) is parameterized by the proper time \(t=\tau \), a nearby trajectory \(\sigma _s(t)\) with \(s\ne 0\) will not be parameterized by its proper time, in general. It is necessary to choose this particular gauge as \(S^a\) can be viewed as a spatial vector field relative to \(T^a\) as long as \(T^a\) can be interpreted as the 4-velocity of an observer.

3.1 Fermi normal coordinates

In this subsection, the relative acceleration will be expressed in the Fermi normal coordinate system of the observer \(\sigma _0(\tau )\) with \(\tau \) the proper time. Let the observer \(\sigma _0(\tau )\) carry a pseudo-orthonormal tetrad \(\{(e_{{\hat{0}}})^a=u^a,(e_{{\hat{1}}})^a,(e_{{\hat{2}}})^a,(e_{{\hat{3}}})^a\}\), which satisfies \(g_{ab}(e_{{\hat{\mu }}})^a(e_{{\hat{\nu }}})^b=\eta _{\hat{\mu }{\hat{\nu }}}\) and is Fermi-Walker transported along \(\sigma _0(\tau )\). The observer \(\sigma _0(\tau )\) will measure the deviation in its own proper reference frame, in which the metric takes the following form [44],

$$\begin{aligned} \mathrm {d}s^2=-(1+2A_{{\hat{j}}}x^{{\hat{j}}})\mathrm {d}\tau ^2+\delta _{{\hat{j}}{\hat{k}}}\mathrm {d}x^{{\hat{j}}}\mathrm {d}x^{{\hat{k}}}+O(|x^{{\hat{j}}}|^2), \end{aligned}$$
(18)

where \(j,\,k=1,2,3\) and the acceleration of \(\sigma _0(\tau )\) has no time component (\(A^{{\hat{0}}}=-u_aA^a =0\)). Similarly, \(S^a=S^{{\hat{j}}}(e_{{\hat{j}}})^a\), so the relative acceleration has the following spatial components

$$\begin{aligned} \begin{aligned} A_\text {rel}^{{\hat{j}}}&=-R_{{\hat{0}}{\hat{k}}{\hat{0}}}{}^{{\hat{j}}}S^{{\hat{k}}}+S^{{\hat{k}}}\nabla _{{\hat{k}}}A^{{\hat{j}}}\\&=-R_{{\hat{0}}{\hat{k}}{\hat{0}}}{}^{{\hat{j}}}S^{{\hat{k}}}+S^{{\hat{k}}}\partial _{{\hat{k}}}A^{{\hat{j}}}, \end{aligned} \end{aligned}$$
(19)

since the only nonvanishing components of the Christoffel symbol are

$$\begin{aligned} \varGamma ^{{\hat{0}}}{}_{{\hat{0}}{\hat{j}}}=\varGamma ^{{\hat{j}}}{}_{{\hat{0}}{\hat{0}}}=A_{{\hat{j}}}. \end{aligned}$$
(20)

The relative acceleration can also be expanded as

$$\begin{aligned} \begin{aligned} A^{{\hat{j}}}_\mathrm {rel}&=u^{{\hat{\mu }}}\nabla _{{\hat{\mu }}}(u^{{\hat{\nu }}}\nabla _{{\hat{\nu }}}S^{{\hat{j}}})\\&=\frac{\mathrm {d}^2S^{{\hat{j}}}}{\mathrm {d}\tau ^2}+A^{{\hat{j}}}A_{{\hat{k}}}S^{{\hat{k}}}. \end{aligned} \end{aligned}$$
(21)

Therefore, one gets

$$\begin{aligned} \frac{\mathrm {d}^2S^{{\hat{j}}}}{\mathrm {d}\tau ^2}=-R_{{\hat{0}}{\hat{k}}{\hat{0}}}{}^{{\hat{j}}}S^{{\hat{k}}}+S^{{\hat{k}}}\partial _{{\hat{k}}}A^{{\hat{j}}}-A^{{\hat{j}}}A_{{\hat{k}}}S^{{\hat{k}}}. \end{aligned}$$
(22)

Similar expression was also found in Ref. [45]. Due to the requirement \(T^b\nabla _b(S^aT_a)=0\), one knows that \(\mathrm {d}^2S^{{\hat{0}}}/\mathrm {d}\tau ^2=T^c\nabla _c[T^b\nabla _b(S^aT_a)]=0\), so \(S^{{\hat{0}}}=0\) is really preserved while \(S^a\) is propagated along the integral curves of \(T^a\). Whenever the observer \(\sigma _0(\tau )\) is moving on a geodesic, \(A^a=0\), then Eq. (22) becomes the usual geodesic deviation equation used to analyze the polarizations of GWs [44].

4 The trajectory of a self-gravitating object in Horndeski theory

The most general scalar–tensor theory with second order equations of motion is the Horndeski theory [23], whose action is given by [46],

$$\begin{aligned} S=\int \mathrm {d}^4x\sqrt{-g}({\mathscr {L}}_2+{\mathscr {L}}_3+\mathscr {L}_4+{\mathscr {L}}_5)+S_m[\psi _m,g_{\mu \nu }],\nonumber \\ \end{aligned}$$
(23)

where \(S_m[\psi _m,g_{\mu \nu }]\) is the action for the matter field \(\psi _m\), and it is assumed that \(\psi _m\) non-minimally couples with the metric only. The individual terms in the integrand are

$$\begin{aligned} {\mathscr {L}}_2= & {} K(\phi ,X), \end{aligned}$$
(24)
$$\begin{aligned} \mathscr {L}_3= & {} -G_3(\phi ,X)\Box \phi , \end{aligned}$$
(25)
$$\begin{aligned} \mathscr {L}_4= & {} G_4(\phi ,X)R+G_{4X}[(\Box \phi )^2-(\phi _{;\mu \nu })^2], \end{aligned}$$
(26)
$$\begin{aligned} \mathscr {L}_5= & {} G_5(\phi ,X)G_{\mu \nu }\phi ^{;\mu \nu }\nonumber \\&-\frac{G_{5X}}{6}[(\Box \phi )^3-3(\Box \phi )(\phi _{;\mu \nu })^2+2(\phi _{;\mu \nu })^3]. \end{aligned}$$
(27)

In these expressions, \(X=-\phi _{;\mu }\phi ^{;\mu }/2\) with \(\phi _{;\mu }=\nabla _\mu \phi \), \(\phi _{;\mu \nu }=\nabla _\nu \nabla _\mu \phi \), \(\Box \phi =g^{\mu \nu }\phi _{;\mu \nu }\), \((\phi _{;\mu \nu })^2=\phi _{;\mu \nu }\phi ^{;\mu \nu }\) and \((\phi _{;\mu \nu })^3=\phi _{;\mu \nu }\phi ^{;\mu \rho }\phi ^{;\nu }_{;\rho }\) for simplicity. \(K, G_3, G_4, G_5\) are arbitrary analytic functions of \(\phi \) and X, and \(G_{iX}=\partial _XG_{i}, i=3,4,5\). For any binary function \(f(\phi ,X)\), define the following symbol

$$\begin{aligned} f_{(m,n)}=\frac{\partial ^{m+n}f(\phi ,X)}{\partial \phi ^m\partial X^n}\Big |_{\phi =\phi _0,X=0}, \end{aligned}$$
(28)

where \(\phi _0\) is a constant value for the scalar field evaluated at infinity. Varying the action (23) with respect to \(g_{\mu \nu }\) and \(\phi \) gives rise to the equations of motion, which are too complicated to write down. Please refer to Refs. [46, 47].

There have been experimental constraints on Horndeski theory. Reference [35] discussed the bounds on it from some solar system tests and the observations on pulsars. GW170817 and its electromagnetic counterpart GRB 170817A together set a strong constraint on the speed of GWs [7, 48]. Based on this result, the Lagrangian takes a simpler form [49,50,51,52,53, 53,54,55,56],

$$\begin{aligned} {\mathscr {L}}=K(\phi ,X)-G_3(\phi ,X)\Box \phi +G_4(\phi )R. \end{aligned}$$
(29)

Although Horndeski theory is highly constrained, we will still work with the original theory in the following discussion.

In this theory, WEP is respected due to the non-minimal coupling between \(\psi _m\) and \(g_{\mu \nu }\). However, due to the indirect interaction between \(\psi _m\) and \(\phi \) mediated by \(g_{\mu \nu }\) via the equations of motion, SEP is violated. In fact, calculations have shown that the effective gravitational “constant” actually depends on \(\phi \) [57]. Therefore, the gravitational binding energy of a compact object, viewed as a system of point particles, will also depend on the local value of \(\phi \). Because of the mass-energy equivalence \(E=m\), the mass of the compact object, i.e., the total mass of the system of point particles, also depends on \(\phi \). This would affect the motion of the compact object. Following Eardley’s suggestion, the matter action can be described by [58]

$$\begin{aligned} S_m=-\int m(\phi (x^\rho ))\sqrt{-g_{\mu \nu }(x^\rho )\dot{x}^\mu \dot{x}^\nu }\mathrm {d}\lambda , \end{aligned}$$
(30)

with \(\dot{x}^\mu =\mathrm {d}x^\mu /\mathrm {d}\lambda \), when the compact object can be treated as a self-gravitating particle. In this action, \(\phi \) and \(g_{\mu \nu }\) also depend on the trajectory. In this treatment, the spin and the multipole moment structure are ignored. To obtain the equation of motion, one applies Euler–Lagrange equation and at the same time, assumes that the parameter \(\lambda \) parameterizes the trajectory such that \(g_{\mu \nu }\dot{x}^\mu \dot{x}^\nu \) is a constant along the trajectory. Usually, one parameterizes particle trajectories with the proper time \(\tau \). This is not necessary, as one can always reparameterize. A generic parametrization is convenient for the following discussion.

The Euler–Lagrange equation reads,

$$\begin{aligned} \begin{aligned} A^a&=u^b\nabla _bu^a\\&=-\frac{\mathrm {d}\ln m}{\mathrm {d}\ln \phi }(-g^{ab}u_cu^c+u^au^b)\nabla _b\ln \phi , \end{aligned} \end{aligned}$$
(31)

where \(u^a=(\partial /\partial \lambda )^a\). Therefore, the self-gravitating particle no longer moves on a geodesic. The failure of its trajectory being a geodesic is described by \(\frac{\mathrm {d}\ln m}{\mathrm {d}\ln \phi }\), which is called the “sensitivity”. One can check that \(u_au^b\nabla _bu^a=0\), which is consistent with the parametrization. This means that the 4-acceleration of the particle is a spatial vector with respect to \(u^a\). If one chooses the proper time \(\tau \) to parameterize the trajectory, the above expression gets simplified,

$$\begin{aligned} A^a=-\frac{\mathrm {d}\ln m}{\mathrm {d}\ln \phi }(\delta ^a_b+u^au_b)\nabla ^b\ln \phi , \end{aligned}$$
(32)

where \(\delta ^a_b+u^au_b\) is actually the projection operator for \(u^a\). Therefore, a self-gravitating object moves along an accelerated trajectory when only gravity acts on it, and its acceleration is due to the gradient in the scalar field \(\phi \).

Now, consider two infinitesimally nearby self-gravitating particles, one of which travels along \(\sigma _0(\lambda )\). The deviation vector connecting \(\sigma _0(\lambda )\) to its nearby company is \(S^a\). It is useful to parameterize \(\sigma _0(\lambda )\) by its proper time \(\tau \) so that \(u^a\) is a unit timelike vector associated with an observer. The relative acceleration is thus given by

$$\begin{aligned} \begin{aligned} A^a_\mathrm {rel}&=-R_{cbd}{}^au^cS^bu^d\\&-S^b\nabla _b\left[ \frac{\mathrm {d}\ln m}{\mathrm {d}\ln \phi }(-g^{ac}u_du^d+u^au^c)\nabla _c\ln \phi \right] . \end{aligned} \end{aligned}$$
(33)

Note that the right hand side is evaluated at \(\sigma _0(\tau )\). The deviation vector \(S^a\) should satisfy

$$\begin{aligned} u^b\nabla _bS^a= & {} S^b\nabla _bu^a, \end{aligned}$$
(34)
$$\begin{aligned} S^b\nabla _b(u^au_a)= & {} 2\frac{\mathrm {d}\ln m}{\mathrm {d}\ln \phi }S^a\nabla _a\ln \phi , \end{aligned}$$
(35)

according to Eq. (17), which explains why \(u_du^d\) inside of the brackets of Eq. (33) is not set to \(-1\). The relative acceleration can be expressed entirely in terms of \(u^a\) of the particle \(\sigma _0(\tau )\) by expanding the brackets and using Eq. (34) together with Eq. (35),

$$\begin{aligned} \begin{aligned} A^a_\mathrm {rel}&=-R_{cbd}{}^au^cS^bu^d\\&-(g^{ac}+u^au^c)S^b\nabla _b\left( \frac{\mathrm {d}\ln m}{\mathrm {d}\ln \phi }\nabla _c\ln \phi \right) \\&-\frac{\mathrm {d}\ln m}{\mathrm {d}\ln \phi }(\nabla _c\ln \phi )\bigg [u^cu^b\nabla _bS^a+u^au^b\nabla _bS^c\\&-2g^{ac}\frac{\mathrm {d}\ln m}{\mathrm {d}\ln \phi }S^b\nabla _b\ln \phi \bigg ]. \end{aligned} \end{aligned}$$
(36)

Again, the right hand side is evaluated along \(\sigma _0(\tau )\).

In the Fermi normal coordinates, the spatial components of \(A^\mu \) are given by

$$\begin{aligned} A^{{\hat{j}}}=-\frac{\mathrm {d}\ln m}{\mathrm {d}\ln \phi }\partial ^{{\hat{j}}}\ln \phi , \end{aligned}$$
(37)

according to Eq. (32). By Eq. (22), one obtains

$$\begin{aligned} \begin{aligned} \frac{\mathrm {d}^2S^{{\hat{j}}}}{\mathrm {d}\tau ^2}&=-R_{{\hat{0}}{\hat{k}}{\hat{0}}}{}^{{\hat{j}}}S^{{\hat{k}}}-S^{{\hat{k}}}\partial _{{\hat{k}}}\left( \frac{\mathrm {d}\ln m}{\mathrm {d}\ln \phi }\partial ^{{\hat{j}}}\ln \phi \right) \\&+\left( \frac{\mathrm {d}\ln m}{\mathrm {d}\ln \phi }\right) ^2(\partial ^{{\hat{j}}}\ln \phi ) S^{{\hat{k}}}\partial _{{\hat{k}}}\ln \phi . \end{aligned} \end{aligned}$$
(38)

When the scalar field is not excited,i.e., \(\phi =\phi _0\), a constant, Eq. (38) reduces to the geodesic deviation equation,

$$\begin{aligned} \frac{\mathrm {d}^2S^{{\hat{j}}}}{\mathrm {d}\tau ^2}=-R_{{\hat{0}}{\hat{k}}{\hat{0}}}{}^{{\hat{j}}}S^{{\hat{k}}}. \end{aligned}$$
(39)

This is expected as vSEP is caused by a dynamical scalar field. In the next section, Eq. (38) will be used to analyze the polarization content of GWs in Horndeski theory.

5 The polarizations of gravitational waves in Horndeski gravity

In Ref. [59], the GW solutions for Horndeski theory [23] in the vacuum background have been obtained. The polarization content of the theory was also determined using the linearized geodesic deviation equation, as the vSEP was completely ignored. In this section, the GW solution will be substituted into Eq. (38) to take into account the effect of the scalar field on the trajectories of self-gravitating test particles. This will lead to a different polarization content of GWs in Horndeski theory.

Now, one expands the fields around the flat background such that \(g_{\mu \nu }=\eta _{\mu \nu }+h_{\mu \nu }\) and \(\phi =\phi _0+\varphi \). At the leading order, one obtains

$$\begin{aligned} G_{2(0,0)}=0,\quad G_{2(1,0)}=0. \end{aligned}$$
(40)

At the first order, the linearized equations of motion can be written in the following form,

$$\begin{aligned}&(\Box -m_s^2)\varphi = 0, \end{aligned}$$
(41)
$$\begin{aligned}&\Box {\tilde{h}}_{\mu \nu } = 0, \end{aligned}$$
(42)

where the scalar field \(\varphi \) is generally massive with the squared mass given by

$$\begin{aligned} m_s^2=-\frac{K_{(2,0)}}{K_{(0,1)}-2G_{3(1,0)}+3G_{4(1,0)}^2/G_{4(0,0)}}, \end{aligned}$$
(43)

and \({\tilde{h}}_{\mu \nu }\) is an auxiliary field defined as

$$\begin{aligned} \tilde{h}_{\mu \nu }=h_{\mu \nu }-\frac{1}{2}\eta _{\mu \nu }\eta ^{\alpha \beta }h_{\alpha \beta }-\chi \eta _{\mu \nu }\varphi , \end{aligned}$$
(44)

with \(\chi =\frac{G_{4(1,0)}}{G_{4(0,0)}}\). Note that the transverse-traceless (TT) gauge \(\partial _\mu {\tilde{h}}^{\mu \nu }=0\), \(\eta ^{\mu \nu }{\tilde{h}}_{\mu \nu }=0\) has been made. A GW propagating in the \(+z\) direction is given below

$$\begin{aligned} {\tilde{h}}_{\mu \nu }= & {} e_{\mu \nu }\cos \varOmega (t-z), \end{aligned}$$
(45)
$$\begin{aligned} \varphi= & {} \varphi _0\cos (\omega t-kz), \end{aligned}$$
(46)

where \(\omega ^2-k^2=m_s^2\) and the only nonvanishing components of tensor wave amplitude \(e_{\mu \nu }\) are \(e_{11}=-e_{22}\) and \(e_{12}\). The coordinate system in which the TT gauge is chosen is called the TT coordinate system.

One is interested in studying the relative acceleration of two nearby particles which were at rest before the arrival of the GW. Because of the presence of the GW induced by the scalar field, one expects \(\sigma _0(\tau )\) to deviate from a straight line in the TT coordinates, so one assumes its 3-velocity is \(\mathbf {v}\) and \(u^\mu =u^0(1,\mathbf {v})\). The normalization of \(u^a\) implies that

$$\begin{aligned} u^0=1+\frac{1}{2}h_{00}+O(v^2). \end{aligned}$$
(47)

The acceleration of \(\sigma _0(\tau )\) can be approximated as

$$\begin{aligned} A^\mu \approx -\frac{s}{\phi _0}(\eta ^{\mu \nu }+{\underline{u}}^\mu {\underline{u}}^\nu )\nabla _\nu \varphi , \end{aligned}$$
(48)

with \(s=(\mathrm {d}\ln m/\mathrm {d}\ln \phi )|_{\phi _0}\) called the sensitivity and \({\underline{u}}^\mu =(1,\mathbf {0})\) the background value. Written in component form, the acceleration is given by

$$\begin{aligned}&A^0=0, \end{aligned}$$
(49)
$$\begin{aligned}&A^j=-\delta ^j_3ks\frac{\varphi _0}{\phi _0}\sin (\omega t-kz). \end{aligned}$$
(50)

On the other hand, the left hand side of Eq. (48) is, in coordinate basis,

$$\begin{aligned} \begin{aligned} A^\mu&=\frac{\mathrm {d}^2x^\mu }{\mathrm {d}\tau ^2}+\varGamma ^\mu {}_{\rho \nu }\frac{\mathrm {d}x^\rho }{\mathrm {d}\tau }\frac{\mathrm {d}x^\nu }{\mathrm {d}\tau }\\&\approx (u^0)^2\left( \frac{\mathrm {d}^2x^\mu }{\mathrm {d}t^2}+\varGamma ^\mu {}_{00}\right) +u^0\frac{\mathrm {d}u^0}{\mathrm {d}t}\frac{\mathrm {d}x^\mu }{\mathrm {d}t}. \end{aligned} \end{aligned}$$
(51)

Consider a trivial motion, i.e., \(x=y=0\). Then one obtains

$$\begin{aligned}&v_3\approx -\frac{k}{2\omega }\left( \chi -\frac{2s}{\phi _0}\right) \varphi _0\cos \omega t, \end{aligned}$$
(52)
$$\begin{aligned}&z\approx -\frac{k}{2\omega ^2}\left( \chi -\frac{2s}{\phi _0}\right) \varphi _0\sin \omega t. \end{aligned}$$
(53)

Here, the initial position of \(\sigma _0(\tau )\) is chosen to be \(x_0=y_0=z_0=0\). In addition,

$$\begin{aligned} u^0=\frac{\mathrm {d}t}{\mathrm {d}\tau }\approx 1+\frac{1}{2}\chi \varphi _0\cos \omega t, \end{aligned}$$
(54)

according to Eq. (47), which implies that

$$\begin{aligned} \tau \approx t-\frac{\chi \varphi _0}{2\omega }\sin \omega t. \end{aligned}$$
(55)

From this, one clearly sees that the TT coordinate system is not the proper reference frame for the observer \(\sigma _0(\tau )\).

Therefore, the trajectory of \(\sigma _0(\tau )\) in the TT coordinate system is described by

$$\begin{aligned} \tau= & {} t-\frac{\chi \varphi _0}{2\omega }\sin \omega t, \end{aligned}$$
(56)
$$\begin{aligned} x= & {} y=0, \end{aligned}$$
(57)
$$\begin{aligned} z= & {} -\frac{k}{2\omega ^2}\left( \chi -\frac{2s}{\phi _0}\right) \varphi _0\sin \omega t, \end{aligned}$$
(58)

up to the linear order. Because of the scalar field, the observer oscillates with the same frequency of the GW in the TT coordinate system according to Eq. (58). The time dilatation also oscillates by Eq. (56).

In the limit of GR (\(\chi =s=0\)), the trajectory of \(\sigma _0(\tau )\) is thus \(t=\tau , x^j=0\) up to the linear order, i.e., a geodesic of the background metric. If vSEP is weak, i.e. \(s\approx 0\), the trajectory is

$$\begin{aligned} \tau= & {} t-\frac{\chi \varphi _0}{2\omega }\sin \omega t, \end{aligned}$$
(59)
$$\begin{aligned} x= & {} y=0, \end{aligned}$$
(60)
$$\begin{aligned} z= & {} -\frac{k\chi \varphi _0}{2\omega ^2}\sin \omega t. \end{aligned}$$
(61)

This agrees with Ref. [59]. Although the particle \(\sigma _0(\tau )\) does not follow a geodesic of the background metric, it still travels along a geodesic of the full metric.

5.1 The relative acceleration in the Fermi normal coordinates

In this subsection, one obtains the relative acceleration in the Fermi normal coordinates using Eq. (38). This discussion will also reveal the polarization content of GWs. The 4-velocity of the observer is

$$\begin{aligned} u^a=(e_{{\hat{0}}})^a=(1+h_{00}/2,0,0,v_3), \end{aligned}$$
(62)

so the following triad can be chosen,

$$\begin{aligned} (e_{{\hat{1}}})^a= & {} (0,1-h_{11}/2,-h_{12}/2,0), \end{aligned}$$
(63)
$$\begin{aligned} (e_{{\hat{2}}})^a= & {} (0,-h_{12}/2,1-h_{22}/2,0), \end{aligned}$$
(64)
$$\begin{aligned} (e_{{\hat{3}}})^a= & {} (v_3,0,0,1+h_{00}/2). \end{aligned}$$
(65)

These basic vectors are Fermi–Walker transported and evaluated along \(\sigma _0(\tau )\). The dual basis is denoted as \(\{(e^{\hat{\mu }})_a\}\) and \((e^{\hat{\mu }})_\nu \approx \delta ^{\hat{\mu }}_\nu \) is sufficient. Up to the linear order in perturbations, Eq. (22) is given by

$$\begin{aligned} \frac{\mathrm {d}^2 S^{{\hat{j}}}}{\mathrm {d}t^2}=-R_{{\hat{0}}{\hat{k}}{\hat{0}}}{}^{{\hat{j}}}S^{{\hat{k}}}+S^{{\hat{k}}}\partial _{{\hat{k}}}A^{{\hat{j}}}, \end{aligned}$$
(66)

since the acceleration \(A^{{\hat{j}}}\) is of the linear order, and the last term in Eq. (22) should be dropped. Normally, one has to find the Fermi normal coordinates explicitly [60, 61]. However, the Fermi normal coordinates differ from the TT coordinates by quantities of order one, and the Riemann tensor and the 4-acceleration of the test particle are both of linear order, so any changes in their components caused by the coordinate transformation are of the second order in perturbations. Therefore, one only has to calculate the components of the Riemann tensor and the 4-acceleration in the TT coordinates, and then simply substitutes them in Eq. (66).

More explicitly, the driving force matrix is given by

$$\begin{aligned} \begin{aligned}&S_{{\hat{k}}}{}^{{\hat{j}}}=R_{{\hat{0}}{\hat{k}}{\hat{0}}}{}^{{\hat{j}}}-\partial _{{\hat{k}}}A^{{\hat{j}}}\\&\approx R_{ 0 k 0}{}^{ j}-\partial _{ k}A^{ j} \\&\approx \left( \begin{array}{ccc} -\frac{\omega ^2}{2}\chi \varphi +\frac{\varOmega ^2}{2}{\tilde{h}}_{11} &{} \frac{\varOmega ^2}{2}{\tilde{h}}_{12} &{} 0 \\ \frac{\varOmega ^2}{2}{\tilde{h}}_{12} &{} -\frac{\omega ^2}{2}\chi \varphi -\frac{\varOmega ^2}{2}{\tilde{h}}_{11} &{} 0 \\ 0 &{} 0 &{} -\frac{m_s^2}{2}\chi \varphi -\frac{k^2s}{\phi _0}\varphi \end{array} \right) , \nonumber \\ \end{aligned}\\ \end{aligned}$$
(67)

where \({\tilde{h}}_{\mu \nu }\) and \(\varphi \) are evaluated at \((t,\mathbf {x}=0)\). Comparing this matrix with the one (Eq. (29)) in Ref. [59], one finds out that vSEP introduces an order one correction \(-k^2s\varphi /\phi _0\) to the longitudinal polarization. This means that the longitudinal polarization gets enhanced. Even if the scalar field is massless, the longitudinal polarization persists because the test particles are accelerated.

However, the enhancement is very extremely small for objects such as the mirrors used in detectors such as LIGO. According to Refs. [29, 62], white dwarfs have typical sensitivities \(s\sim 10^{-4}\), so a test particle, like the mirror used by LIGO, would have an even smaller sensitivity. So it would be still very difficult to use interferometers to detect the enhanced longitudinal polarization as in the previous case [59]. In contrast, neutron stars are compact objects. Their sensitivity could be about 0.2 [29, 62]. They violate SEP relatively strongly, which might be detected by PTAs.

6 Pulsar timing arrays

In this section, the cross-correlation function will be calculated for PTAs. The possibility to detect the vSEP is thus inferred. A pulsar is a strongly magnetized, rotating neutron star or a white dwarf, which emits a beam of the radio wave along its magnetic pole. When the beam points towards the Earth, the radiation is observed, and this leads to the pulsed appearance of the radiation. The rotation of some “recycled” pulsars is stable enough so that they can be used as “cosmic light-house” [63]. Among them, millisecond pulsars are found to be more stable [64] and used as stable clocks [65]. When there is no GW, the radio pulses arrive at the Earth at a steady rate. The presence of the GW will affect the propagation time of the radiation and thus alter this rate. This results in a change in the time-of-arrival (TOA), called timing residual R(t). Timing residuals caused by the stochastic GW background is correlated between pulsars, and the cross-correlation function is \(C(\theta )=\langle R_a(t)R_b(t)\rangle \) with \(\theta \) the angular separation of pulsars a and b, and the brackets \(\langle \,\rangle \) implying the ensemble average over the stochastic background. This makes it possible to detect GWs and probe the polarizations [37,38,39,40, 66,67,68,69,70,71,72,73]. The effect of vSEP can also be detected, as the longitudinal polarization of the scalar–tensor theory is enhanced due to vSEP.

One sets up a coordinate system shown in Fig. 1 to calculate the timing residual R(t) caused by the GW solution (45) and (46). Before the GW comes, the Earth is at the origin, and the distant pulsar is at rest at \(\mathbf {x}_p=(L\cos \beta ,0,L\sin \beta )\) in this coordinate system. The GW is propagating in the direction of a unit vector \({\hat{k}}\), and \({\hat{n}}\) is the unit vector pointing to the pulsar from the Earth. \({\hat{l}}={\hat{k}}\wedge ({\hat{n}}\wedge {\hat{k}})/\cos \beta =[{\hat{n}}-{\hat{k}}(\hat{n}\cdot {\hat{k}})]/\cos \beta \) is actually the unit vector parallel to the y axis.

Fig. 1
figure 1

The GW is propagating in the direction of \({\hat{k}}\), and the photon is traveling in \(-{\hat{n}}\) direction at the leading order. \({\hat{l}}\) is perpendicular to \({\hat{k}}\) and in the same plane determined by \({\hat{k}}\) and \({\hat{n}}\). The angle between \({\hat{n}}\) and \({\hat{l}}\) is \(\beta \)

Where there is no GW, the photon is assumed to have a 4-velocity given by \({\underline{u}}^\mu =\gamma _0(1,-\cos \beta ,0,-\sin \beta )\) with \(\gamma _0=\mathrm {d}t/\mathrm {d}\lambda \) a constant and \(\lambda \) an arbitrary affine parameter. Let the perturbed photon 4-velocity be \(u^\mu ={\underline{u}}^\mu +v^\mu \). The condition \(g_{\mu \nu }u^\mu u^\nu =0\) together with the photon geodesic equation lead to

$$\begin{aligned} v^0= & {} \gamma _0\{\chi \varphi _0\cos [(\omega +k\sin \beta )t-k(L+t_e)\sin \beta ]-\frac{e_{11}}{2} \nonumber \\&\times (1-\sin \beta )\cos [\varOmega (1+\sin \beta )t-\varOmega (L+t_e)\sin \beta ]\}, \nonumber \\ \end{aligned}$$
(68)
$$\begin{aligned} v_1= & {} \gamma _0\{-\chi \varphi _0\cos \beta \cos [(\omega +k\sin \beta )t-k(L+t_e)\sin \beta ]\nonumber \\&+{\,}e_{11}\cos \beta \cos [\varOmega (1+\sin \beta )t-\varOmega (L+t_e)\sin \beta ]\}, \nonumber \\ \end{aligned}$$
(69)
$$\begin{aligned} v_2= & {} \gamma _0e_{12}\cos \beta \cos \varOmega [(1+\sin \beta )t-(L+t_e)\sin \beta ], \nonumber \\ \end{aligned}$$
(70)
$$\begin{aligned} v_3= & {} \gamma _0\{-\chi \varphi _0\sin \beta \cos [(\omega +k\sin \beta )t\nonumber \\&\quad -k(L+t_e)\sin \beta ] -\frac{e_{11}}{2}\nonumber \\&(1-\sin \beta )\cos [\varOmega (1+\sin \beta )t-\varOmega (L+t_e)\sin \beta ]\}, \nonumber \\ \end{aligned}$$
(71)

where \(t_e\) is the time when the photon is emitted from the pulsar.

The 4-velocity of an observer on the Earth has been obtained in Sect. 5, which reads

$$\begin{aligned} \begin{aligned} T_e^\mu&=\biggl (1+\frac{1}{2}\chi \varphi _0\cos \omega t,0,\\&0,-\frac{k}{2\omega }\left( \chi -\frac{2s_r}{\phi _0}\right) \varphi _0\cos \omega t\biggr ), \end{aligned} \end{aligned}$$
(72)

where \(s_r\) is the sensitivity of the Earth. The 4-velocity of another observer comoving with the pulsar can be derived in a similar way. In fact, the translational symmetry in the background spacetime (i.e., Minkowskian spacetime) gives

$$\begin{aligned} \begin{aligned} T_p^\mu&=\biggl (1+\frac{1}{2}\chi \varphi _0\cos (\omega t-kL\sin \beta ),0,\\&0,-\frac{k}{2\omega }\left( \chi -\frac{2s_e}{\phi _0}\right) \varphi _0\cos (\omega t-kL\sin \beta )\biggr ), \end{aligned} \end{aligned}$$
(73)

which agrees with the result from the direct calculation. Here, \(s_e\) is the sensitivity of the pulsar.

So the measured frequency by the observer on the Earth is

$$\begin{aligned} \begin{aligned} f_r&=-u_\mu T^\mu _e\\&=\gamma _0\Bigg [1+\left( \frac{\omega -k\sin \beta }{2\omega }\chi +\frac{s_rk}{\phi _0\omega }\sin \beta \right) \varphi _0\times \\&\cos \omega (t_e+L)-\frac{e_{11}}{2}(1-\sin \beta )\cos \varOmega (t_e+L)\Bigg ], \end{aligned} \end{aligned}$$
(74)

and the one by the observer comoving with the pulsar is

$$\begin{aligned} \begin{aligned} f_e&=-u_\mu T^\mu _p\\&=\gamma _0\Bigg [1+\left( \frac{\omega -k\sin \beta }{2\omega }\chi +\frac{s_ek}{\phi _0\omega }\sin \beta \right) \varphi _0\cos (\omega t_e\\&-kL\sin \beta )-\frac{e_{11}}{2}(1-\sin \beta )\cos \varOmega (t_e-L\sin \beta )\Bigg ].\nonumber \\ \end{aligned}\\ \end{aligned}$$
(75)

Therefore, the frequency shift is given by

$$\begin{aligned} \begin{aligned} \frac{f_e-f_r}{f_r}&=\frac{\omega -k{\hat{k}}\cdot {\hat{n}}}{2\omega }\chi \left[ \varphi (t-L,L{\hat{n}})-\varphi (t,0)\right] \\&-\frac{e_{jk}{\hat{n}}^j{\hat{n}}^k}{2(1+{\hat{k}}\cdot {\hat{n}})}\left[ {\tilde{h}}_{jk}(t-L,L{\hat{n}})-{\tilde{h}}_{jk}(t,0)\right] \\&+\frac{k}{\omega \phi _0}{\hat{k}}\cdot {\hat{n}}[s_e\varphi (t-L,L{\hat{n}})-s_r\varphi (t,0)], \end{aligned} \end{aligned}$$
(76)

where \(t=t_e+L\) is the time when the photon arrives at the Earth at the leading order. This equation has been expressed in a coordinate independent way, so it can be straightforwardly used in any coordinate system with arbitrary orientation and at rest relative to the original one. Note that the first two lines reproduce the result in Ref. [59], and the third line comes from the effect of vSEP. This effect is completely determined by the scalar perturbation \(\varphi \), as expected.

Therefore, the focus will be on the cross-correlation function for the scalar GW in the following discussion. Eq. (76) is the frequency shift due to a monochromatic wave. Now, consider the contribution of a stochastic GW background which consists of monochromatic GWs,

$$\begin{aligned} \varphi (t,\mathbf {x})=\int _{-\infty }^{\infty }\frac{\mathrm {d}\omega }{2\pi }\int \mathrm {d}^2{\hat{k}} \{\varphi _0(\omega ,{\hat{k}} )\exp [i(\omega t-k {\hat{k}} \cdot \mathbf {x})]\},\nonumber \\ \end{aligned}$$
(77)

where \(\varphi _0(\omega ,{\hat{k}})\) is the amplitude for the scalar GW propagating in the direction \({\hat{k}} \) at the angular frequency \(\omega \). Usually, one assumes that the GW background is isotropic, stationary and independently polarized, then one can define the characteristic strains \(\varphi _c\) given by,

$$\begin{aligned} \langle \varphi _0^*(\omega ,{\hat{k}} )\varphi _0(\omega ',{\hat{k}}')\rangle =\delta (\omega -\omega )\delta ({\hat{k}} -{\hat{k}}')\frac{|\varphi _c(\omega )|^2}{\omega }, \end{aligned}$$
(78)

where the star \(*\) implies the complex conjugation.

The total timing residual in TOA due to the stochastic GW background is

$$\begin{aligned} R(T)=\int _{-\infty }^{\infty }\frac{\mathrm {d}\omega }{2\pi }\int \mathrm {d}^2{\hat{k}} \int _{0}^{T}\mathrm {d}t\frac{f_e-f_r}{f_r}, \end{aligned}$$
(79)

where the argument T is the total observation time. Insert Eq. (76) in, neglecting the second line, to obtain

$$\begin{aligned} \begin{aligned} R(T)&=\int _{-\infty }^{\infty }\frac{\mathrm {d}\omega }{2\pi }\int \mathrm {d}^2{\hat{k}} \varphi _0(\omega ,{\hat{k}} )(e^{i\omega T}-1)\left\{ \frac{\omega -k {\hat{k}} \cdot {\hat{n}}}{i2\omega ^2}\chi \right. \\&\left. \times [e^{-i(\omega +k {\hat{k}} \cdot n)L}-1]+\frac{k{\hat{k}}\cdot {\hat{n}}}{i\omega ^2\phi _0}[s_ee^{-i(\omega \!+\!k {\hat{k}} \cdot n)L}\!-\!s_r]\right\} .\nonumber \end{aligned}\\ \end{aligned}$$
(80)

With this result, consider the correlation between two pulsars a and b located at \(\mathbf {x}_a=L_1{\hat{n}}_1\) and \(\mathbf {x}_b=L_2{\hat{n}}_2\), respectively. The angular separation is \(\theta =\arccos ({\hat{n}}_1\cdot {\hat{n}}_2)\). The cross-correlation function is thus given by

$$\begin{aligned} \begin{aligned} C(\theta )&=\langle R_a(T)R_b(T)\rangle \\&=\int _{m_s}^\infty \mathrm {d}\omega \int \mathrm {d}^2{\hat{k}}\frac{|\varphi _c(\omega )|^2}{\pi \omega ^5}\biggl [\frac{k^2{\hat{k}}\cdot {\hat{n}}_1{\hat{k}}\cdot {\hat{n}}_2}{\phi _0^2}{\mathscr {P}}_4\\&+\frac{k{\hat{k}}\cdot {\hat{n}}_1(\omega -k{\hat{k}}\cdot {\hat{n}}_2)}{2\phi _0}\chi {\mathscr {P}}_2\\&+\frac{k{\hat{k}}\cdot {\hat{n}}_2(\omega -k{\hat{k}}\cdot {\hat{n}}_1)}{2\phi _0}\chi {\mathscr {P}}_3\\&+\frac{(\omega -k{\hat{k}}\cdot {\hat{n}}_1)(\omega -k\hat{k}\cdot \hat{n}_2)}{4}\chi ^2{\mathscr {P}}_1\biggr ], \end{aligned} \end{aligned}$$
(81)

where \({\mathscr {P}}_1,\,{\mathscr {P}}_2,\,{\mathscr {P}}_3\) and \(\mathscr {P}_4\) are defined to be

$$\begin{aligned} {\mathscr {P}}_1= & {} 1-\cos \varDelta _1-\cos \varDelta _2+\cos (\varDelta _1-\varDelta _2), \end{aligned}$$
(82)
$$\begin{aligned} {\mathscr {P}}_2= & {} s_r-s_r\cos \varDelta _2-s_e\cos \varDelta _1+s_e\cos (\varDelta _1-\varDelta _2), \end{aligned}$$
(83)
$$\begin{aligned} {\mathscr {P}}_3= & {} s_r-s_r\cos \varDelta _1-s_e\cos \varDelta _2+s_e\cos (\varDelta _1-\varDelta _2), \end{aligned}$$
(84)
$$\begin{aligned} {\mathscr {P}}_4= & {} s_r^2-s_rs_e\cos \varDelta _1-s_rs_e\cos \varDelta _2+s_e^2\cos (\varDelta _1-\varDelta _2),\nonumber \\ \end{aligned}$$
(85)

with \(\varDelta _j=(\omega +k {\hat{k}} \cdot {\hat{n}}_j)L_j\) for \(j=1,2\). To obtain this result, Eq. (78) is used, and the real part is taken. In addition, T drops out, as the ensemble average also implies the averaging over the time [38].

Fig. 2
figure 2

The normalized cross-correlation functions \(\zeta (\theta )=C(\theta )/C(0)\). The upper panel shows the cross-correlations when the scalar field is massless, i.e., when there is no longitudinal polarization. The solid curve is for familiar GR polarizations (i.e., the plus or cross ones), the dashed red curve for the breathing polarization with SEP and the dotted purple curve for the breathing polarization with vSEP. The lower panel shows the normalized cross-correlations induced together by the transverse breathing and longitudinal polarizations when the mass of the scalar field is taken to be \(m_s=7.7\times 10^{-23}\,\mathrm {eV}/c^2\). The solid curves are for the cases where SEP is satisfied, while the dashed curves are for those where SEP is violated. The power-law index \(\alpha =0,-2/3,-1\). The calculation was done assuming \(T=5\) years

Because of the isotropy of the GW background, one sets

$$\begin{aligned} {\hat{n}}_1= & {} (0,0,1), \end{aligned}$$
(86)
$$\begin{aligned} {\hat{n}}_2= & {} (\sin \theta ,0,\cos \theta ). \end{aligned}$$
(87)

Also, let \({\hat{k}} =(\sin \theta _g\cos \phi _g,\sin \theta _g\sin \phi _g,\cos \theta _g)\), so

$$\begin{aligned} \varDelta _1= & {} (\omega +k \cos \theta _g)L_1, \end{aligned}$$
(88)
$$\begin{aligned} \varDelta _2= & {} [\omega +k (\sin \theta _g\cos \phi _g\sin \theta +\cos \theta _g\cos \theta )]L_2, \end{aligned}$$
(89)

Working in the limit that \(\omega L_j\gg 1\), one can drop the cosines in the definitions (82)–(85) of \({\mathscr {P}}_j\) (\(j=1,2,3,4\)), when \(\theta \ne 0\). The integration can be partially done, resulting in

$$\begin{aligned} C(\theta )\!=\!\int _{m_s}^\infty \mathrm {d}\omega \frac{|\varphi _c(\omega )|^2}{\omega ^3}\chi ^2\left[ 1+\frac{k^2}{3\omega ^2}\left( 1-\frac{2s_r}{\phi _0\chi }\right) ^2\cos \theta \right] .\nonumber \\ \end{aligned}$$
(90)

But for \(\theta =0\), one considers the auto-correlation function, so set \({\hat{n}}_1={\hat{n}}_2=(0,0,1)\) and \(L_1=L_2=L\). The auto-correlation function is thus given by

$$\begin{aligned} \begin{aligned} C(0)=&\int _{m_s}^\infty \mathrm {d}\omega \frac{|\varphi _c(\omega )|^2}{\omega ^3}\chi ^2\biggl [2+\frac{k^2}{3\omega ^2}\left( 1-\frac{2s_r}{\phi _0\chi }\right) ^2\\&+\frac{k^2}{3\omega ^2}\left( 1-\frac{2s_e}{\phi _0\chi }\right) ^2\biggr ], \end{aligned} \end{aligned}$$
(91)

where the terms containing L are dropped as they barely contribute according to the experience in Ref. [59]. Finally, the observation time T sets a natural cutoff for the angular frequency, i.e., \(\omega \ge 2\pi /T\), so the lower integration limits in Eqs. (90) and (91) should be replaced by \(\text {Max}\{m_s,2\pi /T\}\).

As usual, assume \(\varphi _c(\omega )\propto (\omega /\omega _c)^\alpha \) with \(\omega _c\) the characteristic angular frequency. Here, \(\alpha \) is called the power-law index, and usually, \(\alpha =0,\,-2/3\) or \(-1\) [38, 74]. Numerically integrating Eqs. (90) and (91) gives the so-called normalized correlation function \(\zeta (\theta )=C(\theta )/C(0)\). In the integration, set the observation time \(T=5\) years. The sensitivities of the Earth and the pulsar are taken to be \(s_r=0\) and \(s_e=0.2\), respectively. This leads to Fig. 2, where the power-law index \(\alpha \) takes different values.

If the scalar field is massless, the results are shown in the upper panel which displays the normalized correlation functions for the plus and cross polarizations – Hellings–Downs curve (labeled by “GR”) [69]. The remaining two curves are for the breathing polarization: the dashed one is for the case where SEP is respected, while the dotted one is for the case where SEP is violated. They are independent of the power-law index \(\alpha \). As one can see that vSEP makes \(\zeta (\theta )\) bigger by about 5%. If the scalar field has a mass \(m_s=7.7\times 10^{-23}\,\mathrm {eV}/c^2\), the results are shown in the lower panel. In this panel, the cross-correlation functions for the scalar polarization are drawn for different values of \(\alpha \). The solid curves correspond to the case where SEP is satisfied, and the dashed curves are for the case where SEP is violated. Since the cross correlation for the plus and cross polarizations does not change, we do not plot them again in the lower panel. In the massive case, vSEP also increases \(\zeta (\theta )\) by about 2–3%.

Reference [75] published the constraint on the stochastic GW background based on the recently released 11-year dataset from the North American Nanohertz Observatory for Gravitational Waves (NANOGrav). Assuming the background is isotropic and \(\alpha =-2/3\), the strain amplitude of the GW is less than \(1.45\times 10^{-15}\) at \(f=1\text { yr}^{-1}\). In addition, the top panel in Figure 6 shows the observed cross correlation. As one can clearly see, the error bars are very large.Footnote 2 More observations are needed to improve the statistics.

7 Conclusion

This work discusses the effects of the vSEP on the polarization content of GWs in Horndeski theory and calculates the cross-correlation functions for PTAs. Because of the vSEP, self-gravitating particles no longer travel along geodesics, and this leads to the enhancement of the longitudinal polarization in Horndeski theory, so even if the scalar field is massless, the longitudinal polarization still exists. This is in contrast with the previous results [59, 76,77,78] that the massive scalar field excites the longitudinal polarization, while the massless scalar field does not. The enhanced longitudinal polarization is nevertheless difficult for aLIGO to detect, as the mirrors does not violate SEP enough. However, pulsars are highly compact objects with sufficient self-gravitating energy such that their trajectories deviate from geodesics enough. Using PTAs, one can measure the change in TOAs of electromagnetic radiation from pulsars and obtain the cross-correlation function to tell whether vSEP effect exits. The results show that the vSEP leads to large changes in the behaviors of the cross-correlation functions. In principle, PTAs are capable of detecting the vSEP if it exists.