1 Introduction

Gravitational radiation plays an important role in astrophysics and has attracted much attention since 1918 [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]. The first detection of gravitational wave (GW) signal, GW150914 [10], was published by LIGO and Virgo Collaboration on February 11, 2016, which verified the prediction of Einstein’s general theory of relativity. Along with the increasing gravitational wave detection events [11,12,13,14,15], the era of gravitational wave astronomy has come. Gravitational waves emitted by a coalescing compact binary system carry information about the source, which can be extracted from gravitational wave signals by using the matched filtering technique [16], i.e., through cross-correlating the incoming noisy signals with the theoretical templates. Since gravitational wave signals are much weaker than the background noises, it becomes crucial to study gravitational wave template in order to dig out real gravitational wave signals hidden under noises.

Based on the post-Newtonian (PN) approximation, Buonanno and Damour introduced an effective-one-body (EOB) theory in 1999 [17], which was then applied to study the gravitational waves emitted by a coalescing binary system of compact objects [18,19,20,21,22,23,24,25]. The basic idea of the theory is to map the relativistic two-body problem onto an effective one-body problem, i.e., the motion of a test particle in an effective background spacetime. In the EOB theory, one uses the following notations for the one-body quantities related to the masses \(m_{1}\) and \(m_{2}\) of the real two bodies

$$\begin{aligned} M=m_{1}+m_{2},\quad m_0=\frac{m_{1}m_{2}}{m_{1}+m_{2}}, \end{aligned}$$
(1.1)

where M is the mass parameter that appeared in the effective metric, and \(m_0\) denotes the mass of a test particle.

The calculation of the reaction force is a key step to construct the EOB theory, since the motion of a test particle is affected by a reaction force due to the loss of energy and angular momentum of the system in the process of the gravitational radiation. In a circular orbit, the reaction force can be obtained from the relation [18], \( {\mathcal {F}}^{circ}_{\varphi }\simeq -\frac{(\frac{dE}{dt})_{circ}}{{\dot{\varphi }}}, \) where \((\frac{dE}{dt})_{circ}\) is the averaged energy-loss-rate along circular orbits and \({\dot{\varphi }}\) is the angular velocity.

The energy-loss-rate for the gravitational radiation of compact binaries has been studied extensively [26,27,28,29,30,31,32], based on the model that a test particle with mass \(m_0\) travels along a circular orbit around a Schwarzschild black hole with the mass M (\(M\gg m_0\)). In such a model the stress-energy tensor \(T^{\mu \nu }\) of the system is described by [33, 34]

$$\begin{aligned} T^{\mu \nu }=m_0\int ^{+\infty }_{-\infty }\frac{U^{\mu }U^{\nu }}{\sqrt{-g}}\delta ^4[x^{\mu }-x^{\mu }_{p}(\tau )]d\tau , \end{aligned}$$
(1.2)

where \(x^{\mu }\) and \(x^{\mu }_{p}(\tau )\) represent spacetime events and the particle’s world line with tangent vector \(U^{\mu }=dx^{\mu }_{p}/d\tau \) (\(\tau \) denotes the proper time), respectively. Gravitational waves, induced by the motion of a particle with mass \(m_0\) in the background, may be explored in terms of black hole perturbation theory. For the Schwarzschild spacetime, by using the Newman–Penrose formalism, one found that the perturbed Weyl fields can be described by the following inhomogeneous Teukolsky equation [35, 36]

$$\begin{aligned} \left[ {\Delta ^2\frac{d}{ dr}\left( {\frac{1}{\Delta } \frac{d}{ dr}}\right) +U(r)}\right] R_{\ell m\omega }(r) =T_{\ell m\omega }(r), \end{aligned}$$
(1.3)

where \(\Delta =r(r-2M)\), U(r) is the effective potential and \(T_{\ell m\omega }(r)\) is the source term which is related to the stress-energy tensor \(T^{\mu \nu }\). Based on Eq. (1.3), the energy-loss-rate was first obtained analytically up to \({\mathcal {O}}(v^{3})\) beyond Newtonian by Poisson [26] and numerically up to \({\mathcal {O}}(v^{6})\) by Cutler et al. [27]. By using the post-Newtonian expansion of the ingoing-wave Regge–Wheeler functions \(X^{in}_{\ell m}\), a highly accurate analytical calculation up to \({\mathcal {O}}(v^{8})\) was done by Tagoshi and Sasaki [28, 29]. In addition, the gravitational radiation of the Kerr black hole was studied in Refs. [37,38,39,40], and a system emits gravitational waves in which the particle’s orbit evolves under radiation reaction was discussed in Ref. [41].

The post-Minkowskian (PM) approach is another useful approximation method to deal with the relativistic two-body problem, which has attracted extensive attention in recent years. Cristofoli describe the computation of post-Minkowskian Hamiltonians in general relativity from scattering amplitudes [42]. Energetics of two-body Hamiltonians and the amplitude for classical scattering of gravitationally interacting massive scalars are conducted at third post-Minkowskian order [43, 44]. Blanchet discuss the equations of motion of N self-interacting massive particles in the first post-Minkowskian approximation of general relativity [45]. Post-Minkowskian approach has been further explored and extended in more recent papers [46,47,48,49,50]. In contrast to the PN approach in which \(\frac{v}{c}\) is assumed to be small, the PM approach uses the gravitational constant G as an expansion parameter and \(\frac{v}{c}\) is not required to be small anymore [51,52,53,54,55]. The EOB theory based on the PM approximation was then introduced by Damour in 2016 [56]. Through calculating the scattering angles [33], it was found that, at the 2PM approximation, the energy map between the real two-body energy \({\mathcal {E}}\) and the energy \({\mathcal {E}}_{0}\) of the effective particle takes the form [56,57,58]

$$\begin{aligned} \frac{{\mathcal {E}}_{0}}{m_{0} c^{2}}=\frac{{\mathcal {E}}^{2}-m_{1}^{2} c^{4}-m_{2}^{2} c^{4}}{2 m_{1} m_{2} c^{4}}. \end{aligned}$$
(1.4)

They also presented an effective Hamiltonian, but they did not give the effective metric at the 2PM order.

In our previous paper [59], we derived the 2PM effective Hamiltonian from the investigation of the bounded states and show that the energy map Eq. (1.4) is still held. Moreover, we have constructed an effective metric at 2PM order by calculating the action variables and precession angles. In this manuscript, we take a step forward to study the radiation reaction force by calculating the energy-loss-rate dE/dt based on the effective metric in the EOB theory at the 2PM approximation.

The paper is organized as follows. In Sect. 2, we derive the inhomogeneous Teukolsky equation and the corresponding formal solutions via the Green function method. Then we transform the homogeneous Teukolsky equation to the Sasaki–Nakamura equation. We obtain the solutions of the Teukolsky equation with the source in Sect. 3, and calculate the reaction force acting on the particle in Sect. 4. Section 5 is devoted to conclusions and discussions. We use the unit with \(c = 1\) throughout the paper in the PM framework.

2 General formulation

In this section, we present explicit expression of the inhomogeneous Teukolsky equation, which describes the gravitational radiation induced by the motion of an effective particle in an effective background. The Teukolsky equation without source is then transformed to the corresponding Sasaki–Nakamura equation.

2.1 The Teukolsky equation

We study the model that an effective particle with mass \(m_0\) moves around a spherical symmetry black hole with total mass \(M=m_{1}+m_{2}\) in a circular orbit. The background geometry is given by the effective metric at 2PM which corresponds to a spinless real two-body system

$$\begin{aligned} d s^{2}_{eff}=A d t^{2}-B d r^{2}-r^{2}\left( d \theta ^{2}+\sin ^{2}\theta d \varphi ^{2}\right) , \end{aligned}$$
(2.1)

with

$$\begin{aligned} A= & {} 1-\frac{2GM}{r} +a_{2}\left( \frac{GM}{r}\right) ^{2}= \frac{(r-{r_{1}}) (r-{r_{h}})}{r^2}, \nonumber \\ B= & {} \frac{{D_1}^2}{A},\quad {D_1}=1+d_{2} \left( \frac{{GM}}{r}\right) ^2, \end{aligned}$$
(2.2)

where \(a_{2}\) and \(d_{2}\) are dimensionless parameters, \(r_{1}=(1-\sqrt{1-a_{2}})GM\) and \(r_{h}=(1+\sqrt{1-a_{2}})GM\).

To calculate gravitational radiation from a particle orbiting around a spherically symmetric black hole, we first present the Teukolsky equation. In a spherical spacetime given by Eq. (2.1), the decoupled equation for \(\Psi _{4}\) becomes [35]

$$\begin{aligned}&\biggl [(\Delta +3\gamma -{\bar{\gamma }}+4\mu +{\bar{\mu }})(D-\rho ) -({\bar{\delta }}+2\alpha ) (\delta -4\alpha )-3\Psi ^{B}_{2} \biggr ]\Psi _{4}\nonumber \\&\quad =\frac{{\hat{\kappa }}}{2}T_{4}, \end{aligned}$$
(2.3)

where \(\gamma \), \(\mu \), \(\rho \), \(\alpha \) are spin coefficients, \(\Delta \), D, \(\delta \) are differential operators, \(\Psi _{4}\) is a component of Weyl tensor and \({\hat{\kappa }}=8\pi G\) is a constant. By decomposing \(\Psi _{4}\)

$$\begin{aligned} \Psi _{4}=\int _{-\infty }^{+\infty }d\omega \sum _{\ell m}\frac{R_{\ell m \omega }(r)}{r^{4}}\,_{-2}Y_{\ell m}(\theta ,\varphi )e^{-i\omega t}, \end{aligned}$$
(2.4)

and considering the integral relationship between \(T_{4}\) and \(T_{\ell m \omega }(r)\)

$$\begin{aligned} \frac{{\hat{\kappa }}}{2}T_{4}=\int _{-\infty }^{+\infty }d\omega \sum _{\ell m}\frac{- e^{-i\omega t} }{2r^{6}}\,_{-2}Y_{\ell m}(\theta ,\varphi )T_{\ell m \omega }(r), \end{aligned}$$
(2.5)

from Eq. (2.3), one may obtain the inhomogeneous Teukolsky equation

$$\begin{aligned} \left[ r^{4}A \sqrt{\frac{A}{B}} \frac{d}{d r}\left( \frac{1}{r^{2} A^{2}} \sqrt{\frac{A}{B}}\frac{d}{d r}\right) +V(r)\right] R_{\ell m \omega }(r)=T_{\ell m \omega }(r)\nonumber \\, \end{aligned}$$
(2.6)

with

$$\begin{aligned} V(r)= & {} \frac{1}{4 A^{2} B^{2}}\biggl \{r^{2} A^{\prime }\left( 8 i \omega B \sqrt{A B}-B A^{\prime }\right) \nonumber \\&+4 A^{2}\left( B+2 r B^{\prime }\right) +r A\left[ 4 r \omega ^{2} B^{2}-r A^{\prime } B^{\prime }\right. \nonumber \\&\left. +2 B\left( -8 i \omega \sqrt{A B}+8 A^{\prime }+r A^{\prime \prime }\right) \right] \biggr \}-1-\lambda , \nonumber \\ \end{aligned}$$
(2.7)

where \(\lambda =(\ell -1)(\ell +2)\) is the eigenvalue of spin-weighted spherical harmonics \(_{-2}Y_{\ell m}(\theta ,\varphi )\), and \(T_{\ell m\omega }(r)\) is the source term. For a particle with the energy momentum tensor [34]

$$\begin{aligned} T^{\mu \nu }=\frac{m_0}{\sqrt{A B} r^{2} \sin \theta {\dot{t}}} U^{\mu } U^{\nu } \delta \left( r-r_{0}\right) \delta \left( \theta -\frac{\pi }{2}\right) \delta (\varphi -\Omega t)\nonumber \\, \end{aligned}$$
(2.8)

where \({\dot{t}}=dt/d\tau \), the source term \(T_{\ell m\omega }(r)\) is given by

$$\begin{aligned} T_{\ell m \omega }= & {} \frac{{\hat{\kappa }}}{4} \biggl [2 \sqrt{(\ell -1) \ell (\ell +1)(\ell +2) } r^{4} { }_{0} T_{\ell m \omega }(r)\nonumber \\&+ 2 \sqrt{2(\ell -1)(\ell +2) }r^{2} A {\mathcal {L}} r^{3} A^{-1} { }_{-1} T_{\ell m \omega }(r)\nonumber \\&+ r A {\mathcal {L}} r^{4} A^{-1} {\mathcal {L}} r { }_{-2} T_{\ell m \omega }(r) \biggr ], \end{aligned}$$
(2.9)

with

$$\begin{aligned} { }_{0} T_{\ell m \omega }(r)= & {} \frac{m_0 {\tilde{E}}}{8 \pi r^{2}} \sqrt{\frac{A}{B}} \delta \left( r-r_{0}\right) { }_{0} Y_{\ell , m}\left( \frac{\pi }{2}, 0\right) \delta (\omega -m \Omega ), \nonumber \\ { }_{-1} T_{\ell m \omega }(r)= & {} \frac{i m_0 {\tilde{L}}}{4 \sqrt{2} \pi r^{3}} \sqrt{\frac{A}{B}} \delta \left( r-r_{0}\right) { }_{-1} Y_{\ell , m}\left( \frac{\pi }{2}, 0\right) \delta (\omega -m \Omega ),\nonumber \\ { }_{-2} T_{\ell m \omega }(r)= & {} \frac{-m_0 {\tilde{L}}^{2}}{4 \pi r^{4}{\tilde{E}}} \sqrt{\frac{A}{B}} \delta \left( r-r_{0}\right) { }_{-2} Y_{\ell , m}\left( \frac{\pi }{2}, 0\right) \delta (\omega -m \Omega ),\nonumber \!\!\!\!\!\!\\ \end{aligned}$$
(2.10)

where \(\delta (r)\) is the Dirac delta function, \({\mathcal {L}}=\sqrt{\frac{A}{B}}\partial _{r}+i \omega \), \({\tilde{E}}\) and \({\tilde{L}}\) are energy and angular momentum of the particle in a circular orbit which are given by

$$\begin{aligned}&{\tilde{E}}= \frac{a_{2}(GM)^{2}+r(r-2GM)}{r\sqrt{2a_{2}(GM)^{2}+r(r-3GM)}}, \nonumber \\&\quad {\tilde{L}}=\frac{r\sqrt{GM(r-a_{2}GM)}}{\sqrt{2a_{2}(GM)^{2}+r(r-3GM)}}. \end{aligned}$$
(2.11)

By using the Green function method [60], the solution of Eq. (2.6) at infinity can be expressed as

$$\begin{aligned} R_{\ell m \omega }(r \rightarrow \infty )= & {} \frac{r^{3} e^{i \omega r^{*}}}{2 i \omega B_{\ell \omega }^{i n}} \int _{r_{h}}^{\infty } d r R_{\ell \omega }^{\mathrm {in}} T_{\ell m \omega }(r)r^{-4} \sqrt{B/A^{3}}\nonumber \\, \end{aligned}$$
(2.12)

where \(R^{in}_{\ell \omega }(r)\) is the homogeneous solution of Eq. (2.6) which is given by

$$\begin{aligned} R_{\ell \omega }^{\mathrm {in}}(r)=\left\{ \begin{array}{ll} D_{\ell \omega } A^{2} e^{-i \omega r^{*}} &{}\quad r^{*} \rightarrow -\infty , \\ r^{3} B_{\ell \omega }^{ {out }} e^{i \omega r^{*}}+r^{-1} B_{\ell \omega }^{ {in }} e^{-i \omega r^{*}} &{} \quad r^{*} \rightarrow +\infty , \end{array}\right. \end{aligned}$$
(2.13)

and where the tortoise coordinate is given by \(r^{*}= r + 2GM b_{1} \ln (r - r_{1})-2GM b_{2} \ln (r - r_{h})\), with

$$\begin{aligned} {b_{1}}=\frac{{d_{2}(GM)^{2}}+r_{1}^2}{2GM(r_{1}-r_{h})}, \quad {b_{2}}=\frac{{d_{2}(GM)^{2}}+r_{h}^2}{2GM(r_{1}-r_{h})}. \end{aligned}$$

2.2 The Sasaki–Nakamura equation

It is difficult to solve Eq. (2.6) directly, since for this case the source term is included. Therefore, we first consider the solution of the homogeneous equation without the source term, and then study the role of the source term. To do so, we transform the Teukolsky equation without a source into the Sasaki–Nakamura equation.

Teukolsky equation (2.6) without source term can be rewritten as

$$\begin{aligned} \biggl [{\mathcal {A}}\frac{d}{dr}{\mathcal {B}}\frac{d}{dr}+V(r) \biggr ]R_{\ell \omega }(r)=0, \end{aligned}$$
(2.14)

with

$$\begin{aligned} {\mathcal {A}}=r^{4}A\sqrt{\frac{A}{B}}, \quad {\mathcal {B}}=\frac{1}{r^{2}A^{2}}\sqrt{\frac{A}{B}}. \end{aligned}$$
(2.15)

The general transformation of \(R_{ \ell \omega }\) can be taken as

$$\begin{aligned} \chi _{ \ell \omega }=\alpha R_{ \ell \omega }+\frac{\beta }{{{\mathcal {A}}}{{\mathcal {B}}}}R_{ \ell \omega }', \end{aligned}$$
(2.16)

where \(\alpha \) and \(\beta \) are functions of r, and a prime denotes the derivative with respect to r. By using Eq. (2.14) and taking the first derivative of Eq. (2.16) with respect to r, one obtains the inverse transformations

$$\begin{aligned} R_{ \ell \omega }= & {} \frac{1}{\gamma }\left( Z\chi _{ \ell \omega }-\frac{\beta }{{{\mathcal {A}}}{{\mathcal {B}}}}\chi _{ \ell \omega }^{\prime } \right) , \end{aligned}$$
(2.17)
$$\begin{aligned} R^{\prime }_{ \ell \omega }= & {} \frac{1}{\gamma }\left[ -\left( \alpha ^{\prime }-\frac{V(r)\beta }{{\mathcal {A}}^{2}{\mathcal {B}}^{2}} \right) \chi _{ \ell \omega }+\alpha \chi _{ \ell \omega }^{\prime } \right] , \end{aligned}$$
(2.18)

where

$$\begin{aligned} Z= & {} \alpha +\left( \frac{\beta }{{{\mathcal {A}}}{{\mathcal {B}}}} \right) '-\frac{\beta }{{{\mathcal {A}}}{{\mathcal {B}}}} \frac{{\mathcal {B}}'}{{\mathcal {B}}},\\ \gamma= & {} \alpha Z-\frac{\beta }{{{\mathcal {A}}}{{\mathcal {B}}}}\left[ \alpha '- \frac{\beta }{({{\mathcal {A}}}{{\mathcal {B}}})^{2}}V(r)\right] . \end{aligned}$$

By taking the first derivative of Eq. (2.17) and using Eq. (2.18), we obtain the equation of motion of \(\chi _{ \ell \omega }\)

$$\begin{aligned} {{\mathcal {A}}}{{\mathcal {B}}}\frac{d^{2}\chi _{ \ell \omega }}{dr^{2}}+\left( {\mathcal {A}}\frac{d{\mathcal {B}}}{dr} -{{\mathcal {A}}}{{\mathcal {B}}}\frac{\gamma '}{\gamma }\right) \frac{d\chi _{ \ell \omega }}{dr}-U(r)\chi _{ \ell \omega }=0, \nonumber \\ \end{aligned}$$
(2.19)

where

$$\begin{aligned} U(r)=-V(r)+\frac{({{\mathcal {A}}}{{\mathcal {B}}})^{2}}{\beta } \left[ \frac{d}{dr}(\alpha +Z)-Z\frac{\gamma '}{\gamma }\right] . \end{aligned}$$

Taking the transformation

$$\begin{aligned} \chi _{ \ell \omega }=r^{4}L_{-}\left[ \frac{r^{2}}{{{\mathcal {A}}}{{\mathcal {B}}}}L_{-}\left( \frac{R_{ \ell \omega }}{r^{2}}\right) \right] , \end{aligned}$$
(2.20)

where \(L_{-}=\frac{d}{dr^{*}}-i\omega =\sqrt{\frac{A}{B}}\frac{d}{dr}-i\omega \), and comparing the coefficients of Eqs. (2.16) and (2.20), we get \(\alpha \) and \(\beta \) as

$$\begin{aligned} \alpha= & {} r^{4}L_{-}\left[ \frac{r^{2}}{{{\mathcal {A}}} {{\mathcal {B}}}}L_{-}\left( \frac{1}{r^{2}}\right) \right] -ABV(r), \nonumber \\ \beta= & {} r{{\mathcal {A}}}{{\mathcal {B}}}\left[ -2ir\omega \sqrt{AB} +\frac{r^{3}}{B}\frac{d}{dr}\left( \frac{AB}{r^{2}}\right) +r\frac{dA}{dr}\right] .\nonumber \\ \end{aligned}$$
(2.21)

Now we introduce a new function

$$\begin{aligned} \chi _{ \ell \omega }=H X_{ \ell \omega }, \end{aligned}$$
(2.22)

then Eq. (2.19) reduces to the Sasaki–Nakamura equation

$$\begin{aligned} r^{3}\left[ \frac{d^{2}}{dr^{*2}}-\sqrt{\frac{A}{B}} \frac{\gamma '}{\gamma }\frac{d}{dr^{*}}+{{\tilde{U}}(r)}\right] X_{ \ell \omega }=0, \end{aligned}$$
(2.23)

with

$$\begin{aligned}&{\tilde{U}}(r)=-\frac{1}{r^{3}}\left[ HU(r)+{\mathcal {A}} {\mathcal {B}}\frac{\gamma '}{\gamma }H'-{\mathcal {A}} \frac{d}{dr}({\mathcal {B}}H')\right] .\nonumber \\&H=rA. \end{aligned}$$

Then the asymptotic solutions of \(X^{in}_{\ell \omega }\) can be expressed as

$$\begin{aligned} X_{\ell \omega }^{\mathrm {in}}(r)=\left\{ \begin{array}{ll} C_{\ell \omega } e^{-i \omega r^{*}} &{} r^{*} \rightarrow -\infty , \\ A_{\ell \omega }^{ {out }} e^{i \omega r^{*}}+A_{\ell \omega }^{ {in }} e^{-i \omega r^{*}} &{} r^{*} \rightarrow +\infty , \end{array}\right. \end{aligned}$$
(2.24)

where \(A^{in}_{\ell \omega }\) is related to \(B^{in}_{\ell \omega }\) appeared in Eq. (2.13) as

$$\begin{aligned} B^{in}_{\ell \omega }= -\frac{1}{4\omega ^2}A^{in}_{\ell \omega }. \end{aligned}$$
(2.25)

3 The solutions of the Teukolsky equation

In this section, we first study the solutions \(X^{in}_{\ell \omega }\) and amplitudes \(A^{in}_{\ell \omega }\) of the Sasaki–Nakamura equation, and then look for the solutions of the Teukolsky equation with source.

3.1 The solutions of the Sasaki–Nakamura equation

The method employed in this subsection is mainly based on the work of Sasaki [28] and Mino et al. [38]. We first introduce the variable \(z=\omega r\), and take the ansatz

$$\begin{aligned} X_{\ell \omega }^{in}=e^{- 2iGM\omega \left[ b_{1} \ln (z - 2c_{1}GM\omega )- b_{2} \ln (z - 2c_{2}GM\omega )\right] } z \xi _{\ell } (z),\nonumber \!\!\!\!\!\\ \end{aligned}$$
(3.1)

where \(c_{1}=\frac{1+\sqrt{1-a_{2}}}{2}\) and \(c_{2}=\frac{1-\sqrt{1-a_{2}}}{2}\). Inserting Eq. (3.1) into Eq. (2.23) and expanding it in powers of \(2GM\omega \), we obtain

$$\begin{aligned} L^{(0)}[\xi _{\ell }]=2GM\omega L^{(1)}[\xi _{\ell }]+(2GM\omega )^{2} L^{(2)}[\xi _{\ell }], \end{aligned}$$
(3.2)

where

$$\begin{aligned} L^{(0)}= & {} \frac{d^{2}}{ dz^2}+\frac{2}{ z}\frac{d}{dz} +\biggl [1-\frac{\ell (\ell +1)}{ z^2}\biggr ],\nonumber \\ \end{aligned}$$
(3.3)
$$\begin{aligned} L^{(1)}= & {} \frac{1}{z}\frac{d^{2}}{ dz^2}+\frac{1+2iz}{z^{2}} \frac{d}{ dz} -\frac{4+z(-i+z)}{z^{3}},\nonumber \\ \end{aligned}$$
(3.4)
$$\begin{aligned} L^{(2)}= & {} -\frac{a_{2}}{4z^{2}}\frac{d^{2}}{ dz^2}-\biggl [\frac{a_{2}(12+iz(12+\lambda (2+\lambda ))) }{2z^{3}\lambda (2+\lambda )}\nonumber \\&+\frac{d_{2}(-12+5\lambda (2+\lambda )-iz(12+\lambda (6+\lambda )))}{2z^{3}\lambda (2+\lambda )} \biggr ]\frac{d}{ dz}\nonumber \\&-\biggl [\frac{a_{2}(48+48iz-4\lambda -8\lambda ^{2}-z^{2}(24+\lambda (2+\lambda )))}{4z^{4}\lambda (2+\lambda )}\nonumber \\&+\frac{2d_{2}(-24-4iz(6+\lambda ))\!-\!\lambda (3\!+\!\lambda ) (8\!+\!\lambda )\!+\!z^{2}(12\!+\!\lambda (6\!+\!\lambda ))}{4z^{4}\lambda (2\!+\!\lambda )}\!\biggr ].\nonumber \\ \end{aligned}$$
(3.5)

In the low frequency limit and by noting that \(2GM\omega \) only appears on the right-hand side of Eq. (3.2), we may look for the solution of \(\xi _{\ell }(z)\) perturbatively in terms of \(\epsilon =2GM\omega \), i.e.

$$\begin{aligned} \xi _{\ell }(z) =\sum _{n=0}^{\infty }\epsilon ^n \xi ^{(n)}_{\ell }(z), \end{aligned}$$
(3.6)

and one obtains the recursive equations from Eq. (3.2)

$$\begin{aligned} L^{(0)}[\xi ^{(n)}_{\ell }]=W^{(n)}_{\ell }, \end{aligned}$$
(3.7)

where

$$\begin{aligned} W^{(0)}_{\ell }= & {} 0 , \end{aligned}$$
(3.8)
$$\begin{aligned} W^{(1)}_{\ell }= & {} L^{(1)}[\xi ^{(0)}_{\ell }], \end{aligned}$$
(3.9)
$$\begin{aligned} W^{(2)}_{\ell }= & {} L^{(1)}[\xi ^{(1)}_{\ell }]+L^{(2)}[\xi ^{(0)}_{\ell }]. \end{aligned}$$
(3.10)

The general solution of Eq. (3.8) corresponding to the case of \(n=0\) is

$$\begin{aligned} \xi ^{(0)}_{\ell }=\alpha ^{(0)}j_{\ell }+\beta ^{(0)}n_{\ell }, \end{aligned}$$
(3.11)

where \(j_{\ell }\) and \(n_{\ell }\) are usual spherical Bessel functions. Considering the boundary condition that \(\xi ^{(n)}_\ell \) is regular at \(z = 0\) for \(n< 1\), we have \(\xi ^{(0)}_{\ell }=\alpha ^{(0)}j_{\ell }\), and for convenience and without loss of generality we set \(\alpha ^{(0)}= 1\), and thus we have \(\xi ^{(0)}_{\ell }=j_{\ell }\). To calculate the cases for \(n \ge 1\), we rewrite Eqs. (3.9) and (3.10) in the indefinite integral form by using the spherical Bessel functions

$$\begin{aligned} \xi ^{(n)}_{\ell }=n_{\ell }\int ^{z}dz z^{2}j_{\ell }W^{(n)}_{\ell }-j_{\ell }\int ^{z}dz z^{2}n_{\ell }W^{(n)}_{\ell } \quad (n=1,2). \end{aligned}$$
(3.12)

The solution of Eq. (3.9) corresponding to the case of \(n=1\) is given by

$$\begin{aligned}&\xi ^{(1)}_{\ell } =\frac{(\ell -1)(\ell +3)}{2(\ell +1)(2\ell +1)}j_{\ell +1} -\left[ \frac{\ell ^2-4}{2\ell (2\ell +1)} +\frac{2\ell -1}{\ell (\ell -1)}\right] j_{\ell -1} \nonumber \\&\quad +z^2(n_\ell j_0-j_\ell n_0)j_0 +\sum _{k=1}^{\ell -2}\left( \frac{1}{k}+\frac{1}{k+1}\right) z^2(n_\ell j_k-j_\ell n_k)j_k \nonumber \\&\quad +n_\ell \left( \mathrm{Ci}2z-\gamma -\ln 2z\right) -j_\ell \mathrm{Si}2z +ij_\ell \ln z+\alpha _\ell ^{(1)}j_\ell +\beta _\ell ^{(1)}n_\ell ,\nonumber \\ \end{aligned}$$
(3.13)

where \(\mathrm{Si}x=\int _0^xdt\sin t/t\) and \(\mathrm{Ci}x=-\int _x^\infty dt\cos t/t\) are the sine and cosine integral functions, respectively. We set the integration constant \(\alpha ^{(1)}_{\ell }=0\) for simplicity, and then the regular boundary condition of \(z \xi ^{(1)}_{\ell }\) lead to \(\beta ^{(1)}_\ell =0\).

Then \(X^{in}_{\ell \omega }\) may be obtained from \(\xi _\ell \) straightforwardly. In general, \(\xi ^{(n)}_\ell \) may be decomposed into real and imaginary parts, denoted by \(f^{(n)}_\ell \) and \(g^{(n)}_\ell \), respectively, i.e.

$$\begin{aligned} \xi ^{(n)}_\ell =f^{(n)}_\ell +ig^{(n)}_\ell . \end{aligned}$$
(3.14)

Inserting this expression into Eq. (3.1) and expanding the result with respect to \(\epsilon \), we obtain the conventional expansion of \(X^{in}_{\ell \omega }\) as

$$\begin{aligned}&X^{(0)}_{\ell \omega }=zf^{(0)}_\ell =zj_\ell \,\quad X^{(1)}_{\ell \omega }=zf^{(1)}_\ell \,\nonumber \\&\quad X^{(2)}_{\ell \omega }=z\left[ f^{(2)}_\ell +\frac{1}{2}(b_{1}-b_{2})^{2}j_\ell (\ln z)^2\right] , \ldots . \end{aligned}$$
(3.15)

To evaluate \(A^{in}_{\ell \omega }\), we only need to examine the asymptotic behavior of \(\xi ^{(n)}_{\ell }\) at infinity. For this purpose, we present the asymptotic form of \(f^{(n)}_\ell \) at \(z\rightarrow \infty \) as

$$\begin{aligned} f^{(n)}_\ell \rightarrow P^{(n)}_\ell j_\ell +Q^{(n)}_\ell n_\ell \quad (n=1,2), \end{aligned}$$
(3.16)

with

$$\begin{aligned} P^{(1)}_\ell= & {} p^{(1)}_\ell ,\nonumber \\ Q^{(1)}_\ell= & {} q^{(1)}_\ell -(b_{1}-b_{2})\ln z,\nonumber \\ P^{(2)}_\ell= & {} p^{(2)}_\ell +(b_{1}-b_{2})q^{(1)}_\ell \ln z-(b_{1}-b_{2})^{2}(\ln z)^2,\nonumber \\ Q^{(2)}_\ell= & {} q^{(2)}_\ell -(b_{1}-b_{2})p^{(1)}_\ell \ln z, \end{aligned}$$
(3.17)

where \(p^{(n)}_\ell \) and \(q^{(n)}_\ell \) are constants, and \(p^{(1)}_\ell \) and \(q^{(1)}_\ell \) can be obtained from the asymptotic behavior of \(\xi ^{(1)}_\ell \) at \(z\rightarrow \infty \).

Now we can read off the incident amplitude \(A^{in}_{\ell \omega }\) form the asymptotic form of \(X^{in}_{\ell \omega }\), which is

$$\begin{aligned} A^{in}_{\ell \omega }= & {} \frac{1}{2}i^{\ell +1}e^{-i\epsilon \ln \epsilon (\sqrt{1-a_{2}})} \left[ 1+\epsilon \left( p^{(1)}_\ell +iq^{(1)}_\ell \right) \right. \nonumber \\&\left. +\epsilon ^2\left( p^{(2)}_\ell +iq^{(2)}_\ell \right) +\cdots \right] . \end{aligned}$$
(3.18)

The iterative equation (3.10) for the case of \(n=2\) consists of the terms of \(L^{(1)}[\xi ^{(1)}_{\ell }]\) and \(L^{(2)}[\xi ^{(0)}_{\ell }]\). The result of \(L^{(1)}[\xi ^{(1)}_{\ell }]\) is the same as the counterpart obtained by Sasaki [28], and \(L^{(2)}[\xi ^{(0)}_{\ell }]\) is our correction term. Note that Eq. (3.15) is related to the real parts \(f^{(n)}_{\ell }\), where \(f^{(1)}_{\ell }\) can be easily obtained from the Eq. (3.13), \(f^{(2)}_{\ell }\) can be acquired from Eq. (3.12), and for the case of \(\ell =2\) we have

$$\begin{aligned} f^{(2)}_{2}= & {} -\frac{389}{70z^2}j_0-\frac{113}{420z}j_1+\frac{1}{7z}j_3 +B_n(z) j_2\nonumber \\&-B_j(z) n_2 +{C}(z)\left( \frac{5}{6z}n_2-\frac{5}{3}n_1-\frac{3}{z}n_0-\frac{107}{210}j_2\right) \nonumber \\&-S(z)\left( \frac{5}{6z}j_2-\frac{5}{3}j_1-\frac{3}{ z}j_0\right. \nonumber \\&\left. +\frac{107}{210}n_2\right) -\frac{107}{210}j_2\ln z-\frac{1}{2}j_2(\ln z)^2+\alpha ^{(2)}_2j_2 \nonumber \\&+ {\mathcal {N}}_{1}\left( \frac{1}{840} C(z) j_2+\frac{1}{840}\ln zj_2+\frac{1}{840} S(z) n_2-\frac{1}{420z}n_2\right. \nonumber \\&\left. +\frac{1}{420}n_0\frac{3-z^{2}}{z^{3}}\right) -\frac{{\mathcal {N}}_{2}}{840z}\left( \frac{2}{z}j_2-j_3\right) +\frac{{\mathcal {N}}_{3}j_2}{420 z^{2}}, \end{aligned}$$
(3.19)

with

$$\begin{aligned} {\mathcal {N}}_{1}= & {} 4 (21 {{\mathcal {I}}}+11 {a_{2}}+2 {{\mathcal {J}}}+{{\mathcal {Y}}}),\nonumber \\ {\mathcal {N}}_{2}= & {} -168 {\mathcal {I}}-188a_{2}+4 {\mathcal {J}}+2{\mathcal {Y}}, \nonumber \\ {\mathcal {N}}_{3}= & {} 252 {\mathcal {I}}+102a_{2}+9 {\mathcal {J}}-48{\mathcal {Y}}+7(2a_{2}+{\mathcal {Y}})z^{2}, \nonumber \\ \end{aligned}$$
(3.20)

where

$$\begin{aligned} {\mathcal {I}}= & {} \frac{a_{2} [24 + \lambda (2 + \lambda )]}{\lambda (2 + \lambda )}-\frac{2 d_{2} [12 + \lambda (6 + \lambda )]}{\lambda (2 + \lambda )},\nonumber \\ {\mathcal {J}}= & {} -\left[ -\frac{4 {a_{2}} \left( 2 \lambda ^2+\lambda -12\right) }{\lambda (\lambda +2)}-\frac{2 {d_{2}} (\lambda +3) (\lambda +8)}{\lambda +2}-\frac{48 {d_{2}}}{\lambda (\lambda +2)}\right] , \nonumber \\ {\mathcal {Y}}= & {} -\left[ \frac{24 {a_{2}}}{\lambda (\lambda +2)}-\frac{24 {d_{2}}}{\lambda (\lambda +2)}+10 {d_{2}}\right] . \end{aligned}$$
(3.21)

Note that here \(B_{n}(z)\), \(B_{j}(z)\), C(z) and S(z) are as defined in [28]. The asymptotic behavior of \(f^{(2)}_2\) is easy to evaluate at \(z\rightarrow \infty \), by comparing Eq. (3.16), the amplitude \(A^{in}_{2\omega }\) upto \(O(\epsilon ^2)\) is found to be

$$\begin{aligned} A^{in}_{2\omega }= & {} -\frac{1}{2}ie^{-i\epsilon (\ln 2\epsilon (\sqrt{1-a_{2}})+\gamma )} \exp \left( i\epsilon \frac{5}{3}-i\epsilon ^2\frac{428-{\mathcal {N}}_{1}}{1680}\pi \right) \nonumber \\&\times \biggl \{1-\epsilon \frac{\pi }{2} +\epsilon ^2 \biggl [\frac{25}{18}+\frac{5}{24}\pi ^2+\left( {\frac{107}{210}-\frac{{\mathcal {N}}_{1}}{840}}\right) (\gamma +\ln 2)\nonumber \\&+\frac{2a_{2}+{\mathcal {Y}}}{60} \biggr ]+\cdots \biggr \}. \end{aligned}$$
(3.22)

Similarly, we may derive

$$\begin{aligned} A^{in}_{3\omega }= & {} \frac{1}{2}e^{-i\epsilon (\ln 2\epsilon (\sqrt{1-a_{2}})+\gamma )} \exp \left( i\epsilon \frac{13}{6}-i\epsilon ^2\frac{780-{\mathcal {N}}_{4}}{5040}\pi \right) \nonumber \\&\times \biggl \{1-\epsilon \frac{\pi }{2} +\epsilon ^2 \biggl [\frac{169}{72}+\frac{5}{24}\pi ^2+\biggl ({\frac{13}{42}-\frac{{\mathcal {N}}_{4}}{2520}}\biggr )(\gamma +\ln 2)\nonumber \\&+\frac{2a_{2}+{\mathcal {Y}}}{140} \biggr ]+\cdots \biggr \},\nonumber \\ A^{in}_{4\omega }= & {} \frac{1}{2}ie^{-i\epsilon (\ln 2\epsilon (\sqrt{1-a_{2}})+\gamma )} \exp \left( i\epsilon \frac{149}{60}-i\epsilon ^2\frac{37704-5{\mathcal {N}}_{5}}{332640}\pi \right) \nonumber \\&\times \biggl \{1-\epsilon \frac{\pi }{2} +\epsilon ^2 \biggl [\frac{22201}{7200}+\frac{5}{24}\pi ^2+\biggl ({\frac{1571}{6930}-\frac{{\mathcal {N}}_{5}}{33264}}\biggr )\nonumber \\&(\gamma +\ln 2)+\frac{2a_{2}+{\mathcal {Y}}}{252} \biggr ]+\cdots \biggr \}, \end{aligned}$$
(3.23)

with

$$\begin{aligned} {{\mathcal {N}}_{4}}= & {} 4 (45 {{\mathcal {I}}}+23 {a_{2}}+2 {{\mathcal {J}}}+{{\mathcal {Y}}}),\nonumber \\ {{\mathcal {N}}_{5}}= & {} 24 (77 {{\mathcal {I}}}+39 {a_{2}}+2 {{\mathcal {J}}}+{{\mathcal {Y}}}). \end{aligned}$$
(3.24)

The corresponding incident amplitudes \(B^{in}_{\ell \omega }\) for the Teukolsky function can be obtained from Eq. (2.25)

$$\begin{aligned}&B_{2 \omega }^{i n}=\frac{i}{8\omega ^{2}} e^{-i \epsilon [\ln (\epsilon \sqrt{1-a_{2}})]}\nonumber \\&\quad \times \biggl \{1+\epsilon \biggl [\frac{5i}{3}-\frac{\pi }{2}-i(\gamma +\ln 2)\biggr ]+\epsilon ^{2}\biggl [\frac{a_{2}-9d_{2}}{60}\nonumber \\&\quad +\frac{i \pi (-457+60a_{2}-40 d_{2})}{420}\nonumber \\&\quad +\frac{5 \pi ^{2}}{24}+\frac{457-60a_{2}+40d_{2}}{210}(\gamma +\ln 2)\nonumber \\&\quad +\frac{i\pi }{2} (\gamma +\ln 2)-\frac{(\gamma +\ln 2)^{2}}{2}\biggr ]\biggr \},\nonumber \\&B_{3 \omega }^{i n}=-\frac{1}{8\omega ^{2}} e^{-i \epsilon [\ln (\epsilon \sqrt{1-a_{2}})]}\nonumber \\&\quad \times \biggl \{1+\epsilon \biggl [\frac{13 i}{6}-\frac{\pi }{2}-i(\gamma +\ln 2)\biggr ]\nonumber \\&\quad +\epsilon ^{2}\biggl [\frac{9a_{2}-49d_{2}}{700} +\frac{i \pi (-52+3a_{2}-2 d_{2})}{42} +\frac{5 \pi ^{2}}{24}\nonumber \\&\quad +\frac{52-3a_{2}+2d_{2}}{21}(\gamma +\ln 2)\nonumber \\&\quad +\frac{i\pi }{2} (\gamma +\ln 2)-\frac{(\gamma +\ln 2)^{2}}{2}\biggr ]\biggr \}, \nonumber \\&B_{4 \omega }^{i n}=-\frac{i}{8\omega ^{2}} e^{-i \epsilon [\ln (\epsilon \sqrt{1-a_{2}})]}\biggl \{1+\epsilon \biggl [\frac{149 i}{60}-\frac{\pi }{2}-i(\gamma +\ln 2)\biggr ]\nonumber \\&\quad +\epsilon ^{2}\biggl [\frac{29a_{2}-149d_{2}}{3780}\nonumber \\&\quad -\frac{i \pi (37561-1356a_{2}+904 d_{2})}{27720}\nonumber \\&\quad +\frac{5 \pi ^{2}}{24}+\frac{37561-1356a_{2}+904d_{2}}{13860}(\gamma +\ln 2)\nonumber \\&\quad +\frac{i\pi }{2} (\gamma +\ln 2)-\frac{(\gamma +\ln 2)^{2}}{2}\biggr ]\biggr \}. \end{aligned}$$
(3.25)

Next, we consider \(X^{in}_{\ell \omega }\). Inserting \(f^{(n)}_{\ell }\) into Eq. (3.15) we obtain

$$\begin{aligned} X^{in}_{2\omega }= & {} \frac{z^{3}}{15}-\frac{z^{5}}{210}+\frac{z^{7}}{7560}+\epsilon \biggl ( -\frac{13z^{4}}{630} \biggr )+\epsilon ^{2}\biggl ( -\frac{d_{2}z}{15} \biggr ) , \nonumber \\ X^{in}_{3\omega }= & {} \frac{z^{4}}{105}-\frac{z^{6}}{1890}+\frac{z^{8}}{83160}\nonumber \\&+\epsilon \biggl ( -\frac{z^{3}}{126} -\frac{z^{5}}{630} \biggr )+\epsilon ^{2}\biggl ( -\frac{{d_{2}}z^{2}}{210} \biggr ), \nonumber \\ X^{in}_{4\omega }= & {} \frac{z^{5}}{945}-\frac{z^{7}}{20790}+\frac{z^{9}}{1081080}+\epsilon \biggl ( -\frac{z^{4}}{630} -\frac{z^{6}}{9900}\biggr )\nonumber \\&+\epsilon ^{2}\biggl (\frac{1}{1764}+\frac{a_{2}}{5292}-\frac{d_{2}}{1764}\biggr )z^{3}. \end{aligned}$$
(3.26)

By using the transformation given by Eq. (2.17), we obtain the corresponding solutions \(R^{in}_{\ell \omega }\) of the Teukolsky equation without the source

$$\begin{aligned}&\omega R_{2 \omega }^{\mathrm {in}}=\frac{z^{4}}{30}+\frac{iz^{5}}{45} -\frac{11 z^{6}}{1260}-\frac{iz^{7}}{420} +\frac{23 z^{8}}{45360}+\frac{iz^{9}}{11340}\nonumber \\&\quad +\epsilon \left( \frac{-z^{3}}{15}-\frac{iz^{4}}{60} -\frac{41z^{5}}{3780}-\frac{31iz^{6}}{3780} \right) \nonumber \\&\quad +\epsilon ^{2}\biggl [\frac{(4+2a_{2}-3d_{2})z^{2}}{120}-\frac{i(12+3a_{2}-2d_{2})z^{3}}{720}\biggr ],\nonumber \\&\omega R_{3 \omega }^{\mathrm {in}}=\frac{z^{5}}{630}+\frac{iz^{6}}{1260} -\frac{ z^{7}}{3780}-\frac{iz^{8}}{16200}+\frac{29z^{9}}{2494800}\nonumber \\&\quad +\epsilon \left( \frac{-z^{4}}{252}-\frac{iz^{5}}{756} -\frac{iz^{7}}{9450}\right) \nonumber \\&\quad +\epsilon ^{2}\biggl [\frac{(20+5a_{2}-7d_{2})z^{3}}{6300} \biggr ],\nonumber \\&\omega R_{4 \omega }^{\mathrm {in}}=\frac{z^{6}}{11340}+\frac{iz^{7}}{28350} -\frac{13 z^{8}}{1247400}-\frac{iz^{9}}{467775}\nonumber \\&\quad +\frac{71 z^{10}}{194594400} +\epsilon \left( \frac{-z^{5}}{3780}-\frac{11iz^{6}}{136080} +\frac{131z^{7}}{18711000} \right) \nonumber \\&\quad +\epsilon ^{2}\biggl [\frac{(540+103a_{2}-141d_{2})z^{4}}{1905120} \biggr ]. \end{aligned}$$
(3.27)

3.2 The solution of the Teukolsky equation with the source

In order to obtain the solution of the Teukolsky equation with the source term, we rewrite the source term \(T_{\ell m \omega }\) schematically in the form of summation, by applying a differential operator to \({ }_{s} T_{\ell m \omega }(r)\)

$$\begin{aligned} T_{\ell m \omega }=2 \pi G\sum _{s}~{ }_{s} p_{\ell }~{ }_{s} D_{\omega }~{ }_{s} T_{\ell m \omega }(r), \end{aligned}$$
(3.28)

where the functions \({ }_{s} T_{\ell m \omega }(r)\) is given in Eq. (2.10), the constants \({ }_{s} p_{\ell }\) are

$$\begin{aligned} { }_{s} p_{\ell }=\left\{ \begin{array}{ll} 2[(\ell -1) \ell (\ell +1)(\ell +2)]^{1 / 2} &{}\quad s=0, \\ 2[2(\ell -1)(\ell +2)]^{1 / 2} &{}\quad s=-1, \\ 1 &{} s=-2, \end{array}\right. \end{aligned}$$
(3.29)

and the differential operators are

$$\begin{aligned} { }_{s} D_{\omega }=\left\{ \begin{array}{ll} r^{4} &{}\quad s=0, \\ r^{2} A {\mathcal {L}} r^{3} A^{-1} &{}\quad s=-1, \\ r A {\mathcal {L}} r^{4} A^{-1} {\mathcal {L}} r &{}\quad s=-2. \end{array}\right. \end{aligned}$$
(3.30)

Now by using the inhomogeneous Teukolsky variable at infinity Eq. (2.12), we get

$$\begin{aligned} {\tilde{Z}}_{\ell m \omega }=\frac{R_{\ell m \omega }}{r^3 e^{i \omega r^*}}=\left[ 2 i \omega B_{\ell \omega }^{i n}\right] ^{-1} \int _{r_{h}}^{\infty } d r r^{-4} \sqrt{B/A^{3}} R_{\ell \omega }^{\mathrm {in}} T_{\ell m \omega }(r),\nonumber \\ \end{aligned}$$
(3.31)

where \({\tilde{Z}}_{\ell m \omega }\equiv {\tilde{Z}}_{\ell m \omega }(r)\). From Eqs. (2.9) and (3.31), we see that \({\tilde{Z}}_{\ell m \omega }\) takes the form

$$\begin{aligned} {\tilde{Z}}_{\ell m \omega }=Z_{\ell m} \delta (\omega -m \Omega ). \end{aligned}$$
(3.32)

To carry out the integral in Eq. (3.31), we define an inner product as

$$\begin{aligned} \left( g_{1}, g_{2}\right) :=\int _{r_{h}}^{+\infty } d r r^{-4} \sqrt{B/A^{3}} g_{2} g_{1}. \end{aligned}$$
(3.33)

By making use of the relations

$$\begin{aligned}&\left( {\mathcal {L}}~_{s} T, R\right) =\left( { }_{s} T, {\mathcal {L}}^{\dagger } R\right) , \end{aligned}$$
(3.34)
$$\begin{aligned}&\left( { }_{s} D_{\omega }~{ }_{s} T, R\right) =\left( { }_{s} T,{ }_{s} D_{\omega }^{\dagger }~ R\right) , \end{aligned}$$
(3.35)

with

$$\begin{aligned}&{\mathcal {L}}^{\dagger }=-r^{4}A \overline{{\mathcal {L}}} r^{-4}A^{-1}, \end{aligned}$$
(3.36)
$$\begin{aligned}&\overline{{\mathcal {L}}}=\sqrt{\frac{A}{B}} \frac{d}{d r}-i \omega , \end{aligned}$$
(3.37)
$$\begin{aligned}&{ }_{s} D_{\omega }^{\dagger }=\left\{ \begin{array}{ll} r^{4} &{} s=0, \\ -r^{7} \overline{{\mathcal {L}}}r^{-2} &{} s=-1, \\ r^{5} A \overline{{\mathcal {L}}}r^{4} A^{-1} \overline{{\mathcal {L}}}r^{-3} &{} s=-2, \end{array}\right. \end{aligned}$$
(3.38)

it can be deduced that

$$\begin{aligned}&Z_{\ell m} =2\pi \frac{\pi G}{ i\omega B^{in}_{\ell \omega }}\Biggl \{ \Biggl [\sqrt{B/A^{3}}\,_0p_{\ell } \,_0T_{\ell m} +2\,A^{-1}\,{ }_{-1}p_{\ell }\,_{-1}T_{\ell m}\nonumber \\&\quad \times \left( 1+\frac{i\omega r_0}{2\sqrt{A/B}}\right) +2\sqrt{\frac{1}{AB}}\,{ }_{-2}p_{\ell }\,_{-2}T_{\ell m}\nonumber \\&\quad \times \Biggl [ i \omega r_{0} \frac{B}{A}\Biggl (\frac{1}{2} i \omega r_{0}+\sqrt{\frac{A}{B}}+\frac{r_{0} A^{\prime }}{2A} \sqrt{\frac{A}{B}} \Biggr )\nonumber \\&\quad +\frac{3r_{0}}{4}\Biggl ( \frac{A^{\prime }}{A}+\frac{B^{\prime }}{B}\Biggr ) \Biggr ] \Biggr ] R^{in}_{\ell \omega }(r_0)\nonumber \\&\quad +\Biggl [-A^{-1}\,{ }_{-1}p_{\ell }\,_{-1}T_{\ell m} -2\sqrt{\frac{1}{AB}}\,{ }_{-2}p_{\ell }\,_{-2}T_{\ell m}\nonumber \\&\quad \times \Biggl [i \omega r_{0} \sqrt{\frac{B}{A}}+1 +\frac{r_{0}}{4}\Biggl ( \frac{A^{\prime }}{A}+\frac{B^{\prime }}{B}\Biggr )\Biggr ]\Biggr ] r_0 R^{in}_{\ell \omega }{}'(r_0) \nonumber \\&\quad +\sqrt{\frac{1}{AB}}\,{ }_{-2}p_{\ell }\,_{-2}T_{\ell m}\,{r_0}^2 R^{in}_{\ell \omega }{}''(r_0) \Biggr \}. \end{aligned}$$
(3.39)

4 Reaction force

In this section, we first present an analytical expression for the energy-loss-rate dE/dt of gravitational waves, and then derive the radiation reaction force acting on a particle.

4.1 Energy-loss-rate

By noticing the symmetry of the spin weighted spherical harmonics \(\,_{s}Y_{\ell ,- m}(\frac{\pi }{2},0)=(-1)^{s+\ell }\,_{s}Y_{\ell m}(\frac{\pi }{2},0)\) and by using Eq. (3.39), we have \(Z_{\ell , -m}=(-1)^{\ell }\bar{Z}_{\ell m}\), where \(\bar{Z}_{\ell m}\) is the conjugate of \(Z_{\ell , -m}\). In terms of the amplitudes \(Z_{\ell m}\), the gravitational waveform at infinity are given by [26, 28, 61]

$$\begin{aligned} h_{+}-ih_{\times }=\frac{2}{r}\sum _{\ell m}\frac{1}{\omega ^2} {Z_{\ell m}}\,_{-2}Y_{\ell m}(\theta ,\varphi )e^{-i\omega (t-r^*)}, \end{aligned}$$
(4.1)

and the energy-loss-rate is described by [26, 29]

$$\begin{aligned} \frac{dE}{dt}=\sum _{\scriptstyle \ell =2}^{\infty }\sum _{m=1}^\ell \dfrac{\mid Z_{\ell m}\mid ^2}{2\pi \omega ^2}. \end{aligned}$$
(4.2)

By substituting Eq. (3.39) into Eq. (4.2), we may obtain the explicit form of the energy-loss-rate for the case we considered in this paper.

In order to show the difference between our results and that obtained by Tagoshi and Sasaki [29], we list a few cases of the energy-loss-rate (4.2) we obtained as follows

$$\begin{aligned}&dE/dt_{(\ell =2,m=2)}= G^{2}r^{4}_{0} m_0^{2}\omega ^{6}\biggl (\frac{1}{10}-\frac{r^{2}_{0}\omega ^{2}}{126}+\frac{r^{4}_{0}\omega ^{4}}{2646}\biggr )\\&\quad - G^{5/2}M^{1/2}r^{9/2}_{0} m_0^{2}\omega ^{7}\biggl (\frac{2}{45}-\frac{r^{2}_{0}\omega ^{2}}{189}\biggr )\\&\quad + G^{3}M r^{3}_{0} m_0^{2}\omega ^{6}\biggl (-\frac{7}{18}+\frac{\pi r_{0}\omega }{5}\\&\quad -\frac{16r^{2}_{0}\omega ^{2}}{189}\biggr ) - G^{7/2}M^{3/2}r^{7/2}_{0} m_0^{2}\omega ^{7}\biggl (\frac{2}{27}-\frac{a_{2}}{45}\biggr )\\&\quad +G^{4}M^{2}r^{2}_{0} m_0^{2}\omega ^{6}\biggl [\frac{(36+4a_{2}-27d_{2})}{45}\biggr ], \\&dE/dt_{(\ell =2,m=1)}=G^{3}M r^{3}_{0} m_0^{2}\omega ^{6}\biggl (\frac{8}{45}-\frac{22 r^{2}_{0}\omega ^{2}}{945}\biggr )\\&\quad -G^{7/2}M^{3/2}r^{7/2}_{0} m_0^{2}\omega ^{7}\biggl (\frac{8}{135}\biggr )\\&\quad +G^{4}M^{2}r^{2}_{0} m_0^{2}\omega ^{6}\biggl (-\frac{2}{15}-\frac{8 a_{2}}{45}\biggr ),\\&dE/dt_{(\ell =3,m=3)}=G^{2}r^{6}_{0} m_0^{2}\omega ^{8}\biggl (\frac{1}{756}-\frac{r^{2}_{0}\omega ^{2}}{9072}\biggr )\\&\quad - G^{5/2}M^{1/2}r^{13/2}_{0} m_0^{2}\omega ^{9}\biggl (\frac{1}{1512}\biggr )\\&\quad +G^{3}M r^{5}_{0} m_0^{2}\omega ^{8}\biggl (-\frac{23}{3024}\biggr ), \end{aligned}$$
$$\begin{aligned}&dE/dt_{(\ell =3,m=2)}=G^{3}M r^{5}_{0} m_0^{2}\omega ^{8}\biggl (\frac{1}{504}\biggr ),\nonumber \\&dE/dt_{(\ell =3,m=1)}=G^{2}r^{6}_{0} m_0^{2}\omega ^{8}\biggl (\frac{1}{1260}\nonumber \\&\quad -\frac{r^{2}_{0}\omega ^{2}}{15120}\biggr )- G^{5/2}M^{1/2}r^{13/2}_{0} m_0^{2}\omega ^{9}\biggl (\frac{1}{7560}\biggr )\nonumber \\&\quad -G^{3}M r^{5}_{0} m_0^{2}\omega ^{8}\biggl (\frac{61}{15120}\biggr ),\nonumber \\&dE/dt_{(\ell =4,m=4)}=G^{2}r^{8}_{0} m_0^{2}\omega ^{10}\biggl (\frac{1}{72576}\biggr ),\nonumber \\&dE/dt_{(\ell =4,m=2)}=G^{2}r^{8}_{0} m_0^{2}\omega ^{10}\biggl (\frac{1}{127008}\biggr ). \end{aligned}$$
(4.3)

Noting \(\omega =m\sqrt{GM/r_{0}^{3}}\) for a given radius \(r_0\) and using the above results, we find that the energy-loss-rate at 2PM order can be expressed as

$$\begin{aligned} \frac{dE}{dt}= & {} \frac{32}{5}\frac{m_0^{2}}{M^{2}}\biggl (\frac{GM}{r_{0}}\biggr )^{5}\biggl [1-\frac{1247}{336}\frac{GM}{r_{0}} +4\pi \biggl (\frac{GM}{r_{0}}\biggr )^{3/2}\nonumber \\- & {} \biggl (\frac{44711}{9072}-\frac{47a_{2}}{36}+6d_{2} \biggr )\biggl (\frac{GM}{r_{0}}\biggr )^{2}\biggr ]. \end{aligned}$$
(4.4)

It is shown explicitly that the energy-loss-rate dE/dt is related to the parameters \(a_{2}\) and \(d_{2}\), and the 2PN results obtained in the Schwarzschild background [29] can be recovered when \(a_{2} = 0\) and \(d_{2} = 0\).

4.2 Reaction force

In this subsection, we calculate the reaction force acting on a particle. Considering quasi-circular motions and ignoring the radial damping force \({\mathcal {F}}_{r}\), Damour and Buonanno showed that the difference between the numerical results in the case of \({\mathcal {F}}_{r}=0\) and non-zero radial force is quite small. Therefore, we can obtain the reaction force by using the rate of energy-loss

$$\begin{aligned} {\mathcal {F}}^{circ}_{\varphi }=-\frac{1}{{\dot{\varphi }}}\sum _{\scriptstyle \ell =2}^{\infty }\sum _{m=1}^\ell \dfrac{\mid Z_{\ell m}\mid ^2}{2\pi \omega ^2}. \end{aligned}$$
(4.5)

For simplicity, we consider the case with equal mass, i.e., \(\nu =\frac{1}{4}\), and study the reaction force \({\mathcal {F}}^{circ}_{\varphi }\) with an orbital phase \(\varphi =0\) at \(r_{0} = 20\) for different \(\ell \) and m modes on the 2PM effective metric in the EOB theory. We present the energy-loss-rate upto the 2PM approximation with \(a_{2}=6(1-\frac{m_{1}+m_{2}}{{\mathcal {E}}})\), \(b_{2}=1+3 \frac{m_{1}+m_{2}}{{\mathcal {E}}}\) and \(d_{2}=-2+\frac{a_{2}+b_{2}}{2}\) in Table 1, and show the reaction force for the case that \(Z_{\ell m}\) does not expanded in Table 2.

Table 1 The reaction force for different \(\ell \),m modes at 2PM approximation of energy-loss-rate in the EOB theory, the total reaction force \({\mathcal {F}}^{circ}_{\varphi }=4.36336\times 10^{-5}\) in this case
Table 2 The reaction force for different \(\ell \), m modes for \(Z_{\ell m}\) does not expanded in the EOB theory, the total reaction force \({\mathcal {F}}^{circ}_{\varphi }=4.23409 \times 10^{-5}\) in this case

It can be seen from these tables that the reaction force is mainly determined by the \(\ell = m = 2\) mode. Therefore, we only compare our results with those obtained in other backgrounds for the particular case of the \(\ell = m = 2\) mode. From Table 1, we obtain the reaction force \({\mathcal {F}}_{2PM}=4.14183 \times 10^{-5}\) at the 2PM approximation of the energy-loss-rate. We also derive the reaction force \({\mathcal {F}}_{2PM.Sch}=4.12319\times 10^{-5}\) for the case \(a_{2} =d_{2}=0\), which is the same as the result obtained by using the 2PN expansion of the energy-loss-rate in Schwarzschild background [29].

5 Conclusion

Gravitational waves produced by coalescing binary system of compact objects have become the promising candidates for the ground based laser interferometric detectors. In order to detect gravitational waves, the theoretical templates are necessary. To construct gravitational waveform templates, the EOB formalism was put forward with the aim of analytically describing the gravitational wave signals emitted by coalescing binary black holes. The basic idea of the EOB theory is to map the relativistic dynamics of a two-body system (with masses \(m_{1}\), \(m_{2}\)) onto the relativistic dynamics of an effective test particle with mass \(m_0=m_{1}m_{2}/(m_{1}+m_{2})\) moving in an effective metric background of mass \(M = m_{1}+m_{2}\). The EOB theory is initially based on the PN approximation (v/c is assumed to be small). In 2016, Damour [56] introduced another EOB theory based on the PM approximation in which \(\frac{v}{c}\) is not required to be small anymore.

To construct the EOB theory based on the PM perturbation, the calculation of reaction force is a key step because the motion of test particle is affected by a reaction force due to the loss of energy and angular momentum of system in the process of gravitational radiation. The reaction force for the case that a test particle with mass \(m_0\) travels along a circular orbit around a Schwarzschild black hole based on the PN approximation was studied in Refs. [26,27,28,29,30,31,32]. In this paper we analytically studied the reaction force acting on the effective particle during the gravitational radiation of compact binaries based on the PM approximation in the EOB theory. First, we derived the inhomogeneous Teukolsky equation of gravitational perturbation on the effective metric, and transformed the corresponding homogeneous counterpart into the Sasaki–Nakamura equation. Second, we obtained the analytic solution \(X^{in}_{\ell \omega }\) of the Sasaki–Nakamura equation and the corresponding amplitude expressions \(A^{in}_{\ell \omega }\). Third, we got the corresponding solution \(R^{in}_{\ell \omega }\) of the Teukolsky equation without the source and amplitudes \(B^{in}_{\ell \omega }\) through the transformation equations. Finally, we found the solution of the Teukolsky equation with the source, which was expressed in terms of amplitudes \(Z_{\ell m}\). With the solution of the Teukolsky equation at hand, we may derive the energy-loss-rate dE/dt and the corresponding radiation reaction force.

We observed that, the \(\ell =m=2\) mode is the dominant term for the reaction force with the 2PM energy-loss-rate. Our results are quite close to the values obtained by Buonanno and Damour with the Pade approximation method [18]. We also show that the reaction force at the 2PM order for the case of \(a_{2}=0=d_{2}\) is the same as the result obtained by using the 2PN expansion in Schwarzschild background [29]. In the next step we would like to calculate the reaction force to higher-order PM approximations and generalize it to a binary system with spin.