1 Introduction

The structure formation at the early universe is one of the most important issues in modern cosmology [1,2,3]. The large scale structure formation is explained by the gravitational instabilities of the space-time metric and matter. To study these instabilities, it is necessary to eliminate the gauge ambiguities coming from the freedom in the coordinate choice in general relativity. To do this, usually a specific gauge is chosen (called gauge fixing), and then the results are expressed in terms of gauge invariant variables which represent the physical quantities.

On the other hand, the inflationary mechanism provides the initial condition for cosmic perturbations observed in the cosmic background radiation (CMB) today. The gravitational instability is amplified during inflationary era and at the end of it, the curvature perturbation, remains frozen at superhorizon scales. It is a gauge invariant quantity which provides the seed of galaxy formation at the time of horizon crossing during the radiation dominated era. Thus, we need to work out the equations of motion for the gauge invariant quantities, introduced by the scalar and tensor perturbations of metric and the matter field. The initial conditions of these equations can be found by considering the fluctuations at sufficiently early time of inflation when the perturbations are deep inside the horizon. At these very early times, the perturbations are essentially quantum fields which can be expanded in terms of creation and annihilation operators satisfying the standard commutation relations. Moreover, it is usually assumed that the initial quantum state of the inflaton field is the standard vacuum state,Footnote 1 Bunch–Davies vacuum. Putting all these points together, one can find the initial conditions of the classical equations governing the evolution of perturbations at inflationary era.

Here we are interested in deriving the initial conditions of modified Mukhanov–Sasaki equation in generalized scalar tensor gravity, described by the Lagrangian density \(\sqrt{-g}f(R,\phi )\) [11,12,13,14]. This is one of the most general modification of general relativity and includes the higher order terms of the scalar curvature, R and also a scalar field \(\phi \) as a dynamical degree of freedom in addition to the usual metric tensor \(g_{\mu \nu }\). The scalar field can be coupled to gravity in an arbitrary way, minimally or non-minimally. It is well-known that \(f(R,\phi )\) gravity can be formulated either in Jordan or Einstein frames which are related to each other by a conformal transformation. In Einstein frame, \(f(R,\phi )\) gravity is equivalent to a scalar tensor theory which contains two scalar fields.Footnote 2 This theory has a number of observational and experimental footprints. For a detailed discussion see [11] and references therein. The cosmological dynamics of dark energy models based on \(f(R,\phi )\) gravity leads to some conditions on the form of the potential ( of \(\phi \)) to realize a sequence of radiation, matter, and accelerated epochs [15,16,17]. This theory can give the correct amount of inflation in agreement with the cosmological data while provides at the same time a graceful exit from the inflationary epoch [18, 19]. Also the problem of rotation curves of spiral galaxies can be addressed in \(f(R,\phi )\) gravity [20]. Many studies of primordial perturbations in this theory was firstly formulated by Hwang and his collaborators [21,22,23,24] and then discussed in [25]. Hwang et al. have applied the conformal equivalence of generalized gravity theories with general relativity minimally coupled to a scalar field, in the absence of ordinary matter. Then they have studied the evolution of the curvature perturbation considering the second order action.

Here we shall use a straightforward calculations based on using some appropriate gauges and after obtaining the generalized Mukhanov–Sasaki equation, we shall find its initial condition. The organization of this paper is as follows: In the next section, we will consider the scalar metric perturbations and also the scalar field fluctuations. After deriving the equations governing the dynamics of these perturbations, these equations are simplified imposing the Newtonian gauge condition. Then we have paid special attention to these equations at sufficiently early times, when \(q/a\gg H\), and used WKB approximation. This specifies solutions to an unknown coefficient. To find it, we quantize the scalar field perturbations and demand that the creation and annihilation operators satisfy the standard commutation relations. In Sect. 3, by introducing a special gauge, the generalized Mokhanov-Sasaki equation is obtained. The initial condition of this equation is given using the results of the previous section. In the Sect. 4, we have followed a similar procedure to find the initial conditions for tensor metric perturbation. As an example a new inflationary solution for a power-law modified gravity \(f(R,\phi )= f_0 R^m \phi ^n\) with a power-law potential is found in Sect. 3 and then using the obtained solution as the background metric and scalar field, the evolution of scalar and tensor perturbations is discussed in Sects. 3 and 4. We have found the appropriate range of the free parameters of the solution to cover the observational constraints.

2 Scalar perturbations in generalized scalar-tensor gravity

Consider inflation driven by a single scalar field which is coupled non-minimally with gravity and represented by the following action in the Jordan frame:

$$\begin{aligned} S=\int \sqrt{-g}\left( \frac{1}{2 \kappa ^2}f(R,\phi )-\frac{1}{2}\omega (\phi )g^{\alpha \beta }\partial _\alpha \phi \partial _\beta \phi -V(\phi )\right) ~d^4x.\nonumber \\ \end{aligned}$$
(1)

where \(\kappa ^2=8\pi G\).Footnote 3 For a spatially flat FRW universe described by the metric: \(ds^2 = -dt^2+a^2(t)\vec {dx}^2\), varying above action with respect to the metric and scalar field yields:

$$\begin{aligned}&3FH^2=\frac{1}{2}(RF-f)-3H{\dot{F}}+\kappa ^2\left[\frac{1}{2}\omega {{\dot{\phi }}}^2+V(\phi )\right]\end{aligned}$$
(2)
$$\begin{aligned}&-\,2F{\dot{H}}={\ddot{F}}-H{\dot{F}}+\kappa ^2\omega {{\dot{\phi }}}^2 \end{aligned}$$
(3)
$$\begin{aligned}&{\ddot{\phi }}+3H{{\dot{\phi }}}+\frac{1}{2\omega }\left( \omega _{,\phi }{{\dot{\phi }}}^2+2V_{,\phi }-\frac{f_{,\phi }}{\kappa ^2}\right) =0 \end{aligned}$$
(4)

where \(F(R,\phi )\equiv \frac{\partial f}{\partial R}\), H is the Hubble parameter, \(R=6(2H^2+{\dot{H}})\) is the Ricci scalar and dot denotes the time derivative. Putting \(f=R\), we see that, the above equations reduce to the equations of Einstein gravity coupled minimally to a scalar field.

Perturbing the scalar field as \(\phi =\phi _0+\delta \phi \) and take the line element of perturbed FRW universe as:

$$\begin{aligned} ds^2&=-(1+2\Phi )dt^2-2a(t) \partial _i\beta dtdx^i+a^2(t)(\delta _{ij}\nonumber \\&-\,2\Psi \delta _{ij}+2\partial _i\partial _j\gamma +{\mathcal {D}}_{ij})dx^idx^j. \end{aligned}$$
(5)

where \(\Phi , \beta , \Psi \) and \(\gamma \) are scalar perturbations and \({\mathcal {D}}_{ij}\) is tensor perturbation which is the traceless, divergenceless and symmetric. Under these conditions, we can see that the corresponding Fourier modes, \({\mathcal {D}}_{ij}(q,t)\), have two independent components and can be expressed in terms of the polarization tensors, \({\hat{e}}^+_{ij}\) and \({\hat{e}}^\times _{ij}\), as following

$$\begin{aligned} {\mathcal {D}}_{ij}(q,t)&={\hat{e}}^+_{ij}{\mathcal {D}}_+(q,t)+{\hat{e}}^\times _{ij} {\mathcal {D}}_\times (q,t) \nonumber \\ e_{xx}^+&=-e_{yy}^+=1,e_{xy}^\times =e_{yx}^\times =1 \end{aligned}$$
(6)

where the momentum q is considered along the z-axis. We will come back to the above relation in Sect. 4.

Now, perturbing the background field Eqs. (2)–(4) at linear order for scalar perturbations gives [11]:

$$\begin{aligned}&-\frac{\Delta }{a^2}\Psi +H A =-\frac{1}{2F}\left[ \left( 3H^2+3{\dot{H}}+\frac{\Delta }{a^2}\right) \delta F -3H\dot{\delta F}\right. \nonumber \\&\quad \left. +\frac{1}{2}\left( \kappa ^2\omega _{,\phi }{{{\dot{\phi }}}_0}^2+2\kappa ^2V_{,\phi }-f_{,\phi }\right) \delta \phi +\kappa ^2\omega {{\dot{\phi }}}_0{\dot{\delta \phi }}\right. \nonumber \\&\quad + \left. \left( 3H{\dot{F}}-\kappa ^2\omega {{\dot{\phi }}}_0^2\right) \Phi +{\dot{F}} A\right] \end{aligned}$$
(7)
$$\begin{aligned}&H\Phi +{{\dot{\Psi }}}=\frac{1}{2F}\left[\kappa ^2\omega {{\dot{\phi }}}_0\delta \phi +\dot{\delta F}-H\delta F-{\dot{F}} \Phi \right]\end{aligned}$$
(8)
$$\begin{aligned}&{{\dot{\chi }}}+H\chi -\Phi +\Psi =\frac{1}{F}\left( \delta F-{\dot{F}}\chi \right) \end{aligned}$$
(9)
$$\begin{aligned}&\dot{A}+ 2H A+\left( 3{\dot{H}}+\frac{\Delta }{a^2}\right) \Phi =\frac{1}{2F}\left[ 3\ddot{\delta { F}}+3H\dot{\delta {F}}\right. \nonumber \\&\quad \left. -\left( 6H^2+\frac{\Delta }{a^2}\right) \delta F+4\kappa ^2\omega {{\dot{\phi }}}_0\dot{\delta {\phi }}\right. \nonumber \\&\quad \left. +(2\kappa ^2\omega _{,\phi }{{\dot{\phi }}}_0^2-2\kappa ^2V_{,\phi }+f_{,\phi })\delta \phi -3{\dot{F}}{{\dot{\Phi }}}\right. \nonumber \\&\quad \left. -{\dot{F}} A-(4\kappa ^2 \omega {{\dot{\phi }}} _0^2+3H{\dot{F}}+6{\ddot{F}})\Phi \right] \end{aligned}$$
(10)
$$\begin{aligned}&\ddot{\delta {F}}+3H\dot{\delta {F}}-\left( \frac{\Delta }{a^2}+\frac{R}{3}\right) \delta F+\frac{2}{3}\kappa ^2\omega \dot{\phi _0}\dot{\delta {\phi }}\nonumber \\&\quad +\frac{1}{3}(\kappa ^2\omega _{,\phi }\dot{\phi _0^2}-4\kappa ^2 V_{,\phi }+2f_{,\phi })\delta \phi ={\dot{F}}(A+{{\dot{\Phi }}})\nonumber \\&\quad +\left( 2{\ddot{F}}+3H{\dot{F}}+\frac{2}{3}\kappa ^2\omega \dot{\phi _0^2}\right) \Phi -\frac{1}{3}F\delta R \end{aligned}$$
(11)
$$\begin{aligned}&\ddot{\delta \phi }+\left( 3H+\frac{\omega _{,\phi }}{\omega }{{\dot{\phi }}}_0\right) \delta {{\dot{\phi }}}\nonumber \\&\quad +\left[-\frac{\Delta }{a^2}+\left( \frac{\omega _{,\phi }}{\omega }\right) _{,\phi }\frac{\dot{\phi _0^2}}{2}+\left( \frac{2V_{,\phi }-\frac{f_{,\phi }}{\kappa ^2}}{2\omega }\right) _{,\phi }\right]\delta \phi \nonumber \\&={{\dot{\phi }}}_0{{\dot{\Phi }}}+\left( 2{\ddot{\phi }}_0+3H{{\dot{\phi }}}_0+\frac{\omega _{,\phi }}{\omega }\dot{\phi _0^2}\right) \Phi \nonumber \\&\quad +{{\dot{\phi }}}_0 A+\frac{1}{2\omega \kappa ^2}F_{,\phi }\delta R \end{aligned}$$
(12)

where

$$\begin{aligned} \chi \equiv a(\beta +a{{\dot{\gamma }}})\quad A\equiv 3(H\Phi +{{\dot{\Psi }}})-\frac{\Delta }{a^2}\chi \end{aligned}$$
(13)

\(\Delta \) is the Laplacian operator and the Ricci perturbation is given by:

$$\begin{aligned} \delta R=-2\left[\dot{A}+4H A+\left( \frac{\Delta }{a^2}+3{\dot{H}}\right) \Phi -2\frac{\Delta }{a^2}\Psi \right]. \end{aligned}$$
(14)

Working in Fourier space significantly simplifies calculations. Thus we write the perturbation equations in Fourier space and choose the Newtonian gauge. It is a particularly simple gauge to use for the scalar mode of metric perturbations and does not leave a residual gauge symmetry. In this gauge, \(\gamma =\beta =0\) and thus (13) yields:

$$\begin{aligned} \chi = 0 A=3(H\Phi +{{\dot{\Psi }}}). \end{aligned}$$
(15)

Denoting the perturbations of metric and the scalar field by their corresponding Fourier transforms:

$$\begin{aligned} \Phi ( x,t)&=\int {\frac{d^3q}{(2\pi )^{3/2}}\Phi _q(t)e^{-i\mathbf{q }.\mathbf x }}. \end{aligned}$$
(16)
$$\begin{aligned} \Psi (\mathbf x ,t)&=\int {\frac{d^3q}{(2\pi )^{3/2}}\Psi _q(t)e^{-i\mathbf{q }.\mathbf x }}. \end{aligned}$$
(17)
$$\begin{aligned} \delta \phi (\mathbf x ,t)&=\int {\frac{d^3q}{(2\pi )^{3/2}}\delta \phi _q(t)e^{-i\mathbf q .\mathbf{x }}}. \end{aligned}$$
(18)

Inserting these relation and (15) into Eqs. (7)–(12), leads to the following equations for the Fourier modes:

$$\begin{aligned}&\left[3\left( H^2-{\dot{H}}+3H\frac{{\dot{F}}}{F}\right) +\frac{q^2}{a^2}-\frac{\kappa ^2\omega {{\dot{\phi }}}_0^2}{F}\right]\Phi _q\nonumber \\&\quad +\left[3\left( H^2+{\dot{H}}-H\frac{{\dot{F}}}{F}\right) +\frac{q^2}{a^2}\right]\Psi _q\nonumber \\&\quad +3\left( H+\frac{{\dot{F}}}{F}\right) {{\dot{\Psi }}}_q+3H{{\dot{\Phi }}}_q =-\frac{\kappa ^2\omega {{\dot{\phi }}}_0}{F}\delta {{\dot{\phi }}}_q\nonumber \\&\quad -\frac{\kappa ^2}{2F}\left( \omega _{,\phi }{{\dot{\phi }}}_0^2+2V_{,\phi }-\frac{f_{,\phi }}{\kappa ^2}\right) \delta \phi _q, \end{aligned}$$
(19)
$$\begin{aligned}&\quad \left( H+2\frac{{\dot{F}}}{F}\right) \Phi _q+{{\dot{\Phi }}}_q+\left( H-\frac{{\dot{F}}}{F}\right) \Psi _q\nonumber \\&\quad +{{\dot{\Psi }}}_q-\frac{\kappa ^2\omega {{\dot{\phi }}}_0}{F}\delta \phi _q=0 \end{aligned}$$
(20)
$$\begin{aligned} \begin{aligned}&{\ddot{\Psi }}_q+{\ddot{\Phi }}_q-\left( \frac{{\dot{F}}}{F}-3H\right) {{\dot{\Psi }}}_q+3\left( \frac{{\dot{F}}}{F}+H\right) {{\dot{\Phi }}}_q\\&\quad -\left[ \frac{{\ddot{F}}}{F}+H\frac{{\dot{F}}}{F}-\frac{1}{3}\left( 6H^2-\frac{q^2}{a^2}\right) \right] \Psi _q\\&\quad +\left[ 2\left( 2{\dot{H}}-H^2+H\frac{{\dot{F}}}{F}\right) -\frac{q^2}{3a^2}\right. \\&\quad \left. +\frac{1}{3F}(4\kappa ^2 \omega {{\dot{\phi }}} _0^2+3H{\dot{F}}+6{\ddot{F}})\right] \Phi _q\\&=\frac{8\kappa ^2\omega {{\dot{\phi }}}_0}{3}\dot{\delta \phi _q}+\frac{2}{3}(2\kappa ^2\omega _{,\phi }{{\dot{\phi }}}_0^2-2\kappa ^2V_{,\phi }+f_{,\phi })\delta \phi _q \end{aligned} \end{aligned}$$
(21)
$$\begin{aligned} \begin{aligned}&{\ddot{\Psi }}_q+{\ddot{\Phi }}_q+\left( \frac{{\dot{F}}}{F}+5H\right) {{\dot{\Psi }}}_q+\left( 3\frac{{\dot{F}}}{F}+5H\right) {{\dot{\Phi }}}_q\\&\quad +\left( \frac{q^2}{3a^2}+\frac{R}{3}-3H\frac{{\dot{F}}}{F}-\frac{{\ddot{F}}}{F}\right) \Psi _q\\&\quad +\left[ \frac{q^2}{3a^2}-\frac{R}{3}+3\frac{{\ddot{F}}}{F}+8H^2+4{\dot{H}}+9H\frac{{\dot{F}}}{F}+\frac{2}{3F}\kappa ^2\omega {{\dot{\phi }}}_0^2\right] \Phi _q\\&=\frac{2}{3}\kappa ^2\omega {{\dot{\phi }}}_0\dot{\delta \phi }+\frac{1}{3}(\kappa ^2\omega _{,\phi }\dot{\phi _0^2}-4\kappa ^2 V_{,\phi }+2f_{,\phi })\delta \phi \end{aligned} \end{aligned}$$
(22)
$$\begin{aligned}&\delta {{\ddot{\phi }}}_q+\left( 3H+\frac{\omega _{,\phi }}{\omega }{{\dot{\phi }}}_0\right) \delta {{\dot{\phi }}}_q\nonumber \\&\quad +\left[\frac{q^2}{a^2}+(\frac{\omega _{,\phi }}{\omega })_{,\phi }\frac{\dot{\phi _0^2}}{2}+(\frac{2V_{,\phi }-\frac{f_{,\phi }}{\kappa ^2}}{2\omega })_{,\phi }\right]\delta \phi _q\nonumber \\&=\left[2{\ddot{\phi }}_0+6H{{\dot{\phi }}}_0+\frac{\omega _{,\phi }}{\omega }\dot{\phi _0^2}-\frac{F_{,\phi }}{\omega \kappa ^2}\left( 6{\dot{H}}+12H^2-\frac{q^2}{a^2}\right) \right]\Phi _q\nonumber \\&\quad +\left( {{\dot{\phi }}}_0-3H\frac{F_{,\phi }}{\kappa ^2\omega }\right) {{\dot{\Phi }}}_q\\&\quad +3\left( {{\dot{\phi }}}_0-4H\frac{F_{,\phi }}{\kappa ^2\omega }\right) {{\dot{\Psi }}}_q-2\frac{q^2F_{,\phi }}{a^2\omega \kappa ^2}\Psi _q-3\frac{F_{,\phi }}{\omega \kappa ^2}{\ddot{\Psi }}_q\nonumber \end{aligned}$$
(23)

where in Eqs. (19)–(23), \(\delta F\) and \(\delta R\) are eliminated by using the following relations

$$\begin{aligned}&{-}\Phi _{q}+\Psi _{q}=\frac{\delta F}{F} \end{aligned}$$
(24)
$$\begin{aligned}&\quad \delta R=-2\left[ \left( 12H^2+6{\dot{H}}-\frac{q^2}{a^2}\right) \Phi _{q}\right. \nonumber \\&\quad \quad \left. +3H{{\dot{\Phi }}}_{q}+2\frac{q^2}{a^2}\Psi _q+3{\ddot{\Psi }}_{q}+12H{{\dot{\Psi }}}_{q}\right] . \end{aligned}$$
(25)

Equations (21), (22) and (23) are the dynamical equations of motion for \(\Phi , \Psi \) and \(\delta \phi \) and two remaining Eqs. (19) and (20), are two constraints imposed on the fluctuations. This is similar to general relativity in which there are two dynamical equations for the metric and scalar fluctuations together with a constraint equation in the Newtonian gauge.Footnote 4

Now the above perturbation equations, can be used to find the functions \(\Psi _q\), \(\Phi _q\), \(\delta \phi _q\) at sufficiently early times by WKB approximation. At these very early times, all perturbations are inside the horizon and the perturbation modes oscillate more quickly than the expansion rate of the universe, i.e. \(q/a\gg H\). The general solutions in the WKB approximation reads:

$$\begin{aligned} \begin{aligned}&\Psi _q(t)\longrightarrow g(t)\exp \left( -iq\int _{t^*}^{t}\frac{dt'}{a(t')}\right) \\&\Phi _q(t)\longrightarrow h(t)\exp \left( -iq\int _{t^*}^{t}\frac{dt'}{a(t')}\right) \\&\delta \phi _q(t)\longrightarrow y(t)\exp \left( -iq\int _{t^*}^{t}\frac{dt'}{a(t')}\right) \end{aligned} \end{aligned}$$
(26)

where the rate of change of g(t), h(t) and y(t) are much smaller than q / a and \(t^*\) is an arbitrary time. Also since the fluctuation of fields are real functions, the complex conjugate of any above solution, is another independent solution. Substituting (26) into (19)–(23) and working up to leading order in q / a, one quickly finds that the terms in (23) of second order in q / a cancel each other. The other equations are satisfied if:

$$\begin{aligned} \frac{g+h}{y}=i\frac{a\kappa ^2\omega {{\dot{\phi }}}_0}{q F} \end{aligned}$$
(27)

This equation shows that at sufficiently early times, the metric fluctuations are one order smaller than the scalar fluctuations and thus they will be insignificant. This is a useful result where we see that it holds not only for the scalar field minimally coupled to gravity in general relativity but also in \(f(R,\phi )\) theory. To first order in q / a, Eq. (23) leads:

$$\begin{aligned} {\dot{y}}+\left( H+\frac{\omega _{,\phi }}{2\omega }{{\dot{\phi }}}_0\right) y=i\frac{qF_{,\phi }}{2a\omega \kappa ^2}(g+h) \end{aligned}$$
(28)

This equation gives the time dependence of scalar field perturbations. Setting \(F=\omega =1\) and using (23), reduces the Eqs. (27) and (28) to:

$$\begin{aligned} g=h \frac{g}{y}=i\frac{a\kappa ^2{{\dot{\phi }}}_0}{2q} \end{aligned}$$
(29)

which are agree with those are found in GR [1].

From (27) and (28) we see immediately that

$$\begin{aligned} {\dot{y}}+Hy+\frac{{{\dot{\phi }}}_0}{2}\left( \frac{\omega _{,\phi }}{\omega }+\frac{F_{,\phi }}{F}\right) y=0 \end{aligned}$$
(30)

which is explicitly independent of the functional form of the scalar field potential but is implicitly dependent on it through the equations of motion (2)–(4). Solving (30), one finds:

$$\begin{aligned} y(t)=\frac{C}{a\sqrt{\omega }}\exp \left( -\int ^t{\dot{\phi }}\frac{F_{,\phi }}{2F}dt'\right) \end{aligned}$$
(31)

where C is an integration constant and needs to be determined.

As explained above, we are dealing with the very early universe in which the fluctuations of inflaton field have quantum nature . According to (27), ignoring the metric perturbation, we quantize the scalar field in the unperturbed FRW background such as general relativity. Expanding the scalar field perturbation in terms of two independent solutions, we have:

$$\begin{aligned}&\delta \phi (\mathbf{x },t) =\int \frac{Cd^3q}{a\sqrt{(2\pi )^{3}\omega }}\exp \left( -\int ^t{\dot{\phi }}\frac{F_{,\phi }}{2F}dt'\right) \nonumber \\&\quad \times \,\left[ \alpha (\mathbf{q })e^{i\mathbf{q }}.\mathbf x \exp \left( -iq\int ^t\frac{dt'}{a(t')}\right) \right. \nonumber \\&\quad \left. +\,\alpha ^*(\mathbf{q })e^{i\mathbf{q }}.\mathbf x \exp \left( iq\int ^t\frac{dt'}{a(t')}\right) \right] \end{aligned}$$
(32)

where \(\alpha \) and \(\alpha ^*\) are normalized annihilation and creation operators. One has to imply these operators obey the standard commutation relations:

$$\begin{aligned} \left[\alpha (\mathbf{q }),\alpha (\mathbf q ')\right]=0,\quad \left[\alpha (\mathbf q ),\alpha ^*(\mathbf{q }')\right]=\delta ^3(\mathbf{q }-\mathbf{q }') \end{aligned}$$
(33)

Now using (32) and (33) one can derive the following equal time commutation relations:

$$\begin{aligned}&\left[\delta \phi (\mathbf{x },t),\delta {\dot{\phi }}(\mathbf y ,t)\right]\nonumber \\&\quad =i\frac{2q(2\pi )^3C^2\exp \left( -\int ^t{\dot{\phi }}\frac{F_{,\phi }}{F}dt'\right) }{a^3\omega }\delta ^3(\mathbf x -\mathbf{y }) \end{aligned}$$
(34)

which fixes the constant C. It must be such that for \(F=\omega =1\), we get the correct commutation relation of general relativity coupled minimally to a scalar field. Thus \(C=\frac{1}{(2\pi )^{3/2}\sqrt{2q}}\). This means that the initial conditions of metric and scalar field fluctuations, for \(a\longrightarrow 0\), are:

$$\begin{aligned}&\Psi _q(t)+\Phi _q(t)\longrightarrow \frac{i\kappa ^2\dot{\phi _0}}{(2\pi )^{3/2}}\sqrt{\frac{\omega }{2q^3}}\exp \nonumber \\&\quad \times \left( -\int ^t{\dot{\phi }}\frac{3F_{,\phi }}{2F}dt'\right) \exp \left( -iq\int ^{t}\frac{dt'}{a(t')}\right) \end{aligned}$$
(35)
$$\begin{aligned}&\delta \phi _q(t)\longrightarrow \frac{1}{(2\pi )^{3/2}a\sqrt{2q\omega }}\exp \nonumber \\&\quad \times \left( -\int ^t{\dot{\phi }}\frac{F_{,\phi }}{2F}dt'\right) \exp \left( -iq\int ^{t}\frac{dt'}{a(t')}\right) \end{aligned}$$
(36)

Again putting \(F=\omega =1\) and \(\Psi _q(t)=\Phi _q(t)\), we recover the familiar initial conditions of the fluctuations in general relativity coupled to a scalar field. The above initial conditions are obtained using the Newtonian gauge (15). We note here that the Eqs. (20)–(23) are derived using (24) which along with the Eq. (9) ensure that \(\chi =0\). Therefore the Newtonian gauge condition remains valid in all times.

3 Mukhanov–Sasaki equation

In \(f(R,\phi )\) gravity due to having more additional degrees of freedom, F and \(\phi \), there exists a number of gauge invariant quantities [11]. It is a simple task to show that the following quantities are invariant under an infinitesimal coordinate transformation, \(x^\mu \rightarrow x^\mu + \delta x^\mu \):Footnote 5

$$\begin{aligned} {\mathcal {R}}= & {} \Psi +\frac{H}{\rho +P}\delta q,\nonumber \\ {\mathcal {R}}_{\delta \phi }= & {} \Psi -\frac{H}{{{\dot{\phi }}}}\delta \phi ,\nonumber \\ {\mathcal {R}}_{\delta F}= & {} \Psi -\frac{H}{{\dot{F}}}\delta F \end{aligned}$$
(37)

where \(\delta q = (\rho + P )v\). Also \(\rho \), P are the energy density and pressure of the matter field and v characterizes the velocity potential. \({\mathcal {R}}\) is the standard curvature perturbation defined in general relativity. \({\mathcal {R}}_{\delta F}\) arises due to the arbitrary form of F and \({\mathcal {R}}_{\delta \phi }\) comes from the scalar field. To have a better description of the above quantities, let us point out that in the absence of matter fields, \({\mathcal {R}}\) which is the comoving curvature perturbation on the uniform-field hypersurface, is equal to \(\Psi \). For a single field with a potential \(V(\phi )\), using \(\rho ={{\dot{\phi }}} ^2/2 + V(\phi )\) and \(P={{\dot{\phi }}}^2/2 - V(\phi )\), we have \(\delta q = {{\dot{\phi }}} \delta \phi \) and thus \({\mathcal {R}}_{\delta \phi }\) is identical to \({\mathcal {R}}\). Also since by a conformal transformation together with introducing a new scalar field , one can bring the action (1) into the Einstein form, thus \({\mathcal {R}}_{\delta F}\) is identical to \({\mathcal {R}}_{\delta \phi }\) (See the footnote of page 3 for further details).

To study the scalar perturbations generated during inflation, it is useful to derive the evolution equation for the curvature perturbation which is known as the Mukhanov–Sasaki equation. This equation can be easily derived by an appropriate choice of gauge. As mentioned above, in \(f(R,\phi )\) gravity, there exists much more gauge invariant quantities in comparison with general relativity. This has led to more different choices for an appropriate gauge. A convenient gauge is spatially-flat (or uniform-curvature) gauge fixed by the conditions \(\Psi =\gamma =0\). Taking this as the gauge condition, the authors of [21], derive the Mukhanov–Sasaki equation for \({\mathcal {R}}_{\delta \phi }\) in the Einstein frame. Then the resulted equation can be mapped back into the first frame (Jordan frame) by the inverse conformal transformation mentioned above. In [11], the authors focus on the coupling of the form of \(f(\phi )R\) and choose the gauge \(\delta \phi =\delta F=0\) in which \({\mathcal {R}}={\mathcal {R}}_{\delta \phi }={\mathcal {R}}_{\delta F}\). They have pointed out that this gauge can not be applied for a general \(f(R,\phi )\) gravity including the non linear terms in curvature.Footnote 6

Here we concentrate on a general \(f(R,\phi )\) theory. Looking at (37), we choose a different gauge defined by:Footnote 7

$$\begin{aligned} \frac{\delta F}{{\dot{F}}}=\frac{\delta \phi }{\dot{\phi _0}}, \quad \Psi =0 \end{aligned}$$
(38)

The main advantage of this gauge is that \({\mathcal {R}}=0\) and there is a scalar curvature defied as \(\tilde{{\mathcal {R}}}\equiv {\mathcal {R}}_{\delta F}={\mathcal {R}}_{\delta \phi }=-\frac{H}{{{\dot{\phi }}}}\delta \phi \) and also simplifies the perturbations Eqs. (7), (8), (9) and (12) as following:

$$\begin{aligned}&\left( 3H^2+3\frac{H{\dot{F}}}{F}-\frac{\kappa ^2\omega \phi ^2_0}{2F}\right) \Phi -\left( H+\frac{{\dot{F}}}{2F}\right) \frac{\Delta }{a^2}\chi \nonumber \\&\quad =-\frac{1}{2F}\left\{ \left( \kappa ^2\omega {{\dot{\phi }}}_0-3\frac{H{\dot{F}}}{{{\dot{\phi }}}_0}\right) \dot{\delta \phi }\right. \nonumber \\&\quad \quad \left. +\left[ \frac{{\dot{F}}}{{{\dot{\phi }}}_0}\left( 3H^2+3{\dot{H}}+\frac{\Delta }{a^2}\right) -3H\left( \frac{{\dot{F}}}{{{\dot{\phi }}}_0}\right) ^.\right. \right. \nonumber \\&\quad \quad \left. \left. +\frac{1}{2}\left( \kappa ^2\omega _{,\phi }{{\dot{\phi }}}_0^2+\kappa ^2V_{,\phi }-\frac{f_{,\phi }}{2}\right) \right] \delta \phi \right\} \end{aligned}$$
(39)
$$\begin{aligned}&\left[ \frac{\kappa ^2\omega {{\dot{\phi }}}_0}{2F}-\frac{H{\dot{F}}}{2F{{\dot{\phi }}}_0}+\frac{1}{2F}\left( \frac{{\dot{F}}}{\dot{\phi _0}}\right) ^.\right] \delta \phi \nonumber \\&\quad \quad +\frac{{\dot{F}}}{2F\dot{\phi _0}}\dot{\delta \phi }=\left( H+\frac{{\dot{F}}}{2F}\right) \Phi \end{aligned}$$
(40)
$$\begin{aligned}&\quad \quad \left( H+\frac{{\dot{F}}}{F}\right) \chi +{\dot{\chi }}=\frac{{\dot{F}}}{F\dot{\phi _0}}\delta \phi +\Phi \end{aligned}$$
(41)
$$\begin{aligned}&\ddot{\delta \phi }+\left( 3H+\frac{\omega _{,\phi }}{\omega }{{\dot{\phi }}}_0\right) \dot{\delta \phi }\nonumber \\&\quad \quad +\left[ -\frac{\Delta }{a^2}+\left( \frac{\omega _{,\phi }}{\omega }\right) _{,\phi }\frac{\dot{\phi ^2}}{2}+\left( \frac{2V_{,\phi }-f_{,\phi }/\kappa ^2}{2\omega }\right) _{,\phi }\right] \delta \phi \nonumber \\&\quad =\dot{\phi _0}{{\dot{\Phi }}}+\left( 2\ddot{\phi _0}+6H{{\dot{\phi }}}_0+\frac{\omega _{,\phi }}{\omega }\phi _0^2\right) \Phi \nonumber \\&\quad \quad -\dot{\phi _0}\frac{\Delta }{a^2}\chi +\frac{F_{,\phi }}{2\omega \kappa ^2}\delta R \end{aligned}$$
(42)

By using these equations and also relations (13) and (14), we could derive two following equations in terms of \(\delta \phi \) and \(\Phi \):

$$\begin{aligned}&\ddot{\delta \phi }+\left[ 3H+\frac{\omega _{,\phi }}{\omega }\dot{\phi _0}\right. \nonumber \\&\quad \left. +\frac{\dot{\phi _0}+\frac{F_{,\phi }}{\kappa ^2\omega }\left( \frac{{\dot{F}}}{F}-H\right) }{2FH+{\dot{F}}}\left( \kappa ^2\omega \dot{\phi _0}-3\frac{{\dot{F}}H}{{{\dot{\phi }}}_0}\right) \right] \dot{\delta \phi }\nonumber \\&+\left\{ -\frac{\Delta }{a^2}+\frac{{{\dot{\phi }}}_0+\frac{F_{,\phi }}{\kappa ^2\omega }\left( \frac{{\dot{F}}}{F}-H\right) }{2FH+{\dot{F}}}\right. \nonumber \\&\quad \left[ \frac{{\dot{F}}}{{{\dot{\phi }}}}\left( 3H^2+3{\dot{H}}+\frac{\Delta }{a^2}\right) \right. \nonumber \\&\quad \left. -3H\left( \frac{{\dot{F}}}{\dot{\phi _0}}\right) ^.-\kappa ^2\omega (\ddot{\phi _0}+3H\dot{\phi _0})\right] \nonumber \\&\left. +\left( \frac{\omega _{,\phi }}{\omega }\right) _{,\phi }\frac{{{\dot{\phi }}}^2}{2}+\left( \frac{2V_{,\phi }-f_{,\phi }}{2\omega }\right) _{,\phi }\right\} \delta \phi \nonumber \\&\quad =\left( \dot{\phi _0}-3\frac{F_{,\phi }}{\kappa ^2\omega }H\right) {{\dot{\Phi }}}+\left[ \frac{\omega _{,\phi }}{\omega }{{\dot{\phi }}}_0^2 +2{\ddot{\phi }}_0+6H\dot{\phi _0}\right. \nonumber \\&-\frac{F_{,\phi }}{\omega \kappa ^2}(6{\dot{H}}+12H^2)-2F\frac{\dot{\phi _0}+\frac{F_{,\phi }}{\kappa ^2\omega }\left( \frac{{\dot{F}}}{F}-H\right) }{2FH+{\dot{F}}}\nonumber \\&\quad \left. \left( 3H^2 +3H\frac{{\dot{F}}}{F}-\frac{\kappa ^2\omega {{\dot{\phi }}}_0^2}{2F}\right) \right] \Phi \end{aligned}$$
(43)
$$\begin{aligned}&\Phi =\frac{1}{2FH+{\dot{F}}}\left( \kappa ^2\omega {{\dot{\phi }}}_0-\frac{H{\dot{F}}}{\dot{\phi _0}}+\left( \frac{{\dot{F}}}{\dot{\phi _0}}\right) ^.\right) \delta \phi \nonumber \\&\quad +\frac{{\dot{F}}}{\dot{\phi _0}(2FH+{\dot{F}})}\dot{\delta \phi } \end{aligned}$$
(44)

and then after some lengthy and tedious calculations we obtain the Mukhanov–Sasaki equation [21]:

$$\begin{aligned}&\ddot{\delta \phi }+\left\{ 3H+\frac{(1+{\dot{F}}/2FH)^2}{\omega +3{\dot{F}}^2/2F\kappa ^2{{\dot{\phi }}}_0^2}\left[ \frac{\left( \omega +\frac{3{\dot{F}}^2}{2F\dot{\phi _0^2}\kappa ^2}\right) }{\left( 1+\frac{{\dot{F}}}{2HF}\right) ^2}\right] ^.\right\} \dot{\delta \phi }\nonumber \\&\quad -\left\{ \frac{\Delta }{a^2}+\frac{H}{a^3\dot{\phi _0}}\frac{(1+{\dot{F}}/2FH)^2}{\omega +3{\dot{F}}^2/2F\kappa ^2{{\dot{\phi }}}_0^2}\right. \nonumber \\&\qquad \left. \left[ \frac{\left( \omega +\frac{3{\dot{F}}^2}{2F\dot{\phi _0^2}\kappa ^2}\right) }{\left( 1+\frac{{\dot{F}}}{2HF}\right) ^2}a^3\left( \frac{\dot{\phi _0}}{H}\right) ^.\right] ^.\right\} \delta \phi =0 \end{aligned}$$
(45)

Changing variable to \(\delta \phi =-\frac{\dot{\phi _0}}{H}\tilde{{\mathcal {R}}}\) allows us to rewrite the above equation as:

$$\begin{aligned} \frac{\left( H+\frac{{\dot{F}}}{2F}\right) ^2}{a^3\left( \omega {{\dot{\phi }}}_0^2+\frac{3{\dot{F}}^2}{2F\kappa ^2}\right) }\left[ \frac{a^3\left( \omega {{\dot{\phi }}}_0^2+\frac{3{\dot{F}}^2}{2F\kappa ^2}\right) }{\left( H+\frac{{\dot{F}}}{2F}\right) ^2}\dot{\tilde{{\mathcal {R}}}}\right] ^.-\frac{1}{a^2}\Delta \tilde{\mathcal R}=0 \end{aligned}$$
(46)

Now in order to obtain the initial condition of this equation, we return to the Newtonian gauge. Substituting (35) and (36) into (37) and noting that according to (27), at the beginning of inflation era, only the scalar field fluctuations contributes in \(\tilde{{\mathcal {R}}}\). Therefore we find that:

$$\begin{aligned}&\tilde{\mathcal {R}}_q\rightarrow -\frac{H}{(2\pi )^{3/2}a\sqrt{2q\omega }{{\dot{\phi }}}_0}\exp \left( -\int ^t{\dot{\phi }}\frac{F_{,\phi }}{2F}dt'\right) \nonumber \\&\quad \exp \left( -iq\int ^{t}\frac{dt'}{a(t')}\right) \end{aligned}$$
(47)

where \(\tilde{{\mathcal {R}}}_q\) is the Fourier transform of \(\tilde{{\mathcal {R}}}\). Beyond the horizon where \(q/a\ll H\), Eq. (46) has two growing and decaying perturbation modes [21]. The growing mode is a non-zero constant which is exactly the same as obtained in general relativity.

For any given function \(f(R,\phi )\), one can solve Eq. (46) with the initial condition (47) which shows that, the curvature perturbation at sufficiently very early times, is independent of the behaviour of the scalar potential. This means that the constant curvature perturbation, outside the horizon, depends only to the scalar potential evaluated when the perturbation mode leaves the horizon. This feature is also similar to what is obtained in general relativity [1].

As an example, we consider an inflationary model \(f(R,\phi )= f_0 R^m\phi ^n\) with the potential of the form \(V(\phi )=V_0\phi ^p\) and also \(\omega (\phi )=1\). The Eqs. (2)–(4) have the following solution

$$\begin{aligned} \phi =\phi _0 t^l a=t^\lambda \end{aligned}$$
(48)

with \(1/\lambda =-{\dot{H}}/H^2<1\) and the constants \(f_0, \phi _0, m, n, l\) are given by:

$$\begin{aligned} f_0= & {} \frac{\kappa ^2\phi _0^{2-n}(1+2V_0\phi _0^{-2/l})\left( \frac{1-m}{1-n/2}\right) ^2}{\left[ 6\lambda \left( 2\lambda -1\right) \right] ^m\left[ 1-m+\frac{m}{2\lambda -1}\left( \lambda +2\frac{1-m}{1-n/2}\right) \right] } \end{aligned}$$
(49)
$$\begin{aligned} l= & {} \frac{1-m}{1-n/2} \end{aligned}$$
(50)
$$\begin{aligned} 1= & {} \frac{m}{2\lambda -1}\left( \frac{4\lambda }{3}-\frac{5l}{3}-\frac{l(2l-1)}{3\lambda }-1\right) \end{aligned}$$
(51)
$$\begin{aligned}&\left[ 1-m+\frac{m\lambda }{2\lambda -1}+\frac{2ml}{2\lambda -1}\right] \left[ l(l-1)+3\lambda \right] \nonumber \\&\quad =\frac{nl^2}{2}-2 V_0 \phi _0^{-2/l}\frac{1}{l^2}\left( 1-\frac{1}{l}\right) \nonumber \\&\quad \left( 1-m+\frac{m\lambda }{2\lambda -1}+\frac{2m\lambda }{2\lambda -1}\right) \end{aligned}$$
(52)

where \(l=\frac{1}{1-p/2}\). Replacing the cosmic time with conformal time, \(\tau =\int ^t{\frac{dt'}{a(t')}}\), the Mukhanov–Sasaki equation (46) now reads:

$$\begin{aligned} \tilde{{\mathcal {R}}}''_q+\frac{l+\lambda }{2(1-\lambda )\tau }\tilde{\mathcal {R}}'_q+q^2\tilde{{\mathcal {R}}}_q=0 \end{aligned}$$
(53)

The general solution to this equation is given by:

$$\begin{aligned} \tilde{{\mathcal {R}}}_q=\tau ^\nu \left( C_1 H_\nu ^{(1)}(-q\tau )+C_2 H_\nu ^{(2)}(-q\tau )\right) \end{aligned}$$
(54)

where \( H_\nu ^{(1)}\) and \(H_\nu ^{(2)}\) are the Hankel’s functions of the first and second kind of order \(\nu =-\frac{l+\lambda }{1-\lambda }+\frac{1}{2}\). Considering the asymptotic behaviour of the first Hankel’s function, \(H_\nu ^{(1)} (x)\rightarrow \sqrt{\frac{2}{\pi x}}\exp \left( {ix-i\pi \nu /2-i\pi /4}\right) \), this matches the initial conditions (47) if we choose \(C_2=0\) and

$$\begin{aligned}&-nl+m-3=0 \end{aligned}$$
(55)
$$\begin{aligned}&C_1=\frac{\lambda (1-\lambda )^{-\frac{l+\lambda }{1-\lambda }}}{4\sqrt{2}\pi l\phi _0}\sqrt{\frac{6\lambda (2\lambda -1)}{mf_0\phi _0^n\left[ 6\lambda \left( 2\lambda -1\right) \right] ^m}}\nonumber \\&\quad \times \exp ^{i\pi \nu /2+i\pi /4} \end{aligned}$$
(56)

On using the behaviour of Hankel’s function for small argument, \(H_\nu ^{(1)} (x)\rightarrow \frac{-i\Gamma (\nu )}{\pi }\left( x/2\right) ^{-\nu }\), we see that beyond the horizon, \(\tilde{{\mathcal {R}}}_q\) is a constant equal to \(\tilde{{\mathcal {R}}}^0_q=\frac{(-1)^{1-\nu }i\Gamma (\nu )C_1}{\pi }\left( q/2\right) ^{-\nu }\).

The deviation from the scale-invariant behaviour is given by the scalar spectral index, \(n_s\):

$$\begin{aligned} n_s\equiv 4-2\nu =1+\frac{(2+l)}{1-\lambda } \end{aligned}$$
(57)

In comparison, the calculated deviation in a power law inflationary epoch in general relativity gives \(n_s=1+2/(1-\lambda )\) [1]. According to Planck 2015 results [26], \(n_s = 0.968 \pm 0.006\), this shows that in this model \(\lambda , l, m\) and n are constrained as

$$\begin{aligned} \begin{aligned}&\lambda =26.338\pm 4.587 \quad l=-1.215\pm 0.006\\&m=1.430\pm 0.111 \quad n=1.292\pm 0.016 \end{aligned} \end{aligned}$$
(58)

Also Eqs. (49) and (52) fix \(f_0\) and \(V_0\) up to the value of \(\phi _0\). In the next section we find an upper bound for \(\phi _0\) from the observational data.

4 Tensor perturbations in generalized scalar-tensor gravity

In this section we would like to consider the initial condition for tensor metric perturbation. It plays an important role in cosmology since the quantum tensor fluctuation of metric generates the gravitational wave fluctuation which can be also observed. The tensor metric perturbations is a gauge invariant quantity and the initial condition problem for it can be treated in the same way as it is done for the scalar perturbations in the previous section.

Varying the action (1) with respect to the metric (5), considering only the tensor perturbation, gives the equation of motion of \({\mathcal {D}}_{ij}\) as following [11]

$$\begin{aligned} \ddot{{\mathcal {D}}}_{ij}+\frac{(a^3F)^.}{a^3F}\dot{{\mathcal {D}}}_{ij}-\frac{\Delta }{a^2}{\mathcal {D}}_{ij}=0 \end{aligned}$$
(59)

Since for a single scalar field, the tenor component of the anisotropic inertia is zero, the scalar field is explicitly appeared in the above equation through function F. It is straightforward to see that the tensor fluctuations have plane-wave solutions of the form \(e_{ij}{\mathcal {D}}_q(t)e^{i\mathbf{q }.\mathbf{x }}\) where \({\mathcal {D}}_q(t)\) satisfies

$$\begin{aligned} \ddot{{\mathcal {D}}}_q+\frac{(a^3F)^.}{a^3F}\dot{{\mathcal {D}}}_q+\frac{q^2}{a^2}{\mathcal {D}}_q=0 \end{aligned}$$
(60)

Just like the scalar perturbations, at sufficiently very early times where \(q/a\gg H\), the Eq. (60) can be solved using the WKB approximation

$$\begin{aligned} {\mathcal {D}}_q(t)\rightarrow z(t)\exp {\left( -iq\int ^t\frac{dt'}{a(t')}\right) } \end{aligned}$$
(61)

in which z(t) is a slowly varying function in comparison with q / a. Substituting (61) into (60), up to leading order in q / a we get

$$\begin{aligned} {\dot{z}}+\left( H+\frac{{\dot{F}}}{2F}\right) z=0 \end{aligned}$$
(62)

Therefore

$$\begin{aligned} z(t)=\frac{K}{a\sqrt{F}} \end{aligned}$$
(63)

where K is an arbitrary constant of integration whose q-dependence must be determined. Following [1] and comparing (63) and (31), we choose \(K=\frac{\kappa }{(2\pi )^{3/2}\sqrt{2q}}\). Thus the initial condition of tensor fluctuation for \(a\longrightarrow 0\) is

$$\begin{aligned} {\mathcal {D}}_q(t)\rightarrow \frac{\kappa }{(2\pi )^{3/2}a\sqrt{2qF}}\exp {\left( -iq\int ^t\frac{dt'}{a(t')}\right) } \end{aligned}$$
(64)

Thus the general solution of (60), satisfying the conditions (6) would be

$$\begin{aligned}&{\mathcal {D}}_{ij}(\mathbf x ,t)=\sum _{\lambda =+,\times }\int d^3q\nonumber \\&\quad \times \left[ {\mathcal {D}}_q(t)e^{i\mathbf q .\mathbf x }\beta (\mathbf q ,\lambda ){\hat{e}}_{{ij}}({\hat{q}},\lambda )+{\mathcal {D}}^*_q(t)e^{-i\mathbf q .\mathbf x }\beta ^*(\mathbf q ,\lambda ){\hat{e}}_{ij}^*({\hat{q}},\lambda )\right] \end{aligned}$$
(65)

where \({\mathcal {D}}_q(t)\) is given by (64) determined by the evolution of fluctuations in the early universe. As mentioned before, at very early universe, the fluctuations are described by quantum fields. This means that, in the above relation \(\beta (q,\lambda )\) and \(\beta ^*(q,\lambda )\) are the annihilation and creation operator for graviton and satisfy the standard commutation relations

$$\begin{aligned}&\left[ \beta (\mathbf q ,\lambda ),\beta (\mathbf q ', \lambda ')\right] =0,\nonumber \\&\left[ \beta (\mathbf q ,\lambda ),\beta ^*(\mathbf q ',\lambda ')\right] =\delta ^3(\mathbf q -\mathbf q ')\delta _{\lambda \lambda '} \end{aligned}$$
(66)

In the limit of \(q/a\ll H\), the Eq. (60) has two independent solutions which one of them is a non-zero constant while the other one is decaying [21].

We proceed now to discuss how the tensor fluctuation grows in the inflationary model \(f(R,\phi )= f_0{\mathcal {R}}^m\phi ^n\) mentioned before. Writing Eq. (60) in terms of the conformal time yields:

$$\begin{aligned} {\mathcal {D}}''_q-\frac{2\lambda +nl-2m+2}{(\lambda -1)\tau }{\mathcal {D}}'_q+q^2 {\mathcal {D}}_q=0 \end{aligned}$$
(67)

This is similar to the equation (53) and thus their solutions are the same except that \(\nu \) and \(C_1\) are replaced with \({\tilde{\nu }}=\frac{\lambda +nl/2-m+1}{\lambda -1}+\frac{1}{2}\) and

$$\begin{aligned}&\tilde{C_1}=\frac{\kappa (1-\lambda )^{\frac{\lambda +nl/2-m+1}{1-\lambda }}}{4\sqrt{2}\pi }\nonumber \\&\quad \sqrt{\frac{6\lambda (2\lambda -1)}{mf_0\phi _0^n\left[ 6\lambda \left( 2\lambda -1\right) \right] ^m}} \exp ^{i\pi {\tilde{\nu }}/2+i\pi /4} \end{aligned}$$
(68)

Denoting the value of \({\mathcal {D}}_q\) outside the horizon by \({\mathcal {D}}^0_q\), the scalar tensor ratio, defined by \(r_q\equiv 4{\vert {\mathcal {D}}^0_q/{\mathcal {R}}^0_q\vert }^2\), has an upper bound 0.09 according to the recent Planck results [26]. Therefore:

$$\begin{aligned} \left| \frac{4 l^2 \phi _0^2}{\lambda ^2}(1-\lambda )^{\frac{2(2\lambda -m+l+1)+nl}{1-\lambda }}\right| < 0.09 \end{aligned}$$
(69)

which gives the the required upper bound of \(\phi _0\).

5 Concluding remarks

The evolution of cosmological fluctuations for radiation dominated universe depends on the initial conditions through scalar and tensor gauge invariant quantities, \({\mathcal {R}}_q^0\) and \({\mathcal {D}}_q^0\) respectively. These are the conserved curvature perturbation and gravitational wave amplitude outside the horizon respectively. To find these, it is required to go back to the inflationary era and obtain the equations governing the scalar and tensor perturbations. The initial conditions of these equations come from the very early stages of inflation when the fluctuations are indeed quantum fields. Furthermore, in general relativity [1], at very early times of inflation when the fluctuations are inside the horizon, the behavior of the scalar and tensor fluctuations are independent of the inflaton’s potential. While outside the horizon, the constant quantities \({\mathcal {R}}_q^0\) and \({\mathcal {D}}_q^0\), will be depended on the nature of the potential near the horizon crossing. For studying the fluctuations inside the horizon, we need only solve the equations governing the gauge invariant quantities with their initial conditions; without any arbitrary assumptions about the strength of cosmological fluctuations [1].

In the present paper, we have studied in more details, the points explained above in the context of generalized scalar tensor gravity. We have discussed the initial conditions of the scalar and tensor perturbations in the generalized gravity. The equations of the scalar perturbations have been derived in the Newtonian gauge and then their solution have been obtained at very early times using WKB approximation. It has been shown that at very early times, we could ignore the scalar metric perturbations with respect to scalar field fluctuation just as general relativity. To find the initial condition, we have quantized the scalar and tensor fluctuations.

In order to achieve the equation governing the evolution of curvature perturbation, we have introduced a new gauge in which \(\Psi =0\) and furthermore, the fluctuations of the scalar field is related to the fluctuations of the arbitrary function F via \(\delta F/{\dot{F}}=\delta \phi /\dot{\phi _0}\). Then, in this gauge, we have derived the generalized Mukhanov–Sasaki equation using the equations, governing the evolution of the scalar perturbations and doing some straightforward algebra. Using the initial condition already obtained for the scalar perturbations, we could set the initial condition of the curvature perturbation. Also in the generalized scalar-tensor gravity, the equation of curvature perturbation has two independent adiabatic solutions outside the horizon. One is a non-zero constant and the other presents a decaying mode just as we have seen in general relativity. This non-zero constant curvature perturbation allows us to connect the past distant to the nearly present times.

Since the tensor fluctuation of metric is the origin of the gravitational wave, we have studied the tensor perturbation of metric and obtained the initial condition of it and then quantized it in the context of \(f(R,\phi )\) gravity.

As an example, for the model \(f(R,\phi )= f_0 R^m \phi ^n\), we have derived an analytical solution describing an inflationary scenario. It is shown that the free parameters of of this solution can be determined using the equation of motion and also the observational data governing the scalar spectral index and the scalar tensor ratio.