Higgs decay into two photons in a warped extra dimension

Abstract

A detailed five-dimensional calculation of the Higgs-boson decay into two photons is performed in both the minimal and the custodially protected Randall–Sundrum (RS) model, where the Standard Model (SM) fields propagate in the bulk and the scalar sector lives on or near the IR brane. It is explicitly shown that the $R_\xi$ gauge invariance of the sum of diagrams involving bosonic fields in the SM also applies to the case of these RS scenarios. An exact expression for the $h\rightarrow \gamma \gamma$ amplitude in terms of the five-dimensional (5D) gauge-boson and fermion propagators is presented, which includes the full dependence on the Higgs-boson mass. Closed expressions for the 5D $W$-boson propagators in the minimal and the custodial RS model are derived, which are valid to all orders in $v^2/M_{\text {KK}}^2$. In contrast to the fermion case, the result for the bosonic contributions to the $h\rightarrow \gamma \gamma$ amplitude is insensitive to the details of the localization of the Higgs profile on or near the IR brane. The various RS predictions for the rate of the $pp\rightarrow h\rightarrow \gamma \gamma$ process are compared with the latest LHC data, and exclusion regions for the RS model parameters are derived.

1 Introduction

After the discovery of the Higgs boson in July 2012 [1, 2], a solution to the hierarchy problem—the question why the electroweak scale is so much lower than the Planck scale—is more urgently needed than ever. Among the numerous possibilities to solve the hierarchy problem, the most popular approach is low-scale supersymmetry. A promising alternative is given by models with a warped extra dimension [3], in which the Standard Model (SM) is embedded in a slice of anti-de Sitter space, while the Higgs field is localized on the “infra-red (IR) brane”, one of the two four-dimensional hyper-manifolds that bound the extra dimension. These so-called Randall–Sundrum (RS) models can provide a natural explanation for this vast hierarchy, since the fundamental ultra-violet (UV) cutoff is the warped Planck scale, which near the IR brane takes values in the TeV range. Moreover, by allowing the fermion fields to propagate into the bulk, warped extra-dimension models can also provide an explanation for the hierarchies observed in the flavor sector [4–6] and the smallness of flavor-changing neutral currents [7–13].

Unfortunately, none of the Kaluza–Klein (KK) particles predicted in extra-dimensional extensions of the SM have been observed yet, and electroweak precision measurements indicate that these particles are probably too massive for a direct detection at the LHC. Thus, indirect searches like the precision measurements of the Higgs-boson couplings to SM particles, which are accessible via studies of both the Higgs production cross section and its decay rates, become more and more attractive. Concerning warped extra dimensions, especially the loop-mediated Higgs couplings to gluons and photons could give hints about the existence of additional KK particles. While the gluon-fusion process has been discussed extensively in several works [14–24], the present paper focuses on the Higgs decay into two photons, which was investigated in [16–20]. The first analysis of the effects of the KK tower of the $W$ boson on the $h\rightarrow \gamma \gamma$ amplitude was performed in [18]. The first complete calculation of the $h\rightarrow \gamma \gamma$ decay rate, in which both the Yukawa couplings to the $Z_2$-even and to the $Z_2$-odd fermions were included, was performed in [19]. It was found in this paper that the Higgs decay rate into two photons is enhanced relative to the SM due to the effect of the KK fermions, which turned out to give the dominating correction. At about the same time, an independent analysis of the $h\rightarrow \gamma \gamma$ decay rate came to the opposite conclusion [20]. An explanation for these deviating results was first given in [23], followed by a five-dimensional (5D) analysis in [24], which found that the two results belong to two different scenarios of the RS model.¹ While these papers focused on the contribution stemming from the fermionic KK states, in the present work we provide a detailed analysis of the bosonic loop contributions to the $h\rightarrow \gamma \gamma$ amplitude, which in unitary gauge stem from the $W$ bosons and their KK excitations. The advantages of this approach are that we are able to derive an exact result, which includes the full dependence on the Higgs-boson mass and holds to all orders in $v^2/M_{\text {KK}}^2$, where $v$ is the Higgs vacuum expectation value (VEV), and $M_\mathrm{KK}$ sets the mass scale for the low-lying KK excitations of the SM particles. It is also straightforward to extend our formulas to the case where the Higgs boson lives in the bulk of the extra dimension. Our approach also allows us to carefully study the effects of the fifth components of the gauge fields, whose profiles are discontinuous on the IR brane, similar to the $Z_2$-odd fermion profiles which indeed require a careful treatment.

Our paper is structured as follows: In Sect. 2 we define the set-up and comment on the necessity to distinguish between the so-called brane-Higgs and narrow bulk-Higgs scenarios when calculating the fermionic loop contributions to the $gg\rightarrow h$ and $h\rightarrow \gamma \gamma$ amplitudes. In Sect. 3 we discuss the general structure of the $h\rightarrow \gamma \gamma$ amplitude and summarize the results for the fermionic contributions from charged quarks and leptons propagating in the loop. We then focus on the bosonic loop contributions, calculate them in the KK-decomposed theory and show that the result for the contributions of each individual KK mode is gauge invariant. In the next step we resum the KK towers and derive an exact formula for the $h\rightarrow \gamma \gamma$ amplitude in terms of an overlap integral over the Higgs-boson profile and the transverse part of the 5D gauge-boson propagator, including the exact dependence on the Higgs-boson mass. To the best of our knowledge, such a formula has not been obtained before. We derive an explicit, closed expression for the 5D propagator and show that the overlap integral is insensitive to the precise details of shape of the Higgs-boson profile, once this profile is localized very close to the IR brane. By expanding our results in powers of $v^2/M_{\text {KK}}^2$, we can identify the contributions from the $W$ bosons (with modified couplings to the Higgs boson) and their KK towers, confirming the results of [18]. In Sect. 4 we generalize our findings to an extended version of the RS model with a custodial symmetry protecting electroweak precision observables [25–27]. Again, we obtain an exact formula for the $h\rightarrow \gamma \gamma$ amplitude and, for the first time, for the 5D gauge-boson propagator in the custodial RS model. When expanded to order $v^2/M_{\text {KK}}^2$, our findings for the contributions of the $W$ boson and its KK excitations are consistent with the findings of [19]. Phenomenological implications of our results in the context of the latest LHC data are discussed in Sect. 5, where we study the Higgs decay into two photons in two different versions of the minimal and the custodially protected RS model. We illustrate the magnitude of the effects as a function of the mass of the lightest KK gluon state and the scale of the 5D Yukawa couplings, and derive the regions in parameter space that are already excluded by recent LHC measurements. Our main results are summarized in the conclusions.

2 Preliminaries

We focus on RS models where the electroweak symmetry-breaking sector is localized on or near the IR brane. The extra dimension is chosen to be an $S^1/Z_2$ orbifold parametrized by a coordinate $\phi \in [-\pi , \pi ]$, with two branes localized on the orbifold fixed-points at $\phi =0$ (UV brane) and $|\phi |=\pi$ (IR brane). The RS metric reads [3]

$$\begin{aligned} \mathrm{d}s^2&= e^{-2\sigma (\phi )}\,\eta _{\mu \nu }\,\mathrm{d}x^\mu \mathrm{d}x^\nu - r^2\mathrm{d}\phi ^2 \nonumber \\&= \frac{\epsilon ^2}{t^2} \left( \eta _{\mu \nu }\,\mathrm{d}x^\mu \mathrm{d}x^\nu - \frac{1}{M_\mathrm{KK}^2}\,\mathrm{d}t^2\right) , \end{aligned}$$

(1)

where $e^{-\sigma (\phi )}$ with $\sigma (\phi )=kr|\phi |$ is referred to as the warp factor, and the size $r$ and curvature $k$ of the extra dimension are assumed to be of Planck size, $k\sim 1/r\sim M_\mathrm{Pl}$. The quantity $L=\sigma (\pi )=kr\pi$ measures the size of the extra dimension and is chosen to be $L\approx 33{-}34$ in order to explain the hierarchy between the Planck and the TeV scales. With the help of the curvature $k$ and the warp factor evaluated at the IR brane, $\epsilon =e^{-\sigma (\pi )}$, one defines the KK mass scale as $M_\mathrm{KK}=k\epsilon$. On the right-hand side of (1) we have introduced the dimensionless coordinate $t$ defined by $t=\epsilon \,\mathrm{e}^{\sigma (\phi )}\in [\epsilon ,1]$,² which will be used throughout this work.

Our paper deals with the minimal RS model based on the SM gauge group, as well as with an extended RS model with a larger bulk gauge group, which after electroweak symmetry breaking includes an $SU(2)$ custodial symmetry protecting the $T$ parameter and suppressing larger corrections to the $Z{\bar{b}} b$ couplings [25–27]. In both versions of the RS model, all fermions and gauge bosons are allowed to propagate into the bulk, resulting in infinite towers of heavy KK copies of the SM particles. In contrast, the scalar sector is assumed to reside on or near the IR brane so as to provide a solution to the hierarchy problem. We recall that RS models are effective field theories valid up to a position-dependent UV cutoff [28–31]

$$\begin{aligned} \Lambda _\mathrm{UV}(t)\approx M_\mathrm{Pl}\,e^{-\sigma (\phi )} = M_\mathrm{Pl}\,\frac{\epsilon }{t} \equiv \frac{\Lambda _\mathrm{TeV}}{t}. \end{aligned}$$

(2)

At this scale, gravity becomes strong and the model needs to be UV-completed into a theory of quantum gravity. The fact that RS models are defined with an inherent UV cutoff can be used to distinguish between different scenarios for the localization of the Higgs sector. We shall consider a scalar field $\Phi$ localized on or very near the IR brane. Its profile along the extra dimension is described by a normalized distribution $\delta ^\eta (t-1)$ with width $\eta \ll 1$. For $\eta \rightarrow 0$ this profile becomes a $\delta$-function, corresponding to a strictly brane-localized scalar field. Following [24], we introduce the characteristic width of the Higgs field along the extra dimension as $\Delta _h^{-1}\equiv \eta /v$. If the inverse width is larger than the inherent UV cutoff near the IR brane, $\Delta _h\gg \Lambda _\mathrm{TeV}$, then the Higgs profile cannot be resolved by the high-momentum modes contained in the effective Lagrangian of the RS model, and hence for all practical purposes such a scalar field can be regarded as a brane-localized Higgs field. On the other hand, if the inverse Higgs width is smaller than the cutoff scale, then the modes of the effective theory can resolve its profile, and we speak of a bulk Higgs field. While calculations in generic bulk-Higgs models are rather complicated, it has been shown in [24, 32] that there is the possibility of obtaining analytical results for the special case of a narrow bulk-Higgs field, whose inverse width is such that $M_\mathrm{KK}\ll \Delta _h\ll \Lambda _\mathrm{TeV}$. As has been explained in [23, 24], the fermionic loop contributions to the $gg\rightarrow h$ and $h\rightarrow \gamma \gamma$ amplitudes are sensitive to the details of the localization mechanism, and indeed the results obtained in a brane-localized Higgs scenario [19, 22] differ significantly from those derived for a narrow bulk-Higgs field [20, 33]. We thus need to distinguish between the two types of localization mechanisms in our phenomenological analysis. As we will show, however, the bosonic contributions to the $h\rightarrow \gamma \gamma$ amplitude are insensitive to the precise localization of the scalar sector and approach an unambiguous result in the limit where $\eta \ll 1$.

3 5D analysis of the $h\rightarrow \gamma \gamma$ amplitude

Our goal is to calculate the $h\rightarrow \gamma \gamma$ decay amplitude entirely in terms of the 5D propagators for both gauge bosons and fermions. While the contributions from quarks and charged leptons can easily be deduced from the corresponding results for the $gg\rightarrow h$ amplitude, a detailed consideration of the gauge-boson contribution has not yet been performed in 5D language. Our approach in the present work is the following: In a first step, we calculate the bosonic contributions to the $h\rightarrow \gamma \gamma$ amplitude in the KK-decomposed, 4D effective theory and show that at each KK level the sum of all diagrams is gauge invariant. The only contributing diagrams in the unitary gauge are those with vector bosons propagating in the loop. We can then rewrite the amplitude, summed over KK states, as an expression involving the 5D gauge-boson propagator in the mixed momentum-position representation [28, 34–37]. We show that in the limit of a very narrow Higgs profile the amplitude approaches an unambiguous value, which is insensitive to the details of the Higgs localization mechanism. At the end of this section, we employ our exact results to derive expressions for the contributions of the zero modes (the standard $W$ bosons) and their infinite towers of KK excitations to the $h\rightarrow \gamma \gamma$ amplitude.

We begin with the calculation in the minimal RS model with the SM gauge group in the bulk, broken to $U(1)_\mathrm{EM}$ on the IR brane, where the Higgs field develops a VEV. Details for the implementation of the Higgs, gauge-boson, and gauge-fixing sectors in the context of this model and using our notation have been given in [10], while Appendix A includes a summary of the relevant Feynman rules needed for our analysis. Here, it suffices to mention that we decompose the 5D $W$-boson field into 4D mass eigenstates

$$\begin{aligned} \begin{array}{rcl} W_\mu ^\pm (x,t) &{}=&{} \displaystyle \frac{1}{\sqrt{r}} \sum _{n=0}^\infty {\chi }_n^{\,W}(t)\,W_\mu ^{\pm (n)}(x), \\ W_\phi ^\pm (x,t) &{}= &{}\displaystyle - \frac{1}{\sqrt{r}}\,\frac{L}{\pi } \sum _{n=0}^\infty \frac{1}{m_n^W}\,t\, \partial _t\chi _n^W(t)\,\varphi _W^{\pm (n)}(x), \end{array} \end{aligned}$$

(3)

Fig. 1

One-loop Feynman diagrams for the process $h\rightarrow \gamma \gamma$. Diagram (a) contains the fermion loops, while diagrams (b)–(k) show the contributions from the gauge sector in a general $R_\xi$ gauge. Solid lines represent fermion mass eigenstates, wavy lines vector-boson mass eigenstates $W_\mu ^{\pm (n)}$, dashed lines scalar mass eigenstates $\varphi _W^{\pm (n)}$, and dotted lines ghost mass eigenstates $c_W^{\pm (n)}$. The ghost masses and profiles are the same as for the $W$ bosons and their KK excitations [38]

where $W_\mu ^{\pm (n)}$ are the KK modes of the $W$ bosons with masses $m_n^W$. The scalar particles $\varphi _W^{\pm (n)}$ are “unphysical” in the sense that they provide the longitudinal degrees of freedom of the $W$ bosons ($n=0$) and the massive $W$-boson KK modes ($n\ge 1$), and thus they can be gauged away. Indeed, the scalar fields $W_\phi ^\pm$ mix with the charged Goldstone bosons arising from the Higgs sector. Assuming for the time being that the scalar sector is localized on the IR brane, we parameterize the Higgs doublet after electroweak symmetry breaking in the usual form

$$\begin{aligned} \Phi (x) = \frac{1}{\sqrt{2}} \begin{pmatrix} -i\sqrt{2} \varphi ^+(x) \\ v + h(x) + i\varphi ^3(x) &{} \end{pmatrix}. \end{aligned}$$

(4)

Throughout this paper, $v$ denotes the Higgs VEV in the RS model, which differs from the SM value $v_\text{ SM }\approx 246\,\text{ GeV }$ by a small amount [18]. We determine $v$ from the shift to the Fermi constant derived in the RS model by considering (at tree level) the effect of the exchange of the infinite tower of KK gauge bosons on the rate for muon decay, using the definition $v_\mathrm{SM}=(\sqrt{2}\,G_F)^{-1/2}$. This yields [24]

$$\begin{aligned} \kappa _v = \frac{v}{v_\mathrm{SM}} = 1 + \frac{L m_W^2}{4c_\vartheta ^2M_{\text {KK}}^2} + \mathcal{O}\bigg (\frac{v^4}{M_{\text {KK}}^4}\bigg ), \end{aligned}$$

(5)

where $c_\vartheta =1$ in the minimal RS model. When we generalize our analysis to the case of an extended RS model with a custodial symmetry in Sect. 4, this relation will still hold, but the parameter $c_\vartheta$ will then take on a non-trivial value. The decomposition of the scalar fields $\varphi ^\pm$ into the mass eigenstates $\varphi _W^{\pm (n)}$ reads [10]

$$\begin{aligned} \varphi ^\pm (x)&= \sum _{n=0}^\infty \frac{{\tilde{m}}_W}{m_n^W}\, \sqrt{2\pi }\,\chi _n^W(1)\,\varphi _W^{\pm (n)}(x),\nonumber \\ {\tilde{m}}_W&= \frac{g_5}{\sqrt{2\pi r}}\,\frac{v}{2}, \end{aligned}$$

(6)

where ${\tilde{m}}_W$ is the leading contribution to the $W$-boson mass in an expansion in powers of $v^2/M_{\text {KK}}^2$. The relation between the two parameters can be written as [10]

$$\begin{aligned} {\tilde{m}}_W^2=m_W^2\left[ 1+\frac{m_W^2}{2M_{\text {KK}}^2} \left( \frac{L}{c_\vartheta ^2}-1+\frac{1}{2L} \right) +\mathcal{O}\bigg ( \frac{v^4}{M_{\text {KK}}^4}\bigg )\right] ,\nonumber \\ \end{aligned}$$

(7)

where once again $c_\vartheta =1$ in the minimal RS model. Since the profile of the zero mode is flat up to corrections of order $v^2/M_{\text {KK}}^2$, it follows that

$$\begin{aligned} \sqrt{2\pi }\,\chi _0^W(1)=1-\frac{m_W^2}{2M_{\text {KK}}^2}\left( L-1+ \frac{1}{2L} \right) +\mathcal{O}\bigg ( \frac{v^4}{M_{\text {KK}}^4}\bigg )\nonumber \\ \end{aligned}$$

(8)

is close to 1, and hence the fields $\varphi ^\pm$ coincide with $\varphi _W^{\pm (0)}$ to leading order. Mixing effects arise at order $v^2/M_{\text {KK}}^2$ and higher. Note also that one can adjust the gauge-fixing Lagrangian so as to cancel any mixings between $W_\mu ^\pm$ and the scalar fields $W_\phi ^\pm$ and $\varphi ^\pm$ [10].

The one-loop Feynman diagrams contributing to the $h\rightarrow \gamma \gamma$ decay amplitude are shown in Fig. 1 for a general $R_\xi$ gauge. In the subsequent section we will demonstrate that the full amplitude is gauge invariant. In the unitary gauge only the diagrams (a)–(c) contribute. In order to present our results, we find it convenient to parameterize the $h\rightarrow \gamma \gamma$ amplitude, including the contributions from SM particles, by means of two Wilson coefficients $C_{1\gamma }$ and $C_{5\gamma }$ define via

$$\begin{aligned} {\mathcal A}(h\rightarrow \gamma \gamma )&= C_{1\gamma }\,\frac{\alpha }{6\pi v}\,\langle \gamma \gamma | F_{\mu \nu } F^{\mu \nu } |0\rangle \nonumber \\&-C_{5\gamma }\,\frac{\alpha }{4\pi v}\,\langle \gamma \gamma | F_{\mu \nu } {\tilde{F}}^{\mu \nu } |0\rangle , \end{aligned}$$

(9)

where ${\tilde{F}}^{\mu \nu }=-\frac{1}{2}\epsilon ^{\mu \nu \alpha \beta } F_{\alpha \beta }$ with $\epsilon ^{0123}=-1$. Each Wilson coefficient can be written as a sum of three terms,

$$\begin{aligned} C_i = C_i^W+C_i^q+C_i^l, \end{aligned}$$

(10)

where in a general gauge $C_i^W$ includes the bosonic contributions from gauge bosons, scalar bosons, and ghosts. The calculation of these bosonic contributions is the main subject of this work. The fermionic loop contributions due to virtual quarks and leptons shown in diagram (a) can be readily deduced from expressions derived in [24]. They will be summarized in Sect. 3.1.

In our analysis we will also discuss the case of a very narrow Higgs boson localized near the IR brane, where the Higgs profile $\delta ^\eta (t-1)$ has a characteristic width $\eta$ subject to the condition $M_{\text {KK}}\ll v/\eta \ll \Lambda _\mathrm{TeV}$. In principle, such a scenario gives rise to a tower of physical scalar particles $\phi _W^{\pm (n)}$, which in some sense are the KK excitations of the charged components of the Higgs doublet. As discussed in detail in [38], these fields are defined in terms of a gauge-invariant superposition of $W_\phi ^\pm$ and $\varphi ^\pm$. It has been shown in the same reference that the effect of these heavy scalar particles on the $h\rightarrow \gamma \gamma$ amplitude is

$$\begin{aligned} C_{1\gamma }^\phi = \frac{1}{8} \sum _{n=1}^\infty \frac{v g_{h\phi \phi }^{(n,n)}}{\big (m_n^\phi \big )^2}\,A_\phi (\tau _n^\phi ), \quad C_{5\gamma }^\phi = 0, \end{aligned}$$

(11)

where $\tau _n^\phi =4(m_n^\phi )^2/m_h^2$, and the function

$$\begin{aligned} A_\phi (\tau ) = 3\tau \,\big [ \tau f(\tau ) - 1 \big ], \quad \text{ with }\ f(\tau ) = \arctan ^2\frac{1}{\sqrt{\tau -1}},\nonumber \\ \end{aligned}$$

(12)

approaches 1 for $\tau \rightarrow \infty$. In the limit of a very narrow Higgs profile the couplings $g_{h\phi \phi }^{(n,n)}$ scale like $1/\eta$, while the masses of the heavy scalar particles scale like $M_{\text {KK}}/\eta$. It follows that $C_{1\gamma }^\phi =\mathcal{O}(\eta )$, and hence this contribution decouples in the limit $\eta \rightarrow 0$, as expected. We will therefore not consider the corresponding Feynman diagrams in our analysis.

3.1 Fermionic contributions to the Wilson coefficients

The one-loop contributions to the $h\rightarrow \gamma \gamma$ amplitude due to the exchange of virtual quarks and leptons can be derived in a straightforward way from analogous results for the quark contributions to the $gg\rightarrow h$ amplitude, which were studied in [19–24]. Here we will use expressions derived in our previous work [24], where a variety of RS models were considered using the same 5D approach employed in the present work. All that is necessary is to include appropriate factor of color and electric charges. The exact result can be written in the form

$$\begin{aligned} C_{1\gamma }^q&= 3 N_c \sum _{f=u,d}\,Q_q^2\,\int \limits _0^1\mathrm{d}x \int \limits _0^{1-x}\mathrm{d}y\,(1-4xy)\nonumber \\&\times \left[ T_+^q(-xy m_h^2) - T_+^q(\Lambda _\mathrm{TeV}^2)\right] , \nonumber \\ C_{5\gamma }^q&= 2 N_c \sum _{f=u,d}\,Q_q^2\,\int \limits _0^1\mathrm{d}x \int \limits _0^{1-x}\mathrm{d}y\nonumber \\&\times \left[ T_-^q(-xy m_h^2) - T_-^q(\Lambda _\mathrm{TeV}^2)\right] , \end{aligned}$$

(13)

where $Q_u=2/3$ and $Q_d=-1/3$ are the electric charges of the quarks, and $N_c=3$ is the number of colors. The functions $T_\pm ^q(-p^2)$ are defined in terms of linear combinations of overlap integrals of the Higgs-boson profile with the chirality-odd components of the 5D fermion propagator (see eq. (15) in [24] for more details). An analogous expression, with $N_c$ replaced by 1 and $Q_q$ replaced by $Q_e=-1$ holds for the charged-lepton contribution. These exact results can be simplified by neglecting some terms of order $v^4/M_{\text {KK}}^4$ and chirally suppressed $\mathcal{O}(v^2/M_{\text {KK}}^2)$ terms, which is an excellent approximation numerically. This leads to the explicit expressions [24]

$$\begin{aligned} C_{1\gamma }^q&\approx \left[ 1 - \frac{v^2}{3M_{\text {KK}}^2}\,\mathrm{Re}\, \frac{(\varvec{Y}_u \varvec{Y}_u^\dagger \varvec{Y}_u )_{33}}{(\varvec{Y}_u)_{33}} \right] N_c\,Q_u^2\,A_q(\tau _t) \nonumber \\&+ N_c\,Q_d^2\,A_q(\tau _b) + \sum _{q=u,d} N_c\,Q_q^2\,\mathrm{Re}\,\text{ Tr }\,g(\varvec{X}_q), \nonumber \\ C_{5\gamma }^q&\approx - \frac{v^2}{3M_{\text {KK}}^2}\,\mathrm{Im} \left[ \frac{(\varvec{Y}_u \varvec{Y}_u^\dagger \varvec{Y}_u )_{33}}{(\varvec{Y}_u)_{33}} \right] N_c\,Q_u^2\,B_q(\tau _t) \nonumber \\&+ \sum _{q=u,d} N_c\,Q_q^2\,\mathrm{Im}\,\text{ Tr }\,g(\varvec{X}_q), \end{aligned}$$

(14)

and

$$\begin{aligned} C_{1\gamma }^l + i C_{5\gamma }^l \approx Q_{e}^2\,\,\text{ Tr }\,g(\varvec{X}_e), \end{aligned}$$

(15)

where the contributions from the SM fermions and the KK excitations can now readily be identified. The loop functions are given by (with $\tau _i=4m_i^2/m_h^2$)

$$\begin{aligned} A_q(\tau ) = \frac{3\tau }{2}\,\big [ 1 + (1-\tau ) f(\tau )\big ], \quad B_q(\tau ) = \tau f(\tau ). \end{aligned}$$

(16)

They both approach 1 for $\tau \rightarrow \infty$. For values $\tau <1$ the function $f(\tau )$ in (12) must be analytically continued, with $\tau \rightarrow \tau -i0$. The quantities

$$\begin{aligned} \varvec{X}_f = \frac{v}{\sqrt{2}M_{\text {KK}}}\,\sqrt{\varvec{Y}_f\varvec{Y}_f^\dagger }; \quad f = u,d,e \end{aligned}$$

(17)

are defined in terms of the dimensionless 5D Yukawa matrices of the RS model, whose entries are assumed to be random complex numbers of order 1. The observed hierarchies in the spectrum of fermion masses and mixing angles are generated when these anarchic Yukawa matrices are combined with the values of the fermion profiles near the IR brane, which are exponentially small for all light fermions [4–6]. Note that with the hermitian matrices $\varvec{X}_f$ the traces over matrix-valued functions $g(\varvec{X}_f)$ are real, so that $C_{5\gamma }^l=0$ and the only contribution to the coefficient $C_{5\gamma }^q$ arises from the top-quark contribution given by the first term on the right-hand side of (14). The precise form of the function $g(\varvec{X}_f)$ depends on the details of the localization of the scalar sector on or near the IR brane. For the two scenarios with a brane-localized Higgs and a narrow bulk Higgs, as defined in Sect. 2, one finds [23, 24]

$$\begin{aligned}&g(\varvec{X}_f) \big |_\mathrm{brane\;Higgs} = - \frac{\varvec{X}_f\tanh \varvec{X}_f}{\cosh 2\varvec{X}_f} = - \varvec{X}_f^2 + \mathcal{O}\bigg ( \frac{v^4}{M_{\text {KK}}^4} \bigg ),\quad \nonumber \\&g(\varvec{X}_f) \big |_\mathrm{narrow\;bulk\;Higgs} \!=\! \varvec{X}_f\tanh \varvec{X}_f \!=\! \varvec{X}_f^2 + \mathcal{O}\bigg (\!\frac{v^4}{M_{\text {KK}}^4}\! \bigg ),\qquad \end{aligned}$$

(18)

so that the effect of the KK tower is approximately equal but of opposite sign in the two cases. The difference is due to a “resonance effect” in the narrow bulk-Higgs scenario, where very heavy KK modes with masses of order the inverse Higgs width $\Delta _h=v/\eta$ give an unsuppressed contribution to the loop amplitude [23, 32]. At a technical level, the difference arises from the subtraction term at large Euclidean momentum in (13), which is relevant for the function $T_+^f(-p^2)$ only. For a brane-localized Higgs, this function approaches a plateau at large momenta, such that $T_+^f(\Lambda _\mathrm{TeV}^2)=\text{ Tr }\,\varvec{X}_f\tanh 2\varvec{X}_f$. For a narrow bulk Higgs, on the other hand, the function $T_+^f(p_E^2)$ vanishes like $1/p_E$ in the region of large Euclidean momenta $p_E^2=-p^2\gg (v/\eta )^2$, and hence $T_+^f(\Lambda _\mathrm{TeV}^2)$ can be set to zero. In [24] we have also considered a variant of the brane-Higgs scenario with two different Yukawa matrices $\varvec{Y}_f^C$ and $\varvec{Y}_f^S$ for the $Z_2$-even and $Z_2$-odd fermion fields. In this so-called type-II brane-Higgs model the matrices $\varvec{X}_f$ are no longer hermitian, but to leading order

$$\begin{aligned} g(\varvec{Y}_f^C,\varvec{Y}_f^S) \big |_{\mathrm{type}\text {-}\mathrm{II\;brane\;Higgs}}&= - \frac{v^2}{2M_{\text {KK}}^2}\,\varvec{Y}_f^C \varvec{Y}_f^{C\dagger }+ \mathcal{O}\bigg ( \frac{v^4}{M_{\text {KK}}^4} \bigg )\nonumber \\ \end{aligned}$$

(19)

is still a hermitian matrix. The type-II brane-Higgs scenario is thus rather similar to the original brane-Higgs model with identical Yukawa matrices $\varvec{Y}_f^C=\varvec{Y}_f^S=\varvec{Y}_f$. Numerically, we find that the main difference is a slightly larger spread of the distribution of scatter points when one scans over the parameter space of the model. In our phenomenological analysis in Sect. 5, we will therefore restrict ourselves to a study of the two cases shown in (18).

3.2 Gauge invariance of the amplitude

In the SM, a recent paper [39] has thoroughly discussed the $\xi$ independence of the $h\rightarrow \gamma \gamma$ amplitude in dimensional regularization and has shown that the calculation can be performed consistently in the unitary gauge $\xi \rightarrow \infty$. In the case of the RS model, it is convenient to first work in the KK-decomposed theory, where 4D Feynman propagators have the same structure as in the SM. The Feynman rules required to evaluate the one-loop diagrams shown in Fig. 1 are summarized in Appendix A. From these rules, it follows that:

• All vertices involving one or two external photons but no Higgs boson are diagonal in KK number after one integrates over the extra-dimensional coordinate of the vertex with measure $\int _{-\pi }^\pi \mathrm{d}\phi =(2\pi /L)\int _\epsilon ^1\mathrm{d}t/t$. The Feynman rules for these vertices have the same form as in the SM after one identifies the 4D electromagnetic coupling as $e=e_5/\sqrt{2\pi r}$ [40, 41]. This is a consequence of $U(1)_\mathrm{EM}$ gauge invariance. For the mass-dependent vertex connecting a photon to $W_\mu ^{\pm (n)}\varphi _W^{\mp (n)}$, one must replace $m_W\rightarrow m_n^W$.

• As a result, all one-loop diagrams contributing to the $h\rightarrow \gamma \gamma$ amplitude involve a single KK particle in the loop. Hence, only KK-diagonal Higgs couplings are required in the calculation.

• All KK-diagonal Higgs couplings have the same structure as in the SM but come with an overall prefactor

  $$\begin{aligned} \frac{v}{2}\,\frac{g_5^2}{2\pi r}\,2\pi \left[ \chi _n^W(1) \right] ^2 = \frac{2{\tilde{m}}_W^2}{v}\,2\pi \left[ \chi _n^W(1) \right] ^2, \end{aligned}$$

  (20)

  which replaces the corresponding factor $g m_W=2m_W^2/ v_\mathrm{SM}$ in the SM. In addition, for each scalar boson $\varphi _W^{\pm (n)}$ a factor $1/m_n^W$ appears, which replaces $1/m_W$ in the corresponding SM Feynman rule for vertices involving the Goldstone bosons $\varphi ^\pm$.

It follows from these observations that, diagram by diagram and in a general $R_\xi$ gauge, the bosonic loop contributions obtained in the RS model resemble those of the SM up to trivial substitutions, such that

$$\begin{aligned}&\!\!\!\!{\mathcal A}_\mathrm{RS}^W(h\rightarrow \gamma \gamma ) \nonumber \\&\ =\frac{{\tilde{m}}_W^2}{v}\sum _{n=0}^\infty \,2\pi \!\left[ \! \chi _n^W(1)\!\right] ^2 \!\left[ \! \frac{v_\mathrm{SM}}{m_W^2}{\mathcal A}_\text{ SM }^W(h\rightarrow \gamma \gamma ) \right] _{m_W\rightarrow m_n^W}.\nonumber \\ \end{aligned}$$

(21)

For vertices involving a photon and a pair of vector bosons, fermions or ghosts, the statement that the interactions are diagonal in KK number (first bullet) is a direct consequence of the flatness of the photon profile and the orthogonality of the relevant vector-boson and fermion profiles. For interactions involving the scalar bosons $\varphi _W^{\pm (n)}$, however, which according to (3) and (6) receive contributions from $W_\phi ^\pm$ and $\varphi ^\pm$, the vertices become diagonal only after one adds up these two contributions. This is discussed in more detail in Appendix A.

Let us now explore the consequences of the general result (21). Obviously, this relation implies that for each single KK mode the $h\rightarrow \gamma \gamma$ amplitude in the RS model is gauge invariant provided the amplitude is gauge invariant in the SM. Since, as we will demonstrate below, the sum over KK modes is convergent, it follows that gauge invariance is maintained also in the 5D theory. We recall that to show gauge invariance in the SM one divides the $W$-boson propagator in $R_\xi$ gauge into two parts,

$$\begin{aligned}&\!\!\!\frac{i}{p^2-m_W^2} \left[ \frac{(1-\xi )\,p^\mu p^\nu }{p^2-\xi m_W^2} - \eta ^{\mu \nu } \right] \nonumber \\&\quad = \frac{i}{p^2-m_W^2} \left( \frac{p^\mu p^\nu }{m_W^2} - \eta ^{\mu \nu } \right) - \frac{i}{p^2-\xi m_W^2}\,\frac{p^\mu p^\nu }{m_W^2},\nonumber \\ \end{aligned}$$

(22)

where the first part coincides with the propagator in unitary gauge and the second part has the same structure as the scalar-boson and ghost propagators. It has been shown in [39] that, after adding up all diagrams, many intricate cancelations occur, and at the end only the diagrams (b) and (c) in Fig. 1 with the $W$-boson propagators in unitary gauge, as well as the fermion loop contributions shown in diagram (a), remain. We have repeated this analysis and checked these cancelations by explicit calculation, thereby confirming that it is justified to use unitary gauge also in the RS model.

3.3 5D analysis of the bosonic loop contributions to ${h\rightarrow \gamma \gamma }$

We now repeat the calculation of the bosonic loop contributions to the $h\rightarrow \gamma \gamma$ amplitude using a 5D approach. Based on the findings of the previous section we adopt unitary gauge and consider only the contributions of diagrams (b) and (c) in Fig. 1. We employ the mixed momentum-position representation of the 5D gauge-boson propagator $D^\xi _{W,\mu \nu }(t,t';p)$ [28, 34–37], in which the extra-dimensional coordinate is kept in position space. This is very convenient from a technical point of view, but it is also physically well motivated, as by AdS/CFT correspondence the position along the extra dimension defines the natural mass scale of the model [28, 42]. It is well known that even in the SM the loop-momentum integral must be regularized dimensionally in order to preserve gauge invariance. We will thus introduce a dimensional regulator $d=4-2\hat{\epsilon }$ on the loop integral in intermediate steps. This regulator can be removed at the end of the calculation. We also regularize the Higgs profile by replacing the $\delta$-function profile of the brane-localized Higgs field by a smooth function $\delta ^\eta (t-1)$ of width $\eta \ll 1$. Such a regularization is important in the calculation of the fermionic loop contributions to the $gg\rightarrow h$ and $h\rightarrow \gamma \gamma$ amplitudes. However, we will find that in the calculation of the bosonic loop contributions the limit $\eta \rightarrow 0$ can be taken without encountering any ambiguities.

Diagrams (b) and (c) give rise to the amplitude

$$\begin{aligned}&i {\mathcal A}(h\rightarrow \gamma \gamma ) = - \frac{2{\tilde{m}}_W^2}{v}\,2\pi {e}^2\,\epsilon ^*_\mu (k_1)\,\epsilon ^*_\nu (k_2) \,\eta ^{\alpha \beta } \int \limits \frac{\mathrm{d}^dp}{(2\pi )^d} \int \limits _\epsilon ^1\mathrm{d}t\,\delta ^\eta (t-1)\, \frac{2\pi }{L} \int \limits _\epsilon ^1\frac{\mathrm{d}t_1}{t_1} \nonumber \\&\quad \times \left[ \frac{2\pi }{L} \int \limits _\epsilon ^1\!\frac{\mathrm{d}t_2}{t_2}\, 2V^{\gamma \mu \lambda \rho \nu \delta } D^{\xi \rightarrow \infty }_{W,\alpha \gamma }(t,t_1,p+k_1) D^{\xi \rightarrow \infty }_{W,\lambda \rho }(t_1,t_2,p)\, D^{\xi \rightarrow \infty }_{W,\delta \beta }(t_2,t,p-k_2) \right. \nonumber \\&\quad + \left( 2\eta ^{\gamma \delta } \eta ^{\mu \nu } - \eta ^{\delta \nu } \eta ^{\gamma \mu } - \eta ^{\nu \gamma } \eta ^{\mu \delta } \right) \left. \times D^{\xi \rightarrow \infty }_{W,\alpha \gamma }(t,t_1,p+k_1)\, D^{\xi \rightarrow \infty }_{W,\beta \delta }(t_1,t,p-k_2)\right] , \end{aligned}$$

(23)

where $V^{\gamma \mu \lambda \rho \nu \delta }=V^{\gamma \mu \lambda }(p+k_1,-k_1,-p)\,V^{\rho \nu \delta }(p,-k_2, -p+k_2)$ arises from the product of two triple gauge-boson vertices, with $V^{\mu \nu \rho }(k,p,q)=\eta ^{\mu \nu } (k-p)^\rho + \eta ^{\nu \rho } (p-q)^\mu + \eta ^{\rho \mu } (q-k)^\nu$. Our goal is to rewrite this result as a Feynman parameter integral over a single 5D gauge-boson propagator, which should be possible since in the KK-decomposed theory only a single KK mode propagates in the loops. In order to simplify the answer, we decompose the 5D propagator as

$$\begin{aligned}&D^\xi _{W,\mu \nu }(t,t';p) = B_W(t,t';-p^2-i0) \left( \eta _{\mu \nu }-\frac{p_\mu p_\nu }{p^2} \right) \nonumber \\&\quad + B_W(t,t';-p^2/\xi -i0)\,\frac{p_\mu p_\nu }{p^2}, \end{aligned}$$

(24)

and use the KK decomposition

$$\begin{aligned} B_W(t,t';-p^2-i0)=\sum _{n=0}^\infty \, \frac{\chi ^W_n(t)\,\chi ^W_n(t')}{\left( m^W_n\right) ^2-p^2-i0} \end{aligned}$$

(25)

in an intermediate step. The use of the KK representation is merely a mathematical trick, similar to the use of Feynman parameters in conventional loop calculations. It is justified because all expressions are finite and the KK sum converges. At the end of the calculation we obtain an expression without any reference to KK modes.

Due to the mode-diagonality of the vertices involving a photon, we can perform the integration over $t_1$ and $t_2$ using the orthonormality relation [10, 40, 41]

$$\begin{aligned} \frac{2\pi }{L}\int \limits _\epsilon ^1\frac{\mathrm{d}t}{t} \,\chi _m^W(t)\,\chi _n^W(t) = \delta _{mn} \end{aligned}$$

(26)

for the gauge-boson profiles. Working out the Dirac algebra and making use of Passarino–Veltman reductions, we can reduce the answer to a simple Feynman parameter integral. After the contributions from the various diagrams have been combined, the dimensional regulator $\hat{\epsilon }$ can be set to 0. We find

$$\begin{aligned} C_{1\gamma }^W&= - 3\pi \,{\tilde{m}}_W^2 \int \limits _\epsilon ^1\mathrm{d}t\,\delta ^\eta (t-1) \sum _{n=0}^\infty \left[ \chi _n^W(t) \right] ^2 \left[ \frac{1}{\left( m_n^W\right) ^2} \right. \nonumber \\&\left. + 6\int \limits _0^1\mathrm{d}x\int \limits _0^{1-x}\mathrm{d}y\, \frac{1-2xy}{\left( m_n^W\right) ^2-xy m_h^2 - i0} \right] , \end{aligned}$$

(27)

and $C_{5\gamma }^W=0$. It is now a simple exercise to recast the answer in terms of the 5D propagator function $B_W(t,t';-p^2)$ defined in (24). We obtain

$$\begin{aligned} C_{1\gamma }^W&= - 3\pi {\tilde{m}}_W^2 \left[ T_W(0) \right. \nonumber \\&+\left. 6\int \limits _0^1\!\mathrm{d}x\int \limits _0^{1-x}\!\mathrm{d}y\,(1-2xy)\,T_W(-xy m_h^2) \right] , \end{aligned}$$

(28)

where $T_W(-p^2)$ denotes the overlap integral of the Higgs profile with the transverse part of the 5D $W$-boson propagator evaluated at $t=t'$,

$$\begin{aligned} T_W(-p^2)&= \int \limits _\epsilon ^1\!\mathrm{d}t\,\delta ^\eta (t-1)\,B_W(t,t;-p^2-i0) \nonumber \\&= B_W(1,1;-p^2-i0) + \mathcal{O}(\eta ). \end{aligned}$$

(29)

For the case of a brane-localized scalar sector, an explicit expression for the function $B_W$ will be derived in the following section. The more general case of a narrow bulk Higgs will be briefly considered at the end of Sect. 3.5 and will be described in more detail elsewhere. Such an analysis demonstrates that the above integral exhibits a smooth behavior in the limit of small $\eta$, so that the last identity holds and the regulator on the Higgs profile can be taken to zero without encountering any ambiguities.

Relation (28) is one of the main results of this work. It shows the exact result for the Wilson coefficient $C_{1\gamma }$ in dependence of overlap integrals of the Higgs profile and the 5D gauge-boson propagator. With the help of the findings in [38], it can be shown that this relation also holds for an arbitrary bulk-Higgs profile $\chi _h(t)$, provided one uses the corresponding 5D gauge-boson propagator in the bulk-Higgs model. Then the regularized $\delta$-function in (29) must be replaced by

$$\begin{aligned} \delta ^\eta (t-1)\rightarrow \frac{2\pi }{Lt}\,\frac{v(t)}{v}\,\chi _h(t) = 2(1+\beta )\,t^{1+2\beta } +\cdots , \end{aligned}$$

(30)

where $v(t)$ is the profile of the Higgs VEV, and we use the conventions of [24]. Note, however, that in this case it is necessary to also include the contribution (11) due to the physical scalar excitations of the bulk Higgs field. The last equation in (30) holds (up to very small corrections) in the particular bulk-Higgs model analyzed in [43, 44]. The parameter $\beta >0$ is related to the 5D mass parameter of the bulk scalar field. In the region where $\beta \gg 1$, the function on the right-hand side indeed approaches a regularized $\delta$-distribution, with a characteristic width given by $\eta =1/(2\beta )$.

Note that relation (27) results after integrating a Feynman loop integrand of the type $1/[p_E^2+(m_n^W)^2-xy m_h^2]^3$ over $\mathrm{d}^4p_E$ (after the Wick rotation), which corresponds to the integral over the second derivative $\partial _{p_E^2}^2 T_W(p_E^2-xym_h^2)$. In order for this integral to exist, we need to require that both $T_W(p_E^2)$ and $p_E\,\partial _{p_E} T_W(p_E^2)$ vanish for very large Euclidean momenta. We will show in the following section that this is indeed the case.

3.4 Calculation of the 5D gauge-boson propagator

The calculation of the propagator function $B_W$ in (24) in the RS model with a brane-localized Higgs field is a straightforward exercise and has been performed, for instance, in [28, 45, 46]. This function is the solution to the differential equation (with $p^2\equiv p^2+i0$)

$$\begin{aligned}&\!\!\!\!\left( t\,\partial _t\,\frac{1}{t}\,\partial _t + \hat{p}^2 \right) B_W(t,t';-p^2) \nonumber \\&\quad = - \frac{Lt}{2\pi M_{\text {KK}}^2}\,\delta (t-t');\quad \hat{p}^2 = \frac{p^2}{M_{\text {KK}}^2}, \end{aligned}$$

(31)

subject to the boundary conditions [10]

$$\begin{aligned}&\left. \partial _t B_W(t,t';-p^2) \right| _{t=\epsilon } =0, \nonumber \\&\left. \partial _t B_W(t,t';-p^2) \right| _{t=1^-} \!=\! b_1 B_W({1},t';-p^2), b_1 \!=\! -\! \frac{L{\tilde{m}}_W^2}{M_{\text {KK}}^2}.\nonumber \\ \end{aligned}$$

(32)

Note the close similarity with the corresponding equations for the gauge-boson profiles $\chi _n^W(t)$ given in relations (A.3) and (A.4) of Appendix A. The notation $t=1^-$ means that the IR brane is approached from the left ($t<1$). Such a prescription is necessary because the derivative of the propagator function is discontinuous on the IR brane. Integrating the differential equation (31) over an infinitesimal interval around $t=t'$, we derive the jump condition

$$\begin{aligned} \left. \partial _t B_W(t,t';-p^2) \right| _{t=t'-0}^{~t'+0} = - \frac{Lt'}{2\pi M_{\text {KK}}^2}. \end{aligned}$$

(33)

The propagator itself is continuous at $t=t'$. In the region of time-like momenta ($p^2\ge 0$), the general solution can be written in the form

$$\begin{aligned}&B_W(t,t';-p^2) \nonumber \\&\quad = \frac{Ltt'}{4M_{\text {KK}}^2}\, \frac{\left[ \hat{p} D_{10}(t_>,1) - b_1 D_{11}(t_>,1) \right] D_{10}(t_<,\epsilon )}{\hat{p} D_{00}(1,\epsilon ) - b_1 D_{10}(1,\epsilon )},\nonumber \\ \end{aligned}$$

(34)

where

$$\begin{aligned} D_{ij}(t,t') = J_i(\hat{p} t)\,Y_j(\hat{p} t') - Y_i(\hat{p} t)\,J_j(\hat{p} t'). \end{aligned}$$

(35)

For space-like momenta, we find instead (with $p_E^2=-p^2>0$ and $\hat{p}_E^2=p_E^2/M_{\text {KK}}^2$)

$$\begin{aligned}&B_W(t,t';p_E^2) \nonumber \\&\quad = \frac{Ltt'}{2\pi M_{\text {KK}}^2}\, \frac{\left[ {\hat{p}_E}D_{10}(t_>,1) + b_1 D_{11}(t_>,1) \right] D_{10}(t_<,\epsilon )}{{\hat{p}_E}D_{00}(1,\epsilon ) - b_1 D_{10}(1,\epsilon )},\nonumber \\ \end{aligned}$$

(36)

with

$$\begin{aligned} D_{ij}(t,t') \!=\! I_i({\hat{p}_E}t)\,K_j({\hat{p}_E}t') {\!-\!} (-\!1)^{i+j} K_i({\hat{p}_E}t)\,I_j({\hat{p}_E}t').\nonumber \\ \end{aligned}$$

(37)

For our result (29) we need the propagator in the time-like region, evaluated at $t=t'=1$. Using the general solution in (34), we obtain (with $\hat{p}\equiv p/M_{\text {KK}}+i0$)

$$\begin{aligned}&T_W(-p^2) \nonumber \\&\quad = \frac{1}{2\pi {\tilde{m}}_W^2}\! \left[ \! 1 \!+\! \frac{\hat{p}M_{\text {KK}}^2}{L{\tilde{m}}_W^2} \frac{J_0(\hat{p})\,Y_0(\hat{p}\epsilon ) \!-\! Y_0(\hat{p})\,J_0(\hat{p}\epsilon )}{J_1(\hat{p})\,Y_0(\hat{p}\epsilon ) \!-\! Y_1(\hat{p})\,J_0(\hat{p}\epsilon )}\!\right] ^{-1} \nonumber \\&\quad \equiv \frac{1}{2\pi {\tilde{m}}_W^2}\,\hat{T}_W(-p^2), \end{aligned}$$

(38)

which is exact to all orders in $v^2/M_{\text {KK}}^2$.³ It follows from this expression that $\hat{T}_W(0)=1$. We have thus succeeded in deriving a closed analytic expression for the Wilson coefficient $C^W_{1\gamma }$ in (28), valid for the minimal RS model with a Higgs sector localized on the IR brane. Note that we have kept the quantity ${\tilde{m}}_W$, which is the leading-order contribution to the mass of the physical $W$ boson, in the prefactor above, since it will cancel against a corresponding factor in the definition of the Wilson coefficient (28). Indeed, our final result for this coefficient takes the form

$$\begin{aligned} C_{1\gamma }^W = - \frac{3}{2}\!\left[ \!1 + 6\int \limits _0^1\!\mathrm{d}x\int \limits _0^{1-x}\!\mathrm{d}y\,(1-2xy)\, \hat{T}_W(-xy m_h^2)\!\right] .\nonumber \\ \end{aligned}$$

(39)

Before proceeding, we briefly study the behavior of the propagator function in the region of large space-like momenta. For large Euclidean momenta $p_E\gg M_{\text {KK}}$, this function approaches an inverse power-law behavior given by

$$\begin{aligned} T_W(p_E^2) = \frac{L}{2\pi M_{\text {KK}}}\,\frac{1}{p_E} + \mathcal{O}(p_E^{-2}). \end{aligned}$$

(40)

It follows that both $T_W(p_E^2)$ and $p_E\,\partial _{p_E} T_W(p_E^2)$ vanish for large Euclidean momenta $p_E^2=-p^2\rightarrow \infty$, and hence the conditions required for the validity of our relation (28) are indeed satisfied.

3.5 Analysis of the zero-mode and KK contributions

Our exact expression for the overlap integral $T_W(-p^2)$ in (38) contains the contribution of the zero mode—the standard $W$ boson with its modified coupling to the Higgs field—as well as the infinite tower of KK excitations. It is instructive to isolate the contribution from the zero mode and the KK tower explicitly. To this end, we expand the exact formula in powers of $v^2/M_{\text {KK}}^2$, using that we need this function for values $p^2=\mathcal{O}(m_h^2)$ much smaller than the KK scale $M_{\text {KK}}^2$. We find

$$\begin{aligned}&\!\!\!\!\hat{T}_W(-p^2) \nonumber \\&\quad = \frac{m_W^2}{m_W^2 - p^2 - i0} \left[ 1 - \frac{m_W^2}{2M_{\text {KK}}^2} \left( \frac{L}{c_\vartheta ^2} - 1 + \frac{1}{2L} \right) \right] \nonumber \\&\qquad + \frac{m_W^2}{2M_{\text {KK}}^2} \left( \frac{L}{c_\vartheta ^2} - 1 + \frac{1}{2L} \right) + \mathcal{O}\bigg ( \frac{v^4}{M_{\text {KK}}^4} \bigg ), \end{aligned}$$

(41)

where $c_\vartheta =1$ in the minimal RS model. In Sect. 4.4 we will show that the same result holds in the custodial RS model, where, however, the parameter $c_\vartheta =1/\sqrt{2}$ takes a different value. In the above result we have replaced the parameter ${\tilde{m}}_W$ by the physical $W$-boson mass $m_W$ using relation (7), which was derived in [10] by solving the eigenvalue equation for the $W$-boson profiles and extracting the lowest eigenvalue. In Appendix B, we present an alternative approach, where the above relation is derived with the help of our expressions for the 5D gauge-boson propagator.

Based on the formulas above, we can perform the integration over the Feynman parameters in (28) and find the Wilson coefficient

$$\begin{aligned} C_{1\gamma }^W&\!=\!&-\! \frac{21}{4}\,\big [ \kappa _W A_W(\tau _W) \!+\! \nu _W \big ] \!+\! \mathcal{O}\bigg ( \frac{v^4}{M_{\text {KK}}^4} \bigg ),\quad C_{5\gamma }^W=0,\nonumber \\ \end{aligned}$$

(42)

where $\tau _W=4m_W^2/m_h^2$, and the function

$$\begin{aligned} A_W(\tau ) = \frac{1}{7}\,\big [ 2 + 3\tau + 3\tau (2-\tau )\,f(\tau ) \big ] \end{aligned}$$

(43)

with $f(\tau )$ from (12) approaching 1 for $\tau \rightarrow \infty$. The former contribution to $C_{1\gamma }$ arises from the standard $W$ boson, whose coupling to the Higgs boson is modified, compared with the SM, by a factor $\kappa _W$ times $v_\mathrm{SM}/v$. The latter factor is accounted for by using the Higgs VEV in the RS model in the definition of the effective operators in (9). The term $\nu _W$ in (42) is due to the KK excitations. Explicitly, we obtain

$$\begin{aligned}&\kappa _W = 1- \frac{m_W^2}{2M_{\text {KK}}^2} \left( \frac{L}{c_\vartheta ^2} -1 + \frac{1}{2L} \right) ,\nonumber \\&\quad \nu _W = \frac{m_W^2}{2M_{\text {KK}}^2} \left( \frac{L}{c_\vartheta ^2} - 1 + \frac{1}{2L} \right) . \end{aligned}$$

(44)

Note that at this order $\nu _W=(1-\kappa _W)$, such that the RS corrections to $C_{1\gamma }$ in (42) would cancel in the limit $\tau _W\rightarrow \infty$. This simple relation is, however, not preserved in higher orders. Our result for $C_{1\gamma }$ agrees with a corresponding expression derived in [18]. Notice also that the value of $\kappa _W$ is consistent with relation (20), which gives $\kappa _W=\frac{{\tilde{m}}_W^2}{m_W^2}\,2\pi [\chi _0^W(1)]^2$.

We close this section by returning briefly to the case of a narrow bulk-Higgs model, in which the scalar sector is localized not on but near the IR brane. As a concrete model, we adopt the scenario discussed in [24, 43, 44] featuring a bulk scalar field, which acquires a VEV due to a Mexican-hat potential localized on the IR brane. As discussed earlier, relation (29) still holds in this model provided one makes the replacement (30) and calculated the gauge-boson propagator in the background of a bulk-Higgs field. It is the solution to the differential equation

$$\begin{aligned}&\!\!\!\left( t\,\partial _t\,\frac{1}{t}\,\partial _t + \hat{p}^2 - \frac{2\pi {\tilde{m}}_W^2}{M_{\text {KK}}^2}\,\frac{v^2(t)}{v^2} \right) B_W(t,t';-p^2) \nonumber \\&\quad = - \frac{Lt}{2\pi M_{\text {KK}}^2}\,\delta (t-t'), \end{aligned}$$

(45)

with Neumann boundary conditions $\partial _t B_W(t,t',-p^2)=0$ for $t=\epsilon , 1$. We will present the details of such an analysis elsewhere. Here it suffices to note that in the limit where $\eta =1/(2\beta )\ll 1$, one obtains

$$\begin{aligned}&\!\!\!\kappa _W^\mathrm{bulk\,Higgs} = \kappa _W + \frac{3L m_W^2}{2M_{\text {KK}}^2}\,\eta + \mathcal{O}(\eta ^2),\nonumber \\&\quad \nu _W^\mathrm{bulk\,Higgs} = \nu _W - \frac{L m_W^2}{M_{\text {KK}}^2}\,\eta + \mathcal{O}(\eta ^2) \end{aligned}$$

(46)

instead of (42). This demonstrates that the result for the bosonic loop contributions to the $h\rightarrow \gamma \gamma$ amplitude interpolates smoothly from the narrow bulk-Higgs scenario into a scenario with a brane-localized scalar sector.

4 Extension to the RS model with custodial symmetry

We will now present the generalization of the above results to the RS model with custodial protection, which has been proposed to mitigate the large corrections to electroweak precision observables, so that the lightest KK particles are in reach for the direct detection at the LHC [47–50]. We will consider an RS model based on the enlarged bulk symmetry $SU(3)_c\times SU(2)_L\times SU(2)_R\times U(1)_X\times P_{LR}$, whose $SU(2)$ subgroups are broken on the IR brane via the symmetry-breaking pattern $SU(2)_L\times SU(2)_R\rightarrow SU(2)_V$. This symmetry breaking is accomplished by means of the Higgs field transforming as a bi-doublet under the two $SU(2)$ gauge groups. In component notation, it is given by

$$\begin{aligned} \Phi (x) = \frac{1}{\sqrt{2}} \!\begin{pmatrix} v + h(x) - i\varphi ^3(x) &{} -i\sqrt{2}\,\varphi ^+(x) \\ -i\sqrt{2}\,\varphi ^-(x) &{} v + h(x) + i\varphi ^3(x) \end{pmatrix}\!,\nonumber \\ \end{aligned}$$

(47)

where $\varphi ^i$ are real scalar fields, $\varphi ^\pm =(\varphi ^1\mp i\varphi ^2)/\sqrt{2}$, and $v$ denotes the Higgs VEV in the custodial RS model. The resulting $SU(2)_V$ supplies the custodial symmetry and protects the $T$ parameter [25, 26]. The $P_{LR}$ symmetry prevents the left-handed $Zb{\bar{b}}$ coupling [27] and its flavor-changing counterparts [11] from receiving too large corrections. On the UV brane, the symmetry breaking $SU(2)_R\times U(1)_X\rightarrow U(1)_Y$ generates the SM gauge group. This is achieved by an interplay between UV and IR boundary conditions. Thorough discussions of this model containing many technical details can be found in [19, 51], and we will adopt the notation of the first reference throughout this analysis.

4.1 Quark contributions to the Wilson coefficients

As a consequence of the discrete $P_{LR}$ symmetry, the left-handed bottom quark needs to be embedded in an $SU(2)_L\times SU(2)_R$ bi-doublet with isospin quantum numbers $T_L^3=-T_R^3=-1/2$. This assignment fixes the quantum numbers of the remaining quark fields uniquely. In particular, the right-handed down-type quarks have to be embedded in an $SU(2)_R$ triplet in order to obtain a $U(1)_X$-invariant Yukawa coupling. One arrives at the following multiplet structure for the quark fields with even $Z_2$ parity:

$$\begin{aligned} Q_L&= \left( \begin{array}{ll} {u_L^{(+)}}_{\frac{2}{3}} &{} {\lambda _L^{(-)}}_{\frac{5}{3}} \\ {d_L^{(+)}}_{-\frac{1}{3}} &{} {u_L^{\prime \,(-)}}_{\frac{2}{3}} \end{array} \right) _{\!\!\frac{2}{3}},\quad u_R^c = \left( {u_R^{c\,(+)}}_{\frac{2}{3}} \right) _{\frac{2}{3}}, \nonumber \\ \mathcal{T}_R&= \mathcal{T}_{1R}\oplus \mathcal{T}_{2R} = \left( \begin{array}{l} {\Lambda _R^{\prime \,(-)}}_{\frac{5}{3}} \\ {U_R^{\prime \,(-)}}_{\frac{2}{3}} \\ {D_R^{\prime \,(-)}}_{-\frac{1}{3}} \end{array} \right) _{\!\!\frac{2}{3}} \nonumber \\&\oplus \left( {D_R^{(+)}}_{-\frac{1}{3}} \quad {U_R^{(-)}}_{\frac{2}{3}} \quad {\Lambda _R^{(-)}}_{\frac{5}{3}} \right) _{\frac{2}{3}}. \end{aligned}$$

(48)

The field $Q_L$ transforms as $(\varvec{2},\varvec{2})$ under $SU(2)_L\times SU(2)_R$, while $\mathcal{T}_R$ transforms as $(\varvec{3},\varvec{1})\oplus (\varvec{1},\varvec{3})$. The fields with odd $Z_2$ parity have the opposite chirality. Their profiles are related to those of the $Z_2$-even fields by the field equations. The inner and outer subscripts on the various fields denote their $U(1)_\mathrm{EM}$ and $U(1)_X$ charges, respectively, which are connected through the relations $Y=-T_R^3+Q_X$ and $Q=T_L^3+Y$. The superscripts on the fields specify the type of boundary conditions they obey on the UV boundary. Fields with superscript $(+)$ obey the usual mixed boundary conditions allowing for a light zero mode, meaning that we impose a Dirichlet boundary condition on the profile functions of the corresponding $Z_2$-odd fields. These zero modes correspond to the SM quarks. Fields with superscripts $(-)$ correspond to heavy, exotic fermions with no counterparts in the SM. For these states, the Dirichlet boundary condition is imposed on the $Z_2$-even fields, so as to avoid the presence of a zero mode. The UV boundary conditions for the fields of opposite $Z_2$ parity are of mixed type and follow from the field equations.

Note that we chose the same $SU(2)_L\times SU(2)_R$ quantum numbers for all three quark generations, which is necessary to consistently incorporate quark mixing in the anarchic approach to flavor in warped extra dimensions. Altogether, there are 15 different quark states in the up sector and nine in the down sector (for three generations). The boundary conditions give rise to three light modes in each sector, which are identified with the SM quarks. These are accompanied by KK towers consisting of groups of 15 and nine modes of similar masses in the up and down sectors, respectively. In addition, there is a KK tower of exotic fermion states with electric charge $Q_\lambda =5/3$, which exhibits nine excitations in each KK level.

The fermionic loop contributions to the $h\rightarrow \gamma \gamma$ amplitude in the custodial RS model can be parameterized in terms of the same $3\times 3$ Yukawa matrices appearing in the minimal model, however, with different coefficients that reflect the embeddings of the various fermion species under the enlarged bulk gauge group. The generalizations of relations (14) for the quark contributions have been worked out in [24]. They are

$$\begin{aligned} C_{1\gamma }^q&\approx \left[ 1 - \frac{2v^2}{3M_{\text {KK}}^2}\,\mathrm{Re}\, \frac{(\varvec{Y}_u \varvec{Y}_u^\dagger \varvec{Y}_u )_{33}}{(\varvec{Y}_u)_{33}} \right] N_c\,Q_u^2\,A_q(\tau _t)\nonumber \\&+ N_c\,Q_d^2\,A_q(\tau _b) \nonumber \\&+ N_c\,Q_u^2\,\mathrm{Re}\,\text{ Tr }\,g(\sqrt{2}\varvec{X}_u)\nonumber \\&+ N_c \left( Q_u^2 + Q_d^2 + Q_\lambda ^2 \right) \mathrm{Re}\,\text{ Tr }\,g(\sqrt{2}\varvec{X}_d), \nonumber \\ C_{5\gamma }^q&\approx - \frac{2v^2}{3M_{\text {KK}}^2}\,\mathrm{Im} \left[ \frac{(\varvec{Y}_u \varvec{Y}_u^\dagger \varvec{Y}_u )_{33}}{(\varvec{Y}_u)_{33}} \right] N_c\,Q_u^2\,B_q(\tau _t) \nonumber \\&+ N_c\,Q_u^2\,\mathrm{Im}\,\text{ Tr }\,g(\sqrt{2}\varvec{X}_u) \nonumber \\&+ N_c \left( Q_u^2 + Q_d^2 + Q_\lambda ^2 \right) \mathrm{Im}\,\text{ Tr }\,g(\sqrt{2}\varvec{X}_d). \end{aligned}$$

(49)

For various RS models with a brane-localized scalar sector or a narrow bulk-Higgs sector, the explicit forms of the function $g(\varvec{X}_f)$ have been given in (18) and (19). Recall that the Taylor expansion of these functions starts with $\varvec{X}_f^2$, and thus the factors of $\sqrt{2}$ arising in the quark contributions in the custodial model approximately double the contribution arising in the minimal model. Combined with the large electric charge of the $\lambda$-type quarks, one finds that due to the higher multiplicity of KK quark states the contribution in the custodial RS model is much larger than in the minimal model [22, 24], by approximately a factor 68/5.

4.2 Charged-lepton contributions to the Wilson coefficients

The result for the loop contributions to the $h\rightarrow \gamma \gamma$ amplitude involving charged leptons depends on the way in which the lepton fields are embedded into the extended gauge symmetry of the custodial RS model. As a first possibility, we consider a model in which the lepton multiplets are chosen in analogy to the quark multiplets in (48). This choice was adopted in [51]. In component notation, the corresponding fields are

$$\begin{aligned} \xi _{1L}&= \left( \begin{array}{ll} {\nu _L^{(+)}}_0 &{} {\psi _L^{(-)}}_1 \\ {e_L^{(+)}}_{-1} &{} {\nu _L^{\prime \,(-)}}_0 \end{array} \right) _{\!\! 0},\quad \xi _{2R} = \left( {\nu _R^{c\,(+)}}_0 \right) _0, \nonumber \\ \xi _{3R}&= \mathcal{T}_{3R}\oplus \mathcal{T}_{4R} = \left( \begin{array}{l} {\Psi _R^{\prime \,(-)}}_1 \\ {N_R^{\prime \,(-)}}_0 \\ {E_R^{\prime \,(-)}}_{-1} \end{array} \right) _{\!\! 0} \nonumber \\&\oplus \left( {E_R^{(+)}}_{-1} \quad {N_R^{(-)}}_0 \quad {\Psi _R^{(-)}}_1 \right) _0. \end{aligned}$$

(50)

There are 15 different lepton states in the neutrino sector and nine in the charged-lepton sector. The boundary conditions give rise to three light modes in each sector, which are identified with the SM neutrinos and charged leptons. These are accompanied by KK towers consisting of groups of 15 and nine modes in the two sectors, respectively. In addition, there is a KK tower of exotic lepton states with electric charge $Q_\psi =+1$, which exhibits nine excitations in each KK level. The gauge-invariant Yukawa interactions for these fields are constructed in complete analogy with the quark Yukawa interactions [19, 51]. They can be expressed in terms of two dimensionless $3\times 3$ Yukawa matrices $\varvec{Y}_\nu$ and $\varvec{Y}_e$, which we assume to have an anarchic structure. When dressed with the fermion profiles on the IR brane, these matrices give masses to the SM leptons. The resulting contributions to the Wilson coefficients have the same structure as in (49), except that there are no zero-mode contributions (they are proportional to $m_l^2/m_h^2$ and thus can be neglected) and that we must replace $\varvec{Y}_u\rightarrow \varvec{Y}_\nu$, $\varvec{Y}_d\rightarrow \varvec{Y}_e$, $N_c\rightarrow 1$, and $Q_u\rightarrow Q_\nu =0$, $Q_d\rightarrow Q_e=-1$, $Q_\lambda \rightarrow Q_\psi =+1$. We thus obtain

$$\begin{aligned} C_{1\gamma }^l + iC_{5\gamma }^l \approx \left( Q_{e}^2 + Q_\psi ^2 \right) \text{ Tr }\,g(\sqrt{2}\varvec{X}_e), \end{aligned}$$

(51)

with $\varvec{X}_e$ as defined in (17). It follows that the leptonic contribution in the custodial RS model is approximately 4 times larger than in the minimal model.

As a second possibility, we consider a model with a more minimal embedding of the leptons into the extended gauge group. The simplest assignment is to put the left-handed neutrino and electron into an $SU(2)_L$ doublet (as in the SM) and the right-handed electron along with a new, exotic neutral particle $N_R$ into an $SU(2)_R$ doublet. The lepton fields with even $Z_2$ parity are then chosen as

$$\begin{aligned} L_L = \begin{pmatrix} {\nu _L^{(+)}}_0 \\ {e_L^{(+)}}_{-1} \end{pmatrix}_{-\frac{1}{2}},\quad L_R^c = \begin{pmatrix} {e_R^{c(+)}}_{-1} \\ {N_R^{(-)}}_{0} \end{pmatrix}_{-\frac{1}{2}}, \end{aligned}$$

(52)

and they transform as $(\varvec{2},\varvec{1})$ and $(\varvec{1},\varvec{2})$, respectively. The choice of the boundary conditions is such that the zero modes correspond to the light leptons of the SM, without a right-handed neutrino. The gauge-invariant Yukawa interaction that can be built using these fields is

$$\begin{aligned} \mathcal{L}_Y&= \frac{v}{\sqrt{2}} \int \limits _{-\pi }^\pi \!\mathrm{d}\phi \,\delta (|\phi |-\pi )\,{e}^{-3\sigma (\phi )}\, \frac{2}{k}\,\big ( Y_e \big )_{ij} \nonumber \\&\times \big ( \bar{L}_L^i\Phi \,\varepsilon L_R^{c\,j}+ \bar{L}_R^i\Phi \,\varepsilon L_L^{c\,j} \big ) + \text{ h.c. }, \end{aligned}$$

(53)

where $\varepsilon =i\sigma ^2$. Upon electroweak symmetry breaking this generates a mass term for the zero modes of the charged leptons. The SM neutrinos remain massless at this order. Their masses can be explained by means of higher-dimensional operators. The only additional lepton field is the right-handed neutrino, which is charged under $SU(2)_R$ but electrically neutral, so that it does not affect the $h\rightarrow \gamma \gamma$ decay amplitude. The lepton contribution is therefore the same as in the minimal version of the RS model, namely $C_{1\gamma }^l+iC_{5\gamma }^l\approx Q_\mathrm{e}^2\,\text{ Tr }\,g(\varvec{X}_e)$ as in (15).

4.3 The bosonic sector

In order to derive the Feynman rules and the 5D gauge-boson propagator it is inevitable to understand the bosonic sector of the custodial RS model, whose 5D action reads

$$\begin{aligned} S_\mathrm{gauge} \!=\! \int \mathrm{d}^4x\,\frac{2\pi r}{L} \int \limits _{\epsilon }^1\!\frac{\mathrm{d}t}{t} \left( \mathcal {L}_{L,R,X} \!+\! \mathcal {L}_\mathrm{Higgs} \!+\! \mathcal {L}_\mathrm{GF} \right) .\quad \end{aligned}$$

(54)

Since it is of no significance for our discussion, we refrain from presenting the gauge-fixing term $\mathcal {L}_\mathrm{GF}$, whose explicit form can be found in [19]. The gauge-kinetic terms read

$$\begin{aligned} \mathcal {L}_{L,R,X}&= \frac{\sqrt{G}}{r}\,G^{KM} G^{LN} \left( - \frac{1}{4}\,L_{KL}^a L_{MN}^a \right. \nonumber \\&\left. - \frac{1}{4}\,R_{KL}^a R_{MN}^a - \frac{1}{4}\,X_{KL} X_{MN} \right) , \end{aligned}$$

(55)

where $G^{MN}$ denotes the 5D metric with determinant $G=r^2 {e}^{-8\sigma (\phi )}$, and where a sum over the gauge-group indices $a=1,2,3$ is implied. We choose the 4-components of the gauge fields to be even under the $Z_2$ parity, while the fifth components are chosen to be odd, in order to derive at a low-energy spectrum that is compatible with observation. As in the previous section, it does not make any difference if we consider a narrow bulk- or a brane-localized Higgs sector, and we thus focus on the scenario in which the Higgs Lagrangian is localized on the IR brane. The Higgs transforms as a bi-doublet under $SU(2)_L\times SU(2)_R$ and is neutral with respect to $U(1)_X$; see (47). In order to show how the symmetry breaking $SU(2)_L\times SU(2)_R\rightarrow SU(2)_V$ is accomplished, we use the covariant derivative

$$\begin{aligned} D_\mu \Phi = \partial _\mu \Phi - ig_{L,5} L_\mu ^a T^a_L\,\Phi + ig_{R,5} \Phi R_\mu ^a T^a_R, \end{aligned}$$

(56)

where $g_{L,5}$ and $g_{R,5}$ are the 5D gauge couplings associated with $SU(2)_{L,R}$, and $T_{L,R}^a=\sigma ^a/2$ are the corresponding generators. In order to evaluate the kinetic term for the scalar bi-doublet, it is convenient to rotate the gauge bosons $L_\mu ^a$ and $R_\mu ^a$ into a new basis of fields $\tilde{A}_\mu ^a$ and $V_\mu ^a$, such that [52]

$$\begin{aligned} \begin{pmatrix} \tilde{A}_M^a \\ V_M^a \end{pmatrix} = \begin{pmatrix} \cos \vartheta &{}&{} -\sin \vartheta \\ \sin \vartheta &{}&{} \cos \vartheta \end{pmatrix} \begin{pmatrix} L_M^a \\ R_M^a \end{pmatrix} \equiv \varvec{R}_\vartheta \begin{pmatrix} L_M^a \\ R_M^a \end{pmatrix}, \end{aligned}$$

(57)

where

$$\begin{aligned} \cos \vartheta = \frac{g_{L,5}}{\sqrt{g_{L,5}^2 + g_{R,5}^2}}, \quad \sin \vartheta = \frac{g_{R,5}}{\sqrt{g_{L,5}^2 + g_{R,5}^2}}. \end{aligned}$$

(58)

The $P_{LR}$ symmetry, which is imposed to protect the left-handed $Z{\bar{b}} b$ couplings from receiving large corrections [27], enforces us to have $g_{L,5}=g_{R,5}$, and hence $\cos \vartheta =\sin \vartheta =1/\sqrt{2}$. In our discussion in this section we will, however, keep the value of $\vartheta$ as a free parameter. The Higgs VEV $\langle \Phi \rangle =(v/\sqrt{2})\,\varvec{1}$ then generates a mass term $M_{\tilde{A}}^2=v^2(g_{L,5}^2+g_{R,5}^2)/4$ for the fields $\tilde{A}_\mu ^a$, while the fields $V_\mu ^a$ remain massless. We can also read off the coupling to the Higgs boson, once we replace $v^2$ by $(v+h)^2$. Note that only the fields $\tilde{A}_\mu ^a$ couple to the Higgs boson $h$. This will become important for the derivation of the propagator. The further symmetry breaking via boundary conditions is not relevant for our discussion, and we again refer to [19] for details. Notice that relation (57) represents the connection between the UV basis fields (right), which obey Dirichlet boundary conditions on the UV brane, and the IR basis fields (left), which obey Dirichlet boundary conditions on the IR brane.

We now focus on the charged sector and define the gauge-boson fields

$$\begin{aligned} \vec W_M^\pm \equiv \begin{pmatrix} \tilde{A}_M^\pm \\ V_M^\pm \end{pmatrix} = \varvec{R}_\vartheta \begin{pmatrix} L_M^\pm \\ R_M^\pm \end{pmatrix} = \frac{\varvec{R}_\vartheta }{\sqrt{2}} \begin{pmatrix} L_M^1 \mp i L_M^2 \\ R_M^1 \mp i R_M^2 \end{pmatrix}, \end{aligned}$$

(59)

whose KK decomposition can be written in a form analogous to (3), such that [19]

$$\begin{aligned} \vec W_\mu ^\pm (x,t)&= \frac{\varvec{R}_\vartheta }{\sqrt{r}} \sum _{n=0}^\infty \vec {\chi }_n^{\,W}(t)\,W_\mu ^{\pm (n)}(x), \nonumber \\ \vec W_\phi ^\pm (x,t)&= -\frac{\varvec{R}_\vartheta }{\sqrt{r}}\, \frac{L}{\pi } \sum _{n=0}^\infty \,\frac{1}{m_n^W}\, t\,\partial _t\vec {\chi }_n^{\,W}(t)\,\varphi _W^{\pm (n)}(x). \end{aligned}$$

(60)

The orthonormality relation for the gauge-boson profiles reads

$$\begin{aligned} \frac{2\pi }{L} \int \limits _{\epsilon }^1\!\frac{\mathrm{d}t}{t}\,\vec {\chi }_n^{\,W}(t)^T\, \vec {\chi }_m^{\,W}(t) = \delta _{nm}. \end{aligned}$$

(61)

The profiles $\vec {\chi }_n^{\,W}(t)$ are $Z_2$-even functions on the orbifold. Their upper (lower) components are “untwisted” (“twisted”) functions. Untwisted even functions obey Neumann boundary conditions on the UV brane, allowing for light zero modes. Twisted even functions obey Dirichlet boundary conditions on the UV brane and are thus not smooth at this orbifold fixed point. The upper (lower) components of the rotated profiles $\varvec{R}_\vartheta \vec {\chi }_n^{\,W}(t)$ obey mixed (Neumann) boundary conditions on the IR brane, such that

$$\begin{aligned} \varvec{R}_\vartheta \,\partial _t\vec {\chi }_n^{\,W}(t) \big |_{t=1^-}&= - \frac{L{\tilde{m}}_W^2}{c_\vartheta ^2M_{\text {KK}}^2}\,\varvec{P}_+ \varvec{R}_\vartheta \,\vec {\chi }_n^{\,W}(1);\nonumber \\ {\tilde{m}}_W&= \frac{g_{L,5}}{\sqrt{2\pi r}}\,\frac{v}{2}, \end{aligned}$$

(62)

which generalizes relation (A.4) in Appendix A. Here $\varvec{P}_+=\text{ diag }(1,0)$ is a projector on the upper component, and from now on we use the abbreviations $c_\vartheta \equiv \cos \vartheta$ and $s_\vartheta \equiv \sin \vartheta$. As in the minimal RS model, the parameter ${\tilde{m}}_W$ is the leading contribution to the $W$-boson mass in an expansion in powers of $v^2/M_{\text {KK}}^2$; see (7).

It is now straightforward to deduce the Feynman rules in the custodial model from the ones in the minimal model compiled in Appendix A. Using (61), we can convince ourselves that the $W_M^\pm$ couplings to the photon are not changed at all. This statement is independent of the basis, since the rotation matrix $\varvec{R}_\vartheta$ drops out in the orthonormalization condition. In contrast, as mentioned earlier the Higgs only couples to the IR basis fields $\tilde{A}_\mu ^\pm$ with a strength proportional to $(g_{L,5}^2+g_{R,5}^2)$. This can be taken into account with the help of the projection operator $\varvec{P}_+$ rotated into the IR basis and accompanied by a factor $1/c_\vartheta ^2$. It follows that, compared with the SM, all KK-diagonal Higgs couplings in the custodial RS model come with a prefactor

$$\begin{aligned}&\!\!\!\frac{2{\tilde{m}}_W^2}{c_\vartheta ^2\,v}\,2\pi \, \vec \chi _n^{\,W}(1)^T\,\varvec{R}_\vartheta ^T\,\varvec{P}_+\, \varvec{R}_\vartheta \,\vec \chi _n^{\,W}(1) \nonumber \\&\quad \equiv \frac{2{\tilde{m}}_W^2}{c_\vartheta ^2\,v}\,2\pi \, \vec \chi _n^{\,W}(1)^T\,\varvec{D}_\vartheta \,\vec \chi _n^{\,W}(1), \end{aligned}$$

(63)

which replaces the corresponding factor (20) in the minimal model. Here we have introduced

$$\begin{aligned} \varvec{D}_\vartheta = \varvec{R}_\vartheta ^T\,\varvec{P}_+\,\varvec{R}_\vartheta = \begin{pmatrix} c_\vartheta ^2 &{} -s_\vartheta c_\vartheta \\ -s_\vartheta c_\vartheta &{} ~s_\vartheta ^2 \end{pmatrix}. \end{aligned}$$

(64)

In analogy with (25), we now define the propagator function

$$\begin{aligned} \varvec{B}_W^\mathrm{UV}(t,t';-p^2-i0) = \sum _{n=0}^\infty \, \frac{\vec {\chi }_n^{\,W}(t)\,\vec {\chi }_n^{\,W}(t')^T}{\left( m^W_n\right) ^2-p^2-i0} \end{aligned}$$

(65)

in terms of gauge-boson profiles in the UV basis. An explicit expression for this function will be derived in the next section. In analogy with expression (21) valid in minimal RS model, we find that the $h\rightarrow \gamma \gamma$ amplitude in the custodial RS model can be written as

$$\begin{aligned}&\!\!\!{\mathcal A}_\mathrm{cust.\,RS}^W(h\rightarrow \gamma \gamma )\nonumber \\&\quad = \frac{{\tilde{m}}_W^2}{c_\vartheta ^2\,v}\,\sum _{n=0}^\infty \,2\pi \, \vec \chi _n^W(1)^T \varvec{D}_\vartheta \,\vec \chi _n^W (1)\nonumber \\&\quad \times \left[ \frac{v_\mathrm{SM}}{m_W^2}\,{\mathcal A}_\text{ SM }^W(h\rightarrow \gamma \gamma ) \right] _{m_W\rightarrow m_n^W}. \end{aligned}$$

(66)

It follows that expression (28) for the Wilson coefficient $C_{1\gamma }^W$ derived in Sect. 3.3 remains valid, provided we replace the quantity $T_W(-p^2)$ defined in (29) with

$$\begin{aligned} T_W(-p^2) = \text{ Tr } \left[ \frac{\varvec{D}_\vartheta }{c_\vartheta ^2}\,\varvec{B}_W^\mathrm{UV}(1,1;-p^2-i0) \right] . \end{aligned}$$

(67)

4.4 Calculation of the 5D gauge-boson propagator

We now derive the exact expression for the 5D gauge-boson propagator in the RS model with custodial symmetry, which to the best of our knowledge has not been done before. The differential equation for the propagator function $\varvec{B}_\mathrm{UV}$ is the same as in the minimal model; see (31). However, the boundary conditions are modified to [19]

$$\begin{aligned}&\!\!\!\left( \varvec{P}_+\,\partial _t + \varvec{P}_- \right) \varvec{B}_W^\mathrm{UV}(t,t';-p^2) \big |_{t=\epsilon } = 0, \nonumber \\&\!\!\!\left( \partial _t - b_1 \varvec{D}_\vartheta \right) \varvec{B}_W^\mathrm{UV}(t,t';-p^2) \big |_{t=1^-} = 0;\quad b_1 = - \frac{L{\tilde{m}}_W^2}{c_\vartheta ^2M_{\text {KK}}^2}.\nonumber \\ \end{aligned}$$

(68)

The first equation follows from the boundary conditions for the UV fields $L_M^\pm$ and $R_M^\pm$, where we have defined $\varvec{P}_-=\text{ diag }(0,1)$. The second equation is a direct consequence of (62). We find that, in the region of time-like momenta ($p^2>0$), the general solution for the propagator function reads

$$\begin{aligned}&\varvec{B}_W^\mathrm{UV}(t,t';-p^2) = \frac{Ltt'}{4M_{\text {KK}}^2}\, \frac{1}{\left[ \hat{p} D_{00}(1,\epsilon ) {\!-\!} b_1 D_{10}(1,\epsilon ) \right] D_{01}(1,\epsilon ) {\!-\!} b_1 \frac{4s_\vartheta ^2}{\pi ^2\hat{p}^2\epsilon }} \nonumber \\&\quad \times \left\{ \left[ \big [ \hat{p} D_{10}(t_>,1) - b_1 D_{11}(t_>,1) \big ] D_{01}(1,\epsilon )\right. \right. \left. - b_1 \frac{2s_\vartheta ^2}{\pi \hat{p}}\,D_{11}(t_>,\epsilon ) \right] D_{10}(t_<,\epsilon )\,\varvec{P}_+ \nonumber \\&\quad + \left[ \big [ \hat{p} D_{00}(1,\epsilon ) - b_1 D_{10}(1,\epsilon ) \big ] D_{10}(t_>,1) \right. \left. + b_1 \frac{2s_\vartheta ^2}{\pi \hat{p}}\,D_{10}(t_>,\epsilon ) \right] D_{11}(t_<,\epsilon )\,\varvec{P}_- \nonumber \\&\quad - b_1 \frac{2s_\vartheta c_\vartheta }{\pi \hat{p}} \big [ D_{10}(t,\epsilon )\,D_{11}(t',\epsilon )\,\varvec{P}_{12} \left. + D_{11}(t,\epsilon )\,D_{10}(t',\epsilon )\,\varvec{P}_{21} \big ] \right\} , \end{aligned}$$

(69)

where the functions $D^\pm _{ij}(t,t')$ have been defined in (35), and we have introduced the $2\times 2$ matrices $\varvec{P}_{12}$ and $\varvec{P}_{21}$, which have an entry 1 at the corresponding position indicated by the subscripts and entries 0 otherwise. Note that up to irrelevant $\mathcal{O}(\epsilon ^2)$ terms we can replace $\hat{p}\epsilon D_{n1}(t,\epsilon )=-\frac{2}{\pi }\,J_n(\hat{p} t)$ for $n=0,1$. This gives rise to a simpler expression, in which the spurious $1/\epsilon$ term in the denominator is removed. In the limit $s_\vartheta \rightarrow 0$, we can identify the coefficient of $\varvec{P}_+$ in (69) with the result (34) obtained in the minimal RS model. Moreover, for the special case $p^2=0$ our result reduces to equation (54) in [19]. Our general results above valid for arbitrary momentum are, however, new.

It is now straightforward to calculate the quantity $T_W(-p^2)$ in (67), which we need for the calculation of the Wilson coefficient $C_{1\gamma }$ in (28). Expanding this answer in powers of $v^2/M_{\text {KK}}^2$ and for $p^2=\mathcal{O}(m_h^2)$, we recover expression (41). With respect to the minimal RS model, the only modification concerns the coefficient of the leading $L$-enhanced correction terms, which is enhanced by $1/c_\vartheta ^2$. This affects both the contributions from the $W$ boson and from the KK tower. In the custodial RS model with $P_{LR}$ symmetry, this enhancement factor is equal to 2. Note that with $c_\vartheta ^2=1/2$ the expressions in (44) are compatible with corresponding results obtained in [19]. In this reference the Wilson coefficient $C_{1\gamma }$ belonged to the operator $v h F_{\mu \nu } F^{\mu \nu }$ instead of the one in (9), and hence $\kappa _W^\mathrm{Ref.[19]}=\kappa _W/\kappa _v^2$.

5 Phenomenological implications

We now present a numerical study of the Higgs decay into two photons in both the minimal and the custodial RS model, which can be directly compared to experimental data. As in our recent work on Higgs production [24], we distinguish the two cases of a brane-localized and a narrow bulk-Higgs scenario. We consider the ratio of the measured $pp\rightarrow h\rightarrow \gamma \gamma$ cross section normalized to its SM value,

$$\begin{aligned} R_{\gamma \gamma }&= \frac{(\sigma \cdot \mathrm{Br)}(pp\rightarrow h\rightarrow \gamma \gamma )_\mathrm{RS}}{(\sigma \cdot \mathrm{Br)}(pp\rightarrow h\rightarrow \gamma \gamma )_\text{ SM }} \nonumber \\&= \frac{\big [ \big (|\kappa _g|^2 + |\kappa _{g5}|^2\big ) f_\mathrm{GF} + \kappa _W^2 f_\mathrm{VBF} \big ] \big ( |\kappa _{\gamma }|^2 + |\kappa _{\gamma 5}|^2 \big )}{\kappa _v^2\,\kappa _h},\nonumber \\ \end{aligned}$$

(70)

where we have included the two main Higgs production channels via gluon fusion (GF) and vector-boson fusion (VBF), with probabilities of $f_\mathrm{GF}\approx 0.9$ and $f_\mathrm{VBF}\approx 0.1$ at the LHC with $\sqrt{s}=8$ TeV [53]. Other Higgs production channels, such as the associated production with a $t\bar{t}$ pair or a vector boson, can be neglected to a very good approximation. The quantities $\kappa _i$ and $\kappa _{i5}$ (with $i=g,\gamma$) parameterize the values of the relevant Wilson coefficients normalized to their SM values,

$$\begin{aligned} \kappa _i = \frac{C_{1i}}{C_i^\mathrm{SM}},\quad \kappa _{i5} = \frac{3}{2}\,\frac{C_{5i}}{C_i^\mathrm{SM}}. \end{aligned}$$

(71)

Explicit expressions for the Wilson coefficients $C_{1\gamma }$ and $C_{5\gamma }$ in the RS model have been derived in Sects. 3 and 4. The corresponding SM value is $C_\gamma ^\mathrm{SM}=N_c\big [ Q_u^2\,A_q(\tau _t)+Q_d^2\,A_q(\tau _b) \big ]-\frac{21}{4} A_W(\tau _W)$. The RS effects on the gluon-fusion production process were studied in [24]. The values of the Wilson coefficients $C_{1g}$ and $C_{5g}$ can be obtained from $C_{1\gamma }$ and $C_{5\gamma }$ by replacing $Q_{u,d}\rightarrow 1$, $Q_e\rightarrow 0$, and $N_c\rightarrow 1$. In the SM we have $C_g^\mathrm{SM}=A_q(\tau _t)+A_q(\tau _b)$. Concerning the VBF production process, we note that using $\kappa _W$ as a correction factor in (70) is only approximate but sufficient for our purposes [54].

The parameter $\kappa _v$ in (70) parameterizes the shift of the Higgs VEV in the RS model relative to the SM and has been given in (5). Finally, we take into account the RS corrections to the SM Higgs width $\Gamma ^\text{ SM }_h=4.14$ MeV (for $m_h=125.5$ GeV) [55] by means of the parameter

$$\begin{aligned} \kappa _h&= {\kappa _v^2}\,\frac{\Gamma _h^\text{ RS }}{\Gamma _h^\text{ SM }} \approx 0.57\,\kappa _b^2 + 0.22\,\kappa _W^2 \nonumber \\&+ 0.09\,\big ( |\kappa _g|^2 + |\kappa _{g5}|^2 \big ) + 0.12, \end{aligned}$$

(72)

where the corrections to the decays $h\rightarrow \tau ^+\tau ^-,\,c\bar{c},\,ZZ^{(*)}, \gamma \gamma ,\dots$ have a numerically insignificant effect and therefore can be neglected (the combined branching ratio for these channels is 12 % in the SM). Neglecting some small chirally suppressed terms, the correction to the Higgs coupling to a $b{\bar{b}}$ pair can be well approximated by [19, 22, 23]

$$\begin{aligned} \kappa _b^\mathrm{min.\,RS}&\approx 1 - \frac{v^2}{3M_{\text {KK}}^2}\, \frac{(\varvec{Y}_d \varvec{Y}_d^\dagger \varvec{Y}_d )_{33}}{(\varvec{Y}_d)_{33}}, \nonumber \\ \kappa _b^\mathrm{cust.\,RS}&\approx 1 - \frac{2v^2}{3M_{\text {KK}}^2}\, \frac{(\varvec{Y}_d \varvec{Y}_d^\dagger \varvec{Y}_d )_{33}}{(\varvec{Y}_d)_{33}}. \end{aligned}$$

(73)

It is an important goal of future LHC and ILC analyses to determine as many of the effective Higgs couplings $\kappa _i$ as possible from a global fit to the data. A detailed discussion of the individual effective Higgs couplings to fermions and gauge bosons in the context of RS models will be presented in a future work [54]. At present, however, the experimental groups have not yet presented a detailed, model-independent analysis of Higgs couplings [57–59], and we will thus focus on the ratio $R_{\gamma \gamma }$ in the present work. Note also that, in contrast to the observable $R_{\gamma \gamma }$, the quantities $\kappa _i$ and $\kappa _{i5}$ are not directly observable. The gluon-fusion rate is proportional to the sum of the absolute squares of $\kappa _g$ and $\kappa _{g5}$, and no observable sensitive to a different combination of these parameters is experimentally accessible. In the case of $h\rightarrow \gamma \gamma$ decay, it is in principle possible to access the CP-violating coefficient $\kappa _{\gamma 5}$ by studying the distribution of the two electron–positron pairs in events in which both photons undergo nuclear conversions [56]; however, this will be very challenging experimentally.

Fig. 2

Predictions for the ratio $R_{\gamma \gamma }$ as a function of the lightest KK gluon mass $M_{g^{(1)}}$ and for different values of the parameter $y_\star$ in the minimal RS model, for the cases of a brane-localized Higgs boson (left) and a narrow bulk-Higgs field (right). The dashed curves show the approximation (74) for $y_\star =3$

Figure 2 shows our predictions for $R_{\gamma \gamma }$ obtained in the minimal RS model with a brane-localized Higgs sector (left plot) and a narrow bulk-Higgs state (right plot). The new-physics effects arising in these scenarios scale with $1/M_{\text {KK}}^2$. We find it useful to convert the mass parameter $M_{\text {KK}}$ to the physical mass $M_{g^{(1)}}\approx 2.45M_{\text {KK}}$ of the lightest KK gluon (or KK photon) state, which is independent of the details of the localization of the scalar sector and of the choice of the electroweak gauge group in the bulk [40, 41]. Our numerical results also depend on the RS volume $L=\ln (M_\mathrm{Pl}/\Lambda _\mathrm{TeV})$ and the dimensionless 5D Yukawa matrices $\varvec{Y}_u$, $\varvec{Y}_d$ and $\varvec{Y}_l$. Typical values for $L$ fall in the range $33{-}34$, corresponding to $\Lambda _\mathrm{TeV}\sim 20{-}50$ TeV, and we take $L=33.5$ for concreteness. We work with anarchic Yukawa matrices, whose individual entries are taken to be random complex numbers subject to the condition that $0\le |(\varvec{Y}_f)_{ij}|\le y_\star$. As a further constraint, we impose the condition that these matrices correctly reproduce the Wolfenstein parameters $\rho$ and $\eta$ of the unitarity triangle (see [10] for explicit formulas) and that, with appropriately chosen bulk mass parameters $c_i$, one can reproduce the correct values for the masses of the SM quarks. The dominant corrections to the $gg\rightarrow h$ and $h\rightarrow \gamma \gamma$ amplitudes arise from fermionic loop contributions and scale with $y_\star ^2$ [19–24]. The value of $y_\star$ should be naturally of $\mathcal{O}(1)$, and requiring that one-loop corrections to the Yukawa couplings remain perturbative one can derive an upper bound $y_\star \lesssim 3$ [9] (see also [24] for a detailed discussion). The green, red, and blue scatter points in the figure correspond to RS model points obtained using three different values of $y_\star$. The latest experimental values for $R_{\gamma \gamma }$ reported by the ATLAS and CMS collaborations are $R_{\gamma \gamma }^\mathrm{ATLAS}=1.55^{+0.33}_{-0.28}$ (at $m_h= 125.5$ GeV) [57] and $R_{\gamma \gamma }^\mathrm{CMS}=0.77\pm 0.27$ (at $m_h=125.7$ GeV) [58] which we naively average to obtain $R_{\gamma \gamma }=1.08^{+0.21}_{-0.19}$. The $1\sigma$ error band corresponding to this result is shown by the blue band in the two plots. Model points falling outside these bands are excluded at 68 % confidence level (CL). It is interesting to observe that for relatively large values for $y_\star$ the data already disfavor KK gluon masses in the low TeV range. The tensions between our theoretical predictions and the experimental data are stronger for the brane-Higgs model due to the mild tendency of an enhanced cross section seen in the data, which is in conflict with the suppression of the predicted cross section in this case. We emphasize, however, that using the individual values for $R_{\gamma \gamma }$ reported by ATLAS and CMS one would obtain different conclusions.

The shape of the various bands of scatter points shown in the plots can be understood as follows. For not too small Yukawa couplings, the largest RS corrections are those arising from fermionic loop contributions. In the brane-localized Higgs (narrow bulk-Higgs) scenario, they suppress (enhance) the gluon-fusion cross section and enhance (suppress) the decay rate into photons. Since the dominant SM contribution to $h\rightarrow \gamma \gamma$ involves $W$-boson loops and acts in the opposite direction as the fermionic contributions, the RS corrections to the Higgs production cross section dominate over those to the decay rate. Hence, we find a suppression (an enhancement) of $R_{\gamma \gamma }$ in the brane-Higgs (narrow bulk-Higgs) scenario. To see this more explicitly, it is instructive to expand the various expressions in (70) to first order in $v^2/M_{\text {KK}}^2$ and to approximate $A_q(\tau _t)\approx 1$ and $A_q(\tau _b)\approx 0$. Keeping the dependence on $A_W(\tau _W)\approx 1.19$ explicit, we obtain

$$\begin{aligned} R_{\gamma \gamma }&\approx 1 + \frac{v^2}{2M_{\text {KK}}^2} \left[ \left( f_\mathrm{GF} - \frac{4}{3|C_\gamma ^\mathrm{SM}|} \right) \left( \mp 18 - \frac{10}{3} \right) y_\star ^2\right. - \!\left( \! f_\mathrm{VBF} + \frac{21\big [ A_W(\tau _W) - 1 \big ]}{4|C_\gamma ^\mathrm{SM}|} \right) \!\nonumber \\&\times \left. \frac{2m_W^2}{v^2} \!\left( \! L - 1 + \frac{1}{2L}\! \right) \right. \left. \frac{L m_W^2}{v^2} + 0.57\,\frac{10}{3}\,y_\star ^2 + 0.22\,\frac{2m_W^2}{v^2} \left( L - 1 + \frac{1}{2L} \right) \right. \left. - 0.09 \left( \mp 18 - \frac{10}{3} \right) y_\star ^2 \right] , \end{aligned}$$

(74)

where the first two lines contain the corrections to the production and decay rates, with corrections to the $h\rightarrow \gamma \gamma$ rate being accompanied by a factor of $1/|C_\gamma ^\mathrm{SM}|$ with $C_\gamma ^\mathrm{SM}\approx \frac{4}{3}-\frac{21}{4} A_W(\tau _W)\approx -4.9$. The third line shows the corrections to the Higgs VEV and total width, as parameterized by $\kappa _h$ in (72). The upper sign holds for the brane-localized Higgs scenario, while the lower sign corresponds to the narrow bulk-Higgs case. Above we have used the fact that for a large set of random complex matrices on average [24]

$$\begin{aligned} \left\langle \text{ Tr }\,\varvec{Y}_f\varvec{Y}_f^\dagger \right\rangle&= N_g^2\,\frac{y_\star ^2}{2},\nonumber \\ \bigg \langle \frac{\left( \varvec{Y}_u\varvec{Y}_u^\dagger \varvec{Y}_u \right) _{33}}{\left( \varvec{Y}_u \right) _{33}} \bigg \rangle&= (2N_g-1)\,\frac{y_\star ^2}{2}, \end{aligned}$$

(75)

where $N_g=3$ is the number of fermion generations. We explicitly see from the first term on the right-hand side of (74) that the fermionic contributions to the $gg\rightarrow h$ production process dominate over those to the $h\rightarrow \gamma \gamma$ decay rate and come with opposite sign. Altogether, we find

$$\begin{aligned} R_{\gamma \gamma } \approx 1 - \frac{v^2}{2M_{\text {KK}}^2} \big [ (\pm 9.7 - 0.1 )\,y_\star ^2 + 4.1 \big ]. \end{aligned}$$

(76)

For the case where $y_*=3$ this result is shown by the dashed lines in the figure. Note also that due to the contribution of the VBF production process the observable $R_{\gamma \gamma }$ is bounded from below in the brane-Higgs case. This explains the behavior for very small $M_{\text {KK}}$ values seen in the left plot in Fig. 2. For $y_*=3$, the $gg\rightarrow h$ production cross section vanishes for $M_{g^{(1)}}\approx 3.5$ TeV, because the new-physics contribution cancels the SM amplitude. However, due to the VBF production process a non-zero value of $R_{\gamma \gamma }$ remains.

Fig. 3

Excluded regions of parameter space in the minimal RS model, for the brane-localized Higgs scenario (left) and the narrow bulk-Higgs model (right). The vertical dashed line denotes the lower bound on $M_{g^{(1)}}$ obtained from a tree-level analysis of electroweak precision observables [61]

Even at the present level of precision, the existing measurements of the observable $R_{\gamma \gamma }$ already provide some interesting constraints on the parameter space of the RS models under consideration. In Fig. 3 we show the regions in the $M_{g^{(1)}}$–$y_\star$ parameter space that are excluded by the current data at various confidence levels. For instance, for the particular choice $y_\star =3$ one finds $M_{g^{(1)}}>8.5$ TeV at 95 % CL for the brane-Higgs model and $M_{g^{(1)}}>6.4$ TeV at 68 % CL for the narrow bulk-Higgs model. Weaker constraints are obtained for smaller values for $y_\star$. These bounds can be compared with the constraints derived from a tree-level analysis of the electroweak $S$ and $T$ parameters [60], which in the minimal RS model leads to the lower bound $M_{g^{(1)}}>12$ TeV at 95 % CL [61]. This value is indicated by the vertical dashed line in the figure. At present, this bound is still stronger than the constraints derived from $R_{\gamma \gamma }$.

Fig. 4

Predictions for the ratio $R_{\gamma \gamma }$ as a function of the KK gluon mass $M_{g^{(1)}}$ in the custodial RS model with minimal lepton sector (52), for the cases of a brane-localized Higgs boson (left) and a narrow bulk-Higgs field (right)

Fig. 5

Excluded regions of parameter space in the custodial RS model with minimal lepton sector (52), for the brane-localized Higgs scenario (left) and the narrow bulk-Higgs model (right). The vertical dashed line denotes the lower bound on $M_{g^{(1)}}$ obtained from a tree-level analysis of electroweak precision observables [61]

Softening the constraints from electroweak precision observables by means of a symmetry has been the main motivation for extending the RS model by enlarging the bulk gauge group [25–27]. It has been pointed out in [22, 24] that the large number of heavy fermionic degrees of freedom in such an extended model can potentially give rise to large virtual effects on the Higgs-boson production and decay rates. The corresponding effects on the quantity $R_{\gamma \gamma }$ arising in the RS model with custodial symmetry are studied in Fig. 4. In analogy with (74), we can expand the result in powers of $v^2/M_{\text {KK}}^2$, exploiting the anarchy of the 5D Yukawa matrices and making the same approximations as above. For the model with the minimal lepton sector shown in (52), this yields

$$\begin{aligned}&R_{\gamma \gamma } \approx 1 + \frac{v^2}{2M_{\text {KK}}^2} \left[ \mp \left( 72 f_\mathrm{GF} - \frac{213}{|C_\gamma ^\mathrm{SM}|} \right) y_\star ^2\right. - \frac{20}{3} \left( f_\mathrm{GF} - \frac{4}{3|C_\gamma ^\mathrm{SM}|} \right) y_\star ^2 - \left( f_\mathrm{VBF} + \frac{21\big [ A_W(\tau _W) - 1 \big ]}{4|C_\gamma ^\mathrm{SM}|} \right) \frac{2m_W^2}{v^2} \nonumber \\&\times \left( 2L - 1 + \frac{1}{2L} \right) - \frac{2L m_W^2}{v^2} + 0.57\,\frac{20}{3}\,y_\star ^2 + 0.22\,\frac{2m_W^2}{v^2} \left( 2L - 1 + \frac{1}{2L} \right) \left. - 0.09 \left( \mp 72 - \frac{20}{3} \right) y_\star ^2 \right] \,. \end{aligned}$$

(77)

If instead the extended lepton sector shown in (50) is employed, then the coefficient 213 inside the parentheses in the first term must be replaced by 240. Note that the individual corrections due to fermion loops are huge; however, significant cancelations take place when one adds the corrections to the $gg\rightarrow h$ and $h\rightarrow \gamma \gamma$ rates. Altogether, we obtain for the model with the minimal lepton sector

$$\begin{aligned} R_{\gamma \gamma } \approx 1 - \frac{v^2}{2M_{\text {KK}}^2} \big [ (\pm 15.0 - 0.2 )\,y_\star ^2 + 8.3 \big ]. \end{aligned}$$

(78)

In the model with the extended lepton sector the coefficient $\pm 15.0$ in the first term must be replaced by $\pm 9.5$. We observe that in linearized form the corrections are only moderately larger than in the minimal model; see (76). Once again, for $y_*=3$ this result is shown by the dashed lines in the figure, where we show results for the custodial model with the minimal lepton sector. If instead the model with an extended lepton sector is considered, the distribution of scatter points looks very similar. For the brane-localized Higgs scenario, Fig. 4 shows a similar behavior as in the minimal model, but the new-physics effects are slightly larger. For $y_*=1.5$ and 3, the $gg\rightarrow h$ production cross section vanishes near $M_{g^{(1)}}\approx 3.5$ TeV and 7 TeV, respectively, and the VBF process remains as the only production mechanism. This explains the minimum values for $R_{\gamma \gamma }$ at these points. For even smaller masses the ratio $R_{\gamma \gamma }$ increases and can even exceed 1. In the narrow bulk-Higgs case, on the other hand, the linearized approximation (77) breaks down for large values $y_*$, as is evident from the discrepancy between the dashed curve and the blue band of scatter points. A reasonable approximation, shown by the solid line, is obtained by linearizing the expressions for the various $\kappa _i$ parameters but not further expanding expression (70). It turns out that the negative corrections to the $h\rightarrow \gamma \gamma$ decay rate are so significant in this model that they compensate the large positive corrections to the gluon-fusion rate in the region of large $M_{g^{(1)}}$. For smaller KK masses, these negative corrections become dominant and drive the ratio $R_{\gamma \gamma }$ toward values significantly less than 1. Eventually, for $M_{g^{(1)}}\approx 3$ TeV (for $y_*=1.5$) and 5.5 TeV (for $y_*=3$), the di-photon decay rate even vanishes. It is obvious that in regions of parameter space where such dramatic cancelations occur our predictions are highly model dependent. Given the preliminary pattern of Higgs couplings seen in experiment, which within errors agree with the SM predictions, it appears unlikely (but not impossible) that there could be $\mathcal{O}(1)$ corrections to the $gg\rightarrow h$ and $h\rightarrow \gamma \gamma$ production and decay rates, which cancel each other in the result for the observable $R_{\gamma \gamma }$. Too large corrections to the gluon-fusion rate are also disfavored by the good agreement of the $pp\rightarrow ZZ^{(*)}\rightarrow 4l$ rate with its SM value. A detailed discussion of the corresponding constraints on the RS parameter space has been presented in [24].

Figure 5 shows the excluded regions of RS parameter space derived from the analysis of the observable $R_{\gamma \gamma }$ in the custodial RS model. In the scenario with a brane-localized Higgs sector, we can exclude the ranges $5.9\,\mathrm{TeV}<M_{g^{(1)}}<13.4$ TeV and $M_{g^{(1)}}<3.5$ TeV for $y_\star =3$, while in the narrow bulk-Higgs model we exclude $5.2\,\mathrm{TeV}<M_{g^{(1)}}<8.4$ TeV, both at 95 % CL. Note that there is a small region in the upper left corner of the left plot, which is allowed by both $R_{\gamma \gamma }$ and the $T$ parameter constraint $M_{g^{(1)}}>4.7$ TeV at 95 % CL. However, bounds derived from the analysis of the decay $h\rightarrow ZZ^{(*)}\rightarrow 4l$ exclude this region [24, 54].

The exclusion plots in Figs. 3 and 5 can also be read in a different way. Under the assumption that the first KK gluon resonance is in reach for direct production at the LHC, these plots allow one to impose bounds on $y_\star$. For instance, in the minimal RS model with a hypothetical KK gluon mass $M_{g^{(1)}}=5$ TeV, our results imply an upper bound of $y_\star <1.5$ at 95 % CL in the brane-Higgs model, and $y_\star <2.4$ at 68 % CL in the narrow bulk-Higgs scenario. In the custodial RS model, those bounds are tightened to $y_\star <0.9$ for a brane Higgs and $y_\star <1.7$ for a narrow bulk Higgs, both at 95 % CL. Even though the constraints are rather strong in the case of the custodial RS model, they do not quite compete with those stemming from the decays $h\rightarrow ZZ^{(*)}, WW^{(*)}$ [24, 54]. This is due to the fact that the RS corrections to the decay into two photons partially compensate the huge effect in the gluon-fusion production process. This compensation does not occur in the decays into two weak gauge bosons, whose couplings to the Higgs are only slightly affected by new-physics effects.

6 Conclusions

The discovery of a Higgs boson at the LHC in the summer of 2012 [1, 2] has marked the beginning of a new era in elementary particle physics. The couplings of this new particle are found to be non-universal and indeed very close to those predicted for the elementary scalar boson of the SM. An explanation for the hierarchy problem is thus more pressing than ever. Measuring precisely the Higgs couplings to various SM particles provides an important tool to discover and distinguish between new-physics models that can address the hierarchy problem. Especially interesting are loop-induced processes like Higgs production via gluon fusion $gg\rightarrow h$ and the radiative decay $h\rightarrow \gamma \gamma$.

In this paper, we have focused on the Higgs decay into two photons in different scenarios of RS models, where the Higgs sector is localized on or near the IR brane, while the remaining fields propagate in the bulk of the warped extra dimension. While the contribution from diagrams with virtual fermions (quarks and charged leptons) in the loop has been extensively discussed in recent work, mostly in the context of gluon fusion [19–24], here we have further concentrated our analysis on the diagrams with bosonic fields propagating in the loops, generalizing the findings of [16–20]. We have shown that the relevant diagrams, calculated in the general $R_\xi$ gauge, add up to a gauge-invariant result. Working in unitary gauge, we have derived an exact expression for the $h\rightarrow \gamma \gamma$ decay amplitude in terms of an integral over the 5D $W$-boson propagator with the Higgs-boson profile along the extra dimension, given in (28). This expression can be used to calculate the bosonic contributions to the amplitude as long as one is able to derive an analytic expression for the 5D $W$-boson propagator. We have shown that the 5D loop diagrams with bosonic fields are insensitive to the precise details of the localization of the scalar sector on or near the IR brane. This finding is in contrast to the case of fermionic loops, for which one finds different results for the cases of a strictly localized Higgs sector and a scenario in which the Higgs is a narrow bulk field [23, 24]. In our approach we retain the exact dependence of the amplitude on the Higgs-boson mass. Taking a 5D perspective on the calculation, we have not distinguished between the SM modes and the KK particles. However, in our final results we have been able to identify the contributions from the $W$ boson and the infinite tower of its KK excitations. Finally, we have generalized our findings to the RS model with custodial protection of electroweak precision observables, for which we have discussed two different embeddings of the lepton sector in the extended gauge group of the model.

In the phenomenological part of this paper, we have analyzed the new-physics effects on the cross section times branching ratio for the process $pp\rightarrow h\rightarrow \gamma \gamma$, including effects on the production cross sections via gluon fusion or vector-boson fusion, the $h\rightarrow \gamma \gamma$ decay rate, and the total Higgs width. We have focused on the ratio $R_{\gamma \gamma }$ representing the cross section times branching ratio in the RS model normalized to its SM value, and distinguished between the brane-localized and the narrow bulk-Higgs scenarios of both the minimal and the custodial RS models. We have also derived approximate formulas for $R_{\gamma \gamma }$, which allow for a reasonable parameterization of our results in terms of the KK mass scale and the maximal value $y_\star$ imposed on the magnitude of the individual entries of the anarchic 5D Yukawa matrices. We have pointed out the fact that the RS corrections to Higgs production via gluon fusion act in the opposite direction as those to the decay into two photons, where generally the new-physics effects on the gluon-fusion rate provide the dominant source of corrections to the SM predictions. For not too small values of $y_\star$ and not too large values of $M_{\text {KK}}$, new-physics effects in RS models can lead to significant deviations of $R_{\gamma \gamma }$ from 1. While in all brane-Higgs scenarios considered here $R_{\gamma \gamma }<1$ in most regions of parameter space, RS scenarios with the Higgs implemented as a narrow bulk field allow for the possibility that $R_{\gamma \gamma }>1$.

Comparing our results with the latest ATLAS and CMS data [57, 58], we have derived exclusion regions in the $M_{g^{(1)}}$–$y_\star$ parameter space of the various models, shown in Figs. 3 and 5. In the minimal model, we can exclude KK gluon masses lighter than $8.5\,\text{ TeV }\times (y_\star /3)$ at 95 % CL for the brane-localized scenario, and $6.4\,\text{ TeV }\times (y_\star /3)$ at 68 % CL for the narrow bulk-Higgs scenario. In case of the custodially protected RS model and for a brane-localized Higgs, we can exclude KK masses below $13.4\,\text{ TeV }\times (y_\star /3)$ at 95 % CL. For a narrow bulk-Higgs field, we find the excluded region $M_{g^{(1)}}<8.4\,\text{ TeV }\times (y_\star /3)$ at 95 % CL. More precise measurements of $R_{\gamma \gamma }$ and of other Higgs production and decay rates would be required to differentiate between different incarnations of RS models. Importantly, significant deviations of the Higgs couplings from their SM values could arise even if the KK mass scale is so large that KK modes are out of reach for direct production at the LHC.

Together with our previous work [24], the present paper allows for a full treatment of the loop-mediated Higgs couplings and can be supplemented by the remaining tree-level couplings to arrive at a comprehensive description of Higgs physics in RS models, where the scalar sector lives on or near the IR brane. With increasing experimental precision on the extracted Higgs couplings, it will be exciting to compare our results for various RS models with the data. Even if no KK excitations will be discovered at the LHC, it is conceivable that future precision measurements of Higgs couplings at the LHC and ILC could provide a first hint on the existence of a warped extra dimension.

Appendices

Appendix A: Feynman rules in the 4D effective theory

Here we list the Feynman rules needed for the calculation of the one-loop gauge-boson, scalar, and ghost contributions to the $h\rightarrow \gamma \gamma$ decay amplitude in the KK-decomposed version of the minimal RS model, using the notation of [10]. We work in a general $R_\xi$ gauge and use mass eigenstates of all SM particles and their KK excitations. The Feynman rules for the vertices shown above are

$$\begin{aligned} \begin{array}{ll} (a): \quad &{} \frac{{\tilde{m}}_W^2}{v m_n^W}\,2\pi \,\chi _m^W(1)\,\chi _n^W(1)\,(p_\varphi -p_h)_\nu ,\\ (b):\quad &{}\frac{2i{\tilde{m}}_W^2}{v}\,2\pi \,\chi _m^W(1)\,\chi _n^W(1)\,\eta _{\mu \nu },\\ (c):\quad &{} \pm e\,\frac{{\tilde{m}}_W^2}{v m_n^W}\,2\pi \, \chi _m^W(1)\,\chi _n^W(1)\,\eta _{\mu \nu },\\ (d): \quad &{} - \frac{im_h^2}{v}\,\frac{{\tilde{m}}_W^2}{m_m^W m_n^W}\,2\pi \,\chi _m^W(1)\,\chi _n^W(1), \\ (e): \quad &{} -ie^2\,\big ( 2\eta _{\lambda \rho }\,\eta _{\mu \nu } - \eta _{\lambda \mu }\,\eta _{\rho \nu } - \eta _{\lambda \nu }\,\eta _{\rho \mu } \big )\,\delta _{mn},\\ (f): \quad &{} ie\,(p_+ - p_-)_\mu \,\delta _{mn}, \\ (g): \quad &{} \pm e\,m_m^W\,\eta _{\mu \nu }\,\delta _{mn},\\ (h): \quad &{} 2ie^2\,\eta _{\mu \nu }\,\delta _{mn}, \\ (i): \quad &{} \pm ie\,p_\mu \,\delta _{mn},\\ (j): \quad &{} - \xi \,\frac{i{\tilde{m}}_W^2}{v}\,2\pi \,\chi _m^W(1)\,\chi _n^W(1),\\ (k): \quad &{} ie\,\delta _{mn}\,V_{\rho \mu \sigma }(p_+,p_{\gamma },p_-), \end{array} \end{aligned}$$

(A.1)

where $v$ is the Higgs VEV in the RS model, the parameter ${\tilde{m}}_W$ has been defined in (6), and the tensor structure $V_{\rho \mu \sigma }$ of the 3-boson vertex has been given in the text after (23).

The fact that the vertices $(e)$–$(i)$ and $(k)$ involving one or two photons but no Higgs boson are diagonal in KK number requires some comments. For vertices involving a photon and a pair of vector bosons, fermions or ghosts, this follows from the flatness of the photon profile, enforced by $U(1)_\mathrm{EM}$ gauge invariance, and the orthogonality of the relevant vector-boson and fermion profiles. However, the situation is different for vertices involving the scalar bosons $\varphi _W^{\pm (n)}$, which receive contributions from the original fields $W_\phi ^\pm$ and $\varphi ^\pm$; see (3) and (6). In this case the vertices become diagonal only after one adds up these two contributions. Consider, as an example, the vertex $(h)$ needed for diagram (j) in Fig. 1. After integrating over the coordinate of this vertex, we obtain the Feynman rule

where the first contribution originates from the $W_\phi W_\phi A_\mu A^\mu$ term contained in the Yang–Mills action for the $W$-boson fields using the KK decomposition (3), while the second contribution arises from the $\varphi ^+\varphi ^- A_\mu A^\mu$ term contained in the kinetic term for the Higgs doublet using the KK decomposition (6). We now integrate by parts in the first term use the equations of motion [40, 41]

$$\begin{aligned} - t\,\partial _t\,\frac{1}{t}\,\partial _t\chi _n^W(t) = \frac{\left( m_n^W\right) ^2}{M_{\text {KK}}^2}\,\chi _n^W(t) \end{aligned}$$

(A.3)

for the gauge-boson profiles along with the boundary conditions

$$\begin{aligned} \partial _t\chi _n^W(t) \big |_{t=\epsilon } = 0,\quad \partial _t\chi _n^W(t) \big |_{t=1^-} = - \frac{L{\tilde{m}}_W^2}{M_{\text {KK}}^2}\,\chi _n^W(1) \end{aligned}$$

(A.4)

corresponding to a Neumann boundary condition on the UV brane and a mixed boundary condition on the IR brane. In this way, we obtain the Feynman rule

$$\begin{aligned} 2ie^2\eta _{\mu \nu } \left[ \frac{m_n^W}{m_m^W}\, \frac{2\pi }{L} \int \limits _\epsilon ^1\!\frac{\mathrm{d}t}{t}\,\chi _m^W(t)\,\chi _n^W(t) \right] = 2ie^2\eta _{\mu \nu }\,\delta _{mn}, \end{aligned}$$

(A.5)

where the boundary term cancels the contribution arising from the $\varphi ^+\varphi ^- A_\mu A^\mu$ term. In the last step we have used the orthonormality relation (26) for the gauge-boson profiles.

Appendix B: $\varvec{W}$ mass and profile derived from the 5D propagator

For the analysis of the zero-mode contributions in Sect. 3.5 we need to expand the $W$-boson propagator at $t=t'=1$ in powers of $v^2/M_{\text {KK}}^2$. In this context, we also need the relation between the physical $W$-boson mass and the model parameter ${\tilde{m}}_W$ in (6) beyond the leading order. A corresponding formula was derived in [10] by solving the eigenvalue equation for the $W$-boson profiles and extracting the lowest eigenvalue. We now present an alternative approach based on our exact expression for the 5D gauge-boson propagator, which allows us to derive explicit expressions for the $W$-boson mass and profile to any order in $v^2/M_{\text {KK}}^2$.

Starting from the exact expression (34), we perform an expansion in powers of $v^2/M_{\text {KK}}^2$ while keeping $p^2$ and ${\tilde{m}}_W^2$ fixed and of order $v^2$. This yields

where

which are valid up to terms of order $v^4/M_{\text {KK}}^4$. The zero of the denominator of the propagator in (B.1) defines the physical mass $m_W$ of the ground state, and the residue of the pole determines the corresponding product of profile functions $\chi _0^W(t)\,\chi _0^W(t')$. Indeed, for $p^2\approx m_W^2$ we find

$$\begin{aligned} B_W(t,t';-p^2) = \frac{1}{2\pi }\,\frac{-Z_2(t,t')}{p^2-m_W^2+i0} + \text{ non-singular } \text{ terms, } \end{aligned}$$

(B.3)

where

These results are valid up to $\mathcal{O}(v^4/M_{\text {KK}}^4)$ corrections. The relation in the first line is equivalent to (7). Rewriting the second relation in the form $Z_2(t,t')=2\pi \,\chi _0^W(t)\,\chi _0^W(t')$, we can extract

which gives the first non-trivial correction to the profile of the $W$ boson [10, 45].