Tri-photon at muon collider: a new process to probe the anomalous quartic gauge couplings

The muon collider has recently received a great deal of attention because of its ability to achieve both high energy and high luminosity. It plays as a gauge boson collider because the vector boson scattering (VBS) becomes the dominant production topology for Standard Model processes starting from a few TeV of collision energy. In this paper, we propose that the process of $\mu^+\mu^-$ annihilation into tri-photon is also very sensitive to the search of anomalous quartic gauge couplings (aQGCs). We investigate the projected constraints on the transverse operators contributing to aQGCs through $\mu^+\mu^-\to Z^\ast/\gamma^\ast\to \gamma\gamma\gamma$ at muon colliders. For the muon collider with $\sqrt{s}=3$ TeV and $\mathcal{L}=1\;{\rm ab}^{-1}$, the expected constraints are about two orders of magnitude stronger than those at the 13 TeV LHC.


Introduction
As the foundation of the Standard Model (SM), the non-Abelian nature of SU (2) × U (1) gauge symmetry has received longstanding attention in both experiments and theoretical studies.The precision measurements of the self-couplings of electroweak (EW) gauge bosons drive the test of the electroweak symmetry breaking mechanism and the search for new physics (NP) beyond the SM.Any deviation away from the SM gauge interactions and anomalous gauge couplings would indicate the existence of new physics above the EW scale in terms of higher dimensional effective operators.
Recently, the progress on the studies of muon collider has drawn a lot of attention [52][53][54][55][56][57][58][59][60][61][62].Compared with the composite particles at hadron colliders, the muon collider provides a cleaner environment without suffering from QCD background.Also, the muon collider is able to reach both multi-TeV energies and a high integrated luminosity scaling with energy quadratically.As a result, the muon collider is undoubtedly an ideal place to search for new physics above the EW scale and precisely measure the aQGCs.The aQGCs induced processes can be categorized into the vector boson scattering/fusion (VBS/VBF) processes and the tri-boson productions.At the LHC, the VBS processes are generally more sensitive to the aQGCs than the tri-boson processes [63][64][65][66][67].At the muon collider, the gauge bosons in final states of VBS processes are associated with very energetic muons or neutrinos in the forward region with respect to the beam.They become the dominant production mode for all SM processes starting from a few TeV energies due to the logarithmic enhancement from gauge boson radiation, which also holds for many new physics models [58,60].On the other hand, the tri-boson processes are normally suppressed by 1/s in the s-channel propagator.In spite of this, it is expected that the tri-boson processes without forward objects carrying away energies can also be sensitive to the aQGCs induced by dimension-8 operators with four field strength tensors.This is especially true for the tri-photon production where there are no subsequent decays in final states.This channel was utilized to probe the axion-like particle at muon colliders through µ + µ − → Z * /γ * → γa(→ γγ) [68,69].
In this paper, we study the sensitivity of the tri-photon process to the neutral anomalous couplings γγγZ/γ and the relevant dimension-8 effective operators at muon colliders.The rest of this paper is organized as follows.In Sec. 2, the dimension-8 operators contributing to aQGCs are introduced.We also discuss the tri-photon process and compare with the VBS.The numerical results are presented in Sec. 3. In Sec. 4 we summarize our conclusions.
2 Anomalous quartic gauge couplings and tri-photon process

A brief introduction of the anomalous quartic gauge couplings
The dimension-8 operators frequently used in previous works can be classified as scalar/longitudinal operators O S i , mixed transverse and longitudinal operators O M i and transverse operators where f S i , f M i and f T i are dimensionless coefficients, and Λ is the cut-off scale.O S i and O M i operators are irrelevant for the study of tri-photon channel in this paper.The relevant O T i operators with four field strength tensors are [5,6] Table 1.The existing LHC constraints on the O Ti coefficients obtained at 95% CL.
where W ≡ i W i σ i /2 with σ i being the Pauli matrices.The constraints on the coefficients of these operators were obtained at the LHC, and listed in Table 1.These existing bounds are at the level of f T i /Λ 4 ≃ O(0.1) TeV −4 .Note that the real unitarity analysis was not performed when obtaining the constraints on coefficients in Table 1.Instead, Refs.
[50, 51] provided the maximally allowed energy scales when the coefficients are equal to the constraints in Table 1.With unitarity taken into account, according to our previous studies [10,12,71], the LHC constraints should become weaker.

The annihilation process compared with the VBS processes
At the muon collider, the annihilation and VBS processes used to study the aQGCs are where f and f ′ are the outgoing muons or neutrinos, and V i denotes SM gauge bosons.An important difference between the muon collider and the pp collider is that the annihilation process at pp collider must be led by sea quark partons in initial states.Nevertheless, before starting to study the tri-photon process, we need to compare the annihilation with the VBS process at the muon collider.When a new particle X is produced, there is a relative scaling [59] σ where σ BSM VBF and σ BSM ann are the cross-sections of VBF and annihilation processes with X in the final states, m V,X are the masses of intermediate vector boson and the new particle X, respectively.Well above the threshold for √ s ≫ m V , the initial muons substantially radiate massive EW gauge bosons under an approximately unbroken SM gauge symmetry.This leads to the above double-logarithmic enhancement log 2 s m 2 V and thus the VBF process will dominate over the annihilation at high energies.
To demonstrate that the tri-boson processes can archive competitive sensitivity to aQGCs at the muon collider, we consider the process µ + µ − → γγν ν for illustration below.The typical Feynman diagrams of the aQGC contribution to this process are shown in Fig. 1.For illustration, we only involve O T 5 as an example in which both VBS and triboson contributions are present.The VBS contribution in Fig. 1  effective vector boson approximation [72][73][74].
where s W is the sine of the weak mixing angle θ W , √ ŝ = √ ξ 1 ξ 2 s is the center of mass (c.m.) energy of W + W − → γγ with √ s being the c.m. energy of µ + µ − , and µ f is the factorization scale set to be √ ŝ/4 [74].The cross-section σ triboson of the tri-boson diagram in Fig. 1 (b) is estimated as σ(µ + µ − → Zγγ) × Br(Z → ν ν), and Br(Z → ν ν) is taken as 20% [75] in the numerical calculation.At the leading order of s, the analytical cross-sections are given by (2.6) The numerical results can be obtained by using the MadGraph5_aMC@NLO toolkit [76,77].The cross-sections as a function of c.m. energy √ s with f T 5 /Λ 4 = 0.5 TeV −4 are shown in the left panel of Fig. 2.Although the VBS process overwhelms for larger √ s, one can see that the tri-boson process is of the same order of amplitude as the VBS process.This is because the momentum dependence in the Feynman rule of the Zγγγ or ZZγγ vertex σ VBS and σ triboson are analytically calculated using Eq.(2.6).The numbers demonstrate that the cut m ν ν > (<) 100 GeV would keep most of VBS (tri-boson) contribution.The small discrepancy should be attributed to the effective vector approximation for VBS and the kinematic cuts in the Monte-Carlo simulation.Despite this discrimination, there exists inevitable interference between the two kinds of processes.

The contribution of aQGCs to the tri-photon process
In fact, most of signal channels induced by aQGCs are composed of both VBS and tri-boson production topologies with the decay of massive EW gauge bosons.However, unlike the process µ + µ − → γγν ν, it is interesting to notice that the tri-photon production µ + µ − → γγγ has only one topology induced by one single electroweak vertex.Moreover, the photons in final states have no subsequent decays and thus the production rate would be enhanced compared with the production of massive EW gauge bosons.As a result, the tri-photon process provides a distinctive channel for the study of anomalous O T i operators.We analyze this tri-photon channel in this subsection.
At tree level, there are only two Feynman diagrams (γ or Z mediated) induced by O T i as shown in Fig. 3 (a).The SM background is from the process µ + µ − → γγγ.There are six Feynman diagrams, one of which is shown in Fig. 3 (b).The other five diagrams can be obtained by permutating the photons in the final states.The total cross-section with aQGCs can be written as where σ SM , σ O T i and σ int are the SM cross-section, the O T i operator induced cross-section and their interference, respectively.For a large √ s, one can ignore the masses of muon and Z boson.The analytical results of σ O T i and σ int are given by with (2.9) Here c W denotes cos(θ W ). By using MadGraph5@NLO, the results of σ SM are listed in Table 2.They are obtained with default basic cuts for photons where p T,γ is the transverse momentum of the photon, η γ is the pseudo-rapidity, and ∆R = ∆ϕ 2 + ∆η 2 with ∆ϕ and ∆η being the difference of azimuth angles and pseudorapidities of the photons.The tri-photon process is particularly more sensitive to O T 8,9 operators which only contribute to neutral aQGCs.For illustration, taking O T 9 and assuming non-vanishing f T 9 /Λ 4 = 0.92 TeV −4 , we obtain σ O T 9 = 0.26 pb which is at least 40 times larger than σ SM at √ s = 3 TeV.Meanwhile, σ int = 0.013 pb.Note that σ int ∝ f T i , if the constraints can be at least two orders of magnitude severer, the interference terms become dominant.
At large s, the validity of the SMEFT should be taken into account.Unlike the case of VBS, the amplitude of the triphoton induced by aQGCs is suppressed by a propagator for large s.A similar case was studied for the diboson production induced by dimension-8 operators contributing to neutral triple gauge couplings (nTGCs) at an e + e − collider through the s-channel [36].It has been found that the unitarity constraint is only relevant at very low luminosity.Therefore, we omit the unitarity issue in this work.

Event selection strategy
We use the MadGraph5_aMC@NLO toolkit to perform the numerical calculation.The basic cuts in Eq. (2.10) are also applied at partonic level.A fast detector simulation is then performed by Delphes [78] with the muon collider card.The analyses of the signal and the background are completed by MLAnalysis [79].To study the kinematic features of the signal and background, the signal events are generated by assuming one operator at a time and the largest coefficients allowed by LHC constraints in Table 1 are adopted.We require at least three photons in the following studies.All the numerical results are presented after this photon number cut (denoted as N γ cut).
The aQGC diagrams in Fig. 3 (a) are through s-channel.For the SM diagram in Fig. 3 (b), one can combine two photons as a virtual particle to form 2 → 2 t-channel diagrams as Fig. 4 so that A ∝ 1/t 1 or A ∝ 1/t 2 , where with p a,b and k 1,2,3 shown in Fig. 3.The other five diagrams can be transformed similarly.By contrast, the aQGC diagrams can only transform to s-channel 2 → 2 diagrams.We expect that the distribution of SM background events is dominantly around small t and thus the transverse momentum of one of the photons is small.Denoting p T,min as the smallest transverse momentum of the three hardest photons and E cm as the c.m. energy, the normalized distributions of p T,min /E cm for the signal of O T 0 and the background are displayed in the left panel of Fig. 5.The distributions for other O T i and different √ s are similar.We thus apply hard p T cut to events, i.e., p T,min /E cm > 0.12.
Another observable sensitive to IR divergence is the Gram determinant ∆ 4 [80] which appears in the denominator of the phase space integration, defined as where The cross-sections after event selection strategy and the cut efficiencies are shown in Table 3.Note that in the event selection strategy, p T,min and ∆ 4 are rescaled to be dimensionless.The cut efficiencies for the contributions of O T i should be the same for different √ s, which is also implied in the distributions shown in Fig. 5. Thus, only the cross-sections at √ s = 3 TeV are shown for O T i in Table 3.We find that the cut efficiency  for our signal becomes ∼ 60%.For the SM background, the cut efficiencies decrease with the growth of √ s, and can be as small as 7.24% at √ s = 30 TeV.Another important background would be the beam induced background, such as the radiated photons tangent to electron trajectories inside the collider.The p T,min and ∆ 4 cuts would be helpful to remove such background.However, Delphes does not include the effects of this background contribution at this stage.A detailed quantitative analysis is deferred to future works.

Significance and the expected constraints
The total cross-sections after event selection strategy (denoted by σ ac aQGC ) are obtained as a function of f T i /Λ 4 by scanning some values of coefficient with one operator at a time, and shown in Fig. 6.With the interference, these total cross-sections σ ac aQGC can be fitted as bilinear functions of f T i , as where ϵ SM and ϵ O T i are the cut efficiencies of SM and aQGC events, respectively, and σint is the interference parameter to be fitted.The numerical results of efficiencies are given in Table 3.The fitted results of σint can then be obtained and are given in Table 4.
The sensitivity of the muon colliders to the aQGC operators and the expected constraints on the coefficients can be obtained with respect to the significance defined as [75,81] where N s = N aQGC − N SM and N bg = N SM .The likelihood function for N s is defined as and one can find the maximum by setting ∂lnL/∂N s = 0.The approximate significance is then obtained by taking the square root of the likelihood ratio statistic −2ln(L(0)/L(N s )) after letting n = N bg + N s [81].The number of events N aQGC is obtained by setting certain high-dimensional coefficient f T i in SMEFT.The number of background events N SM denotes the SM prediction with all high-dimensional coefficients vanishing.The integrated luminosities L for both the "conservative" and "optimistic" cases are considered [59].As a result, we can obtain the projected sensitivity on f T i /Λ 4 by taking 2σ, 3σ or 5σ significance.
The results are shown in Tables 5 and 6.One can be seen that the tri-photon process at the muon colliders is very sensitive to O T i operators, in particular the O T 8,9 operators which are only relevant for the neutral aQGCs.For the muon collider with √ s = 3 TeV and L = 1 ab −1 , the expected constraints at S stat = 2 are about two orders of magnitude stronger than those at the √ s = 13 TeV LHC at 95% CL listed in Table 1.At √ s = 30 TeV and L = 90 ab −1 , the projected constraints would be about six orders of magnitude stronger than the 13 TeV LHC.
Recently, a Snowmass paper investigated the searches of aQGCs through the production of W W boson pairs at a muon collider with √ s = 6 TeV and L = 4 ab −1 [82].They studied the W W νν and W W µµ final states with the W bosons' hadronic decay.We also attempted to measure the aQGCs in W + W − → W + W − scattering at muon collider using artificial neural networks [83].In principle, the tri-photon channel can only be used to constrain O T i operators, whereas O M i ,S i operators can be studied in these VBS W + W − production modes.We obtain the 95% CL constraints on O T 0(1),2 operators as 0.1447 TeV

Summary
The future muon collider provides a unique chance to explore both the high-luminosity frontier and the high-energy frontier of the particle physics.It has been realized that the muon collider is an EW gauge boson collider [59].Analogous to the VBS processes at the LHC, one expects that the VBS at the muon collider is very suitable to probe the aQGCs.Furthermore, we propose that the tri-photon process is also a very sensitive process to the O T i operators contributing to aQGCs at the muon collider.
In this paper, we study the contribution of O T i operators to the tri-photon process at muon colliders.The cross-sections induced by O T i operators are calculated.We also investigate the kinematic features of the signal events from O T i operators, based on which the event selection strategy is discussed.As a result, we obtain the projected sensitivities  on the aQGC coefficients.It turns out that, the tri-photon process is especially sensitive to the O T 8,9 operators which only contribute to the neutral aQGCs.At √ s = 3 TeV and L = 1 ab −1 , the expected constraints on the coefficients of O T 8,9 can be about two orders of magnitude stronger than those at the 13 TeV LHC.At √ s = 30 TeV and L = 90 ab −1 , the projected constraints would be about six orders of magnitude stronger.

Figure 3 .
Figure 3. Feynman diagrams for the process µ + µ − → γγγ.The diagrams induced by O Ti operators are shown in the left panel, and a representative diagram in the SM is shown in the right panel.In the SM, there are other five diagrams by permutating the photons in the final state.

Figure 5 .
Figure 5.The normalized distributions of p T,min /E cm (left panel) and |∆ 4 |/s 4 (right panel) for SM and O Ti after the N γ cut.
p a,b and k 1,2,3 shown in Fig. 3, and t 1,2 in Fig. 4. The IR divergence shows up in the region |∆ 4 | → 0, and thus one expects that the SM events sharply peak at small |∆ 4 |.Since the permutation of photons in the final state is symmetric, in the calculation of ∆ 4 , k 1 is assigned as the momentum of the hardest photon, k 2 as the second hardest, and k 3 as the third hardest.The normalized distributions of |∆ 4 | /s 4 for the signals and the background are shown in the right panel of Fig. 5.We require the events satisfying |∆ 4 | /s 4 > 3 × 10 −5 .

Table 3 .
The cross-sections σ SM and σ O T i (in unit of fb) and cut efficiencies after selection strategy.The results of aQGCs are obtained using the upper bound of the coefficients in Table1.

Table 5 .
The projected sensitivities on the aQGC coefficients at the muon colliders with different c.m. energies and integrated luminosities for "conservative" case.

Table 6 .
The projected sensitivities on the aQGC coefficients at the muon colliders with different c.m. energies and integrated luminosities for "optimistic" case.[45]ATLAS collaboration, Observation of electroweak W ± Z boson pair production in association with two jets in pp collisions at √ s = 13 TeV with the ATLAS detector, Phys.Lett.B 793 (2019) 469 [1812.09740].[46] CMS collaboration, Measurement of electroweak WZ boson production and search for new physics in WZ + two jets events in pp collisions at √ s = 13TeV, Phys.Lett.B 795 (2019) 281 [1901.04060].