Probing the CP Structure of the Top Quark Yukawa at the Future Muon Collider

We study the top-Higgs coupling with a CP violating phase $\xi$ at a future multi-TeV muon collider. We focus on processes that are directly sensitive to the top quark Yukawa coupling: $t\bar{t}h$, $tbh\mu\nu$, and $t\bar{t}h\nu\bar{\nu}$ with $h\rightarrow b\bar{b}$ and semileptonic top decays. At different energies, different processes dominate the cross section, providing complementary information. At and above an energy of $\mathcal{O}(10)$ TeV, vector boson fusion processes dominate. As we show, in the Standard Model there is destructive interference in the vector boson fusion processes $t\bar{t}h\nu\bar{\nu}$ and $tbh\mu\nu$ between the top quark Yukawa and Higgs-gauge boson couplings. A CP-violating phase changes this interference, and the cross section measurement is very sensitive to the size of the CP-violating angle. Although we find that the cross sections are measured to $\mathcal{O}(50\%)$ statistical uncertainty at $1\sigma$, a 10 and 30 TeV muon collider can bound the CP-violating angle $|\xi|\lesssim9.0^\circ$ and $|\xi|\lesssim5.4^\circ$, respectively. However, cross section measurements are insensitive to the sign of the CP-violating angle. To determine that the coupling is truly CP violating, observables sensitive to CP-violation must be measured. We find in the $t\bar{t}h$ process the azimuthal angle between the $t+\bar{t}$ plane and the initial state muon+Higgs plane shows good discrimination for $\xi=\pm0.1\pi$. For the $tbh\mu\nu$ and $t\bar{t}h\nu\bar{\nu}$ processes, the operator proportional to $\left(\vec{p}_\mu\times\vec{p}_h\right)\cdot \vec{p}_t$ is sensitive to the sign of CP phase $\xi$. From these observables, we construct asymmetry parameters that show good distinction between different values and signs of the CP violating angle.


Introduction
One of the major goals of the Large Hadron Collider (LHC) and future collider programs is to measure the properties of the Higgs boson [1][2][3][4].Since the top quark is the heaviest particle in the Standard Model (SM) and has the strongest coupling to the Higgs, processes involving the top quark Yukawa coupling provide us an ideal place to study the Higgs properties.There has been much interest in directly probing the top Yukawa coupling at colliders.This is typically done by measuring the Higgs produced in association with a top and anti-top (t th).In the h → b b channel, this was the first direct measurement of top-Higgs Yukawa interactions [5,6], constraining the Yukawa coupling magnitude to be within about 10% of the SM prediction.The high luminosity LHC (HL-LHC) is projected to measure the top quark Yukawa to within 3.4% [1,7] of the SM value.When combined with the HL-LHC results, future electron positron colliders such as Higgs factories with energies of 240 − 500 GeV [2,8,9] are expected to measure the top quark Yukawa with a precision of ∼ 3% [1].Electron positron colliders with higher energies of O(1 TeV) and 100 TeV proton-proton machines will have a precision of ∼ 1 − 2% [1,2].
Of particular interest is the charge conjugation and parity (CP) properties of the Higgs boson.Higgs spin and CP properties are measured to be compatible with J PC = 0 ++ by the Higgs and vector bosons interactions (hV V ) at the LHC [10,11].However, the Higgs may still be an admixture of CP even and odd scalars, and different CP phases can appear in the Higgs-gauge boson couplings and the Higgs-fermion couplings.In this paper we study the CP properties of the top quark Yukawa coupling at a future muon collider.We parameterize the ht t interaction Lagrangian with a CP violating coupling via the relevant SMEFT [12,13] dimension-6 operator [14][15][16][17][18] L htt,SMEFT = −y t QL Φt R − c t Φ † Φ QL Φt R + h.c., where y t and c t are allowed to be complex, Q L = (t L , b L ) T is the left-handed quark doublet, and is the Higgs doublet.The Higgs boson is denoted by h and v = 246 GeV is the Higgs vacuum expectation value (vev).A typical parameterization of the CP violating top Yukawa is through a CP violating phase ξ.The SMEFT coupling constants can be rewritten via the identifications Re(y t ) = − m t √ 2 v (κ htt cos ξ − 3) , Re(c t ) = √ 2m t v 3 (κ htt cos ξ − 1) , Using these, Eq. (1.1) becomes 1L htt,SMEFT = −m t tt − g htt h t (cos ξ + i γ 5 sin ξ) t with where m t is the top quark mass.When ξ = 0 (±π/2), Higgs couples with the top as a SM CP even scalar (CP odd pseudoscalar).The Yukawa coupling is CP violating in other cases.The magnitude of the top Yukawa coupling κ htt can be bound by rate measurements of t th.Due to the strong projected constraints at the HL-LHC we will assume κ htt = 1.
Electron and neutron electric dipole moment (EDM) bounds strongly constrain the CP odd component of Higgs-top quark Yukawa coupling to be |ξ| ≲ 0.08 • [19][20][21][22] at 95% CL.However, these results rely on the Higgs coupling to first generation fermions being SM like.Due to the very small couplings, it is extremely difficult to directly verify SM like couplings between the Higgs and electron and first generation quarks.Allowing the Higgs couplings to first generation fermions to float drastically reduces the EDM bounds on the CP odd component of the top quark Yukawa coupling and the collider bounds are dominant [21,22].That is, without confirmation that the Higgs couplings to first generation fermions are indeed SM-like, direct probes of the CP structure of the top quark Yukawa are needed.
In this work, we explore the CP violating h tt coupling via processes directly dependent on the top quark Yukawa at a future muon collider: µ + µ − → t th, µ + µ − → t thν ν and µ + µ − → tbhµν2 .We consider collider energies of √ s = 1, 3, 10, and 30 TeV.A multi-TeV muon collider has been of interest for a long time [53][54][55][56], and there has been much interest recently [57][58][59][60][61].Despite the technological challenges under investigation [56,60,[62][63][64][65], the muon collider has the merit of both the high energy reach of a hadron collider and the cleanliness in background of the electron-positron collider.Direct measurements of the top Yukawa in t th production at a multi-TeV muon collider are projected to have an uncertainty of 35 − 50% [66].
At multi-TeV energies and above, vector boson fusion (VBF) becomes the dominant production mode for many processes at muon colliders [57,62,67,68].In many SM processes, there is strong destructive interference in longitudinal VBF to guarantee unitarity.Hence, small variations away from the predicted SM couplings can cause rates to grow very quickly with energy.Hence, rate measurements of VBF processes can provide strong bounds on new physics.Indeed, by studying VBF production of top pairs µ + µ − → t tν ν with an off-shell Higgs, the top Yukawa can be indirectly measured to a precision of 6% [57,69].A recent study [70] finds that a 10 TeV muon collider with an integrated luminosity of 10 ab −1 could probe the top Yukawa coupling with a precision surpassing 1.5%, more than one order of magnitude better than the precision from t thν ν channel at muon colliders.
We investigate the CP structure of the top Yukawa at a muon collider in a couple ways.First, we study numerical and semi-analytic arguments about strong destructive interference in the SM calculation for the cross section of the VBF processes µ + µ − → t thν ν, tbhµν and show that the CP violating angle can make this constructive interference.However, the total cross section measurement is CP even with a CP violating angle.Hence, to determine that a measured rate not in agreement with SM predictions is indeed due to a CP violating top Yukawa, we study observables that are genuinely sensitive to CP violation.We then perform a full collider study for semileptonic top quark decays in t th, t thν ν and hadronic top decay in tbhµν, with h → b b.It should be noted that all these processes have the same final states and hence must be considered together.Although we project a measurement of the total cross section with O(50%) 1σ uncertainties at muon collider energies of ≳ 10 TeV, we obtain very strong constraints on the CP violating angle from pure rate measurements.
The paper is organized as follows.In Sec. 2 we study the event rates of t th, tbhµν and t thν ν in the SM and then explore the full −π ≤ ξ ≤ π region of each process.We then study observables and construct asymmetries that can discriminate the magnitude and sign of ξ, i.e. observables that are genuinely sensitive to CP violation.In Sec. 3 we present the collider analysis with h → b b and semileptonic decays.The luminosity needed for discovery and exclusion of different ξ are reported, as well as the expected exclusion at design luminosities.Finally, in Sec. 4 we conclude.

Top-Higgs Production at a muon collider
In Figs. 1, 2, and 3 representative Feynman diagrams are shown for the production of µ − µ + → t th, t bhµ − ν, and t thν ν, respectively.The production of t th is through s-channel photon and Z diagrams, whereas production of t thν ν and t bhµ − ν proceed   coupling V − V − h − h or Higgs trilinear coupling h − h − h where one Higgs splits into a t t pair.
2.1 Pure CP even (ξ = 0) and CP odd (ξ = ±π/2) cases We show in Fig. 4 the SM (ξ = 0) cross section as a function of center of momentum energy, √ s, for (blue) µ + µ − → t th, (green) µ + µ − → t thν ν ≡ t thν ℓ νℓ (ℓ = e, µ and τ ), and with √ s from 500 GeV to 30 TeV.The conjugate processes t bhµ − ν and tbhµ + ν are combined.The cross sections are generated with MadGraph5 aMC@NLO [71].The total cross section is shown as solid lines and the VBF subprocess contributions are shown  separately in dashed lines for tbhµν and t thν ν.To insure a gauge invariant subset of diagrams, the VBF subprocesses are isolated by replacing the initial state µ + with an e + as adopted in Ref. [67].Even though the attraction of a muon collider is to have a multi-TeV lepton collider, we consider √ s ≳ 0.5 TeV to gauge our understanding and reach at lower energy.We terminate the plot at √ s = 30 TeV since for larger energies the electroweak (EW) Sudakov logarithms in the VBF diagrams with massive vector bosons become large and the effective vector boson approximation is needed for a reliable calculation of rates [67,[72][73][74].For tbhµν production, the t-channel type diagrams can also be realized by a photon exchange.The massless photon mediator in this case would cause a singularity when a final state muon is collinear with an initial state muon.A proper treatment also involves the effective vector boson approximation [72][73][74][75][76].We impose a cut on the transverse momentum of the outgoing muon p µ T > 10 GeV to regulate the singularity and avoid the complexity of the effective vector boson approximation.The t th process has only s-channel contributions.Therefore, its cross section decreases as the energy of the collider increases past the mass threshold, from about 2.0 fb at √ s = 1 TeV to 8 × 10 In the following discussion, we will choose √ s = 1, 3, 10 and 30 TeV as the benchmark energies.Table 1 shows the benchmark luminosities and expected number of events at the benchmark energies for pure CP even (ξ = 0) and CP odd (ξ = ±π/2) cases.These rates are estimated using 100,000 events generated in MadGraph5 aMC@NLO [71].The signal model is implemented using FeynRules [77,78].
For energies at and below 10 TeV, the benchmark luminosities are set by the minimal instantaneous luminosity needed to generate 10,000 events for a process with 1 fb cross section within 5 years at √ s = 10 TeV [56]: This is the number of events needed for percent level statistical uncertainty.For collider energies above 10 TeV, we consider the luminosity benchmarks based on the conservative designs of Table 1 of Ref. [57].
For tbhµν and t thν ν, at √ s = 10 and 30 TeV we can expect 10 3 events for the CP even state and 10 3 − 10 5 events for the CP odd state.What is particularly striking is at 10 and 30 TeV, the pure CP odd case has significantly more events than the pure CP even case in the tbhµν and t thν ν channels.We will discuss this phenomena in depth below.

CP admixtures
Figure 5 shows the cross sections for t th, t thν ν and tbhµν as functions of the CP violating phase at (a) 1 TeV, (b) 3 TeV, (c) 10 TeV, and (d) 30 TeV. Green and red dashed lines represent the VBF contributions for t thν ν and tbhµν.The s-channel and VBF style diagrams have different dependencies on ξ.As can be seen, for s-channel processes the SM (ξ = 0) has the maximum cross section, while for processes dominated by VBF the wrong sign Yukawa ξ = ±π cases have the maximum cross section.Also, the periodicity of the cross section dependence on ξ is different for s-channel and VBF contributions.Finally, as the center of momentum energy of the collider increases the cross section dependence on ξ becomes very strong for the VBF dominant processes.
To understand the cross section dependence on ξ, we take an semi-analytical approach.First we discuss the t th production.Since the total cross section is CP even, it can only depend on cos 2 ξ, sin 2 ξ, cos ξ, and a piece independent of ξ.The terms proportional to cos 2 ξ and sin 2 ξ originate from the modulus square of diagrams where the Higgs is radiated off a top quark, such as Fig. 1(a).The terms independent of ξ are from the square of diagrams where the Higgs is radiated off an internal Z-boson3 , such as Fig. 1(b).Finally, the terms proportional to cos ξ are due to interference between diagrams where the Higgs is radiated off of a top quark or Z-boson.Hence, for t th production we use the general parameterization, 3) The numerical results of the coefficients C i tth (i = 0, 1, 2) at √ s = 1, 3, 10 and 30 TeV are summarized in Table 2.All results in Tab. 2 are found by scanning over ξ in MadGraph5 aMC@NLO [71] with 50,000 events at each point.The stability fit is checked by also fitting with additional higher powers to cos ξ and insuring the fit does not change.The dominance of the cos 2 ξ and constant terms explain the overall shape of the t th cross section dependence on ξ, which has an approximate π/2 periodicity.As the collider energy increases the constant term C 0 tth becomes the largest coefficient decreasing the cross section dependence on ξ.The non-zero value of the cos ξ coefficient explains the small asymmetry between ξ = 0 and ξ = ±π, which slightly breaks the π/2 periodicity and reveals the weak interference between the diagrams with htt and hZZ couplings in Fig. 1

(a) and (b), respectively
For tbhµν production, there are still at most single insertions of the top quark Yukawa.Hence, we can use a similar parameterization as in the t th production cross section: Unlike the t th, there are also diagrams where the Higgs is radiated off a b-quark or a W -boson. Together with radiation off the Z-boson, these interactions contribute to the interference term C 1 tbhµν and the constant term C 0 tbhµν .The numerical results are also shown in Table 2.At √ s = 1 TeV, the coefficients C 2 tbhµν and C 0 tbhµν are largest.At this energy the s-channel diagrams dominate, hence the ξ dependence is similar to t th.At higher energies, the VBF diagrams dominate and the interference term C 1 tbhµν becomes more important.Indeed, it can be seen in the SM case (ξ = 0) there is strong destructive interference between the top Yukawa and other Higgs couplings.Due to the smallness of the bottom quark mass, the major contribution to this interference is between the top and massive vector boson couplings with the Higgs.When we move away from the SM limit to the wrong sign Yukawa case (ξ = ±π), the strong destructive interference becomes constructive interference.The cross sections at √ s = 30 TeV are σ tbhµν (ξ = 0) = 0.31 fb and σ tbhµν (ξ = ±π) = 3.5 fb. (2.5) Cross sections at ξ = ±π are an order of magnitude larger than those at ξ = 0.This explains why at high energy the tbhµν rate depends strongly on ξ.It should be noted, similar interference effects between the top and massive vector boson couplings to the Higgs are seen in single t+h production at the LHC [23][24][25][26][27][28].
The cross section dependence on ξ for t thν ν is different than the dependence of the t th and tbhµν cross sections.There are now diagrams with intermediate Higgs bosons that have double insertions of the top Yukawa, as seen in Figs.3(b,e).Additionally, some of these diagrams depend on the coupling between the Higgs and massive vector bosons, potentially complicating the interference story.Keeping in mind that the total cross section is CP even, the double insertion of the top Yukawa introduces new dependence on ξ.The modulus square of the diagrams with two top Yukawas contribute cos 4 ξ, sin 4 ξ, cos 2 ξ sin 2 ξ.There are also diagrams with a single 4-point h − h − t − t coupling from the dimension-6 operator as shown in Fig. 3(f).However, the dependence of the four point top-Higgs coupling on ξ is similar to that as the top Yukawa, i.e. ∝ cos ξ + iγ 5 sin ξ − 1, as can be seen in Eq. (1.4).Hence, this four point coupling does not introduce new dependencies on ξ in the cross section.The interference between diagrams with two top Yukawas and other processes then contribute to cos 3 ξ, cos ξ sin 2 ξ, cos 2 ξ, and sin 2 ξ.We parameterize the ξ dependence as It should be noted, that unlike t th and tbhµν, the t thν ν process also depends on the Higgs trilinear coupling h − h − h and the quartic coupling between the Higgs boson and massive vector boson h The numerical values of the coefficients C i tthνν are given in Table 2.At all energies under consideration, there is destructive interference.The destructive interference becomes very strong at high energies where the numerical values of C 1 tthνν and C 0 tthνν are the largest.At √ s = 30 TeV, the SM cross section is The wrong sign Yukawa cross section is The ξ = ±π cross section is two orders of magnitude larger than the ξ = 0 cross section.This ξ dependence explains why at 10 and 30 TeV, the t thν ν cross section has the strongest dependence on ξ (see Fig. 5).It is clear that a pure rate measurement of top+Higgs production at a high energy muon collider is very sensitive to non-zero ξ.
It is worth noting that at hadron colliders a direct Higgs-top coupling measurement is from gg(q q) → t th production and all the contributed diagrams depend on ξ at leading order.Hence, the total t th cross section at hadron colliders is symmetric between ξ = 0 versus ξ = ±π.As discussed earlier, the rate measurement of single top+Higgs production at hadron colliders is sensitive to the difference between the SM (ξ = 0) and wrong sign Yukawa (ξ = ±π) cases [23][24][25][26][27][28].However, this rate is quite small making it difficult to directly measure small values ξ in this process in the current LHC run.This is unlike the case at lepton colliders with √ s ≳ O(10 TeV), where we can easily disentangle the ξ = 0 and ξ ̸ = 0 just from rate measurements.We will quantify this in Sec. 3.

CP violating observables
The previous section demonstrated that at high energies the cross sections of t thν ν and tbhµν depend very strongly on the CP violating phase ξ.Hence, a pure rate measurement could be very sensitive to the value of ξ.However, the total cross section is CP even.Hence, observing a deviation away from SM predictions would not demonstrate a CP violating top Yukawa.To determine that there is indeed CP violation, we must measure observables that are sensitive to this CP violation, which we now investigate.Additionally, as we will see in Sec. 3, rate measurements at a muon collider with √ s = 3 TeV are insensitive to ξ.Hence, at that energy CP violating observables may be necessary to discover direct evidence of a non-zero CP violating angle in the top-Higgs interaction.For that reason, while the observables under consideration are general, many of the results of this section are presented for √ s = 3 TeV.

t th
We first consider the t th process.For this process the t t is considered as a single subsystem.The total three-momentum of the t t subsystem is where ⃗ p t is the top quark momentum and ⃗ pt the anti-top quark momentum.The t t subsystem defines one plane.Another plane can be created between the Higgs and initial state muon.A CP odd observable can be created by considering the angle between these planes: TeV for CP violating angle (solid) ξ = 0.25π and (dashed) ξ = −0.25π.This is evaluated in the collider center of momentum system.and √ s = 3 TeV in the t th center of momentum frame.The distribution of ξ = 0.25π peaks at positive ϕ while ξ = −0.25πpeaks at negative ϕ.That is, there is a clear separation between ξ = 0.25π and ξ = −0.25πand this observable is sensitive to the sign of the CP violating phase.
Based on this observation, we define an asymmetry parameter between the regions of ϕ > 0 and ϕ < 0: In Fig. 6(b) we show the results of this asymmetry for ξ = ±0.1π,±0.25π and at energies √ s = 1, 3, 10, and 30 TeV.Again, there is a clear separation with negative CP violating angles having negative asymmetry and positive CP violating angles having positive asymmetry.Additionally, this symmetry increases between √ s = 1 and 3 TeV, and slightly decreases again at 30 TeV.To find a rough estimate on uncertainties, the benchmark points in Table 1 are used and the statistical uncertainty on the asymmetry is estimated as ∼ 1/ √ N .This assumes that the number of forward and backward events are roughly equal.We could expect the uncertainty of the asymmetry to be of the order ∼ 0.05 at 1 − 10 TeV and ∼ 0.1 at 30 TeV.

tbhµν and t thν ν
We now turn to investigating observables sensitive to CP violation in t thν ν and tbhµν production.First, we consider tbhµν.This process consists of t bhµ − ν and its CP conjugate tbhµ + ν.We construct triple products between an initial state muon and final state products, which are odd under parity transformations [79][80][81]83].For t bhµ − ν we use the initial state muon and final state top quark and Higgs boson: and for the conjugate process tbhµ + ν we use the initial state µ + and final state t and Higgs boson We show the differential distributions of O µht in Fig. 7 and O µht in Fig. 8(a) for √ s = 3 and ξ = ±0.25π.The observables are evaluated in the lab frame.There is a clear separation for positive and negative CP violating angle ξ, with positive (negative) ξ preferring positive (negative) O µht and O µht .Hence, these observables are sensitive to CP violation.Since these observables are parity odd but even under charge conjugation, the distributions for the charge conjugated processes t bhµ − ν and tbhµ + ν are the same.For the t thν ν process, we can also construct the observable O µht , whose differential distribution is shown in Fig. 8(a).Similarly to t bhµ − ν, positive ξ peak at positive O µht while negative ξ peak at negative O µht .However, at √ s = 3 TeV, the distribution is flatter for t thν ν than it is for t bhµ − ν.

Based on these observations, we define an asymmetry parameter
where and similarly for σ(O µht < 0).The results of this asymmetry for √ s = 1, 3, 10, and 30 TeV are shown in Fig. 8(b) for both ξ = ±0.25πand both t bhµ − ν and t thν ν.There is a clear separation between different signs of the CP violating angle, indicating that this asymmetry is indeed sensitive to CP violation in the top Yukawa.Interestingly, at √ s = 1 an 3 TeV for t bµ − ν, the CP violating angle ξ and the asymmetry A O µht have the same sign, while at √ s = 10 and 30 TeV they have opposite sign.For t thν ν, the sign of the asymmetry and CP violating angle are the same for all energies.
Based on the expected number of events in Table 1, at √ s = 1 TeV we could expect ∼ 0.1 − 0.3 uncertainty on a measurement of A O µht .Hence, a measurement of the asymmetry A ϕ in the tth system may be more sensitive to CP violation.This is unsurprising, since t th is the dominant production mode at √ s = 1 TeV.At √ s = 3 TeV, the expected number of events between processes is more comparable, and the uncertainties on A O µht and A ϕ are expected to be similar: ∼ 0.05−0.1.At √ s = 10 and 30 TeV, the tbhµν and t thν ν are dominant.Ignoring acceptances and branching ratios, the statistical uncertainty on A O µht could be expected to be percent or subpercent level, much more accurate than A ϕ .Hence, measuring these processes at different energies could provide complementary information on the CP violating angles.
The above estimates of uncertainties are rough and only give an order of magnitude estimation.Additionally, they do not take into account any realistic cuts needed to separate signal from background.As shown in Ref. [66] and as we will show below, after a collider analysis the uncertainty on a measurement of the cross section of t th, tbhµν, and t thν ν is expected to be as large as O(50%) at collider energies ≳ 10 TeV.This indicates that in a realistic scenario, the uncertainty of the asymmetry parameters could be expected to be O(1).These are ballpark estimates to give an idea of the size of the expected statistical uncertainties.A dedicated collider analysis of the CP violating observables would be needed to determine the expected uncertainty to higher accuracy.

Collider Analysis and Sensitivity Projections
We now present our collider analysis of the t th, t thν ν, and tbhµν signals at a muon collider.To maximize rates we consider h → b b.In order to balance the need for rates and a clean signal, we study the semi-leptonic decay of the tops t → 2 j +b and t → bℓν.Hence, our signal consists of two bs, two bs, two jets, a lepton ℓ, and missing transverse energy / E T from the neutrinos: where ℓ = e, µ.Since all signal processes result in the same final state, we consider them simultaneously.Signal and background events are simulated in MadGraph5 aMC@NLO [71], where we use new HELAS [84,85] routines to more efficiently simulate the signal.The signal model is implemented using FeynRules [77,78].
The irreducible SM background processes are In the first line, a gluon is radiated from a b b (t t) pair and then splits into t t (b b).The t tb b (EW) background is the inclusive production of t tb b through EW processes, with no QCD contributions.Finally, t tb bν ν is inclusive of all such processes.For the t tb b (EW) and t tb bν ν the underlying s-channel, t-channel, and/or VBF processes are not easily disentangled and cannot be due to the necessity of gauge invariance.Other backgrounds such as 2b2 bjjℓ ± + / E T , t t(g → q q), or t t(g → gg) are expected to be negligible, with requiring four b-tagged jets.We assume a nominal b-tagging efficiency of ϵ b = 0.9 in our study with a percent level rate of light quark and gluon jets being mistagged as b-jets. 4t should be noted that the t tb b (EW) and t tb bν ν background include Feynman diagrams that depend on the top Yukawa couplings and include signal diagrams.To insure that the estimates of backgrounds do not include the signals in a gauge invariant way, we generate the backgrounds with the bottom quark Yukawa set to zero.Additionally, these backgrounds in principle will depend on the CP structure of the top Yukawa.The EW t tb b background is insensitive to ξ.However, at 10 and 30 TeV the t tb bν ν background could increase upwards by a factor of five between ξ = 0 and ξ = ±π/2, and by a factor of 10 between ξ = 0 and ξ = ±π.This could be an indirect probe of the CP structure of the top Yukawa.However, in this paper we are interested in direct probes.Additionally, the importance of the background dependence on ξ would depend significantly on if an analysis is performed using simulated or data driven estimates of the background.All backgrounds that are used for our estimates are calculated for ξ = 0.
For a more realistic analysis, we simulate the detector effects by performing a Gaussian smearing of jet energies with energy resolution For a multi-TeV lepton collider, the expected jet energy resolution can reach 3.5% to 4% especially for high energy jets [88][89][90].We adopt 10% as a more conservative study.
Our results show that even with a pessimistic energy resolution we can effectively distinguish signal and background.All results presented in this section are after jet energy smearing.
For the acceptance cuts for jets and leptons, we require them to be isolated from each other and have minimum transverse momentum, p T , [89,91]: where ∆R mn = (∆ϕ mn ) 2 + (∆η mn ) 2 is the angular distance between particles m and n with ∆ϕ mn being the difference in azimuthal angles and ∆η mn the rapidities.Additionally, since our signal consists of neutrinos which appear as missing energy, we require a minimum missing transverse energy / E T > 50 GeV. (3.5) Muon colliders have significant beam induced backgrounds.These backgrounds arise due to the finite lifetime of the muon.Hence, they decay while in the beam producing a significant amount of charged particles along the beam direction.To shield the detector from the beam induced backgrounds, detector proposals include shielding nozzles [57,59,60,64,65,92].These are expected to be within 10 • of the beamline [57,63,70].We therefore apply maximum rapidity cuts on all reconstructed particles as well: In Table 3 we provide signal and background cross sections for muon collider energies of √ s = 1, 3, 10, and 30 TeV and CP violating angles ξ = 0, 0.1 π, and 0.2 π.
These cross sections are generated from 50,000 events at the generator level. 5After acceptance cuts and for ξ = 0 the signal and background are comparable at √ s = 1, and  The cut on the Higgs reconstructed mass m recon h are those in Eq. (3.8).Dashes indicate negligible cross sections.We assume a nominal b-tagging efficiency of ϵ b = 0.9 in our study but it is not applied to cross sections in this table.
3 TeV.At √ s = 10 and 30 TeV, the t thν ν signal begins to dominate the background while tbhµν is relatively suppressed despite having a comparable cross section before acceptance cuts as shown in Table 1 and Fig. (5).The t th rate is greatly suppressed after acceptance cuts.These effects can be understood with the distributions in Fig. 9.
In (a) we show the minimum angular separation among all particles at √ s = 30 TeV for t th.As can be clearly seen, the minimum rapidity is quite small, below the minimum required separation in Eq. (3.4).At such high center of momentum energy, the t th cross section is dominated by a t t topology with a Higgs radiated collinear with a parent top quark.Since the Higgs and top are collinear, the decay products are collimated and fail the separation cuts in Eq. (3.4).In Fig. 9(b) we show the rapidity of the final state muon in t bhµ − ν.The muon is central for √ s = 1 and 3 TeV, but very far backward for √ s = 10 and 30 TeV.At such high energies, this process is dominated by t-channel and VBF type processes.Since the mass of vector bosons are small compared to the total energy, there are collinear tchannel enhancements and the VBF process is dominated by collinear emissions off the initial state µ − .Hence, the initial and final state µ − s prefer to be in the same direction, causing the final state µ − to be far backward in the detector 6 .There is a similar effect for tbhµ + ν where the µ + is very far forward at √ s = 10 and 30 TeV.Hence, at these high energies, the tbhµν process largely fails the rapidity cuts in Eq. (3.6).The precise shape of these distributions depend on the cut on the final state muon transverse momentum, since that cut places a minimum on the muon rapidity.However, the conclusions are unchanged with different transverse momentum cuts.In these cases the signal cross section could be enhanced by tagging a forward muon.GeV [93].The width of these peaks are dominated by the jet energy smearing due to the extreme smallness of the Higgs width Γ = 4.1 MeV [94].Once all combinations of bs and bs are considered, the gbb/gtt and t tb bν ν backgrounds produce a continuous spectrum, while the EW production of t tb b backgrounds have Z-pole peaks.Even after jet energy smearing, the signal Higgs peak is separated from the background Z peaks.At the detector level, it is a priori unknown which jets originate from the top quark and Higgs.Hence, to use the Higgs mass for separation of signal and background, proper event reconstruction is needed.To resolve the combinatorics in identifying the parent particles of the jets, we find the combination of jets that minimize the following χ 2 : where σ t = 5 GeV, σ W = 5 GeV, σ p T,t = 100 GeV, and σ H = 5 GeV.The χ 2 is constructed to find the combinations of particles that reconstruct the W masses, top quark masses, and Higgs mass most accurately.That is, determine the decay products of each particle.Since the base topology is t t production with a radiated Higgs, so we expect the t and t to have similar transverse momenta.However, this is very rough so the relative uncertainty on the | − → p T | term is significantly larger than the mass reconstruction terms.Note that in the case of t thν ν production, there are two additional neutrinos from VBF processes, adding to the missing transverse momentum and therefore making the full reconstruction of the final state more challenging.However, Eq. (3.7) works well for our purposes to disentangle the combinatorics of the problem.Also, while it is possible that a jet from a W -decay is misidentified as a b-jet, we expect the mistagging rate of light quark jets to be percent level.Since we require exactly four b-tagged jets, the effects of the misidentification of a jet from a W -decay is negligible.
The above minimization procedure of − → p ν aims to find approximate three momentum of the neutrino which originates from the leptonic top quark decay without the missing transverse momentum constraint (see Refs. [95][96][97] for similar reconstruction method using χ 2 ), and recent review [98] on general methods for kinematic reconstruction).
Once the b-jets originating from the Higgs are identified, we impose a cut on reconstructed Higgs invariant mass m recon h around the Higgs mass m h = 125 GeV: As shown in Table 3, this cut does not noticeably change the total signal cross section, but greatly decreases the background.For ξ = 0, at √ s = 1 TeV the signal to background ratio increases from 1.9 to 3.9, at 3 TeV from 0.8 to 4.0 , at 10 TeV from 2.2 to 6.4, and at 30 TeV from 2.6 to 8.8.That is, the reconstructed Higgs mass cut can increase the signal to background ratio by a factor of two to five.The reconstructed Higgs mass cut has a small effect on the signal for ξ = 0.1π and ξ = 0.2π as well.Hence, it increases the signal to background ratio for all ξ.

Significance and sensitivity to the CP violating angle
We now present the main results of this paper.First, we calculate the required luminosity for 2σ exclusion and 5σ discovery at different CP violating angles.The discovery Projected 1σ (2σ) bounds on |ξ| In Fig. 11(a) we show the required luminosity of 5σ discovery and 2σ exclusion as a function of the CP violating angle ξ for √ s = 1, 3, 10, and 30 TeV.We have included the b-tagging efficiencies of ε b = 0.9 for both signal and background.At √ s = 1 TeV, 110 fb −1 is needed to obtain 2σ sensitivity to the SM rate.For ξ ̸ = 0 the required luminosity increases.In this case, as shown in Fig. 5, the cross section at √ s = 1 TeV is maximum at ξ = 0, and, hence, requires less luminosity to observe that ξ ̸ = 0.For √ s = 3 TeV, the SM cross section is more difficult to discover or exclude than ξ ̸ = 0. Indeed, a luminosity of 2.1 ab −1 is needed for 2σ evidence of the SM, a factor of two higher than the benchmark luminosity.This is due to the VBF diagrams beginning to make a significant contribution to the rate, where the cross sections peak at ξ ̸ = 0.The effect at 10 and 30 TeV, where VBF typologies dominate, is even more striking.Luminosities of 4.8 and 3 ab −1 are needed for 2σ evidence of the SM point at 10 and 30 TeV, respectively.
In Fig. 11(b), we show the projected 2σ constraints on the total cross section normalized to the SM value at the benchmark luminosities.The total cross section dependence on ξ is overlaid.To determine the 2σ constraints, we assume that the observations agree with the SM signal plus background.Then we introduce a signal strength µ = σ/σ SM and find bounds on µ according to the likelihood: where S SM are the expected number of SM signal events.For the total cross section dependence on ξ, at √ s = 1 and 3 TeV, the signal cross sections for t th, t thν ν, and tbhµν are added together.At √ s = 10 and 30 TeV, as Table 3 shows, the only signal that makes a significant contribution after the analysis is t thν ν.Hence, at these energies, we only use the t thν ν signal to show the dependence on ξ.These results consider only statistic uncertainties, but we have checked that systematic uncertainties of 5 − 10% have little effect on our conclusions by adding them in quadrature with the statistical uncertainties.At 2σ, O(1) bounds can be placed on a SM-like cross section at √ s = 1 and 10 TeV.At √ s = 3 TeV, we find that only an upper bound the signal cross section can be placed.At 30 TeV the prospects are more optimistic than other energies with projected bounds on the order of 60 − 80%.
In Table 4 we report the projected upper bounds on |ξ| for our benchmark energies and luminosities for a variety of potential b-tagging efficiencies of ε b = 0.6, 0.7, 0.8 and 0.9 [86,87], which are obtained via rescaling the signal and background cross sections by ϵ 4 b corresponding to our requirement of exactly four b-tagged jets.As discussed previously, this requirement is to decrease the backgrounds from light quarks and gluons.At 1 TeV, the cross section measurement translates to an 1σ exclusion of 125 • > |ξ| > 57 • and no exclusion at 2σ for our benchmark ε b = 0.9.At 3 TeV, the cross section measurements are insensitive to the CP violating angle and place no bounds on ξ.As discussed previously, this is due to the VBF and s-channel processes being of similar strengths [Fig.5].Since the s-channel peaks at ξ = 0 and the VBF processes peak at ξ = ±π, when they are summed together the total cross is largely independent of ξ.The results at 10 and 30 TeV are striking.At these energies, the cross section is strongly dependent on the CP violating angle ξ.Hence, even a cross section measurement with an O(50%) uncertainty can exclude quite small ξ.For our benchmark b-tagging efficiency of ε b = 0.9, we find that a 30 TeV (10 TeV) muon collider can exclude |ξ| > 5.4 • (|ξ| > 9.0 • ) at 1σ and |ξ| > 8.0 • (|ξ| > 14 • ) at 2σ.As the b-tagging efficiency decreases, the projected constraints on ξ weaken.

Conclusions
In this paper we presented a comprehensive study of direct probes of the CP properties of the top quark Yukawa coupling at a future muon collider.While there are many indirect probes of this coupling, direct probes are needed to provide unambiguous sensitivity to the top-Higgs coupling.As such, we studied the t th, t thν ν, and tbhµν production.As we showed, and is well-known [57,62,67,68], the s-channel process t th is the dominant cross section at O(1 TeV) scale muon colliders, while the VBF type processes dominate at ≳ O(10 TeV) energies.
The cross over between s-channel and VBF dominance is particularly interesting for constraining a CP violating angle.We have shown that s-channel and VBF processes have different dependencies on this angle.Hence, measurements at different collider energies can provide complementary information about the CP structure of the top-Higgs interaction.Although our total cross section measurement including t th, t thν ν, and tbhµν has O(50 − 100%) uncertainties at the benchmark luminosities in Tab. 1, the constraints on the CP violating angle can be quite strong.At 1σ, rate measurements can constrain |ξ| < 57 • or |ξ| > 125 • at √ s = 1 TeV, |ξ| < 9 • at √ s 10 TeV, and |ξ| < 5.4 • at √ s = 30 TeV.These constraints favorably compare to direct measurements at proposed future high energy hadron colliders [48,50].As we showed in Sec. 2, these strong constraints in VBF-style processes come from strong destructive interference between diagrams involving the top quark Yukawa and Higgs-vector boson coupling.Hence, the additional CP violating coupling alters this destructive interference which enhances the sensitivity to ξ.At √ s = 3 TeV, the rate measurement does not meaningfully constrain the CP violating angle.At this energy, the s-channel and VBF style processes have comparable cross sections.Since these processes exhibit different dependencies on ξ, the addition of all contributing signal rates flattens out the dependence of the total cross section on the CP violating angle.
At √ s = 3 TeV, measuring observables directly sensitive to CP violation may provide better sensitivity to the CP violating angle than a rate measurement.Even at √ s = 1, 10, and 30 TeV where rate measurements are sensitive to ξ, the rates are still CP even.Hence, even if a first indication of a non-zero ξ may appear in the t th, t thν ν, and tbhµν rates, CP violating observables are needed to verify that any excess was originating from genuine CP violation.In Sec. 2, we explored many different CP violating observables.For s-channel t th production, the azimuthal angle between the Higgs+initial state muon plane and t + t plane is quite sensitive to the CP violating angle.Additionally, for the VBF-style processes t thν ν and tbhµν we explored triple products to enhance sensitivity to CP violation.Based on these results, we showed that defining asymmetries with these observables are indeed sensitive to CP violation in the top quark Yukawa.Never-the-less, observation of a non-zero asymmetry would definitively test the CP structure of the top quark Yukawa.While the collider analysis assumed all signal processes to be indistinguishable, the analysis of CP violating observables separated the processes.As shown in Tab. 3, after cuts a different processes dominate at different energies.This property could be used to determine the most promising CP-violating observable to use.Additionally, the t th, tbhµν, and t thν ν have different properties that could be used to distinguish them.The VBF-style processes, t thν ν and tbhµν, will have more missing energy than t th.At high energies, tbhµν has a very forward or backward muon while t th and t thν ν do not.These properties could be used to distinguish processes and guide an analysis of observable sensitive to CP-violation.However, a dedicated collider study of the CP violating observables would be needed to determine how well this can be accomplished.

Figure 1 .
Figure 1.Feynman diagrams for µ + µ − → t th production (a) with and (b) without a top quark Yukawa.Blue dots indicate the top Yukawa couplings.

Figure 3 .
Figure 3. Representative Feynman diagrams for µ + µ − → t thν ν production for s-channel type diagrams with (a) one and (b) two top quark Yukawas; VBF type diagrams with (c) zero, (d) one, and (e) two top quark Yukawas and (f) one four point top-Higgs interaction from the dimension 6 operator.Green dots indicate the top-Higgs couplings.In addition to these diagrams, there are diagrams involving 4-point interactions with W + W − hh and ZZhh and the Higgs trilinear coupling hhh (not shown).

Figure 4 .
Figure 4.The SM (ξ = 0) cross section for (blue) µ − µ + → t th, (red) tbhµν and (green) t thν ν processes as a function of √ s.Solid lines are the total cross sections and dashed lines are the VBF contributions.

Figure 5 .
Figure 5.The cross section as a function of the CP violating phase ξ for (blue) µ + µ − → t th, (green) t thν ν, and (red) tbhµν at (a) 1 TeV, (b) 3 TeV, (c) 10 TeV, and (d) 30 TeV.A cut of p µ T > 10 GeV is applied for tbhµν.Solid lines indicate total cross section while dashed lines are the VBF contribution.

Figure 6 .
Figure 6.(a) Azimuthal angle (ϕ) distribution for t th production.The angle is between the t + t and h + µ − planes as defined in Eq. (2.10).(b) The forward backward asymmetry of ϕ.These are evaluated in the center of momentum frame of the t th system.

Figure 10 .
Figure 10.Normalized b b invariant mass distributions for (blue, green, red) signal and (yellow, maroon, cyan) background before acceptance cuts, after jet energy smearing, and with ξ = 0.The signal histograms use truth level h → b b and backgrounds bin all six possible b and b combinations on an event-by-event basis.

3. 1
Event Reconstruction and Signal/Background Separation The signal consists of a b b pair originating from a Higgs boson.Hence, placing cuts on the reconstructed Higgs invariant mass can further separate signal and background.In Fig. 10 we show the normalized b b invariant mass distributions for √ s = 10 TeV.For signal we use the truth level h → b b decays, while for background we bin all possible bottom quark pairs.The signals have a clear peak at M b b = m h = 125

Table 1 .
Target 5-year integrated luminosities and expected number of events for t th, tbhµν and t thν ν in the pure CP even (ξ = 0) and pure CP odd (ξ = ±π/2) state.tbhµνhave both s-channel and VBF contributions.When the s-channel contributions are dominant, the cross sections decrease similarly to t th.Once the VBF contributions are dominant, there is an EW Sudakov log enhancement that causes the cross sections to increase at high energy.For t thν ν, the VBF contributions become dominant at √ s ≳ 3 TeV, while it is from √ s ≳ 5 TeV that tbhµν receives contribution mainly from VBF subprocesses.

Table 4 .
The 1σ projected bounds on |ξ| at √ s = 1, 10 and 30 TeV for the benchmark luminosities in Table1and a variety of b-tagging efficiencies ε b .The projected 2σ bounds are in parenthesis.At 3 TeV, there are no bounds on ξ from cross section measurements.Stars indicate no bounds on ξ.