Rare top quark decays at a 100 TeV proton-proton collider: $t \rightarrow bWZ$ and $t\rightarrow hc$

We investigate extremely rare top quark decays at a future proton-proton collider with centre-of-mass energy of 100 TeV. We focus on two decay modes: radiative decay with a $Z$ boson, $t \rightarrow b WZ$, and flavour-changing neutral decay with a Higgs boson, $t \rightarrow h c$, the former being kinematically suppressed with a branching ratio of $\mathcal{O}(10^{-6})$, and the latter highly loop-suppresed, with a branching ratio of $\mathcal{O}(10^{-15})$. We find that $t \rightarrow b WZ$ will be very challenging to observe in top quark pair production, even within well-motivated beyond-the-Standard Model scenarios. For the mode $t\rightarrow h c$ we find a stronger sensitivity than that obtained by any future LHC measurement by at least one order of magnitude.


Introduction
The top quark's large mass suggests that it may be intimatelyconnected to the mechanism of electroweak symmetry breaking (EWSB). Furthermore, it is unique as a colour-charged particle because of the fact that it decays before hadronizing. The top quark decays mostly through the channel t → bW with much smaller contributions from other SM processes. The QCD production of top quarks at the Large Hadron Collider (LHC) is expected to be high, with σ (pp → tt) 950 pb at 14 TeV centre-of-mass energy [3], and would be 40 times larger at a proton collider with a centre-of-mass energy of 100 TeV (e.g. the Future Circular hadron-hadron Collider -FCC-hh [4,5]). Given its large production rates, the top quark provides simultaneously interesting tests of QCD (through its production mechanisms) and of electroweak physics (through its decay channels). The Tevatron has already measured several properties of the top following its discovery, and the LHC is already adding further precision measurements. The purpose of the present study is to investigate to which extent a FCC-hh can provide valuable information to our knowledge of the top quark properties and couplings. Here, we focus on the potential for the observability of the radiative SM decay mode, t → bW Z, and on the decay of the top quark through the direct interaction between the top and charm together with the Higgs boson, i.e. t → hc.
The decay of the top quark to a bottom quark, a Z and a W boson, t → bW Z, received some attention 20 years ago, with Send offprint requests to: a e-mail: apapaefs@nikhef.nl b e-mail: gtx@nikhef.nl several studies addressing its rate and potential sensitivity to new physics [1,[6][7][8][9]. A peculiar feature of this process is that it occurs near the kinematical threshold: m t m b + m W + m Z and is thus suppressed within the Standard Model (SM), predicted to be O(10 −6 ). In the studies of Refs. [1,6,8], it was pointed out that one has to take into account the finite widths of the Z and W bosons when calculating this decay mode. Indeed, if the widths are ignored, the current particle data group nominal values [10] of the particles involved: m t 173.1 GeV, m b 4.18 GeV, m Z 91. 19 GeV and m W 80.39 GeV would imply m t < m b + m W + m Z and would suggest a kinematicallyforbidden decay. Nevertheless, if the gauge boson widths are properly taken into account the decay can proceed. This leads to O(10 4 ) top quark pair production events containing the decay during the lifetime of the high-luminosity LHC (HL-LHC) with an integrated luminosity of 3000 fb −1 , inclusively over the decays of the Z and W bosons. Consequently, the process will likely be impossible to observe at the LHC. However, given the substantial increase in cross section at the FCC-hh this channel is expected to yield O(10 6 ) events, justifying a more detailed investigation into its observability. Flavour-changing neutral (FCN) decays of the top quark appear at one loop and have a strong suppression due to the Glashow-Iliopoulos-Maiani (GIM) mechanism and second-third generation mixing [2,[11][12][13][14]. Within the SM, this suppression leads to the following branching ratios: BR(t → γc) ∼ 10 −14 , BR(t → gc) ∼ 10 −12 , BR(t → Zc) ∼ 10 −14 and BR(t → hc) ∼ 10 −15 [2], rendering them unobservable at the current and future colliders. Consequently, the measurement of such processes within current capabilities would clearly signal the presence of BSM phenomena. Here, we focus on the transition t → hc, where the new physics effect is treated as an effec-tive interaction between the top quark, the charm quark and the Higgs boson. This paper is organised as follows: in section 2 we consider theoretical aspects of the t → bW Z decay and construct a simple phenomenological analysis to assess its observability at the FCC-hh within the SM. To complement this analysis, we study the allowed size of an enhancement to the decay rate due to the presence of a charged heavy scalar boson contribution, given current experimental constraints. Subsequently, in section 3, we analyse the decay t → hc, taking into account the effective coupling h-t-c and extract a bound by looking at a clean final state at the FCC-hh, with h → γγ. Finally, we present our conclusions in section 4.
2 Top quark decays to bW Z

Defining the final state
The 'cleanest' channels in which the process t → bW Z contributes are those containing multiple leptons. Here we will focus on the cases t → be + ν e µ + µ − and t → b j jµ + µ − , which receive contributions from other intermediate states in addition to t → bW Z. We examine the observability of these processes by looking at the particle content in the final states only and we do not attempt to separate the t → bW Z contribution. 1 In Ref. [1], the ratio R = BR(t → bµν µ νν)/[BR(W → µν µ )× BR(Z → νν)] was considered as a definition of the process t → bW Z. Taking into account that the top quarks are on-shell, we can use an equivalent definition: .
We have calculated R by using the MG5 aMC@NLO Monte Carlo event generator [15,16]. Within the given errors, the results for R are in good agreement with those for the ratio R appearing in the third column of Table 2 of Ref. [1]. To make a direct comparison, we show the results in Table 1, where we have used the values of the masses and constants given in Table 1 of Ref. [1].

Next-to-leading order corrections
Since the decay process occurs close to the top mass threshold, it is interesting to investigate the impact of next-to-leading order (NLO) QCD corrections. To consider a scenario which might be realistic in a phenomenological study, we examine corrections to the decay process t → be + ν e µ + µ − , with the invariant mass of µ + µ − pair taken to lie above 70 GeV to remove the photon contribution, γ → µ + µ − . We use MG5 aMC@NLO to calculate the SM decay width, Γ , setting m t = 173 GeV and varying the bottom mass between 0.2 and 5.2 GeV. In Figure 1 1 This separation is not possible due to interference of the t → bW Z contribution with other diagrams.  Fig. 1. The decay width of the process t → be + ν e µ + µ − for m(µ + µ − ) > 70 GeV at leading order and next-to-leading order in QCD as a function of the bottom quark mass. The lower inset shows the ratio of NLO to LO.
we show the variation of the decay width as a function of the input bottom mass. In addition, we also present the ratio between the NLO and the leading order (LO) corrections in the lower inset. It is evident that the NLO corrections reduce the branching ratio by about 10%. The impact of increasing the b-quark mass is larger at NLO than at LO, which is due to the phase space becoming even more restricted close to the top mass threshold. Given the null results of the phenomenological analyses (see below), we do not consider higher-order corrections in more detail.

Phenomenological analysis for t → bW Z
We construct a simple phenomenological analysis to determine whether a FCC-hh at 100 TeV will be sensitive to the SM top radiative decay t → bW Z. We focus on the final states that arise through top pair production that contain one "signal" top decaying into 3 leptons and the other top decaying fully hadronically, i.e.
and its charge-conjugate, where j is any light-flavour jet. In addition we also consider the case where one of the top quarks decays to oppositely-charged leptons and the other to only hadronic products: In our analysis, the jet reconstruction is done using the fastjet package [17,18]. We simulate the signal and backgrounds using MG5 aMC@NLO at parton level and the general-purpose Monte Carlo event generator HERWIG 7 [19][20][21][22][23] for the parton shower and non-perturbative effects, such as hadronization and multipleparton interactions.

The three-lepton final state
As a starting point for the analysis of the three-lepton final state: we ask for two opposite-sign same-flavour leptons and one additional lepton, with p T > 20 GeV within a pseudo-rapidity of |η| < 6. Furthermore, we ask that the opposite-sign sameflavour leptons reconstruct the Z mass, m + − ∈ [89, 93] GeV. As before, jets are reconstructed with the anti-k ⊥ algorithm with R = 0.3 and we demand that they satisfy p T > 20 GeV within |η| < 6. Two of these jets are b-jet candidates. Here we assume that b-jets can be tagged with 100% efficiency, so as to give an upper estimate of the sensitivity to the processes under consideration. We then find the best combination of two or three jets that reconstruct the top quark mass by looking at all the combinations of jets. We call this combination the "reconstructed hadronic top" and require m t,reco ∈ [153, 193] GeV. We also ask that the distance ∆ R between 3 the sub-leading b-jet and the reconstructed Z boson is ∆ R < 1.0. Using the assumption that the missing transverse momentum is primarily due to the undetected neutrino from the "signal" top quark decay and using the mass-shell condition m 2 t = (p ν + p + + p + + p − + p b ) 2 we obtain a quadratic equation, and hence two solutions, for the z-component of the neutrino momentum. We use these solutions to reconstruct two corresponding values of the total invariant mass using momenta of the reconstructed top quarks. We ask require both of these values lie within [350, 700] GeV.
To assess the detection prospects of this process we consider the background arising from pp → ttZ, where all three particles are taken to be on-shell in this case. Using the aforementioned basic cuts, and the LO cross sections for the signal, σ signal 3.00×10 −5 pb, and for the background contribution to the final state considered here (i.e. including branching ratios) σ ttZ 0.10 pb, we find an estimate of O(5) events for the signal and O(5000) events for the ttZ at an integrated luminosity of 10 ab −1 , a ballpark estimate of the FCC-hh end-of-lifetime data sample. This implies that this channel will be impossible to observe during the FCC-hh lifetime, and we do not consider it here any further.

The two-lepton final state
To further investigate top quark radiative decays, let us now consider the signal process: As before, we consider only the main background channel: where the pair of muons arises from the decay of an on-shell Z boson. Our selection procedure focuses on the reconstruction of the top quark based on the combination of final state products (b j jµ + µ − ). To reconstruct the Z boson we require two oppositely-charged muons satisfying |η| < 6 such that the combined invariant mass is within the interval [80, 100] GeV. We cluster final state particles, excluding the muons, with p T > 5 GeV into anti-k ⊥ jets of R = 0.3. We require exactly two b-tagged jets. We then recluster the constituents of the R = 0.3 subjets into R = 1.2 "fat jets" using the anti-k ⊥ algorithm, excluding the previously btagged subjets. For each fat jet, we apply a mass drop algorithm as in Ref. [24] with parameters y cut = 1.5 and µ = 0.25. 4 We determine the invariant mass M of the system composed by the pair of muons that reconstruct the Z boson, the lowest p T b-tagged jet and the resulting subjet after the mass-drop application. The event is selected only if there is a fat jet such that 140 GeV < M < 180 GeV after the mass drop conditions are applied. Based on this selection we obtain the cross sections σ signal = 1.4 × 10 −4 pb and σ background = 0.5 pb, for signal and background respectively, corresponding to a significance S/ √ B = 0.6 at an integrated luminosity of 10 ab −1 . Thus, it seems unlikely that this process would be detected in this channel. Fig. 2. The heavy charged Higgs process gb → H − t → (tb)t (left, charge conjugate implied) and VBF to H ± → W ± Z (right) related to the ATLAS experimental analyses at 8 TeV [25,26], used here to extract the limits on the H ± −t − b and H ± −W ± − Z couplings, respectively.

Heavy charged Higgs bosons
The analyses of the previous sections indicate that the SM radiative decay of the top quark seems challenging to detect even at the 100 TeV FCC-hh. Nevertheless, one might ask whether it would be possible to observe an enhanced rate due to beyondthe-SM contributions. One possibility arises with the addition of a heavy charged Higgs boson, H ± that couples to W ± Z as well as top and bottom quarks. Such couplings have been probed in LHC experimental analyses, for example the ATLAS analyses at 8 TeV that appear in Refs. [25,26]. In Ref. [25], the process gb → H − t → (tb)t (and charge conjugate) was searched for by the ATLAS experiment, using data corresponding to an integrated luminosity of 20.3 fb −1 . The diagram contributing to this channel, shown on the left in Fig. 2, has a rate that is directly proportional to the quartic power of the H ± − t − b coupling. Similarly, in Ref. [26], the final state H ± → W ± Z was searched for using an equivalent dataset in vector boson fusion. The latter process, shown on the right in Fig. 2, has a rate proportional to the quartic power of the H ± −W ± − Z coupling. limits on top branching ratio t b + e + (ATLAS, 8 TeV) Fig. 3. The 95% C.L. limits on the decay width of the process t → be + ν e µ + µ − for m(µ + µ − ) > 70 GeV at leading order through a hypothetical heavy charged scalar that couples only to W ± Z and top and bottom quarks. The constraints on the couplings were obtained from the ATLAS experimental analyses at 8 TeV [25,26].
We simulate these two processes at LO using MG5 aMC@NLO and assume that the new scalar only possesses these two interactions. Hence, using the results obtained in the aforementioned articles [25,26], we derive constraints for the maximum and minimum allowed values of the decay width at LO. These are shown in Fig. 3. Evidently the enhancement factor is moderate over the range of scalar boson masses considered, with a maximal value of O(2) for a heavy charged Higgs boson mass of ∼ 200 GeV. Hence we can conclude that the addition of a heavy charged scalar cannot render this process observable at a 100 TeV collider. Note that due to the interference of the SM diagrams with the charged scalar diagrams, which can be negative, the decay width can possess values lower than those of the SM.

Top quark decays to Higgs boson-charm quark 3.1 The h-t-c coupling
We now turn to the investigation of flavour-changing neutral decays of the top quark. We consider gauge-invariant and renormalizable Yukawa interactions of the form where Q i is a left-handed doublet, q j is a right-handed singlet and H is the SM Higgs doublet. We constrain our analysis to real and symmetric couplings: λ h ct = λ h tc = (λ h ct ) † = (λ h tc ) † . Here we focus on the resulting top quark decay t → hc, induced by couplings of the above kind. We also note that these couplings can lead to other interesting final states [27]. Various studies have already examined this process at the LHC [14,27,28], with the current best experimental constraint on BR(t → hc) at the LHC being 0.22% at 95% C.L., coming from AT-LAS 13 TeV data (36.1 fb −1 ) in the di-photon channel [29]. The corresponding best constraint at CMS is currently 0.47% through h → bb decays [31]. Naive extrapolation of the 13 TeV ATLAS result [29] to the high-luminosity LHC demonstrates that an integrated luminosity of 3000 fb −1 implies an ultimate constraint of BR(t → hc) 0.019% through the h → γγ channel alone. 5 It is important to note here that the LHC analyses do not consider the tagging of charm jets in the derivation of these constraints. This implies that these limits are associated with the crucial assumption that the t → hu decay will be either absent or sub-dominant with respect to t → hc. Alternatively, one can use these analyses to impose constraints on t → hu, assuming t → hc is absent or sub-dominant.
In the present study we will analyse the prospect of constraining BR(t → hc) and λ h ct through top quark pair production at the FCC-hh. To the best of our knowledge this represents the first estimate for a constraint on this coupling at the FCC-hh. Among all the decay channels, the one expected to provide the strongest contribution in the combination for the constraint is the one involving the transition h → γγ, and therefore we will focus on it in the present study. In our analysis we consider both the scenario with and that without charm-jet tagging, with values for the tagging efficiencies motivated by current LHC considerations [32]. We have implemented the interaction described by Eq. (3.1) in a UFO [33] model which we interface to MG5 aMC@NLO to generate signal pp → tt → (hc)t (and the charge-conjugate process) events. We also generate parton-level events for the backgrounds using MG5 aMC@NLO and perform shower and hadronization using HERWIG 7 as before. The background processes considered include those that include a Higgs boson in association with other particles: pp → tth, pp → h j jW ± and those that contain non-resonant di-photon production: pp → ttγγ, pp → γγ j jW ± , where the W bosons were decayed to electrons or muons. Generation-level cuts were applied on the non-resonant photon samples: the photon transverse momentum was required to lie in p T,γ > 10 GeV, the distance between either a jet and a photon or between two photons ∆ R(γ, j or γ) > 0.1 and the invariant mass of the two photons to satisfy M γγ ∈ [110, 140] GeV. In all background samples we asked for the generation-level cuts on the jets and final-state leptons of p T > 20 GeV. The jets have been merged to the HERWIG 7 parton shower at tree level using the MLM method via the FxFx add-on module [34]. 6 Our analysis is divided into two cases, either with the "nonsignal" top decaying hadronically, (t → b j j) or semi-leptonically (t → b ν). The starting cross sections for both the hadronic and semi-leptonic final states, with the generation-level cuts described above, are given in the second column of Table 2. For the signal cross section we use a working value of λ h ct = 0.1. To take into account higher-order effects and for the sake of simplicity, we have rescaled the leading-order cross sections for all processes by a k-factor of 2. This approximation does not have a significant impact on our derived constraints and can be fully addressed in a future analysis.

Phenomenological analysis for t → hc
We identify photons and leptons by requiring p T > 25 GeV, within |η| < 2.5 in both cases. We assume flat identification efficiencies of b-jets of 70% and of c-jets of 20% and ask for them to have p T > 25 GeV and lie within |η| < 2.5 GeV. 7 We consider mis-tagging rates for light jets to b-jets of 1%, and to c-jets of 0.5%. The rate of mis-identification of b-jets to cjets was taken to be 12.5% and the rate for the converse was taken to be 10% [32]. We do not consider mis-tagging of light jets to photons in our analysis. We do not apply any detector effects such as momentum smearing and we assume that jets, leptons and photons are detected with 100% efficiency within the considered coverage. 8 In all cases, we ask for exactly one identified b-jet and at least two photons. For the semi-leptonic top case we ask for at least one lepton. We reconstruct the signal top quark from the identified b-jets and di-photon system and ask for the mass to lie within m γγc ∈ [160, 190] GeV. Furthermore, we ask for the di-photon invariant mass to reconstruct the Higgs boson mass within 2 GeV: m γγ ∈ [123, 127] GeV, the distance be-tween the photons to lie within ∆ R(γ, γ) ∈ [1.8, 5.0] and the distance between the di-photon system and the c-jet to lie within ∆ R(γγ, c) < 1.8. In the case of no charm-jet tagging we simply consider all non-b-jets candidates as light. In practice, this amounts to not having a specific tagging weight when considering "true" charm jets. In the semi-leptonic top case we assume that the missing transverse energy is entirely due to the missing neutrino and reconstruct its z-component by solving the quadratic equation m 2 W = (p + p ν ) 2 , where p and p ν represent the 4-momenta of the hardest lepton and the missing neutrino respectively. Here, we take m W = 80.4 GeV. We then ask for one of the two solutions to reconstruct the top mass when   Table 4. The expected signal and background events at an integrated luminosity of L = 10 ab −1 after applying the analyses in the search for the t → hc decay. The resulting event yields are shown for the case where charm-jet tagging is considered for the hadronic and semileptonic cases, see main text for further details. As before, the signal cross sections are shown for the working value λ h ct = 0.1.
We summarise the main features of the analysis in Table 3. The resulting event yields after applying the analyses are shown  Table 5. As for Table 4, but without charm-jet tagging.
in Tables 4 and 5 for the cases with and without charm tagging, respectively, at an integrated luminosity of L = 10 ab −1 .

Constraints for t → hc
To take into account the effect of the presence of systematic uncertainties, we assume that they only affect the total number of background events, B, by inducing a systematic uncertainty ∆ B = αB, with α ≥ 0 parameterising the effect. We add this in quadrature to the statistical uncertainty on the expected number of events. We therefore show results for values of α corresponding to no systematics (α = 0) to demonstrate the ultimate precision at the future collider, low systematic uncertainty (α = 0.05), and high systematic uncertainty (α = 0.2). For the ATLAS analysis of [29] we have deduced that the current systematic uncertainty would correspond to α 0.063 and we derive results for an extrapolation to the high-luminosity LHC data set (3000 fb −1 ) either using this value or setting α = 0 as the best-case scenario.
We show the resulting constraints on the percent branching ratio in Fig. 4. The values for 95% C.L. upper limits are also given in Table 6. Both the analyses with and without charm-jet tagging are able to provide constraints on the branching ratio of O(10 −3 )%. A naive statistical combination of the hadronic and semi-leptonic channels yields 95% C.L. upper limits on BR(t → hc) of (8.5 × 10 −4 )% for the case with charm-jet tagging and (4.4 × 10 −4 )% without for the case of no systematics (α = 0). 9 For the cases α = (0.05, 0.2) the combination yields: BR(t → hc) of (1.8, 6.1) × 10 −3 )% for the case with charmjet tagging and (2.8, 11.1) × 10 −3 )% without, respectively. We also show the extrapolation of the ATLAS constraints of [29], which is at least an order of magnitude worse than the results of the present analysis for comparable systematics, corresponding to BR(t → hc) of (1.9, 9.7) × 10 −2 % for α = (0, 0.063) respectively. 10  going from darker to lighter-shaded, respectively. The blue bars represent the cases with charm-jet tagging applied, whereas the red bars represent the cases without it. The darker grey bar represents a naive statistical extrapolation of the ATLAS constraints that appear in [29] to the full high-luminosity LHC data set (3000 fb −1 ) in the absence of systematic uncertainties (α = 0) and the lighter-shaded grey area roughly takes into account the current estimate of the systematic uncertainties, corresponding to α 0.063.
with c-tagging: analysis: hadr., semi-lept.  Table 6. The 95% C.L. upper limits as calculated by each of the phenomenological analyses of this article for a 100 TeV FCC-hh with an integrated luminosity of 10 ab −1 corresponding to the systematic uncertainty parameter values α = (0, 0.05, 0.2) as described in the main text. We give the limits on the branching ratios for the top quark as a percentage of the total as well as the associated values of the coupling, λ h ct . The results for BR(t → hc) are also given graphically in Fig 4.

Conclusions
We have investigated the rare top quark decay processes t → bW Z and t → hc at a future circular hadron collider running at 100 TeV with 10 ab −1 of integrated luminosity. We have demonstrated that it will be extremely challenging to observe a final state in which the t → bW Z process contributes. This is true even in the case of the presence of new physics contributions allowed by current LHC constraints. On the other hand, the t → hc decay can be constrained to O(10 −3 )%, either with or without considering charm-jet tagging. This estimate is an order of magnitude more stringent than a high-luminosity LHC extrapolation and will allow us to constrain the off-diagonal top quark-charm quark Yukawa couplings to λ h ct ∼ O(10 −3 ). The extremely rare decay modes we have investigated in the present article constitute two of the many interesting ones for top quarks. A future high-energy collider will be able to provide information on other processes, such as t → cWW , t → qγ, t → qZ, t → cγγ and t → cZZ. We leave investigations of such modes to future work.