Search for the t → ch decay at hadron colliders

We study the observability for the flavor-changing decay of a top quark t → ch at the Large Hadron Collider (LHC) and future hadron colliders, namely, High-Luminosity LHC (HL-LHC), High-Energy LHC (HE-LHC) and Future Circular hadron-hadron Collider (FCC-hh). Two scenarios in which the Higgs boson could decay: into a quark bottom pair (bb-channel) and two photons (γγ-channel) are analyzed. A Monte Carlo analysis of the signal and the Standard Model (SM) background is computed. Center-of-mass energies of s=14,27\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \sqrt{s}=14,27 $$\end{document} and 100 TeV and integrated luminosities from 0.3 to 30 ab−1 are explored. The theoretical framework adopted in this work is the Type-III Two-Higgs Doublet Model (THDM-III) for which, constraints on the parameter space from the Higgs boson coupling modifiers κi are presented and used in order to evaluate the branching ratio of the t → ch decay and the pp→tt¯,t→ch\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \left( pp\to t\overline{t},\ t\to ch\right) $$\end{document} production cross section. We find that with the integrated luminosity achieved at the LHC, the t → ch decay is out of the reach of detection. More promising results emerge for the HL-LHC, HE-LHC and FCC-hh in which potential discoveries could be claimed.


Introduction
The SM is the most successful model to explain almost all the experimental data nowadays. However, despite its great success it is well known that it does not offer adequate answers to some questions such as it does not propose a candidate for dark matter, does not incorporate gravitational interaction, does not give an adequate solution to the hierarchy problem, etc. In particular, in the SM Flavor Changing Neutral Currents (FCNC) mediated by the Higgs boson are not induced at tree-level. The branching ratio for the t → ch decay in the context of the SM at one-loop level is of the order of 10 −15 [1][2][3] which is far from being detected with the current sensitivity of the LHC. However, several models predict the existence of FCNC at the tree level [4][5][6][7][8][9] and predict branching ratios of up to 10 −3 , which opens the possibility that experiments carried out at the LHC or future hadron colliders can be done with high expectation for a detection, namely:

JHEP07(2019)041
• High-Luminosity Large Hadron Collider [10]. The HL-LHC is a new stage of the LHC starting about 2026 to a center-of-mass energy of 14 TeV. The upgrade aims at increasing the integrated luminosity by a factor of ten (3 ab −1 , ∼ year 2035) with respect to the final stage of the LHC (300 fb −1 ).
• High-Energy Large Hadron Collider [11]. The HE-LHC is a possible future project at CERN. The HE-LHC will be a 27 TeV pp collider being developed for the 100 TeV Future Circular Collider. This project is designed to reach up to 12 ab −1 which opens a large window for new physics research.
• Future Circular hadron-hadron Collider [12]. The FCC-hh is a future 100 TeV pp hadron collider which will be able to discover rare processes, new interactions up to masses of around 30 TeV and search for a possible substructure of the quarks. Because the great energy and collision rate, billions of Higgs bosons and trillions of top quarks will be produced, this is an unbeatable opportunity to search for the t → ch decay. The FCC-hh will reach up to an integrated luminosity of 30 ab −1 in its final stage.
On the other hand, the ATLAS and CMS collaborations [13,14] searched for the t → qh decay, with q = u, c, in the h → γγ and h → bb channels at 7, 8 and 13 TeV, nevertheless they did not found an excess above the background of the SM. The current upper limits for the t → ch decay by ATLAS collaboration at √ s = 13 TeV corresponding to an integrated luminosity of 36.1 fb −1 are given by: B(t → uh) < 0.19%, while the CMS collaboration at √ s = 13 TeV corresponding to an integrated luminosity of 35.9 fb −1 imposes less restrictive limits given by: In theoretical aspect, the prediction of extension models is in the range of O(10 −6 ) − O(10 −3 ) [6,[15][16][17][18][19][20][21]. As far as the simulation is concerned, the authors of ref. [22] proposed a strategy for the search for t → ch at the LHC in the framework of the general Two-Higgs Doublet Model which predicts a B(t → ch) ∼ O(10 −3 ) by using a value for the coupling htc = √ m t m c /v ∼ 0.006, the Cheng-Sher ansatz [23].
In our work, we study the potential discovery about the t → ch decay within the framework of the Type-III Two-Higgs Doublet Model with four-zero textures (THDM-III). We study h → bb and h → γγ channels that could appear in collisions as pp → tt → W b + ch → νb + cXX (with X = b for the bb−channel and X = γ for the γγ-channel ) at hadron colliders.
The organization of our work is as follows. In section 2 we discuss generalities of the THDM-III including the Yukawa interaction Lagrangian written in terms of mass eigenstates as well as the diagonalization of the mass matrix. Section 3 is devoted to the JHEP07(2019)041 constraints on the relevant model parameter space whose values will be used in our analysis. The section 4 is focused on analysis of pp → tt → W b + ch → νb + cXX production cross sections at the LHC, HL-LHC, HE-LHC and FCC-hh. We also present the Monte Carlo analysis of our signal and its SM main background. Finally, conclusions and outlook are presented in section 5.

Theoretical framework
In this section, we give the theoretical framework on which we rely for our research, i.e. THDM-III with a four-zero texture. We analyze the Yukawa Lagrangian of the THDM-III and obtain the Feynman rules involved in our calculations.

Yukawa Lagrangian
The Yukawa Lagrangian in the THDM-III is given by [4] Here Φ i (i = 1, 2) denotes the Higgs doublets and Y f i stands for 3 × 3 Yukawa matrices. In the Yukawa Lagrangian both Higgs doublets can be coupled to all fermions, so that we would get two Yukawa terms for each doublet. The physical particles are obtained through a rotation depending on mixing angle α, which relates the real part of the Φ i doublets with the neutral physical Higgs bosons as follows: whereas the mixing angle β transforms the imaginary part of the Φ i doublets to the charged and neutral Higgs bosons in the following way:

JHEP07(2019)041
with the angle β given by: After of the spontaneous symmetry breaking, mass matrices are defined by: The physical fermion masses are obtained by rotating the matrices of the eq. (2.7) by a biunitary transformation V f = O f P f . Then, the diagonalized mass matrices can be written as: whereM f are the diagonalized matrices whose elements are the fermion masses, i.e., . V f diagonalizes the mass matrices, although not necessarily it diagonalizes each one Yukawa matrices, which are denoted byỸ f i , with i = 1, 2. Therefore, neutral flavor violating Higgs-fermion interactions will be induced. The explicit form of both O f and P f matrices can be consulted in the appendix A.1. On the other hand, the mass eigenstates for fermions can be obtained in the following way: Once the eqs. (2.2)-(2.6) and (2.9) are introduced in the eq. (2.1), the htc coupling acquires a very simple form [24,25]: ch. (2.10) The complete Yukawa Lagrangian is shown in the appendix A.2. We observe that eq. (2.10) includes FCNC at tree-level. In order to suppress them, we assume that the Yukawa matrices of the eq. (2.7) have the form of an hermitian four-zero texture, i.e., whose elements have the hierarchy: Given the structure of the Yukawa matrices as above, the mass matrix inherits its form, so that: (2.12)

JHEP07(2019)041
The elements of a real matrix of the type (2.12) are related to eigenvalues m i , (i = 1, 2, 3), through the following invariants: where we omit the index f , as of now, so as not to overload the notation. We assume the hierarchy m 3 > A > m 2 > m 1 , with A = m 3 − γm 2 and γ in the interval [0, 1]. From these expressions we find a relation between the components of the four-zero matrix mass and the mass eigenstates, namely: By considering the eqs. (2.8) and (2.11)-(2.14), the terms Ỹ f 2 ij of the eq. (2.10) can be written as: i.e., the Cheng-Sher ansatz multiplied by a term depending on Yukawa matrix elements and phases coming from eqs. (A.1) and (A.2). In particular, we have: In this work, instead of constraining the parameters that come from the explicit form of Yukawa matrices, we restrict the χ tc parameter as a whole.

Model parameter space
In order to evaluate the decay width and the (pp → tt, t → hc) production cross section, it is necessary to have current bounds on the model parameters involved in our calculation. These free model parameters are the following: • χ tc .

JHEP07(2019)041
Parameter The best fit value Table 1. The best fit values and ±1σ uncertainties for κ V .

Constraint on c αβ
To constrain c αβ , we use the most up-to-date constraints on the Higgs boson data reported by CMS collaboration [26]: • The Higgs boson coupling modifiers κ j which, for a production cross section or a decay mode j, are defined as: Effects of new physics arise through σ j and Γ j . Because the hV V coupling coming from THDM-III (g THDM-III The table 1 shows the most up-to-date values for κ V reported by CMS Collaboration [26]. In the figure 1 is presented the allowed region by κ V in the s αβ -κ W (Z) planes. The graphics were obtained through the SpaceMath package [27]. To 2σ, κ W and κ Z impose a low limit for s αβ ∼ 0.8, however, by considering 1σ uncertainties for κ W , its lowest limit (s αβ ∼ 0.93) is more restrictive than κ Z (s αβ ∼ 0.86). We note that in the special case when s αβ = 1, then κ V = 1 and the SM is recovered. Because h is identified with the SM-like Higgs boson, to have a consistent theoretical framework with the SM, we consider s αβ = 0.99, which implies that c αβ ∼ 0.14. These results are in Parameter Values accordance with the analysis reported in the ref. [28], in which (α − β) ∼ π/2 it is the most favorable scenario.

Constraint on t β and χ tc
In addition to c α , also t β and χ tc are free parameters. To constraint them, we consider the direct upper bound on the B(t → tc) < 0.16% imposed by ATLAS collaboration [13], however, with this upper bound a very weak bounds on t β and χ tc are obtained. Nevertheless, the authors of the ref. [29] have obtained a better upper limit than ATLAS, extrapolating the number of events for the signal and backgrounds from 36.1 fb −1 to 3000 fb −1 , assuming that the experimental details and analysis remain unchanged. This upper limit is given by In the figure 2 is presented the allowed region in the t β − χ tc plane by the direct upper bound on the B(t → tc) < 0.16% and by extrapolation B(t → tc) < 0.00769%. Considering the limit by ATLAS, the allowed values for χ tc are in the range from −8 to 8 once the t β ∼ 4, whereas for t β ≤ 1.5, χ tc decreases. On the other hand, if the extrapolation is applied, there will be a more restrictive scenario contemplating values for χ tc in the range from ∼ −2 to ∼ 2 for 1.5 ≤ t β . In order to getỸ u 2 ∼ Cheng-Sher ansatz, values for χ tc between 0.5 − 1.5 are considered, corresponding to values for t β in the (0 − 1) interval. However, the authors of [30] proposed a ansatz modified for a scalar-fermion interaction. In summary, the table 2 presents the values for the free model parameters used in this work.

JHEP07(2019)041 4 Search for t → ch decay at hadron colliders
The main interest in this paper is to study an evidence or a possible discovery of the t → ch decay. The theoretical framework adopted to study the signal is the THDM-III. The analysis is carried out for the LHC and future hadron colliders: 1. High-Luminosity LHC [10].
In this work two channels are explored, namely, the Higgs boson decaying into two photons (γγ-channel ) and two bottom quarks (bb-channel ). Then, the signal and the SM main background processes are as follows: • Signal: the signal is pp → tt → hc + W b → XXc + ν b, with X = γ for the γγchannel and X = b for the bb-channel. Then, final state of the signal is γγbj ν or bbbj ν . The flavor-changing process come from one top quark decaying into a charm quark and a Higgs boson through the production mechanism of top quark pairs.

Number of signal events
We now turn to analyze the number of events produced for the signal as a function of t β and χ tc at the LHC and future hadron colliders, i.e., HL-LHC, HE-LHC and FCC-hh. We consider events if and only if they satisfy the constraint B(t → ch) < 10 −5 , i.e., two orders of magnitude less than the upper limit reported by the ATLAS [13] and CMS [14] collaborations and slightly more restrictive than the one reported in ref. [29]. √ s = 100 TeV, respectively. In all cases (a)-(d), the number of events is high when t β as increase as χ tc , which is expected since the htc coupling behaves as ∼ χ tc /t β . For the benchmark points (t β ∼ 0.3, χ tc ∼ 0.5) and (t β ∼ 1, χ tc ∼ 1.3), the number of signal events are 30, 300, 4 × 10 3 and 7 × 10 4 for LHC, HL-LHC, HE-LHC and FCC-hh, respectively. If t β is fixed and χ tc is scanned, the number of signal events increase. Otherwise, if χ tc is fixed and t β is scanned, the number of signal event decreases.

bb channel
As far as bb-channel is concerned, the figure 4 presents the same as in figure 3 though for the bb-channel. As the γγ-channel, the number of signal events of the bb-channel behave very similar. However, because the B(h → bb) ∼ 10 2 · B(h → γγ), the number of signal events increase about two orders of magnitude being 7 × 10 3 , 7 × 10 4 , 8 × 10 5 , 2 × 10 7 for LHC, HL-LHC, HE-LHC and FCC-hh, respectively. The bb-channel gives a great opportunity to detect the signal, as discussed below.

Signal and SM dominant background simulation
Signal events are produced through tt production at the hadron colliders considered, the first top decays into a Higgs boson and a charm quark and, the second one, into a bottom quark, a light charged lepton plus a neutrino via a W gauge boson. In the γγ-channel the Higgs boson decays into two photons and in the bb-channel the Higgs boson decays into two bottom quarks. As far as the computation scheme is concerned, the Feynman rules in the THDM-III were implemented via LanHEP routines [31] for a UFO model [32]. 10 5 parton-level events were generated for the signal and the SM main background using MadGraph5 [33] and perform shower and hadronization with Pythia8 [34]. The CT10 parton distribution function [35] is used. A Higgs boson mass of 125 GeV and a top quark mass of 173 GeV were considered [36]. Afterwards, the kinematic analysis was done via MadAnalysis5 [37]. As far as the jet reconstruction, the jet finding package FastJet [38] and the anti−k T algorithm, with R = 0.4, were used, which are implemented in MadAnalysis5.

Mass reconstruction
γγ-channel. Since the signal comes from (pp → tt, t → ch, h → γγ), the Higgs boson mass was reconstructed as the invariant mass of the diphoton system, M γγ . Events which JHEP07(2019)041 Figure 5. Invariant mass distribution of the diphoton system, M γγ , without cuts. the invariant mass is between 123 − 127 GeV were selected, as we discussed below. The figure 5 shows the invariant mass distribution M γγ without cuts.
bb-channel. In this channel, the signal comes from (pp → tt, t → ch, h → bb), as for the γγ-channel, the Higgs boson mass was reconstructed as the invariant mass, but now for a bb pair, such that |M b 1 b 2 − m h | ≤ 0.15m h . The figure 6 presents the invariant mass distribution, M bb , without cuts.

Kinematic cuts
In order to isolate the signal, the following kinematic cuts were applied.
γγ-channel. For both signal and background events the following kinematic cuts were imposed: • Exactly one b−jet and two photons.
• The invariant mass of the diphoton system, M γγ , is the main variable for search the Higgs boson decay, events between: 123 ≤ M γγ ≤ 127 GeV are acepted.
• Because the Higgs boson decays into two photons, in order to reconstruct the signal top quark from the identified b−jet and the diphoton system, it is required that: 160 ≤ M γγj ≤ 190 GeV.
• The ATLAS collaboration reported in the ref. [39], that the b tagging efficiency ( b ) is ∼ 70%, the probability that a c-jet is mistagged as a b-jet ( c ) is of the order of 10% [40], while the probability that any other jet is mistagged as a b-jet ( j ) is of the order of 1%. Following it, the tagging and mistagging efficiencies considered in this work are as follows: bb-channel. For both signal and background events there should be: • Exactly four jets: three of them are tagged as b−jets with p j, b T >30 GeV and |η j | < 2.5.
• Because in the final state emerge a neutrino, the missing transverse energy (MET) must be MET> 30 GeV.
• In order to reconstruct the top quark mass associated with the FCNC, it is required that |M b 1 b 2 j − m t | ≤ 26 GeV.
• In order to reconstruct the Higgs boson mass as the invariant mass of the bb system, it is imposed that: • It is required that ∆R is between each jet and that charged lepton pair is ∆φ 2 + ∆η 2 > 0.4 • The tagging and mistagging efficiencies are as follows

Evidence and potential discovery
In this section we compute the signal significance defined as S = N S / √ N S + N B , where N S are the number of signal events and N B is the number of SM background events once the kinematic cuts were applied.

γγ-channel
After applying the kinematic cuts shown in section 4.2.2, evidence for the t → ch decay in the γγ-channel with a integrated luminosity of ∼ 3 ab −1 is found. Density plots of the signal significance as a function of t β and χ tc are presented in the figure 7. Three illustrative integrated luminosities which will be achieved at the HL-LHC, namely, L=2, 2.5, 3 ab −1 are considered. It is found a region between 0.6 ≤ t β ≤ 1 and 0.9 ≤ χ tc ≤ 1.3 intervals, with a signal significance 3σ ≤ S, which allows us to claim evidence for t → ch decay. The figure 8 shows the same as in the figure 7 but for the HE-LHC. We found that with an integrated luminosity of ∼ 0.3 ab −1 (300 fb −1 ), evidence for the t → hc decay would be established. However, higher standard deviations may be achieved which range from 7σ (L=3 ab −1 ) to 14σ (L=12 ab −1 ). This collider could be used, among other things, to perform several cross-checks of the discovery of t → ch decay. Finally, the figure 9 presents density plots for the FCC-hh collider. Signal significances of the order of O(30) are found. This means, along with bb-channel, as we will discuss below, an opportunity to secure new physics and focus on finding new sources of physics beyond the SM.

bb-channel
Once the kinematic cuts of the section 4.2.2 are applied, luminosities larger than ∼500 fb −1 are required to achieve a signal significance of ∼ 3σ at the LHC; although the HL-LHC is more promising. The figure 10 shows density plots for the signal significance as a function of t β and χ tc for the HL-LHC by considering three values of the integrated luminosity, L =2, 2.5, 3 ab −1 . The last value is the aim to search at the HL-LHC. Once the integrated luminosity exceeds a value of L ∼2 ab −1 , a evidence for the t → ch decay could be claimed. With a luminosity of least 2.5 ab −1 , a potential discovery looks promising. Finally, when a luminosity of 3 ab −1 is considered, it is the most encouraging scenario with up to ∼ 6σ's for (t β ∼ 0.4, χ tc ∼ 0.5) and (t β ∼ 0.8, χ tc ∼ 0.9). As far as to the HE-LHC and the FCC-hh are concerned, the results are even more promising than for the HL-LHC. The figure 11 and 12 presents density plots as the figure 10, but for the HE-LHC and the FCC-hh. Three  representative scenarios, for both the HE-LHC and the FCC-hh, are explored also, L =3, 7, 12 ab −1 and L =10, 20, 30 ab −1 , respectively. Both colliders could be used to perform a cross-check since, for instance, at the HE-LHC with a minimum integrated luminosity of 0.5 ab −1 discovery of the t → ch decay could be announced. With higher integrated luminosities, for instance, L =12 ab −1 and with (t β ∼ 0.9, χ tc ∼ 1.1), a signal significance of ∼ 18σ is found. On the other hand, at the FCC-hh, signal significances of up to O(90) are searched, with this values, the FCC-hh could work as a FCNC processes factory.
In the table 3 we show a summary of the main results.

Conclusions
We study the t → ch decay at future hadron colliders, namely, HL-LHC, HE-LHC and FCC-hh with center-of-mass energies associated to each hadron collider, i.e,    Table 3. Integrated luminosities for evidence or dicovery of the t → ch decay at hadron colliders.
photons (γγ − channel) and into two bottom quarks (bb − channel). After studying the constraints on the free model parameters from the most up-to-date Higgs boson coupling and applying several kinematic cuts to the signal and SM background, we find that with the integrated luminosity achieved at the LHC, 0.3 ab −1 , is not possible claim discovery for the t → ch decay. However, in the bb − channel, an integrated luminosity of at least ∼ 0.5 ab −1 is necessary to achieve a signal significance of 3σ. On the other hand, with the forthcoming HL-LHC, once it achieves an integrated luminosity of ∼ 2.5 ab −1 (∼ 3 ab −1 ), discovery (evidence) in the bb−channel (γγ −channel) could be claimed. More favorable results emerge for the HE-LHC since with an integrated luminosity of ∼ 0.5 ab −1 (∼ 1.7 ab −1 ), discovery of the t → ch decay in the bb−channel (γγ −channel) will be announced. With these results, several cross-checks, in both channels, could be performed. Finally, the most promising scenario arises at the FCC-hh, which, among other goals, could work as a FCNC factory rediscovering the t → ch decay with a few fb −1 of integrated luminosity in both channels.