Top quark FCNCs in extended Higgs sectors

The large number of top quarks produced at the LHC and possible future hadron colliders allows to study rare decays of this particle. In many well motivated models of new physics, for example in non-minimal composite-Higgs models, the existence of scalar singlets can induce new flavor-violating top decays surpassing the Higgs contribution by orders of magnitude. We study the discovery prospects of rare top decays within such models and develop new search strategies to test these interactions in top pair-produced events at the LHC. We demonstrate that scales as large as 10–50 TeV can be probed. Improvements by factors of ∼1.5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sim 1.5$$\end{document} and ∼3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sim 3$$\end{document} can be obtained at s=27\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sqrt{s} = 27$$\end{document} TeV and s=100\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sqrt{s} = 100$$\end{document} TeV colliders respectively.

A priori, however, there might be other particles lurking around the Electroweak (EW) scale, into which the top quark can possibly decay. Out of the different possibilities, scalar singlets, S, constitute a prime example at the level of dimension-four interactions. When the singlet mixes with the SM-like Higgs boson, its production is strongly constrained owing to the increasingly precise Higgs signal strength measurements (central values reaching unity along with decreasing uncertainties) in various channels [25,26] and also from measurements of the W -boson mass [27][28][29][30][31]. For a relatively small mixing parameter of sin θ varying between 0.2 and 0.35 [32,33], one can however, have a wide range of allowed singlet mass.
Such scalar particles are predicted by some of the best motivated models of new physics, including supersymmetric extensions (e.g., the NMSSM [34]) and Composite Higgs Models (CHM) [35][36][37][38]. Moreover, we will see that these scalars can induce FCNCs significantly larger than those mediated by the SM-like Higgs boson [39]. The reason for the above is three-fold; (i) The top FCNCs mediated by a new scalar singlet are generally suppressed by one less power of the heavy physics scale, (ii) in principle, the scalar singlet can have a larger decay width into cleaner final states, such as + − , bb or γ γ and (iii) in broad classes of models (for example in CHMs), Higgs mediated FCNCs are forbidden in first approximation [8]. Altogether, top FCNCs mediated by new scalars might be well within the reach of the LHC.
Presently, there are no direct limits on t → q S. However, one can have strong constraints from D 0 −D 0 oscillations [24,40,41] which always come about as a product of two S Yukawas, Y ct and Y ut (and also Y uc ). In order to circumvent these constraints, one can always fall back upon scenarios where Y ut is negligibly small. We will argue in Sect. 3, that the ut FCNCs can be vanishingly small compared to their ct counterpart in explicit models.
In this work, we scrutinise the reach of the LHC for top FCNCs in top-pair produced events. We consider the standard leptonic decay of one of the tops, while the other is assumed to decay into Sc, with either S → bb or S → γ γ (leptonic decays will be analysed in a later work). In principle, the current experimental searches for t → hc might be also sensitive to these signals. However, these searches are only optimised for a 125 GeV scalar resonance and not for the whole range of masses, in which S can potentially lie. Moreover, the latest experimental strategies (see e.g. Ref. [42]) rely on trained BDTs which make them hard to recast for arbitrary scalar masses. In light of these issues, we develop new dedicated analyses tailored for each mass point.
The structure of the paper is as follows. In Sect. 2 we outline a model-independent introduction to the signals of interest. We follow this up in Sect. 3, where we discuss concrete realisations, involving both strongly and weakly coupled models of new physics. This allows us to establish wellmotivated benchmark points (BP). We go on to discuss the analysis for the bb channel in Sect. 4 and in Sect. 5, we present the corresponding analysis for the S → γ γ mode for the 14 TeV LHC machine. We finally conclude in Sect. 6, where we also provide an outlook for the high energy colliders by commenting on naive estimations of the reach of hadron colliders at √ s = 27 TeV and √ s = 100 TeV.

Effective Lagrangian
Let us consider a scenario where the SM Higgs sector is extended by a gauge singlet, S, having a mass m S in the EW regime. For low energies, we write the relevant Yukawa Lagrangian as follows 1 is the SM-like Higgs doublet, q L (u R ) denotes the left-handed (right-handed) quarks, Y, Y , andỸ are arbitrary flavour matrices, v ∼ 246 GeV, is the Higgs vacuum expectation value (vev) and f O(TeV) is the new physics scale. In general, the flavour matrices are not aligned, and thus the FCNCs can arise in the EW phase. Among various new physics effects, these induce top flavourviolating decays, viz., t → hc or t → Sc. In general, the latter dominates over the former, because (i) t → hc is further suppressed by an additional factor of 1/ f and (ii) in several UV-complete models, Y and Y are approximately aligned. Finally, after the EWSB, one obtains where m t ∼ 173 GeV is the top mass and g is an O(1) coupling. Such interactions can be tested to a high accuracy through rare top decays. Upon using Eq. 2.2, one obtains the partial width of t → Sc as follows couplings and α is the fine-structure constant. Thus, in the regime, m S m ψ , one obtains at leading order The suppression factor driven by the small electromagnetic coupling can thus be partially compensated upon scaling with the free parameter m S . B(S → γ γ ) can Footnote1 continued is an accidental symmetry S → −S, provided C P is conserved). Moreover, these are proportional to the vev of S, which triggers the mixing with the Higgs boson and are therefore severely constrained by current Higgs measurements [26,30,31]. In addition, the FCNC currents will be further suppressed by the GIM mechanism. Thus, we would expect B(t → Sc) to be several orders of magnitude smaller than B(t → hc), which in the SM is predicted to be smaller than 10 −13 [9]. be significantly larger than the B(h → γ γ ) ∼ 2 × 10 −3 , with the precise value being model dependent.

Explicit models
The Lagrangian in Eq. 2.1 appears naturally in several UVcomplete models, for example in CHMs. In these classes of models, H and S are pseudo Nambu-Goldstone Bosons (pNGBs) arising in a new global symmetry breaking G/H at a scale ∼ f . A prime example is the CHM based on the coset SO(6)/SO(5) [44], which is the smallest one that admits four-dimensional UV completion [45]. The generators of this coset can be chosen as Among these, X 16 − X 46 expand the coset space of the Higgs doublet, while the broken generator associated to S is provided by X 56 . The SM fermions do not couple directly to the (fully composite) Higgs. Instead, the latter couples to composite fermionic resonances, which in turn mix with the SM fermions, thus explicitly breaking the global symmetry. The Yukawa Lagrangian depends therefore on the quantum numbers of the aforementioned fermionic resonances. For concreteness, we will assume that these fields transform in the fundamental representation 6 of SO (6). The latter can be decomposed as 6 = 1 + 1 + 4 under the custodial symmetry group SO (4). Let us assume that u i R is embedded in both singlets, whereas the one for q i L is fixed. These are listed as and Using the corresponding Goldstone matrix one obtains the Yukawa Lagrangian (3.6) which, to leading order, reads Hence, we obtain Y i j = −Y = y (2) i j andỸ i j = iy (1) i j . Thus, to leading order, scalar mediated FCNCs are only driven by S, provided that y (1) i j and y (2) i j are not aligned. Even in that case, the FCNCs would still arise in the presence of higherdimensional operators, and then undergo suppression by a factor of 1/g 2 * (with g * being a strong coupling) just as in the minimal CHM [8,38]. Similar results hold for other representations, with the exception of those that respect a S → −S parity and those for which the shift symmetry of S remains unbroken, examples being q i L in the 6, t i R in the 15 [46]. We note that althoughỸ i j is in principle arbitrary, one can easily expect sizeable top-charm couplings and still have small top-up FCNCs. Indeed, despite being not directly measurable, in common viable ansätze, Y is hierarchical and nearly block-diagonal, with the maximal mixing occurring in the top-charm sector [47,48]. It can therefore be diagonalised as Y → L † YR, with L and R being block-diagonal as well. Moreover, in CHMs the aforementioned hierarchy reflects the fact that heavier fermions couple stronger to the composite sector, so not only to H but also to S. One can then easily expect a similar block-diagonal structure forỸ. As a consequence, LỸR is also block-diagonal with only the topcharm mixing. Hence, we concentrate on the t → Sc decay channel. However, because we will not use any explicit ctagging in our analyses, our results can easily be translated for the t → Su mode.
We define three Benchmark Points (BP), each including m S = 20, 50, 80, 100, 120 and 150 GeV, as follows in CHMs, contribute to the Higgs width with an effective λ H S ∼ 4m 2 h / f 2 0.05 for a scale, f 1.2 TeV. If we consider only weakly-coupled extensions of the SM+S, the Lagrangian in Eq. 2.1 can also be induced at tree level by the fields listed in Table 1. In particular, this means that the NMSSM [34], which extends the SM scalar sector with an additional SU (2) L doublet with Y = 1/2 (required by SUSY itself) as well as with a singlet (in order to avoid the μ-problem [50]), fits naturally into the targets of our analysis.

LHC prospects for t → Sc, S → bb
In this section, we focus on the scenario where the scalar singlet decays to a pair of b-quarks, yielding a final state comprised of at least four jets, three of them required to be b-tagged and exactly one isolated lepton. As described above, we quantify our results in terms of six benchmark masses, viz., m S = 20, 50, 80, 100, 120 and 150 GeV. Our ultimate goal in this section is to derive an upper bound on B(t → Sc, S → bb) at 95 % Confidence Level (CL).
We have fixed the b-tagging efficiency to 70%. The c ( ) → b mistag rate has been taken as 10% (1%). The most dominant real background ensues from semi-leptonic ttbb production. Besides, the fully leptonic channel from the aforementioned production mode also contributes. The major fake backgrounds that we consider are the semi-leptonic (and leptonic) tt merged up to one extra matrix element parton, the W bb process merged up to two extra matrix element partons and Zbb also merged up to two extra partons with the Z -boson decaying leptonically. For the analysis framework, we use MG5_aMC@NLO v2.6.0 [51] for generating the signal and background samples. We employ the MLM merging scheme [52] embedded in this framework, with appropriate parameter choices. We use very loose parton level cuts, viz., p T ( j) > 15 GeV, p T (b) > 15 GeV and p T ( ) > 10 GeV, as well as |η( j)| < 4, |η(b)| < 4 and |η( )| < 3. Moreover, we require the R separations to be zero for each pair at the generation level. The cross sections of tt, W bb, Zbb and ttbb are multiplied by K -factors of 1.6, 2.3, 1.25 and 1.13, respectively. While the first one can be found in Ref.
[53], the other three have been estimated by computing in MG5_aMC@NLO at NLO in QCD. We use the NNPDF 2.3 [54] at leading order. The analyses in this section and in the next are carried out for the 14 TeV LHC.
Furthermore, we shower the samples with Pythia 8 [55]. 2 Finally, at the analysis level, we construct the jets employing the anti-k T [56] algorithm with a jet parameter R = 0.4 in the FastJet [57] framework. All the jets are required to have p T > 30 GeV and to lie within a pseudorapidity range of |η| < 2.5. Leptons must have a p T > 10 GeV and |η| < 2.5. For the isolation, we require that the total hadronic activity around the lepton within a cone of R = 0.2 is less than 10 % of its transverse momentum. All the aforementioned selected objects are also required to be separated by R > 0.4.
After selecting these events, we look for the closest pair (in terms of R separation) of b-tagged jets and reconstruct the top-quark mass m R t with the additional hardest jet which is not b-tagged. We require this variable to be within a window of 50 GeV from m t . With the remaining b-tagged jet, we construct the transverse mass variable m T and require it to be less than 200 GeV. We show the distributions of m R t , m T and the reconstructed scalar mass, m R S , after the basic cuts (which include the aforementioned p T cuts as well as a requirement for 3 b-tagged jets, at least an additional light jet and one isolated lepton) for two signal benchmark points and four dominant backgrounds, in Fig. 1. The mass-independent cutflow tables for the six benchmark points and six backgrounds are listed in Tables 2 and 3 respectively. To optimise each signal region, we impose an additional cut, viz., 0.8 m S < m R S < m S +10 GeV. In Table 4, we list the final efficiencies for each signal region after this additional cut on top of the aforementioned ones.
Finally, we show our results in Fig. 2. The left plot shows the 95 % upper limit on B R(t → Sc, S → bb) and the right plot shows the minimum integrated luminosity to test the aforementioned branching ratio to 10 −4 at 95 % CL.

LHC prospects for t → Sc, S → γ γ
We closely follow the previous section in terms of the analysis framework. Here we focus on the scenario in which the scalar decays to a pair of photons, yielding the final state with at least two jets, with one being b-tagged, one isolated lepton and two isolated photons. Similar to the leptons, we require the photons to have p T > 10 GeV and require them to lie within a pseudorapidity range of |η| < 2.5. We demand the photons to be isolated with the exact same criteria as for the leptons as discussed in the section above. The R > 0.4 cuts between pairs of all the selected objects are also used for this study. The dominant backgrounds for this channel are the semi-leptonic and di-leptonic tth processes and the QCD-QED production of ttγ γ . The cross section of the former is scaled by a K -factor of 1.68, what takes into account the NLO corrections to both the production and the h decay [58].
For the second, we use a conservative K -factor of 2. We also include the W γ γ background matched up to two hard jets.
However, despite having a cross section of order O(0.1) pb, it becomes irrelevant after imposing all cuts. Consequently, we do not show explicit numbers for this process hereafter. The selection level cuts up until the transverse mass are identical to the previous section. However, because of a much sharper diphoton mass resolution, we demand a very narrow window of 3 GeV around the scalar mass. The shape of the reconstructed top mass distributions as well as m T in this case are shown in Fig. 3 after the basic cuts (which includes objects selected with the p T requirement as mentioned above along with the selection criteria of exactly one b-tagged jet, at least one additional jet, one isolated lepton and two or more isolated photons). The cutflows are listed in Tables 5, 6 and 7. The 95% CL upper limit on the branching ratio B(t → Sc, S → γ γ ) is shown in Fig. 4 along with the minimum integrated luminosity required to probe a branching ratio of 10 −6 . In this analysis we have added a new m S mass point of 125 GeV, where the tth background is much larger. The dominance of this latter process is apparent in the figure.

Conclusions
Flavour-violating top decays into scalar singlets, S, mediated by new physics at a scale f O(TeV), dominate strongly over the ones involving the SM-like Higgs boson. From an effective-field theory point of view, the main reason for the dominance of the new scalar is due to the fact that the latter proceeds via effective operators of dimension six and is hence suppressed by 1/ f 2 , whereas the former is already present at dimension five (and hence suppressed only by 1/ f ). Moreover, the singlet can be much lighter than the Higgs, the corresponding top decay being therefore kinematically enhanced. Since such scalar particles are predicted in several new physics models, we designed novel analyses dedicated for the upcoming runs of the LHC, to search for t → Sc, S → bb/γ γ in events pertaining to top pair production. We restricted our study of S masses varying between 20 GeV < m S < 150 GeV.
In the S → bb channel, the highest reach is obtained for m S ∼ 80 GeV, for which we can probe B(t → Sc, S → bb) > 10 −4 at 95 % CL with an integrated luminosity of L = 3 ab −1 . The reach is about a factor of 5 smaller for low masses. This is due to the fact that at low masses the two b-quarks ensuing from the scalar S, do not always form two resolved b-jets, and hence upon requiring three b-tagged jets, we incur a reduction in the efficiency of the signal. However, one might consider a fat jet in the framework of a boosted analysis to overcome this difficulty. On the other hand, for large masses, the invariant mass of the two b-tagged jets closest in R separation, do not always peak at m S .
In the γ γ channel, the sensitivity of the signal is considerably less dependent on m S , given an excellent resolution of the di-photon mass spectrum. We find that a branching ratio, B(t → Sc, S → γ γ ) > 10 −7 can be tested at the 95 % CL with the same integrated luminosity. We note that, if B(S → γ γ ) is as large as ∼ 1%, then we can indirectly probe new physics scales as large as ∼ 50 TeV. Furthermore, we note that the bound obtained in this channel for m S ∼ m h agrees well with the results obtained for t → hc listed in previous works [22] (which utilise significantly different search strategies and statistical approaches). Reference [22] also showed that the increase in sensitivity at a 100 TeV collider can be roughly estimated by scaling the signal and background cross-sections and the luminosity. In this particular channel, the dominant background is ttγ γ for masses of the singlet, m S , well separated from m h . It turns out that the increase in cross section in this background at √ s = 27 TeV (100 TeV) with respect to that at √ s = 14 TeV is similar to that in the signal, and of order ∼ 4(∼ 40). Thus, assuming an integrated luminosity of 10 ab −1 , we expect an increase in significance of order 4/ √ 4 × √ 10/3 ∼ 3.7 (40/ √ 40× √ 10/3 ∼ 11.5). This implies that one can expect up to an order of magnitude improvement in the bound on B(t → Sc, S → γ γ ). Similar results will hold for the bb channel.