Probing anomalous tqh couplings via single top production in associated with the Higgs boson at the HE-LHC and FCC-hh

We investigate the prospects for discovering the Flavour Changing Neutral Current (FCNC) tqh couplings via the process pp→th\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$pp\rightarrow th$$\end{document} at the proposed High Energy Large Hadron Collider (HE-LHC) and Future Circular Collider in hadron-hadron mode (FCC-hh) including the realistic detector effects. The relevant SM backgrounds are considered in the cut based analysis to obtain the limits on the the branching ratios of t→qh(q=u,c)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t\rightarrow qh~(q=u,c)$$\end{document}, followed by the leptonic decay channel of the top quark and diphoton decay channel of the Higgs boson. The upper limits on the FCNC branching ratios at 95% confidence level (CL) and the 5σ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$5\sigma $$\end{document} discovery reach for the different integrated luminosities are obtained. It is shown that at the 27 TeV HE-LHC with an integrated luminosity of 15 ab-1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{-1}$$\end{document} and at the 100 TeV FCC-hh with an integrated luminosity of 30 ab-1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{-1}$$\end{document}, the BR(t→uh\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t\rightarrow uh$$\end{document}) (BR(t→ch\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t\rightarrow ch$$\end{document})) can be probed, respectively, to 4.4(6.4)×10-5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$4.4~(6.4)\times 10^{-5}$$\end{document} and 1.3(1.6)×10-5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1.3~(1.6) \times 10^{-5}$$\end{document} at the 95% CL.


Introduction
The discovery of the Higgs boson by the Large Hadron Collider (LHC) [1,2] experiments has opened up a new era of the direct searches for new physics (NP) beyond the Standard Model (SM). One of the possible anomalous interactions is the flavour-changing neutral currents (FCNC) process mediated by the Higgs boson, and such processes are forbidden in SM at tree level and are strongly suppressed at loop-level due to the Glashow-Iliopoulos-Maiani (GIM) mechanism [3,4]. Being the most massive elementary particle in the SM, the top quark is an excellent probe not only to search the dynamics of electroweak symmetry breaking, but also to test SM and NP beyond the SM. The flavor-violating Higgs couplings involving a top quark in the SM, i.e., the vertex tqh (q = u, c), a e-mail: yanjuzhang@xxmu.edu.cn b e-mail: shjf@xxmu.edu.cn (corresponding author) are out of range for current LHC experimental sensitivities. While in some NP scenarios such as supersymmetry [5][6][7], two-Higgs-doublet model [8][9][10][11][12], 3-3-1 model [13], and other NP models by extending the Higgs sector [14,15], the branching ratios of top quark FCNC decays t → qh are predicted promisingly at the order of 10 −6 − 10 −3 . Thus searches for such FCNC processes are very important and could be considered as a clear signal for NP beyond the SM [16][17][18].
By searching for pp → tt process with one top decaying to W b and the other assumed to decay to hq, the ATLAS and CMS collaborations have given the upper limits on the top quark FCNC branching ratios at 95% Confidence Level (CL) at √ s = 13 TeV [19][20][21][22]. For example, the combination of these searches with ATLAS yields observed limits [21]: As a more promising and realistic project, the future High-Luminosity LHC (HL-LHC) expects to reach branching Br(t → uh) ≤ 2.4 × 10 −4 and Br(t → ch) ≤ 1.5 × 10 −4 with integrated luminosity of 3000 fb −1 at √ s = 14 TeV with a full simulation of the ATLAS detector upgrade [23,24].
At present there is no any experimental evidence of the FCNC process. One can expect that future research and improvements on these bounds with the proposed higher center-of-mass energy colliders [25], such as High Energy LHC (HE-LHC) with 27 TeV c.m. energy [26] and Future Circular Collider in hadron-hadron mode (FCC-hh) with 100 TeV c.m. energy [27]. Very recently, the authors of Ref. [28] investigated the th j final signal by considering thq FCNC couplings from three different subprocesses at the HE-LHC and FCC-hh. Their results show that the top pair production process with one of top FCNC decay could give dominant contributions. The aim of this work is to investigate the limits on the anomalous tqh couplings by considering the process pp → th at the HE-LHC and FCC-hh projects with the decay chains: t → W + b → + νb and h → γ γ . Realistic detector effects are included in the production of signal and background processes. The sensitivity of anomalous FCNC tqh couplings are searched by using multivariate analysis. Finally, the results are reported respectively for HE-LHC and FCC-hh and compared with other phenomenological predictions.
This paper is arranged as follows. In Sect. 2, the cross section of the the signal process pp → th is calculated at the HE-LHC and FCC-hh. This section also includes the estimations for the signal and background, the simulations and the detector effects for HE-LHC and FCC-hh. We also give the 95% confidence level limits of HE-LHC and FCC-hh and compare with the LHC measured limits. Furthermore, we also consider the effects of the systematic uncertainty on the backgrounds. Finally, we conclude and summarize our main results and findings in Sect. 3.

Analysis strategy and numerical result
In general, the relevant top quark FCNC couplings is explored in a model-independent way considering the most general effective Lagrangian approach [4]. The Lagrangian of the FCNC tqh interactions can be written as where the parameters κ tuh and κ tch represent the strength of tqh FCNC interactions. Such FCNC interactions have been phenomenologically studied in many final states with various sensitivities [29][30][31][32][33][34][35][36][37].
With the above effective interactions, the Feynman diagrams for the production of single top quark associated with the Higgs boson are shown in Fig. 1. Obviously, the conjugate process pp →th production can also occur at the tree level which do not included in Fig. 1. For the simulations of the HE-LHC and FCC-hh collider phenomenology, we first use the FeynRules [38] to generate the Universal FeynRules Output (UFO) files [39]. The leading order cross sections are obtained by using MadGraph5-aMC@NLO [40] with NNPDF23L01 parton distribution functions (PDFs) [41] considering the renormalization and factorization scales to be μ R = μ F = μ 0 /2 = (m t + m h )/2. We take κ tqh ≤ 0.04 to satisfy the upper limit from the current LHC experiments The Feynman diagram for the pp → th process including the leptonic decay of top quark and diphoton decay of the Higgs boson [28]. The numerical values of the input parameters are taken as follows [42]: The numerical values of the cross sections are shown in Table 1 at the leading order. One can see that the cross section ofūg →th is overwhelmed by ug → th due to the difference between the u-quark andū-quark PDF of the proton. Thus, if we consider the leptonic top decay modes, more leptons will be observed than anti-leptons for a given centerof-mass (c.m.) energy and luminosity. However, since the c-quark andc-quark have the similar small PDF, the cross section ofcg →th is same to that of cg → th and they are much smaller than the cross section of ug → th for the same values of κ tqh . This implies that the sensitivity to the FCNC coupling parameter κ tuh will be better than κ tch .

Monte Carlo simulations
The signal is mainly produced through the process where = e, μ. Note that the analysis has included the charge-conjugate process. The final state for the signal is characterised by two photons, one lepton, one b-jet and missing transverse energy from the undetected neutrino, as shown in Fig. 2. Certainly, the pp → tt → th j process should also be a source of the signal if the final light quark is missed by the detector. Due to small production cross sections for c(c) → ht (t) processes, this additional contribution is very significant for detecting the λ tch couplings [29]. Therefore, we also include the pp → tt → th j process and focus on the final topology from the pp → th process without the extra light jet. Considering the signal scenarios, the following relevant background processes which have similar final state need to be taken into account: the resonant backgrounds tth and th j in which the Higgs boson decays to diphotons, and the non-resonant backgrounds ttγ γ , t jγ γ and γ γ W ± j j. We do not consider processes where the light jet is mis-tagged as the photon, which is beyond the scope of this paper. The signal and background samples are generated at leading order (LO) by using the Monte Carlo event generator MadGraph5-aMC@NLO [40], interfaced to Pythia 8.20 [43] for the parton showering. All produced jets are forced to be clustered using the FASTJET 3.2 [44] assuming the anti-k t algorithm with a cone radius of R = 0.4 [45]. All event samples are fed into the Delphes 3.4.2 [46] with the default HE-LHC and FCC-hh detector card. Finally, event analysis is performed by using MadAnalysis5 [47]. Here it should be mentioned that we rescale the leading order cross sections to the corresponding higher order QCD results with the K -factor, i.e., K = 1.5 for the pp → th process [29,48], K = 1.4 for the pp → tt → th j process [49] and K = 1.3 for the pp → tth process [49][50][51]. For the sake of simplicity, others SM background processes, we have rescaled their cross sections by a K -factor of 1.5. This approximation does not have a significant impact on our derived sensitivities.
In order to identify objects, we chose the basic cuts at parton level for the signals and SM backgrounds as follows: where R = 2 + η 2 is the separation in the rapidityazimuth plane, p γ, ,b, j T are the transverse momentum of photons, leptons, b-jets and jets.
In Fig. 3, we plot some differential distributions for signals and SM backgrounds at the HE-LHC, such as the transverse momentum distributions of the two photons, the separations R γ γ and R γ , the invariant mass distributions of the two photons, M γ 1 γ 2 , and the transverse mass distribution for the top quark. Here the transverse masses of the top quark are defined as [52] where p T, and p T,b are the transverse momentums of the charged leptons and b-quark, respectively, and / p T is the miss-ing transverse momentum determined by the negative sum of visible momenta in the transverse direction.
One can see that the two photons in the signal and the resonant backgrounds have the harder p T spectrum than those in the non-resonant backgrounds. Furthermore, the signal and the resonant backgrounds have the diphoton invariant-mass peak at the mass of Higgs boson. Besides, the cut on the transverse mass of the top candidate is needed since the lepton and b-jet are assumed to originate from the leptonic decaying top quark.
According to the above analysis, we can impose the following set of cuts.
• Cut 1: There are only one isolated lepton, especially one positive charge lepton is required for ug → th process, The leading b-tagged jet is required to have p T > 30 GeV. Furthermore, an event is rejected if the p T of the subleading jet or subleading b-tagged jet is greater than 30 GeV. We use the same selection cuts for the FCC-hh analysis at 100 TeV because the distributions are very similar to the case of HE-LHC. The effects of the suitable cuts on the signal and SM background processes are illustrated in Tables 2, 3. One can see that, the jet veto in cut 1 significantly reduce the backgrounds which contain two b jets from the decay of the top quarks. The cut on the invariant-mass of the mass of Higgs boson can reduce the non-resonance backgrounds largely. At the end of the cut flow, the total SM backgrounds are very small, and there are 90 events at the FCC-hh with the integrated luminosity of 30 fb −1 . Furthermore, the cross section for the ug → th process is more than twice that of the pp → tt → th j process, while the cross section for the cg → th process is smaller than that of the pp → tt → th j process. Thus we can not neglect the contribution from the top pair production with the FCNC decay process when considering the sensitivities on κ tch . 1 Due to existence of neutrino in the final state of signal events, the W candidate transverse mass is reconstructed as M W Table 2 The cut flow of the cross sections (in ab) for the signals and SM backgrounds at the HE-LHC with κ tuh = 0.04 and κ tch = 0.04 (in the bracket)

Cuts
Signal Backgrounds

Sensitivity at the HE-LHC and FCC-hh
To estimate the expected discovery and exclusion significance, Z disc and Z excl , for the small number of events, we use the following expression [53]: BR(t → ch) 6.4 × 10 −5 6.5 × 10 −5 6.9 × 10 −5 1.6 × 10 −5 2.5 × 10 −5 3.9 × 10 −5 Here, s and b represent the total signal and SM background events, respectively. δ is the percentage systematic error on the SM background estimate. In this work we choose three cases: δ = 0%, 10%, 20%. In the limit of δ → 0, the above We define the regions with Z excl ≤ 1.645 as those that can be excluded at 95% CL, and Z disc ≥ 5 as discovered regions [53].
In Table 4, we list the exclusion limits at 95% CL at the future HE-LHC and FCC-hh, respectively, with three systematic error cases: δ = 0%, δ = 10% and δ = 20%. One can see that, the sensitivities are slightly weaker than those without any systematic error, being of the order of 10 −5 at the 95% CL both at the HE-LHC and FCC-hh.
In Figs. 4, 5, the 95% CL and 5σ lines are drawn as a function of the integrated luminosity and B R(t → qh). We also plot the results for the case of only considering the signal from the process pp → th (marked as Case 2 in the figures). For the case of δ = 10%, one can see that the 5σ discovery sensitivity of Br(t → qh) (q = u/c) is about 1.5 × 10 −4 (2.2 × 10 −4 ) at the HE-LHC with the integrated luminosity is 15 ab −1 , and about 6 × 10 −5 (9 × 10 −5 ) at the FCC-hh with the integrated luminosity is 30 ab −1 , respectively. However, the numerical results are changed significantly for the sensitivity of Br(t → ch) in Case 2. This is mainly because the cross sections of the ug → th process and pp → tt → th j process are much larger than that of the cg → th process, which has been discussed earlier.