1 Introduction

The discovery of the Higgs boson by the Large Hadron Collider (LHC) [1, 2] experiments has opened up a new era of the direct searches for new physics (NP) beyond the Standard Model (SM). One of the possible anomalous interactions is the flavour-changing neutral currents (FCNC) process mediated by the Higgs boson, and such processes are forbidden in SM at tree level and are strongly suppressed at loop-level due to the Glashow-Iliopoulos-Maiani (GIM) mechanism [3, 4]. Being the most massive elementary particle in the SM, the top quark is an excellent probe not only to search the dynamics of electroweak symmetry breaking, but also to test SM and NP beyond the SM. The flavor-violating Higgs couplings involving a top quark in the SM, i.e., the vertex tqh (\(q=u,c\)), are out of range for current LHC experimental sensitivities. While in some NP scenarios such as supersymmetry [5,6,7], two-Higgs-doublet model [8,9,10,11,12], 3-3-1 model [13], and other NP models by extending the Higgs sector [14, 15], the branching ratios of top quark FCNC decays \(t\rightarrow qh\) are predicted promisingly at the order of \(10^{-6}-10^{-3}\). Thus searches for such FCNC processes are very important and could be considered as a clear signal for NP beyond the SM [16,17,18].

By searching for \(pp\rightarrow t{\bar{t}}\) process with one top decaying to Wb and the other assumed to decay to hq, the ATLAS and CMS collaborations have given the upper limits on the top quark FCNC branching ratios at \(95\%\) Confidence Level (CL) at \(\sqrt{s}=13\) TeV [19,20,21,22]. For example, the combination of these searches with ATLAS yields observed limits [21]: \(Br(t\rightarrow uh)\le 1.2\times 10^{-3}\), and \(Br(t\rightarrow ch)\le 1.1\times 10^{-3}\). As a more promising and realistic project, the future High-Luminosity LHC (HL-LHC) expects to reach branching \(Br(t\rightarrow uh)\le 2.4\times 10^{-4}\) and \(Br(t\rightarrow ch)\le 1.5\times 10^{-4}\) with integrated luminosity of 3000 fb\(^{-1}\) at \(\sqrt{s}=14\) TeV with a full simulation of the ATLAS detector upgrade [23, 24].

At present there is no any experimental evidence of the FCNC process. One can expect that future research and improvements on these bounds with the proposed higher center-of-mass energy colliders [25], such as High Energy LHC (HE-LHC) with 27 TeV c.m. energy [26] and Future Circular Collider in hadron-hadron mode (FCC-hh) with 100 TeV c.m. energy [27]. Very recently, the authors of Ref. [28] investigated the thj final signal by considering thq FCNC couplings from three different subprocesses at the HE-LHC and FCC-hh. Their results show that the top pair production process with one of top FCNC decay could give dominant contributions. The aim of this work is to investigate the limits on the anomalous tqh couplings by considering the process \(pp\rightarrow th\) at the HE-LHC and FCC-hh projects with the decay chains: \(t\rightarrow W^{+}b \rightarrow \ell ^+ \nu b\) and \(h\rightarrow \gamma \gamma \). Realistic detector effects are included in the production of signal and background processes. The sensitivity of anomalous FCNC tqh couplings are searched by using multivariate analysis. Finally, the results are reported respectively for HE-LHC and FCC-hh and compared with other phenomenological predictions.

This paper is arranged as follows. In Sect. 2, the cross section of the the signal process \(pp\rightarrow th\) is calculated at the HE-LHC and FCC-hh. This section also includes the estimations for the signal and background, the simulations and the detector effects for HE-LHC and FCC-hh. We also give the 95% confidence level limits of HE-LHC and FCC-hh and compare with the LHC measured limits. Furthermore, we also consider the effects of the systematic uncertainty on the backgrounds. Finally, we conclude and summarize our main results and findings in Sect. 3.

2 Analysis strategy and numerical result

In general, the relevant top quark FCNC couplings is explored in a model-independent way considering the most general effective Lagrangian approach [4]. The Lagrangian of the FCNC tqh interactions can be written as

$$\begin{aligned} \mathcal{L}= \kappa _{tuh}{\bar{t}}Hu+\kappa _{tch}{\bar{t}}Hc+h.c., \end{aligned}$$
(1)

where the parameters \(\kappa _{tuh}\) and \(\kappa _{tch}\) represent the strength of tqh FCNC interactions. Such FCNC interactions have been phenomenologically studied in many final states with various sensitivities [29,30,31,32,33,34,35,36,37].

Fig. 1
figure 1

The Feynman diagrams for th production at the LHC through FCNC tqh interactions

Full size image

With the above effective interactions, the Feynman diagrams for the production of single top quark associated with the Higgs boson are shown in Fig. 1. Obviously, the conjugate process \(pp\rightarrow {\bar{t}}h\) production can also occur at the tree level which do not included in Fig. 1. For the simulations of the HE-LHC and FCC-hh collider phenomenology, we first use the FeynRules [38] to generate the Universal FeynRules Output (UFO) files [39]. The leading order cross sections are obtained by using MadGraph5-aMC@NLO [40] with NNPDF23L01 parton distribution functions (PDFs) [41] considering the renormalization and factorization scales to be \(\mu _R=\mu _F=\mu _0/2=(m_t + m_h)/2\). We take \(\kappa _{tqh}\le 0.04\) to satisfy the upper limit from the current LHC experiments [28]. The numerical values of the input parameters are taken as follows [42]:

$$\begin{aligned} m_h&=125.1\mathrm{~GeV}, \quad m_t=172.9\mathrm{~GeV}, \quad m_W=80.379\mathrm{~GeV},\nonumber \\ m_Z&=91.1876\mathrm{~GeV}, \quad \alpha _{s}(m_Z)=0.1185, \quad \nonumber \\ G_F&=1.166370\times 10^{-5}\ \mathrm{GeV^{-2}}. \end{aligned}$$
(2)
Table 1 The cross sections (in fb) of the subprocess \(pp\rightarrow th\) with \(\kappa _{tqh}=0.01\) at HE-LHC and FCC-hh
Full size table

The numerical values of the cross sections are shown in Table 1 at the leading order. One can see that the cross section of \({\bar{u}}g\rightarrow {\bar{t}}h\) is overwhelmed by \(ug\rightarrow th\) due to the difference between the u-quark and \({\bar{u}}\)-quark PDF of the proton. Thus, if we consider the leptonic top decay modes, more leptons will be observed than anti-leptons for a given center-of-mass (c.m.) energy and luminosity. However, since the c-quark and \({\bar{c}}\)-quark have the similar small PDF, the cross section of \({\bar{c}}g\rightarrow {\bar{t}}h\) is same to that of \(cg\rightarrow th\) and they are much smaller than the cross section of \(ug\rightarrow th\) for the same values of \(\kappa _{tqh}\). This implies that the sensitivity to the FCNC coupling parameter \(\kappa _{tuh}\) will be better than \(\kappa _{tch}\).

Fig. 2
figure 2

The Feynman diagram for the \(pp\rightarrow th\) process including the leptonic decay of top quark and diphoton decay of the Higgs boson

Full size image

2.1 Monte Carlo simulations

The signal is mainly produced through the process

$$\begin{aligned} pp\rightarrow & {} t(\rightarrow W^{+}b\rightarrow \ell ^{+}\nu b)h(\rightarrow \gamma \gamma ), \end{aligned}$$
(3)

where \(\ell = e, \mu \). Note that the analysis has included the charge-conjugate process. The final state for the signal is characterised by two photons, one lepton, one b-jet and missing transverse energy from the undetected neutrino, as shown in Fig. 2. Certainly, the \(pp\rightarrow t{\bar{t}}\rightarrow thj\) process should also be a source of the signal if the final light quark is missed by the detector. Due to small production cross sections for \(c({\bar{c}})\rightarrow ht({\bar{t}})\) processes, this additional contribution is very significant for detecting the \(\lambda _{tch}\) couplings [29]. Therefore, we also include the \(pp\rightarrow t{\bar{t}}\rightarrow thj\) process and focus on the final topology from the \(pp\rightarrow th\) process without the extra light jet. Considering the signal scenarios, the following relevant background processes which have similar final state need to be taken into account: the resonant backgrounds \(t{\bar{t}}h\) and thj in which the Higgs boson decays to diphotons, and the non-resonant backgrounds \(t{\bar{t}}\gamma \gamma \), \(tj\gamma \gamma \) and \(\gamma \gamma W^{\pm }jj\). We do not consider processes where the light jet is mis-tagged as the photon, which is beyond the scope of this paper.

The signal and background samples are generated at leading order (LO) by using the Monte Carlo event generator MadGraph5-aMC@NLO [40], interfaced to Pythia 8.20 [43] for the parton showering. All produced jets are forced to be clustered using the FASTJET 3.2 [44] assuming the anti-\(k_{t}\) algorithm with a cone radius of \(R=0.4\) [45]. All event samples are fed into the Delphes 3.4.2 [46] with the default HE-LHC and FCC-hh detector card. Finally, event analysis is performed by using MadAnalysis5 [47]. Here it should be mentioned that we rescale the leading order cross sections to the corresponding higher order QCD results with the K-factor, i.e., \(K=1.5\) for the \(pp\rightarrow th\) process [29, 48], \(K=1.4\) for the \(pp\rightarrow t{\bar{t}}\rightarrow thj\) process [49] and \(K=1.3\) for the \(pp\rightarrow t{\bar{t}}h\) process [49,50,51]. For the sake of simplicity, others SM background processes, we have rescaled their cross sections by a K-factor of 1.5. This approximation does not have a significant impact on our derived sensitivities.

In order to identify objects, we chose the basic cuts at parton level for the signals and SM backgrounds as follows:

$$\begin{aligned} \begin{aligned} \mathrm {HE-LHC:}&p_{T}^{\ell /\gamma }>&~25~\text {GeV},&p_{T}^{j/b}>&~30~\text {GeV},&|\eta _{i}|<&~2.5,\\ \mathrm {FCC-hh:}&p_{T}^{\ell /\gamma }>&~25~\text {GeV},&p_{T}^{j/b}>&~30~\text {GeV},&|\eta _{i}|<&~5,\\ \end{aligned}\end{aligned}$$
(4)

where \(\Delta R=\sqrt{\Delta \Phi ^{2}+\Delta \eta ^{2}}\) is the separation in the rapidity-azimuth plane, \(p_{T}^{\gamma ,\ell , b, j}\) are the transverse momentum of photons, leptons, b-jets and jets.

Fig. 3
figure 3

Normalised distributions for the signals and SM backgrounds at the HE-LHC

Full size image

In Fig. 3, we plot some differential distributions for signals and SM backgrounds at the HE-LHC, such as the transverse momentum distributions of the two photons, the separations \(\Delta R_{\gamma \gamma }\) and \(\Delta R_{\gamma \ell }\), the invariant mass distributions of the two photons, \(M_{\gamma _{1}\gamma _{2}}\), and the transverse mass distribution for the top quark. Here the transverse masses of the top quark are defined as [52]

$$\begin{aligned} M_T^{2}(b\ell )\equiv & {} (\sqrt{(p_{\ell }+p_{b})^{2}+|\vec {p}_{T,\ell }+\vec {p}_{T,b}|^{2}}+|\vec {p\!\!/}_T| )^{2}\nonumber \\&-|\vec {p}_{T,\ell }+\vec {p}_{T,b}+\vec {p\!\!/}_T|^{2}, \end{aligned}$$
(5)

where \(\vec {p}_{T,\ell }\) and \(\vec {p}_{T,b}\) are the transverse momentums of the charged leptons and b-quark, respectively, and \(\vec {p\!\!/}_T\) is the missing transverse momentum determined by the negative sum of visible momenta in the transverse direction.

One can see that the two photons in the signal and the resonant backgrounds have the harder \(p_{T}\) spectrum than those in the non-resonant backgrounds. Furthermore, the signal and the resonant backgrounds have the diphoton invariant-mass peak at the mass of Higgs boson. Besides, the cut on the transverse mass of the top candidate is needed since the lepton and b-jet are assumed to originate from the leptonic decaying top quark.

According to the above analysis, we can impose the following set of cuts.

  • Cut 1: There are only one isolated lepton, especially one positive charge lepton is required for \(ug\rightarrow th\) process, The leading b-tagged jet is required to have \(p_{T}>30\) GeV. Furthermore, an event is rejected if the \(p_T\) of the subleading jet or subleading b-tagged jet is greater than 30 GeV.

  • Cut 2: At least two photons and the leading (sub-leading) photons with \(p_{T}>70~(40) \mathrm ~GeV\), \(\Delta R_{\gamma _{1},\gamma _{2}}< 1.8\) and \(\Delta R_{\gamma _{1},\ell }> 2.0\) .

  • Cut 3: The invariant mass of the two diphotons is required as \(|M_{\gamma _{1}\gamma _{2}} - m_h| < 5\) GeV.

  • Cut 4: The transverse mass of the W candidate is required as \(M_T^{W} > 30 \mathrm ~GeV\)Footnote 1, and the transverse mass of the top quark is required as \(100 ~\mathrm{GeV}< M_T^{b\ell } < 180 \mathrm ~GeV\).

Table 2 The cut flow of the cross sections (in ab) for the signals and SM backgrounds at the HE-LHC with \(\kappa _{tuh}=0.04\) and \(\kappa _{tch}=0.04\) (in the bracket)
Full size table
Table 3 The cut flow of the cross sections (in ab) for the signals and SM backgrounds at the FCC-hh with \(\kappa _{tuh}=0.04\) and \(\kappa _{tch}=0.04\) (in the bracket)
Full size table

We use the same selection cuts for the FCC-hh analysis at 100 TeV because the distributions are very similar to the case of HE-LHC. The effects of the suitable cuts on the signal and SM background processes are illustrated in Tables 2, 3. One can see that, the jet veto in cut 1 significantly reduce the backgrounds which contain two b jets from the decay of the top quarks. The cut on the invariant-mass of the mass of Higgs boson can reduce the non-resonance backgrounds largely. At the end of the cut flow, the total SM backgrounds are very small, and there are 90 events at the FCC-hh with the integrated luminosity of 30 fb\(^{-1}\). Furthermore, the cross section for the \(ug\rightarrow th\) process is more than twice that of the \(pp\rightarrow t{\bar{t}}\rightarrow thj\) process, while the cross section for the \(cg\rightarrow th\) process is smaller than that of the \(pp\rightarrow t{\bar{t}}\rightarrow thj\) process. Thus we can not neglect the contribution from the top pair production with the FCNC decay process when considering the sensitivities on \(\kappa _{tch}\).

Table 4 The upper limits on BR\((t\rightarrow qh)\) at 95% CL obtained at the HE-LHC and FCC-hh. We consider systematic errors of 0%, 10% and 20% on the SM background events
Full size table
Fig. 4
figure 4

The \(95\%\) and \(5\sigma \) contour plots for the signal in \(L_{int}-Br(t\rightarrow uh)\) (left) and \(L_{int}-Br(t\rightarrow ch)\) (right) planes at HE-LHC. For comparison, the results also plot the under two figures for the case of only considering the signal from the \(pp\rightarrow th\) process

Full size image
Fig. 5
figure 5

Same for Fig. 4 but at the FCC-hh

Full size image

2.2 Sensitivity at the HE-LHC and FCC-hh

To estimate the expected discovery and exclusion significance, \(Z_\text {disc}\) and \(Z_\text {excl}\), for the small number of events, we use the following expression [53]:

$$\begin{aligned} \begin{aligned}&Z_\text {disc} = \sqrt{2\left[ (s+b)\ln \left( \frac{(s+b)(1+\delta ^2 b)}{b+\delta ^2 b(s+b)}\right) - \frac{1}{\delta ^2 }\ln \left( 1+\delta ^2\frac{s}{1+\delta ^2 b}\right) \right] } \\&Z_\text {excl} \\&\quad =\sqrt{2\left[ s-b\ln \left( \frac{b+s+x}{2b}\right) - \frac{1}{\delta ^2 }\ln \left( \frac{b-s+x}{2b}\right) \right] - \left( b+s-x\right) \left( 1+\frac{1}{\delta ^2 b}\right) }, \end{aligned} \end{aligned}$$
(6)

with

$$\begin{aligned} \begin{aligned} x=\sqrt{(s+b)^2- 4 \delta ^2 s b^2/(1+\delta ^2 b)}. \end{aligned} \end{aligned}$$
(7)

Here, s and b represent the total signal and SM background events, respectively. \(\delta \) is the percentage systematic error on the SM background estimate. In this work we choose three cases: \(\delta =0\%, 10\%, 20\%\). In the limit of \(\delta \rightarrow 0\), the above expressions can be simplified as

$$\begin{aligned}&Z_\text {disc} = \sqrt{2[(s+b)\ln (1+s/b)-s]}, \nonumber \\&Z_\text {excl} = \sqrt{2[s-b\ln (1+s/b)]} \end{aligned}$$
(8)

We define the regions with \(Z_\text {excl} \le 1.645\) as those that can be excluded at 95% CL, and \(Z_\text {disc} \ge 5\) as discovered regions [53].

Table 5 The recent projected 95% CL limits on \(Br(t\rightarrow qh)\) from different channels at the HE-LHC and FCC-hh
Full size table

In Table 4, we list the exclusion limits at 95% CL at the future HE-LHC and FCC-hh, respectively, with three systematic error cases: \(\delta =0\%\), \(\delta =10\%\) and \(\delta =20\%\). One can see that, the sensitivities are slightly weaker than those without any systematic error, being of the order of \(10^{-5}\) at the 95% CL both at the HE-LHC and FCC-hh.

In Figs. 4, 5, the \(95\%\) CL and \(5\sigma \) lines are drawn as a function of the integrated luminosity and \(BR(t\rightarrow qh)\). We also plot the results for the case of only considering the signal from the process \(pp\rightarrow th\) (marked as Case 2 in the figures). For the case of \(\delta =10\%\), one can see that the \(5\sigma \) discovery sensitivity of \(Br(t\rightarrow qh)\) (\(q=u/c\)) is about \(1.5\times 10^{-4}\) (\(2.2\times 10^{-4}\)) at the HE-LHC with the integrated luminosity is 15 ab\(^{-1}\), and about \(6\times 10^{-5}\) (\(9\times 10^{-5}\)) at the FCC-hh with the integrated luminosity is 30 ab\(^{-1}\), respectively. However, the numerical results are changed significantly for the sensitivity of \(Br(t\rightarrow ch)\) in Case 2. This is mainly because the cross sections of the \(ug\rightarrow th\) process and \(pp\rightarrow t{\bar{t}}\rightarrow thj\) process are much larger than that of the \(cg\rightarrow th\) process, which has been discussed earlier. For example, at the HE-LHC with the integrated luminosity is 15 ab\(^{-1}\) and the FCC-hh with the integrated luminosity is 30 ab\(^{-1}\), the upper limits at \(95\%\) CL on \(Br(t\rightarrow ch)\) with \(\delta =10\%\) are changed as \(2.4\times 10^{-4}\) and \(6.6\times 10^{-5}\), respectively, and the \(5\sigma \) CL discovery sensitivity changed as \(8.3\times 10^{-4}\) and \(2.4\times 10^{-4}\), respectively.

Very recently, many phenomenological studies available in literature have extensively investigated the tqh FCNC couplings via the top pair production process at the future hadron colliders, see e.g. Refs. [28, 54,55,56] for the most recent reviews. It is worth comparing the limits obtained in this study with those obtained in other groups, which are summarized in Table 5. One can see that the limits on the branching ratios are expected to be of \({\mathcal {O}}(10^{-5})\). Therefore, we expect to provide the complementary information for detecting such anomalous couplings at the future HE-LHC and FCC-hh.

3 Conclusion

In this work, we have studied the top-Higgs FCNC interactions, i.e., tqh (\(q= u, c\)), at the HE-LHC and FCC-hh by performing a full simulation for the process: \(pp\rightarrow th\) with the top leptonic decay channel and \(h\rightarrow \gamma \gamma \) decay channel. In the case the SM background is known with negligible uncertainty, our results show that 95% CL limits on the branching ratios of \(t\rightarrow u(c)h\), have been found to be \(4.4~(6.4) \times 10^{-5}\) at the HE-LHC with an integrated luminosity of 15 ab\(^{-1}\) and \(1.3~(1.6) \times 10^{-5}\) at the FCC-hh with an integrated luminosity of 30 ab\(^{-1}\). Meanwhile, the \(5\sigma \) discovery sensitivity of \(Br(t\rightarrow u(c)h)\) is about \(1.4\times 10^{-4}\) (\(2.0\times 10^{-4}\)) at the HE-LHC, and about \(4.1\times 10^{-5}\) (\(5\times 10^{-5}\)) at the FCC-hh, respectively. Therefore, we expect to provide the complementary information for detecting such anomalous couplings at the future HE-LHC and FCC-hh.