Search for anomalous couplings via single top quark production in association with a photon at LHC

Abstract

Considering the experimental constraints given by the CMS collaboration at $\sqrt{s} = 8$ TeV on the strength of top quark flavour-changing neutral-current couplings $tq\gamma$ and tqg, we investigate the production of top quark in association with a photon and carry out a full simulation for the signals $\ell \nu b\gamma$ and $jjb\gamma$ at 14 TeV LHC. In our numerical analysis, the contributions of single top production with a photon radiation off the top decay products are also included. The discovery potential for anomalous couplings $tq\gamma$ and tqg with an integrated luminosity of 100 fb$^{-1}$ is examined in detail.

1 Introduction

The top quark is generally considered as a sensitive probe of physics beyond the standard model (BSM) [1], with its large mass close to the electroweak symmetry breaking scale. Transitions between top quarks and other quark flavours mediated by neutral gauge bosons, the flavour-changing neutral-currents (FCNC), are forbidden at tree level in the standard model (SM) and suppressed at the level of quantum loop due to the Glashow–Iliopoulos–Maiani (GIM) mechanism [2]. The branching ratios for the processes $t \rightarrow q\gamma$ and $t \rightarrow qg$ are of the order of $10^{-14}$–$10^{-12}$ in the SM [3, 4]. In contrast, several BSM scenarios, such as the two-Higgs doublet model, supersymmetry or technicolour, predict much larger rates [5, 6], of the order of $10^{-6}$–$10^{-5}$. It implies that observation of the large FCNC-induced couplings $tq\gamma$ and tqg would indicate the existence of BSM.

The enhanced FCNC $tq\gamma$ and tqg interactions are predicted by many extensions of the SM which include new exotic quarks [7], new scalars [8, 9], supersymmetry [5, 10–14], or technicolour [6, 15]. The BSM effects can be described by a minimal set of the higher order effective operators independently from the underlying theory [16]. The effective operators not only simplify multiple free parameters of specific models in a model-independent way, but also they order them and allow us to consistently take into account higher order quantum corrections. This method appears in many studies to search top quark FCNC [17–39]. It can facilitate the analysis of new physics effects in $tq\gamma$ and tqg interactions. So we can give limits on the strength of anomalous top couplings in a model-independent way. The most general effective Lagrangian can be written as

$$\begin{aligned} \mathcal {L}_\mathrm{eff}= & {} -e Q_{t}\bar{u}\frac{i\sigma ^{\mu \nu }q_{\nu }}{\Lambda } (\kappa _{tu\gamma }^{L}P_{L}+\kappa _{tu\gamma }^{R} P_{R})tA_{\mu } \nonumber \\&-\, e Q_{t}\bar{c}\frac{i\sigma ^{\mu \nu }q_{\nu }}{\Lambda } (\kappa _{tc\gamma }^{L}P_{L}+\kappa _{tc\gamma }^{R} P_{R})tA_{\mu } \nonumber \\&-\, g_{s}\bar{u}\frac{i\sigma ^{\mu \nu }q_{\nu }}{\Lambda } (\kappa _{tug}^{L} P_{L}+\kappa _{tug}^{R} P_{R})T^a tG_{a\mu } \nonumber \\&-\, g_{s}\bar{c}\frac{i\sigma ^{\mu \nu }q_{\nu }}{\Lambda } (\kappa _{tcg}^{L} P_{L}+\kappa _{tcg}^{R} P_{R})T^a tG_{a\mu } + h.c.,\quad \end{aligned}$$

(1)

where $Q_{t}$ is the electric charge of the top quark, $g_{s}$ is the strong-coupling constant, $T^{a}=\lambda _{a}/2$ are colour matrices, q is the momentum of the gauge boson, and $P_{L(R)}$ denotes the left (right)-handed projection operators. ${\Lambda }$ is the new physics scale, which is related to the cutoff mass scale above which the effective theory breaks down. The terms with $\sigma ^{\mu \nu }=\frac{1}{2}[\gamma ^{\mu },\gamma ^{\nu }]$ are suppressed by the GIM mechanism, and in consequence are absent at tree level in renormalizable theories, like the SM. Real dimensionless parameters $\kappa _{tqV}^{L,R}$ are the strength of anomalous couplings tqV with $V=\gamma ,g$ and $q=u,c$.

Among FCNC top quark decays, $t \rightarrow qg$ is very difficult to distinguish from generic multijet production via quantum chromodynamics (QCD). It has therefore been suggested to search for FCNC couplings in anomalous single top quark production. The existence of anomalous couplings $tq\gamma$ and tqg would induce production of a top quark in association with a photon, $pp\rightarrow t\gamma$. Next-to-leading order QCD predictions for this process have been studied in [24, 33]. This process has been probed at the CMS experiment, as yet, with no indication of any signal. The strong bounds on the strengths of anomalous couplings have been provided by the CMS experiment with $\sqrt{s}=8$ TeV [40, 41]. We consider the CMS limits and give the discovery potential of 14 TeV LHC.

The aim of this paper is to investigate the limits on anomalous top couplings by considering $t\gamma$ production. Unlike previous studies of $t\gamma$ production, we focus on an analysis of signal and backgrounds based on the CMS detector simulation. In addition, we discuss the sensitivity of 14 TeV LHC to anomalous top couplings and detection potential bounds on the $tq\gamma$ and tqg couplings.

The rest of the paper is organized as follows. In Sect. 2, we provide the cross sections of $t\gamma$ production with $\sqrt{s}=14$ TeV. The simulation of signal and the expected backgrounds is discussed in detail. At the end of this section, we discuss the contribution of single top production with a photon radiation off the top decay products. In Sect. 3, we analyze the sensitivity of 14 TeV LHC to anomalous top couplings in detail. The limits on the branching ratios of top quarks into lighter quarks and photons or gluons are given correspondingly. Finally we summarize our results in Sect. 4.

2 Anomalous top couplings and $t\gamma$ production at LHC

In this section we first study $t\gamma$ production with the on-shell top quark in the final state, and then discuss the signal and background events for two channels $jjb\gamma$ and $\ell \nu b\gamma$ depending on the W decay mode. A realistic and detailed analysis is presented, including object identification and event selection.

Fig. 1

Leading order Feynman diagrams for anomalous $t\gamma$ production. Flavour violation (the dots) occurs either in the weak (a, b) or strong (c, d) sector

2.1 $t\gamma$ production

From the general effective Lagrangian $\mathcal {L}_\mathrm{eff}$ as shown in Eq. (1), one can see that anomalous couplings $tq\gamma$ and tqg induce the process $pp\rightarrow t\gamma$. The $t\gamma$ production provides the opportunity to probe both anomalous tuV and tcV couplings. Less sensitivity to tcV couplings is expected due to the large enhancement of the charm quark parton distribution function (PDF). Therefore, we consider that each type of interaction, $tu\gamma$, $tc\gamma$ and tug, tcg, should be treated independently. In our analysis, we take ${\Lambda }$ as the top quark mass, $m_{t}=173.2$ GeV, $\alpha _s= 0.108$, $\alpha = 1/128.92$ and a simplified scenario with $\kappa _{tq\gamma }^{L}=\kappa _{tq\gamma }^{R}=\kappa _{tq\gamma }$ and $\kappa _{tqg}^{L}=\kappa _{tqg}^{R}=\kappa _{tqg}$.

The presence of the anomalous couplings $tq\gamma$ and tqg leads to the single top production in association with an energetic photon by two main mechanisms related to the strong and weak sector. This process can take place through the s- and t-channel. The corresponding Feynman diagrams are shown in Fig. 1. The effective cross sections $\sigma (s)$ can be evaluated from $\hat{\sigma }(\hat{s})$ by convoluting with $f_{q_1/p}(x_1)$ and $f_{q_2/p}(x_2)$,

$$\begin{aligned} \sigma (s)=\int ^1_{x_\mathrm{min}} \mathrm{d}x_1 \int ^1_{x_\mathrm{min}/x_1} \mathrm{d}x_2 f_{q_1/p}(x_1) f_{q_2/p}(x_2) \hat{\sigma }(\hat{s}), \end{aligned}$$

(2)

where $\hat{s}=x_1x_2s$ is the effective center-of-mass (c. m.) energy squared for the partonic process, and $x_\mathrm{min}=m_{t}^2/s$. For the quark distribution functions $f_{q_1/p}(x_1)$ and $f_{q_2/p}(x_2)$, we will use the form given by the leading order parton distribution function CT14 [42].

The effective Lagrangian is implemented in FeynRules [43] and subsequently passed to Madgraph5/aMC@NLO [44] framework by means of UFO module [45]. We assume that only one anomalous top coupling is nonzero. It is worth mentioning that if the photon is collinear to the initial quark, the cross section will have a divergence. To avoid this divergence, we set a minimum transverse momentum cut on emitted photons, $p_{T}^{\gamma } > 50$ GeV which is adopted by the CMS collaboration. In this case, the $t\gamma$ cross section at 14 TeV are

$$\begin{aligned} \begin{aligned} \sigma _{t\gamma }(\kappa _{tu\gamma })&= 144.4 |\kappa _{tu\gamma }|^2\ (\mathrm{pb}), \\ \sigma _{t\gamma }(\kappa _{tc\gamma })&= 13.7 |\kappa _{tc\gamma }|^2\ (\mathrm{pb}), \\ \sigma _{t\gamma }(\kappa _{tug})&= 401.6 |\kappa _{tug}|^2\ (\mathrm{pb}), \\ \sigma _{t\gamma }(\kappa _{tcg})&= 55.7 |\kappa _{tcg}|^2\ (\mathrm{pb}). \end{aligned} \end{aligned}$$

(3)

Obviously, the cross sections of $t\gamma$ production only depend on the strengths of anomalous top couplings $\kappa _{tq\gamma }$ and $\kappa _{tqg}$.

Fig. 2

The cross sections of anomalous $t\gamma$ production versus the FCNC branching ratios $\mathcal {B}\mathcal {R}(t\rightarrow u\gamma )$, $\mathcal {B}\mathcal {R}(t\rightarrow c\gamma )$, $\mathcal {B}\mathcal {R}(t\rightarrow ug)$ and $\mathcal {B}\mathcal {R}(t\rightarrow cg)$

Table 1 The most stringent experimental upper bounds on the top quark FCNC branching ratios at 95% CL obtained in CDF, D0, ATLAS and CMS from different channels

There are many alternatives for normalisation of coupling constants in $\mathcal {L}_\mathrm{eff}$. The experimental results always use branching ratios. In order to compare with them, we will show our results by using branching ratios of top quark. In our analysis, the width of $t\rightarrow Wb$ is assumed to be approximately top quark total width. The LO prediction for decay width of top quark decay to a bottom quark and a W boson is [46]

$$\begin{aligned} \Gamma (t \rightarrow Wb) = \frac{\alpha }{16 s_w^2} |V_{tb}|^2 \frac{m_t^3}{m_W^2} \left[ 1-3\frac{m_t^4}{m_W^4}+2\frac{m_t^6}{m_W^6}\right] . \end{aligned}$$

(4)

The partial widths of the top FCNC decays $t\rightarrow q\gamma$ and $t\rightarrow qg$ are expressed as follows:

$$\begin{aligned}&\Gamma (t \rightarrow q\gamma ) = \frac{2 \alpha }{9} m_t^3 \frac{|\kappa _{q\gamma }|^2}{\Lambda ^2}, \nonumber \\&\Gamma (t \rightarrow qg) = \frac{2 \alpha _s}{3} m_t^3 \frac{|\kappa _{qg}|^2}{\Lambda ^2}. \end{aligned}$$

(5)

We plot the cross section of $t\gamma$ production originating from different anomalous couplings tqV versus the FCNC branching ratios in Fig. 2. From Fig. 2, we can see that the cross sections of anomalous $t\gamma$ production increase with the FCNC branching ratios increasing. Experimental limits on the branching ratios of the rare top quark decays were established by experiments of top production at the LEP, HERA, Tevatron, and LHC accelerators [40, 41, 47–52]. At present the most stringent upper limits at 95% confidence level (CL) from different sensitive channels are shown in Table 1. Using the upper limits on the branching ratios by the CMS experiment,

$$\begin{aligned}&\mathcal {B}\mathcal {R}(t\rightarrow u\gamma )< 1.61\times 10^{-4},\ \mathcal {B}\mathcal {R}(t\rightarrow c\gamma ) < 1.82\times 10^{-3},\ \end{aligned}$$

(6)

$$\begin{aligned}&\mathcal {B}\mathcal {R}(t\rightarrow ug)< 3.55\times 10^{-4},\ \mathcal {B}\mathcal {R}(t\rightarrow cg) < 3.44\times 10^{-3}, \end{aligned}$$

(7)

we obtain the limits on the cross section in Fig. 2. Then we use Eq. (3) to calculate the strengths of anomalous top couplings

$$\begin{aligned} \kappa _{tu\gamma }< 0.028,\ \kappa _{tc\gamma }< 0.094,\ \kappa _{tug}< 0.036 ,\ \kappa _{tcg} < 0.112, \end{aligned}$$

(8)

which are coincident with that given by [40, 41].

2.2 Signal and background simulation

Different decay channels of the gauge boson W give different experimental signals. There are two kinds of signals, $\ell \nu b\gamma$ and $jjb\gamma$. We have

$$\begin{aligned} pp \rightarrow t \gamma \rightarrow W^+ b \gamma \rightarrow \ell \nu b\gamma \end{aligned}$$

(9)

and

$$\begin{aligned} pp \rightarrow t \gamma \rightarrow W^+ b \gamma \rightarrow jjb\gamma . \end{aligned}$$

(10)

Table 2 The expected number of events with 100 fb$^{-1}$ integrated luminosity at $\sqrt{s}=14$ TeV and the generated events for all processes are displayed

The leptonic mode of $t\gamma$ production is in general characterized by the presence of an isolated charged leptons (electrons or muons) together with a photon, missing transverse energy, and one b-jet. The $\ell \nu b \gamma$ final state is more attractive from the experimental point of view. On the one hand, it is relatively efficient to be searched by experiments. Everything in the final state could be the targeted objects. They can be reconstructed efficiently by subdetector systems of LHC detectors. The lepton reconstruction efficiency with $P_t > 5$ GeV is more than 90% and particle identification can be made at detector level. The average efficiency of single photon reaches 91%. The b-tagging algorithm could be used with the efficiency of about 70%. On the other hand, the relatively clean signal is robust against the contamination of pileups and underlying events, since primary collision vertices of the signal events can be reconstructed. Moreover, $\sigma (\ell ^{+})/\sigma (\ell ^{-})$ can be used to determine whether $t\gamma$ production comes from the up quark or charm quark initiated process [26, 53]. It provides the opportunity to understand the underlying new physics.

We use Madgraph5/aMC@NLO to generate signal and backgrounds events in a collision energy $\sqrt{s} = 14$ TeV. Higher order correction is taken into account for signal by K factor ($K={\sigma _{NLO}}/{\sigma _{LO}}$), which is equal to 1.8 [24]. For numerical estimation, we take coupling constants $\kappa _{tu\gamma }=0.01$, $\kappa _{tc\gamma }=0.02$, $\kappa _{tug}=0.01$, and $\kappa _{tcg}=0.03$. Parton showering and fast detector simulations are subsequently performed by PYTHIA6 [54] and Delphes3 [55]. Jets are clustered by using the anti-$k_{t}$ algorithm with a cone radius $\Delta R = 0.7$ [56].

As demonstrated in [40, 41], the signal of FCNC $t\gamma$ production might suffer more realistic experimental issues of fake photon and mis-tagged b jet. The probability for jets to reconstruct a single photon candidate in the electromagnetic calorimeter is about 0.1% [57], and the misidentification probability of light quarks or gluons as b-jets is approximately 1.5% [58]. Thus the light jet could be misidentified as b-jet (or photon) candidate and therefore $W\gamma +$jets events will be one of backgrounds with a fake b-jet. Similarly, $W+$jets will contribute to backgrounds if two jets are misidentified as an isolated photon and a b-jet simultaneously, respectively. For the background events with more than one b-jet, we take into account $t\bar{t}$ and $t\bar{t}+\gamma$ and three single top processes with an additional photon. The measurement accuracy of the hadronic calorimeter is not enough to distinguish the W or Z boson. Thereby, $Z\gamma +$jets process also contributes to backgrounds. For the fully hadronic final state, the overwhelming QCD multijet backgrounds are large. We consider the main QCD backgrounds 4j and bjjj for the signal $jjb\gamma$.

The background processes can be roughly categorized into the following three types:

• 1. The multijets background processes include $W+$jets, $W\gamma +$jets, $Z\gamma +$jets. To include the QCD effects, we generate multijet events with up to two jets (three for $W+$jets) that are matched to the parton shower using the MLM-scheme [60] with merging scale $x_{q} = 15$ GeV.

• 2. The top processes include $t\bar{t}$, $t\bar{t}+\gamma$ and three single top production with a photon processes. The reconstructed top mass distribution of signal would be similar to this type of background. The multijets and top background processes contribute to both leptonic and hadronic mode.

• 3. The QCD processes include 4j and bjjj. They only affect the $jjb\gamma$ final state.

The cross sections of these processes are listed in Table 2. It is worthy of remarking that $W+$jets in the categories above is the dominant background for the leptonic mode, which can greatly affect the significance.

In order to select the most relevant events, we introduce the following preselection cuts:

• To trigger the signal events, every event is required to have one isolated photon and one b-jet. Additionally, $N(\ell )=1$ is applied by the leptonic mode of signal, and $N(j)<4$ for hadronic mode. The number of targeted objects in each events can help to suppress the background events effectively, especially to the events with fake particles.

• One of the distinctive signatures of the signal is the presence of a high-$p_{T}$ photon in the final state. The photon is expected to carry large momentum because of the recoil against the heavy top quark. Photon candidates with significant energy are required to have transverse momentum $p_T \ge 50$ GeV with $| \eta | \le 2.5$, using the CMS coordinate system presented [59]. Additionally, only leading jet with $p_{T}>30$ GeV for hadronic mode and $p_{t}(\ell )>20$ GeV with 30 GeV for leptonic mode are considered in our analysis (Table 3).

• The particle flow isolation $\Delta R=\sqrt{(\Delta \eta )^2+(\Delta \phi )^2}<0.4$ around the photon candidate is applied, where $\Delta \eta$ is the rapidity gap and $\Delta \phi$ is the azimuthal angle gap between the particle pair. These cuts on photon ensure the events with exactly one photon candidate. In order to have well separated physical objects and remove radiated photons from high $p_{T}$ leptons or final state partons, it is required that $\Delta R$(jet,$\gamma )>0.7$ and $\Delta R($lepton,$\gamma )>0.7$.

Table 3 The preselection cuts in our analysis are tabulated

Fig. 3

Normalized transverse mass distributions of the $m_{T}(\ell \nu )$ (left) and $m_{T}(\ell \nu b)$ (right) in signal and backgrounds before kinematical cuts at 14 TeV LHC

Table 4 The event numbers of the $\ell \nu b\gamma$ signal and backgrounds with $\mathcal {L}=100$ fb$^{-1}$ and $\sqrt{s}=14$ TeV

Fig. 4

Normalized invariant mass distributions of the m(jj) (left) and m(jjb) (right) in signal and backgrounds before kinematical cuts at 14 TeV LHC

In Fig. 3, we plot the transverse mass windows with the preselection cuts by MadAnalysis 5 [61]. The left figure shows the distribution of the transverse mass of the reconstructed W, and the right figure for reconstructed top. The W transverse mass window can reduce $Z\gamma +$jets backgrounds efficiently while affecting the signal slightly. In the right figure, we see that the transverse mass of signal and top processes show narrower peaky distributions, while the multijet backgrounds distribute in a broad transverse mass region as expected. We could use the top transverse mass window to reduce the multijet backgrounds. From Fig. 3, we require transverse mass cuts as

$$\begin{aligned}&45\,\mathrm{GeV}<m_T(\ell \nu )<85\,\mathrm{GeV},\nonumber \\&130\,\mathrm{GeV}<m_T(\ell \nu b)<190\,\mathrm{GeV}. \end{aligned}$$

(11)

We calculate the statistical significance $S/{\sqrt{(S+B)}}$ for the luminosity of 100 fb$^{-1}$, where S and B denote the number of the signal and background events, respectively. After taking into account transverse mass widow cuts to reject backgrounds, we can further suppress background and gain in the significance up to ${\sim }5\sigma$, respectively, as represented in Table 4.

Similar to leptonic mode, we display m(jj) and m(jjb) distributions of $jjb\gamma$ with preselection cuts in Fig. 4. In the left figure, the sharp peak of signal corresponds to the W mass, while the right part with $m(jj)>100$ GeV is caused by unidentified b quark. Other top background processes represent the same distributions as well. Due to the wide distribution in signal, we claim that the m(jj) cut will not be very effective in improving the significance of the signal. The right figure shows all backgrounds distribute in a broad invariant mass region, while a sharp peaky distribution close to top mass in signal events. As a result, we require the invariant mass m(jjb) to be around the top quark mass window,

$$\begin{aligned} |m(jjb)-m_t|<35\,\mathrm{GeV}. \end{aligned}$$

(12)

Table 5 The event numbers of the $jjb\gamma$ signal and backgrounds with $\mathcal {L}=100$ fb$^{-1}$ at 14 TeV LHC

Fig. 5

Additional Feynman diagrams contributing to the signal of top decay products in association with a photon through tqg vertices

Fig. 6

The cross sections of the radiation processes induced by couplings tug (left) and tcg (right) with $p_{T}^\gamma > 15$ GeV

From Table 5, we can find that the top mass window cut can reduce about 4/5 background events (the rejected rates even reach 85–90% for 4j and $t\gamma +X$), whilst the signal only loses about 1/2 events. However, the observability of the $jjb\gamma$ signal with an integrated luminosity of 100 fb$^{-1}$ at 14 TeV LHC is unpromising, with less than $1\sigma$ level statistical significance.

Fig. 7

Normalized distributions of $m(f\bar{f}b)$ (left) and $m(f\bar{f}b\gamma )$ (right) via different processes with $p_{T}^\gamma > 15$ GeV. The $f\bar{f}$ here denotes W decay products jj or $\ell \nu$

Fig. 8

LHC sensitivity to the considered anomalous top couplings as a function of the coupling strengths after applying the kinematical cuts and event selection

2.3 The contribution of photon radiation to signal

For the signal of anomalous top couplings tqg, $t\gamma$ associated production is not the only contribution to $jjb\gamma$ and $\ell \nu b\gamma$ [20]. The additional Feynman diagrams for $qg \rightarrow f\bar{f}b\gamma$ are depicted in Fig. 5. They correspond to direct top production and photon radiation of the b quark, W boson and W decay products, respectively.

We take account of preselection cuts to examine the contributions of the radiation process. By setting basic cuts and $p_T^\gamma > 15$ GeV, we present the cross sections of additional Feynman diagrams with the strengths of anomalous top couplings in Fig. 6. After setting our preselection cuts, we note that this part of contributions to total cross section have a rapid decrease with the photon transverse momentum increasing. When $p_T^\gamma > 5$ GeV, for $jjb\gamma$ ($\ell \nu b\gamma$), the total cross section which contains the contributions of radiation processes is doubling (tripling) of the $t\gamma$ production with subsequent top decay. When $p_T^\gamma > 40$ GeV, the contributions of additional Feynman diagrams to the cross section quickly decrease to less than 10% of $t\gamma$ production with subsequent top decay. Thus we conclude that the contributions from $t\gamma$ production with subsequent top decay dominate in part of $p_T^\gamma > 50$ GeV.

Table 6 $5\sigma$ ($3\sigma$) discovery lower limits on top quark FCNC anomalous couplings

As a probe to research the top quark FCNC, radiation processes could help us to further understand whether it is induced by strong interactions. In this case, it is necessary to consider not only the final states of a top quark plus a photon but also the final state particles reconstructing a top quark. Thus we present the distributions for the $m(f\bar{f}b)$ and $m(f\bar{f}b\gamma )$ in Fig. 7.

Fig. 9

3$\sigma$ and $5\sigma$ detection potential regions for the $\ell \nu b\gamma$ signal in weak (left) and strong (right) sector at 14 TeV LHC

The contributions from these additional Feynman diagrams could enhance the signature to help us searching for top FCNC process. The radiation processes should be considered as a part of the top FCNC signals.

3 Sensitivity of anomalous top couplings at 14 TeV LHC

In this section, we study the sensitivity of anomalous couplings through the $t\gamma$ production at 14 TeV LHC. For an adequate signal modelling, the photon radiation from top quark decay products is taken into account. The sensitivity of four anomalous top couplings at 14 TeV LHC which is defined as $S/\sqrt{S+B}$ (SS) are presented in Fig. 8. The SS is obtained with the selection strategy talk about in section II. In the case of assuming a single non-vanishing coupling at a time, four choices of anomalous coupling parameters are subsequently fitted by polynomial functions so that ${3}\sigma$ and ${5}\sigma$ discovery ranges are extracted. Assuming that LHC at 14 TeV could collect an integrated luminosity of 100 fb$^{-1}$, we obtain the constraints of one-dimensional discovery limits of anomalous couplings as shown in Table 6.

Obviously, the overwhelming QCD multijet backgrounds make the FCNC coupling constants in hadronic mode looser. The $\ell \nu b\gamma$ signal is more sensitive to search for anomalous couplings at 14 TeV LHC.

Actually, we allow for a set of non-vanishing couplings simultaneously, either in the weak sector (non-vanishing $\kappa _{tu\gamma }$ and $\kappa _{tc\gamma }$) or in the strong sector (non-vanishing $\kappa _{tug}$ and $\kappa _{tcg}$). In order to illustrate excluded detection potential regions of anomalous couplings to reach a given statistical significance, we plot the associated $3\sigma$ and $5\sigma$ discovery reaches in the $\kappa _{tu\gamma }$–$\kappa _{tc\gamma }$ ($\kappa _{tug}$–$\kappa _{tcg}$) planes for $\ell \nu b\gamma$ at 14 TeV LHC in Fig.  9. We observe a better sensitivity to flavour-changing interactions with an up quark than with a charm quark, as expected from parton densities, the charm content of the proton being suppressed with respect to its up content.

Since both $tq\gamma$ and tqg operators contribute to the same final state, the interference effects should be considered. If $\kappa _{tu\gamma }=\kappa _{tc\gamma }=\kappa _{tq\gamma }$, $\kappa _{tug}=\kappa _{tcg}=\kappa _{tqg}$, for $p_T^\gamma > 50$ GeV, the total $t\gamma$ cross section with contributions of $tq\gamma$ and tqg operators is

$$\begin{aligned} \sigma _{t\gamma } = 158.2\ \big |\kappa _{tq\gamma }|^2 + 457.3\ \big |\kappa _{tqg}|^2 + 153\ \kappa _{tq\gamma }\cdot \kappa _{tqg} \ (\mathrm{pb}). \end{aligned}$$

(13)

Fig. 10

3$\sigma$ and $5\sigma$ discovery ranges at 14 TeV LHC in the $\kappa _{tq\gamma }-\kappa _{tqg}$ plane

By applying the same selection strategy described as above, we presented excluded 14 TeV LHC detection potential regions for the $\ell \nu b\gamma$ signal in $tq\gamma$ and tqg plane in Fig. 10. Compared to existing research as regards tZ production and same-sign top quark production [27, 32], our results give more sensitive constraints on top anomalous couplings via $t\gamma$ production.

4 Conclusions

Many of the extensions of the SM predict that the tree-level FCNC processes could exist. With Run-II of the LHC, more and more measurements in the top quark sector will be explored with an unprecedented precision. Measurements of single top production allow us to search for deviations from the SM predictions. While these deviations are often interpreted in terms of anomalous top couplings. The possible deviations can be described by the effects of effective operators, and experimental results can be used to determine useful constraints on each effective operator. The established deviations can then be evolved up to high scales, and matched to possible new physics scenarios.

In this paper, we have investigated FCNC couplings $tq\gamma$ and tqg in a model-independent way. These interactions lead to possibly significant production rates for $jjb\gamma$ and $\ell \nu b\gamma$ signals via $t\gamma$ associated production with subsequent decay. We have considered the contribution of the single top production with photon radiation off the top decay products to this process. Once $t\gamma$ production processes are eventually discovered, the single top decay with photon radiation processes could help us find out whether or not the origin of the FCNC interactions is in the strong sector. This contribution dominates when $p_T^\gamma$ is small, and it quickly decreases with the photon transverse momentum increasing. For our cuts, the contributions of these processes could enhance the signature, so we include these processes in signal for our analysis of the sensitivity of anomalous couplings.

The sensitivities of 14 TeV LHC to anomalous FCNC couplings $tq\gamma$ and tqg were calculated for both the $\ell \nu b\gamma$ and the $jjb\gamma$ signals. Due to the overwhelming QCD multijet backgrounds, it is challenging to discover the FCNC $t\gamma$ production via the hadronic mode explored at LHC. We found that it is most promising to observe anomalous top couplings via leptonic mode of $t\gamma$ production at the LHC. We further discussed the interference effects on the cross section from contributions of $tq\gamma$ and tqg operators. Then we presented excluded detection potential regions for the $\ell \nu b\gamma$ signal. With an integrated luminosity of 100 fb$^{-1}$ and $\sqrt{s}=14$ TeV, for a ${5}\sigma$ discovery, the strengths needed of $tq\gamma$ and tqg couplings that are in order are down to 0.001–0.01. We hope our results could help search for the signal of anomalous top couplings at 14 TeV LHC in operation.