Search for flavour-changing neutral-current couplings between the top quark and the Higgs boson in multi-lepton final states in 13 TeV pp collisions with the ATLAS detector

Abstract

A search is presented for flavour-changing neutral-current interactions involving the top quark, the Higgs boson and an up-type quark ($q=u,c$) with the ATLAS detector at the Large Hadron Collider. The analysis considers leptonic decays of the top quark along with Higgs boson decays into two $W$ bosons, two $Z$ bosons or a $\tau ^{+}\tau ^{-}$ pair. It focuses on final states containing either two leptons (electrons or muons) of the same charge or three leptons. The considered processes are $t\bar{t}$ and Ht production. For the $t\bar{t}$ production, one top quark decays via $t\rightarrow Hq$. The proton–proton collision data set analysed amounts to $({140}\,{\hbox {fb}^{-1}})$ at $(\sqrt{s}={13}\,\hbox {TeV})$. No significant excess beyond Standard Model expectations is observed and upper limits are set on the $t\rightarrow Hq$ branching ratios at 95 % confidence level, amounting to observed (expected) limits of $\mathcal {B}(t\rightarrow Hu)<2.8\,(3.0) \times 10^{-4}$ and $\mathcal {B}(t\rightarrow Hc)<3.3\,(3.8) \times 10^{-4}$. Combining this search with other searches for $tHq$ flavour-changing neutral-current interactions previously conducted by ATLAS, considering $H\rightarrow b\bar{b}$ and $H\rightarrow \gamma \gamma$ decays, as well as $H\rightarrow \tau ^{+}\tau ^{-}$ decays with one or two hadronically decaying $\tau$-leptons, yields observed (expected) upper limits on the branching ratios of $\mathcal {B}(t\rightarrow Hu)<2.6\,(1.8) \times 10^{-4}$ and $\mathcal {B}(t\rightarrow Hc)<3.4\,(2.3) \times 10^{-4}$.

1 Introduction

Following the discovery of the Higgs boson at the Large Hadron Collider (LHC) by the ATLAS and CMS experiments in 2012 [1, 2], several measurements have probed properties of the particle. In addition to Higgs boson couplings to fermions and gauge bosons, which agree well with the Standard Model (SM) predictions [3, 4], further interactions can be investigated to search for evidence of possible new physics in the Higgs sector. A possibility is that the Higgs boson has an interaction involving two up-type quarks of different generations, denoted by $tHq$ where $q=(u,c)$, leading to $gq\rightarrow Ht$ production and $t\rightarrow Hq$ decay. These are flavour-changing neutral-current (FCNC) interactions, forbidden at tree-level in the SM and suppressed at higher orders due to the Glashow–Iliopoulos–Maiani (GIM) mechanism [5]. The SM predictions for the $t \rightarrow Hq$ branching ratios $\mathcal {B}(t \rightarrow Hq)$ are very small, around $10^{-15}$ [6,7,8,9,10], and beyond the sensitivity of the LHC. However, in the framework of new physics models such as two-Higgs-doublet models (2HDMs) [11], the minimal supersymmetric SM (MSSM) [12,13,14,15], supersymmetric models with R-parity violation [16], quark-singlet models [17], and warped extra dimensions [18], the branching ratios of these processes are modified making them large enough to be measured at the LHC. For example, branching ratios of up to $10^{-3}$ are possible in 2HDMs without explicit flavour symmetry, since there are no symmetries explicitly forbidding tree-level FCNC interactions [19,20,21,22,23,24,25,26] in this model. The FCNC interaction can also mediate the production of a Higgs boson with a single top quark ($pp \rightarrow tH$) [27]. The SM prediction of the $pp \rightarrow tH$ cross-section is $74.3^{+0.4}_{-0.3}$ fb at $\sqrt{s} = {13}\hbox {TeV}$ [28].

The FCNC interactions are introduced using an effective field theory (EFT) framework, which is used for indirect searches for new physics [29]. Here the SM is regarded as a low-energy approximation of an ultraviolet complete theory containing new particles, whose masses are characterised by an energy scale $\Lambda = {1}\,\hbox {TeV}$. The new physics contributions are parameterised in terms of operators with mass dimension greater than four containing only the SM fields, scaled by dimensionless Wilson coefficients and inverse powers of $\Lambda$. In the case where only the $tHu$ and $tHc$ interactions are considered, the relevant operators are

$$\begin{aligned}&\mathcal {O}_{u\phi }^{qt} =\left( \phi ^\dagger \phi -\frac{v^2}{2}\right) (\bar{q}_L\,t_R)\tilde{\phi } \nonumber \\&\mathcal {O}_{u\phi }^{tq} =\left( \phi ^\dagger \phi -\frac{v^2}{2}\right) (\bar{t}_L\,q_R)\tilde{\phi }, \end{aligned}$$

(1)

where q corresponds to an up or charm quark, depending on the FCNC coupling. The index u is the coupling to any up-type quark, t is the top-quark, $\phi$ denotes the Higgs boson field with v corresponding to the absolute value of its vacuum expectation value. The two left-handed quark doublet fields are $\bar{q}_L$ and $\bar{t}_L$, with $q_R$ and $t_R$ being the corresponding right-handed singlets. The operators are scaled with Wilson coefficients $C_{u\phi } ^{qt}$ and $C_{u\phi } ^{tq}$, and $1/\Lambda ^2$ to give the relevant Lagrangian:

$$\begin{aligned} \mathcal {L}_{\text {EFT}}&=\mathcal {L}_{\text {SM}}+\sum \limits _{q=u,c}\left[ \frac{C_{u\phi }^{qt}}{\Lambda ^2}\mathcal {O}_{u\phi }^{qt} +\frac{C_{u\phi }^{tq}}{\Lambda ^2}\mathcal {O}_{u\phi }^{tq} \right] . \end{aligned}$$

(2)

In the case of $t\bar{t}$ production with a FCNC $t\rightarrow Hq$ decay, no kinematic differences between $\mathcal {O}_{u\phi }^{qt}$ and $\mathcal {O}_{u\phi }^{tq}$ are expected because the top quarks are produced unpolarised and the Higgs boson is a scalar particle. For $t\rightarrow Hq$ production, comparisons of simulations using either $\mathcal {O}_{u\phi }^{qt}$ or $\mathcal {O}_{u\phi }^{tq}$ have shown that differences between production rates and kinematic distributions are negligible in the phase space considered. Thus, the mean of both the couplings $C_{u\phi }^{qt,tq} =\frac{C_{u\phi }^{qt} +C_{u\phi }^{tq}}{2}$ is considered.

Both the ATLAS and CMS Collaborations have undertaken various searches for $tHq$ FCNC processes, split by Higgs boson final state [30, 30,31,32,33,34,35,36,37,38]. In this paper, a search is conducted using final states containing either two leptons (electrons or muons) of the same charge ($2\ell \text {SS}$, $\ell =e,\mu$) or three leptons, exactly two of which have the same charge ($3\ell$, $\ell =e,\mu$). The ATLAS Collaboration searched for $tHq$ FCNC couplings in $2\ell \text {SS}$ and $3\ell$ final states using a partial Run 2 data sample of ${36.1}\,\hbox {fb}^{-1}$ [32], resulting in observed (expected) 95% CL upper limits of $\mathcal {B}(t \rightarrow Hu)<19(15)\times 10^{-4}$ and $\mathcal {B}(t \rightarrow Hc)<16(15)\times 10^{-4}$. The strongest limits set by the ATLAS Collaboration come from a combination of searches in the $H \rightarrow b\bar{b}$, $H \rightarrow \gamma \gamma$ and $H\rightarrow \tau ^{+}\tau ^{-}$ channels using the full Run 2 data sample [35]. The obtained limits are $\mathcal {B}(t \rightarrow Hu)<4.0(2.4)\times 10^{-4}$ and $\mathcal {B}(t \rightarrow Hc)<5.8(3.0)\times 10^{-4}$. For the $tHc$ coupling, these are the most stringent limits published to date. However, for the $tHu$ coupling the strongest limits are set by a search conducted with the CMS detector in the $H\rightarrow \gamma \gamma$ channel [38], amounting to $\mathcal {B}(t \rightarrow Hu)<1.9(3.1)\times 10^{-4}$.

The dominant Higgs-boson decay mode resulting in $2\ell \text {SS}$ and $3\ell$ final states across production and decay processes is $H \rightarrow WW^*$, with $WW^* \rightarrow \ell \nu jj$ or $WW^* \rightarrow \ell \nu \ell \nu$ for the $2\ell \text {SS}$ and $3\ell$ final state respectively. The corresponding Feynman diagrams are shown in Figs. 1, 2. Other Higgs-boson decay modes such as $H \rightarrow ZZ^*$ and $H \rightarrow \tau ^{+}\tau ^{-}$ can also meet the selection criteria. Based on the SM branching ratios of the Higgs boson and the top quark, approximately 73 % of all $2\ell \text {SS}$ signal events are expected from the $H\rightarrow WW^*$ decay mode, while the rest originate from leptonic $H\rightarrow \tau ^{+} \tau ^{-}$ decays. Less than 1 % of events are attributable to the $H\rightarrow ZZ^*$ decay mode. In the $3\ell$ final state, the latter contributes more significantly, being responsible for 14 % of all events. The majority, accounting for 54 % of the $3\ell$ signal events, originate from $H\rightarrow WW^*$ decays, while the remaining 32 % arise from $H\rightarrow \tau ^{+} \tau ^{-}$ events. These numbers do not take into account detector acceptance effects or selection requirements. Although the total number of events meeting the $2\ell \text {SS}$ and $3\ell$ criteria is modest, the low statistical precision is balanced by minimal background contributions.

Fig. 1

Example Feynman diagrams of the $t\bar{t} (t\rightarrow Hq)$ decay signal process resulting in a the $2\ell \text {SS}$ final state and b the $3\ell$ final state. The FCNC vertex is highlighted in red

Fig. 2

Example Feynman diagrams of the $gq\rightarrow Ht$ production signal process resulting in a the $2\ell \text {SS}$ final state and b the $3\ell$ final state. The FCNC vertex is highlighted in red

The strategy of the analysis is to first identify kinematic regions enriched in the signal process. In these regions, various reconstruction algorithms are implemented to create variables that can distinguish between the signal and several background processes. These reconstructed variables are combined into a single discriminant using a feed-forward neural network [39]. The same kinematic regions are used for the $tHu$ and the $tHc$ channel of the analysis, while individual neural networks are trained for each of the channels. Aside from this, the analysis strategy is identical for the $tHu$ and the $tHc$ channel. The distribution of the neural network output is then used as input to a maximum-likelihood fit, which considers statistical and systematic uncertainties. In addition to the signal-enriched regions, several regions enriched in specific background processes are included in the fit, to constrain the normalisation of these processes. If the fit shows evidence of a signal, the corresponding significance is determined. Otherwise, upper limits are set on the FCNC branching ratios and the Wilson coefficients of the EFT dimension-6 operators. Finally, the results of this analysis are statistically combined with those from other ATLAS searches for $tHq$ FCNC interactions in different final states.

2 The ATLAS detector

The ATLAS detector [40] at the LHC covers nearly the entire solid angle around the collision point.¹ It consists of an inner tracking detector surrounded by a thin superconducting solenoid, electromagnetic and hadron calorimeters, and a muon spectrometer incorporating three large superconducting air-core toroidal magnets.

The inner-detector system (ID) is immersed in a ${2}\,\hbox {T}$ axial magnetic field and provides charged-particle tracking in the range of $|\eta | < 2.5$. The high-granularity silicon pixel detector covers the vertex region and typically provides four measurements per track, the first hit normally being in the insertable B-layer (IBL) installed before Run 2 [41, 42]. It is followed by the silicon microstrip tracker (SCT), which usually provides eight measurements per track. These silicon detectors are complemented by the transition radiation tracker (TRT), which enables radially extended track reconstruction up to $|\eta | = 2.0$. The TRT also provides electron identification information based on the fraction of hits (typically 30 in total) above a higher energy-deposit threshold corresponding to transition radiation.

The calorimeter system covers the pseudorapidity range $|\eta | < 4.9$. Within the region $|\eta |< 3.2$, electromagnetic calorimetry is provided by barrel and endcap high-granularity lead/liquid-argon (LAr) calorimeters, with an additional thin LAr presampler covering $|\eta | < 1.8$ to correct for energy loss in material upstream of the calorimeters. Hadron calorimetry is provided by the steel/scintillator-tile calorimeter, segmented into three barrel structures within $|\eta | = 1.7$, and two copper/LAr hadron endcap calorimeters. The solid angle coverage is completed with forward copper/LAr and tungsten/LAr calorimeter modules optimised for electromagnetic and hadronic energy measurements respectively.

The muon spectrometer (MS) comprises separate trigger and high-precision tracking chambers measuring the deflection of muons in a magnetic field generated by the superconducting air-core toroidal magnets. The field integral of the toroids ranges between 2.0 and 6.0 T m across most of the detector. Three layers of precision chambers, each consisting of layers of monitored drift tubes, cover the region $|\eta | < 2.7$ and are complemented by cathode-strip chambers in the forward region, where the detector occupancy is highest. The muon trigger system covers the range of $|\eta | < 2.4$ with resistive-plate chambers in the barrel, and thin-gap chambers in the endcap regions.

Recorded events are selected by the first-level trigger system implemented in custom hardware, followed by selections made by algorithms implemented in software in the high-level trigger [43]. The first-level trigger accepts events from the ${40}\,\hbox {MHz}$ bunch crossings at a rate below ${100}\,\hbox {kHz}$, which the high-level trigger further reduces to record events to disk at about ${1}\,\hbox {kHz}$.

An extensive software suite [44] is used in data simulation, in the reconstruction and analysis of real and simulated data, in detector operations, and in the trigger and data acquisition systems of the experiment.

3 Samples of data and simulated events

Proton–proton (pp) collision data are used recorded with the ATLAS detector in the years 2015 to 2018 at a centre-of-mass energy of 13$\,\hbox {TeV}$. After applying data-quality requirements [45], the data sample corresponds to an integrated luminosity of ${140.1(12)}\,\hbox {fb}^{-1}$ [46]. The LUCID-2 detector [47] was used for the primary luminosity measurements, complemented by measurements using the inner detector and calorimeters.

Events were selected online during data taking by single-electron or single-muon triggers [48, 49]. Multiple unprescaled triggers were combined in a logical OR to increase the selection efficiency. The lowest-threshold triggers utilised isolation requirements to reduce the trigger rate. The higher-level lepton triggers had transverse momentum ($p_{\text {T}}$) thresholds of $20\,\hbox {GeV}$ for muons and $24\,\hbox {GeV}$ for electrons in 2015 data, and $26\,\hbox {GeV}$ for both the lepton types in 2016, 2017 and 2018 data. They were complemented by other triggers with higher $p_{\text {T}}$ thresholds but no isolation requirements to increase the trigger efficiency.

Large sets of simulated events from signal and background processes were produced with Monte Carlo (MC) event generator programs to model the selected data. After event simulation, the response of the ATLAS detector was simulated using the Geant4 toolkit [50] with a full detector model [51] or a fast simulation [51, 52] which employed a parameterisation of the calorimeter response. Samples relying on fast simulation were used to evaluate systematic uncertainties in the event generators and for the modelling of $t\bar{t}$ production with two bosons.

To account for additional inelastic $pp$ collisions in the same and neighbouring bunch crossings (pile-up), minimum-bias interactions were overlaid on the hard-scattering events at the level of energy depositions simulated using Geant4. The minimum-bias events were simulated using Pythia [8.186] [53] with the A3 [54] set of tuned parameters and the NNPDF[2.3lo] set of parton distribution functions (PDF) [55]. The resulting events were weighted to reproduce the observed pile-up distribution. The average number of interactions per bunch crossing during the entire data-taking period from 2015 to 2018 is 33.7.

Finally, the simulated events were reconstructed using the same software as applied to the collision data. The same event selection requirements were applied and the selected events were passed through the same analysis chain. Small corrections were applied to simulated events such that the efficiencies of the utilised reconstruction methods are in better agreement with the response observed in data. More details of the simulated event samples are provided in the following subsections. Except for the events simulated with the Sherpa generator [56, 57], the Evtgen [58] program was used to simulate bottom and charm hadron decays. If not mentioned otherwise, the top-quark mass was set to $m_t={172.5\,\mathrm{\,}}\hbox {GeV}$ and the Higgs-boson mass to $m_H={125\,\mathrm{\,}}\hbox {GeV}$. All processes for which the parton shower is simulated using Pythia[8] use the A14 set of tuned parameters [59] and the NNPDF[2.3lo] PDF set.

3.1 Simulation of ${tHq}$ FCNC signal samples

Two different signal processes are studied: the $t\bar{t} (t\rightarrow Hq)$ decay and the $gq\rightarrow Ht$ production process. The EFT operators parametrising the FCNC coupling are implemented in the TopFCNC [60] model using the Feynrules[2.0] framework [61], allowing for next-to-leading-order ($\text {NLO}$) calculations of the postulated processes. The decay process events were simulated using the $\text {NLO}$ matrix-element generator Powhegbox[v2] [62,63,64,65,66,67,68], with the NNPDF[3.0nlo] PDF set [69]. The production of a top quark–antiquark pair is performed according to SM calculations in the five-flavour scheme, setting the masses of all quarks except for the top quark to zero. One top quark is required to decay via the SM decay mode $t\rightarrow Wb$, while the other performs the FCNC decay $t\rightarrow Hq$. Both top-quark decays are modelled using Madspin [70, 71]. Only leptonic final states for the SM decay of the top quark are considered. The Higgs-boson decay, and parton showers, hadronisation, and the underlying event, were modelled using Pythia[8.308] [72]. The matrix-element-to-parton-shower matching is steered by the $h_\textrm{damp}$ parameter, which controls the $p_{\text {T}}$ of the first additional gluon emission beyond the leading-order ($\text {LO}$) Feynman diagram in the parton shower and therefore regulates the high-$p_{\text {T}}$ emission against which the $t\bar{t}$ system recoils. The event generation was performed with $h_\textrm{damp} =1.5m_t$ [73]. Four separate samples are produced, modelling the $tHu$/ $tHc$ coupling with either the top quark or the top antiquark decaying via the FCNC interaction. For consistency across searches, all $tHq$ FCNC samples are normalised to the cross-section corresponding to a branching ratio of $\mathcal {B}(t\rightarrow Hq)={0.1\,\mathrm{\%}}$. As explained previously, kinematic differences between the left- and right-handed couplings are negligible in the case of $t\rightarrow Hq$ decay process. Therefore, the same samples are used to model the left-handed and the right-handed FCNC couplings. The matrix element generator Mgnlo[2.9.9] [74] was used to model the $gq\rightarrow Ht$ production signal using the TopFCNC model with the NNPDF[3.0nlo] PDF set and the five-flavour scheme. Separate samples are produced, each with exactly one of the Wilson coefficients set to 1.0 and the others set to zero, all of them at $\text {NLO}$ in QCD.

The SM top-quark and Higgs-boson decays are simulated using Madspin. The three relevant Higgs-boson decays ($H\rightarrow WW^*$, $H\rightarrow ZZ^*$, $H\rightarrow \tau ^{+}\tau ^{-}$) are modelled in separate samples. All Higgs-boson decays, parton shower, hadronisation, and the underlying event, were modelled using Pythia[8.307]. The samples are normalised to the cross-section calculated with Mgnlo.

3.2 Simulation of background processes

Samples of simulated events from SM $t\bar{t}$ and single-top-quark production were simulated using the $\text {NLO}$ matrix-element generator Powhegbox[v2]. For $t\bar{t}$ and $tW$ production and s-channel single-top-quark production ($t\bar{b}$ production) the Nnpdf[3.0nlo] PDF set was used with the five-flavour scheme. Following a recommendation given in Ref. [68], single top-quark production in the $t$-channel (tq production) was simulated with the Nnpdf[3.0nlo_nf4] PDF set, which implements the four-flavour scheme. Parton showers, hadronisation, and the underlying event were modelled using Pythia[8.230]. The event simulation used a value of $h_\textrm{damp} =1.5m_t$.

The $t\bar{t}$ production cross-section was scaled to $\sigma (t\bar{t})=832\,$pb, the value obtained from next-to-next-to-leading-order predictions from the Top++2.0 program (see Ref. [75] and references therein), and it includes the resummation of next-to-next-to-leading logarithmic soft-gluon terms. The predicted cross-sections of $tq$ and $\bar{t} q$ production used to scale the corresponding samples of simulated events are $\sigma (tq)=136\,$pb and $\sigma (\bar{t}q)=81\,$pb and were calculated at $\text {NLO}$ in QCD with the Hathor[2.1] program [76, 77]. The total cross-section for $t\bar{b}$ production was also computed at $\text {NLO}$ in QCD with the Hathor[2.1] program and the corresponding sample of simulated events was scaled to $\sigma (t\bar{b} +\bar{t} b)=10.32\,$pb. The cross-section used for normalising the $tW$ sample is $\sigma (tW+\bar{t} W)=71.7\,$pb [78].

The production of a vector boson in association with jets (V+jets, $V=W,Z$), including b- and c-jets, was simulated with the Sherpa[2.2.11] generator. NLO-accurate matrix elements for up to two partons and LO-accurate matrix elements for three to five partons are calculated in the five-flavour scheme using the Comix [79] and Openloops [80,81,82] libraries. The default Sherpa parton shower [83] based on Catani–Seymour dipole factorisation and the cluster hadronisation model [84] are used. The samples are simulated using a dedicated set of tuned parameters developed by the Sherpa authors and use the NNPDF[3.0nnlo] PDF set.

The $t\bar{t}W$ process was simulated using the Sherpa[2.2.10] generator. The matrix elements were calculated for up to one additional parton at $\text {NLO}$ and up to two partons at $\text {LO}$ using Comix and Openloops, and merged with the Sherpa parton shower using the MEPS@NLO prescription [85]. In addition to the nominal prediction at $\text {NLO}$ in QCD, higher-order corrections related to electroweak (EWK) contributions are also included. Event-by-event correction factors are applied that provide virtual $\text {NLO}$ EWK corrections to $\mathcal {O}(\alpha ^{2}\alpha _{\text {s}} ^{2})$ and $\text {LO}$ corrections to $\mathcal {O}(\alpha ^3)$ [57, 86, 87]. An independent Sherpa[2.2.10] sample was produced at $\text {LO}$ to account for the sub-leading EWK corrections to $\mathcal {O}(\alpha ^{3}\alpha _{\text {s}})$ [88]. The $\text {NLO}$ QCD and $\text {NLO}$ EWK contributions from Sherpa are combined following the method of Ref. [89]. The $t\bar{t}W$ samples are normalised using a cross-section of 722  fb computed at $\text {NLO}$ including the hard non-logarithmically enhanced radiation at $\text {NLO}$ in QCD [89].

The modelling of the production of a top quark–antiquark pair with a Z boson decaying leptonically is done using the Mgnlo[2.8.1] generator. It provides matrix elements at $\text {NLO}$ in QCD, employing the NNPDF[2.3lo] PDF set. The generated events are interfaced with Pythia[8.244] for the parton shower and hadronisation stages. The $t\bar{t}Z/\gamma ^*(\rightarrow \ell ^+\ell ^-)$ prediction was normalised to the calculation at $\text {NLO}$ QCD and NLO EWK accuracy based on Ref. [28] with the additional inclusion of off-shell effects, corresponding to 162  fb.

The production of $t\bar{t}H$ events was modelled using the Powhegbox[v2] generator at $\text {NLO}$ in QCD with the NNPDF[3.0nlo] PDF set. The events were interfaced to Pythia[8.230]. The cross-section was calculated at $\text {NLO}$ QCD and $\text {NLO}$ EWK using Mgnlo and amounts to $507\,\hbox {fb}$.

The production of two or three weak vector bosons ($VV$, $VVV$) in $2\ell$, $3\ell$ and $4\ell$ final states was simulated using the Sherpa[2.2.12] generator including the off-shell contributions. Matrix elements including additional partons were calculated at $\text {NLO}$ in QCD for up to one parton and at $\text {LO}$ accuracy for two or three additional partons using Comix and Openloops. For the parton shower simulation, the same method as in the $W$+jets and $Z$+jets samples is used. The $VV$ and $VVV$ samples are normalised to the total cross-sections provided by Sherpa.

The $tZq$ process was modelled at $\text {NLO}$ with the Mgnlo [2.9.5] generator and the NNPDF[3.0nlo] PDF. The generated events were interfaced with Pythia[8.230]. Matrix elements of the $tWZ$ process were also calculated with Mgnlo[2.2.2], using the same PDF set. The events were interfaced with Pythia[8.212] using the same tune and PDF set as used for $tZq$ production. The samples were normalised to the theoretical cross-section at $\text {NLO}$ QCD.

Some rare processes are considered: $t\bar{t}t$, $t\bar{t}t\bar{t}$, $tHW$, $tHq$, $t\bar{t}WW$, $t\bar{t}HH$, $t\bar{t}WH$, $t\bar{t}ZZ$, $t\bar{t}WZ$ and $VH$ production. The processes involving at least one top quark were simulated with Mgnlo, while $VH$ was produced with Powhegbox[v2]. Simulated events of all processes were interfaced to Pythia[8] to simulate parton showers and hadronisation. The processes are normalised to their predicted $\text {NLO}$ cross-sections. Their combined contribution is nearly negligible, ranging from 0.1 to 1 %, depending on the kinematic region considered.

4 Object reconstruction and event preselection

Events are required to have at least one vertex reconstructed from at least two ID tracks with transverse momenta of $p_{\text {T}} >0.5\,\hbox {GeV}$. The primary vertex of an event is defined as the vertex with the highest $p_{\text {T}} ^2$ summed over the ID tracks [90] matched to it.

Electron candidates are reconstructed by matching a track in the ID to clusters of energy deposits in the electromagnetic calorimeter [91]. The pseudorapidity of clusters, $\eta _{\textrm{cluster}}$, is required to be in the range of $|\eta _{\textrm{cluster}}|<2.47$. However, clusters are excluded if they are in the transition region $1.37<|\eta _{\textrm{cluster}}|<1.52$ between the barrel and endcap electromagnetic calorimeters. Electron candidates must have $p_{\text {T}} >10\,\hbox {GeV}$. A likelihood-based discriminant is constructed to simultaneously evaluate several properties of electron candidates, including shower shapes in the electromagnetic calorimeter, track quality, and the detection of transition radiation produced in the TRT [92]. Applying a discriminant requirement enhances the selection of prompt electrons produced in $W$-, $Z$- or Higgs-boson decays and leptonic $\tau$-lepton decays, while effectively rejecting photon conversions and hadrons misidentified as electrons. All selected electrons must meet the Tight selection criteria as defined in Ref. [92]. ig Muon candidates are reconstructed by combining tracks in the MS with tracks in the ID [93]. The tracks must be in the range of $|\eta |<2.5$ and have $p_{\text {T}} >10\,\hbox {GeV}$. Similarly to electrons, likelihood-based identification criteria are applied [93]. Muons are required to satisfy the Medium selection defined in Ref. [93].

The tracks matched to electron and muon candidates must point to the primary vertex, which is ensured by requirements imposed on the transverse impact-parameter significance, $|d_{0}/\sigma (d_{0})|<5.0$ for electrons and $|d_{0}/\sigma (d_{0})|<3.0$ for muons, and on the longitudinal impact parameter, $z_0$, for which $|z_0\sin (\theta )|<0.5$ mm has to be satisfied for both lepton flavours. Non-prompt electrons and muons are leptons produced by mechanisms other than $W$-, $Z$- or Higgs-boson decays and leptonic $\tau$-lepton decays. They are effectively rejected by using multivariate discriminants computed with boosted decision trees (BDT), which integrate electromagnetic shower shapes and track information from the ID [94]. Separate BDTs are trained for electrons in the barrel ($|\eta |<1.37$) and endcap ($|\eta |>1.37$) regions, while a single BDT is employed for muons. The efficiency for correctly identifying prompt muons (electrons) ranges from approximately 80 % to 95 % (65 % to 90 %) for $p_{\text {T}}$ values between 20  GeV and 45  GeV, with the efficiency plateauing beyond 45  GeV. Overall, the BDTs achieve a 71 % (90 %) rejection rate for muons (electrons) from $B$-hadron decay. Additionally, the contribution of electrons reconstructed with an incorrectly reconstructed charge is significantly reduced by using an additional BDT [91], which consolidates information about an electron candidate’s charge, impact parameter, energy, and ID track into a single discriminant. This achieves a 95 % efficiency for electrons with a correct charge assignment while rejecting 94 % of charge-misidentified electrons.

Scale factors are used to correct the efficiencies in simulation to match the efficiencies measured for the electron [48] and muon [49] triggers, and the reconstruction, identification and isolation criteria [91, 93].

Jets are reconstructed from particle-flow objects [95] with the anti-$k_{t}$ clustering algorithm [96, 97] using a radius parameter of 0.4. This algorithm matches topological energy clusters [98] in the calorimeters to selected tracks in the ID. The energy of tracks is subtracted from the matched topological clusters and both the tracks and the energy-subtracted topological clusters are used as input to the jet reconstruction. The jet energy is calibrated by applying several simulation-based corrections and techniques correcting for differences between simulation and data [99]. The jets must satisfy $p_{\text {T}} > {20\,\mathrm{\,}}\hbox {GeV}$ and $|\eta |<2.5$.

To suppress jets originating from pile-up collisions, several track-based variables are combined with a multivariate technique to form the jet-vertex-tagger (JVT) discriminant [100]. Jets with $p_{\text {T}} < {60\,\mathrm{\,}}\hbox {GeV}$ and $|\eta |<2.4$ are required to have a JVT discriminant above 0.5, which corresponds to an efficiency of $92\%$ for jets from the primary vertex, while 98 % of jets from pile-up events are rejected.

Identification of jets containing $B$ hadrons ($b$-tagging) is performed with the DL1r algorithm, which uses a deep feed-forward neural network with several $b$-tagging algorithms as inputs [101]. These input algorithms exploit the impact parameters of charged-particle tracks, the properties of reconstructed secondary vertices and the topology of $b$- and $c$-hadron decays inside the jets. The requirement on the DL1r discriminant is chosen such that the average tagging efficiency of $b$-jets from simulated dileptonic $t \bar{t}$ events is ${70\,\mathrm{\%}}$. The corresponding $p_{\text {T}}$-dependent c-jet rejection factors range from 10 to 14, while those for light-flavour jets range from 100 to 900. Differences between the $b$-tagging efficiency between collision data and simulation are corrected using simulation-to-data scale factors derived from $t\bar{t}$ events. The scale factors depend on the $p_{\text {T}}$ of the jets and are consistent with unity within the uncertainties. The obtained scale factors depend on the parton-shower generator used to produce the $t\bar{t}$ samples. When using samples produced with a different parton-shower generator, for example Sherpa, to model $W$+jets events, or when evaluating systematic uncertainties with a setup based on Herwig, additional correction factors called MC-to-MC scale factors are applied. Since the DL1r algorithm uses measurements from the ID, the identification of $b\text {-jets}$ is limited to the region with $|\eta | < 2.5$.

With the above definitions of physical objects it is possible for ambiguities to arise, since all objects are reconstructed independently of each other. To avoid the double-counting of candidates which satisfy more than one selection criterion, a procedure called overlap removal is applied. Reconstructed physical objects are removed in the following order: electrons sharing an ID track with a muon; jets within $\Delta R = 0.2$ of an electron, thereby avoiding double-counting electron energy deposits as jets; electrons within $\Delta R = 0.4$ of a remaining jet, for reducing the impact of non-prompt electrons; jets within $\Delta R = 0.2$ of a muon if they have two or fewer matched tracks; muons within $\Delta R = 0.4$ of a remaining jet, reducing the rate of non-prompt muons.

The missing transverse momentum $\vec {p}_\textrm{T}^\mathrm {\,miss}$ is reconstructed as the negative vector sum of the transverse momentum of the reconstructed leptons and jets, and ID tracks that point to the primary vertex but are not matched to a reconstructed object [102]. The magnitude of $\vec {p}_\textrm{T}^\mathrm {\,miss}$ is denoted by $E_{\text {T}}^{\text {miss}}$.

Events selected by this analysis are required to have at least one charged lepton with $p_{\text {T}} >{28\,\mathrm{\,}}\hbox {GeV}$. Additional leptons must satisfy a $p_{\text {T}}$ threshold of 10  GeV. In the $2\ell \text {SS}$ final state, exactly two leptons of the same charge are required, while in the $3\ell$ final state the requirement is three leptons with a total charge of $\pm 1e$. The lepton with the highest $p_{\text {T}}$ is referred to as $\ell _0$. The lepton with the second highest $p_{\text {T}}$ is denoted by $\ell _1$ and, for events in the $3\ell$ final state, the lepton with the lowest $p_{\text {T}}$ is called $\ell _2$. In the $3\ell$ final state an alternative labelling of leptons is introduced based on their charge. In any given $3\ell$ event two leptons have the same charge while the remaining lepton has an opposite charge. The latter is labelled $\ell _{\text {OS}}$, the former $\ell _{\text {SS,0}}$ and $\ell _{\text {SS,1}}$ with the number indexing the lepton in order of decreasing $p_{\text {T}}$. In addition to the requirements on leptons, considered events need to contain at least one jet, at least one of which is b-tagged. A summary of the preselection applied to events is given in Table 1.

Table 1 Overview of the preselections applied in the analysis. These selections ensure jet and b-tag multiplicities, lepton momenta, and define the $2\ell \text {SS}$ and $3\ell$ final states

5 Background estimate

Several SM background processes are present in the signal regions. Many of these processes can be simulated by using MC, others require dedicated treatment. Particularly the modelling of background from non-prompt leptons must be validated and potentially corrected. This background is separated into various categories, encompassing electrons from prompt muon decay, electrons from photon conversion and leptons produced in a hadronic jet, where further differentiation depending on the jet primary particle are made. Jets initiated by $b$-quarks, by other quarks or gluons, and by $\tau$-leptons are considered, where leptons from $b$-jets are further categorised into electrons and muons. The production of these non-prompt leptons is fully accounted for by the simulated SM samples. By navigating through the generator-level parent particles of a given lepton, this lepton is sorted into one of the previously discussed lepton-origin categories. Based on this, several MC templates are defined containing the events corresponding to exactly one non-prompt lepton-origin category. The remaining events include only prompt leptons and remain in their original SM background template. All background estimate methods discussed in the following are based on these prompt and non-prompt MC templates. Several of the processes discussed in this section are assigned unconstrained (free-floating) normalisation factors in the final maximum-likelihood fit. These normalisation factors are constrained in the fit through dedicated control regions, enriched in the respective processes. It should be noted that the QCD multijet background is negligible in this analysis, owing to the rarity of the considered $2\ell \text {SS}$ and $3\ell$ final states.

5.1 Non-prompt lepton background

The primary source of non-prompt lepton background is the decay of $B$-hadrons. These particular objects are also referred to as heavy-flavour (HF) decay leptons. They mainly consist of events from $t\bar{t}$ production (75 %), while the remaining events are split evenly between $V$+jets and single-top-quark production. The Template fit method [103] is used to determine the rates of HF-decay electrons and HF-decay muons. It assumes the kinematic distributions of the corresponding processes are well described, while their normalisation may require corrections. The normalisation factors are determined in the final maximum-likelihood fit, which is performed simultaneously in the $2\ell \text {SS}$ and the $3\ell$ final states.

As explained above, various other non-prompt lepton processes contribute to this analysis. Their contribution is minor compared with the $\text {HF-decay}\,e$ and $\text {HF-decay}\,\mu$ processes. In the fit, they are assigned normalisation uncertainties of 50 %.

5.2 Charge misidentification

In addition to the processes discussed above, the phase space of the $2\ell \text {SS}$ final state is contaminated by events containing prompt electrons with a misidentified charge. The modelling of such objects is strongly dependent on their $p_{\text {T}}$. Furthermore, electrons can radiate hard bremsstrahlung through interaction with the detector material, which then undergoes asymmetric conversion into an electron-positron-pair. The probability for electrons to interact with the detector material increases with the absolute value of the pseudorapidity $|\eta |$. Both, charge misidentified electrons and electrons from bremsstrahlung, are considered as a single background labelled $\text {Q-misID}$, which is modelled using a data-driven method based on the one described in Ref. [103]. To avoid double-counting $\text {Q-misID}$ events, MC events featuring an electron from any of the considered processes are identified using generator level information and are excluded from the analysis.

The data-driven $\text {Q-misID}$ background estimate is performed as follows. Events containing two isolated electrons with an invariant mass around the mass of the $Z$ boson are considered, where the electrons can have either an opposite-sign charge (OS) or the same-sign charge (SS). To maintain orthogonality to the kinematic regions of the main analysis, there is an $N_{b\text {-tags}} = 0$ requirement. The $m_{ee}$ distribution in the vicinity of $m_Z={91.19\,\mathrm{\,}}\hbox {GeV}$ [104] is fitted separately for OS and SS events using Breit–Wigner functions. The fitted centres $m_{\text {peak}}$ and widths $\sigma$ are used to define the $Z$-windows up to $4\sigma$ from the centres, and sideband regions $[m_{\text {peak}} -8\sigma ,\,m_{\text {peak}} - 4\sigma ]\,\cup \,[m_{\text {peak}} + 4\sigma ,\,m_{\text {peak}} + 8\sigma ]$. The assumption is made, that events in the $Z$-window consist of prompt $Z\rightarrow e^{+} e^{-}$ events and a uniform background. The uniform background is estimated from the average yield in the sideband regions, separately for the OS and SS events, and are subtracted in the $Z$-window.

$\text {Q-misID}$ efficiencies are extracted in bins of $|\eta |$ and $p_{\text {T}}$, corresponding to the probability for an electron in a given bin to be charge-misidentified. These relate the number of background-corrected SS and OS events in the $Z$-window and can thus be determined from the obtained event yields. The $\text {Q-misID}$ events in a given region of the $2\ell \text {SS}$ phase space are then modelled by selecting data events with two opposite-charge leptons while keeping all other selection requirements of the region unmodified, and weighting these data events with the obtained efficiencies. It should be noted that due to the high accuracy of the ATLAS muon spectrometer, the muon charge misidentification is negligible in this analysis.

5.3 Prompt-lepton background

The $t\bar{t}W$ and $t\bar{t}Z$ processes represent the primary background sources from prompt leptons. While measurements of $t\bar{t}W$ production deviate from theoretical predictions by $1.4\sigma$ [105], measurements of $t\bar{t}Z$ production align closely with SM expectations. However, it’s worth noting that the latest measurement of $t\bar{t}Z$ production is conducted in regions with a jet multiplicity of at least four [106]. This analysis extends to kinematic regions with fewer jets, which may not be adequately modelled by MC simulations. To address this discrepancy, the normalisation of $t\bar{t}W$ and $t\bar{t}Z$ production is left unconstrained in the maximum-likelihood fit. Similar to the non-prompt templates, a single normalisation factor is applied per process across the entire phase space, encompassing both the $2\ell \text {SS}$ and the $3\ell$ final states.

The $2\ell \text {SS}$ and $3\ell$ event selection requires at least one b-jet in the final state. In the $VV$ MC samples used in the analysis these quarks must be produced by the shower generator. To account for potential mismodeling of the jet flavour composition, the $VV$ background is split according to the number of leptons ($2\ell$, $3\ell$, $4\ell$) and the generator-level flavour of additional jets. If the event contains at least one jet originating from the decay of either a bottom or a charm hadron is sorted into the b/c category. If it exclusively contains jets from light quarks or $\tau$-leptons it is included in the $l/\tau$ category. The $VV3\ell +b/c$ template has the largest contribution in the phase space and its normalisation is left free-floating in the maximum-likelihood fit. The remaining templates ($VV2\ell +l/\tau$, $VV2\ell +b/c$, $VV3\ell +l/\tau$, $VV4\ell +b/c$ and $VV4\ell +l/\tau$) are assigned individual 50 % normalisation uncertainties. These minor $VV$ templates, alongside the minor non-prompt templates and several SM processes are combined into one Others category in the following. The rates of the majority of processes in this category are similar in the analysis phase space.

6 Event categorisation

Preselected events are categorised into several kinematic regions. These regions are either enriched in signal events, to improve the overall sensitivity of the analysis, or in certain background events, to control the normalisation of specific processes.

6.1 Definition of signal regions

After event preselection, four signal regions are defined in this analysis, two in the $2\ell \text {SS}$ final state and two in the $3\ell$ final state. One region per final state is enriched in the $t\rightarrow Hq$ decay signal. These two regions are referred to as $\text {SR}2\ell \text {Dec}$ and $\text {SR}3\ell \text {Dec}$. The other two signal regions contain a larger fraction of $gq\rightarrow tH$ production signal and are correspondingly named $\text {SR}2\ell \text {Prod}$ and $\text {SR}3\ell \text {Prod}$.

Both the $2\ell \text {SS}$ signal regions require exactly one b-tagged jet. Given that a higher jet multiplicity is anticipated for the decay signal compared to the production signal, the $\text {SR}2\ell \text {Dec}$ is defined by requiring $N_{\text {jets}} \ge 4$. A cut of $N_{\text {jets}} \le 3$ in the $\text {SR}2\ell \text {Prod}$ ensures the orthogonality of the two regions. To minimise the contribution from $\text {HF-decay}$ leptons, the $p_{\text {T}}$ of $\ell _1$ is required to be larger than 12  GeV in the $\text {SR}2\ell \text {Dec}$ and 16  GeV in the $\text {SR}2\ell \text {Prod}$. As $\text {Q-misID}$ events strongly contaminate the $2\ell \text {SS}$ final state, a cut on the invariant mass of the two leptons is imposed in both the regions. If both the leptons are identified as electrons, $|m(e,e)-m_Z|\ge {10\,\mathrm{\,}}\hbox {GeV}$ must be satisfied.

Similar to the $2\ell \text {SS}$ final state, both the signal regions in the $3\ell$ final state require exactly one b-tagged jet. The primary difference between the two regions is again the jet multiplicity. Overall fewer jets are expected for the signal in the $3\ell$ final state. Hence, $N_{\text {jets}} \ge 3$ is required for the $\text {SR}3\ell \text {Dec}$, while $N_{\text {jets}} \le 2$ is required for the $\text {SR}3\ell \text {Prod}$. To reduce the $\text {HF-decay}$ contribution, the definition of both the regions require cuts on the $p_{\text {T}}$ of the two sub-leading leptons: $p_{\text {T}} (\ell _1)\ge {20\,\mathrm{\,}}\hbox {GeV}$ and $p_{\text {T}} (\ell _2)\ge {16\,\mathrm{\,}}\hbox {GeV}$. As the $\text {Q-misID}$ process does not contribute to the $3\ell$ final state, no further restrictions to the event selection are made. The definition of all signal regions is summarised in Table 2, while their respective composition in terms of background processes is shown in Fig. 3.

Table 2 Overview of the event selections applied to define the signal regions of the analysis

Fig. 3

The composition of background processes in the a $\text {SR}2\ell \text {Dec}$, b $\text {SR}2\ell \text {Prod}$, c $\text {SR}3\ell \text {Dec}$ and d $\text {SR}3\ell \text {Prod}$. The Others template contains various minor processes, which individually only have small contributions. No correction to the normalisation of any process is applied

6.2 Control regions for non-prompt-lepton backgrounds

The template-fit method assumes that corrective normalisation factors are applicable independently of the considered final state. To ensure this assumption is correct, two control regions are defined for each of the $\text {HF-decay}$ processes: one in the $2\ell \text {SS}$ final state ($\text {CR}2\ell \text {HF}e$ and $\text {CR}2\ell \text {HF}\mu$) and one in the $3\ell$ final state ($\text {CR}3\ell \text {HF}e$ and $\text {CR}3\ell \text {HF}\mu$).

Overall, the average $p_{\text {T}}$ of leptons originating from a HF decay is smaller than that of prompt leptons. Thus, all $\text {HF-decay}$ control regions are defined by imposing upper boundaries on the $p_{\text {T}}$ of the lowest-$p_{\text {T}}$ lepton. The boundaries are chosen such that an orthogonality to the signal regions of the respective final state is ensured. Requiring the flavour of the lowest-$p_{\text {T}}$ lepton to be either e or $\mu$ ensures that the region is exclusively enriched in $\text {HF-decay}\,e$ or $\text {HF-decay}\,\mu$ events. In the $\text {CR}2\ell \text {HF}e$ and the $\text {CR}2\ell \text {HF}\mu$ the $p_{\text {T}}$-leading lepton is required to be a muon, reducing the contamination of $\text {Q-misID}$ events. Due to high statistical uncertainty, only the event yields of the $\text {CR}2\ell \text {HF}e$ and the $\text {CR}2\ell \text {HF}\mu$ enter the maximum-likelihood fit. For the $\text {CR}3\ell \text {HF}e$ and the $\text {CR}3\ell \text {HF}\mu$, the distributions of the $p_{\text {T}} (\ell _1)$ variable are included. A summary of the definition of all non-prompt control regions can be found in Table 3.

6.3 Control regions for $t\bar{t}W$ and $t\bar{t}Z$ backgrounds

Three additional control regions are defined to determine the normalisation of the background from $t\bar{t}W$ and $t\bar{t}Z$ production. Studies on various kinematic distributions in the signal regions of the $2\ell \text {SS}$ final state show a strong similarity between the two processes. Therefore, in this final state only one control region (denoted $\text {CR}2\ell t\bar{t}V$) is defined to constrain the sum of $t\bar{t}W$ and $t\bar{t}Z$. This region requires at least four jets, exactly two of which are b-tagged, because both the top quarks in the process are expected to produce a b-quark. An additional requirement of $p_{\text {T}} (\ell _1)\ge {18\,\mathrm{\,}}\hbox {GeV}$ further increases the purity of the region in $t\bar{t}V$ events by reducing contamination from HF-decay events. The $p_{\text {T}} (\ell _1)$ variable is used as input for the maximum-likelihood fit.

In the $3\ell$ final state, distinct shape differences between $t\bar{t}W$ and $t\bar{t}Z$ production are observed due to the absence of any requirement on the invariant mass of lepton pairs. Therefore, two control regions, $\text {CR}3\ell t\bar{t}W$ and $\text {CR}3\ell t\bar{t}Z$, are defined. Both the regions require at least two jets, two of which are b-tagged. Events are then assigned to the $\text {CR}3\ell t\bar{t}Z$ if at least one pair of leptons with the same flavour and opposite charge can be found satisfying $|m(\ell ^+,\ell ^-)-m_Z|<{10\,\mathrm{\,}}\hbox {GeV}$. In cases where this criterion is not met, events are assigned to $\text {CR}3\ell t\bar{t}W$. In both the control regions, the transverse momentum of the $p_{\text {T}}$-leading $b$-tagged jet $p_{\text {T}} (b\text {-jet}_0)$ is used as input for the maximum-likelihood fit. A summary of the definition of all $t\bar{t}V$ control regions can be found in Table 4.

Table 3 Overview of the event selections applied to define the non-prompt lepton background control regions of the analysis

Table 4 Overview of the event selections applied to define the $t\bar{t}W$ and $t\bar{t}Z$ control regions of the analysis

7 Separation of signal and background

Artificial neural networks (NNs) are used to separate signal and background in all four signal regions combining several kinematic variables into optimised NN discriminants. In addition to variables derived from the reconstructed objects the NNs build on a diverse range of reconstructed kinematic variables. Detailed information regarding the reconstruction algorithms is provided in Sect. 7.1, while a comprehensive description of the NN training process can be found in Sect. 7.2.

7.1 Event reconstruction in signal regions

The reconstruction algorithms employed are described below. In $tHq$ FCNC interactions, multiple decay modes of the Higgs boson that contribute to the $2\ell \text {SS}$ and $3\ell$ final states are considered. However, the $H\rightarrow WW^*$ decay mode has by far the largest contribution with $\ge {75\,\mathrm{\%}}$ in all signal regions. Thus, all reconstruction efforts presented here focus on this specific Higgs-boson decay mode. Most algorithms are customised to fit one of the two considered final states. The relevant final states for each algorithm are given in parentheses.

Recursive jigsaw reconstruction ($2\ell \text {SS}$ and $3\ell$)

The Recursive Jigsaw Reconstruction (RJR) technique [107] is a method for resolving combinatorial and kinematic ambiguities which arise in hadron collider events. RJR relies on decay trees which describe the topologies of interest, and relates the reference frames of the intermediate particles in the decay. Lepton and jet assignment, along with the splitting of ${E}_{\text {T}}^{\text {miss}}$, is governed by a set of jigsaw rules which rely on known properties of the decay tree. For instance the top-quark and Higgs-boson masses can be used to constrain the assignment of the jets, or the W-boson mass can be used to facilitate ${E}_{\text {T}}^{\text {miss}}$ splitting and matching to leptons.

This process leads to the computation of four-vectors of the reconstructed intermediate states, accessible in any of the frames defined in the decay tree. For each of the four topologies ($2\ell 1b2j$, $2\ell 1b3j$, $3\ell 1b0j$, $3\ell 1b1j$, where b refers to $N_{b\text {-jets}}$ and j to $N_{\text {jets}}$) there are different intermediate states, split by production versus decay and again by final state. The particles that are reconstructed for each individual region are listed in Table 5. These include W and Higgs bosons, and top quarks. Figure 4 depicts the probability densities of two RJR reconstructed variables for the $tHu$ signal and the combination of all background processes, showing clear shape differences between the two.

Table 5 Particles from the $tHq$ processes which are reconstructed using the Recursive Jigsaw Reconstruction (RJR) in the respective signal regions

Fig. 4

Probability densities of a $m(\ell _1,t_{\text {FCNC}})$ in the $\text {SR}2\ell \text {Dec}$ and b $m(W_t)$ in the $\text {SR}3\ell \text {Dec}$ for the $tHu$ signal process and the sum of all background processes. The vertical lines on bins represent MC-statistical uncertainties. The last bin includes the overflow

Neutrino independent combinatorics estimator ($3\ell$) The considered decay scenario of a signal event in the $3\ell$ final state involves one top quark decaying into a b-quark, a lepton and a neutrino, while the Higgs-boson decay is expected to yield two leptons and two neutrinos. Studies of the MC signal samples show that on average the two leptons from the Higgs-boson decay have the smallest angular separation out of any lepton pair in the event. Based on this information, the two leptons with the smallest angular separation $\Delta R$ are labelled $\ell _{H,0}$ and $\ell _{H,1}$, ordered by their transverse momentum. The remaining lepton is labelled $\ell _{t}$, while the b-tagged jet is denoted $b_t$. Various kinematic parameters are calculated for these objects, two of which are shown as examples in Fig. 5 for the full $tHu$ signal and the combination of all background processes, demonstrating their high separation power. However, these parameters are discarded, if the angular separation of $b_t$ from either $\ell _{H,i}$ is smaller than the separation from $\ell _t$. Conversely, when the angular separation of $b_t$ from $\ell _t$ is found to be smaller than from any other lepton, and further, when $\ell _{H,0}$ and $\ell _{H,1}$ have opposite charges the Neutrino independent combinatorics estimator reconstruction (NICE Reco) condition is defined as satisfied. In this scenario, the calculated kinematic parameters have a particularly high separation power between signal and background processes. This information is provided as additional input to the NNs.

Fig. 5

Probability densities of a $m(\ell _{H,0},\ell _{H,1})$ and b $m(\ell _t,b_t)$ in the $\text {SR}3\ell \text {Dec}$ for the $tHu$ signal process and the sum of all background processes. The vertical lines on bins represent MC-statistical uncertainties. The last bin includes the overflow

7.2 Training of feed-forward neural networks

The NNs are implemented using the NeuroBayes package [108, 109], which combines a three-layer feed-forward NN with a complex and robust preprocessing of the input variables before they are presented to the NN. The preprocessing produces a ranking of the input variables based on an algorithm employing the total correlation of a set of variables to the target function which assumes the value $1$ for signal and $0$ for background events. The input variable selection is performed once per signal region.

The selected variables of all signal regions are listed in Tables 11, 12, 13, 14. Overall, invariant masses of reconstructed objects, including RJR objects, compose the largest fraction. The most significant variables in both the $3\ell$ signal regions are the invariant mass of the opposite charge and either of the two same charge leptons $m(\ell _{\text {OS}}, \ell _{\text {SS,0}})$ and $m(\ell _{\text {OS}}, \ell _{\text {SS,1}})$. In the $\text {SR}2\ell \text {Prod}$, the invariant mass of the subleading-$p_{\text {T}}$ lepton and the RJR Higgs boson $m(\ell _1,H)$ is the most important variable, followed by the number of jets $N_{\text {jets}}$. The most significant variable of the $\text {SR}2\ell \text {Dec}$ is the scalar $p_{\text {T}}$ sum of all jets $H_T(\text {jets})$, a quantity that is also highly ranked in other trainings. The invariant mass of the leading-$p_{\text {T}}$ lepton and the event’s $b$-jet $m(\ell _0,b\text {-jet})$ is the second-highest ranked in the training.

Individual NNs are created for both the signal processes and every signal region, summing up to a total of eight NN architectures. The training uses the data-driven $\text {Q-misID}$ estimate together with unmodified MC templates as input, before any corrections to their normalisation by the maximum-likelihood fit. Cross-training is employed to prevent overfitting, where the signal and background MC events are split into half. The splitting is performed in a pseudo-random way based on the parity of the unique event identification number. For each half, a separate NN model is trained, which is then applied to the other half of MC events in the final analysis.

During the training process of all the NNs, the signal is trained against all considered backgrounds with a fraction of 50 % signal events and 50 % background events. The different background processes are weighted relative to each other according to their expected number of events. NeuroBayes uses Bayesian regularisation techniques for the training process to improve the generalisation performance and to avoid overfitting. The network infrastructure consists of one input node for each input variable, a single hidden layer, and one output node giving a continuous output in the interval $(-1,+1)$. The number of nodes in the hidden layer is optimised individually for each NN model. As a non-linear activation function NeuroBayes uses the symmetric sigmoid function. In the region close to zero, the sigmoid function has a linear response. The final discriminant $D_{\text {NN}}$ is obtained by linearly scaling the output of the NNs to the interval $(0,1)$.

8 Systematic uncertainties

Several sources of systematic uncertainty affect the expected event yield from signal and background processes and the shape of the NN discriminants used in the maximum-likelihood fits. The systematic uncertainties are divided into two major categories. There are experimental uncertainties in the reconstruction of the four-momenta of the final-state objects: electrons, muons, jets, $b\text {-tagged}$ jets, and ${E}_{\text {T}}^{\text {miss}}$ as a sign of a high-$p_{\text {T}}$ neutrino. The second category of uncertainties is related to the modelling of scattering processes with event generators and of the interaction of particles with the detector. All uncertainties are propagated through the analysis and their effects on the expected event yields and discriminant distributions are accounted for by including corresponding nuisance parameters in the fit. In the following, the estimate of experimental and modelling uncertainties is explained in more detail.

8.1 Experimental uncertainties

The uncertainty in the integrated luminosity of the combined 2015–2018 data set is 0.83% and is based on a calibration of the luminosity scale using x–y beam-separation scans [46]. The luminosity uncertainty is applied to the expected signal and background event yields.

Scale factors are applied to simulated events to correct for reconstruction, identification, isolation and trigger performance differences between data and detector simulation for electrons and muons. These scale factors and their systematic uncertainties, and the lepton momentum scale and resolution, were assessed using $Z\rightarrow e^+e^-$ and $Z\rightarrow \mu ^+\mu ^-$ events in simulation and data [91, 93].

The jet energy scale (JES) was calibrated using a combination of test-beam data, simulation and in situ techniques [99]. The JES is parameterised in bins of jet $p_{\text {T}}$ and $\eta$. Its uncertainty is decomposed into a set of 30 uncorrelated components, of which 29 are non-zero in a given event depending on the type of simulation used. Sources of uncertainty contributing to the JES uncertainties include the $\eta$ intercalibration of forward jets in the range of $0.8< |\eta _\textrm{det}| < 4.5$ with those in the central barrel region ($|\eta _\textrm{det}| < 0.8$), pile-up modelling, jet flavour composition and response, differences between jets induced by $b\text {-quarks}$ and those from gluons or light-quarks, single-particle response, detector modelling, non-closure, and effects of jets not fully contained in the calorimeter.

The uncertainty of the jet energy resolution (JER) is evaluated by smearing jet energies according to a Gaussian function [99]. Thirteen orthogonal components account for jet-$p_{\text {T}}$ and $\eta$-dependent differences between simulation and data which were determined using dijet events and noise measurements based on random cones. The smearing is applied to simulated events if the resolution in data is larger than in MC simulation, and to pseudo-data when the resolution is larger in simulation than in collision data. The JER uncertainties are defined by comparing both types of smearing and thereby taking the anti-correlation between different components into account. The nominal data remains unchanged. The uncertainty in the efficiency to satisfy the JVT requirement for pile-up suppression was derived in $Z(\rightarrow \mu ^+\mu ^-)$+jets events and is also considered [100]. The uncertainty in $E_{\text {T}}^{\text {miss}}$ due to a possible miscalibration of its soft-track component was derived from data–simulation comparisons of the $p_{\text {T}}$ balance between the hard and soft $E_{\text {T}}^{\text {miss}}$ components [102].

The $b\text {-tagging}$ requirement made in the measurement has uncertainties in the $b\text {-tagging}$ efficiency of true $b\text {-jets}$ and in the mistagging rates of light-quark jets and $c$-jets. The $b\text {-tagging}$ efficiency is measured in dileptonic $t\bar{t}$ events. Differences between data and detector simulation are corrected by $p_{\text {T}}$-dependent scale factors applied to simulated events. The uncertainty in the scale factors is decomposed into 45 orthogonal components [110]. The uncertainties are propagated through the analysis as weights. The set of uncertainties used covers both the 70 % working point used in this analysis, as well as other working points of the DL1r discriminant. The rate of mistagging $c$-jets as $b\text {-jets}$ was measured in semileptonic $t\bar{t}$ events, where one of the W bosons decays into an electron or a muon and a neutrino and the other decays into a quark–antiquark pair [111]. This event sample allows to utilise the relatively large and known $W\rightarrow cs$ contribution. The mistagging rate of $c$-jets depends on the jet $p_{\text {T}}$ and has a total uncertainty in the range of 317%. The uncertainties are decomposed into 20 orthogonal components. The misidentification rate of light-quark jets was evaluated based on the techniques described in Ref. [112]. The resulting calibration factors are in the range of about 1.5–3.0 with uncertainties up to 50 %. The uncertainties are decomposed into 20 independent eigenvectors.

To account for differences between simulation and data in the pile-up distribution, the pile-up profile in the simulation is corrected to match the one in data. The uncertainty in the correction factor is applied in the measurement as a variation of the event weight.

Fig. 6

Distributions of the most important NN input variable for the signal regions a $\text {SR}2\ell \text {Dec}$, b $\text {SR}2\ell \text {Prod}$, c $\text {SR}3\ell \text {Dec}$ and d $\text {SR}3\ell \text {Prod}$. The corrections from a background-only fit are applied. Only events with $D_{\text {NN}} <0.5$ were used in the fit and in the plots. The first bin contains all events below its lower boundary, while the last bin contains all events falling above its upper boundary. The Others template contains various minor processes, which individually only have small contributions. The total statistical and systematic uncertainty is indicated by the hatched band. The dashed line depicts the sum of all background processes prior to the fit. The last bin includes the overflow

Fig. 7

The $D_{\text {NN}}$ distributions in the a $\text {SR}2\ell \text {Dec}$, b $\text {SR}2\ell \text {Prod}$, c $\text {SR}3\ell \text {Dec}$ and d $\text {SR}3\ell \text {Prod}$, obtained from the signal-plus-background fit to data in the $tHc$ channel. The Others template contains various minor processes, which individually only have small contributions. The total statistical and systematic uncertainty is indicated by the hatched band. The dotted line represents the distribution of the signal, scaled to the number of background events. The dashed line depicts the sum of all background processes prior to the fit

Fig. 8

Summary plot of the fitted distributions in all control regions obtained from the signal-plus-background fit to data in the $tHc$ channel. The bin boundaries for each region are depicted by the parenthesised numbers. The Others template contains various minor processes, which individually only have small contributions. The total statistical and systematic uncertainty is indicated by the hatched band. The dashed line depicts the sum of all background processes prior to the fit

8.2 Modelling uncertainties

Normalisation uncertainties are applied to all processes, except for the ones whose normalisation is a free parameter of the maximum-likelihood fit, specifically for the $\text {HF-decay}$ templates, for $t\bar{t}W$ and $t\bar{t}Z$ production, and for the largest $VV$ template $VV3\ell +b/c$. Thus, no normalisation uncertainty is applied to $t\bar{t}$, single-top-quark, $Z$+jets and $W$+jets production, all of which exclusively contribute to non-prompt templates. Minor non-prompt templates are assigned 50 % normalisation uncertainties. The same is true for the minor $VV$ templates and all rare processes discussed in Sect. 3.2. For $tZq$ and $tWZ$ production, a respective uncertainty of 30 % is assigned based on a previous ATLAS measurement of the processes [113]. Asymmetric uncertainties of $^{+{7}\%}_{-{10}{\%}}$ are assigned to $t\bar{t}H$ production, estimated by varying the renormalisation and factorisation scales, and the combined PDF+$\alpha _{\text {s}}$ uncertainties [28].

Uncertainties in modelling parton showers and hadronisation are assigned to the $tHq$ signal samples, and the background originating from $t\bar{t}$, $t\bar{t}H$, $t\bar{t}Z$ and $t\bar{t}W$ production. For the signal and $t\bar{t}$, $t\bar{t}H$ and $t\bar{t}Z$ production the nominal samples are compared with alternative samples, for which the individual matrix element generators were interfaced to Herwig[7.1.3] [114, 115] (for $t\bar{t}$ production), Herwig[7.0.4] (for $t\bar{t}H$ production) and Herwig[7.2.1] (for $t\bar{t}Z$ production) instead of the respective versions of Pythia[8]. For $t\bar{t}W$ production, two alternative samples are generated. The matrix element of both the samples is generated with Powhegbox[v2], which is interfaced to Pythia[8] for one sample and to Herwig[7] for the other. The relative differences between the two samples is determined for all bins in the analysis and applied to the nominal Sherpa[2.2.10] $t\bar{t}W$ sample.

The impact of the generator choice for $t\bar{t}W$ is evaluated using an alternative sample and is applied as systematic uncertainties for both the normalisation and shape. This alternative sample is generated with up to one additional parton in the final state at $\text {NLO}$ accuracy using Mgnlo[2.9.3], with different parton multiplicities being merged using the FxFx $\text {NLO}$ matrix-element and parton-shower merging prescription [89] with a merging scale of 30  GeV. The events are interfaced with Pythia[8.245] using the A14 set of tuned parameters and the NNPDF[2.3lo] PDF set to simulate hadronisation and showering. The uncertainty is split into three components: a shape uncertainty in each region, a region migration uncertainty and a global normalisation uncertainty.

Table 6 The predicted and observed yields in all $2\ell \text {SS}$ regions of the analysis from the signal-plus-background fit to signal and control regions. The pre-fit predictions for the two signal components are presented also, scaled to a cross-section equivalent to a branching ratio $\mathcal {B}(t\rightarrow Hq)={0.1\,\mathrm{\%}}$. The Others template contains various minor processes, which individually only have small contributions. The uncertainties in non-signal MC yields reflect the sum in quadrature of all post-fit systematic and statistical uncertainties. The row labelled Post-fit BG shows the sum of all above background (BG) processes with post-fit uncertainties, while the row labelled Pre-fit BG shows the total pre-fit background yield with the corresponding pre-fit uncertainties. Hyphens signify that the corresponding process does not contribute to the given region

Table 7 The predicted and observed yields in all $3\ell$ regions of the analysis from the signal-plus-background fit to signal and control regions. The pre-fit predictions for the two signal components are presented as well, scaled to a cross-section equivalent to a branching ratio $\mathcal {B}(t\rightarrow Hq)={0.1\,\mathrm{\%}}$. The Others template contains various minor processes, which individually only have small contributions. The uncertainties in non-signal MC yields reflect the sum in quadrature of all post-fit systematic and statistical uncertainties. The row labelled Post-fit BG shows the sum of all above background (BG) processes with post-fit uncertainties, while the row labelled Pre-fit BG shows the total pre-fit background yield with the corresponding pre-fit uncertainties. Hyphens signify that the corresponding process does not contribute to the given region

Uncertainties related to the choice of the renormalisation scale $\mu _\textrm{r}$ and the factorisation scale $\mu _\textrm{f}$ for the matrix-element calculations are evaluated by varying the scales independently by factors of $2$ and $0.5$, separately for $VV$ production and each of the top-quark production processes, including production with an additional boson and the two signal processes. The scale variations are implemented as generator weights in the nominal sample. The uncertainties are treated as uncorrelated across individual processes.

The uncertainty in matching the NLO matrix elements to the parton shower when generating $t\bar{t}$ events is evaluated by comparing the nominal samples of simulated events to samples with an alternative setting of the $p_{\text {T}} ^{\textrm{hard}}$ parameter in the matching code, using one instead of the default setting of zero. This parameter regulates the definition of the vetoed region of the parton shower, which is needed to avoid overlap in the phase space filled by Powheg and Pythia. This estimate of the uncertainty follows the description in Ref. [116]. The uncertainty in the choice of the $h_\textrm{damp}$ parameter for the $t\bar{t}$ event generation is estimated by using an additional $t\bar{t}$ sample produced with the $h_\textrm{damp}$ parameter set to $3 m_t$, while keeping all other generator settings the same as used for the nominal sample of events.

Three uncertainties are defined for the modelling of the $\text {Q-misID}$ template. The statistical uncertainty in the $\text {Q-misID}$ efficiencies obtained from the likelihood fit is taken into account. An uncertainty in the selection of the Z-window is calculated by redefining the Z-window with $3\sigma$ and $5\sigma$, where $\sigma$ corresponds to the width of the fitted Breit–Wigner distribution. The width of the Z-window is varied and the efficiencies are determined again, with the symmetrised difference to the nominal values comprising the uncertainty, which is then propagated through the analysis. Finally, a non-closure uncertainty is implemented by applying the $\text {Q-misID}$ estimation method to a $Z\rightarrow ee$ MC sample and comparing the resulting misidentification rates to the MC prediction. The symmetrised difference applied to the nominal $\text {Q-misID}$ rates constitutes the uncertainty.

Uncertainties in the amount of initial-state radiation are defined for the top-quark production processes $t\bar{t}$, $t\bar{t}Z$ and $t\bar{t}H$. They are assessed by varying the eigentune Var3c of the A14 Pythia[8] tune consistent with the uncertainties of the tune. An uncertainty in the final-state radiation is introduced by varying the renormalisation scale $\mu _\textrm{r}$ in the parton shower, at which the strong coupling constant $\alpha _\textrm{s}$ is evaluated, by factors of $0.5$ and $2.0$.

Table 8 Normalisation factors for various backgrounds determined from the signal-plus-background fit to signal and control regions. The results for the $tHu$ fit and the $tHc$ fit are shown

Table 9 The expected 95 % upper limits on the branching ratio $\mathcal {B}(t\rightarrow Hq)$ for the nominal and alternative fit configurations. One fit is performed in the full phase space considering only statistical uncertainties. Two other fits are performed using the full set of uncertainties in only the $2\ell \text {SS}$ or $3\ell$ final states. The $\pm 1\,\sigma$ interval of the expected limit is indicated by the upper and lower indices. For the $tHu$ channel, the assumption of $\mathcal {B}(t\rightarrow Hc)=0$ is made and vice versa

Table 10 The observed (expected) 95 % upper limits on the branching ratio $\mathcal {B}(t\rightarrow Hq)$ and the absolute value of the Wilson coefficient $|C_{u\phi } |$ under the assumption of a cutoff-scale of $\Lambda ={1}\,\hbox {TeV}$. For the $tHu$ channel, the assumption of $\mathcal {B}(t\rightarrow Hc)=0$ is made and vice versa

In all uncertainty evaluations mentioned above the alternative samples or reweighted samples are normalised to the total cross-section of the nominal samples.

Uncertainties in the PDFs are evaluated for the background processes simulated using Powhegbox, meaning $t\bar{t}$, $\text {single-}t$ and $t\bar{t}H$ production, using the PDF4LHC15 prescription with 30 eigenvectors [117]. Simulated events are reweighted to the central value and the eigenvectors of the combined PDF set. Systematically varied templates are constructed by taking the differences between the samples reweighted to the central value and those reweighted to the eigenvectors. In the likelihood fit, the PDF uncertainties are treated as correlated across the top-quark production processes.

The uncertainties due to the finite number of simulated events, also called the MC statistical uncertainty, is accounted for by adding a nuisance parameter for each bin of the $D_{\text {NN}}$ distributions and the distributions in the CRs, implementing the Barlow–Beeston approach [118].

9 Statistical analysis

The normalisation $\mu$ of the $tHu$ and $tHc$ signal couplings is determined in binned profile maximum-likelihood fits. The fits are performed simultaneously in all signal and control regions. In the signal regions, the respective $D_{\text {NN}}$ distributions optimised for the considered signal coupling are used. For the $\text {CR}2\ell \text {HF}e$ and $\text {CR}2\ell \text {HF}\mu$ only event yields are employed in the fit, whereas for the $\text {CR}3\ell \text {HF}e$ and $\text {CR}3\ell \text {HF}\mu$ the transverse momentum of the third-leading-$p_{\text {T}}$ lepton is fitted. In the $\text {CR}2\ell t\bar{t}V$, the distribution of the subleading-$p_{\text {T}}$ lepton’s transverse momentum is used, as it provides separation between prompt and non-prompt leptons. The $\text {CR}3\ell t\bar{t}W$ and $\text {CR}3\ell t\bar{t}Z$ are incorporated into the fit with the distribution of the transverse momentum of the leading-$p_{\text {T}}$ $b$-tagged jet. This distribution separates the $t\bar{t}Z$ background from remaining $VV$ events in the $\text {CR}3\ell t\bar{t}Z$ and is also adopted in the $\text {CR}3\ell t\bar{t}W$ for reasons of consistency. In the fits, each of the signal processes are treated as maximally anti-correlated, implying that the presence of one signal automatically precludes the presence of the other signal.

The likelihood function is constructed as a product of Poisson probability terms over all considered bins. The fitted event yields in the bins depend on nuisance parameters $\theta$ which include the effects of systematic uncertainties. Each nuisance parameter, except those representing the MC statistical uncertainties, is constrained by a Gaussian distribution term in the likelihood function. Some systematically varied discriminant distributions are smoothed and nuisance parameters of systematic uncertainties with negligible impact on the parameter of interest are entirely removed to reduce spurious effects in minimisation, improve convergence of the fit, and reduce the computing time. Normalisation and shape effects from a source of systematic uncertainty are treated separately in this removal process. Single-sided systematic variations are turned into symmetric variations by taking the difference between the nominal model and the alternative model and mirroring this difference in the opposite direction. This is done for the event yield and for the event shape, where the distributions are subtracted from the nominal one. For most sources with two variations, their effects are made symmetric by using the average deviation from the nominal prediction. Exceptions are the uncertainties in the JER, for which the asymmetric variations are kept because the underlying effects are known to be asymmetric. Free-floating normalisation factors are assigned to the templates modelling the $\text {HF-decay}\,e$ and $\text {HF-decay}\,\mu$ background, as well as the $VV 3\ell$+$b$/$c$, $t\bar{t}W$, and $t\bar{t}Z$ templates.

The test statistic $q_\mu$ is defined as the profile likelihood ratio

$$ q_\mu =-2 \ln \frac{ \mathcal {L}(\mu , \hat{\hat{\theta }}) }{ \mathcal {L}(\hat{\mu }, \hat{\theta }) }, $$

where $\hat{\mu }$ and $\hat{\theta }$ are the values of the parameters that maximise the likelihood function and $\hat{\hat{\theta }}$ are the values of the nuisance parameters that maximise the likelihood function for a given value of $\mu$. The test statistic is evaluated with the RooFit package [119]. If the observed signal normalisation $\mu$ is compatible with the background hypothesis, that is $\mu =0$, the test statistic $q_\mu$ is used in the $\text {CL}_{\text {S}}$ method [120] to obtain exclusion limits on the signal normalisation. For a given signal scenario, values of $\mu$ yielding $\text {CL}_{\text {S}} \le 0.05$, where $\text {CL}_{\text {S}}$ is computed using the asymptotic approximation [121], are excluded at $\ge {95\,\mathrm{\%}}$ CL. The obtained upper limits on the signal strength $\mu$ are then transformed into limits on the respective Wilson coefficient $C_{u\phi }$ and the corresponding branching ratio $\mathcal {B}(t\rightarrow Hq)$ through their dimension-6 operators [60].

10 Results

This section presents the results of the profile-likelihood fit. Prior to the full fit to extract the signal normalisation, a background-only fit using only regions with little signal contribution is performed to check the modelling of NN input variables. Once the modelling is checked and validity of the fit model is established, the signal normalisation is determined in a full fit to data using the entire phase space of the analysis.

10.1 Cross-checks on the modelling of the neural network input variables

Before the application of the NNs to the full collision data, modelling of the input variables is checked in parts of the signal regions. For this purpose, the events populating the respective signal regions up to a threshold value of $D_{\text {NN}} =0.5$ are selected. The threshold of $D_{\text {NN}} =0.5$ is chosen to maximise the available statistics while avoiding a signal contamination of more than 10 %. The full fit is performed with both statistical and systematic uncertainties considered in all control regions and the low-$D_{\text {NN}}$ signal regions. The corrections made by the fit are then applied to the input variables of the various NNs, using only events with $D_{\text {NN}} <0.5$, to check the variable modelling. The resulting distributions of the most important NN input variables in each signal region are shown in Fig. 6. Overall, no discrepancies between the post-fit distributions and data could be observed in any of the input variables.

Fig. 9

The 95 % CL upper limits on a $\mathcal {B}(t \rightarrow Hu)$ assuming $\mathcal {B}(t \rightarrow Hc) = 0$ and b $\mathcal {B}(t \rightarrow Hc)$ assuming $\mathcal {B}(t \rightarrow Hu) = 0$ for the individual searches and their combination. The observed limits (solid lines) are compared with the expected (median) limits under the background-only hypothesis (dotted lines). The surrounding shaded bands correspond to the 68 % and 95 % CL intervals around the expected limits, denoted by $\pm 1\sigma$ and $\pm 2 \sigma$, respectively. The $H\rightarrow VV^*$ also includes events from leptonic $H\rightarrow \tau ^+\tau ^-$ decays

10.2 Full fit to data

Figure 7 shows the $D_{\text {NN}} (tHc)$ distributions in all four signal regions after the fit. A summary plot of all $2\ell \text {SS}$ and $3\ell$ control regions is shown in Fig. 8. Overall, a good agreement between MC and data is observed. The event yields shown in Tables 6, 7 confirm this. The best fit value of the normalisation of the $tHu$ ($tHc$) signal is found to be $\mu _{tHq}=-0.03\pm 0.15(-0.08\pm 0.19)$. No pulls beyond $1\sigma$ or strong constraints are observed for any of the nuisance parameters. The observed post-fit values of all normalisation factors and nuisance parameters are in agreement with the background-only fit consistent with the statistical uncertainties, suggesting a good modelling of the most signal-sensitive $D_{\text {NN}}$ bins. The obtained normalisation factors for free-floating background processes are listed in Table 8. For the $\text {HF-decay}\,e$ and $\text {HF-decay}\,\mu$ processes they are compatible with one. A 40 % increase can be observed for the $VV3\ell +b/c$ template, owing to the poorer modelling of b-jets by the shower generator. The normalisations of the $t\bar{t}W$ and $t\bar{t}Z$ processes are increased by approximately 15 %, which is compatible at a $1\sigma$ level. The increased normalisation of $t\bar{t}W$ production is expected based on dedicated measurements of the process, while for $t\bar{t}Z$, the slightly higher normalisation comes from the inclusion of events with low jet multiplicity. Studies on the process confirm that for higher values of $N_{\text {jets}}$ the normalisation is compatible with one.

As the best fit value of the signal strength is compatible with $\mu =0$, upper limits are determined with the $\text {CL}_{\text {S}}$ method. In addition to the nominal limits of the analysis, expected limits are determined for a fit considering only statistical uncertainties, as well as for fits in the $2\ell \text {SS}$ and $3\ell$ channels separately. A comparison of the expected upper limits on the branching ratio $\mathcal {B}(t\rightarrow Hq)$ for each configuration is shown in Table 9. A degradation in the expected upper limit of approximately ${20\,\mathrm{\%}}$ can be observed for both the signal processes when systematic uncertainties are included. This shows that the sensitivity of this analysis is primarily limited by available statistics. When comparing the two final states with each other, a clear dominance of the $2\ell \text {SS}$ final state can be observed. Including the $3\ell$ final states yields an improvement of 20 % in the $tHu$ channel and 13 % in the $tHc$ channel.

The observed upper limits on the branching ratio $\mathcal {B}(t\rightarrow Hq)$ together with a reinterpretation as limits on the absolute value of the Wilson coefficient $|C_{u\phi } |$ for the EFT dimension-6 operators can be found in Table 10. The signal does not interfere with other SM processes, meaning that the signal normalisation is proportional to $|C_{u\phi } |^2$ and limits can only be set on the coefficient’s absolute value. Additionally, owing to the averaging of the left-handed and the right-handed signal component, these limits are set on the average of the two respective coefficients $|C_{u\phi }^{tq} |$ and $|C_{u\phi }^{qt} |$. The final observed (expected) upper limits on the branching ratio are $\mathcal {B}(t\rightarrow Hu)<2.8\,(3.0)\times 10^{-4}$ and $\mathcal {B}(t\rightarrow Hc)<3.3\,(3.8)\times 10^{-4}$ for a cutoff scale $\Lambda ={1}\,\hbox {TeV}$. The limits on the Wilson coefficients amount to $|C_{u\phi }^{ut,tu} |<0.71\,(0.73)$ and $|C_{u\phi }^{ct,tc} |<0.76\,(0.82)$. Limits for each of the two signal couplings are determined with the assumption that the other signal coupling do not contribute.

Compared with a previous ATLAS search for $tHq$ FCNC couplings in $2\ell \text {SS}$/ $3\ell$ final states using a partial Run 2 data sample of ${36}\,\hbox {fb}^{-1}$ [32], a luminosity-adjusted improvement by a factor of two for the $tHc$ channel and three for the $tHu$ channel can be observed. This is determined by creating an Asimov-data sample, scaling its luminosity to that of the previous analysis, and comparing the resulting expected upper limits to the original analysis. This improvement primarily results from using improved analysis techniques such as new methods for event reconstruction and a more thorough multivariate analysis, and more precise detector calibrations. The additional improvement in the $tHu$ channel is achieved because of the inclusion of the $gq\rightarrow Ht$ production process in the analysis. This process primarily plays a role in the $tHu$ channel because the up quark is a valence quark of the proton.

11 Combination of results with other searches

The reported results are combined with the corresponding ATLAS searches that use $H \rightarrow \tau ^+\tau ^-$ decays with one or two hadronically decaying $\tau$-leptons [33], and $H \rightarrow b\bar{b}$ [34] and $H \rightarrow \gamma \gamma$ [35] decays. For clarity, this analysis is referred to as the $H\rightarrow VV^*$ channel in the following, as the $H\rightarrow WW^*/ZZ^*$ decay modes are by far the most dominant ones contributing to $2\ell \text {SS}$ and $3\ell$ final states. Orthogonality among all analyses is ensured by their respective event selections. The $H\rightarrow \tau ^{+} \tau ^{-}$ and the $H\rightarrow VV^*$ analysis, specifically, are orthogonal to each other, because the $H\rightarrow \tau ^{+} \tau ^{-}$ analysis exclusively uses events with exactly one or zero leptons. The combination is performed through a simultaneous profile-likelihood fit of all four analyses. The correlations between the uncertainties in the different channels are assessed. In each search, the dominant systematic uncertainties are different. In addition, the $H \rightarrow \tau ^+\tau ^-$, $H \rightarrow \gamma \gamma$ and $H\rightarrow VV^*$ channels are dominated by the statistical uncertainty in the data. Therefore, the combination exhibits minimal sensitivity to possible correlations of uncertainties across channels. Specifically, uncertainties pertaining to luminosity, pile-up modelling, and jet energy scale and resolution are correlated among the four channels. The uncertainties related to $b$-tagging are correlated between the $H \rightarrow \tau ^+\tau ^-$, the $H \rightarrow b\bar{b}$ and the $H\rightarrow VV^*$ analyses, but uncorrelated with the $H\rightarrow \gamma \gamma$ analysis, which uses a simplified $b$-tagging scheme. The remaining uncertainties (mostly from experimental sources, and signal and background modelling) are taken as uncorrelated. Some of the sources of systematic uncertainties (especially related to electron and muon identification and signal modelling) are common to the three search channels, but are treated as uncorrelated due to slight differences in their treatment by individual analyses. The combined $p$-value for the $tHu$ fit is $0.134$, while that for the $tHc$ fit it is $0.117$, showing an overall good agreement among the four analyses.

The observed (expected) 95 % CL combined upper limits on the branching ratios are $\mathcal {B}(t\rightarrow Hu)<2.6\,(1.8)\times 10^{-4}$ and $\mathcal {B}(t\rightarrow Hc)<3.4\,(2.3)\times 10^{-4}$. For the $tHc$ coupling the observed limit is larger than that of the individual $H\rightarrow VV^*$ analysis, because of the strong upward-fluctuation in the $H\rightarrow \tau ^{+} \tau ^{-}$ analysis. A summary of the upper limits on the branching ratios from the individual searches, and their combination, is given in Fig. 9. In the EFT framework, the limits observed for the respective branching ratios translate to a limit on the Wilson coefficients of the $tHu$ dimension-6 operators of $|C_{u\phi }^{ut,tu} |< 0.68(0.56)$ at 95 % CL, assuming $C_{u\phi }^{ct,tc} = 0$, and for a mass scale $\Lambda ={1}\,\hbox {TeV}$. The analogous limit on the Wilson coefficient of the $tHc$ dimension-6 operator amounts to $|C_{u\phi }^{ct,tc} |< 0.78(0.64)$ assuming $C_{u\phi }^{ut,tu} = 0$.

12 Conclusion

A search is reported for FCNC couplings between the top quark, the Higgs boson and a second up-type quark in final states containing either two leptons of the same charge or three leptons. This search uses ${140}\,\hbox {fb}^{-1}$ of proton–proton collision data collected with the ATLAS detector at the LHC between 2015 and 2018. The FCNC processes are considered in $t\bar{t}$ production where either the top quark or the top antiquark decays via $t\rightarrow Hq$, and in the production of a single top quark with a Higgs boson via the $q\rightarrow tH$ FCNC process. The results are compatible with the SM and no evidence of FCNC couplings is observed. Upper limits at 95 % CL are set on the branching ratio $\mathcal {B}(t\rightarrow Hq)$. The observed (expected) upper limits on the branching ratio are $\mathcal {B}(t\rightarrow Hu)<2.8\,(3.0)\times 10^{-4}$ and $\mathcal {B}(t\rightarrow Hc)<3.3\,(3.8)\times 10^{-4}$. They are reinterpreted as limits on the average of the Wilson coefficients for the left-handed and the right-handed dimension-6 operators modelling the effective $tHq$ couplings with a new physics scale at $\Lambda ={1}\,\hbox {TeV}$. These limits amount to $|C_{u\phi }^{ut,tu} |<0.71\,(0.73)$ and $|C_{u\phi }^{ct,tc} |<0.76\,(0.82)$. These are the most stringent upper limits reported by any analysis searching for $tHq$ FCNC couplings with the ATLAS detector. They are also lower than any other search for these couplings in $2\ell \text {SS}$ and $3\ell$ final states.

The results are combined with three other searches for $tHq$ FCNC couplings with the ATLAS detector. The combined observed (expected) limits set by all four analyses, considering correlations between the individual searches, are $\mathcal {B}(t\rightarrow Hu)<2.6\,(1.8)\times 10^{-4}$ and $\mathcal {B}(t\rightarrow Hc)<3.4\,(2.3)\times 10^{-4}$ on the branching ratios, and $|C_{u\phi }^{ut,tu} |< 0.68(0.56)$ and $|C_{u\phi }^{ct,tc} |< 0.78(0.64)$ on the dimension-6 Wilson coefficients.

Appendix

Tables 11, 12, 13, 14 list the input variables to the NNs used in the four signal regions, approximately ordered by the increase in significance provided by each variable. The exact order differs among the two signal processes.

Table 11 List of input variables to the NN in the $\text {SR}3\ell \text {Prod}$, approximately ordered by the increase in significance provided by each variable. The exact order differs among the two signal processes

Table 12 List of input variables to the NN in the $\text {SR}3\ell \text {Dec}$, approximately ordered by the increase in significance provided by each variable. The exact order differs among the two signal processes. Variables labelled NICE were reconstructed with a fulfilled NICE Reco condition

Table 13 List of input variables to the NN in the $\text {SR}2\ell \text {Prod}$, approximately ordered by the increase in significance provided by each variable. The exact order differs among the various signal processes

Table 14 List of input variables to the NN in the $\text {SR}2\ell \text {Dec}$, approximately ordered by the increase in significance provided by each variable. The exact order differs among the various signal processes