Search for new physics in top quark production in dilepton final states in proton-proton collisions at $\sqrt{s} = 13\,\text {TeV}$

Abstract

A search for new physics in top quark production is performed in proton-proton collisions at $13\,\text {TeV}$. The data set corresponds to an integrated luminosity of $35.9{\,\text {fb}^{-1}}$ collected in 2016 with the CMS detector. Events with two opposite-sign isolated leptons (electrons or muons), and $\mathrm{b}$ quark jets in the final state are selected. The search is sensitive to new physics in top quark pair production and in single top quark production in association with a $\mathrm{W}$ boson. No significant deviation from the standard model expectation is observed. Results are interpreted in the framework of effective field theory and constraints on the relevant effective couplings are set, one at a time, using a dedicated multivariate analysis. This analysis differs from previous searches for new physics in the top quark sector by explicitly separating $\mathrm{t}\mathrm{W}$ from $\mathrm{t}{\bar{\mathrm{t}}}$ events and exploiting the specific sensitivity of the $\mathrm{t}\mathrm{W}$ process to new physics.

1 Introduction

Because of its large mass, close to the electroweak (EW) symmetry breaking scale, the top quark is predicted to play an important role in several new physics scenarios. If the new physics scale is in the available energy range of the CERN LHC, the existence of new physics could be directly observed via the production of new particles. Otherwise, new physics could affect standard model (SM) interactions indirectly, through modifications of SM couplings or enhancements of rare SM processes. In this case, it is useful to introduce a model independent approach to parametrize and constrain possible deviations from SM predictions, independently of the fundamental theory of new physics.

Several searches for new physics in the top quark sector including new non-SM couplings of the top quark have been performed at the Tevatron and LHC colliders [1,2,3,4, 4,5,6,7,8,9,10]. Most of the previous analyses followed the anomalous coupling approach in which the SM Lagrangian is extended for possible new interactions. Another powerful framework to parametrize deviations with respect to the SM prediction is the effective field theory (EFT) [11, 12]. Constraints obtained on anomalous couplings can be translated to the effective coupling bounds [1, 13]. Several groups have performed global fits of top quark EFT to unfolded experimental data from the Tevatron and LHC colliders [14, 15]. Due to the limited access to data and details of the associated uncertainties, correlations between various cross section measurements and related uncertainties are neglected in a global fit on various unfolded measurements. On the other hand, EFT operators could affect backgrounds for some processes constructively or destructively while cross sections are measured with the SM assumptions for background processes. Inside the CMS Collaboration and with direct access to data, all mentioned points could be considered properly.

In this paper, the EFT approach is followed to search for new physics in the top quark sector in the dilepton final states. In Refs. [13, 16], all dimension-six operators that contribute to top quark pair ($\mathrm{t}{\bar{\mathrm{t}}}$) production and single top quark production in association with a $\mathrm{W}$ boson ($\mathrm{t}\mathrm{W}$) are investigated. The operators and the related effective Lagrangians, which are relevant for dilepton final states, can be written as [12]:

$$\begin{aligned} O_{\phi \mathrm{q}}^{(3)}&= (\phi ^+\uptau ^iD_\upmu \phi )({\bar{\mathrm{q}}}\upgamma ^\upmu \uptau ^i\mathrm{q}), \nonumber \\ L_{\mathrm {eff}}&=\frac{{\mathrm {C}}_{\phi \mathrm{q}}^{(3)}}{{\sqrt{2}}{\varLambda }^2} \mathrm{g}v^2{\bar{\mathrm{b}}}\upgamma ^\upmu P_{\mathrm {L}}\mathrm{t}\mathrm{W}^-_\upmu +\text {h.c.}, \end{aligned}$$

(1)

$$\begin{aligned} O_{\mathrm{t}\mathrm{W}}&= ({\bar{\mathrm{q}}}\sigma ^{\upmu \upnu }\uptau ^i\mathrm{t})\tilde{\phi }\mathrm{W}^i_{\upmu \upnu }, \nonumber \\ L_{\mathrm {eff}}&=-2\frac{{\mathrm {C}}_{\mathrm{t}\mathrm{W}}}{{\varLambda }^2}v{\bar{\mathrm{b}}}\sigma ^{\upmu \upnu }P_{\mathrm {R}}\mathrm{t}\partial _\upnu \mathrm{W}^-_\upmu +\text {h.c.}, \end{aligned}$$

(2)

$$\begin{aligned} O_{\mathrm{t}\mathrm{G}}&= ({\bar{\mathrm{q}}}\sigma ^{\upmu \upnu }\lambda ^a \mathrm{t})\tilde{\phi }\mathrm{G}^a_{\upmu \upnu }, \nonumber \\ L_{\mathrm {eff}}&= \frac{{\mathrm {C}}_{\mathrm{t}\mathrm{G}}}{{\sqrt{2}}{\varLambda }^2}v\left( {\bar{\mathrm{t}}}\sigma ^{\upmu \upnu }\lambda ^a \mathrm{t}\right) \mathrm{G}_{\upmu \upnu }^a+\text {h.c.}, \end{aligned}$$

(3)

$$\begin{aligned} O_{\mathrm{G}}&= f_{abc} \mathrm{G}^{a\upnu }_{\upmu } \mathrm{G}^{b\uprho }_{\upnu } \mathrm{G}^{\mathrm{c}\upmu }_{\uprho }, \nonumber \\ L_{\mathrm {eff}}&= \frac{{\mathrm {C}}_{\mathrm{G}}}{{\varLambda }^2} f_{abc}\mathrm{G}^{a\upnu }_\upmu \mathrm{G}^{b\uprho }_\upnu \mathrm{G}^{\mathrm{c}\upmu }_{\uprho }, \end{aligned}$$

(4)

$$\begin{aligned} O_{\mathrm{u}(\mathrm{c})\mathrm{G}}&= ({\bar{\mathrm{q}}}\sigma ^{\upmu \upnu }\lambda ^a \mathrm{t})\tilde{\phi }\mathrm{G}^a_{\upmu \upnu }, \nonumber \\ L_{\mathrm {eff}}&= \frac{{\mathrm {C}}_{\mathrm{u}(\mathrm{c})\mathrm{G}}}{{\sqrt{2}}{\varLambda }^2}v\left( {\bar{\mathrm{u}}}\left( {\bar{\mathrm{c}}}\right) \sigma ^{\upmu \upnu }\lambda ^a \mathrm{t}\right) \mathrm{G}_{\upmu \upnu }^a+\text {h.c.}, \end{aligned}$$

(5)

where $D_\upmu =\partial _\upmu -i\mathrm{g}_s\frac{1}{2}\lambda ^a\mathrm{G}_\upmu ^a-i\mathrm{g}\frac{1}{2}\uptau ^i\mathrm{W}_\upmu ^i-i\mathrm{g}^{'}YB_\upmu$, $\mathrm{W}_{\upmu \upnu }^i=\partial _\upmu \mathrm{W}_\upnu ^i-\partial _\upnu \mathrm{W}^i_\upmu +\mathrm{g}\epsilon _{ijk}\mathrm{W}^j_\upmu \mathrm{W}^k_\upnu$, $\mathrm{G}_{\upmu \upnu }^a=\partial _\upmu \mathrm{G}_\upnu ^a-\partial _\upnu \mathrm{G}^a_\upmu +\mathrm{g}_sf^{abc}\mathrm{G}^b_\upmu \mathrm{G}^c_\upnu$, $\sigma ^{\upmu \upnu } = \frac{1}{2}[\upgamma ^{\upmu },\upgamma ^{\upnu }]$, $P_{\mathrm {L,R}}=\frac{1}{2} (1\mp \upgamma ^5)$, and the symbols $\mathrm{q}$, $\mathrm{t}$ and $\phi$ ($\tilde{\phi }=\epsilon \phi ^*$) in the operators represent the left-handed quark doublet, the right-handed top quark singlet, and the Higgs boson doublet fields, respectively. The parameters ${\hbox {C}}_{\phi \mathrm{q}}^{(3)}$, ${\hbox {C}}_{\mathrm{t}\mathrm{W}}$, ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$, ${\hbox {C}}_{\mathrm{G}}$ and ${\hbox {C}}_{\mathrm{u}(\mathrm{c})\mathrm{G}}$ stand for the dimensionless Wilson coefficients, also called effective couplings. The variable ${\varLambda }$ represents the energy scale beyond which new physics becomes relevant. A detailed description of the operators is given in Refs. [13, 16, 17]. In this analysis, four-fermion operators involved in $\mathrm{t}{\bar{\mathrm{t}}}$ production are not probed. Up to order ${\varLambda }^{-2}$, the $\mathrm{t}\mathrm{W}$ and $\mathrm{t}{\bar{\mathrm{t}}}$ production cross sections and most of the differential observables considered in this analysis do not receive CP-odd contributions. Therefore, we only probe CP-even operators with real coefficients. The operators $O_{\phi \mathrm{q}}^{(3)}$ and $O_{\mathrm{t}\mathrm{W}}$ modify the SM interaction between the $\mathrm{W}$ boson, top quark, and $\mathrm{b}$ quark ($\mathrm{W}\mathrm{t}\mathrm{b}$). We consider the EFT effects in the production of top quarks not in their decays [18]. The operator $O_{\mathrm{t}\mathrm{G}}$ is called the chromomagnetic dipole moment operator of the top quark and can arise from various models of new physics [19, 20]. The triple-gluon field strength operator $O_{\mathrm{G}}$ represents the only genuinely gluonic CP conserving term that can appear at dimension six within an effective strong interaction Lagrangian. Although it is shown that jet production at the LHC can set a tight constraint on the ${\hbox {C}}_{\mathrm{G}}$ [21], $\mathrm{t}{\bar{\mathrm{t}}}$ production is also considered as a promising channel [22, 23]. The operators $O_{\mathrm{u}\mathrm{G}}$ and $O_{\mathrm{c}\mathrm{G}}$ lead to flavor-changing neutral current (FCNC) interactions of the top quark and contribute to $\mathrm{t}\mathrm{W}$ production. The effect of introducing new couplings ${\hbox {C}}_{\phi \mathrm{q}}^{(3)}$, ${\hbox {C}}_{\mathrm{t}\mathrm{W}}$, ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$ and ${\hbox {C}}_{\mathrm{u}(\mathrm{c})\mathrm{G}}$ can be investigated in $\mathrm{t}\mathrm{W}$ production. The chromomagnetic dipole moment operator of the top quark also affects $\mathrm{t}{\bar{\mathrm{t}}}$ production. In the case of ${\hbox {C}}_{\mathrm{G}}$ coupling, only $\mathrm{t}{\bar{\mathrm{t}}}$ production is modified. It should be noted that the $O_{\mathrm{t}\mathrm{W}}$ and $O_{\mathrm{t}\mathrm{G}}$ operators with imaginary coefficients lead to CP-violating effects. Representative Feynman diagrams for SM and new physics contributions in $\mathrm{t}\mathrm{W}$ and $\mathrm{t}{\bar{\mathrm{t}}}$ production are shown in Fig. 1.

A variety of limits have been set on the $\mathrm{W}\mathrm{t}\mathrm{b}$ anomalous coupling through single top quark t-channel production and measurements of the $\mathrm{W}$ boson polarization from top quark decay by the D0 [1], ATLAS [2, 3] and CMS [4, 5] Collaborations. Direct limits on the top quark chromomagnetic dipole moment have been obtained by the CMS Collaboration at 7 and $13\,\text {TeV}$ using top quark pair production events [6, 10]. Searches for top quark FCNC interactions have been performed at the Tevatron [7, 8] and LHC [4, 9] via single top quark production and limits are set on related anomalous couplings.

Fig. 1

Representative Feynman diagrams for the $\mathrm{t}\mathrm{W}$ (left panel) and $\mathrm{t}{\bar{\mathrm{t}}}$ (right panel) production at leading order. The upper row presents the SM diagrams, the middle and lower rows present diagrams corresponding to the $O_{\phi \mathrm{q}}^{(3)}$, $O_{\mathrm{t}\mathrm{W}}$,$O_{\mathrm{t}\mathrm{G}}$, $O_{\mathrm{G}}$ and $O_{\mathrm {u/cG}}$ contributions

In this paper, a search for new physics in top quark production using an EFT framework is reported. This is the first such search for new physics that uses the $\mathrm{t}\mathrm{W}$ process. Final states with two opposite-sign isolated leptons (electrons or muons) in association with jets identified as originating from the fragmentation of a bottom quark (“$\mathrm{b}$jets”) are analyzed. The search is sensitive to new physics contributions to $\mathrm{t}\mathrm{W}$ and $\mathrm{t}{\bar{\mathrm{t}}}$ production, and the six effective couplings, ${\hbox {C}}_{\mathrm{G}}$, ${\hbox {C}}_{\phi \mathrm{q}}^{(3)}$, ${\hbox {C}}_{\mathrm{t}\mathrm{W}}$, ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$, ${\hbox {C}}_{\mathrm{u}\mathrm{G}}$, and ${\hbox {C}}_{\mathrm{c}\mathrm{G}}$, are constrained assuming one non-zero effective coupling at a time. The effective couplings affect both the rate of $\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ production and the kinematic distributions of final state particles. For the ${\hbox {C}}_{\phi \mathrm{q}}^{(3)}$, ${\hbox {C}}_{\mathrm{t}\mathrm{W}}$, ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$, and ${\hbox {C}}_{\mathrm{G}}$ effective couplings, the deviation from the SM prediction is dominated by the interference term between SM and new physics diagrams, which is linear with respect to the effective coupling. Therefore, the kinematic distributions of the final-state particles vary as a function of the Wilson coefficients. For small effective couplings the kinematic distributions approach those predicted by the SM. On the other hand, the new physics terms due to the ${\hbox {C}}_{\mathrm{u}\mathrm{G}}$ and ${\hbox {C}}_{\mathrm{c}\mathrm{G}}$ effective couplings do not interfere with the SM $\mathrm{t}\mathrm{W}$ process, and the kinematic distributions of final-state particles are determined by the new physics terms independently of the SM prediction. In this analysis, we use the rates of $\mathrm{t}\mathrm{W}$ and $\mathrm{t}{\bar{\mathrm{t}}}$ production to probe the ${\hbox {C}}_{\phi \mathrm{q}}^{(3)}$, ${\hbox {C}}_{\mathrm{t}\mathrm{W}}$, ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$, and ${\hbox {C}}_{\mathrm{G}}$ effective couplings. Variations in both rate and kinematic distributions of final-state particles are employed to probe the ${\hbox {C}}_{\mathrm{u}\mathrm{G}}$ and ${\hbox {C}}_{\mathrm{c}\mathrm{G}}$ effective couplings. The analysis utilizes proton-proton ($\mathrm{p}\mathrm{p}$) collision data collected by the CMS experiment in 2016 at a center-of-mass energy of $13\,\text {TeV}$, corresponding to an integrated luminosity of $35.9{\,\text {fb}^{-1}}$.

The paper is structured as follows. In Sect. 2, a description of the CMS detector is given and the simulated samples used in the analysis are detailed. The event selection and the SM background estimation are presented in Sect. 3. Section 4 presents a description of the signal extraction procedure. An overview of the systematic uncertainty treatment is given in Sect. 5. Finally, the constraints on the effective couplings are presented in Sect. 6, and a summary is given in Sect. 7.

2 The CMS detector and event simulation

The central feature of the CMS apparatus is a superconducting solenoid of 6 m internal diameter, providing a magnetic field of 3.8 T. Within the solenoid volume are a silicon pixel and strip tracker, a lead tungstate crystal electromagnetic calorimeter (ECAL), and a brass and scintillator hadron calorimeter, each composed of a barrel and two endcap sections. Forward calorimeters extend the pseudorapidity ($\eta$) coverage provided by the barrel and endcap detectors. Muons are detected in gas-ionisation chambers embedded in the steel flux-return yoke outside the solenoid. A more detailed description of the CMS detector, together with a definition of the coordinate system used and the relevant kinematic variables, can be found in Ref. [24].

The Monte Carlo (MC) samples for the $\mathrm{t}{\bar{\mathrm{t}}}$, $\mathrm{t}\mathrm{W}$ and diboson ($\mathrm {VV} = \mathrm{W}\mathrm{W}$, $\mathrm{W}\mathrm{Z}$, $\mathrm{Z}\mathrm{Z}$) SM processes are simulated using the Powheg-Box event generator (v1 for $\mathrm{t}\mathrm{W}$, v2 for $\mathrm{t}{\bar{\mathrm{t}}}$ and diboson) [25,26,27,28] at the next-to-leading order (NLO), interfaced with pythia (v8.205) [29] to simulate parton showering and to match soft radiations with the contributions from the matrix elements. The pythia tune CUETP8M1 [30] is used for all samples except for the $\mathrm{t}{\bar{\mathrm{t}}}$ sample, for which the tune CUETP8M2 [31] is used. The NNPDF3.0 [32] set of the parton distribution functions (PDFs) is used. The $\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ samples are normalized to the next-to-next-to-leading order (NNLO) and approximate NNLO cross section calculations, respectively [33, 34]. In order to better describe the transverse momentum ($p_{\mathrm {T}}$) distribution of the top quark in $\mathrm{t}{\bar{\mathrm{t}}}$ events, the top quark $p_{\mathrm {T}}$ spectrum simulated with powheg is reweighted to match the differential top quark $p_{\mathrm {T}}$ distribution at NNLO quantum chromoDynamics (QCD) accuracy and including EW corrections calculated in Ref. [35]. Other SM background contributions, from Drell–Yan (DY), $\mathrm{t}{\bar{\mathrm{t}}}{+}\hbox {V}$, $\mathrm{t}{\bar{\mathrm{t}}}{+}\upgamma$, and $\mathrm{W}+\upgamma$ processes, are simulated at NLO using the MadGraph 5_amc@nlo (v2.2.2) event generator [36,37,38], interfaced with pythia v8 for parton showering and hadronization. The events include the effects of additional $\mathrm{p}\mathrm{p}$ interactions in the same or nearby bunch crossings (pileup) and are weighted according to the observed pileup distribution in the analyzed data. The CMS detector response is simulated using Geant4  (v9.4) [39, 40], followed by a detailed trigger simulation. Simulated events are reconstructed with the same algorithms as used for data.

In order to calculate the total cross sections for the $\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ processes and generate events in the presence of new effective interactions, the operators of Eqs. 1–5 have been implemented in the universal FeynRules output (UFO) format [41] through the FeynRules package [42]. The output EFT model is used in the MadGraph 5_amc@nlo (v2.2.2) event generator [36, 37]. If we allow for the presence of one operator at a time, the total cross section up to $\mathcal {O}({\varLambda }^{-4})$ can be parametrized as

$$\begin{aligned} \sigma =\sigma _{\mathrm {SM}}+ {\mathrm {C}}_{i}\sigma _i^{(1)}+ {\mathrm {C}}_{i}^2\sigma _{i}^{(2)}, \end{aligned}$$

(6)

where the ${\hbox {C}}_i$s are effective couplings introduced in Eqs. 1–5. Here, $\sigma _i^{(1)}$ is the contribution to the cross section due to the interference term between the SM diagrams and diagrams with one EFT vertex. The cross section $\sigma _i^{(2)}$ is the pure new physics contribution. We use the most precise available SM cross section prediction, which are $\sigma _{\text {SM}}^{\mathrm{t}{\bar{\mathrm{t}}}}=832^{+20}_{-29}\,\text {(scales)}\pm 35\,(\text {PDF}+\alpha _S) \, \hbox {pb}$ and $\sigma _{\text {SM}}^{\mathrm{t}\mathrm{W}}=71.7\pm 1.8\,\text {(scales)}\pm 3.4\,(\text {PDF}+\alpha _S) \, \hbox {pb}$ for $\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ production, respectively [33, 34], where the $\alpha _S$ is strong coupling constant. The first uncertainty reflects the uncertainties in the factorization and renormalization scales. In the framework of EFT, the $\sigma _i^{(1)}$ and $\sigma _i^{(2)}$ terms have been calculated at NLO accuracy for all of the operators, except $O_{\mathrm{G}}$ [16, 43, 44]. At the time the work for this paper was concluded, there was no available UFO model including the $O_{\mathrm{G}}$ operator at the NLO. The values of $\sigma _i^{(1)}$ and $\sigma _i^{(2)}$ for various effective couplings at LO and available K factors are given in Table 1.

Table 1 Contribution to the cross section due to the interference between the SM diagrams and diagrams with one EFT vertex ($\sigma _i^{(1)}$), and the pure new physics ($\sigma _i^{(2)}$) for $\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ production [in pb] for the various effective couplings for ${\varLambda }= 1\,\text {TeV}$. The respective K factors ($\sigma _i^{\mathrm {NLO}}/\sigma _i^{\mathrm {LO}}$) are also shown

3 Event selection and background estimation

The event selection for this analysis is similar to the one used in Ref. [10]. The events of interest are recorded by the CMS detector using a combination of dilepton and single-lepton triggers. Single-lepton triggers require at least one isolated electron (muon) with $p_{\mathrm {T}} >27\,(24)\,\text {GeV}$. The dilepton triggers select events with at least two leptons with loose isolation requirements and $p_{\mathrm {T}}$ for the leading and sub-leading leptons greater than 23 and 12 (17 and 8) $\text {GeV}$ for the $\mathrm{e}\mathrm{e}$ ($\upmu \upmu$) final state. In the $\mathrm{e}\upmu$ final state, in the case of the leading lepton being an electron, the events are triggered if the electron-muon pair has a $p_{\mathrm {T}}$ greater than 23 and $8 \,\text {GeV}$ for the electron and muon, respectively. In the case of the leading lepton being a muon, the trigger thresholds are 23 and $12\,\text {GeV}$ for the muon and electron, respectively [45].

Offline, the particle-flow (PF) algorithm [46] aims to reconstruct and identify each individual particle with an optimized combination of information from the various elements of the CMS detector. Electron candidates are reconstructed using tracking and ECAL information [47]. Requirements on electron identification variables based on shower shape and track-cluster matching are further applied to the reconstructed electron candidates, together with isolation criteria [10, 47]. Electron candidates are selected with $p_{\mathrm {T}} >20\,\text {GeV}$ and $|\eta |< 2.4$. Electron candidates within the range $1.444< |\eta |< 1.566$, which corresponds to the transition region between the barrel and endcap regions of the ECAL, are not considered. Information from the tracker and the muon spectrometer are combined in a global fit to reconstruct muon candidates [48]. Muon candidates are further required to have a high-quality fit including a minimum number of hits in both systems, and to be isolated [10, 48]. The muons used in this analysis are selected inside the fiducial region of the muon spectrometer, $|\eta |<2.4$, with a minimum $p_{\mathrm {T}}$ of $20\,\text {GeV}$.

The PF candidates are clustered into jets using the anti-$k_{\mathrm {T}}$ algorithm with a distance parameter of 0.4 [49,50,51]. Jets are calibrated in data and simulation, accounting for energy deposits of particles from pileup [52]. Jets with $p_{\mathrm {T}} >30\,\text {GeV}$ and $|\eta | <2.4$ are selected; loose jets are defined as jets with the $p_{\mathrm {T}}$ range between 20 and $30\,\text {GeV}$. Jets originating from the hadronization of $\mathrm{b}$ quarks are identified using the combined secondary vertex algorithm [53]; this algorithm combines information from track impact parameters and secondary vertices identified within a given jet. The chosen working point provides a signal identification efficiency of approximately 68% with a probability to misidentify light-flavor jets as $\mathrm{b}$jets of approximately 1% [53]. The missing transverse momentum vector ${\vec {p}}_{\mathrm {T}}^{\text {miss}}$ is defined as the projection on the plane perpendicular to the proton beams axis of the negative vector sum of the momenta of all reconstructed PF candidates in the event [54]. Corrections to the jet energies are propagated to ${\vec {p}}_{\mathrm {T}}^{\text {miss}}$. Its magnitude is referred to as $p_{\mathrm {T}} ^\text {miss}$.

Events are required to have at least two leptons (electrons or muons) with opposite sign and with an invariant mass above $20\,\text {GeV}$. The leading lepton must fulfill $p_{\mathrm {T}} >25\,\text {GeV}$. For the same-flavor lepton channels, to suppress the DY background, the dilepton invariant mass must not be within $15\,\text {GeV}$ of the $\mathrm{Z}$ boson mass and a minimal value (of $60\,\text {GeV}$) on $p_{\mathrm {T}} ^\text {miss}$ is applied.

The events are divided into the $\mathrm{e}\mathrm{e}$, $\mathrm{e}\upmu$, and $\upmu \upmu$ channels according to the flavors of the two leptons with the highest $p_{\mathrm {T}}$ and are further categorized in different bins depending on the number of jets (“n-jets”) and number $\mathrm{b}$-tagged jets (“m-tags”) in the final state. The largest number of $\mathrm{t}\mathrm{W}$ events is expected in the category with exactly one $\mathrm{b}$-tagged jet (1-jet,1-tag) followed by the category with two jets, of which one a $\mathrm{b}$jet (2-jets,1-tag). Events in the categories with more than two jets and exactly two $\mathrm{b}$-tagged jets are dominated by the $\mathrm{t}{\bar{\mathrm{t}}}$ process ($\ge$2-jets,2-tags). Categories with zero $\mathrm{b}$jets are dominated by DY events in the $\mathrm{e}\mathrm{e}$ and $\upmu \upmu$ channels and are not used in the analysis. However, in the $\mathrm{e}\upmu$ channel, the contamination of DY events is lower and a significant number of $\mathrm{t}\mathrm{W}$ events is present in the category with one jet and zero $\mathrm{b}$-tagged jets (1-jet,0-tag). The latter category is included in this analysis. In Fig. 2, the data in the ten search regions are shown together with the SM background predictions.

Fig. 2

The observed number of events and SM background predictions in the search regions of the analysis for the $\mathrm{e}\mathrm{e}$ (upper left), $\upmu \upmu$ (upper right), and $\mathrm{e}\upmu$ (lower) channels. The hatched bands correspond to the quadratic sum of statistical and systematic uncertainties in the event yield for the SM background predictions. The ratios of data to the sum of the predicted yields are shown at the bottom of each plot. The narrow hatched bands represent the contribution from the statistical uncertainty in the MC simulation

The contributions of SM processes leading to two prompt leptons in the final state are estimated from simulated samples and are normalised to the integrated luminosity of the data. These contributions originate mainly from $\mathrm{t}{\bar{\mathrm{t}}}$, $\mathrm{t}\mathrm{W}$ and DY production. Other SM processes, such as diboson, $\mathrm{t}{\bar{\mathrm{t}}}{+} \hbox {V}$ and $\mathrm{t}{\bar{\mathrm{t}}}{+}\upgamma$ have significantly smaller contributions.

To correct the DY simulation for the efficiency of the $p_{\mathrm {T}} ^\text {miss}$ threshold and for the mismodeling of the heavy-flavour content, scale factors are derived using the ratio of the numbers of simulated events inside and outside the dilepton invariant mass window, 76–$106 \,\text {GeV}$. The observed event yield inside the window is scaled to estimate the DY background outside the mass window [55].

The nonprompt lepton backgrounds which contain fake lepton(s) from a misreconstructed $\upgamma$ or jet(s) are also considered. The contribution of misidentified or converted $\upgamma$ events from the $\mathrm{W}\upgamma$ process is estimated from MC simulation. The contribution from $\mathrm{W}$+jets and multijet processes is estimated by a data-based technique using events with same-sign leptons. The method is based on the assumption that the probability of assigning positive or negative charge to the fake lepton is equal. Therefore, the background contribution from fake leptons in the final selection (opposite-sign sample) can be estimated from the corresponding sample with same-sign leptons. In this latter same-sign event sample, the remaining small contribution from prompt-lepton backgrounds is subtracted from data using MC samples.

After all selections, the expected numbers of events from $\mathrm{t}\mathrm{W}$, $\mathrm{t}{\bar{\mathrm{t}}}$, DY, and remaining background contributions mentioned above, as well as the total number of background events are reported in Table 2 for the $\mathrm{e}\mathrm{e}$, $\mathrm{e}\upmu$, and $\upmu \upmu$ channels and for the various (n-jets,m-tags) categories. We find generally very good agreement between data and predictions, within the uncertainties of the data.

Table 2 Number of expected events from $\mathrm{t}\mathrm{W}$, $\mathrm{t}{\bar{\mathrm{t}}}$ and DY production, from the remaining backgrounds (other), total background contribution and observed events in data after all selections for the $\mathrm{e}\mathrm{e}$, $\mathrm{e}\upmu$, and $\upmu \upmu$ channels and for different (n-jets,m-tags) categories. The uncertainties correspond to the statistical contribution only for the individual background predictions and to the quadratic sum of the statistical and systematic contributions for the total background predictions

4 Signal extraction using neural networks tools

The purpose of the analysis is to search for deviations from the SM predictions in the $\mathrm{t}\mathrm{W}$ and $\mathrm{t}{\bar{\mathrm{t}}}$ production due to new physics, parametrized with the presence of new effective couplings. In order to investigate the effect of the non-zero effective couplings, it is important to find suitable variables with high discrimination power between the signal and the background. Depending on the couplings, the total yield or the distribution of the output of a neural network (NN) algorithm is employed, as summarized in Table 3. The NN algorithm used in this analysis is a multilayer perceptron [56].

Table 3 Summary of the observables used to probe the effective couplings in various (n-jets,m-tags) categories in the $\mathrm{e}\mathrm{e}$, $\mathrm{e}\upmu$, and $\upmu \upmu$ channels

All the effective couplings introduced in Sect. 1 can contribute to $\mathrm{t}\mathrm{W}$ production except the triple gluon field strength operator, $O_{\mathrm{G}}$ which only affects $\mathrm{t}{\bar{\mathrm{t}}}$ production. As observed in previous analysis [22] and confirmed here, the top quark $p_{\mathrm {T}}$ distribution is sensitive to the triple-gluon field-strength operator. The kinematic distributions of final-state particles show less discrimination power than the top quark $p_{\mathrm {T}}$ distribution. In addition, they vary with the value of ${\hbox {C}}_{\mathrm{G}}$ and approach the SM prediction for decreasing values of ${\hbox {C}}_{\mathrm{G}}$. Therefore, we use the total yield in various categories to constrain the ${\hbox {C}}_{\mathrm{G}}$ effective coupling.

The deviation from the SM $\mathrm{t}\mathrm{W}$ production because of the interference between the SM and the $O_{\mathrm{t}\mathrm{G}}$, $O_{\phi \mathrm{q}}^{(3)}$, and $O_{\mathrm{t}\mathrm{W}}$ operators is of the order of $1/{\varLambda }^2$. It is assumed that the new physics scale ${\varLambda }$ is larger than the scale we probe. Therefore, $1/{\varLambda }^4$ contributions from the new physics terms are small compared to the contribution from the interference term. The operator $O_{\phi \mathrm{q}}^{(3)}$ is similar to the SM $\mathrm{W}\mathrm{t}\mathrm{b}$ operator and leads to a rescaling of the SM $\mathrm{W}\mathrm{t}\mathrm{b}$ vertex [13]. The $O_{\mathrm{t}\mathrm{W}}$ and $O_{\mathrm{t}\mathrm{G}}$ operators lead to the right-handed Wtb interaction and a tensor-like ttg interaction, respectively, which are absent in the SM at the first order. Their effects have been investigated via the various kinematic distributions of the final-state particles considered in this analysis and are found to be not distinguishable from the SM $\mathrm{t}\mathrm{W}$ and $\mathrm{t}{\bar{\mathrm{t}}}$ processes for unconstrained values of the effective couplings within the current precision on data. After the selection described in Section 3, the dominant background comes from $\mathrm{t}{\bar{\mathrm{t}}}$ production, with a contribution of about 90%. In order to observe deviations from SM $\mathrm{t}\mathrm{W}$ production in the presence of the $O_{\phi \mathrm{q}}^{(3)}$, $O_{\mathrm{t}\mathrm{W}}$, and $O_{\mathrm{t}\mathrm{G}}$ effective operators, we need to separate $\mathrm{t}\mathrm{W}$ events from the large number of $\mathrm{t}{\bar{\mathrm{t}}}$ events. Two independent NNs are trained to separate $\mathrm{t}{\bar{\mathrm{t}}}$ events (the background) and $\mathrm{t}\mathrm{W}$ events (considered as the signal) in the (1-jet, 1-tag) (${\hbox {NN}}_{11}$) and (2-jets, 1-tag) (${\hbox {NN}}_{21}$) categories, which have significant signal contributions [57]. For the $\mathrm{e}\upmu$ channel, another NN is used for the (1-jet, 0-tag) (${\hbox {NN}}_{10}$) category, in which the $\mathrm{t}{\bar{\mathrm{t}}}$, $\mathrm{W}\mathrm{W}$, and DY events are combined and are considered as the background. A comparison between the observed data and the SM background prediction of the NN output shape in various (n-jets, m-tags) categories is shown for the $\mathrm{e}\mathrm{e}$ and $\upmu \upmu$ channels in Fig. 3 and for the $\mathrm{e}\upmu$ channel in Fig. 4 (left column).

The presence of the $O_{\mathrm{u}\mathrm{G}}$ and $O_{\mathrm{c}\mathrm{G}}$ operators changes the initial-state particle (see Fig. 1), and leads to different kinematic distributions for the final-state particles, compared to the SM $\mathrm{t}\mathrm{W}$ process. For these FCNC operators, new physics effects on final-state particle distributions are expected to be distinguishable from SM processes. In order to search for new physics due to the $O_{\mathrm{u}\mathrm{G}}$ and $O_{\mathrm{c}\mathrm{G}}$ effective operators, an NN (${\hbox {NN}}_{\mathrm {FCNC}}$) is used to separate SM backgrounds ($\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ events together) and new physics signals for events with exactly one $\mathrm{b}$-tagged jet with no requirement on the number of light-flavor jets (n-jets, 1-tag). The comparison of the NN output for data, SM background and signal ($\mathrm{t}\mathrm{W}$ events via FCNC interactions) is shown in Fig. 4 (right column) for the $\mathrm{e}\mathrm{e}$, $\mathrm{e}\upmu$, and $\upmu \upmu$ channels.

Fig. 3

The NN output distributions for data and simulation for the $\mathrm{e}\mathrm{e}$ (left) and $\upmu \upmu$ (right) channels in 1-jet, 1-tag (upper) and 2-jets, 1-tag (lower) categories. The hatched bands correspond to the quadratic sum of the statistical and systematic uncertainties in the event yield for the sum of signal and background predictions. The ratios of data to the sum of the predicted yields are shown at the lower panel of each graph. The narrow hatched bands represent the contribution from the statistical uncertainty in the MC simulation. In each plot, the expected distributions assuming specific values for the effective couplings (given in the legend) are shown as the solid curves

Fig. 4

The NN output distributions for (left) data and simulation for the $\mathrm{e}\upmu$ channel in 1-jet, 1-tag (upper) and 2-jets, 1-tag (middle) and 1-jet, 0-tag (lower) categories; and for (right) data, simulation, and FCNC signals in the n-jets, 1-tag category used in the limit setting for the $\mathrm{e}\mathrm{e}$ (upper), $\mathrm{e}\upmu$ (middle), and $\upmu \upmu$ (lower) channels. The hatched bands correspond to the quadratic sum of the statistical and systematic uncertainties in the event yield for the sum of signal and background predictions. The ratios of data to the sum of the predicted yields are shown at the lower panel of each graph. The narrow hatched bands represent the contribution from the statistical uncertainty in the MC simulation. In each plot, the expected distributions assuming specific values for the effective couplings (given in the legend) are shown as the solid curves

The various input variables for training the NN introduced above are described below and are shown in Table  4.

• M$_{\ell \ell }$ (where $\ell = \mathrm{e}$  or $\upmu$), invariant mass of dilepton system;

• $p_{\mathrm {T}} ^{\ell \ell }$, $p_{\mathrm {T}}$ of dilepton system;

• ${\varDelta }p_{\mathrm {T}} (\ell _1,\, \ell _2)$, $p_{\mathrm {T}} ^{\text {leading lepton}} - p_{\mathrm {T}} ^{\text {sub-leading lepton}}$;

• $p_{\mathrm {T}} ^{\ell _1}$, $p_{\mathrm {T}}$ of leading lepton;

• Centrality($\ell _1$, jet$_1$), scalar sum of $p_{\mathrm {T}}$ of the leading lepton and leading jet, over total energy of selected leptons and jets;

• Centrality($\ell _1,\, \ell _2$), scalar sum of $p_{\mathrm {T}}$ of the leading and sub-leading leptons, over total energy of selected leptons and jets;

• ${\varDelta }{\varPhi }(\ell \ell$, jet$_1$), ${\varDelta }{\varPhi }$ between dilepton system and leading jet where ${\varPhi }$ is azimuthal angle;

• $p_{\mathrm {T}}$ ($\ell \ell$, jet$_1$), $p_{\mathrm {T}}$ of dilepton and leading jet system;

• $p_{\mathrm {T}}$ ($\ell _1$, jet$_1$), $p_{\mathrm {T}}$ of leading lepton and leading jet system;

• Centrality($\ell \ell$, jet$_1$), scalar sum of $p_{\mathrm {T}}$ of the dilepton system and leading jet, over total energy of selected leptons and jets;

• ${\varDelta }$R($\ell _1,\, \ell _2$), $\sqrt{\smash [b]{ \, ( \eta ^{\ell _1} - \eta ^{\ell _2})^2 \, + \, ({\varPhi }^{\ell _1} - {\varPhi }^{\ell _2})^2}}$;

• ${\varDelta }$R($\ell _1$, jet$_1$), $\sqrt{\smash [b]{ \, ( \eta ^{\ell _1} - \eta ^{\text {jet}_1})^2 \, + \, ({\varPhi }^{\ell _1} - {\varPhi }^{ \text {jet}_1})^2}}$;

• M($\ell _1$, jet$_1$), invariant mass of leading lepton and leading jet;

• M(jet$_1$, jet$_2$), invariant mass of leading jet and sub-leading jet;

• ${\varDelta }$R($\ell _1$, jet$_2$), $\sqrt{\smash [b]{ \, ( \eta ^{\ell _1} - \eta ^{\text {jet}_2})^2 \, + \, ({\varPhi }^{\ell _1} - {\varPhi }^{\text {jet}_2})^2}}$;

• ${\varDelta }$R($\ell \ell$, jet$_1$), $\sqrt{\smash [b]{ \, ( \eta ^{\ell \ell } - \eta ^{\text {jet}_1})^2 \, + \, ({\varPhi }^{\ell \ell } - {\varPhi }^{\text {jet}_1})^2}}$;

• ${\varDelta }p_{\mathrm {T}} (\ell _2,~ \text {jet}_2)$, $p_{\mathrm {T}} ^{\ell _2} - p_{\mathrm {T}} ^{\text {jet}_2}$;

• M($\ell _2$, jet$_1$), invariant mass of sub-leading lepton and leading jet.

Table 4 Input variables for the NN used in the analysis in various bins of n-jets and m-tags. The symbols “$\times$” indicate the input variables used in the four NNs

5 Systematic uncertainties

The normalization and shape of the signal and the backgrounds are both affected by different sources of systematic uncertainty. For each source, an induced variation can be parametrized, and treated as a nuisance parameter in the fit that is described in the next section.

A systematic uncertainty of 2.5% is assigned to the integrated luminosity and is used for signal and background rates [58]. The efficiency corrections for trigger and offline selection of leptons were estimated by comparing the efficiency measured in data and in MC simulation using $\mathrm{Z}\rightarrow \ell \ell$ events, based on a “tag-and-probe” method as in Ref. [59]. The scale factors obtained are varied by one standard deviation to take into account the corresponding uncertainties in the efficiency. The jet energy scale and resolution uncertainties depend on $p_{\mathrm {T}}$ and $\eta$ of the jet and are computed by shifting the energy of each jet and propagating the variation to ${\vec {p}}_{\mathrm {T}}^{\text {miss}}$ coherently [60].

The uncertainty in the $\mathrm{b}$ tagging is estimated by varying the $\mathrm{b}$ tagging scale factors within one standard deviation [53]. Effects of the uncertainty in the distribution of the number of pileup interactions are evaluated by varying the effective inelastic $\mathrm{p}\mathrm{p}$ cross section used to predict the number of pileup interactions in MC simulation by ± 4.6% of its nominal value [61].

The uncertainty in the DY contribution in categories with one or two $\mathrm{b}$-tagged jets is considered to be 50 and 30% in the $\mathrm{e}\upmu$ and same-flavor dilepton channels, respectively [10, 57]. For the DY normalization in the (1-jet, 0-tag) category, an uncertainty of 15% is assigned [62]. In addition, systematic uncertainties related to the PDF, and to the renormalization and factorization scale uncertainty are taken into account for DY process in the (1-jet, 0-tag) category. The uncertainty in the yield of nonprompt lepton backgrounds is considered to be 50% [57]. Contributions to the background from $\mathrm{t}{\bar{\mathrm{t}}}$ production in association with a boson, as well as diboson production, are estimated from simulation and a systematic uncertainty of 50% is conservatively assigned [63].

Various uncertainties originate from the theoretical predictions. The effect of the renormalization and factorization scale uncertainty from the $\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ MC generators is estimated by varying the scales used during the generation of the simulation sample independently by a factor 0.5, 1 or 2. Unphysical cases, where one scale fluctuates up while the other fluctuates down, are not considered. The top quark $p_{\mathrm {T}}$ reweighting procedure, discussed in Sect. 2, is applied on top of the nominal powheg prediction at NLO to account for the higher-order corrections.

The uncertainty in the PDFs for each simulated signal process is obtained using the replicas of the NNPDF 3.0 set [64]. The most recent measurement of the top quark mass by CMS yields a total uncertainty of $\pm 0.49 \,\text {GeV}$ [65]. We consider variations of the top quark mass due to this uncertainty and they are found to be insignificant. At NLO QCD, $\mathrm{t}\mathrm{W}$ production is expected to interfere with $\mathrm{t}{\bar{\mathrm{t}}}$ production [66]. Two schemes for defining the $\mathrm{t}\mathrm{W}$ signal in a way that distinguishes it from the $\mathrm{t}{\bar{\mathrm{t}}}$ production have been compared in the analysis: the “diagram removal” (DR), in which all doubly resonant NLO $\mathrm{t}\mathrm{W}$ diagrams are removed, and the “diagram subtraction” (DS), where a gauge-invariant subtractive term modifies the NLO $\mathrm{t}\mathrm{W}$ cross section to locally cancel the contribution from $\mathrm{t}{\bar{\mathrm{t}}}$ production [66,67,68]. The DR method is used for the nominal $\mathrm{t}\mathrm{W}$ sample and the difference with respect to the sample simulated using the DS method is taken as a systematic uncertainty. The model parameter $h_{\text {damp}}$ in $\mathrm{t}{\bar{\mathrm{t}}}$ powheg [25] that controls the matching of the matrix elements to the pythia parton showers is varied from a top quark mass default value of $172.5 \,\text {GeV}$ by factors of 0.5 and 2 for estimating the uncertainties from the matching between jets from matrix element calculations and parton shower emissions. The renormalization scale for QCD emissions in the initial- and final-state radiation (ISR and FSR) is varied up and down by factors of 2 and $\sqrt{2}$, respectively, to account for parton shower QCD scale variation error in both $\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ samples [69]. In addition, several dedicated $\mathrm{t}{\bar{\mathrm{t}}}$ samples are used to estimate shower modeling uncertainties in both underlying event and color re-connections [10, 31, 69]. To estimate model uncertainties, $\mathrm{t}\mathrm{W}$ and $\mathrm{t}{\bar{\mathrm{t}}}$ samples are generated with powheg as described in Sect. 2, varying the relevant model parameters with respect to the nominal samples.

6 Constraints on the effective couplings

The six Wilson coefficients sensitive to new physics contributions in top quark interactions, as defined in Eqs. 1–5, are tested separately in the observed data. The event yields and the NN output distributions in each analysis category, summarized in Table 3, are used to construct a binned likelihood function. All sources of systematic uncertainty, described in Sect. 5, are taken into account as nuisance parameters in the fit. A simultaneous binned maximum-likelihood fit is performed to find the best fit value for each Wilson coefficient together with 68 and 95% confidence intervals (CIs) [70]. In this section, distributions of the log-likelihood functions are shown with one nonzero effective coupling at a time for ${\varLambda }= 1\,\text {TeV}$.

The SM cross section prediction for the $\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ processes, $\sigma _i^{(1)}$ and $\sigma _i^{(2)}$ (see Table 1), are accompanied by uncertainties in scales and PDFs. These theoretical uncertainties can affect the bounds on the Wilson coefficients. In order to study this effect, the fit is performed on data, while theoretical uncertainties are varied within one standard deviation and are shown together with the nominal results in the likelihood scan plots in Fig. 5. The nominal theoretical cross sections for $\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ processes are varied by [$+4.8\%,-5.5\%$] and [$+5.4\%,-5.4\%$] , respectively. These variations cover the uncertainties arising from the variations of factorization and renormalization scales and PDFs [10, 34]. The scale variations for $\sigma _i^{(1)}$ and $\sigma _i^{(2)}$ are evaluated to be within 1–25%. We assumed that the scale uncertainty is 100% correlated among the terms $\sigma _{\mathrm {SM}}$, $\sigma _i^{(1)}$, and $\sigma _i^{(2)}$.

6.1 Exclusion limits on the ${\hbox {C}}_{\mathrm{G}}$ effective coupling

In order to constrain the ${\hbox {C}}_{\mathrm{G}}$ coupling, the effect on the $\mathrm{t}{\bar{\mathrm{t}}}$ rate in various (n-jets,m-tags) categories is considered. The impact of the difference between the kinematic distributions of $\mathrm{t}{\bar{\mathrm{t}}}$ events from the $O_{\mathrm{G}}$ interaction and from the SM interactions on the acceptance is evaluated to be 3% for ${\hbox {C}}_{\mathrm{G}}\sim 1$. This uncertainty is considered only for the ${\hbox {C}}_{\mathrm{G}}$ coupling since the top $p_{\mathrm {T}}$  spectrum is affected considerably by this operator, while other operators lead to a $p_{\mathrm {T}}$  spectrum similar to the SM prediction for unconstrained values of the probed Wilson coefficients. The fit is performed simultaneously on the observed event yields in the categories presented in Fig. 2 in the (1-jet, 1-tag), (2-jets, 1-tag), and ($\ge$2-jets, 2-tags) categories for the $\mathrm{e}\mathrm{e}$, $\mathrm{e}\upmu$, and $\upmu \upmu$ channels. In addition, the (1-jet, 0-tag) category is included only for the $\mathrm{e}\upmu$ channel. The main limiting factor on the constraints in the ${\hbox {C}}_{\mathrm{G}}$ coupling is the uncertainty in the signal acceptance found after maximizing the likelihood, followed by uncertainties in the integrated luminosity calibration and the trigger scale factor.

The results of the fit for the individual channels and for all channels combined are listed in the first row of Table 5.

The results of the likelihood scans of the ${\hbox {C}}_{\mathrm{G}}$ coupling are shown in Fig. 5 (upper left plot). The likelihood scan result of the nominal fit, in which the nominal values of $\sigma _{\mathrm {SM}}$, $\sigma _i^{(1)}$, and $\sigma _i^{(2)}$ terms are assumed, is shown as the thick curve. The thin dashed curves are the results of the fit to the observed data when the assumed values of the $\sigma _{\mathrm {SM}}$, $\sigma _i^{(1)}$, and $\sigma _i^{(2)}$ terms are varied due to the scale and PDF uncertainties. As a second-order parametrization, given by Eq. 6 is used to fit the data, the resulting likelihood function could have two minima, as can be seen in some of the plots in Fig. 5.

6.2 Exclusion limits on the ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$, ${\hbox {C}}_{\phi \mathrm{q}}^{(3)}$, and ${\hbox {C}}_{\mathrm{t}\mathrm{W}}$ effective couplings

In order to set limits on the effective couplings ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$, ${\hbox {C}}_{\phi \mathrm{q}}^{(3)}$, and ${\hbox {C}}_{\mathrm{t}\mathrm{W}}$, we utilize the NN output distributions for both data and MC expectation in the (1-jet, 1-tag) and (2-jets, 1-tag) regions and event yields in the ($\ge$2-jets, 2-tags) region for the three dilepton channels. The inclusion of the ($\ge$2-jets, 2-tags) and (2-jets, 1-tag) categories provides a constraint of the normalization and systematic uncertainties in the $\mathrm{t}{\bar{\mathrm{t}}}$ background. In addition, the (1-jet, 0-tag) category is included for the $\mathrm{e}\upmu$ channel to increase the signal sensitivity. The results of the likelihood scans of the ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$, ${\hbox {C}}_{\phi \mathrm{q}}^{(3)}$, and ${\hbox {C}}_{\mathrm{t}\mathrm{W}}$ Wilson coefficients are shown in Fig. 5 for the combination of all channels. The inclusion of the ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$ coupling to the tW process tightens the 2 standard deviations band by 7%. The results for the individual channels, and the combined results are listed in Table 5 (second, third, and fourth rows). The three main sources of uncertainty that affect the interval determination are uncertainties in the DY estimation, integrated luminosity, and lepton identification scale factors for ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$; jet energy scale, tt and tW interference at NLO, and statistical uncertainty in MC samples for ${\hbox {C}}_{\phi \mathrm{q}}^{(3)}$; statistical uncertainty in data, jet energy scale, and the powheg matching method for ${\hbox {C}}_{\mathrm{t}\mathrm{W}}$ effective couplings.

6.3 Exclusion limits on the ${\hbox {C}}_{\mathrm{u}\mathrm{G}}$ and ${\hbox {C}}_{\mathrm{c}\mathrm{G}}$ effective couplings

Since the $\mathrm{t}\mathrm{W}$ production via FCNC interactions does not interfere with the SM $\mathrm{t}\mathrm{W}$ process (with the assumption of $|\text {V}_{\mathrm{t}\mathrm{d}} | = |\text {V}_{\mathrm{t}\mathrm{s}} | = 0$), the FCNC signal sample is used to set upper bounds on the related Wilson coefficients. Events with exactly one $\mathrm{b}$-tagged jet are included in the limit setting procedure with no requirement on the number of light-flavor jets (n-jets, 1-tag). The observed (median expected) $95\%$ confidence level ($\text {CL}$) upper limits on the product of cross section times branching fractions $\sigma (\mathrm{p}\mathrm{p}\rightarrow \mathrm{t}\mathrm{W})\mathcal {B}(\mathrm{W}\rightarrow \ell \upnu )^2$ for the ${\hbox {C}}_{\mathrm{u}\mathrm{G}}$ and ${\hbox {C}}_{\mathrm{c}\mathrm{G}}$ FCNC signals for the combination of the $\mathrm{e}\mathrm{e}$, $\upmu \upmu$, and $\mathrm{e}\upmu$ channels are found to be 0.11 (0.20) pb and 0.13 (0.26) pb, respectively. These results are used to calculate upper limits on the Wilson coefficients ${\hbox {C}}_{\mathrm{u}\mathrm{G}}$, ${\hbox {C}}_{\mathrm{c}\mathrm{G}}$, and on the branching fractions $\mathcal {B}(\mathrm{t}\rightarrow \mathrm{u}\mathrm{g})$ and $\mathcal {B}(\mathrm{t}\rightarrow \mathrm{c}\mathrm{g})$. The limits on the ${\hbox {C}}_{\mathrm{u}\mathrm{G}}$ and ${\hbox {C}}_{\mathrm{c}\mathrm{G}}$ couplings are summarized in the last two rows of Table 5, and correspond to the observed (expected) limits on $\mathcal {B}(\mathrm{t}\rightarrow \mathrm{u}\mathrm{g})<$ 0.12 (0.22)% and $\mathcal {B}(\mathrm{t}\rightarrow \mathrm{c}\mathrm{g})<$ 0.53 (1.05)% at $95\% \, \text {CL}$. The statistical uncertainty in data is the dominant source of uncertainty affecting the limits on the FCNC couplings. The second and third most important uncertainties originate from $\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ interferences at NLO and FSR in $\mathrm{t}{\bar{\mathrm{t}}}$ events.

The observed best fit together with one and two standard deviation bounds on the six Wilson coefficients, ${\hbox {C}}_{\phi \mathrm{q}}^{(3)}$, ${\hbox {C}}_{\mathrm{t}\mathrm{W}}$, ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$, ${\hbox {C}}_{\mathrm{G}}$, ${\hbox {C}}_{\mathrm{u}\mathrm{G}}$, and ${\hbox {C}}_{\mathrm{c}\mathrm{G}}$, obtained from the combination of all channels are shown in Fig. 6. Table 6 summarizes the effect of the most important uncertainty sources on the observed allowed intervals.

Table 5 Summary of the observed and expected allowed intervals on the effective couplings obtained in the $\mathrm{e}\mathrm{e}$, $\mathrm{e}\upmu$, and $\upmu \upmu$ channels, and all channels combined. All sources of systematic uncertainty, described in Sect. 5, are taken into account with the exception of the uncertainties on the SM cross section predictions for the tt and tW processes

Table 6 Estimation of the effect of the most important uncertainty sources on the observed allowed intervals of in the fit

Fig. 5

Observed (solid) and expected (dotted) log likelihoods for the effective couplings: ${\hbox {C}}_{\mathrm{G}}$ (upper left), ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$ (upper right), ${\hbox {C}}_{\mathrm{t}\mathrm{W}}$ (middle left), ${\hbox {C}}_{\phi \mathrm{q}}$ (middle right), ${\hbox {C}}_{\mathrm{u}\mathrm{G}}$ (lower left), and ${\hbox {C}}_{\mathrm{c}\mathrm{G}}$ (lower right). The dashed curves represent fits to the observed data with the variations of normalization due to the theoretical uncertainties

Fig. 6

Observed best fits together with one and two standard deviation bounds on the top quark effective couplings. The dashed line shows the SM expectation and the vertical lines indicate the $95\% \, \text {CL}$ bounds including the theoretical uncertainties

7 Summary

A search for new physics in top quark interactions is performed using $\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ events in dilepton final states. The analysis is based on data collected in $\mathrm{p}\mathrm{p}$ collisions at $13\,\text {TeV}$ by the CMS detector in 2016, corresponding to an integrated luminosity of $35.9{\,\text {fb}^{-1}}$. No significant excess above the standard model background expectation is observed. For the first time, both $\mathrm{t}{\bar{\mathrm{t}}}$ and $\mathrm{t}\mathrm{W}$ production are used simultaneously in a model independent search for effective couplings. The six effective couplings, ${\hbox {C}}_{\mathrm{G}}$, ${\hbox {C}}_{\mathrm{t}\mathrm{G}}$, ${\hbox {C}}_{\mathrm{t}\mathrm{W}}$, ${\hbox {C}}_{\phi \mathrm{q}}^{(3)}$, ${\hbox {C}}_{\mathrm{u}\mathrm{G}}$, and ${\hbox {C}}_{\mathrm{c}\mathrm{G}}$ are constrained using a dedicated multivariate analysis. The constraints presented, obtained by considering one operator at a time, are a useful first step toward more global approaches.