Search for new physics in top quark production in dilepton final states in proton-proton collisions at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sqrt{s} = 13\,\text {TeV} $$\end{document}s=13TeV

A search for new physics in top quark production is performed in proton-proton collisions at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$13\,\text {TeV} $$\end{document}13TeV. The data set corresponds to an integrated luminosity of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$35.9{\,\text {fb}^{-1}} $$\end{document}35.9fb-1 collected in 2016 with the CMS detector. Events with two opposite-sign isolated leptons (electrons or muons), and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm{b}$$\end{document}b quark jets in the final state are selected. The search is sensitive to new physics in top quark pair production and in single top quark production in association with a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm{W}$$\end{document}W boson. No significant deviation from the standard model expectation is observed. Results are interpreted in the framework of effective field theory and constraints on the relevant effective couplings are set, one at a time, using a dedicated multivariate analysis. This analysis differs from previous searches for new physics in the top quark sector by explicitly separating \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm{t}\mathrm{W}$$\end{document}tW from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm{t}{\bar{\mathrm{t}}}$$\end{document}tt¯ events and exploiting the specific sensitivity of the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm{t}\mathrm{W}$$\end{document}tW process to new physics.


Introduction
Because of its large mass, close to the electroweak (EW) symmetry breaking scale, the top quark is predicted to play an important role in several new physics scenarios. If the new physics scale is in the available energy range of the CERN LHC, the existence of new physics could be directly observed via the production of new particles. Otherwise, new physics could affect standard model (SM) interactions indirectly, through modifications of SM couplings or enhancements of rare SM processes. In this case, it is useful to introduce a model independent approach to parametrize and constrain possible deviations from SM predictions, independently of the fundamental theory of new physics.
Several searches for new physics in the top quark sector including new non-SM couplings of the top quark have been performed at the Tevatron and LHC colliders [1-4,4-10]. Most of the previous analyses followed the anomalous coupling approach in which the SM Lagrangian is extended for possible new interactions. Another powerful framework e-mail: cms-publication-committee-chair@cern.ch to parametrize deviations with respect to the SM prediction is the effective field theory (EFT) [11,12]. Constraints obtained on anomalous couplings can be translated to the effective coupling bounds [1,13]. Several groups have performed global fits of top quark EFT to unfolded experimental data from the Tevatron and LHC colliders [14,15]. Due to the limited access to data and details of the associated uncertainties, correlations between various cross section measurements and related uncertainties are neglected in a global fit on various unfolded measurements. On the other hand, EFT operators could affect backgrounds for some processes constructively or destructively while cross sections are measured with the SM assumptions for background processes. Inside the CMS Collaboration and with direct access to data, all mentioned points could be considered properly.
In this paper, the EFT approach is followed to search for new physics in the top quark sector in the dilepton final states. In Refs. [13,16], all dimension-six operators that contribute to top quark pair (tt) production and single top quark production in association with a W boson (tW) are investigated. The operators and the related effective Lagrangians, which are relevant for dilepton final states, can be written as [12]: O tG = (qσ μν λ a t)φG a μν , O G = f abc G aν μ G bρ ν G cμ ρ , O u(c)G = (qσ μν λ a t)φG a μν , where μ G c ν , σ μν = 1 2 [γ μ , γ ν ], P L,R = 1 2 (1 ∓ γ 5 ), and the symbols q, t and φ (φ = φ * ) in the operators represent the left-handed quark doublet, the right-handed top quark singlet, and the Higgs boson doublet fields, respectively. The parameters C (3) φq , C tW , C tG , C G and C u(c)G stand for the dimensionless Wilson coefficients, also called effective couplings. The variable Λ represents the energy scale beyond which new physics becomes relevant. A detailed description of the operators is given in Refs. [13,16,17]. In this analysis, fourfermion operators involved in tt production are not probed. Up to order Λ −2 , the tW and tt production cross sections and most of the differential observables considered in this analysis do not receive CP-odd contributions. Therefore, we only probe CP-even operators with real coefficients. The operators O (3) φq and O tW modify the SM interaction between the W boson, top quark, and b quark (Wtb). We consider the EFT effects in the production of top quarks not in their decays [18]. The operator O tG is called the chromomagnetic dipole moment operator of the top quark and can arise from various models of new physics [19,20]. The triple-gluon field strength operator O G represents the only genuinely gluonic CP conserving term that can appear at dimension six within an effective strong interaction Lagrangian. Although it is shown that jet production at the LHC can set a tight constraint on the C G [21], tt production is also considered as a promising channel [22,23]. The operators O uG and O cG lead to flavor-changing neutral current (FCNC) interactions of the top quark and contribute to tW production. The effect of introducing new couplings C (3) φq , C tW , C tG and C u(c)G can be investigated in tW production. The chromomagnetic dipole moment operator of the top quark also affects tt production. In the case of C G coupling, only tt production is modified. It should be noted that the O tW and O tG operators with imaginary coefficients lead to CP-violating effects. Representative Feynman diagrams for SM and new physics contributions in tW and tt production are shown in Fig. 1 [4,9] via single top quark production and limits are set on related anomalous couplings.
In this paper, a search for new physics in top quark production using an EFT framework is reported. This is the first such search for new physics that uses the tW process. Final states with two opposite-sign isolated leptons (electrons or muons) in association with jets identified as originating from the fragmentation of a bottom quark ("bjets") are analyzed. The search is sensitive to new physics contributions to tW and tt production, and the six effective couplings, C G , C (3) φq , C tW , C tG , C uG , and C cG , are constrained assuming one non-zero effective coupling at a time. The effective couplings affect both the rate of tt and tW production and the kinematic distributions of final state particles. For the C (3) φq , C tW , C tG , and C G effective couplings, the deviation from the SM prediction is dominated by the interference term between SM and new physics diagrams, which is linear with respect to the effective coupling. Therefore, the kinematic distributions of the final-state particles vary as a function of the Wilson coefficients. For small effective couplings the kinematic distributions approach those predicted by the SM. On the other hand, the new physics terms due to the C uG and C cG effective couplings do not interfere with the SM tW process, and the kinematic distributions of final-state particles are determined by the new physics terms independently of the SM prediction. In this analysis, we use the rates of tW and tt production to probe the C (3) φq , C tW , C tG , and C G effective couplings. Variations in both rate and kinematic distributions of final-state particles are employed to probe the C uG and C cG effective couplings. The analysis utilizes proton-proton (pp) collision data collected by the CMS experiment in 2016 at a centerof-mass energy of 13 TeV, corresponding to an integrated luminosity of 35.9 fb −1 .
The paper is structured as follows. In Sect. 2, a description of the CMS detector is given and the simulated samples used in the analysis are detailed. The event selection and the SM background estimation are presented in Sect. 3. Section 4 presents a description of the signal extraction procedure. An overview of the systematic uncertainty treatment is given in Sect. 5. Finally, the constraints on the effective couplings are presented in Sect. 6, and a summary is given in Sect. 7.

The CMS detector and event simulation
The central feature of the CMS apparatus is a superconducting solenoid of 6 m internal diameter, providing a magnetic field of 3.8 T. Within the solenoid volume are a silicon pixel and strip tracker, a lead tungstate crystal electromagnetic calorimeter (ECAL), and a brass and scintillator hadron calorimeter, each composed of a barrel and two endcap sections. Forward calorimeters extend the pseudorapidity (η) coverage provided by the barrel and endcap detectors. Muons are detected in gas-ionisation chambers embedded in the steel flux-return yoke outside the solenoid. A more detailed description of the CMS detector, together with a definition of the coordinate system used and the relevant kinematic variables, can be found in Ref. [24].
The Monte Carlo (MC) samples for the tt, tW and diboson (VV = WW, WZ, ZZ) SM processes are simulated using the Powheg-Box event generator (v1 for tW, v2 for tt and diboson) [25][26][27][28] at the next-to-leading order (NLO), interfaced with pythia (v8.205) [29] to simulate parton showering and to match soft radiations with the contributions from the matrix elements. The pythia tune CUETP8M1 [30] is used for all samples except for the tt sample, for which the tune CUETP8M2 [31] is used. The NNPDF3.0 [32] set of the parton distribution functions (PDFs) is used. The tt and tW samples are normalized to the next-to-next-to-leading order (NNLO) and approximate NNLO cross section calculations, respectively [33,34]. In order to better describe the transverse momentum ( p T ) distribution of the top quark in tt events, the top quark p T spectrum simulated with powheg is reweighted to match the differential top quark p T distribution at NNLO quantum chromoDynamics (QCD) accuracy and including EW corrections calculated in Ref. [35]. Other SM background contributions, from Drell-Yan (DY), tt+V, tt+γ, and W + γ processes, are simulated at NLO using the MadGraph5_amc@nlo (v2.2.2) event generator [36][37][38], interfaced with pythia v8 for parton showering and hadronization. The events include the effects of additional pp interactions in the same or nearby bunch crossings (pileup) and are weighted according to the observed pileup distribution in the analyzed data. The CMS detector response is simulated using Geant4 (v9.4) [39,40], followed by a detailed trigger simulation. Simulated events are reconstructed with the same algorithms as used for data.
In order to calculate the total cross sections for the tt and tW processes and generate events in the presence of new effective interactions, the operators of Eqs. 1-5 have been implemented in the universal FeynRules output (UFO) format [41] through the FeynRules package [42]. The output EFT model is used in the MadGraph5_amc@nlo (v2.2.2) event generator [36,37]. If we allow for the presence of one operator at a time, the total cross section up to O(Λ −4 ) can be parametrized as where the C i s are effective couplings introduced in Eqs. 1-5. Here, σ (1) i is the contribution to the cross section due to the interference term between the SM diagrams and diagrams with one EFT vertex. The cross section σ (2) i is the pure new physics contribution. We use the most precise available SM cross section prediction, which are σ tt SM = 832 +20 −29 (scales) ± 35 (PDF + α S ) pb and σ tW SM = 71.7 ± 1.8 (scales) ± 3.4 (PDF + α S ) pb for tt and tW production, respectively [33,34], where the α S is strong coupling constant. The first uncertainty reflects the uncertainties in the factorization and renormalization scales. In the framework of EFT, the σ (1) i and σ (2) i terms have been calculated at NLO accuracy for all of the operators, except O G [16,43,44]. At the time the work for this paper was concluded, there was no Contribution i for various effective couplings at LO and available K factors are given in Table 1.

Event selection and background estimation
The event selection for this analysis is similar to the one used in Ref.
[10]. The events of interest are recorded by the CMS detector using a combination of dilepton and single-lepton triggers. Single-lepton triggers require at least one isolated electron (muon) with p T > 27 (24) GeV. The dilepton triggers select events with at least two leptons with loose isolation requirements and p T for the leading and sub-leading leptons greater than 23 and 12 (17 and 8) GeV for the ee (μμ) final state. In the eμ final state, in the case of the leading lepton being an electron, the events are triggered if the electron-muon pair has a p T greater than 23 and 8 GeV for the electron and muon, respectively. In the case of the leading lepton being a muon, the trigger thresholds are 23 and 12 GeV for the muon and electron, respectively [45].
Offline, the particle-flow (PF) algorithm [46] aims to reconstruct and identify each individual particle with an optimized combination of information from the various elements of the CMS detector. Electron candidates are reconstructed using tracking and ECAL information [47]. Requirements on electron identification variables based on shower shape and track-cluster matching are further applied to the reconstructed electron candidates, together with isolation criteria [10,47]. Electron candidates are selected with p T > 20 GeV and |η| < 2.4. Electron candidates within the range 1.444 < |η| < 1.566, which corresponds to the transition region between the barrel and endcap regions of the ECAL, are not considered. Information from the tracker and the muon spectrometer are combined in a global fit to reconstruct muon candidates [48]. Muon candidates are further required to have a high-quality fit including a minimum number of hits in both systems, and to be isolated [10,48]. The muons used in this analysis are selected inside the fiducial region of the muon spectrometer, |η| < 2.4, with a minimum p T of 20 GeV.
The PF candidates are clustered into jets using the antik T algorithm with a distance parameter of 0.4 [49][50][51]. Jets are calibrated in data and simulation, accounting for energy deposits of particles from pileup [52]. Jets with p T > 30 GeV and |η| < 2.4 are selected; loose jets are defined as jets with the p T range between 20 and 30 GeV. Jets originating from the hadronization of b quarks are identified using the combined secondary vertex algorithm [53]; this algorithm combines information from track impact parameters and secondary vertices identified within a given jet. The chosen working point provides a signal identification efficiency of approximately 68% with a probability to misidentify lightflavor jets as bjets of approximately 1% [53]. The missing transverse momentum vector p miss T is defined as the projection on the plane perpendicular to the proton beams axis of the negative vector sum of the momenta of all reconstructed PF candidates in the event [54]. Corrections to the jet energies are propagated to p miss T . Its magnitude is referred to as p miss T .
Events are required to have at least two leptons (electrons or muons) with opposite sign and with an invariant mass above 20 GeV. The leading lepton must fulfill p T > 25 GeV. For the same-flavor lepton channels, to suppress the DY background, the dilepton invariant mass must not be within 15 GeV of the Z boson mass and a minimal value (of 60 GeV) on p miss T is applied. The events are divided into the ee, eμ, and μμ channels according to the flavors of the two leptons with the highest p T and are further categorized in different bins depending on the number of jets ("n-jets") and number b-tagged jets ("mtags") in the final state. The largest number of tW events is expected in the category with exactly one b-tagged jet (1jet,1-tag) followed by the category with two jets, of which one a bjet (2-jets,1-tag). Events in the categories with more than two jets and exactly two b-tagged jets are dominated by the tt process (≥2-jets,2-tags). Categories with zero bjets are dominated by DY events in the ee and μμ channels and are not used in the analysis. However, in the eμ channel, the contamination of DY events is lower and a significant number of tW events is present in the category with one jet and zero b-tagged jets (1-jet,0-tag). The latter category is included in this analysis. In Fig. 2, the data in the ten search regions are shown together with the SM background predictions.
The contributions of SM processes leading to two prompt leptons in the final state are estimated from simulated samples and are normalised to the integrated luminosity of the data. These contributions originate mainly from tt, tW and DY production. Other SM processes, such as diboson, tt+V and tt+γ have significantly smaller contributions.
To correct the DY simulation for the efficiency of the p miss T threshold and for the mismodeling of the heavy-flavour con-tent, scale factors are derived using the ratio of the numbers of simulated events inside and outside the dilepton invariant mass window, 76-106 GeV. The observed event yield inside the window is scaled to estimate the DY background outside the mass window [55]. The nonprompt lepton backgrounds which contain fake lepton(s) from a misreconstructed γ or jet(s) are also considered. The contribution of misidentified or converted γ events from the Wγ process is estimated from MC simulation. The contribution from W+jets and multijet processes is estimated by a data-based technique using events with same-sign leptons. The method is based on the assumption that the probability of assigning positive or negative charge to the fake lepton is equal. Therefore, the background contribution from Table 2 Number of expected events from tW, tt and DY production, from the remaining backgrounds (other), total background contribution and observed events in data after all selections for the ee, eμ, and μμ channels and for different (n-jets,m-tags) categories. The uncer-tainties correspond to the statistical contribution only for the individual background predictions and to the quadratic sum of the statistical and systematic contributions for the total background predictions fake leptons in the final selection (opposite-sign sample) can be estimated from the corresponding sample with same-sign leptons. In this latter same-sign event sample, the remaining small contribution from prompt-lepton backgrounds is subtracted from data using MC samples. After all selections, the expected numbers of events from tW, tt, DY, and remaining background contributions mentioned above, as well as the total number of background events are reported in Table 2 for the ee, eμ, and μμ channels and for the various (n-jets,m-tags) categories. We find generally very good agreement between data and predictions, within the uncertainties of the data.

Signal extraction using neural networks tools
The purpose of the analysis is to search for deviations from the SM predictions in the tW and tt production due to new physics, parametrized with the presence of new effective cou-plings. In order to investigate the effect of the non-zero effective couplings, it is important to find suitable variables with high discrimination power between the signal and the background. Depending on the couplings, the total yield or the distribution of the output of a neural network (NN) algorithm is employed, as summarized in Table 3. The NN algorithm used in this analysis is a multilayer perceptron [56].
All the effective couplings introduced in Sect. 1 can contribute to tW production except the triple gluon field strength operator, O G which only affects tt production. As observed in previous analysis [22] and confirmed here, the top quark p T distribution is sensitive to the triple-gluon field-strength operator. The kinematic distributions of final-state particles show less discrimination power than the top quark p T distribution. In addition, they vary with the value of C G and approach the SM prediction for decreasing values of C G . Therefore, we use the total yield in various categories to constrain the C G effective coupling.
The deviation from the SM tW production because of the interference between the SM and the O tG , O (3) φq , and O tW operators is of the order of 1/Λ 2 . It is assumed that the new physics scale Λ is larger than the scale we probe. Therefore, 1/Λ 4 contributions from the new physics terms are small compared to the contribution from the interference term.
φq is similar to the SM Wtb operator and leads to a rescaling of the SM Wtb vertex [13]. The O tW and O tG operators lead to the right-handed Wtb interaction and a tensor-like ttg interaction, respectively, which are absent in the SM at the first order. Their effects have been investigated via the various kinematic distributions of the final-state particles considered in this analysis and are found to be not distinguishable from the SM tW and tt processes for unconstrained values of the effective couplings within the current precision on data. After the selection described in Section 3, the dominant background comes from tt production, with a contribution of about 90%. In order to observe deviations from SM tW production in the presence of the O φq , O tW , and O tG effective operators, we need to separate tW events from the large number of tt events. Two independent NNs are trained to separate tt events (the background) and tW events (considered as the signal) in the (1-jet, 1-tag) (NN 11 ) and (2-jets, 1-tag) (NN 21 ) categories, which have significant signal contributions [57]. For the eμ channel, another NN is used for the (1-jet, 0-tag) (NN 10 ) category, in which the tt, WW, and DY events are combined and are considered as the background. A comparison between the observed data and the SM background prediction of the NN output shape in various (n-jets, m-tags) categories is shown for the ee and μμ channels in Fig. 3 and for the eμ channel in Fig. 4 (left  column).
The presence of the O uG and O cG operators changes the initial-state particle (see Fig. 1), and leads to different kinematic distributions for the final-state particles, compared to the SM tW process. For these FCNC operators, new physics effects on final-state particle distributions are expected to be distinguishable from SM processes. In order to search for new physics due to the O uG and O cG effective operators, an NN (NN FCNC ) is used to separate SM backgrounds (tt and tW events together) and new physics signals for events with exactly one b-tagged jet with no requirement on the number of light-flavor jets (n-jets, 1-tag). The comparison of the NN output for data, SM background and signal (tW events via FCNC interactions) is shown in Fig. 4 (right column) for the ee, eμ, and μμ channels.
The various input variables for training the NN introduced above are described below and are shown in Table 4.
-M (where = e or μ), invariant mass of dilepton system; p T , p T of dilepton system; -Δp T ( 1 , 2 ), p leading lepton T − p sub-leading lepton T ; p 1 T , p T of leading lepton; -Centrality( 1 , jet 1 ), scalar sum of p T of the leading lepton and leading jet, over total energy of selected leptons and jets; -Centrality( 1 , 2 ), scalar sum of p T of the leading and sub-leading leptons, over total energy of selected leptons and jets; -ΔΦ( , jet 1 ), ΔΦ between dilepton system and leading jet where Φ is azimuthal angle; p T ( , jet 1 ), p T of dilepton and leading jet system; p T ( 1 , jet 1 ), p T of leading lepton and leading jet system; -Centrality( , jet 1 ), scalar sum of p T of the dilepton system and leading jet, over total energy of selected leptons and jets; -ΔR( 1 , 2 ), jet 1 ), invariant mass of leading lepton and leading jet; -M(jet 1 , jet 2 ), invariant mass of leading jet and subleading jet; -ΔR( 1 , jet 2 ), T ; -M( 2 , jet 1 ), invariant mass of sub-leading lepton and leading jet.

Systematic uncertainties
The normalization and shape of the signal and the backgrounds are both affected by different sources of systematic uncertainty. For each source, an induced variation can be parametrized, and treated as a nuisance parameter in the fit that is described in the next section. A systematic uncertainty of 2.5% is assigned to the integrated luminosity and is used for signal and background rates [58]. The efficiency corrections for trigger and offline selection of leptons were estimated by comparing the efficiency measured in data and in MC simulation using Z → events, based on a "tag-and-probe" method as in Ref.
[59]. The scale factors obtained are varied by one standard deviation to take into account the corresponding uncertainties in the efficiency. The jet energy scale and resolution uncertainties depend on p T and η of the jet and are computed by shifting the energy of each jet and propagating the variation to p miss T coherently [60].
The uncertainty in the b tagging is estimated by varying the b tagging scale factors within one standard deviation [53]. Effects of the uncertainty in the distribution of the number of pileup interactions are evaluated by varying the effective inelastic pp cross section used to predict the num- The uncertainty in the DY contribution in categories with one or two b-tagged jets is considered to be 50 and 30% in the eμ and same-flavor dilepton channels, respectively [10,57]. For the DY normalization in the (1-jet, 0-tag) category, an uncertainty of 15% is assigned [62]. In addition, systematic uncertainties related to the PDF, and to the renormalization and factorization scale uncertainty are taken into account for DY process in the (1-jet, 0-tag) category. The uncertainty in the yield of nonprompt lepton backgrounds is considered to be 50% [57]. Contributions to the background from tt production in association with a boson, as well as diboson production, are estimated from simulation and a systematic uncertainty of 50% is conservatively assigned [63].
Various uncertainties originate from the theoretical predictions. The effect of the renormalization and factorization scale uncertainty from the tt and tW MC generators is estimated by varying the scales used during the generation of the simulation sample independently by a factor 0.5, 1 or 2. Unphysical cases, where one scale fluctuates up while the other fluctuates down, are not considered. The top quark p T reweighting procedure, discussed in Sect. 2, is applied on top of the nominal powheg prediction at NLO to account for the higher-order corrections.
The uncertainty in the PDFs for each simulated signal process is obtained using the replicas of the NNPDF 3.
they are found to be insignificant. At NLO QCD, tW production is expected to interfere with tt production [66]. Two schemes for defining the tW signal in a way that distinguishes it from the tt production have been compared in the analysis: the "diagram removal" (DR), in which all doubly resonant NLO tW diagrams are removed, and the "diagram subtraction" (DS), where a gauge-invariant subtractive term modifies the NLO tW cross section to locally cancel the contribution from tt production [66-68]. The DR method is used for the nominal tW sample and the difference with respect to the sample simulated using the DS method is taken as a systematic uncertainty. The model parameter h damp in tt powheg [25] that controls the matching of the matrix elements to the pythia parton showers is varied from a top quark mass default value of 172.5 GeV by factors of 0.5 and 2 for estimating the uncertainties from the matching between jets from matrix element calculations and parton shower emissions. The renormalization scale for QCD emissions in the initialand final-state radiation (ISR and FSR) is varied up and down by factors of 2 and √ 2, respectively, to account for parton shower QCD scale variation error in both tt and tW samples [69]. In addition, several dedicated tt samples are used to estimate shower modeling uncertainties in both underlying event and color re-connections [10, 31,69]. To estimate model uncertainties, tW and tt samples are generated with powheg as described in Sect. 2, varying the relevant model parameters with respect to the nominal samples.

Constraints on the effective couplings
The six Wilson coefficients sensitive to new physics contributions in top quark interactions, as defined in Eqs. 1-5, are tested separately in the observed data. The event yields and the NN output distributions in each analysis category, summarized in Table 3, are used to construct a binned likelihood function. All sources of systematic uncertainty, described in Sect. 5, are taken into account as nuisance parameters in the fit. A simultaneous binned maximum-likelihood fit is performed to find the best fit value for each Wilson coefficient together with 68 and 95% confidence intervals (CIs) [70]. In this section, distributions of the log-likelihood functions are shown with one nonzero effective coupling at a time for Λ = 1 TeV.
The SM cross section prediction for the tt and tW processes, σ (1) i and σ (2) i (see Table 1), are accompanied by uncertainties in scales and PDFs. These theoretical uncertainties can affect the bounds on the Wilson coefficients. In order to study this effect, the fit is performed on data, while theoretical uncertainties are varied within one standard deviation and are shown together with the nominal results in the likelihood scan plots in Fig. 5. The nominal theoretical cross sections for tt and tW processes are varied by [+4.8%, −5.5%] and [+5.4%, −5.4%] , respectively. These variations cover the uncertainties arising from the variations of factorization and renormalization scales and PDFs [10,34]. The scale variations for σ (1) i and σ (2) i are evaluated to be within 1-25%. We assumed that the scale uncertainty is 100% correlated among the terms σ SM , σ (1) i , and σ (2) i .

Exclusion limits on the C G effective coupling
In order to constrain the C G coupling, the effect on the tt rate in various (n-jets,m-tags) categories is considered. The impact of the difference between the kinematic distributions of tt events from the O G interaction and from the SM interactions on the acceptance is evaluated to be 3% for C G ∼ 1. This uncertainty is considered only for the C G coupling since the top p T spectrum is affected considerably by this operator, while other operators lead to a p T spectrum similar to the SM prediction for unconstrained values of the probed Wilson coefficients. The fit is performed simultaneously on the observed event yields in the categories presented in Fig. 2 in the (1-jet, 1-tag), (2-jets, 1-tag), and (≥2-jets, 2-tags) categories for the ee, eμ, and μμ channels. In addition, the (1-jet, 0-tag) category is included only for the eμ channel. The main limiting factor on the constraints in the C G coupling is the uncertainty in the signal acceptance found after maximizing the likelihood, followed by uncertainties in the integrated luminosity calibration and the trigger scale factor.
The results of the fit for the individual channels and for all channels combined are listed in the first row of Table 5.   Table 5 Summary of the observed and expected allowed intervals on the effective couplings obtained in the ee, eμ, and μμ channels, and all channels combined. All sources of systematic uncertainty, described in Sect. 5, are taken into account with the exception of the uncertainties on the SM cross section predictions for the tt and tW processes The results of the likelihood scans of the C G coupling are shown in Fig i terms are varied due to the scale and PDF uncertainties. As a secondorder parametrization, given by Eq. 6 is used to fit the data, the resulting likelihood function could have two minima, as can be seen in some of the plots in Fig. 5. 6.2 Exclusion limits on the C tG , C φq , and C tW effective couplings In order to set limits on the effective couplings C tG , C φq , and C tW , we utilize the NN output distributions for both data and MC expectation in the (1-jet, 1-tag) and (2-jets, 1-tag) regions and event yields in the (≥2-jets, 2-tags) region for the three dilepton channels. The inclusion of the (≥2-jets,  2-tags) and (2-jets, 1-tag) categories provides a constraint of the normalization and systematic uncertainties in the tt background. In addition, the (1-jet, 0-tag) category is included for the eμ channel to increase the signal sensitivity. The results of the likelihood scans of the C tG , C (3) φq , and C tW Wilson coefficients are shown in Fig. 5 for the combination of all channels. The inclusion of the C tG coupling to the tW process tightens the 2 standard deviations band by 7%. The results for the individual channels, and the combined results are listed in Table 5 (second, third, and fourth rows). The three main sources of uncertainty that affect the interval determination are uncertainties in the DY estimation, integrated luminosity, and lepton identification scale factors for C tG ; jet energy scale, tt and tW interference at NLO, and statistical uncertainty in MC samples for C (3) φq ; statistical uncertainty in data, jet energy scale, and the powheg matching method for C tW effective couplings.

Exclusion limits on the C uG and C cG effective couplings
Since the tW production via FCNC interactions does not interfere with the SM tW process (with the assumption of |V td | = |V ts | = 0), the FCNC signal sample is used to set upper bounds on the related Wilson coefficients. Events with exactly one b-tagged jet are included in the limit setting procedure with no requirement on the number of light-flavor jets (n-jets, 1-tag). The observed (median expected) 95% confidence level (CL) upper limits on the product of cross section times branching fractions σ (pp → tW)B(W → ν) 2 for the C uG and C cG FCNC signals for the combination of the ee, μμ, and eμ channels are found to be 0.11 (0.20) pb and 0.13 (0.26) pb, respectively. These results are used to calculate upper limits on the Wilson coefficients C uG , C cG , and on the branching fractions B(t → ug) and B(t → cg). The limits on the C uG and C cG couplings are summarized in the last two rows of φq , C tW , C tG , C G , C uG , and C cG , obtained from the combination of all channels are shown in Fig. 6. Table 6 summarizes the effect of the most important uncertainty sources on the observed allowed intervals.

Summary
A search for new physics in top quark interactions is performed using tt and tW events in dilepton final states. The analysis is based on data collected in pp collisions at 13 TeV by the CMS detector in 2016, corresponding to an integrated luminosity of 35.9 fb −1 . No significant excess above the standard model background expectation is observed. For the first time, both tt and tW production are used simultaneously in a model independent search for effective couplings. The six effective couplings, C G , C tG , C tW , C φq , C uG , and C cG are constrained using a dedicated multivariate analysis. The constraints presented, obtained by considering one operator at a time, are a useful first step toward more global approaches.