Search for third-generation scalar leptoquarks decaying to a top quark and a $\tau$ lepton at $\sqrt{s}=13\,\text {Te}\text {V}$

Abstract

A search for pair production of heavy scalar leptoquarks (LQs), each decaying into a top quark and a $\tau$ lepton, is presented. The search considers final states with an electron or a muon, one or two $\tau$ leptons that decayed to hadrons, and additional jets. The data were collected in 2016 in proton–proton collisions at $\sqrt{s}=13\,\text {Te}\text {V}$ with the CMS detector at the LHC, and correspond to an integrated luminosity of 35.9$\,\text {fb}^{-1}$. No evidence for pair production of LQs is found. Assuming a branching fraction of unity for the decay $\mathrm {LQ} \rightarrow \mathrm {t}\tau$, upper limits on the production cross section are set as a function of LQ mass, excluding masses below 900$\,\text {Ge}\text {V}$ at 95% confidence level. These results provide the most stringent limits to date on the production of scalar LQs that decay to a top quark and a $\tau$ lepton.

1 Introduction

Leptoquarks (LQs) are hypothetical particles that carry non-zero baryon and lepton quantum numbers. They are charged under all standard model (SM) gauge groups, and their possible quantum numbers can be restricted by the assumption that their interactions with SM fermions are renormalizable and gauge invariant [1]. The spin of an LQ state is either 0 (scalar LQ) or 1 (vector LQ). Leptoquarks appear in theories beyond the SM such as grand unified theories [2,3,4], technicolor models [5, 6] and other compositeness scenarios [7, 8], and R-parity-violating (RPV) supersymmetric models [9, 10].

Third-generation scalar LQs ($\mathrm {LQ}_3$ s) have recently received considerable theoretical interest, as their existence can explain the anomaly in the $\mathrm {\overline{B}}\rightarrow \mathrm {D}\tau \overline{\nu }$ and $\mathrm {\overline{B}}\rightarrow \mathrm {D}^* \tau \overline{\nu }$ decay rates reported by the BaBar [11, 12], Belle [13,14,15], and LHCb [16] Collaborations. These decay rates deviate from the SM predictions by about four standard deviations [17], and studies of the flavor structure of LQ couplings reveal that large couplings to third-generation quarks and leptons could explain this anomaly [18,19,20,21]. Third-generation LQs can appear in models in which only third-generation quarks and leptons are unified [22, 23] and therefore their existence is not constrained by proton decay experiments. All models that predict LQs with masses at the TeV scale and sizable couplings to top quarks and $\tau$ leptons can be probed by the CMS experiment at the CERN LHC.

In proton–proton ($\mathrm {p}\mathrm {p}$) collisions LQs are mainly pair produced through the quantum chromodynamic (QCD) quark-antiquark annihilation and gluon-gluon fusion s- and t-channel subprocesses as shown in Fig. 1. There are also lepton-mediated t- and u-channel contributions that depend on the unknown lepton-quark-LQ Yukawa coupling, but these contributions to $\mathrm {LQ}_3$ production are negligible at the LHC as they require third-generation quarks in the initial state. Hence, the LQ pair-production cross section can be taken to depend only on the assumed values of the LQ spin and mass, and on the center-of-mass energy. The corresponding pair production cross sections have been calculated up to next-to-leading order (NLO) in perturbative QCD [24].

This paper presents the first search for the production of an $\mathrm {LQ}_3$ decaying into a top quark and a $\tau$ lepton at $\sqrt{s} = 13\,\text {Te}\text {V}$. The search targets $\mathrm {LQ}_3$ s with electric charges $-5/3\,e$ and $-1/3\,e$, where e is the proton charge, and with various possible weak isospin configurations, depending on the model. A previous search for this channel at $\sqrt{s} = 8\,\text {Te}\text {V}$ by the CMS Collaboration resulted in a lower mass limit of 685$\,\text {Ge}\text {V}$ for an $\mathrm {LQ}_3$ with branching fraction $\mathcal {B}=1$ into a top quark and a $\tau$ lepton [25]. Other searches for an $\mathrm {LQ}_3$ have targeted the decays $\mathrm {LQ}_3 \rightarrow \mathrm {b}\nu$ and $\mathrm {LQ}_3 \rightarrow \mathrm {b}\tau$ [26,27,28,29,30,31,32,33,34,35,36,37,38,39]. The results of the search presented here are also interpreted in the context of RPV supersymmetric models, where the supersymmetric partner of the bottom quark (bottom squark) decays into a top quark and a $\tau$ lepton via the RPV coupling.

Fig. 1

Dominant leading order Feynman diagrams for the production of leptoquark pairs in proton–proton collisions

We consider events with at least one electron or muon and at least one $\tau$ lepton, where the $\tau$ lepton undergoes a one- or three-prong hadronic decay, $\tau _\mathrm {h} \rightarrow \text {hadron(s)}+\nu _\tau$. In $\mathrm {LQ}_3 \overline{\mathrm {LQ}}_3$ events, $\tau$ leptons arise directly from $\mathrm {LQ}_3$ decays, as well as from $\mathrm {W}$ bosons in the top quark decay chain. Electrons and muons are produced in leptonic decays of $\mathrm {W}$ bosons or $\tau$ leptons. Two search regions are used in this analysis: a di-$\tau$ region with the signature $\ell \tau _\mathrm {h} \tau _\mathrm {h}$+jets and small background levels from SM processes, which provides high sensitivity for $\mathrm {LQ}_3$ masses below 500$\,\text {Ge}\text {V}$, and a region with a single $\tau$ lepton in the final state, $\ell \tau _\mathrm {h}$+jets, which has higher sensitivity for $\mathrm {LQ}_3$ masses above 500$\,\text {Ge}\text {V}$ because of a larger signal efficiency. Here, $\ell$ denotes either an electron or a muon. The dominant backgrounds in this search come from ${\mathrm {t}\overline{\mathrm {t}}}$+jets and $\mathrm {W}+\text {jets}$ production, with jets misidentified as hadronically decaying $\tau$ leptons. These backgrounds are estimated through measurements in control regions and extrapolated to the signal region.

In this paper, Sect. 2 describes the CMS detector, while Sect. 3 discusses the data samples and the properties of simulated events utilized in the analysis. Section 4 outlines the techniques used for event reconstruction and Sect. 5 describes the selection criteria applied in each analysis channel. The method used for the background estimation is reported in Sect. 6, and systematic uncertainties are detailed in Sect. 7. Finally, Sect. 8 contains the results of the analysis, and Sect. 9 summarizes this work.

2 The CMS detector

The central feature of the CMS apparatus [40] is a superconducting solenoid of 6$\,\text {m}$ internal diameter, providing a magnetic field of 3.8$\,\text {T}$. Within the solenoid volume are a silicon pixel and strip tracker, a lead tungstate crystal electromagnetic calorimeter (ECAL), and a brass and scintillator hadron calorimeter (HCAL), each composed of a barrel and two endcap sections. Forward calorimeters extend the pseudorapidity ($\eta$) coverage provided by the barrel and endcap detectors. Electron momenta are estimated by combining the energy measurement in the ECAL with the momentum measurement in the tracker. Muons are measured in gas-ionization detectors embedded in the steel flux-return yoke outside the solenoid. A more detailed description of the CMS detector, together with a definition of the coordinate system used and the relevant kinematic variables, can be found in Ref. [40].

Events of interest are selected using a two-tiered trigger system [41], where the first level is composed of custom hardware processors and selects events at a rate of around 100$\,\text {kHz}$ within a time interval of less than 4$\,\upmu \text {s}$. The second level, known as the high-level trigger, uses a version of the full event reconstruction software optimized for fast processing, and reduces the event rate to around 1$\,\text {kHz}$ before data storage.

3 Data sample and simulated events

The search for $\mathrm {LQ}_3$ s presented here uses $\mathrm {p}\mathrm {p}$ collisions at $\sqrt{s}=13\,\text {Te}\text {V}$ recorded with the CMS detector in 2016. The data sample corresponds to an integrated luminosity of 35.9$\,\text {fb}^{-1}$  [42].

The leading order (LO) Monte Carlo (MC) program pythia  8.205 [43] is used to simulate the $\mathrm {LQ}_3$ pair production signal process. Both $\mathrm {LQ}_3$ s are required to decay into a top quark and a $\tau$ lepton, and polarization effects from the chiralities of the top quark and the $\tau$ lepton have been neglected. The signal samples are generated for $\mathrm {LQ}_3$ masses ranging from 200 to 2000$\,\text {Ge}\text {V}$.

The principal background processes, top quark pair production (${\mathrm {t}\overline{\mathrm {t}}}$) via the strong interaction and electroweak single top quark production in the t-channel and tW processes, are simulated with the NLO generator powheg (v1 is used for the single top $\mathrm {t}$ $\mathrm {W}$ processes and v2 for the single top t-channel and ${\mathrm {t}\overline{\mathrm {t}}}$ processes) [44,45,46,47,48,49]. The s-channel process of single top quark production is generated at NLO using the program MadGraph 5_amc@nlo  (v2.2.2) [50]. Other background processes involve $\mathrm {W}$ and $\mathrm {Z}$ boson production in association with jet radiation. These processes are generated with MadGraph 5_amc@nlo  (v2.2.2), with $\mathrm {W}$ boson production at NLO and $\mathrm {Z}$ boson production at LO level. The matrix element generation of $\mathrm {W}$ and $\mathrm {Z}$ boson production is matched to the parton shower emissions with the Frederix and Frixione [51] and MLM [52] algorithms, respectively. Background processes from QCD multijet production are simulated with pythia  8.205. For all generated events, pythia  8.205 is used for the description of the parton shower and hadronization. In the parton shower, the underlying event tune CUETP8M1 [53, 54] has been applied for all samples except for ${\mathrm {t}\overline{\mathrm {t}}}$ and single top quark production in the t-channel, which use the underlying event tune CUETP8M2T4 [53, 54]. The event generation is performed using the NNPDF 3.0 parton distribution functions (PDFs) [55], for all events. The detector response is modeled with the Geant4  [56] suite of programs.

4 Event reconstruction

Event reconstruction is based on the CMS particle-flow (PF) algorithm [57], which combines information from all subdetectors, including measurements from the tracking system, energy deposits in the ECAL and HCAL, and tracks reconstructed in the muon detectors. Based on this information, all particles in the event are reconstructed as electrons, muons, photons, charged hadrons, or neutral hadrons.

Interaction vertices are reconstructed using a deterministic annealing filtering algorithm [58, 59]. The reconstructed vertex with the largest value of summed physics-objects $p_{\mathrm {T}} ^2$ is taken to be the primary $\mathrm {p}\mathrm {p}$ interaction vertex. The physics objects are jets, clustered using the jet finding algorithm [60, 61] with the tracks assigned to the vertex as inputs, and the associated missing transverse momentum, taken as the negative vector sum of the $p_{\mathrm {T}}$ of those jets. Charged particles associated with other interaction vertices are removed from further consideration.

Muons are reconstructed using the information collected in the muon detectors and the inner tracking detectors, and are measured in the range $|\eta |< 2.4$. Tracks associated with muon candidates must be consistent with muons originating from the primary vertex, and are required to satisfy a set of identification requirements. Matching muon detector information to tracks measured in the silicon tracker results in a $p_{\mathrm {T}}$ resolution for muons with $20<p_{\mathrm {T}} < 100\,\text {Ge}\text {V}$ of 1.3–2.0% in the barrel and better than 6% in the endcaps. The $p_{\mathrm {T}}$ resolution in the barrel is better than 10% for muons with $p_{\mathrm {T}}$ up to 1$\,\text {Te}\text {V}$  [62].

Electron candidates are reconstructed in the range $|\eta |<2.5$ by combining tracking information with energy deposits in the ECAL. Candidates are identified [63] using information on the spatial distribution of the shower, the track quality and the spatial match between the track and electromagnetic cluster, the fraction of total cluster energy in the HCAL, and the level of activity in the surrounding tracker and calorimeter regions. The transverse momentum $p_{\mathrm {T}}$ resolution for electrons with $p_{\mathrm {T}} \approx 45\,\text {Ge}\text {V}$ from $\mathrm {Z} \rightarrow \mathrm {e}\mathrm {e}$ decays ranges from 1.7% for nonshowering electrons in the barrel region to 4.5% for electrons showering in the endcaps [63].

Jets are clustered using PF candidates as inputs to the anti-$k_{\mathrm {T}}$ algorithm [60] in the FastJet  3.0 software package [61], using a distance parameter of 0.4. For all jets, corrections based on the jet area [64] are applied to the energy of the jets to remove the energy contributions from neutral hadrons from additional pp interactions in the same or adjacent bunch crossings (pileup collisions). Subsequent corrections are used to account for the nonlinear calorimetric response in both jet energy and mass, as a function of $\eta$ and $p_{\mathrm {T}}$ [65]. The jet energy resolution amounts typically to 15% at 10$\,\text {Ge}\text {V}$, 8% at 100$\,\text {Ge}\text {V}$, and 4% at 1$\,\text {Te}\text {V}$  [66]. Corrections to the jet energy scale and the jet energy resolution are propagated to the determination of the missing transverse momentum [66]. Jets associated with $\mathrm {b}$ quarks are identified using the combined secondary vertex v2 algorithm [67, 68]. The working point used for jet $\mathrm {b}$ tagging in this analysis has an efficiency of $\approx$65% (in ${\mathrm {t}\overline{\mathrm {t}}}$ simulated events) and a mistag rate (the rate at which light-flavor jets are incorrectly tagged) of approximately 1% [68].

Hadronically decaying $\tau$ leptons are reconstructed with the hadron-plus-strips (HPS) algorithm [69] and are denoted by $\tau _\mathrm {h}$. The HPS algorithm is based on PF jets and additionally includes photons originating from neutral pion decays. Energy depositions in the ECAL are reconstructed in “strips” elongated in the direction of the azimuthal angle $\phi$, to take account of interactions in the material of the detector and the axial magnetic field. These deposits are associated with one or three charged tracks to reconstruct various hadronic decay modes of $\tau$ leptons. To suppress backgrounds from light-quark or gluon jets, a $\tau _\mathrm {h}$ candidate is required to be isolated from other energy deposits in the event. The isolation criterion is based on the scalar $p_{\mathrm {T}}$ sum $I_\tau$ of charged and neutral PF candidates within a cone of radius $\smash [b]{\sqrt{(\varDelta \eta )^2 + (\varDelta \phi )^2} = 0.5}$ around the $\tau _\mathrm {h}$ direction, excluding the $\tau _\mathrm {h}$ candidate. The isolation criterion is $I_\tau < 1.5\,\text {Ge}\text {V}$ [70].

The energies and resolutions as well as the selection efficiencies for all reconstructed jets and leptons are studied in data and simulated events [62, 63, 66, 68, 70]. Based on these studies, the simulation is corrected to match the data.

5 Event selection and categorization

In the online trigger system, events with an isolated muon (or electron) with $p_{\mathrm {T}} >24\,(27)\,\text {Ge}\text {V}$ and $|\eta |<2.4\,(2.1)$ are selected in the muon (electron) channel. We select events offline containing exactly one isolated muon (or electron) with $p_{\mathrm {T}} >30\,\text {Ge}\text {V}$ and $|\eta |<2.4\,(2.1)$. For the electron channel, a veto is applied to events with a muon to avoid overlap between the two channels. At least one $\tau _\mathrm {h}$ lepton with $p_{\mathrm {T}} >20\,\text {Ge}\text {V}$ and $|\eta |<2.1$ and at least two jets with $p_{\mathrm {T}} >50\,\text {Ge}\text {V}$ and $|\eta | < 2.4$ are required. Events are selected if a third jet with $p_{\mathrm {T}} >30\,\text {Ge}\text {V}$ and $|\eta | < 2.4$ is present, and any additional jets are only considered if they have ${p_{\mathrm {T}} >30\,\text {Ge}\text {V}}$. The magnitude of the missing transverse momentum, $p_{\mathrm {T}} ^\text {miss}$, is required to be above 50$\,\text {Ge}\text {V}$. Further, the events are divided into two categories corresponding to the number of observed LQ candidates, allowing the sensitivity to be enhanced over a broad range of LQ masses. The event selection was chosen to maximize the expected significance of a possible LQ signal. A summary of the selection criteria for both categories is given in Table 1 and described below.

Table 1 Summary of selection criteria in event categories A ($\ell \tau _\mathrm {h}$ + jets) and B ($\ell \tau _\mathrm {h} \tau _\mathrm {h}$ + jets), where $\ell = \mu , \mathrm {e}$. In category A, the two subcategories, OS and SS, are defined by the charge of the $\ell \tau _h$ pair. The fit variable used in each category is also shown

5.1 Category A: $\ell \tau _\mathrm {h}$ + jets

In this category, exactly one $\tau _\mathrm {h}$ lepton is required in addition to the presence of one electron or muon. High $p_{\mathrm {T}}$ requirements are applied to maximize the sensitivity at high LQ masses. The leading jet is required to have ${p_{\mathrm {T}} >150\,\text {Ge}\text {V}}$. In addition we define two subcategories based on the electric charges of the particles in the $\ell \tau _\mathrm {h}$ pair: opposite-sign (OS) and same-sign (SS). Events passing the OS $\ell \tau _\mathrm {h}$ pair requirement must contain at least four jets and have $p_{\mathrm {T}} ^\text {miss} >100\,\text {Ge}\text {V}$. For both subcategories, we require that the leading tau lepton has $p_{\mathrm {T}} >100\,\text {Ge}\text {V}$ and that there is at least one $\mathrm {b}$-tagged jet. Finally the events are divided into two regions of $S_{\mathrm {T}}$, where $S_{\mathrm {T}}$ is the scalar $p_{\mathrm {T}}$ sum of all selected jets, leptons, and $p_{\mathrm {T}} ^\text {miss}$. In the low (high)-$S_{\mathrm {T}}$ search regions, events must satisfy $S_{\mathrm {T}} <1200\,(\ge 1200)\,\text {Ge}\text {V}$. This division adds sensitivity for $\mathrm {LQ}_3$ masses of 600$\,\text {Ge}\text {V}$ and higher.

The top quarks originating from the decay of a heavy $\mathrm {LQ}_3$ are expected to be produced with larger $p_{\mathrm {T}}$ than the top quarks produced in background processes. Therefore, the transverse momentum distribution of the top quark candidate decaying into hadronic jets ($p_{\mathrm {T}}^{\mathrm {t}}$) gives discrimination power between background and signal events, and a measurement of the $p_{\mathrm {T}}^{\mathrm {t}}$ spectrum is performed in category A.

A kinematic reconstruction of the top quark candidate is performed by building top quark hypotheses using between one and five jets. Because of the presence of multiple hypotheses in each event, we choose the hypothesis in which the reconstructed top quark mass is closest to the value of 172.5$\,\text {Ge}\text {V}$.

The statistical evaluation in this category is performed through a template-based fit to the measured $p_{\mathrm {T}}^{\mathrm {t}}$ distribution.

5.2 Category B: $\ell \tau _\mathrm {h} \tau _\mathrm {h}$ + jets

In this category events are required to have at least two $\tau _\mathrm {h}$ leptons and one electron or muon. This requirement of two $\tau _\mathrm {h}$ leptons removes a large fraction of the SM background processes. The exception to this exclusion of SM backgrounds are diboson production events that contain one or more $\tau _\mathrm {h}$ leptons, but the cross sections for these processes are small. The selection criteria in this category are adapted to provide good sensitivity for low LQ masses.

Each event is required to contain an OS $\tau _\mathrm {h} \tau _\mathrm {h}$ pair. If the event contains more than one $\tau _\mathrm {h} \tau _\mathrm {h}$ pair, the OS pair with the largest scalar $p_{\mathrm {T}}$ sum is selected. Moreover, the leading and subleading $\tau _\mathrm {h}$ must satisfy $p_{\mathrm {T}} >65$ and $35\,\text {Ge}\text {V}$, respectively.

In this category a counting experiment is performed, as the number of expected background events is too small for results to benefit from a shape-based analysis.

6 Background estimation

The background in this analysis consists of samples of events that are selected because of jets misidentified as $\tau _\mathrm {h}$ leptons and events with one electron or muon together with one or more $\tau _\mathrm {h}$ leptons.

In the following, events from ${\mathrm {t}\overline{\mathrm {t}}}$ and $\mathrm {W}+\text {jets}$ production that contain at least one misidentified $\tau _\mathrm {h}$ lepton are obtained from control regions (CRs) separately defined for the two search regions (SRs) A and B. We consider the following contributions: the ${\mathrm {t}\overline{\mathrm {t}}}$ background that consists of only misidentified $\tau _\mathrm {h}$ leptons (or exactly one misidentified $\tau _\mathrm {h}$ lepton as in category A), denoted by ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {f}}$, the ${\mathrm {t}\overline{\mathrm {t}}}$ background that consists of (at least) one $\tau _\mathrm {h}$ lepton and (at least) one misidentified $\tau _\mathrm {h}$ lepton (only used in category B), denoted by ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {p+f}}$, and the ${\mathrm {t}\overline{\mathrm {t}}}$ background that consists of one $\tau _\mathrm {h}$ lepton, denoted by ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {p}}$.

An extrapolation method is used to derive the background due to misidentified $\tau _\mathrm {h}$ leptons. The normalization, and in category A also the shape, of the ${\mathrm {t}\overline{\mathrm {t}}}$ background is estimated using

$$\begin{aligned} N^{{\mathrm {t}\overline{\mathrm {t}}}, \, \text {data}}_{\text {SR}} = \left( N^{\text {data}}_{\text {CR}}-N^{\text {other, MC}}_{\text {CR}} \right) \, \frac{N^{{\mathrm {t}\overline{\mathrm {t}}}, \,\text {MC}}_{\text {SR}}}{N^{{\mathrm {t}\overline{\mathrm {t}}}, \, \text {MC}}_{\text {CR}}}, \end{aligned}$$

(1)

where N is the total number of events for the respective process in the signal region or control region and where “other” denotes all non-${\mathrm {t}\overline{\mathrm {t}}}$ background processes that are estimated from simulation. The contribution to the background from events with $\tau _\mathrm {h}$ leptons only is estimated from simulated events.

6.1 Backgrounds in category A

In each subcategory of category A, the largest fraction of background events originates from ${\mathrm {t}\overline{\mathrm {t}}}$ production. The second largest source of background events arises from $\mathrm {W}+\text {jets}$ production, while minor contributions come from single top quark and $\mathrm {Z} +\text {jets}$ production.

The ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {f}}$ background and the $\mathrm {W}+\text {jets}$ background that contain a misidentified $\tau _\mathrm {h}$ lepton are derived from a single control region ($\mathrm {CR}_{\mathrm {A}}$), which is defined through the same selection requirements as for the SR, but with an inverted isolation requirement for the $\tau _\mathrm {h}$ lepton.

The shape of the $p_{\mathrm {T}}^{\mathrm {t}}$ distribution is compared between the $\mathrm {CR}_{\mathrm {A}}$ and SR in simulated ${\mathrm {t}\overline{\mathrm {t}}}$ and $\mathrm {W}+\text {jets}$ events. Since the inversion of the $\tau _\mathrm {h}$ isolation criterion introduces kinematic differences between the SRs and CRs, the jet multiplicity and $p_{\mathrm {T}}^{\mathrm {t}}$ are corrected in order to reproduce the shape of the ${\mathrm {t}\overline{\mathrm {t}}}$ and $\mathrm {W}+\text {jets}$ backgrounds in the SRs [71], as shown in Fig. 2.

Fig. 2

Shape comparison between the category A signal region and the corresponding control region, as a function of $p_{\mathrm {T}}^{\mathrm {t}}$, for simulated ${\mathrm {t}\overline{\mathrm {t}}}$ and $\mathrm {W}+\text {jets}$ events. Events with an opposite-sign $\mu \tau _\mathrm {h}$ pair are shown in the upper panel, while those with a same-sign $\mu \tau _\mathrm {h}$ pair are shown in the lower panel. The full selection is applied and the $S_{\mathrm {T}}$ categories are combined. All histograms are normalized to the total number of entries. Uncertainties of the signal region and control region are indicated by red error bars and gray hatched areas, respectively. The gray band in the ratio plot corresponds to the statistical uncertainty in the simulated samples

Once the kinematic distributions in the $\mathrm {CR}_{\mathrm {A}}$ are corrected, we use Eq. (1) to extrapolate the ${\mathrm {t}\overline{\mathrm {t}}}$ and $\mathrm {W}+\text {jets}$ background yields to the SR. In this equation, we replace $N^{{\mathrm {t}\overline{\mathrm {t}}}}$ with $N^{{\mathrm {t}\overline{\mathrm {t}}},\,\mathrm {W}+\text {jets}}$ for category A.

6.2 Backgrounds in category B

In category B, the dominant background also originates from ${\mathrm {t}\overline{\mathrm {t}}}$ production. As the fraction of misidentified electrons and muons was found to be negligible in this analysis, at least one of the two $\tau _\mathrm {h}$ leptons is mimicked by a jet. Thus, background events from ${\mathrm {t}\overline{\mathrm {t}}}$ production consist either of only misidentified $\tau _\mathrm {h}$ leptons or one $\tau _\mathrm {h}$ lepton and one misidentified $\tau _\mathrm {h}$ lepton, plus an electron or a muon. A separate CR is defined for each component. The strategy for determining this background in category B is shown in Fig. 3.

The first control region ($\mathrm {CR}_{\mathrm {B1}}$) is defined by inverting the isolation criterion for all $\tau _\mathrm {h}$ leptons with respect to the isolation criterion applied in the SR. The region $\mathrm {CR}_{\mathrm {B1}}$ is used to extrapolate the ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {f}}$ background to the SR. In contrast to the SR, the charge criterion on the $\tau _\mathrm {h}$ lepton is removed and the leading $\tau _\mathrm {h}$ lepton must have $p_{\mathrm {T}} <100\,\text {Ge}\text {V}$ to avoid overlap between the control region $\mathrm {CR}_{\mathrm {B1}}$ and control region $\mathrm {CR}_{\mathrm {A}}$. The ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {f}}$ background normalization is then derived as in Eq. (1).

A second control region ($\mathrm {CR}_{\mathrm {B2}}$) to estimate the ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {p+f}}$ background is defined, in which at least one isolated and at least one nonisolated $\tau _\mathrm {h}$ lepton are required. In contrast to the SR, the charge criterion on the $\tau _\mathrm {h}$ lepton is removed and the leading $\tau _\mathrm {h}$ lepton must have $p_{\mathrm {T}} <45\,\text {Ge}\text {V}$. The event must have an opposite-sign $\ell \tau _\mathrm {h}$ pair. For this requirement, the pair with the largest summed $p_{\mathrm {T}}$ is chosen. In addition, the events must satisfy $M_{\mathrm {T}}(\ell ,p_{\mathrm {T}} ^\text {miss})>100\,\text {Ge}\text {V}$, where $M_{\mathrm {T}}(\ell ,p_{\mathrm {T}} ^\text {miss})$ is the transverse mass of the lepton-$\vec {p}_{\mathrm {T}}^{\,\text {miss}}$ system and defined as

$$\begin{aligned} \smash [b]{M_{\mathrm {T}}(\ell ,p_{\mathrm {T}} ^\text {miss})=\sqrt{2p_{\mathrm {T}} ^\ell p_{\mathrm {T}} ^\text {miss} \left( 1-\cos [\varDelta \phi (\vec {p}_{\mathrm {T}}^{\,\ell },\vec {p}_{\mathrm {T}}^{\,\text {miss}})]\right) }}. \end{aligned}$$

The largest non-${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {p+f}}$ fraction in control region $\mathrm {CR}_{\mathrm {B2}}$ arises from the ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {f}}$ events. The estimate of this background is derived from the control region $\mathrm {CR}_{\mathrm {B1}}$ and extrapolated to the control region $\mathrm {CR}_{\mathrm {B2}}$ by using the extrapolation method as in Eq. (1). Once the ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {f}}$ background is estimated from $\mathrm {CR}_{\mathrm {B1}}$, it is subtracted from $\mathrm {CR}_{\mathrm {B2}}$. The ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {p+f}}$ background is extrapolated to the SR by using the extrapolation method as in Eq. (1).

Fig. 3

Strategy for the background estimation in category B. The ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {f}}$ background in the signal region is derived from the control region $\mathrm {CR}_{\mathrm {B1}}$. The ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {p+f}}$ background in the signal region is derived from the control region $\mathrm {CR}_{\mathrm {B2}}$. To obtain an estimate of the ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {f}}$ background in the control region $\mathrm {CR}_{\mathrm {B2}}$, the control region $\mathrm {CR}_{\mathrm {B1}}$ is used

7 Systematic uncertainties

Systematic uncertainties can affect both the overall normalization of background components, and the shapes of the $p_{\mathrm {T}}^{\mathrm {t}}$ distributions for signal and background processes. Uncertainties in the MC simulation are applied to all simulated events used in the signal and in the various control regions. For each systematic uncertainty, the background estimation procedure described in Sect. 6 is repeated to study the impact of the respective systematic variation on the final result of the analysis. In the following, the systematic uncertainties applied to the analysis are summarized.

• The uncertainty in the integrated luminosity measurement recorded with the CMS detector in the 2016 run at $\sqrt{s}=13\,\text {Te}\text {V}$ is 2.5% [42].

• The following uncertainties in the normalization of the background processes are included:

  • 5.6% in the ${\mathrm {t}\overline{\mathrm {t}}}$ production cross sect. [72] for ${\mathrm {t}\overline{\mathrm {t}}}$ events that include $\tau$ leptons,

  • 10% for single top quark [73,74,75], W+jets, and Z+jets production [76],

  • 20% for diboson production [77,78,79].

• The estimation of pileup effects is based on the total inelastic cross section. This cross section is determined to be 69.2$\,\text {mb}$. The uncertainty is taken into account by varying the total inelastic cross section by 5% [80].

• Simulated events are corrected for lepton identification, trigger, and isolation efficiencies. The corresponding scale factors are applied as functions of $|\eta |$ and $p_{\mathrm {T}}$. The systematic uncertainties due to these corrections are taken into account by varying each scale factor within its uncertainty.

• The scale factors for the jet energy scale and the jet energy resolution are determined as functions of $|\eta |$ and $p_{\mathrm {T}}$ [66]. The effect of the uncertainties in these scale factors are considered by varying the scale factors within their uncertainties. These variations are propagated to the measurement of the $p_{\mathrm {T}} ^\text {miss}$.

• Scale factors for the $\mathrm {b}$ tagging efficiencies are applied. These scale factors are measured as a function of the jet $p_{\mathrm {T}}$ [68]. The corresponding uncertainty is taken into account by varying the scale factors within their uncertainties.

Table 2 Summary of largest systematic uncertainties for the ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {f}}$ (and $\mathrm {W}+\text {jets}$) and ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {p+f}}$ backgrounds derived from data, for the ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {p}}$ background obtained from simulation and for a leptoquark signal with a mass of 700$\,\text {Ge}\text {V}$. Shown are the ranges of uncertainties, which are dependent on the search regions and the lepton channel type

• Various uncertainties in the $\tau$ lepton reconstruction are considered. An uncertainty of 5% in the $\tau$ lepton identification is applied, with an additional uncertainty of $0.2\,p_{\mathrm {T}}/(1\,\text {Te}\text {V})$. An uncertainty of 3% in the $\tau$ lepton energy scale is taken into account, and an uncertainty in the charge misidentification rate of 2% is applied [70].

• Parton distribution functions from the NNPDF 3.0 set are used to generate simulated events for both background and signal samples. The uncertainties in the PDFs are determined according to the procedure described in Ref. [81]. The associated PDF uncertainties in the signal acceptance are estimated following the prescription for the LHC [81].

• We consider uncertainties in the renormalization ($\mu _{\mathrm {R}}$) and factorization ($\mu _{\mathrm {F}}$) scales by varying the respective scales, both simultaneously and independently, by factors between 0.5 and 2.

• We apply an uncertainty in the background estimation method by varying the extrapolation factors for background processes without $\tau$ leptons within their uncertainties. An additional uncertainty due to the correction factors used to reweight events in control region $\mathrm {CR}_{\mathrm {A}}$ is applied.

Fig. 4

Distributions of $p_{\mathrm {T}}^{\mathrm {t}}$ for events in the electron channel passing the full selection in category A. The events are separated into OS (upper), SS (lower), low $S_{\mathrm {T}}$ (left) and high $S_{\mathrm {T}}$ (right) categories. The hatched areas represent the total uncertainties of the SM background. In the bottom panel, the ratio of data to SM background is shown together with statistical (dark gray) and total (light gray) uncertainties of the total SM background

The systematic uncertainties with the largest effects on the most important background processes and on the signal are summarized in Table 2. The most important background processes are the ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {f}}$, ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {f}}$ and $\mathrm {W}+\text {jets}$, and ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {p+f}}$ backgrounds derived from data, and the ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {p}}$ background taken from simulation. Also shown is the systematic uncertainty associated with the signal produced by an $\mathrm {LQ}_3$ whose mass is 700$\,\text {Ge}\text {V}$. The impact of the different sources of uncertainty varies for different processes. The uncertainty due to the variation in the scales $\mu _{\mathrm {R}}$ and $\mu _{\mathrm {F}}$ has a large impact on the ${\mathrm {t}\overline{\mathrm {t}}} _{\mathrm {p}}$ background, and is derived from simulation. The uncertainty in the $\tau$ lepton identification has the largest effect on the signal sample. For the backgrounds derived from several CRs, the uncertainty in the extrapolation factor has the largest impact.

8 Results

The results of all search categories in the electron and muon channels are combined in a binned-likelihood fit. A statistical template-based analysis, using the measured $p_{\mathrm {T}}^{\mathrm {t}}$ distributions in category A and a counting experiment with the events measured in category B, is performed by using the Theta software package [82]. Each systematic uncertainty discussed in Sect. 7 is accounted for by a nuisance parameter in the likelihood formation.

The post-fit $p_{\mathrm {T}}^{\mathrm {t}}$ distributions in the electron and muon channels in category A are shown in Figs. 4 and  5, respectively. Contributions from ${\mathrm {t}\overline{\mathrm {t}}}$ and $\mathrm {W}+\text {jets}$ production with a misidentified $\tau _\mathrm {h}$ lepton are derived from control region $\mathrm {CR}_{\mathrm {A}}$, whereas SM backgrounds with a $\tau _\mathrm {h}$ lepton and other small backgrounds are taken from simulation.

Fig. 5

Distributions of $p_{\mathrm {T}}^{\mathrm {t}}$ for events in the muon channel passing the full selection in category A. The events are separated into OS (upper), SS (lower), low $S_{\mathrm {T}}$ (left) and high $S_{\mathrm {T}}$ (right) categories. The hatched areas represent the total uncertainties of the SM background. In the bottom panel, the ratio of data to SM background is shown together with statistical (dark gray) and total (light gray) uncertainties of the total SM background

Table 3 Final event yield in category B in the muon and electron channels for different leptoquark mass hypotheses, the background processes, and data. The total uncertainties for the signal and the background processes are shown

In Table 3, the total number of events from background processes and signal processes in category B is summarized. No significant deviation from the SM prediction is observed in the data in either category A or category B.

A Bayesian statistical method [82, 83] is used to derive 95% confidence level ($\text {CL}$) upper limits on the product of the cross section and the branching fraction squared for $\mathrm {LQ}_3$ pair production. Pseudo-experiments are performed to extract expected limits under a background-only hypothesis. For the signal cross section parameter, we use a uniform prior distribution. For the nuisance parameters, log-normal prior distributions are used. These are randomly varied within their ranges of validity to estimate the 68 and 95% $\text {CL}$ expected limits. Correlations between the systematic uncertainties across all channels are taken into account. The statistical uncertainties of simulated samples are treated as an additional Poisson nuisance parameter in each bin of the $p_{\mathrm {T}}^{\mathrm {t}}$ distribution.

The 95% $\text {CL}$ upper limits on the product of the cross section and the branching fraction squared $\mathcal {B}^2$ as a function of $\mathrm {LQ}_3$ mass and the 95% $\text {CL}$ upper limits on the $\mathrm {LQ}_3$ mass as a function of $\mathcal {B}$ are shown in Fig. 6 (top). The cross section for pair production of scalar LQs at NLO accuracy [24] is shown as the dashed line. The dotted lines indicate the uncertainty due to the PDFs and to variations of the renormalization and factorization scales by factors of 0.5 and 2.

Production cross sections of 0.6$\,\text {pb}$ for $\mathrm {LQ}_3$ masses of 300$\,\text {Ge}\text {V}$ and of about 0.01$\,\text {pb}$ for masses up to 1.5$\,\text {Te}\text {V}$ are excluded at 95% $\text {CL}$ under the assumption of $\mathcal {B}=1$ for $\mathrm {LQ}_3$ decays to a top quark and $\tau$ lepton. Comparing these limits with the NLO cross sections, $\mathrm {LQ}_3$ masses up to 900$\,\text {Ge}\text {V}$ (930$\,\text {Ge}\text {V}$ expected) can be excluded.

Exclusion limits with varying branching fractions $\mathcal {B}$ are presented in Fig. 6 (bottom), where limits on the complementary $\mathrm {LQ}_3 \rightarrow \mathrm {b} \nu$ ($\mathcal {B}=0$) decay channel are also included. The results for $\mathcal {B}=0$ are obtained from a search for pair-produced bottom squarks [38] with subsequent decays into $\mathrm {b}$ quark and neutralino pairs, in the limit of vanishing neutralino masses. Scalar $\mathrm {LQ}_3$ s can be excluded for masses below 1150$\,\text {Ge}\text {V}$ for $\mathcal {B}=0$ and for masses below 700$\,\text {Ge}\text {V}$ over the full $\mathcal {B}$ range. For the assumptions of a LQ with symmetric couplings under the SM gauge symmetry and with decays to only $\mathrm {b} \nu$ and $\mathrm {t}\tau$, $\mathcal {B}$ can only take values of 1 or 0.5. When these assumptions are lifted, $\mathcal {B}$ can take all possible values between 0 and 1. Note that if upper limits on $\mathcal {B}$ are to be used to constrain the lepton-quark-$\mathrm {LQ}_3$ Yukawa couplings, $\lambda _{\mathrm {b} \nu }$ and $\lambda _{\mathrm {t} \tau }$, kinematic suppression factors that favor $\mathrm {b} \nu$ decay over the $\mathrm {t} \tau$ decay have to be considered as well [26, 27].

The results presented here can be directly reinterpreted in the context of pair produced down-type squarks decaying into top quark and $\tau$ lepton pairs. Such squarks appear in RPV SUSY scenarios and correspond to LQs with $\mathcal {B} = 0.5$. These squarks are excluded up to a mass of 810$\,\text {Ge}\text {V}$, and the decay mode is dominated by the RPV coupling $\lambda ^{\prime }_{333}$ [84].

Fig. 6

Upper limits at 95% confidence level on the product of the cross section and the branching fraction squared (upper), and on the leptoquark mass as a function of the branching fraction (lower), for the pair production of scalar LQs decaying to a top quark and a $\tau$ lepton. In the top plot, the theoretical curve corresponds to the NLO cross section with uncertainties from PDF and scale variations [24], shown by the dotted lines. The bottom plot additionally includes results from a search for pair-produced bottom squarks [38]

9 Summary

A search has been conducted for pair production of third-generation scalar leptoquarks ($\mathrm {LQ}_3$ s) decaying into a top quark and a $\tau$ lepton. Proton–proton collision data recorded in 2016 at a center-of-mass energy of 13$\,\text {Te}\text {V}$, corresponding to an integrated luminosity of 35.9$\,\text {fb}^{-1}$, has been analyzed. The search has been carried out in the $\ell \tau _\mathrm {h}$+jets and $\ell \tau _\mathrm {h} \tau _\mathrm {h}$+jets channels, where $\ell$ is either an electron or muon and $\tau _\mathrm {h}$ indicates a tau lepton decaying to hadrons. Standard model backgrounds due to misidentified $\tau _\mathrm {h}$ leptons are derived from control regions. The measured transverse momentum distributions for the reconstructed top quark candidate are analyzed in four search regions in the $\ell \tau _\mathrm {h}$+jets channel. The observed number of events are found to be in agreement with the background predictions.

Upper limits on the production cross section of $\mathrm {LQ}_3$ pairs are set between 0.6 and 0.01$\,\text {pb}$ at 95% confidence level for $\mathrm {LQ}_3$ masses between 300 and 1700$\,\text {Ge}\text {V}$, assuming a branching fraction of $\mathcal {B} = 1$. The scalar $\mathrm {LQ}_3$ s are excluded with masses below 900$\,\text {Ge}\text {V}$, for $\mathcal {B}=1$. This result represents the most stringent limits to date on $\mathrm {LQ}_3$ s coupled to $\tau$ leptons and top quarks and constrains models explaining flavor anomalies in the $\mathrm {b}$ quark sector through contributions from scalar LQs.