Search for a heavy resonance decaying to a top quark and a vector-like top quark in the lepton + jets final state in pp collisions at $\sqrt{s} = 13\,\text {TeV}$

Abstract

A search is presented for a heavy spin-1 resonance $\mathrm {Z}'$ decaying to a top quark and a vector-like top quark partner $\text {T}$ in the lepton + jets final state. The search is performed using a data set of $\mathrm {p}$ $\mathrm {p}$ collisions at a centre-of-mass energy of 13$\,\text {TeV}$ corresponding to an integrated luminosity of $35.9{\,\text {fb}^{-1}}$ as recorded by the CMS experiment at the CERN LHC in the year 2016. The analysis is optimised for final states arising from the $\text {T}$ decay modes to a top quark and a Higgs or $\mathrm {Z}$ boson ($\text {T} \rightarrow \mathrm {H} \mathrm {t}, \mathrm {Z} \mathrm {t}$). The event selection makes use of resolved and merged top quark decay products, as well as decays of boosted Higgs bosons and $\mathrm {Z}$ and $\mathrm {W}$ bosons using jet substructure techniques. No significant deviation from the standard model background expectation is observed. Exclusion limits on the product of the cross section and branching fraction for $\mathrm {Z}'\rightarrow \mathrm {t}\text {T}, \text {T} \rightarrow \mathrm {H} \mathrm {t}, \mathrm {Z} \mathrm {t}, \mathrm {W}\mathrm {b}$ are presented for various combinations of the $\mathrm {Z}'$ resonance mass and the vector-like $\text {T}$ quark mass. These results represent the most stringent limits to date for the decay mode $\mathrm {Z}'\rightarrow \mathrm {t}\text {T} \rightarrow \mathrm {t}\mathrm {H} \mathrm {t}$. In a benchmark model with extra dimensions, invoking a heavy spin-1 resonance $\mathrm {G}^*$, masses of the $\mathrm {G}^*$ between 1.5 and 2.3$\,\text {TeV}$ and between 2.0 and 2.4$\,\text {TeV}$ are excluded for $\text {T}$ masses of 1.2 and 1.5$\,\text {TeV}$, respectively.

1 Introduction

Many extensions of the standard model (SM) predict the existence of heavy bosonic resonances, such as the composite spin-1 resonances found in the $\rho ^0$ model [1], or the lightest Kaluza–Klein excitation of the gluon [2] in Randall–Sundrum models [3, 4]. In models that invoke such states to address the hierarchy problem, these resonances are required to cancel top quark loop contributions to the radiative corrections that would otherwise drive the Higgs boson mass up to the Planck scale. As a consequence, these resonances typically feature an enhanced coupling to third-generation SM quarks, resulting in decays predominantly to SM top quarks. The ATLAS and CMS Collaborations have performed searches for such resonances in their proton–proton ($\mathrm {p}\mathrm {p}$) collision data sets at centre-of-mass energies $\sqrt{s} = 8$ [5,6,7] and 13$\,\text {TeV}$ [8,9,10], leading to stringent exclusion limits on the product of the resonance cross section and branching fraction, and therefore also on the resonance masses when interpreted in the context of specific models. However, in some models describing physics beyond the standard model (BSM) the new heavy resonance state is accompanied by an additional fermionic sector realised as a non-chiral (or vector-like) fourth generation of quarks. Vector-like quarks (VLQs) are fermions whose left- and right-handed components transform in the same way under the electroweak symmetry group. For this reason, direct mass terms for VLQs are not forbidden in the BSM Lagrangian. Furthermore, unlike sequential fourth-generation chiral quarks, the existence of VLQs is not yet excluded by recent Higgs boson measurements [11].

Examples of BSM models with these additional particles are models with a heavy gluon [2], a composite Higgs boson [12,13,14,15], or extra spatial dimensions [3, 16]. In these models, decays of the heavy resonance into final states with VLQs are allowed. The analysis presented in this paper searches for evidence of the production of a heavy spin-1 resonance, denoted as $\mathrm {Z}'$, with decays to an SM top quark and a vector-like top quark partner $\text {T}$, $\mathrm {Z}'\rightarrow \mathrm {t}\text {T}$. This decay mode is dominant for the intermediate $\mathrm {Z}'$ mass region in which the decay is kinematically allowed ($M_{\mathrm {t}}+M_{\text {T}} <M_{\mathrm {Z}'}$) and the decay to two VLQ top quark partners ($\mathrm {Z}'\rightarrow \text {T} \text {T}$) is kinematically forbidden ($M_{\mathrm {Z}'} <2M_{\text {T}}$). The VLQ $\text {T}$ considered in this analysis has three decay channels: to a Higgs boson and a top quark $(\text {T} \rightarrow \mathrm {H} \mathrm {t})$, to a $\mathrm {Z}$ boson and a top quark $(\text {T} \rightarrow \mathrm {Z} \mathrm {t})$, and to a $\mathrm {W}$ boson and a bottom quark $(\text {T} \rightarrow \mathrm {W}\mathrm {b})$. The leading-order Feynman diagram showing the production mode of the $\mathrm {Z}'$ boson and its subsequent decay, including the decay of the VLQ, is shown in Fig. 1.

Fig. 1

Leading order Feynman diagram for the production of a spin-1 resonance $\mathrm {Z}'$ and its decay, along with the possible decays of the vector-like quark $\text {T}$

A first search for the production of a heavy $\mathrm {Z}'$ resonance decaying to $\mathrm {t}\text {T}$ was performed by the CMS Collaboration [17] using a data set corresponding to an integrated luminosity of 2.6$\,\text {fb}^{-1}$, recorded at a centre-of-mass energy of $\sqrt{s}=13\hbox {TeV}$. The analysis was optimised for the decay mode $\text {T} \rightarrow \mathrm {W}\mathrm {b}$ and made use of the all-hadronic final state, where both the top quark and $\mathrm {W}$ boson are highly Lorentz-boosted, resulting in a three-jet event topology. No deviation from the SM expectation was observed and upper limits on the cross section were obtained, ranging from 0.13 to 10$\hbox {\,pb}$ depending on the masses of the $\mathrm {Z}'$ and $\text {T}$. A search for the single production of a vector-like quark $\text {T}$ decaying to a $\mathrm {Z}$ boson and a top quark in the dilepton + jets final state has also been performed, with an interpretation of the results in the context of a $\mathrm {Z}'$ decaying into $\mathrm {t}\text {T}$ [18]. Upper limits on the cross section ranging from 0.06 to 0.13$\hbox {\,pb}$ were obtained, for the production of a $\mathrm {Z}'$ with the subsequent decays to $\mathrm {t}\text {T}$ and $\text {T} \rightarrow \mathrm {Z} \mathrm {t}$. Searches for $\text {T}$ pair production have been performed by the ATLAS [19,20,21,22,23,24] and CMS [25, 26] Collaborations, placing a lower bound of $\approx 1.3\hbox { TeV}$ on the VLQ mass.

The analysis presented here is optimised for $\mathrm {Z}'\rightarrow \mathrm {t}\text {T}$ with the $\text {T}$ decay modes $\text {T} \rightarrow \mathrm {Z} \mathrm {t}$ and $\text {T} \rightarrow \mathrm {H} \mathrm {t}$, in the monolepton + jets final state. The search is performed in a data set of $\mathrm {p}$ $\mathrm {p}$ collisions at $\sqrt{s}=13\,\text {TeV}$ corresponding to $35.9{\,\text {fb}^{-1}}$ [27] as recorded by the CMS experiment during the year 2016.

The two decay channels ($\mathrm {Z}'\rightarrow \mathrm {t}\text {T} \rightarrow \mathrm {t}\mathrm {H} \mathrm {t}$, $\mathrm {Z}'\rightarrow \mathrm {t}\text {T} \rightarrow \mathrm {t}\mathrm {Z} \mathrm {t}$) each produce two top quarks accompanied by one boson. The event selection in the single lepton + jets final state relies on the leptonic decay of one of these top quarks. The $\mathrm {H}$ or $\mathrm {Z}$ boson from the $\text {T}$ decay is expected to receive a large Lorentz boost because of the large mass difference between the $\text {T}$ and the boson. The resultant hadronic decay products will be merged and are thus reconstructed as a single broad jet. Jet substructure techniques are used for the boson identification [28] and in the event categorisation. Additional categories for both resolved and merged decays of the other hadronically decaying top quark are considered to ensure a high sensitivity over a broad range of the $\mathrm {Z}'$ mass. In all categories, the $\mathrm {Z}'$ mass is reconstructed by considering various combinations of reconstructed objects, with the final combination determined by a $\chi ^2$ metric.

Limits at 95% confidence level ($\text {CL}$) are derived for all three $\text {T}$ decay channels ($\text {T} \rightarrow \mathrm {H} \mathrm {t}$, $\text {T} \rightarrow \mathrm {Z} \mathrm {t}$, $\text {T} \rightarrow \mathrm {W}\mathrm {b}$) using a template-based statistical evaluation of the reconstructed mass spectra of the $\mathrm {Z}'$ boson from all categories. A mass range of the $\mathrm {Z}'$ from 1.5 to 4.0$\,\text {TeV}$ and of the $\text {T}$ from 0.7 to 3.0$\,\text {TeV}$ is considered. The rate of the dominant SM background processes (${\mathrm {t}\overline{\mathrm {t}}}$ and $\mathrm {W}\text {+jets}$) predicted by simulation is constrained using the mass spectra in dedicated control regions that enhance these background processes, and are fit simultaneously with the signal regions.

The search is performed in a model-independent manner by scanning over a large range of possible masses of the $\mathrm {Z}'$ and $\text {T}$ and couplings of the $\text {T}$ to various final states. The results are then interpreted in the context of two theory benchmark models.

In the $\mathrm {G}^*$ model [2], ten new VLQs ($\text {T}$, ${\mathrm {B}}$, ${\tilde{\text {T}}}$, ${\tilde{{\mathrm {B}}}}$, $\text {T} _{5/3}$, $\text {T} _{2/3}$, $\text {T} ^\prime$, ${\mathrm {B}}^\prime$, ${\mathrm {B}}_{-1/3}$, ${\mathrm {B}}_{-4/3}$) are predicted with well-defined relationships between their masses. In this analysis, the $\text {T}$ mass is varied, whilst other masses are related by $M_{\text {T} _{5/3}} = M_{\text {T} _{2/3}} = M_{\text {T}} \cos \phi _{\mathrm {L}}$. The mixing angle $\cos \phi _{\mathrm {L}}$ governs the degree of compositeness of the left-handed quark doublet $(\mathrm {t}_{\mathrm {L}}, \mathrm {b}_{\mathrm {L}})$, and hence the relative coupling of the lightest spin-1 Kaluza–Klein excitation of the gluon, $\mathrm {G}^*$, to third-generation quarks compared to the other two generations of quarks. A benchmark scenario with parameters $\tan \theta = 0.44$, $\sin \phi _{\mathrm {tR}} = 0.6$, and $Y_{*} = 3$ is used in this analysis, leading to $\cos \phi _{\mathrm {L}} = 0.84$. A description of the benchmark and its parameters can be found in Ref. [2]. In this model the branching fractions (${\mathcal {B}}$) of the $\text {T}$ decay to $\mathrm {W}\mathrm {b}$, $\mathrm {H} \mathrm {t}$, and $\mathrm {Z} \mathrm {t}$ are chosen to be 0.5, 0.25, and 0.25, respectively.

The $\rho ^0$ model [1] predicts a heavy spin-1 resonance, $\rho$, along with a multiplet of four new vector-like quarks, with two of the vector-like quarks ($\text {T}$, $\text {B}$) representing the heavy partners of the top and $\mathrm {b}$ quarks, respectively. Other exotic vector-like quarks are also predicted: $\text {X}_{2/3}$ with a charge of $2\,e/3$, and $\text {X}_{5/3}$ with a charge of $5\,e/3$, where e is the magnitude of the charge of the electron. A benchmark scenario with parameters $y_{\mathrm {L}}=c_2=c_3=1$ and $g_{\rho _{\mathrm {L}}}=3$ is used in this analysis, where a description of the benchmark and its parameters can be found in Ref. [1]. In this model the branching fractions of the $\text {T}$ decay to $\mathrm {W}\mathrm {b}$, $\mathrm {H} \mathrm {t}$, and $\mathrm {Z} \mathrm {t}$ are chosen to be 0, 0.5, and 0.5, respectively.

After a short description of the CMS experiment (Sect. 2), the reconstruction of events is discussed in Sect. 3, and the data sets and the simulated samples are introduced in Sect. 4. The event selection, the categorisation, and the procedures for the mass reconstruction are discussed in Sect. 5. In Sect. 6 the background estimation procedure is explained, and Sect. 7 gives an overview of the systematic uncertainties. Finally, the search results and the interpretation in the benchmark models are presented in Sect. 8. This paper concludes with a summary in Sect. 9.

2 The CMS detector

The central feature of the CMS apparatus is a superconducting solenoid of $6\hbox { m}$ internal diameter, providing a magnetic field of $3.8\hbox { T}$. Contained within the solenoid volume are a silicon pixel and strip tracker, a lead tungstate crystal electromagnetic calorimeter (ECAL), and a brass and scintillator hadron calorimeter (HCAL), each composed of a barrel and two endcap sections. Muons are detected in gas-ionization chambers embedded in the steel flux-return yoke outside the solenoid. Extensive forward calorimetry complements the pseudorapidity ($\eta$) coverage provided by the barrel and endcap detectors. Events of interest are selected using a two-tiered trigger system [29]. The first level, composed of custom hardware processors, uses information from the calorimeters and muon detectors to select events at a rate of around $100\hbox { kHz}$ within a time interval of less than 4$\,\upmu \text {s}$. The second level, known as the high-level trigger, consists of a farm of processors running a version of the full event reconstruction software optimised for fast processing, and reduces the event rate to around $1\hbox { kHz}$ before data storage. A more detailed description of the CMS detector, together with a definition of the coordinate system used and the relevant kinematic variables, can be found in Ref. [30].

3 Event reconstruction

The particle-flow (PF) algorithm [31] deployed by the CMS Collaboration aims to reconstruct and identify each individual particle in an event, with an optimised combination of information from the various elements of the CMS detector. The energy of photons is obtained from the ECAL measurement. The energy of electrons is determined from a combination of the electron momentum at the primary interaction vertex as determined by the tracker, the energy of the corresponding ECAL cluster, and the energy sum of all bremsstrahlung photons spatially compatible with originating from the electron track. The energy of muons is obtained from the curvature of the corresponding track. The energy of charged hadrons is determined from a combination of their momentum measured in the tracker and the matching ECAL and HCAL energy deposits, corrected for zero-suppression effects and for the response function of the calorimeters to hadronic showers. Finally, the energy of neutral hadrons is obtained from the corresponding corrected ECAL and HCAL energies.

The primary $\mathrm {p}\mathrm {p}$ interaction vertex is taken to be the reconstructed vertex with the largest value of summed physics-object $p_{\mathrm {T}} ^2$, where $p_{\mathrm {T}}$ is the transverse momentum. Here the physics objects are the objects returned by a jet finding algorithm [32, 33] applied to all tracks associated with the vertex, plus the negative vector sum of the $p_{\mathrm {T}}$ of those jets.

Muons are reconstructed through a fit to hits in both the inner tracking system and the muon spectrometer [34, 35]. Muons must satisfy identification and reconstruction requirements on the impact parameters of the track, the number of hits reconstructed in both the silicon tracker and the muon detectors, and the uncertainty in the $p_{\mathrm {T}}$. These quality criteria ensure a precise measurement of the four-momentum, and rejection of badly reconstructed muons.

Electron candidates are required to have a match between the energy deposited in the ECAL and the momentum determined from the reconstructed track [36]. To suppress the multijet background, electron candidates must pass stringent identification criteria. These include requirements on the geometrical matching between ECAL deposits and position of reconstructed tracks, the ratio of the energies deposited in the HCAL and ECAL, the spatial distribution of the ECAL depositions, the impact parameters of the track, and the number of reconstructed hits in the silicon tracker.

In the $\mathrm {Z}'$ signal targeted by this analysis, the lepton is emitted in the decay chain of a top quark ($\mathrm {t}\rightarrow \mathrm {b}\ell \nu _{\ell }$) at high $p_{\mathrm {T}}$. Because of the Lorentz boost of the top quark, the lepton is expected to be in angular proximity to a $\mathrm {b}$ quark, and therefore conventional lepton isolation criteria would lead to a loss in signal efficiency. Instead, a dedicated two-dimensional criterion is used to reduce the background of leptons arising from heavy-flavour quark decays in multijet events produced through the strong interaction. This criterion is discussed in Sect. 5.1.

In this analysis hadronic jets are reconstructed from PF candidates using the anti-$k_{\mathrm {T}}$ algorithm [32] as implemented in the FastJet software package [33]. Since the analysis targets both resolved and merged top quark decays, jets are clustered with two values of the distance parameter R; $R = 0.4$ (AK4 jets) for the reconstruction of resolved top quark decay products, and $R = 0.8$ (AK8 jets) for the reconstruction of merged $\mathrm {W}$, $\mathrm {Z}$, and Higgs boson decay products as well as merged top quark decays. In the jet clustering procedure, charged PF candidates associated with non-primary vertices are excluded. The jet momentum is determined as the vectorial sum of all particle momenta in the jet, and is found from simulation to be within 5–10% of the true momentum over the whole $p_{\mathrm {T}}$ spectrum and detector acceptance. A correction based on the area of the jet, projected on the front face of the calorimeter, is used to correct for the extra energy clustered in jets due to additional inelastic proton–proton interactions within the same or adjacent bunch crossings (pileup) [33]. Jet energy corrections are derived from simulation in order to bring the measured response of jets to that of particle level jets on average. Dijet, multijet, photon + jet, and leptonically-decaying $\mathrm {Z}$ + jet events are used to perform in situ measurements of the momentum balance to derive corrections for residual differences in jet energy scale in data and simulation [37]. Additional selection criteria are applied to each event to remove spurious jet-like features originating from isolated noise patterns in certain HCAL regions [38]. The clustered jets also contain leptons. To avoid double counting of the lepton momentum in an event, the lepton used for the reconstruction of the $\mathrm {W}$ boson from the leptonic top quark decay is removed from an AK4 jet if the lepton overlaps with the jet within the jet’s radius parameter, $\varDelta R(\ell ,\, \mathrm {j}) < 0.4$, where $\varDelta R (\ell ,\, \mathrm {j}) = \sqrt{\smash [b]{[\varDelta \eta (\ell ,\, \mathrm {j})]^2+[\varDelta \phi (\ell ,\, \mathrm {j})]^2}}$, and $\varDelta \eta (\ell ,\, \mathrm {j})$ and $\varDelta \phi (\ell ,\, \mathrm {j})$ are the separations in pseudorapidity and azimuthal angle, respectively, between the lepton and jet. The momentum of the lepton is subtracted from that of the jet before jet energy corrections are applied. Larger radius AK8 jets that overlap with the lepton within $\varDelta R(\ell ,\, \mathrm {j}) < 0.8$ are not considered in this analysis. Jets that are produced by the decay of $\mathrm {b}$ quarks can be identified using the combined secondary vertex discriminator [39]. An AK4 jet is denoted as being $\mathrm {b}$ tagged if it fulfils the medium working point of the discriminator, which has an efficiency of 63% for correctly identifying a $\mathrm {b}$ quark jet, with a 1% probability of misidentifying a light-flavour jet as $\mathrm {b}$ tagged (a mistag).

The boosted bosons and merged top quark decays are identified by applying so-called taggers to AK8 jets. Each tagger requires the jet mass to be within a certain range, along with additional criteria on substructure variables such as N-subjettiness [40] or subjet $\mathrm {b}$ tagging [39]. The jet mass is computed after applying a modified mass-drop algorithm [41, 42], known as the soft drop algorithm [43], which eliminates soft, large-angle radiation resulting from SM quantum chromodynamics (QCD) processes. This improves the jet mass resolution for the reconstructed boson and top quark. It also reduces the mass of jets initiated by single quarks or gluons, thereby improving discrimination between jets from true boson or top quark decays, and those from background QCD multijet events. Furthermore, it helps mitigate the effect of pileup [28]. The N-subjettiness variable $\tau _{\mathrm {N}}$ quantifies the compatibility of the jet clustering with the hypothesis that exactly N subjets are present, with small values of $\tau _{\mathrm {N}}$ indicating greater compatibility. The N-subjettiness ratios $\tau _{21} = \tau _2 / \tau _1$ and $\tau _{32} = \tau _3 / \tau _2$ are calculated prior to the application of the soft drop algorithm, and are used to reject background jets arising from the hadronization of single quarks or gluons. Jets from hadronic $\mathrm {Z}/\mathrm {W}$ boson decays in signal events are characterized by smaller values of $\tau _{21}$ in comparison to jets from QCD multijet background processes, and similarly jets from merged hadronic top quark decays have smaller values of $\tau _{32}$ than background jets. For each AK8 jet, two subjets are obtained from the soft drop algorithm. An AK8 jet can have up to two subjet $\mathrm {b}$ tags depending on how many subjets fulfil the loose working point of the $\mathrm {b}$ tagging discriminator. In contrast to the medium working point applied to AK4 jets, the loose working point has a larger $\mathrm {b}$ tagging efficiency of 83%, at the expense of a larger mistag probability of 9%.

The missing transverse momentum $p_{\mathrm {T}} ^\text {miss}$ is defined as the magnitude of the vector sum of the transverse momenta of the reconstructed PF objects, ${\vec p}_{\mathrm {T}}^{\text {miss}}$. The value of $p_{\mathrm {T}} ^\text {miss}$ is modified to account for corrections to the energy scale of the reconstructed AK4 jets in the event.

4 Data and simulated samples

The analysis is based on the data set of $\mathrm {p}\mathrm {p}$ collisions recorded by the CMS detector during the year 2016. Events targeting the decay of a top quark to a final state including a muon are selected with a high-level single-muon trigger that requires the presence of at least one muon candidate with $p_{\mathrm {T}} >50\,\text {GeV}$ and $|\eta |<2.4$. For events targeting a final state with an electron, the high-level trigger requires the presence of at least one electron candidate with $p_{\mathrm {T}} >115\,\text {GeV}$ and $|\eta |<2.5$, or at least one photon with $p_{\mathrm {T}} >175\,\text {GeV}$ and $|\eta |<2.5$. The latter requirement ensures events containing electrons with a high $p_{\mathrm {T}}$ are efficiently selected, as the requirements on ECAL shower shapes are less stringent for photons than for electrons. Given the highly boosted topology of the final-state objects, no isolation requirements are applied to the lepton candidates at the trigger level. The electron trigger threshold is significantly larger than the muon trigger threshold, since the non-isolated electron trigger selects a large number of hadrons incorrectly identified as electrons. Both recorded data sets correspond to an integrated luminosity of $35.9{\,\text {fb}^{-1}}$ [27] .

The spin-1 resonance signal samples are generated with the leading-order (LO) mode of MadGraph5_amc@nlo 2.2.2 [44] as a high mass resonance with SM-like couplings using the $\mathrm {G}^*$ model [2]. The pythia 8.212 [45] event generator with the CUETP8M1 underlying event tune [46, 47] is used to model the parton showering and underlying event. Separate samples for the different decay channels of the $\text {T}$ are produced, so that each sample has a branching fraction of 100% to the chosen decay channel. Throughout this paper, a generic spin-1 heavy resonance will be referred to as $\mathrm {Z}'$, whilst interpretations within a given model will refer to their specific resonance names.

We consider only $\mathrm {Z}'$ and $\text {T}$ masses that result in a significant branching fraction of the $\mathrm {Z}'$ to $\mathrm {t}\text {T}$. Scenarios where the mass of the $\mathrm {Z}'$ is smaller than the sum of the top quark mass and the $\text {T}$ mass are not considered, since the $\mathrm {Z}'$ would then decay to SM quark pairs, and such scenarios have been largely excluded by previous searches [5, 9]. Masses of the $\mathrm {Z}'$ larger than twice the $\text {T}$ mass are also not considered, as in such cases the $\mathrm {Z}'$ decays predominantly to $\text {T}$ pairs, resulting in a large $\mathrm {Z}'$ width. Such large masses are better targeted by direct searches for $\text {T}$ pair production.

Two values of the $\mathrm {Z}'$ width are considered, corresponding to 1% or 30% of its mass. The $\text {T}$ width is set to 1% of its mass. For the $\mathrm {Z}'$ and $\text {T}$ mass parameter space considered in this analysis the total $\mathrm {Z}'$ decay width in the two considered theoretical models is always less than 20% of its mass. Since the experimental resolution is approximately 15%, the samples with the $\mathrm {Z}'$ width set to 1% are dominated by the experimental resolution, and are thus used in the interpretation of the results. The samples generated with the width of 30% are used as cross-checks and help to confirm that the conclusions do not change for scenarios with $\mathrm {Z}'$ widths somewhat larger than the experimental resolution for high masses of the $\mathrm {Z}'$. Furthermore, it was checked that scenarios with $\text {T}$ widths of up to 30%, with a $\mathrm {Z}'$ width equal to or larger than that of the $\text {T}$, do not significantly affect the resolution of the $\mathrm {Z}'$ mass, and therefore the experimental limits obtained with the $\text {T}$ width set to 1% are also valid for larger $\text {T}$ width scenarios.

The $\mathrm {G}^*$ model considers only left-handed $\text {T}$ quarks. The $\rho ^0$ model also allows for a right-handed $\rho _{\mathrm {R}}$ coupling to $\text {T}$ quarks. For the $\text {T} \rightarrow \mathrm {H} \mathrm {t}$ decay mode the kinematic distributions in the $\mathrm {G}^*$ model and $\rho ^0$ model are the same. While for the $\text {T} \rightarrow \mathrm {Z} \mathrm {t}$ and $\text {T} \rightarrow \mathrm {W}\mathrm {b}$ decay modes the $\mathrm {Z}/\mathrm {W}$ boson $p_{\mathrm {T}}$ spectra are similar for the left-handed $\rho _{\mathrm {L}}$ and the $\mathrm {G}^*$, the ratio of the distributions for left- and right-handed scenarios in the $\rho ^0$ model deviates from unity by up to 30%. In this analysis only the decays of the left-handed $\rho _{\mathrm {L}}$ are considered.

Simulated event samples for the SM background processes Drell–Yan (DY) + jets, also referred to as $\mathrm {Z} \text {+jets}$, and $\mathrm {W}\text {+jets}$ are computed at next-to-leading-order (NLO) precision in QCD with MadGraph5_amc@nlo. The parton showering is calculated using pythia 8 following the FxFx merging scheme [48]. Background events from QCD multijet processes are simulated using pythia 8. For the simulation of the underlying event, the tune CUETP8M1 is used in pythia 8 for the $\mathrm {W}\text {+jets}$, $\mathrm {Z} \text {+jets}$, and QCD multijets samples.

The simulation of SM ${\mathrm {t}\overline{\mathrm {t}}}$ and single top quark (ST) background events is performed with the powheg event generator [49,50,51,52,53,54,55,56,57], using powheg v1.0 for the simulation of $\mathrm {t}\mathrm {W}$ events, whilst powheg v2.0 was used for the simulation of ${\mathrm {t}\overline{\mathrm {t}}}$ and all other single top quark processes. The pythia 8 generator was used for the showering in both versions of powheg. An observed discrepancy between simulation and data in the top quark $p_{\mathrm {T}}$ spectrum is corrected with a reweighting procedure based on measurements of the top quark $p_{\mathrm {T}}$ spectrum [58, 59]. The underlying event tune CUETP8M2T4 [60] is used in pythia 8 for the ${\mathrm {t}\overline{\mathrm {t}}}$ and single top quark samples.

All events are generated with the NNPDF 3.0 parton distribution functions (PDFs) [61]. The detector response is simulated with the Geant4 package [62]. Simulated events are processed through the same software chain as used for collision data. All simulated event samples include the simulation of pileup, and are reweighted to match the observed distribution of the number of pileup interactions in data.

5 Event selection, categorisation, and mass reconstruction

5.1 Event selection

All events are required to contain at least one reconstructed interaction vertex within a volume 24$\,\text {cm}$ in length and 2$\,\text {cm}$ in radius, centred on the mean $\mathrm {p}\mathrm {p}$ collision point [63].

Since there are differences in the way the electrons and muons from top quark decays are treated, the analysis considers the $\mathrm {e}\text {+jets}$ and $\mu \text {+jets}$ channels separately. In the $\mu \text {+jets}$ channel exactly one reconstructed muon with $p_{\mathrm {T}} > 53\,\text {GeV}$ and $|\eta | < 2.4$ is required. In the $\mathrm {e}\text {+jets}$ channel exactly one electron with $p_{\mathrm {T}} > 125\,\text {GeV}$ and $|\eta | < 2.5$ is required. Events with an electron candidate located inside the transition region between the ECAL barrel and endcaps ($1.44< |\eta | <1.57$) are rejected. In the $\mathrm {e}\text {+jets}$ channel, an additional requirement of $p_{\mathrm {T}} ^\text {miss} > 90 \,\text {GeV}$ from the associated neutrino is introduced to reduce the background of hadrons falsely identified as electrons in QCD multijet events.

Because of the boosted nature of the signal, conventional lepton isolation criteria are not applicable. Instead, in both the $\mathrm {e}\text {+jets}$ and $\mu \text {+jets}$ channels, events are required to pass a two-dimensional selection of either $\varDelta R (\ell ,\, \mathrm {j}) > 0.4$ or $p_{\mathrm {T},\text {rel}} (\ell ,\, \mathrm {j}) > 40\,\text {GeV}$, where $\mathrm {j}$ is the AK4 jet with minimal angular separation $\varDelta R$ from the lepton $\ell$ (electron or muon), and $p_{\mathrm {T},\text {rel}} (\ell ,\, \mathrm {j})$ is the component of the lepton momentum orthogonal to the axis of jet $\mathrm {j}$. Only AK4 jets with $p_{\mathrm {T}} >15\,\text {GeV}$ are considered in this criterion. The chosen working point has an efficiency of ${\approx }30\%$ for a lepton with $p_{\mathrm {T}} = 100\,\text {GeV}$, increasing with $p_{\mathrm {T}}$ and reaching a plateau of ${\approx }90\%$ at $p_{\mathrm {T}} =400\,\text {GeV}$. The background rejection rate is 99.5% at $p_{\mathrm {T}} = 100\,\text {GeV}$, decreasing to ${\approx }94\%$ at $p_{\mathrm {T}} = 400\,\text {GeV}$.

In order to reconstruct the boosted $\mathrm {H}$/$\mathrm {Z}$/$\mathrm {W}$ boson or merged top quark decays, events are required to contain at least one AK8 jet with $p_{\mathrm {T}} > 250\,\text {GeV}$ and a soft drop jet mass $M_{\text {AK8}}^{\text {SD}}> 30\,\text {GeV}$.

5.2 Event categorisation

After the event selection, different taggers are used for the identification of hadronic decays of boosted $\mathrm {Z}/\mathrm {W}$ bosons, Higgs bosons, and top quarks, called in the following $\mathrm {Z}/\mathrm {W}$ tagger, $\mathrm {H}$ tagger, and $\mathrm {t}$ tagger, respectively. These taggers make use of the soft drop mass of AK8 jets, whose distribution after the event selection is shown in Fig. 2 for data, a simulated signal for each $\text {T}$ decay mode, and the simulated SM backgrounds. No corrections to the SM backgrounds from the fit to data (explained in Sect. 6) are applied in this figure. The selection criteria of the different taggers are:

• $\mathrm {Z}/\mathrm {W}$ tagger: AK8 jets are denoted $\mathrm {Z}/\mathrm {W}$-tagged if their soft drop jet mass is in the range $60< M_{\text {AK8}}^{\text {SD}}< 115\,\text {GeV}$ and their N-subjettiness ratio fulfils $\tau _{21} < 0.5$.

• $\mathrm {H}$ tagger: two different $\mathrm {H}$ taggers are used:

  • $\mathrm {H} _{2\mathrm {b}}$ tagger: AK8 jets are denoted $\mathrm {H} _{2\mathrm {b}}$-tagged if their soft drop jet mass is in the range $100< M_{\text {AK8}}^{\text {SD}}< 150\,\text {GeV}$ and two subjet $\mathrm {b}$ tags are found. This more stringent selection is used to reduce backgrounds in regions with significant background contributions.

  • $\mathrm {H} _{1\mathrm {b}}$ tagger: AK8 jets are denoted $\mathrm {H} _{1\mathrm {b}}$-tagged if their soft drop jet mass is in the range $100< M_{\text {AK8}}^{\text {SD}}< 150\,\text {GeV}$ and exactly one subjet $\mathrm {b}$ tag is found. This less stringent selection is used in regions with low background contributions.

• $\mathrm {t}$ tagger: AK8 jets are denoted $\mathrm {t}$-tagged if their soft drop jet mass is in the range $150< M_{\text {AK8}}^{\text {SD}}< 220\,\text {GeV}$ and their N-subjettiness ratio fulfils $\tau _{32} < 0.57$.

Fig. 2

Distribution of the soft drop mass of jets as reconstructed with the anti-$k_{\mathrm {T}}$ jet algorithm with $R=0.8$ after the event selection. Events are shown in the combined lepton + jets channel, with contributions from data, simulated signal samples, and the simulated SM backgrounds without corrections from the fit to data (explained in Sect. 6). The expected signal distribution from various $\text {T}$ decay modes is shown for the example mass configuration $M_{\mathrm {Z}'} =1.5\,\text {TeV}$ and $M_{\text {T}} =1.3\,\text {TeV}$ with a nominal cross section $\sigma (\mathrm {Z}'\rightarrow \mathrm {t}\text {T})$ of 1$\hbox {\,pb}$. The lower panel shows the ratio of data to predicted background. Here the darker grey band indicates the statistical uncertainty, whilst the lighter grey band shows the combined statistical and systematic uncertainty

Events are required to contain at least one $\mathrm {H} _{2\mathrm {b}}$-tagged jet, one $\mathrm {H} _{1\mathrm {b}}$-tagged jet, or one $\mathrm {Z}/\mathrm {W}$-tagged jet. Because of the overlapping mass windows in the tagger definitions, an AK8 jet may be tagged by several taggers. In this case, the priority is given in the following order to ensure the best signal sensitivity: $\mathrm {H} _{2\mathrm {b}}$, then $\mathrm {H} _{1\mathrm {b}}$, and finally $\mathrm {Z}/\mathrm {W}$, such that events can be categorised into three exclusive event categories based on the presence of $\mathrm {H} _{2\mathrm {b}}$-, $\mathrm {H} _{1\mathrm {b}}$-, and $\mathrm {Z}/\mathrm {W}$-tagged jets. To maintain sensitivity to both merged and resolved top quark decays, each of these three categories is further split into two subcategories, containing events with and without a $\mathrm {t}$-tagged AK8 jet, respectively. The resulting six exclusive event categories are listed in the first column of Table 1, which shows the selection efficiency for each decay channel of the $\text {T}$ in each event category for a signal with $M_{\mathrm {Z}'} =2.5\,\text {TeV}$ and $M_{\text {T}} =1.3\,\text {TeV}$. The selection requirements include all aforementioned requirements, along with requirements on the reconstructed ${\mathrm {t}\overline{\mathrm {t}}}$ system, detailed in the following section. The four $\mathrm {H}$ tag categories feature a higher selection efficiency for the decay $\text {T} \rightarrow \mathrm {H} \mathrm {t}$, while the decays $\text {T} \rightarrow \mathrm {Z} \mathrm {t}$ and $\text {T} \rightarrow \mathrm {W}\mathrm {b}$ are selected in the two $\mathrm {Z}/\mathrm {W}$ tag categories for both the resolved and boosted top quark scenarios. These decays channels are also selected by the $\mathrm {H}$ tag categories, as they are prioritised over the $\mathrm {Z}/\mathrm {W}$ tag categories. Signal events with the $\mathrm {t}\mathrm {H} \mathrm {t}$ final state are reconstructed in the two $\mathrm {Z}/\mathrm {W}$ tag categories if the $\mathrm {b}$ tag criteria for the subjets are not fulfilled, but the $\mathrm {Z}/\mathrm {W}$ tagging requirements are met. The selection efficiency is lower in categories requiring a $\mathrm {t}$ tag, since any top quark produced by this chosen signal will not be significantly boosted, and therefore will not be efficiently identified by the $\mathrm {t}$ tagger.

Table 1 Signal selection efficiency for the three $\text {T}$ decay modes in each category for a signal with $M_{\mathrm {Z}'} =2.5\,\text {TeV}$ and $M_{\text {T}} =1.3\,\text {TeV}$, taking into account branching fractions ${\mathcal {B}}(\mathrm {t}\mathrm {H} \mathrm {t}\rightarrow \ell \text {+jets}) = 0.294$, ${\mathcal {B}}(\mathrm {t}\mathrm {Z} \mathrm {t}\rightarrow \ell \text {+jets}) = 0.317$, and ${\mathcal {B}}(\mathrm {t}\mathrm {W}\mathrm {b}\rightarrow \ell \text {+jets}) = 0.255$ [64], where $\ell \text {+jets}$ is a final state with exactly one electron or muon originating from the decay of one of the top quarks, including electrons and muons from tau lepton decays. The last row of the table shows the total selection efficiency summed over all six categories. The efficiencies are shown after all selection requirements, including those on the reconstructed ${\mathrm {t}\overline{\mathrm {t}}}$ system as detailed in Sect. 5.3

5.3 Mass reconstruction

The reconstructed $\mathrm {Z}'$ mass $M_{\mathrm {Z}'}^{\text {rec}}$ is used as the discriminating observable between background and signal in this analysis. In addition to the $\mathrm {Z}/\mathrm {W}$ or Higgs boson, the signature also requires a ${\mathrm {t}\overline{\mathrm {t}}}$ pair. The reconstruction of a fully resolved ${\mathrm {t}\overline{\mathrm {t}}}$ system is performed by defining top quark candidates, built from the four-momenta of the reconstructed objects. One candidate is constructed for the hadronic decay of the (anti)top quark (denoted the hadronic top quark candidate), and one for the leptonic decay of the (anti)top quark (denoted the leptonic top quark candidate). Objects that are used in the reconstruction are the ${\vec p}_{\mathrm {T}}^{\text {miss}}$, leptons, AK4 jets, and a $\mathrm {t}$-tagged AK8 jet, if present. Only AK4 jets with $p_{\mathrm {T}} > 30\,\text {GeV}$ and $|\eta | < 2.4$ are considered. The $\mathrm {b}$ tag information of the AK4 jets is not used in the reconstruction of the ${\mathrm {t}\overline{\mathrm {t}}}$ system, since it was found that applying it did not improve the assignment of jets to the top quark candidates. To ensure that there is no overlap between the two jet collections, AK4 jets that overlap with the $\mathrm {Z}/\mathrm {W}$- or $\mathrm {H}$-tagged jet within $\varDelta R(\text {tagged AK8 jet, AK4 jet}) < 1.2$ are removed from the event. If an event has a $\mathrm {t}$-tagged jet, AK4 jets with $\varDelta R(\mathrm {t}\text {-tagged AK8 jet, AK4 jet}) < 1.2$ are removed from the analysis. Each possible possibility for assigning these objects to the ${\mathrm {t}\overline{\mathrm {t}}}$ system is considered a hypothesis. The best hypothesis is chosen by a $\chi ^2$ discriminator that is a measure of the quality of the reconstruction, combining information from both the hadronic and leptonic reconstructed top quark candidates. The procedure of building the hypotheses and calculating $\chi ^2$ is described in detail below.

The reconstruction of the leptonic top quark candidate requires a neutrino. Since neutrinos are not measurable in the detector, ${\vec p}_{\mathrm {T}}^{\text {miss}}$ is used to infer the four-momentum of the neutrino. It is assumed that neutrinos are the only source that contributes to ${\vec p}_{\mathrm {T}}^{\text {miss}}$, and thus the x and y components of the neutrino’s four-momentum are taken directly from ${\vec p}_{\mathrm {T}}^{\text {miss}}$. Assuming an on-shell $\mathrm {W}$ boson, the z component of the neutrino can be calculated by solving the quadratic equation relating the four-momenta of the $\mathrm {W}$ and its decay products:

$$\begin{aligned} p_{\mathrm {W}}^2 = M_{\mathrm {W}}^2 = (p_{\ell } + p_{\nu })^2. \end{aligned}$$

(1)

This quadratic equation can have zero, one, or two real solutions. In the case of zero real solutions only the real part of the complex solution is taken. Candidates are built for each of the neutrino solutions. In addition to the estimated neutrino momentum, the lepton momentum is assigned to the leptonic top quark candidate.

If an event has a $\mathrm {t}$-tagged jet, at least one additional AK4 jet is required. Since the hadronic top quark candidate is already determined, all remaining AK4 jets in the event are assigned either to the leptonic top quark or are not assigned to a candidate at all. If an event has no $\mathrm {t}$-tagged jet, at least two additional AK4 jets are required. All AK4 jets in the event are assigned either to the leptonic top quark candidate, the hadronic top quark candidate, or neither candidate, constructing all possible candidates. The four-momenta of the leptonically and hadronically decaying top quarks are then calculated by summing the four-momenta of the corresponding objects (lepton, $p_{\mathrm {T}} ^\text {miss}$, AK4 jets, and $\mathrm {t}$-tagged AK8 jet, if present).

Fig. 3

Distribution of the smallest $\chi ^2$ discriminator in each event for the combination of both top tag and no top tag categories, after the ${\mathrm {t}\overline{\mathrm {t}}}$ reconstruction, combining both lepton channels. The simulated SM backgrounds are shown without corrections from the fit to data (explained in Sect. 6). The expected signal distribution is shown for various $M_{\mathrm {Z}'}$ masses for a fixed mass $M_{\text {T}} =1.3 \,\text {TeV}$ in the $\text {T} \rightarrow \mathrm {H} \mathrm {t}$ decay channel, each with a nominal cross section $\sigma (\mathrm {Z}'\rightarrow \mathrm {t}\text {T})$ of 1$\hbox {\,pb}$. The lower panel shows the ratio of data to predicted background. Here the darker grey band indicates the statistical uncertainty, whilst the lighter grey band shows the combined statistical and systematic uncertainty

Table 2 Signal selection efficiency after each step in the selection requirements for a signal with $M_{\mathrm {Z}'} =2.5\,\text {TeV}$ and $M_{\text {T}} =1.3\,\text {TeV}$, taking into account branching fractions ${\mathcal {B}}(\mathrm {t}\mathrm {H} \mathrm {t}\rightarrow \ell \text {+jets}) = 0.294$, ${\mathcal {B}}(\mathrm {t}\mathrm {Z} \mathrm {t}\rightarrow \ell \text {+jets}) = 0.317$, and ${\mathcal {B}}(\mathrm {t}\mathrm {W}\mathrm {b}\rightarrow \ell \text {+jets}) = 0.255$ [64], where $\ell \text {+jets}$ is a final state with exactly one electron or muon originating from the decay of one of the top quarks, including electrons and muons from tau lepton decays

Out of all possible ${\mathrm {t}\overline{\mathrm {t}}}$ hypotheses only one is chosen, based on the smallest value of $\chi ^2$, defined as

$$\begin{aligned} \chi ^2 = \left[ \frac{M_{\text {lep}} - {\overline{M}}_{\text {lep}}}{\sigma _{M_{\text {lep}}}} \right] ^2 + \left[ \frac{M_{\text {had}} - {\overline{M}}_{\text {had}}}{\sigma _{M_{\text {had}}}} \right] ^2, \end{aligned}$$

(2)

where $M_{\text {lep}/\text {had}}$ is the invariant mass of the reconstructed leptonic/hadronic top quark, and ${\overline{M}}_{\text {lep}/\text {had}}$ and $\sigma _{M_{\text {lep}/\text {had}}}$ are the average mass and resolution, respectively, of reconstructed top quark candidates in simulation. The quantities ${\overline{M}}_{\text {lep}/\text {had}}$ and $\sigma _{M_{\text {lep}/\text {had}}}$ are determined from ${\mathrm {t}\overline{\mathrm {t}}}$ simulation by fitting each of the reconstructed top quark mass distributions with a Gaussian distribution. The $\mathrm {b}$ quark of the leptonic top quark decay from simulation is required to match the assigned reconstructed jet within $\varDelta R (\mathrm {b},\, \mathrm {j}) <0.4$, whilst the jets used to reconstruct the hadronic top quark are required to match with the quarks from the hadronic top quark decay from simulation within $\varDelta R (\mathrm {q},\, \mathrm {j}) <0.4$. The distribution of the smallest $\chi ^2$ discriminator in each event is shown in Fig. 3. The $\chi ^2$ discriminator tends to zero for well-reconstructed ${\mathrm {t}\overline{\mathrm {t}}}$ systems, and to higher values for poor quality reconstructions and background events. In events where only one top quark is well-reconstructed, the $\chi ^2$ peaks at values of $\chi ^2 \approx 120$. From optimisation studies, it was found that requiring events in the signal region to have $\chi ^2 < 50$ ensured the best sensitivity.

Finally, $M_{\mathrm {Z}'}^{\text {rec}}$ is calculated by summing all four momenta of the chosen ${\mathrm {t}\overline{\mathrm {t}}}$ hypothesis along with that of the tagged $\mathrm {Z}/\mathrm {W}$ or Higgs boson.

The impact of each stage of the selection requirements on the signal selection efficiency for each decay channel of the $\text {T}$ is shown in Table 2 for a signal with $M_{\mathrm {Z}'} =2.5\,\text {TeV}$ and $M_{\text {T}} =1.3\,\text {TeV}$. The lower efficiency in the $\text {T} \rightarrow \mathrm {W}\mathrm {b}$ decay channel is primarily due to the requirement placed on $\chi ^2$, since there is only one $\mathrm {t}$ quark emitted in the decay chain for this channel.

6 Background estimation

A multistep procedure is performed to ensure that the Monte Carlo (MC) simulation (Sect. 4), used to estimate the backgrounds, accurately describes the data.

We apply scale factors to the simulation to account for the measured differences between simulation and data in the mistag rates and tagging efficiencies for the $\mathrm {Z}/\mathrm {W}$, $\mathrm {H}$, and $\mathrm {t}$ taggers. The mistag rates are measured both in data and in simulation using a QCD multijet-enriched region, while the tagging efficiencies are measured in a ${\mathrm {t}\overline{\mathrm {t}}}$-enriched region. Finally, in the statistical analysis simulations are constrained using control regions in data. These are fit simultaneously with the signal regions, constraining the normalizations and shapes of the background distributions while efficiently searching for a signal.

Fig. 4

Distribution of the reconstructed $\mathrm {Z}'$ boson mass in the $\mu \text {+jets}$ channel (upper row) and $\mathrm {e}\text {+jets}$ channel (lower row) for the ${\mathrm {t}\overline{\mathrm {t}}}$-enriched control region (left) and for the $\mathrm {W}\text {+jets}$-enriched region (right). The expected signal distribution is shown for various $M_{\mathrm {Z}'}$ masses for a fixed mass $M_{\text {T}} =1.3\,\text {TeV}$ in the $\text {T} \rightarrow \mathrm {H} \mathrm {t}$ decay channel, each with a nominal cross section $\sigma (\mathrm {Z}'\rightarrow \mathrm {t}\text {T})$ of 1$\hbox {\,pb}$. The lower panel shows the ratio of data to predicted background. Here the darker grey band indicates the statistical uncertainty, whilst the lighter grey band shows the combined statistical and systematic uncertainty

The mistag rate is determined from a QCD multijet-enriched data sample where the contribution from real $\mathrm {Z}$, $\mathrm {W}$, or Higgs bosons is negligible. The mistag rate is defined as the number of AK8 jets after the tagger is applied, divided by the number of AK8 jets before the tagger is applied. The data are selected with a trigger requiring the scalar $p_{\mathrm {T}}$ sum of the jets in the event, defined as $H_{\mathrm {T}}$, to be $H_{\mathrm {T}} >900\,\text {GeV}$. The selected data events are then required to have $H_{\mathrm {T}} > 1000\,\text {GeV}$ to ensure that events are selected in a region of phase space where the trigger is fully efficient. The AK8 jets must have $|\eta | < 2.5$, $p_{\mathrm {T}} > 200\,\text {GeV}$, and $M_{\text {AK8}}^{\text {SD}}> 30\,\text {GeV}$. The uncertainties in the mistag rate scale factors receive contributions from statistical uncertainties, and from effects associated with differences in the quark and gluon compositions and in the kinematic distributions between the QCD multijet and $\mathrm {Z}/\mathrm {W}$ + jets samples. The mistag rate for the $\mathrm {Z}/\mathrm {W}$ tagger is measured both in data and in simulation, resulting in a scale factor of $1.05 \pm 0.08$ that is applied to simulation in the signal regions. The scale factor is only applied to jets that are $\mathrm {Z}/\mathrm {W}$ tagged and are not matched to a $\mathrm {Z}/\mathrm {W}$ boson at generator level, where a jet is considered matched if both quarks of the hadronic boson decay are within $\varDelta R (\mathrm {q},\, \text {tagged AK8 jet}) < 0.8$. The mistag rate scale factors for the $\mathrm {H} _{2\mathrm {b}}$ and $\mathrm {H} _{1\mathrm {b}}$ taggers are $1.15 \pm 0.18$ and $1.22 \pm 0.05$, respectively, and are applied to all MC samples (except ${\mathrm {t}\overline{\mathrm {t}}}$) that do not contain real Higgs bosons. Since the fraction of jets initiated by a $\mathrm {b}$ quark is significantly higher in the ${\mathrm {t}\overline{\mathrm {t}}}$ background than in the $\mathrm {W}\text {+jets}$ and QCD multijet backgrounds, a dedicated scale factor is calculated for the mistag rate of the $\mathrm {H} _{2\mathrm {b}}$ and $\mathrm {H} _{1\mathrm {b}}$ taggers in a sample of ${\mathrm {t}\overline{\mathrm {t}}}$ events with back-to-back topology in the $\ell \text {+jets}$ final state. These mistag rate scale factors are measured to be $1.01 \pm 0.18$ and $0.99 \pm 0.03$ for the $\mathrm {H} _{2\mathrm {b}}$ and $\mathrm {H} _{1\mathrm {b}}$ taggers, respectively. The mistag rate scale factor for the $\mathrm {t}$ tagger is measured to be $0.95 \pm 0.02$. The scale factor is only applied to jets that are $\mathrm {t}$ tagged and do not match to a top quark at generator level, where a match requires the three quarks from the hadronic top quark decay to have $\varDelta R (\mathrm {q},\, \text {tagged AK8 jet}) < 0.8$.

The efficiency scale factor for the $\mathrm {t}$ tagger is measured using the procedure described in Ref. [28] and is found to be $1.06^{+0.07}_{-0.04}$.

Table 3 List of systematic uncertainties considered in the statistical analysis, with the size of their impact, the type(s) of effect they have, and the categories they affect. The impact size of each uncertainty is based on a signal sample with $M_{\mathrm {Z}'} = 1.5\,\text {TeV}$ and $M_{\text {T}} = 1.3\,\text {TeV}$. All uncertainties affect the normalizations of the $M_{\mathrm {Z}'}^{\text {rec}}$ distributions. The ones also affecting the shapes are indicated by a tick mark. Uncertainties that affect control regions are denoted by CR, whilst those that affect signal regions are denoted by SR

The efficiency of the $\mathrm {Z}/\mathrm {W}$ tagger is measured in a ${\mathrm {t}\overline{\mathrm {t}}}$-enriched region with a back-to-back topology. One of the top quarks is required to decay leptonically into a $\mathrm {b}$ quark and a $\mathrm {W}$ boson, with the $\mathrm {W}$ boson decaying into a lepton and a neutrino, whilst the other top quark decays hadronically, resulting in a $\mathrm {b}$ quark reconstructed as an AK4 jet and a $\mathrm {W}$ boson reconstructed as an AK8 jet. This AK8 jet is used to measure the efficiency of the $\mathrm {Z}/\mathrm {W}$ tagger. The sample is selected following the same procedure as in Sect. 5.1, additionally requiring at least two AK4 jets, where each jet must have $p_{\mathrm {T}} > 30\,\text {GeV}$, $|\eta | < 2.4$, must pass the medium working point of the $\mathrm {b}$ tagging discriminator, and not overlap with the AK8 jet. It is required that the angular separation $\varDelta R$ between the leptonic top quark and the hadronic top quark is greater than $\pi /2$ in order to reconstruct $\mathrm {W}$ bosons well-separated from nearby $\mathrm {b}$ quark jets. The $\mathrm {Z}/\mathrm {W}$ tagger efficiency scale factor is then estimated with a procedure similar to that used for the $\mathrm {t}$ tagger, and is found to be $0.91 \pm 0.08$ for events in the signal region. The systematic uncertainty for the dependence of the scale factor on the choice of the fit model used to extract the boosted $\mathrm {W}$ contribution from the combinatorial ${\mathrm {t}\overline{\mathrm {t}}}$ background is estimated to be 1%. A direct measurement of the tagger efficiency scale factor using data is only possible for jet $p_{\mathrm {T}} \lesssim 200\,\text {GeV}$. For larger jet $p_{\mathrm {T}}$, the difference between ${\mathrm {t}\overline{\mathrm {t}}}$ samples simulated with two different shower and hadronization models (pythia 8 and herwig++  2.7.1 [65]) contributes an additional uncertainty to the $p_{\mathrm {T}}$ dependence of the scale factor parameterised as $4.1\% \times \ln (p_{\mathrm {T}}/200{\,\text {GeV}})$.

For the $\mathrm {H} _{2\mathrm {b}}$ and $\mathrm {H} _{1\mathrm {b}}$ taggers, the efficiency scale factor of the $\mathrm {Z}/\mathrm {W}$ tagger is used, taking into account its associated uncertainty. Since the $\mathrm {H}$ taggers do not utilise a requirement on $\tau _{21}$, they are not assigned a corresponding $p_{\mathrm {T}}$-dependent uncertainty. In addition, two further uncertainties are considered. Firstly, the uncertainty in the extrapolation of the scale factor from the $\mathrm {Z}/\mathrm {W}$ selection to the $\mathrm {H} _{2\mathrm {b}}$ and $\mathrm {H} _{1\mathrm {b}}$ selections is estimated from the difference between the two shower and hadronization models (pythia 8 and herwig++). Secondly, uncertainties in the subjet $\mathrm {b}$ tagging efficiencies are also included, as described in Ref. [39].

The main background processes that contribute to this analysis are ${\mathrm {t}\overline{\mathrm {t}}}$ and $\mathrm {W}\text {+jets}$. Two control regions, each chosen to enhance a background process, are used to constrain the production rate of these processes and reduce potential mismodelling of the event kinematic variables. The control regions are also used to verify the agreement of the simulation with data. They are based on the selection described in Sect. 5.1. In addition to this selection, we require $\chi ^2 < 50$, and the mass of the $\mathrm {H} _{2\mathrm {b}}$-/$\mathrm {H} _{1\mathrm {b}}$-/$\mathrm {Z}/\mathrm {W}$-tagged AK8 jet to be either $M_{\text {AK8}}^{\text {SD}}< 60\,\text {GeV}$ or $M_{\text {AK8}}^{\text {SD}}> 150\,\text {GeV}$. This last requirement ensures events in the control regions are not also found in the signal regions. The first control region is designed to be enriched in ${\mathrm {t}\overline{\mathrm {t}}}$ events, and is obtained by requiring at least one additional $\mathrm {b}$-tagged AK4 jet. The second control region is designed to be enriched in $\mathrm {W}\text {+jets}$ events, and is obtained by requiring no additional $\mathrm {b}$-tagged AK4 jets.

Fig. 5

Distribution of the reconstructed $\mathrm {Z}'$ resonance mass after the full selection in the $\mu \text {+jets}$ channel for the data, the expected SM background, and for the signal with different $\mathrm {Z}'$ masses for a fixed $\text {T}$ mass of 1.3$\,\text {TeV}$. In the left (right) column the results in the top tag (no top tag) category are shown. Different rows display the distributions of events accepted by different taggers as well as the signal for the respective $\text {T}$ decays: $\mathrm {H} _{2\mathrm {b}}$ tagger and $\text {T} \rightarrow \mathrm {H} \mathrm {t}$ decay (upper), $\mathrm {H} _{1\mathrm {b}}$ tagger and $\text {T} \rightarrow \mathrm {H} \mathrm {t}$ decay (middle), and $\mathrm {Z}/\mathrm {W}$ tagger and $\text {T} \rightarrow \mathrm {Z} \mathrm {t}$ decay (lower). The signal histograms correspond to a nominal cross section $\sigma (\mathrm {Z}'\rightarrow \mathrm {t}\text {T})$ of 1$\hbox {\,pb}$. The lower panel shows the ratio of data to predicted background. Here the darker grey band indicates the statistical uncertainty, whilst the lighter grey band shows the combined statistical and systematic uncertainty

Fig. 6

Distribution of the reconstructed $\mathrm {Z}'$ resonance mass after the full selection in the $\mathrm {e}\text {+jets}$ channel for the data, the expected SM background, and for the signal with different $\mathrm {Z}'$ masses for a fixed $\text {T}$ mass of 1.3$\,\text {TeV}$. In the left (right) column the results in the top tag (no top tag) category are shown. Different rows display the distributions of events accepted by different taggers as well as the signal for the respective $\text {T}$ decays: $\mathrm {H} _{2\mathrm {b}}$ tagger and $\text {T} \rightarrow \mathrm {H} \mathrm {t}$ decay (upper), $\mathrm {H} _{1\mathrm {b}}$ tagger and $\text {T} \rightarrow \mathrm {H} \mathrm {t}$ decay (middle), and $\mathrm {Z}/\mathrm {W}$ tagger and $\text {T} \rightarrow \mathrm {Z} \mathrm {t}$ decay (lower). The signal histograms correspond to a nominal cross section $\sigma (\mathrm {Z}'\rightarrow \mathrm {t}\text {T})$ of 1$\hbox {\,pb}$. The lower panel shows the ratio of data to predicted background. Here the darker grey band indicates the statistical uncertainty, whilst the lighter grey band shows the combined statistical and systematic uncertainty

Fig. 7

Observed exclusion limits at 95% $\text {CL}$ on the production cross section for various ($M_{\mathrm {Z}'}, M_{\text {T}}$) combinations for the decay channels $\text {T} \rightarrow \mathrm {H} \mathrm {t}$ (upper left), $\text {T} \rightarrow \mathrm {Z} \mathrm {t}$ (upper right), and $\text {T} \rightarrow \mathrm {W}\mathrm {b}$ (lower). The hatched area in the upper left indicates the region where the $\mathrm {Z}'\rightarrow \mathrm {t}\text {T}$ decay is kinematically forbidden, while in the lower right $\mathrm {Z}'\rightarrow \mathrm {t}\text {T}$ is suppressed by the preferred $\mathrm {Z}'\rightarrow \text {T} \text {T}$ mode. White areas indicate regions where signal samples have not been generated

Fig. 8

Exclusion limits at 95% $\text {CL}$ on the product of the cross section and branching fraction for three $\text {T}$ masses of 1.2$\,\text {TeV}$ (upper row), 1.5$\,\text {TeV}$ (lower left), and 2.1$\,\text {TeV}$ (lower right), as a function of the resonance mass. The branching fraction is defined as ${\mathcal {B}}= {\mathcal {B}}(\text {T} \rightarrow \mathrm {W}\mathrm {b}) + {\mathcal {B}}(\text {T} \rightarrow \mathrm {H} \mathrm {t}) + {\mathcal {B}}(\text {T} \rightarrow \mathrm {Z} \mathrm {t})$. Observed and expected limits are compared to the predictions from two different theory benchmark models: the $\mathrm {G}^*$ model (upper left and lower row), and the left-handed $\rho _{\mathrm {L}}$ in the $\rho ^0$ model (upper right)

Figure 4 shows the $M_{\mathrm {Z}'}^{\text {rec}}$ distribution in the control regions for the muon and electron channels, after fitting the ${\mathrm {t}\overline{\mathrm {t}}}$ and $\mathrm {W}\text {+jets}$ backgrounds simultaneously in both control and signal regions. It can be seen that there is good agreement between data and simulation. Similar agreement is found in kinematic distributions of the objects used to reconstruct the $\mathrm {Z}'$ resonance mass. Both control regions are included in the maximum-likelihood based fit described in Sect. 8. The fit estimates the size of a possible signal, whilst simultaneously constraining the background simulation normalizations using the data in the control regions.

7 Systematic uncertainties

Systematic uncertainties can affect both the normalization and the shape of the $M_{\mathrm {Z}'}^{\text {rec}}$ distributions. The uncertainties considered in this analysis are explained in the following and listed in Table 3.

For each AK8 jet, the combined statistical and systematic uncertainty in the $\mathrm {Z}/\mathrm {W}$, $\mathrm {H} _{2\mathrm {b}/1\mathrm {b}}$ and $\mathrm {t}$ tagger efficiency scale factors and misidentification rate scale factors for the $\mathrm {Z}/\mathrm {W}$, $\mathrm {H} _{2\mathrm {b}/1\mathrm {b}}$, and $\mathrm {t}$ taggers, are propagated to variations of signal and background distributions.

Uncertainties in the jet energy scale [37] have been measured as a function of $p_{\mathrm {T}}$ and $\eta$. The jet energy scale is varied within ${\pm }1$ standard deviation for AK8 and AK4 jets simultaneously. The jet energy scale uncertainty is also propagated to $p_{\mathrm {T}} ^\text {miss}$.

The uncertainty in the jet energy resolution has been measured in different $\eta$ bins [37]. This uncertainty is applied to AK4 and AK8 jets simultaneously, assessing the impact of varying their resolutions by ${\pm }1$ standard deviation. The variation is also propagated to $p_{\mathrm {T}} ^\text {miss}$.

The $\mathrm {b}$ tagging efficiencies are measured in a sample enriched with heavy-flavour jets, whilst the probability to tag a jet originating from a different flavour as a $\mathrm {b}$ quark jet (a mistag) is measured in a sample enriched with light-flavour jets. These are applied to jets in signal and background events [39]. The uncertainties in these measurements are propagated to variations of signal and background normalizations and shapes.

Data-to-simulation scale factors for muon and electron identification and trigger efficiencies are applied as a function of $p_{\mathrm {T}}$ and $\eta$. The effect of varying each scale factor by ${\pm }1$ standard deviation is studied to estimate uncertainties in the normalizations and shapes of the signal and background distributions.

The uncertainty in the integrated luminosity of the 2016 data set is 2.5% [27]. The effect of pileup is studied by comparing simulated samples where the distribution of pileup interactions is varied according to its uncertainty.

The uncertainties in the factorisation and renormalization scales $\mu _{\mathrm {F}}$ and $\mu _{\mathrm {R}}$ are taken into account for $\mathrm {W}\text {+jets}$, $\mathrm {Z} \text {+jets}$, ${\mathrm {t}\overline{\mathrm {t}}}$, and single top quark backgrounds, as well as for the signal. The uncertainty related to the choice of $\mu _{\mathrm {F}}$ and $\mu _{\mathrm {R}}$ scales is evaluated following the proposal in Refs. [66, 67] by varying the default choice of scales by the following six combinations of factors, $(\mu _{\mathrm {F}}, \mu _{\mathrm {R}}) \times (1/2, 1/2)$, (1 / 2, 1), (1, 1 / 2), (2, 2), (2, 1), and (1, 2). The maximum and minimum of the six variations are computed for each bin of the $M_{\mathrm {Z}'}^{\text {rec}}$ distribution, producing an uncertainty “envelope” that affects both normalization and shape.

For samples generated at LO and NLO, uncertainties based on the NNPDF 3.0 PDF sets [61] and PDF4LHC15 (NLO 100) [68, 69], respectively, have been evaluated using the PDF4LHC procedure [68], where the root-mean-square of 100 pseudo-experiments provided by the PDF sets represent the uncertainty envelope. For the $\mathrm {W}\text {+jets}$, $\mathrm {Z} \text {+jets}$, ${\mathrm {t}\overline{\mathrm {t}}}$, and single top quark background processes, the full uncertainty in normalization and shape due to the variations in cross section is evaluated. For signal samples, only the uncertainty in normalization and shape due to the variations in event selection and reconstruction efficiency is taken into account, and overall uncertainties in the inclusive cross section due to PDF variations are only displayed as error bands on benchmark theory model predictions.

8 Results

The final reconstructed $\mathrm {Z}'$ invariant mass distribution is shown in each of the various categories in Fig. 5 for the muon channel, and in Fig. 6 for the electron channel. The smaller number of events in the electron channel is due to the higher electron trigger threshold in comparison to the muon trigger threshold. The binning in the figures is chosen such that the statistical uncertainty of the background MC simulation in each bin does not exceed 30%, leading to some categories represented by a single bin only. As a consequence of the higher electron trigger threshold, the categories “$\mathrm {Z}/\mathrm {W}$ with top tag” and “$\mathrm {H} _{2\mathrm {b}}$ with no top tag” contain significantly fewer events than the corresponding categories in the muon channel, and are thus only represented by a single bin as well.

A binned likelihood combining the reconstructed $\mathrm {Z}'$ invariant mass distributions in all categories and channels is constructed to compare the signal and SM background hypotheses. A Poisson probability is calculated in each bin of the mass distribution for each category in each channel. The uncertainty due to the limited number of events in the templates is taken into account using a simplified Barlow–Beeston method that defines one additional nuisance parameter with a Gaussian distribution for each bin [70]. The systematic uncertainties are taken into account as nuisance parameters in the likelihood. For each systematic uncertainty that affects the shape of the reconstructed $\mathrm {Z}'$ mass, an interpolation between the nominal template and the shifted template with a Gaussian prior is performed. Systematic uncertainties that affect only the normalization are taken into account as nuisance parameters with log-normal priors. The likelihood is maximized with respect to these parameters. The parameters representing the Poisson means of the signal strength and the background processes are determined in a maximum likelihood fit to the data, using a flat prior for the signal strength.

No significant excess of events in the data over the expectation from SM backgrounds was found for the ranges of $\mathrm {Z}'$ and $\text {T}$ masses considered. A Bayesian calculation with priors known to yield good frequentist properties [64, 71, 72] is used to derive 95% $\text {CL}$ upper limits on the product of the cross section and branching fraction, $\sigma (\mathrm {p}\mathrm {p}\rightarrow \mathrm {Z}'\rightarrow \mathrm {t}\text {T}) {\mathcal {B}}(\text {T} \rightarrow \mathrm {H} \mathrm {t}, \mathrm {Z} \mathrm {t}, \mathrm {W}\mathrm {b})$, for a heavy resonance $\mathrm {Z}'$ decaying into a top quark and a vector-like quark $\text {T}$. The calculation is implemented in the Theta software package [73]. The median of the distribution of the upper limits at 95% $\text {CL}$ in the pseudo-experiments and the central 68% (95%) interval define the expected upper limit and the 1 (2) standard deviation band, respectively.

Figure 7 shows observed limits as a function of $\mathrm {Z}'$ mass, $\text {T}$ mass, and $\text {T}$ decay mode. The limits are obtained using only decays to the indicated decay mode. Limits are shown for combinations of $\mathrm {Z}'$ and $\text {T}$ masses where the decay $\mathrm {Z}'\rightarrow \mathrm {t}\text {T}$ is kinematically allowed and the decay $\mathrm {Z}'\rightarrow \text {T} \text {T}$ is kinematically forbidden.

Figure 8 shows limits on the product of the cross sections and branching fractions ${\mathcal {B}} = {\mathcal {B}}(\text {T} \rightarrow \mathrm {W}\mathrm {b}) + {\mathcal {B}}(\text {T} \rightarrow \mathrm {H} \mathrm {t}) + {\mathcal {B}}(\text {T} \rightarrow \mathrm {Z} \mathrm {t})$, for fixed ratios of the individual branching fractions. The upper row in Fig. 8 shows the limit as a function of the $\mathrm {Z}'$ mass for a fixed $\text {T}$ mass of 1.2$\,\text {TeV}$. The upper left plot compares the limit with a prediction from the $\mathrm {G}^*$ model, showing that $\mathrm {G}^*$ masses between 1.5 and 2.3$\,\text {TeV}$ are excluded by this search, for a $\text {T}$ mass of 1.2$\,\text {TeV}$. The decrease in the predicted $\mathrm {G}^*$ cross section at a mass of approximately 2$\,\text {TeV}$ is due to the custodian VLQ $\text {T} _{5/3}$ with a mass of 1$\,\text {TeV}$, such that the $\mathrm {G}^* \rightarrow \text {T} _{5/3} \text {T} _{5/3}$ decay mode then becomes kinematically viable. At a mass of 2.4$\,\text {TeV}$ there is another decrease in the predicted cross section due to the availability of the $\mathrm {G}^* \rightarrow \text {T} \, \text {T}$ decay. This has the effect of drastically increasing the width of the $\mathrm {G}^*$, and also lowering the branching ratio (and hence predicted cross section) for the decay mode $\mathrm {G}^* \rightarrow \mathrm {t}\text {T}$. In the upper right plot of Fig. 8, the limit is compared with a prediction for the left-handed $\rho _{\mathrm {L}}$ from the $\rho ^0$ model, showing that this search is not sensitive to this model. The lower row of plots in Fig. 8 shows the observed and expected limits in the context of the $\mathrm {G}^*$ model for two other $\text {T}$ masses. For a $\text {T}$ mass of 1.5$\,\text {TeV}$ (lower left), $\mathrm {G}^*$ masses between 2.0 and 2.4$\,\text {TeV}$ are excluded by this search, whilst for a $\text {T}$ mass of 2.1$\,\text {TeV}$ (lower right) this analysis is not able to exclude the model scenario.

Fig. 9

Model-independent observed exclusion limits at 95% $\text {CL}$ on the product of the cross section and branching fraction ${\mathcal {B}}= {\mathcal {B}}(\text {T} \rightarrow \mathrm {W}\mathrm {b}) + {\mathcal {B}}(\text {T} \rightarrow \mathrm {H} \mathrm {t}) + {\mathcal {B}}(\text {T} \rightarrow \mathrm {Z} \mathrm {t})$ for an example mass configuration of $M_{\mathrm {Z}'} =1.5$ $\,\text {TeV}$ and $M_{\text {T}} =1.3$ $\,\text {TeV}$ as a function of the branching fractions ${\mathcal {B}}(\text {T} \rightarrow \mathrm {H} \mathrm {t})$ and ${\mathcal {B}}(\text {T} \rightarrow \mathrm {Z} \mathrm {t})$

Finally, Fig. 9 shows the observed limits on the product of the cross section and branching fraction as a function of the branching fractions ${\mathcal {B}}(\text {T} \rightarrow \mathrm {H} \mathrm {t})$ and ${\mathcal {B}}(\text {T} \rightarrow \mathrm {Z} \mathrm {t})$ for a $\mathrm {Z}'$ mass of 1.5$\,\text {TeV}$ and a $\text {T}$ mass of 1.3$\,\text {TeV}$, demonstrating the dependence of the limit on both branching fractions.

The expected and observed limits are 30% weaker for a $\mathrm {Z}'$ width of 30% when compared to a width of 1% for the mass range $1.5< M_{\mathrm {Z}'} < 2.5\,\text {TeV}$. However for $M_{\mathrm {Z}'} > 3\,\text {TeV}$ there is no significant difference between the limits obtained with each of the widths.

9 Summary

A search for a heavy spin-1 resonance $\mathrm {Z}'$ decaying to a standard model top quark and a vector-like quark partner $\text {T}$ has been presented. The data used in this search were recorded with the CMS detector at the LHC at $\sqrt{s} = 13\,\text {TeV}$ and correspond to an integrated luminosity of $35.9{\,\text {fb}^{-1}}$. The analysis is primarily optimised to study the decay modes of the vector-like quark to a Higgs boson and a top quark ($\text {T} \rightarrow \mathrm {H} \mathrm {t}$), and to a $\mathrm {Z}$ boson and a top quark ($\text {T} \rightarrow \mathrm {Z} \mathrm {t}$), although the decay to a $\mathrm {W}$ boson and a bottom quark ($\text {T} \rightarrow \mathrm {W}\mathrm {b}$) is also considered. This is the first direct search for the decay $\mathrm {Z}'\rightarrow \mathrm {t}\text {T} \rightarrow \mathrm {t}\mathrm {H} \mathrm {t}$. No significant excess of events over the expectation from standard model backgrounds is found. Limits on the production cross section are presented for a narrow $\mathrm {Z}'$ resonance in the mass range from 1.5 to 4.0$\,\text {TeV}$ and a narrow $\text {T}$ resonance in the mass range from 0.7 to 3.0$\,\text {TeV}$. Interpretation of these limits within the context of the $\mathrm {G}^*$ benchmark model results in the exclusion of $\mathrm {G}^*$ resonance masses in the range from 1.5 to 2.3$\,\text {TeV}$ and from 2.0 to 2.4$\,\text {TeV}$, for a $\text {T}$ mass of 1.2 and 1.5$\,\text {TeV}$, respectively. The presented limits are the most stringent to date for the decay mode $\mathrm {Z}'\rightarrow \mathrm {t}\text {T} \rightarrow \mathrm {t}\mathrm {H} \mathrm {t}$.