Search for a W' boson decaying to a vector-like quark and a top or bottom quark in the all-jets final state at $\sqrt{s} =$ 13 TeV

A search is presented for a heavy W' boson resonance decaying to a B or T vector-like quark and a t or a b quark, respectively. The analysis is performed using proton-proton collisions collected with the CMS detector at the LHC. The data correspond to an integrated luminosity of 138 fb$^{-1}$ at a center-of-mass energy of 13 TeV. Both decay channels result in a signature with a t quark, a Higgs or Z boson, and a b quark, each produced with a significant Lorentz boost. The all-hadronic decays of the Higgs or Z boson and of the t quark are selected using jet substructure techniques to reduce standard model backgrounds, resulting in a distinct three-jet W' boson decay signature. No significant deviation in data with respect to the standard model background prediction is observed. Upper limits are set at 95% confidence level on the product of the W' boson cross section and the final state branching fraction. A W' boson with a mass below 3.1 TeV is excluded, given the benchmark model assumption of democratic branching fractions. In addition, limits are set based on generalizations of these assumptions. These are the most sensitive limits to date for this final state.


Introduction
New massive charged gauge bosons are predicted by various extensions of the standard model (SM) [1][2][3]. The W boson is a hypothetical heavy partner of the SM W gauge boson that could be produced in proton-proton (pp) collisions at the CERN LHC. Searches for W bosons have most recently been performed at a center-of-mass energy of 13 TeV by the CMS and ATLAS Collaborations in the lepton-neutrino [4,5], diboson [6,7], and diquark [8,9] final states. The decay of the W boson to a heavy B or T vector-like quark (VLQ) and a t or b quark, respectively, is predicted, e.g., in composite Higgs boson models with custodial symmetry protection [10][11][12]. The VLQs are hypothetical heavy partners of SM quarks that contain left-and right-handed chiralities, which transform identically under the SM gauge groups. These models stabilize the quantum corrections to the Higgs boson mass and preserve naturalness. The W branching fraction to a quark and a VLQ depends on the VLQ mass, with a maximum of 50% [13]. Searches for VLQs have been performed by CMS and ATLAS in both the single-[14-18] and pair-production [19-22] channels. This paper describes a search for the W boson in the all-hadronic final state, where the B VLQ decays into a Higgs or Z boson and a b quark, or the T VLQ decays into a Higgs or Z boson and a t quark. The separate studies of the two decay channels are referred to as the tHb analysis and the tZb analysis throughout the paper. Both the B-and T-mediated decays result in the same signature, as shown in Fig. 1.
Owing to the high W boson and VLQ masses considered, the decay products are highly Lorentz boosted, and are combined into large-radius jets with distinct substructure. The challenge is to distinguish between these events and the main background processes, which arise from events comprised of light-quark and gluon jets produced via the strong interaction, referred to as quantum chromodynamics (QCD) multijet events, and t quark pair (tt) events. These backgrounds are modeled using a combination of Monte Carlo simulation and control regions in data. The invariant mass distribution of the three-jet system (m tHb or m tZb ) is used to search for the W boson signal. The data correspond to an integrated luminosity of 138 fb −1 [23][24][25] of pp collisions at √ s = 13 TeV, recorded in the years 2016 to 2018. We consider a theoretical framework where the t quark and W boson are superpositions of elementary and composite modes, with the t quark degree of compositeness given by s L , and the mixing angle of the elementary and composite W states given by θ 2 [13]. The W boson is assumed to be produced in a Drell-Yan process, with a cross section that is inversely proportional to cot 2 (θ 2 ). Small values of cot(θ 2 ) tend to be dominated by the leptonic W boson decay mode. Large values of the s L parameter increase the relative phase space for the decay into two VLQs, whereas small s L values enhance the W diboson decays. The analysis assumes this theoretical framework as evaluated at s L = 0.5 and cot(θ 2 ) = 3, which is chosen to enhance sensitivity to the single VLQ decay channel of the W boson. The expected signal cross sections are evaluated at √ s = 13 TeV, considering W boson masses in the 1.5 to 5.0 TeV range with the assumption that the W → VLQ branching fraction is equally distributed between the tB and tT final states. For the benchmark, the VLQ branching fractions for the decays to qH and qZ are both assumed to be 50%. Additionally, we present generalizations of the relative and total branching fraction assumptions for both the W and VLQ decays.
Tabulated results are provided in the HEPData record for this analysis [26].

The CMS detector
The central feature of the CMS apparatus is a superconducting solenoid of 6 m internal diameter, providing a magnetic field of 3.8 T. Within the solenoid volume are a silicon pixel and strip tracker, a lead tungstate crystal electromagnetic calorimeter (ECAL), and a brass and scintillator hadron calorimeter (HCAL), each composed of a barrel and two endcap sections. Forward calorimeters extend the pseudorapidity (η) coverage provided by the barrel and endcap detectors. Muons are detected in gas-ionization chambers embedded in the steel flux-return yoke outside the solenoid.
Events of interest are selected using a two-tiered trigger system. The first level, composed of custom hardware processors, uses information from the calorimeters and muon detectors to select events at a rate of around 100 kHz within a fixed latency of about 4 µs [27]. The second level, known as the high-level trigger, consists of a farm of processors running a version of the full event reconstruction software optimized for fast processing, and reduces the event rate to around 1 kHz before data storage [28].
A more detailed description of the CMS detector, together with a definition of the coordinate system used and the relevant kinematic variables, can be found in Ref. [29].
The particle-flow algorithm [30] aims to reconstruct and identify each individual particle with an optimized combination of information from the various elements of the CMS detector. The energy of each photon is obtained from the ECAL measurement. The energy of each electron is determined from a combination of the electron momentum at the primary interaction vertex as determined by the tracker, the energy of the corresponding ECAL cluster, and the energy sum of all bremsstrahlung photons spatially compatible with originating from the electron track. The energy of each muon is obtained from the momentum, which is measured by the curvature of the corresponding track. The energy of charged hadrons is determined from a combination of their momentum measured in the tracker and the matching ECAL and HCAL energy deposits, corrected for zero-suppression effects and for the response function of the calorimeters to hadronic showers. Finally, the energy of each neutral hadron is obtained from the corresponding corrected ECAL and HCAL energies that are not associated with a charged hadron track.
Jets are clustered with the anti-k T algorithm [31] using the FASTJET 3.0 [32] software package with distance parameters of 0.8 (AK8 jets) and 0.4 (AK4 jets). Jet momentum is determined as the vectorial sum of all particle momenta in the jet, and is found from simulation to be within 5 to 10% of the true momentum over the entire p T spectrum and detector acceptance. Additional pp interactions within the same or nearby bunch crossings (pileup) can contribute additional tracks and calorimetric energy depositions to the jet momentum. To mitigate this effect, tracks identified to be originating from pileup vertices are discarded and an offset correction is applied to correct for remaining contributions.
The candidate vertex with the largest value of summed physics-object p 2 T is taken to be the primary pp interaction vertex. The physics objects used for this determination are the jets, clustered using the jet finding algorithm [31,32] with the tracks assigned to candidate vertices as inputs, and the associated missing transverse momentum, taken as the negative vector sum of the p T of those jets.
To account for the neutral pileup component of the AK8 jets, the pileup per particle identification (PUPPI) algorithm [33] is used, which applies weights that rescale the jet p T based on the per-particle probability of originating from the primary vertex prior to jet clustering.
Jet energy corrections are derived from simulation studies so that the average measured response of jets becomes identical to that of particle level jets. In situ measurements of the momentum balance in dijet, photon + jet, Z + jet, and multijet events are used to determine any residual differences between the jet energy scale (JES) in data and in simulation, and appropriate corrections are made [34]. Additional selection criteria are applied to each jet to remove jets potentially dominated by instrumental effects or reconstruction failures. The jet energy resolution (JER) amounts typically to 15% at 10 GeV, 8% at 100 GeV, and 4% at 1 TeV, to be compared to about 40, 12, and 5% obtained when the calorimeters alone are used for jet clustering.

Simulated samples
The tt and single t quark production backgrounds are estimated from simulation generated at next-to-leading order with POWHEG 2.0 [35][36][37][38][39][40]. The signal and QCD background events are generated at leading order using the MADGRAPH5 aMC@NLO 2.3.3 [41,42] package. Signal events are generated in the mass range from 1.5 to 5.0 TeV in 0.5 TeV increments. Events are generated using either the NNPDF 3.0 [43] or 3.1 [44] parton distribution functions (PDFs), with parton showering and hadronization simulated with PYTHIA 8.212 [45].
The simulated events are produced with either the CUETP8M2T4 [46], CUETP8M1 [47], or CP5 [48] underlying-event tunes. The CMS detector is simulated using GEANT4 [49]. All simulated events include pileup simulation and are weighted such that the distribution of the number of interactions per bunch crossing agrees with that in data assuming a total inelastic cross section of 69.2 mb.
For each W boson mass (m W ) point, three characteristic VLQ mass points are generated, corresponding to low (1/2 m W ), medium (2/3 m W ), and high (3/4 m W ) mass ratios, in order to allow the full sensitive phase space of the boosted W boson decay products to be explored.
The generated W boson and VLQ widths are chosen to be narrow (less than 10%) relative to the experiment resolution, as required by the theoretical predictions for the region of phase space under investigation [13].

Event reconstruction
The final state is characterized by three high-p T jets from the decay of the t and b quarks, and from either the Higgs or Z boson. Because the signal of interest corresponds to a high mass resonance decaying to multiple high-p T jets, data events are triggered by the presence of either a large value of H T (the scalar sum of all AK4 jet p T in the event), an AK8 jet with a large p T , or an AK8 jet with a large jet mass.
The efficiency of the trigger selection is studied using a sample of events that are triggered by at least one high-p T muon. The fraction of these events that pass the full trigger selection is defined as the trigger efficiency. The offline event selection requires that the H T is larger than 1 TeV and the maximum jet mass is larger than 140 GeV, which ensures that the trigger is close to fully efficient. The small inefficiency (less than 1%) due to the trigger selection is taken into account as an event weight when processing simulated events.
The jets from the t quark (t jet) and from the Higgs or Z boson (Higgs or Z jet) decay tend to be wide, massive, and have a distinct substructure, whereas the jet from the b quark (b jet) tends to be narrower and have a lower mass. Therefore, an AK8 jet with p T > 400 GeV is required for the Higgs or Z boson candidate, and similarly for the t quark candidate, together with an AK4 jet with p T > 200 GeV for the b candidate. The angular separation ∆R = √ (∆φ) 2 + (∆η) 2 , where φ is the azimuthal angle, between the two AK8 jets is required to be at least 1.6 to reduce the confusion of jet shapes arising from the abutment of jet boundaries, which can bias the background estimate. The AK8 jets are then selected as being consistent with a t quark or a Higgs or Z boson decay using the jet identification ("tagging") procedures defined below. The jet considered for the b quark candidate is the leading AK4 jet with a ∆R of at least 1.2 from the tagged AK8 jets. All jets are required to be within the tracker acceptance (|η| < 2.4). The high-p T requirement for the AK8 jets provides a strong reduction in the background for the selection of boosted resonances. However, for the lowest mass W boson signal hypothesis considered, it results in a small kinematic acceptance, thus this mass region would also benefit from nonboosted techniques that are beyond the scope of this paper.

Tagging t jets
For t quarks with p T > 400 GeV, the decay products, one b quark and two light quarks, can merge into a single AK8 jet. The t quark jets are identified by a novel algorithm designed to separate t jets from jets arising from light quarks and gluons using a deep convolutional neural network called imageTop [50]. The network uses the particle momentum and particle-flow identification category as an analogy to the pixel intensity and color that would be used as inputs to a conventional image-classification algorithm. Additionally, the algorithm includes b tagging information simultaneously in the training in order to identify the b quark from the t quark decay.
The version used in this analysis includes a network that is trained on QCD jets that are drawn from a mass distribution that is constrained to be identical to the top mass distribution, so as to decorrelate the discriminator from the jet mass, and is referred to as the mass-decorrelated imageTop or imageTop MD . This analysis is the first application of the imageTop MD tagger, which is the most sensitive decorrelated t tagger developed by the CMS Collaboration, and the mass decorrelation is valid over a large range of t jet mass and p T . This tagger shows an improvement of approximately a factor of six in background rejection at a constant efficiency when compared to the best t quark taggers used previously. The imageTop MD tagger is used to select t quark jets at an operating point with ≈60% efficiency and a mistag probability of ≈2% in QCD multijet events.
Owing to the decorrelated nature of the t tagging algorithm, we also apply a selection on the mass of the t jet. The modified mass-drop tagger algorithm [51], also known as the "soft-drop" (SD) algorithm [52], with β = 0 and z = 0.1, is used to calculate the mass variable m SD (t) to discriminate between t jets and the typically lighter background jets. This variable is corrected by applying AK4 jet energy corrections to the subjets identified by the algorithm. For events in the signal region, we require 140 < m SD (t) < 220 GeV, to be consistent with the reconstructed mass of a t quark.
A simulation-to-data scale factor for the t tagging requirement is extracted by fitting the SD mass distribution in a highly enriched sample of SM tt events in data and simulation. This scale factor is applied to correct differences between the top tagging efficiencies for simulated events and data. The scale factor is p T dependent with a typical value of ≈0.95 and does not vary significantly between different running periods.

Tagging Higgs jets
In the case of a highly boosted Higgs boson in the bb decay mode, the decay products tend to merge into a single jet with a mass that is consistent with that of a Higgs boson and contains two b hadron decays. The SD algorithm is used to extract the variable m SD (H) as a measure of the jet mass, but in this case, the jet is scaled using a p T -and η-dependent correction suitable for two-prong resonances below the t quark mass [53].
To be selected, the jet should fulfill the condition 105 < m SD (H) < 140 GeV, to be consistent with the reconstructed Higgs boson mass. Scale factors are used for the jet mass scale and resolution (JMS and JMR), which are derived from a fit to the SD mass distribution of the W boson in a sample enriched in lepton+jets tt events using the technique outlined in Ref. [54].
To identify the two b hadrons clustered into the merged Higgs jet, a dedicated double-b tagging algorithm (Dbtag) is used at an operating point that has an efficiency of ≈60% and a mistag probability of ≈7% for u, d, s quarks and gluon jets. This algorithm aims to identify two b quarks from the Higgs decay by using variables correlated with b hadron lifetime and mass combined using a boosted decision tree. Data enriched in QCD-produced bb and tt events are used to establish scale factors for Dbtag for signal (which is ≈0.9) and mistagged t quarks, respectively [55]. Figure 2 shows the simulated distributions of discriminants that are used for t and Higgs jet tagging in tt, QCD, and signal for the tHb analysis.

Tagging Z jets
In contrast to the Higgs boson, the Z boson has significant decay branching fractions to all five kinematically accessible quark flavors, so we require a jet energy deposition pattern that is consistent with coming from two hard partons, along with a jet mass that is consistent with a Z boson.
The N-subjettiness [56] algorithm defines the τ N variable, which quantifies the consistency of the jet energy pattern with the expectation from N or fewer hard partons. The N-subjettiness is defined as where the sum is over jet constituents. The τ N discriminant is bound between zero and one, with low values being more consistent with N or fewer partons. For a Z boson hadronic decay, the ratio of τ 2 to τ 1 (τ 21 ) is used at an operating point of ≈85% efficiency and a mistag probability of ≈20% in QCD multijet events. Scale factors used for the N-subjettiness selection are derived simultaneously with those for the JMS and JMR as discussed in Section 4.2, and have a value of ≈0.98.
The m SD (Z) is required to be within the range 65 < m SD (Z) < 105 GeV, to be consistent with the mass of a Z boson. Figure 3 shows the distributions of discriminants that are used for t and Z jet tagging in tt, QCD, and signal simulation for the tZb analysis.

Tagging b jets
The b quark from the VLQ or W boson decay is reconstructed as an AK4 jet that is required to pass the DEEPJET b tagging algorithm [57,58] at an operating point defined by a 1% lightquark and gluon misidentification probability, approximately 10% c quark misidentification probability, and approximately 80% b quark efficiency. A simulation-to-data scale factor for the b tagging requirement is used to improve the agreement between data and simulation and has a value of ≈0.95.

Event selection
We categorize events into selection regions that are either used to search for the W boson signal, predict the background in the signal region, or validate this background estimate. Event selection details are presented in Table 1. The signal region is required to contain a tight t jet, a Higgs or Z jet, and a b jet. Regions used for background estimation and validation require combinations of loosened tagger criteria.
The sensitivity of the selections has been studied both in the context of the expected cross section upper limit and of the W boson discovery potential. After identifying the t, Higgs or Z, and b jets, the W boson candidate mass is taken as the invariant mass of the three jets. Table 2 shows the signal efficiency for all samples considered.

Background estimation
The primary background is from QCD multijet production and is estimated from data. For this estimate, we use control regions that are selected with identical kinematic criteria to the signal region, but with a reduced acceptance for signal events. Table 1 and Fig. 4 define various selection regions. A transfer function TF(p T , η) is extracted from data by inverting the Higgs or Z jet candidate selection. After this inversion, TF(p T , η) is defined as the ratio of the jet p T spectrum of the t candidate in the case that it passes (region B) to the case that it fails (region A) the t tagging algorithm. The TF(p T , η) is extracted in two η ranges (central, |η| < 1.2, and forward, |η| > 1.2, regions).
The TF(p T , η) distribution is used to predict the background in the signal region. This is accomplished by defining the control region D, which has identical Higgs or Z and b jet candidate selections to the signal region, but with an inverted t jet selection. Events that pass the region D selection use TF(p T , η) as an event weight in a given t-candidate jet (p T , η) bin. These weighted events are then used to provide the QCD background estimate in the signal region. The primary assumption for this background estimate method is that the t jet substructure selection can be inverted without significantly biasing the Higgs or Z jet substructure selection.
In the TF(p T , η) extraction procedure, the tt production component derived from simulation is subtracted from data to ensure that TF(p T , η) refers only to the QCD component. Additionally, the tt contamination of the QCD background estimate in the signal region must be subtracted. This is performed by applying the QCD background estimation procedure to simulated tt events using the same TF(p T , η) as is used when extracting the QCD estimate from data. The following relations are used to extract the QCD component of the signal region: Here, the subscripts "data" and "tt" indicate that the distributions are obtained from data and tt events, respectively.
To test the validity of the background estimate in data, a series of "validation" regions are defined (F, K, and H, as defined in Fig. 4). The transfer function used for predicting the background in the K and F regions (TF v (p T , η)) is estimated from the ratio of regions E to A, whereas the transfer function for the H region is the same as for the signal region. The following rela-  Table 1.
tions are used to extract the QCD background component of the validation regions: The background validation tests in the F, K, and H regions are shown in Fig. 5. These validation regions confirm the background estimate agreement with χ 2 /ndf values of 1.7, 1.0, and 1.3, respectively for the combined fit which considers systematic uncertainties (as discussed in Section 6), where ndf is the number of degrees of freedom. The tt background component in these validation regions is subtracted using the same procedure that is used when performing the signal region background estimate. The agreement demonstrates that the t jet selection can be inverted without biasing the Higgs jet selection. Figure 5 also shows for comparison the distributions expected from a potential signal for several W mass values, assuming the medium VLQ mass hypothesis. The corresponding distributions for the low and high VLQ mass values have shapes that are similar to these, differing mainly in the normalization.
The tt background component is estimated using simulated events with an additional event weight that corrects the generator t jet p T distribution [59]. This correction reduces the disagreement between simulation and data due to a known generator-level mismodeling effect and is validated using a tt control region. The final state for this control region contains two boosted AK8 t jets, so the selections on the AK4 b jet are loosened, and both of the AK8 jets are required to pass the same imageTop MD selection as the signal region. After this selection, the composition is a nearly pure (approximately 99%) sample of hadronic tt events, demonstrating the discrimination power of the imageTop MD tagger. The QCD background estimate in this control region is performed in an analogous way to the signal region, and is estimated by weighting events with one inverted and one tight t tag by TF(p T , η). This control region is used to extract an additional correction, which is taken as a simple ratio of QCD-subtracted data to the tt simulation estimate. The uncertainty in this procedure is taken as the uncorrelated combination of the statistical uncertainty in the ratio, 100% of the QCD estimate, and the t tag scale factor uncertainty. This correction is applied to all simulated tt distributions.

Systematic uncertainties
A wide range of systematic uncertainties are considered, and organized into those that impact only the event yields, which are assumed to follow log-normal distributions [60], and those that affect the shape of the three-jet invariant mass distribution as well, which are assumed to follow Gaussian distributions. All of the systematic uncertainties are summarized in Table 3. ±1σ(p T , η) QCD tt contamination ±1σ(p T , η) QCD
The uncertainty in the tt normalization measurement (as discussed in Section 5) used to correct the estimate in the signal region accounts for the full uncertainty in the tt yield. It is calculated  separately for data collected in 2016, 2017, and 2018, and ranges from 11 to 15%. This is considered a measurement of tt uncertainties that have a large normalization effect, including top p T reweighting, PDF, and renormalization and factorization (µ R and µ F ) scale uncertainties. Uncertainties that have a pronounced shape effect on the invariant mass distribution are included separately when setting limits (as shown in Table 3).

Shape uncertainties
The uncertainty in the JES is taken into account by varying the JES within its uncertainty and observing the effect on the invariant mass distribution. The JES variation impacts the invariant mass distribution shape through a horizontal shift but also causes a normalization difference when the jet falls above or below a kinematic threshold. This uncertainty is propagated to the t jet m SD (t) estimate. The uncertainty in the JER is also taken into account by the uncertainty in the JER correction used for simulated events. This uncertainty is applied to all simulated events and has only a small impact on the final limit.
The uncertainty in the JMS and JMR is measured in a highly enriched sample of tt events containing one final-state lepton. In this sample, a fit is performed to the W boson PUPPI m SD distribution, from which the mean and width are extracted. The JMS uncertainty is estimated from the uncertainty in the shift of the W mass peak, and the uncertainty in the JMR is estimated from the uncertainty in the width. These uncertainties are applied to the Higgs or Z boson m SD estimates for all simulated events and have an ≈4% effect on the signal estimate.
The uncertainty in the τ 21 selection associated with Z jet tagging includes a constant uncertainty and a p T -dependent uncertainty based on an extrapolation to momenta beyond the kinematic region of the scale factor measurement. This uncertainty is applied to all simulated events and has an ≈15% effect.
The uncertainty used for the b tagging requirement on the AK4 jet is evaluated by varying the b tagging and b mistagging scale factors within their uncertainties [55]. These uncertainty sources are considered uncorrelated and are applied to all simulated events and result in an ≈5% effect on the signal estimate.
The uncertainty in the Dbtag method used for the Higgs jet tagging [55] selection is evaluated by varying the Dbtag scale factor by the uncertainty. The scale factor is parameterized using three regions in p T . Also evaluated is the scale factor for a Higgs boson that is mistagged as a t quark (as discussed in Section 4). The uncertainties in both the Higgs jet tagging efficiency and the mistag probability are applied to all simulated events and are treated as uncorrelated during limit setting. The Higgs boson Dbtag uncertainty has an ≈10% effect on the signal estimate.
The uncertainty in the imageTop MD selection used for t tagging is p T dependent and evaluated by varying the imageTop MD scale factor within the uncertainty. This has an ≈6% effect on the signal estimate.
The kinematic requirements ensure that the trigger is close to being fully efficient, and all simulated events are corrected using the method outlined in Section 4. The uncertainty in this correction is taken to be half of the trigger inefficiency. This uncertainty is small and is applied to all simulated events.
As mentioned in Section 3, the simulated pileup distribution is reweighted to match data using an effective total inelastic cross section. The uncertainty in this procedure is evaluated by varying the total inelastic cross section by ±4.6% [61]. The resultant uncertainty is applied to all simulated events and has only a small impact on the final limit.
The PDF uncertainty is evaluated using the NNPDF3.1 [43,44] set for the signal, and the PDF4LHC15 combined set [62,63] for single t quark events. The uncertainty is calculated using the symmetric Hessian approach.
The uncertainty in the µ R and µ F scales are evaluated by varying the µ R or µ F scales up and down by a factor of two. Both independent and simultaneous variations of µ R and µ F are considered, resulting in six weights after excluding the two most extreme variations. The envelope of these weights is used as the uncertainty. The µ R , µ F , and PDF uncertainties are taken as the signal cross section theoretical uncertainty. For the single t quark prediction, these uncertainties are considered when setting limits in addition to the parton shower variation in the initialand final-state radiation (ISR and FSR). Together, the effect of these theoretical uncertainties amount to ≈65%, and dominate the total uncertainty in the single top quark estimate.
The primary uncertainty in the shape of the QCD background estimate is taken from the statistical uncertainty in the TF(p T , η). This is propagated to the invariant mass spectrum by evaluating the TF(p T , η) weight at ±1σ in a given (p T , η) bin. The uncertainty from each TF(p T , η) bin is added in quadrature to form the full uncertainty, and the effect on the QCD background estimate is ≈20%. The effect of the uncertainty in the tt subtraction procedure is evaluated by varying the amount subtracted within the range of the tt normalization uncertainty. The effect on the QCD background estimate is ≈6%.
The statistical uncertainty of the simulation is taken into account during limit setting by using the "Barlow-Beeston lite" method [64].

Results
The final distributions of reconstructed m W for the signal region are shown in Fig. 6. The distributions expected from a potential signal for several W mass values, assuming the medium VLQ mass hypothesis, are shown for comparison. Since no deviation from the SM expectation is observed, we place limits on the W boson production cross section.
The 95% confidence level (CL) upper limits are obtained using the CL s approach [65,66] in the asymptotic approximation [67] on the product of the W boson production cross section for the s L = 0.5 and cot(θ 2 ) = 3 hypothesis [13], and for the benchmark branching fraction. A binned maximum likelihood fit to the data is used to calculate upper limits, in a process where all systematic uncertainties are included as nuisance parameters. For the signal template, the sum of the invariant mass distributions from the tB and bT decay channels are used. Upper limits on the product of cross section and branching fraction are shown in Fig. 7. A W boson with a mass below 3.1 TeV is excluded at 95% CL for the medium VLQ mass case. The low and high VLQ mass benchmarks have a lower W → VLQ branching fraction, and the sensitivity is not sufficient to set mass exclusion limits.
Variations of the primary assumptions of this benchmark are shown in Fig. 8 for the purpose of generalizing the results. Specifically, the fraction of bT and tB from the W decay is varied between 0 and 1, and the VLQ branching fractions to qH and qZ are varied within the same bounds. Therefore, for both generalized coupling figures, the mass exclusion of the benchmark model corresponds to the point with coordinates [0.5, 0.5], indicated with an asterisk.   Figure 8: Expected (upper) and observed (lower) 95% CL limits for generalized hypotheses varying the fraction of tB (F(VLQ=B)) and bT (F(VLQ=T)) from the W decay (left), and the VLQ branching fraction to qH and qZ (right). The asterisk marker signifies the branching fractions for the benchmark model

Summary
A search has been presented for a heavy W boson decaying to a B or T vector-like quark and a t or b quark, respectively. The data correspond to an integrated luminosity of 138 fb −1 collected between 2016 and 2018 with the CMS detector at the LHC in proton-proton collisions at √ s = 13 TeV. The signature considered for both decay modes is a t quark and a Higgs or Z boson, each decaying hadronically, and a b quark jet. Boosted heavy-resonance identification techniques are used to select the events containing three energetic jets and to suppress standard model backgrounds. No significant deviation from the standard model background prediction is observed. Upper limits are placed on the product of the W boson cross section and the final state branching fraction as a function of the m W . A W boson with a mass below 3.1 TeV is excluded at 95% confidence level, given the benchmark model assumption of democratic branching fractions. In addition, limits are set based on generalizations of these assumptions. These are the most sensitive limits to date for this final state.