Collider constraints and new tests of color octet vectors

We analyze the collider sensitivity for new colored resonances in $t\bar{t}$, $b \bar{b}$, and $jj$ final states. While searches in the single production channel are model-dependent, the pair production rate is model independent and the existing $(JJ)(JJ)$ and $4t$ searches impose strong constraints on the relevant branching fractions, where $J = j$ or $b$. We point out the missing, complementary searches in the mixed decay modes, $t\bar{t}(jj)$, $t\bar{t}(b\bar{b})$, and $(b\bar{b})(jj)$. We propose analysis strategies for the $t\bar{t}(jj)$ and $t\bar{t}(b\bar{b})$ decays and find their sensivity surpasses that of existing searches when the decay widths to tops and light jets are comparable. If no other decays are present, collective lower limits on the resonance mass can be set at 1.5~TeV using 37~fb$^{-1}$ of 13~TeV data.


I. INTRODUCTION
Searches for new particles and new forces beyond the Standard Model (BSM) are a critical endeavor for the ATLAS and CMS experiments at the Large Hadron Collider (LHC), but such results have thus far come up null. The largest rates come from colored particle pair production, where leading order cross sections can be calculated knowing only the color charge and mass [1].
Even so, the colored particle decay patterns and corresponding collider signatures are highly model dependent.
Given the possibility that the color octet vector has flavor dependent branching fractions to quark pairs, and because color octet vectors have a model independent pair production rate, the mixed decay signature of a paired ditop and dijet resonance is strongly motivated. This channel is complementary to the existing (JJ)(JJ) and 4t searches and even offers superior sensitivity when the branching fractions of the resonance to dijets and ditops are comparable.
In Sec. II, we review the theory motivation and collider phenomenology of massive color octet vectors. In Sec. III, we analyze the prospects for discovering color octet vector resonances in the tt(jj) and tt(bb) mixed decay modes at the LHC and compare with the current constraints. We conclude in Sec. IV.

II. THEORY BACKGROUND
Massive color octet vector resonances arise in various different beyond the Standard Model extensions, such as models with extended color gauge groups or models with extra dimensions. In the unbroken phase of electroweak symmetry, the general interaction Lagrangian between quarks and a massive color octet vector X µ is L ⊃ g s Q L γ µ λT a X µ a Q L +ū R γ µ κT a X µ a u R +d R γ µ T a ηX µ a d R , where g S is the strong coupling constant, T a are the SU (3) generators, and λ, κ, and η are the flavor-dependent couplings in the quark gauge basis. While these matrices must be symmetric by CP T , nonzero off-diagonal entries are possible in principle and correspond to tree-level flavor violation. Since these processes are strongly constrained by low-energy precision flavor measurements, such as meson mixing measurements and b → X s γ transitions [49,50], the simplest ansatz is to adopt a flavor-universal coupling structure, evading the most stringest flavor bounds. In such a scenario, however, the branching ratio for X to light quarks vs. tops is fixed, and there is no model freedom in the complementarity between searching in dijet vs. ditop resonances. Hence, we will focus on the scenario where the λ, κ, and η matrices are diagonal but not universal in the quark gauge basis. Moving to the quark mass basis will then induce off-diagonal entries proportional to Cabibbo-Kobayashi-Maskawa (CKM) mixing. We will first briefly review models of massive color octet vectors and then discuss the collider and flavor physics phenomenology of the general Lagrangian, Eq. (1).

A. Color octet vectors from extended color gauge groups
In models with an extended color symmetry, such as a SU (3) 1 × SU (3) 2 gauge group, the two parent gauge groups are typically broken to the diagonal subgroup by the vacuum expectation value (vev) of a bifundamental, complex scalar field Σ [3,8]. The diagonal subgroup is then identified with the SM SU (3) c gauge group, which imposes the requirement where h 1 and h 2 are the gauge couplings of SU (3) 1 and SU (3) 2 , respectively. The Goldstone modes of the complex scalar field become the longitudinal degrees of freedom for the heavy color octet vector, leaving one real and one pseudoreal color singlet scalar and one real color octet scalar as the dynamical scalar fields below Σ . Explicitly, the covariant derivative for Σ is and after Higgsing SU (3) 1 × SU (3) 2 → SU (3) c , we get the SM gluon and coloron fields, In the coloron model [3,8], all SM quarks transform as under one SU (3) gauge group and singlets under the other. Hence, the coloron has flavor universal, purely vector couplings to the SM quarks as g s tan θqγ µ T a X a µ q .
An alternative prescription is the chiral color model [4,9,51], where left-handed (LH) and right- where X a µ is commonly referred as an axigluon in the literature. To motivate a non-universal yet diagonal coupling structure in Eq. (1), we can straightforwardly assign different quark flavors to different gauge representations under SU For example, the topcolor model charges the third generation quarks differently than the first two This assignment also requires additional matter to cancel anomalies, for which a minimal solution involves two electroweak singlet quarks transforming as (1, ) and ( , 1), each with hypercharge −2/3. The corresponding massive color octet vector does not have flavor-changing couplings in the quark gauge basis, but instead features distinct couplings to light and heavy generation quarks: with tan θ = h 1 /h 2 as before. Although topcolor models are generally motivated by composite Higgs scenarios triggered by top quark condensation [2], we will only focus on the motivated possibility that X µ has non-universal couplings.

B. Color octet vectors from extra dimensions
Models with extra spacetime dimensions provide an alternative framework for realizing massive color octet vector resonances [11][12][13]. In such models, the SM fields are the lowest-lying states of a Kaluza-Klein tower, whose masses and dynamics result from solving the five-dimensional equations of motion [16][17][18][19][20]. In minimal universal extra dimensions [52], the level-2 KK gluon obtains a coupling to SM quarks from one-loop diagrams with level-1 KK particles running in the loop. These couplings are generated from boundary conditions on the KK gluon and the bulk masses, which provide the only source of translational invariance breaking, and read where Y u = 2 for up-type quarks, Y d = 1/2 for down-type quarks, Λ is an ultraviolet scale larger than the inverse size of the extra dimension, and µ is the renormalization scale to evaluate the coupling.
In Randall-Sundrum warped scenarios [14,15], the fermion mass hierarchy can be explained by allowing fermions to propagate in the bulk, where the observed charged fermion mass hierarchy originates as O(1) differences in bulk mass parameters. Typically, the KK mass scale must then be O(5 − 10 TeV) to satisfy low-energy flavor violation probes, especiallyK − K mixing [24], but this scale can be lowered in the case that the bulk fermions obey a flavor symmetry [53,54]. Since KK parity is absent, the first KK gluon decays to SM zero-mode quarks, and this coupling is given by calculating the wavefunction overlap between the zero modes and the KK gluon in the extra dimension. Using the general coupling structure in Eq. (1) and identifying X µ with the first KK gluon, where RH up-type couplings are obtained by replacing λ → κ and Q → u, RH down-type couplings are obtained by replacing λ → η and Q → d, m X ≈ 2.4M KK is fixed by the boundary conditions of the 5D gluon, L is the length of the extra dimension, and c Q , c u , and c d are the bulk mass parameters for LH and RH quark fields [25]. In order to reproduce the known SM quark masses, these bulk mass parameters must be chosen to maximize the top quark wavefunction overlap with the TeV-scale infrared brane while the wavefunctions for the light quarks are skewed towards the ultraviolet brane. As a result, the first KK gluon preferentially decays to top quarks, with branching fractions that can exceed 80% [25].
We see that these constraints are easily satisfied for O(TeV) scale color octets as long asλ ∼ O(0.1).
Moreover, with this structure,λ 11 =λ 22 , the stronger constraints come fromB 0 TeV and |C 1 Bs | requires Λ > 240 TeV. Using the tree-level, CKM-induced off-diagonal elements ofλ D , we obtain which are again weakened to the sub-TeV scale forλ ∼ O(0.1). As a result, we can realistically expect O(TeV) color octet vectors with dominant branching fractions to either tops,λ 33 λ 11 or jets, λ 11 λ 33 , entirely consistent with flavor bounds. We remark that an alternative assumption of the flavor structure can relax these bounds further. In particular, if instead V † CKMλ V CKM is diagonal, then the tree-level flavor violating couplings are shifted to the LH up-quark couplings [57,58]. We can defineλ = V † CKMλ V CKM as diagonal, and then the LH up-quark couples via V CKMλ V † CKM , which is constrained by D 0 −D 0 mixing, where Λ > 700 TeV [49]. The corresponding bound is The ∆F = 1 constraints, including b → X s γ and b → X s g penguin amplitudes, are also immediately satisfied if the X µ couplings are aligned in the down-quark sector [50,58].

D. Collider phenomenology of color octet vectors
Color octet vectors can be singly and doubly produced at colliders. The single-production rate scales with the corresponding partial width into qq, q = d, u, s, c, since single production via gluons is forbidden at tree-level. In our scenario, the induced flavor-violating couplingsλ D are at most 5% ×λ 11 ,λ 33 , which we will neglect and hence approximateλ D ≈λ. The partial width of X to a pair of quarks q is where g L and g R are the corresponding diagonal entries inλ,κ, andη 1 . If the masses of the quarks can be neglected compared to the X mass, the partial width simplifies to 1 For reference, the generalized flavor violating partial width is where g ij L = g ji L and g ij R = g ji R .
The branching fractions for jj, bb, and tt final states are then simply ratios of the corresponding sum of squared couplings. We see that large branching fractions to top quarks can easily be realized by increasingκ 33 , while large branching fractions to bottom quarks corresponds to increasingη 33 . This is the effective description of the warped extra dimension scenario with t R and b R wavefunction profiles peaked close to the infrared brane [25], and small hierarchies in these third generation couplings are consistent with electroweak precision tests [57,58].
As previously highlighted, the single dijet resonance constraints scale with the overal partial width into light flavor quarks. Pair-production rates, however, are robustly calculable knowing only m X and its color octet representation. Hence, all searches in pair-production modes provide   [65]. We rescale the background by a flat K-factor of 1.5, adopted from Sherpa+BlackHat [65,66]. The inclusive SM tt + jets production cross section is known at next-to-next-to-leading order and next-to-next-to-leading log (NNLL) precision [67], whereas differential cross sections can be obtained at next-to-leading order and NNLL precision [68]. Since our X resonance should appear as a resonant peak over a smooth continuum background, a p T -dependent K-factor should not significantly affect our projected results.

A. Event selection strategy
We first discuss the semi-leptonic top decay channel, where our signal process is XX * → bb ± νjj(JJ), with J = j or b. Jets are clustered using the anti-k T algorithm [69] with R = 0.5, and we require jets to have p T > 50 GeV and |η| < 4.9. We identify b-jets using a Delphes v3.1.2 [70] detector simulation, with a p T -and η-dependent tagging efficiency of about 70% for displaced tracks from B-mesons within ∆R < 0.3 from the main jet axis; the charm misidentification rate is roughly 15% and the light flavor mistag rate is 0.1%.
Events must have at least 5 jets, at least two of which must be b-tagged and one must be untagged. The leading b-tagged and untagged jets must have p T ≥ 250 GeV, and the subleading untagged jet, if present, must have p T ≥ 80 GeV. Events must also have exactly one isolated lepton with p T > 20 GeV and |η| < 2.5, with isolation using Delphes default parameters. We furthermore cut on missing transverse energy (MET), requiring / E T > 80 GeV, as well as signal mass-optimized H T = Σ j |p T,j | ≥ (4/3) m X .
Naively, the X → JJ signal resonance is extracted from forming the invariant mass of the two leading jets. The jet combinatorics, however, present a major hurdle against using the dijet and ditop masses to discriminate the signal from the irreducible background. Thus, the main goal of our collider strategy is to solve the combinatorial ambiguity, taking advantage of the resonant high mass dijet signal, the large p T of the resonant dijet system, and the relevant angular spread between the b-jets, lepton, and jets. Difficulties in resolving multijet combinatorial ambiguities are discussed in, e.g., [71].
For this purpose, we define two signal regions. We target the bb mode by selecting events with more than two b-tagged jets, while the jj mode is optimized by requiring exactly two b-tagged jets.
For the bb decay, we assume that the leading b-tagged jet comes from the X → JJ decay directly, not from tt. We then reconstruct the bb system by adding the remaining leading jet, whether tagged or untagged. We find that this reconstruction best reflects the dijet resonance in spite of the underlying combinatorial ambiguity.
For the jj mode, we start with the leading light jet j 1 and add to it the hardest light jet j 2 which satisfies ∆R j 1 j 2 ≤ π. We also add in the next hardest light jet j 3 with ∆R j 1 j 3 ≤ π if j 1 and j 2 are not balanced in p T , when p T,j 2 /p T,j 1 ≤ 0.15. This follows the hemisphere intuition, where the X → jj decays should be untagged and relatively hard. Adding the third hardest light jet accounts for the wide-angle final state radiation of our signal quarks.
For both bb and jj decay modes, we construct the dijet invariant mass as our final kinematic discriminant. The signal and background cut efficiencies for the semileptonic analysis are shown in Table I, while the most salient kinematic distributions for the background stacked with different signal hypotheses are shown in Fig. 1. The upper left panel in Fig. 1 shows the hardening of the H T distribution coming from the signal jets. The dijet p T,JJ distribution, however, only offers an overal rate shift from the additional signal events and no strong correlation with the signal X mass, which is because of the combinatorial ambiguity among the jets. The bottom panels show the invariant mass distributions in the jj and bb targetted modes. Again, the broad peak structure arises mostly from the combinatorial ambiguity in capturing the correct signal jets to reconstruct the resonance.
We note that b-tagging efficiency and the combinatorial ambiguity also cause X → jj events to populate the m bb signal region and vice versa.
The search in the fully leptonic top quark decay channel is very similar. We again select jets and leptons as in the semi-leptonic channel, but we require only 4 jets, at least two of which are b-tagged, and also require exactly two isolated leptons of opposite charge. We loosen the MET cut, / E T > 50 GeV, and keep the H T cut, H T ≥ (4/3) m X . To avoid the anticipated Z+jets background we veto events with m ≤ 115 GeV. Again we distinguish between bb and jj dijet resonance decay modes, using the same method as the semi-leptonic search. The corresponding cut flow is presented in Table II. The final signal rates lead to the same relative signal to background ratios as the semi-leptonic analysis, but the absolute rate is only an ≈ 5% additional contribution compared to the semi-leptonic channel.

B. Comparison with current searches
We now compare the projected sensitivity from the mixed tt(JJ) searches in combination with the recasted exclusions from ATLAS and CMS for (JJ)(JJ) and 4t searches. We also show the single production limits for dijet and ditop resonance searches. Since we assume that our new physics state X only decays to quark pairs, we can express the pair production constraints in the branching fraction vs. mass plane, where Br(X → tt) = 1 − Br(X → JJ) .
So, the pair production limits can be translated according to: with Br JJ = Br(X → JJ). The 4t constraint and the tt(JJ) projected exclusion after replacing (Br JJ ) 2 with (Br tt ) 2 and 2(Br JJ Br tt ), respectively, where Br tt = Br(X → tt).
The single dijet resonance limits are determined only after specifying the total width of the resonance. If the total width is narrow, a reference cross section can be rescaled by the partial width into light quarks. We see that In the case of tt resonances, the above constraint becomes a bounded requirement on the branching fraction to tops. Hence, the tt resonance constraint follows Note that here we drop the acceptance factor since the experiments unfold the acceptance when presenting their results. We combine all four orthogonal signal regions of the semi-leptonic and fully leptonic search and compute the 95% C.L. limits based on the respective JJ invariant mass shapes, where we assume a 10% systematic uncertainty on signal and background. The result can be found in Fig. 2.
We remark that the width of our resonance, Γ X = 5 × 10 −4 m X , corresponds to diagonal entries ofλ,κ,η 0.1. As discussed in Subsec. II C, such LH quark couplings to X readily satisfy flavor violation bounds from meson oscillation measurements and neutral current transitions given m X > 1 TeV. On the other hand, these couplings generally arise from mixing the SM quarks with heavy vectorlike states, since otherwise perturbativity in the parent extended color gauge symmetry is violated [48]. The corresponding flavor violation bounds, collider constraints, electroweak precision observable tests are more model-dependent, but realistic and complete scenarios can be constructed [50,57,58].
As mentioned previously, the sensitivity from single production compared to the sensitivity from pair production strongly depends on the ratio Γ X /m X . Any direct resonance bounds from JJ searches could in principle be evaded completely with a suitably small choice of Γ X , but choosing Γ X /m X = 5 × 10 −4 allows complementary dijet constraints from CMS and ATLAS [39,40]. We remark that the tt resonance limits [47] are absent from Fig. 2 for Γ X /m X = 5 × 10 −4 . These constraints are only relevant once Γ X /m X 7 × 10 −4 for m X around 1 TeV. As shown in Eq. (26), the tt resonance limit is symmetric around Br tt = 50% since the maximum rate in this channel corresponds to equal partial widths to dijets and ditops.
The strongest existing bounds in Fig. 2  In Fig. 2, we assume that all quark couplings are flavour universal except for the top. We relax this assumption in Fig. 3 the current searches for pair production of the 125 GeV Higgs [72,73], where the main novelty would be varying the (bb) mass window to test for new resonances. The multijet background, however, is very challenging to simulate and thus substantial statistics in the b-tagged backgrounds would be required to suitably smooth paired invariant mass spectrum in the (bb)(bb) and (bb)(jj) channels.

IV. CONCLUSION
In this work, we have highlighted the tt(jj) and tt(bb) mixed decay channels of a massive, color octet vector as new targets for ATLAS and CMS searches. Hierarchies in the underlying couplings of the X resonance to light quarks, bottom pairs, or top pairs are entirely consistent with low energy FCNC constraints if the X mass is above 1 TeV and its couplings to quarks are at most 0.1. As a result, the LHC provides the leading reach to TeV-scale color octet vectors with variable couplings to heavy and light flavor quarks by virtue of the model independent pair production rate.
In our tt(JJ) analyses, we focused on resolving the jet combinatorial ambiguity to reconstruct the dijet or dibottom resonance. In principle, reconstructing the (tt) resonance is also possible, but our scenario with its many resolved jets did not afford any additional signal discrimination in this regard.
Nevertheless, our results show that new tt(jj) and tt(bb) searches will fill a sensitivity gap between the (JJ)(JJ) and 4t searches. This gap is clear in the Br JJ vs. m X plane, which itself provides a useful tool for easily presenting the results from different collider searches of pairproduced resonances. We stress that a post-discovery scenario of a new resonance greatly benefits  [30] The ATLAS collaboration (ATLAS), "A search for pair produced resonances in four jets final states in proton-proton collisions at √ s=13 TeV with the ATLAS experiment," (2016).
[31] The ATLAS collaboration (ATLAS), "A search for pair-produced resonances in four-jet final states at √ s = 13 TeV with the ATLAS detector," (2017).