Hunting for Vectorlike Quarks

We analyze decays of vectorlike quarks in extensions of the standard model and a two Higgs doublet model. We identify several typical patterns of branching ratios of the lightest new up-type quark, $t_4$, and down-type quark, $b_4$, depending on the structure of Yukawa couplings that mix the vectorlike and standard model quarks (we assume only mixing with the third generation) and also on their doublet or singlet nature. We find that decays into heavy neutral or charged Higgs bosons, when kinematically open, can easily dominate and even be close to 100\%: $b_4\to H b$ at medium to large $\tan \beta$, $t_4\to H t$ at small $\tan \beta$ and $b_4\to H^\pm t$, $t_4\to H^\pm b$ at both large and small $\tan \beta$. The pair production of vectorlike quarks leads to $6 t$, $4t 2b$, $2t4b$ and $6b$ final states. The decay modes into $W$, $Z$ and $h$ follow the pattern expected from the Goldstone boson equivalence limit that we generalize to scenarios with all possible couplings. We also discuss in detail the structure of Yukawa couplings required to significantly deviate from the pattern characteristic of the Goldstone boson equivalence limit that can result in essentially arbitrary branching ratios.


Introduction
Models with extra Higgs bosons or vectorlike quarks and leptons are among the simplest extensions of the standard model (SM). Consequently many strategies have been designed to search for them at collider experiments [1][2][3][4][5][6][7][8][9][10][11][12][13][14]. The success of this effort depends on understanding the decay patterns of new Higgs bosons or vectorlike matter. It is often assumed that only one particle or a specific coupling of a new particle is present. However, if two new particles are present, or more than one coupling of a new particle is sizable, the decay patterns can be dramatically altered.
We analyze decays of new quarks in extensions of the standard model and a two Higgs doublet model (type-II) by vectorlike pairs of new quarks (VLQ), corresponding to a copy of SM quark SU (2) doublets and singlets and their vectorlike partners. We identify several typical patterns of branching ratios of the lightest new up-type quark, t 4 , and down-type quark, b 4 , depending on the structure of Yukawa couplings that mix the vectorlike and standard model quarks and also on their doublet or singlet nature. We assume only mixing with the third generation of SM quarks, nevertheless the results can be straightforwardly generalized for cases of mixing with the first or second generation.
We find that decays into heavy neutral (H) or charged (H ± ) Higgs bosons, when kinematically open, can easily dominate and even be close to 100%: b 4 → Hb at medium to large tan β, t 4 → Ht at small tan β and b 4 → H ± t, t 4 → H ± b at both large and small tan β. Thus, the pair production of vectorlike quarks leads to 6t, 4t2b, 2t4b and 6b final states (and similar final states for single production). The SM backgrounds for these final states (at large invariant mass) are very small and thus searching for these processes could lead to the simultaneous discovery of a new Higgs boson and a new quark.
The usual decay modes into W , Z and the SM Higgs boson, h, cluster around the pattern expected from the Goldstone boson equivalence limit (GBEL), corresponding to sending all vectorlike quark masses to infinity, that we generalize to scenarios with all possible couplings. For singlet-like new quarks this leads to 2:1:1 branching ratios into W , Z and h. For doublet-like new quarks this leads to a one parameter family of branching ratios characterized by an arbitrary branching ratio to W and equal branching ratios to Z and h. We also discuss in detail the structure of Yukawa couplings required to significantly deviate from the pattern characteristic of Goldstone boson equivalence limit that can result in essentially arbitrary branching ratios.
Extensions of the SM, two Higgs doublet models or the minimal supersymmetric model (MSSM) with vectorlike matter were previously explored in a variety of contexts. Examples include studies of their effects on gauge and Yukawa couplings in the framework of grand unification [15][16][17][18][19][20][21][22][23] and on electroweak symmetry breaking and the Higgs boson mass [24][25][26]. The supersymmetric extension with a complete vectorlike family provides a very sharp prediction for the weak mixing angle [22,27] and also a possibility to understand the values of all large couplings in the SM from the IR fixed point structure of the renormalization group equations [28]. In addition, vectorlike fermions are often introduced on purely phenomenological grounds to explain various anomalies. Examples include discrepancies in precision Z-pole observables [29][30][31][32] and the muon g-2 anomaly [33][34][35] among many others.
In this paper we focus on vectorlike quarks with the same quantum numbers as the quarks in the SM. Examples of signatures of related scenarios with vectorlike leptons can be found in Refs. [36][37][38][39][40][41]. For related studies and especially for studies of decay modes and signatures in scenarios with different quantum numbers of vectorlike matter see also Refs. [42][43][44][45][46][47][48][49][50][51][52] and references therein. This paper is organized as follows. In Sec. 2, we outline the model and assumptions. Details of the analysis and experimental constraints are discussed in Sec. 3. The main results and their discussion are contained in Sec. 4 and we conclude in Sec. 5. The appendix contains details of the model, formulas for couplings and relevant partial widths, and approximate formulas that are useful to understand the results.

Model
We consider an extension of a two Higgs doublet model by vectorlike pairs of new quarks: SU(2) doublets Q L,R and SU(2) singlets T L,R and B L,R . The quantum numbers of new particles are summarized in table 1. The Q L , T R and B R have the same quantum numbers as the SM quark doublet q L and the right-handed quark singlets u R and d R , respectively. We further assume that quarks couple to the two Higgs doublets as in the type-II model, namely the down sector couples to H d and the up sector couples to H u . This can be achieved by the Z 2 symmetry specified in table 1. The generalization to the whole vectorlike family of new fermions, including the lepton sector (which has been studied in ref. [37]), is straightforward. The most general renormalizable Lagrangian consistent with Table 1. Quantum numbers of standard model quarks (q i L , u i R , d i R for i = 1, 2, 3), extra vectorlike quarks and the two Higgs doublets. The electric charge is given by Q = T 3 + Y , where T 3 is the weak isospin, which is +1/2 for the first component of a doublet and -1/2 for the second component. our assumptions contains the following Yukawa and mass terms for the SM and vectorlike quarks: where the first term represents the Yukawa couplings of the SM down-type quarks, followed by Yukawa couplings of vectorlike quarks to H d (denoted by various λs), Yukawa couplings of the SM up-type quarks, Yukawa couplings of vectorlike quarks to H u (denoted by various κs), and finally by mass terms for vectorlike quarks. Note that the explicit mass terms mixing SM and vectorlike quarks, can be removed by redefinitions of Q L , T R , B R and the Yukawa couplings. The components of doublets are labeled as follows: We assume that the neutral Higgs components develop real and positive vacuum expectation values, H 0 u = v u and H 0 d = v d , as in the CP conserving two Higgs doublet model GeV and we define tan β ≡ v u /v d . For simplicity, we further assume that the new quarks mix only with one family of SM quarks and we consider the mixing with the third family as an example; the mixing of new quarks with more than one SM family simultaneously is strongly constrained by various flavor changing processes and we will not pursue this direction here. In the basis in which the SM quark Yukawas are diagonal, the mass matrices describing the mixing between the third generation and the vectorlike quarks are (see the appendix): Note that the corresponding mass matrices in the case of a single Higgs doublet can be obtained by setting v u = v d = 174 GeV. A complete discussion of the mass eigenstates and of their couplings to the W , Z, and Higgs bosons can be found in the appendix.

Parameter space scan and experimental constraints
We study the branching ratio patterns that can be obtained in the model by varying the relevant parameters as follows: Note that the upper range of the couplings has no impact on presented results as long as it is common for all couplings. We impose the experimental constraints from precision electroweak measurements [53] 1 , h → (γγ, 4 ) [55, 56] 2 and direct searches for vectorlike quarks pair produced at the LHC [1][2][3]60]. For the latter constraints, we directly use the data points from hepdata.net [61] obtained from Ref. [2] where the limits in terms of mass of vectorlike quarks and the branching ratios into W, Z and the SM Higgs boson are similar to the other search results.
Note that searches for the single production of VLQ via t-channel quark gluon interactions also exist but the experimental constraints are not stronger than those for the pair production of VLQ [8,62] although the production cross section can be larger than that of the pair production for VLQ heavier than ∼ 800 GeV [44]. Hence, we do not consider the corresponding experimental bounds here but leave it to a future work [63]. In scenarios where the lightest VLQ can decay into heavy neutral or charged Higgs boson, the currently existing searches for VLQ, focussed on decay modes into W , Z, and h, do not directly apply.
In the model we are considering we expect potentially large contributions to B → X s γ. While the charged Higgs loop is suppressed for Higgs masses above 1 TeV, the b R W t R 1 See also Ref. [54] for a detailed discussion about imposing constraints from precision electroweak measurements in models with vectorlike quarks.
, yielding contributions which can be at most 1.5% for vectorlike quarks heavier than 1 TeV.
Finally we note that in our framework there are no flavor changing neutral interactions in the Higgs nor Z sector. Therefore, tan β enhanced tree-level Higgs contributions to B d,s → + − , which are present in supersymmetric models with loop-induced nonholomorphic Higgs couplings, are absent.

Results
The main features of the decays of vectorlike quarks can be understood from the dominant couplings that appear in table 2, which can be easily read from the approximate expressions presented in the appendix. An important quantity that controls the decay of the lightest vectorlike quark is its doublet or singlet fraction, which add up to unity and, for up-type vectorlike quarks t a (a = 4, 5), are defined respectively as The doublet and singlet fractions for down-type vectorlike quarks b a (a = 4, 5) are obtained by replacement V u L,R → V d L,R . Note that in the absence of couplings to H u (H d ), i.e., in the case when all λ's (κ's) in (2.1) vanish, the doublet fraction of t 4 (b 4 ) defined in eq. (4.1) is exactly 1 or 0. The latter case corresponds to a vectorlike singlet quark which is completely uncoupled (and thus of no interest for our study).  In the next two subsections we discuss decays into heavy Higgses and then dwell into the intricacies of standard decays into W , Z and h in situations in which subdominant couplings become important.

Decays into heavy Higgses
In figure 1 we present the branching ratios of vectorlike quarks into heavy CP-even neutral and charged Higgses that we obtain from the parameter space scan described in section 3. For simplicity, we assume that only one (either charged or neutral heavy Higgs) decay channel is kinematically open. 3 In all panels orange (green) points correspond to allowing only couplings to H u (H d ); gray points show the additional branching ratio values that we obtain by allowing simultaneous couplings to H u and H d .
The interpretation of these plots follows immediately from the couplings presented in table 2 which yield the branching ratios collected in tables 4-6. The channel t 4 → Ht is allowed only if couplings to H u are present and is sizable only at small tan β independently of the t 4 doublet fraction. Conversely, the channel b 4 → Hb requires couplings to H d and can easily dominate at medium-to-large tan β independently of the b 4 doublet fraction.
For the decay t 4 → H ± b, couplings to H u (H d ) only, result in a mostly singlet (doublet) t 4 with large branching ratio at small (medium-to-large) tan β. Similarly, for the decay b 4 → H ± t, couplings to H u (H d ) only, result in a mostly doublet (singlet) b 4 with large branching ratio at small (medium-to-large) tan β. These features can be inferred directly from tables 2 and 6. Inspection of the bottom panels of figure 1 reveals also that, in presence of coupling to H d only and at large tan β, the t 4 → H + b branching ratio is typically close to unity while the b 4 → H − t one can be small. This happens because in presence of couplings to H d only, t 4 is either 100% doublet or 100% singlet (the doublet fraction in eq. (4.1) is unity but it is possible to have a singlet t 4 by lowering M T ) while the b 4 doublet fraction varies continuously between 0 and 100%. This implies that the t 4 → H + b branching ratio is either close to 100% (doublet t 4 ) or close to 0 (singlet t 4 ), while the b 4 → H − t one covers smoothly the entire range, with large (small) branching ratios corresponding to a mostly singlet (doublet) b 4 . The presence of couplings to both H u and H d (gray points) allows the t 4 → H ± b branching ratio to acquire any value by tuning the contributions of the two sets of couplings.

Decays into W , Z and h
The results of the parameter space scan for decays into W , Z and h are summarized in Fig. 2, where we show branching ratios of the lightest up-type and down-type vectorlike quarks t 4 and b 4 . In the top two panels we consider the limiting cases of couplings to H u and H d only for which we consider only the vectrolike quark with non-trivial branching Table 4. Branching ratios of the vectorlike quarks t 4 and b 4 into W , Z, h and H as a function of t β = tan β. These expressions are exact for heavy vectorlike masses and in the limit in which the couplings in  Table 5. Branching ratios of the vectorlike quarks t 4 and b 4 into W , Z, h and H ± as a function of t β = tan β. These expressions are exact for heavy vectorlike masses and in the limit in which the couplings in  Table 6. Branching ratios of the vectorlike quarks (t 4 , b 4 ) into (H, H ± ) normalized to the total branching ratio into SM particles (W , Z and h) as a function of t β = tan β. These expressions are exact for heavy vectorlike masses and in the limit in which the couplings in table 2 dominate. In the bottom panels we consider the general case. In presence of couplings to both b4 is a doublet, in absence of mixing its mass is exactly MQ. The mass matrix for t4 and t5 has MQ and MT on the diagonal with some small mixing, implying that mt 4 < min(MQ, MT ). In both cases the decay t4 → b4W is kinematically not allowed: for doublet t4  factors that appear in the formulae for the decay widths, see the discussion below (A.88). We start by considering the case of a mostly doublet t 4 with couplings to H u only (i.e. all λ's set to zero). In the Goldstone boson equivalence limit we see that only couplings to Z and h are non-zero and that they are controlled by the parameter κ Q , implying BR(t 4 → Zt) = BR(t 4 → ht) = 1/2, that corresponds to the point [1/2, 0]  For κ Q similar in size to (v u /M T )κκ a scan deviates mildly from the red ellipse (yellow points); if κ Q is allowed to be very small, the subleading terms mentioned above (green points) fill completely the green ellipse. For κ Q = 0 the six couplings depend on the two combinations (v u /M T )κ Tκ sin β and (v u M Q /M 2 T )κ T κ sin β, implying that the branching ratios lie exactly on the green ellipse. 5 In particular, note that unlessκ κM Q /M T only the region of the ellipse near the point (0, 1/3) is allowed; this is reflected in the high density of points near this limit at small κ Q . This feature can be seen also in the full scan presented in the upper panels of figure 2, in which the right-hand side of the ellipse is sparsely populated.
Finally, allowing also couplings to H d all branching ratios on the segment between The resulting path in the plane for three representative points is described by the red, blue and orange ellipses. The minor axes of the ellipses decreases for heavier vectorlike quark masses. Since the points on the right side of ellipses are considerably fine-tuned, as soon as we change the vectorlike quark mass (without changing the other parameters), the points move towards the left-hand side (as is the case for the orange curves).
(see eqs. (4.3) and (4.5)) which are now controlled by two independent combinations of parameters, implying that decay into Z is suppressed and the remaining two channels can have any branching ratios. This can be seen by the purple points clustered on the vertical line corresponding to BR(t 4 → Zt) = 0 in figure 3. In the high-density scan presented in the bottom two panels of figure 2 we see that every branching ratio is possible, but that the region in between the two purple lines in figure 3 is significantly more populated. Note that, if there are couplings only to H d , t 4 decays exclusively into the channels W b and W b 4 .
In the case of a mostly singlet t 4 with couplings to H u only (i.e. all λ's set to zero) the situation is very different. Keeping only the leading terms in 1/M Q,T,B for each coupling we obtain the following very simplified expressions for the h, W and Z couplings for a mostly singlet t 4 : (4.14) In the GBEL the h, W and Z couplings are controlled by the parameter κ T , implying BR(t 4 → ht) = BR(t 4 → Zt) = 1 2 BR(t 4 → W b) = 1/4, that corresponds to the point [1/4, 1/2] in the [BR(t 4 → Zt), BR(t 4 → W b)] plane. Corrections to this limiting case are suppressed by at least one mixing parameter (κ,κ) and by additional powers of the heavy doublet mass M Q (which for mostly singlet t 4 is much larger than M T ).
As κ T decreases below v/M Q , the largest coupling becomes λ h t 4 t , implying that the branching ratio into h increases and we move along a line towards the origin in the [BR(t 4 → Zt), BR(t 4 → W b)] plane (see the red line in the right panel of figure 3). As κ T further decreases below vM T /M 2 Q , the right-handed Z coupling becomes sizable and a large t 4 → Zt branching ratio becomes possible (see green shaded triangle in the right panel of figure 3). For κ T = 0, the W channel disappears and the points collapse on the line at BR(t 4 → W b) = 0 (green line in the right panel of figure 3); note that points with large BR(t 4 → Zt) require some degree of fine tuning. 6 Finally, allowing also couplings to H d , we see that the t 4 → W b branching ratio becomes effectively independent of the others for κ T < vM T /M 2 Q as we see from eq. (4.12). Under these conditions we start moving along a line towards the point [0,1] in the [BR(t 4 → Zt), BR(t 4 → W b)] plane (see purple points in the right panel of figure 3).
The features described here help understanding the results of the full scan presented in the figure 2. Note that the b 4 case is almost identical to the t 4 one, the only difference being the replacement of the top Yukawa and mass with the bottom ones.

Conclusions
We have analyzed decays of new quarks in extensions of the standard model and a two Higgs doublet model (type-II) by vectorlike pairs of new quarks, corresponding to a copy of SM quark SU(2) doublets and singlets and their vectorlike partners. We assumed only mixing with the third generation of SM quarks, nevertheless the results can be straightforwardly generalized for cases of mixing with the first or second generation.
We have identified several typical patterns of branching ratios of the lightest new uptype quark, t 4 , and down-type quark, b 4 , depending on the structure of Yukawa couplings that mix the vectorlike and standard model quarks and also on their doublet or singlet nature. Among the most striking results is the finding that decays into heavy neutral or charged Higgs bosons, when kinematically open, can easily dominate and even be close to 100%: b 4 → Hb at medium to large tan β, t 4 → Ht at small tan β and b 4 → H ± t, t 4 → H ± b at both large and small tan β.
We found that the conventional decay modes into W , Z and the SM Higgs boson, h, follow the pattern expected from the Goldstone boson equivalence limit that we have generalized to scenarios with all possible couplings. For doublet-like new quarks this leads to a one parameter family of branching ratios characterized by an arbitrary branching ratio to W and equal branching ratios to Z and h.
We have also discussed in very detail the structure of Yukawa couplings required to significantly deviate from the pattern characteristic of Goldstone boson equivalence limit that can result in essentially arbitrary branching ratios. For vectorlike quark masses within the reach of the LHC and near future colliders this does not require very special choices of model parameters. We found that large deviations from the GBEL are possible for (κ Q , λ Q ) < v/M T and κ T < vM T /M 2 Q for doublet-like and singlet-like t 4 , respectively (similar relations hold for b 4 with M T → M B and κ T → λ B ).
The new decay modes of vectorlike quarks through heavy Higgs bosons and the fact that they easily dominate imply that the usual search strategies are not sufficient and the current exclusion limits might not necessarily apply. Among the smoking gun signatures are the 6t, 4t2b, 2t4b and 6b final states resulting from the pair production of vectorlike quarks (and similar final states for single production). The SM backgrounds for these final states (at large invariant mass) are very small and thus searching for these processes could lead to the simultaneous discovery of a new Higgs boson and a new quark.

A.1 Mass eigenstates
From the Lagrangian in eq. (2.1) we read the 5 × 5 mass matrices in the up and down sectors in the gauge eigenstate basis: where i, j = 1, 2, 3. It is convenient to define the following vectors of flavor eigenstates: u La = (u L1 , u L2 , u L3 , T Q L , T L ) and d La = (d L1 , d L2 , d L3 , B Q L , B L ), and similarly for u Ra and d Ra .
We then use four unitary matrices to diagonalize the Yukawa matrices y ij u and y ij d : The resulting up and down mass matrices are: where the requirement of mixing with the third generation only has been imposed by setting Finally we diagonalize the 3 × 3 mass matrices M t and M b : The vectorsû La andd La of mass eigenstates (and similarly for right-handed fields) are explicitly given by: It is useful to have approximate expressions in the limit If the lightest vectorlike quarks (t 4 , b 4 ) are mostly SU (2) doublets, we find: The corresponding formulas for the lightest vectorlike quarks being mostly SU (2) singlets can be obtained by swapping the second and third column in each mixing matrix. Note that in the numerical analysis we use the exact expressions.

A.2 Couplings of the W boson
The Lagrangian in eq. (2.1) yields the following W boson interactions for left-handed and right-handed fermions: Note that the matrixV CKM ≡ U † L D L that appears in eq. (A.16) is not the standard CKM matrix. In fact, the matrix that controls the SM quark interactions with the W boson is given by: where in the last step we used the fact that (V u L ) 33

A.3 Couplings of the Z boson
The couplings of the Z boson to the quarks can be obtained from the kinetic terms: where the covariant derivative is 28) and the weak isospin T 3 a is +1/2 (-1/2) for up (down) components of SU(2) doublets and 0 for singlets. Following the same steps as in the previous section (with the simplification that flavor changing couplings are absent after the rotations in eqs. (A.3) and (A.4)) we obtain the following Z interactions: where f a =û a ord a and As expected there are no FCNC among the SM quark generations.

A.4 Couplings of the Higgs bosons
The couplings of neutral Higgs bosons to up and down quarks follow from the Yukawa interactions in the Lagrangian. The couplings of the first two generations to neutral Higgs bosons are not affected by mixing with the vectorlike fermions. In the basis in which the SM quark Yukawas are diagonal and keeping only the heaviest SM quarks and the vectorlike fermions we have: The CP even (h and H), CP odd (A) and Goldstone boson (G) mass eigenstates are: .
(A. 40) We require that in the CP conserving two Higgs doublet model we consider, the light Higgs h couplings to gauge bosons are identical to those in the SM. This implies that the heavy CP even Higgs H has no couplings to gauge bosons. In this limit, α = β − π/2 and the mass eigenstates h and H read: In terms of the mass eigenstates the Lagrangian for h and H reads: where the 3 × 3 matrices Y u and Y d are given by Since these matrices are not proportional to the corresponding 3 × 3 minors in the mass matrices given in eqs. (A.5) and (A.6), the Higgs couplings are in general flavor violating. The resulting Lagrangian is: , the Higgs boson couplings in the up and down sectors can be written as: where we used v u = v sin β and v d = v cos β. These expressions show explicitly that in the absence of vectorlike fermions the lightest Higgs (h) couplings are SM-like, while the heavier scalar Higgs (H) couplings in the up (down) sector are suppressed (enhanced) by tan β.
The Lagrangian for the CP-odd Higgs A reads: where the 3 × 3 matrices Y A u and Y A d are given by Thus we get: Note that in the limit in which there are no CP-violating phases, the left and right components of the diagonal pseudoscalar interactions combine to yield a single coupling proportional to γ 5 . Charged Higgs interactions involving SM quarks are controlled by the CKM and can be obtained following the procedure detailed in section A.2. We focus only on interactions involving the third generation of SM quarks and the heavy vectorlike fermions for which CKM effects are negligible. In the basis in which the SM quark Yukawas are diagonal and keeping only the heaviest SM quarks and the vectorlike fermions we have: where we adopt the conventional notation The charged Higgs (H ± ) and Goldstone bosons (G ± ) mass eigenstates are given by: (A. 60) In terms of the mass eigenstates the Lagrangian for H ± reads: and Y H ± d are given by Thus we get: where Explicit expressions for the couplings relevant to our analysis are: A.5 Partial decay widths of vectorlike quarks into the W , Z, h, H and H ± In this section, we present explicit expressions for the partial widths of the vectorlike quarks. The partial width for t i → W b j for i, j = 3, 4, 5, if kinematically allowed, is given by: The decay width for b i → W t j can be obtained by replacing t i → b i and b j → t j . Similarly, the partial width for t i → Zt j for i, j = 3, 4, 5 is given by: For almost doublet t j ( u Q u T ) the off-diagonal Yukawa and gauge couplings read: . For almost singlet t j ( u T u Q ) the off-diagonal Yukawa and gauge couplings read:

A.7 On ellipses
Here we show that if a vectorlike quark decays into three channels (Z, W , h) whose effective couplings are linear combinations of two fundamental parameters a and b, any two branching ratios lie on an ellipse. By assumption the widths Γ W,Z,h are linear combinations of a 2 , b 2 and ab. A generic ellipse in the [BR Z , BR W ] plane is given by the equation: