Cascade decays of Heavy Higgs bosons through vectorlike quarks

We study cascade decays of heavy neutral Higgs bosons through vectorlike quarks. We focus on scenarios where decay modes into pairs of vectorlike quarks are not kinematically open which extends the sensitivity of the LHC to larger masses. Assuming only mixing with the third family of standard model quarks the new decay modes of heavy Higgs bosons are: H → t4t → Wbt, Ztt, htt and H → b4b → Wtb, Zbb, hbb, where t4 (b4) is the new up-type (down-type) quark mass eigenstate. We identify the region of the parameter space where these decay modes are significant or can even dominate. We also find that the rates for these processes can be much larger than the rates for a single production of vectorlike quarks. Thus, in the identified regions, they provide the best opportunities for the discovery of a new Higgs boson and vectorlike quarks. In the numerical analysis we assume the CP even Higgs boson in the two Higgs doublet model type-II but the signatures are relevant for many other scenarios. ar X iv :1 90 7. 07 18 8v 1 [ he pph ] 1 6 Ju l 2 01 9

In addition, a heavy neutral Higgs boson could also decay into pairs of vectorlike quarks. However, we focus on the range of masses where H → t 4t4 , b 4b4 are not kinematically open which extends the sensitivity of the LHC to larger masses of vectorlike quarks (moreover, the final states would be the same as for the pair production of vectorlike quarks). We also find that the rates for processes (1.1) and (1.2) can be much larger than the rates for a single production of vectorlike quarks. Thus, in the identified regions, they provide the best opportunities for the discovery of a new Higgs boson and vectorlike quarks. Although in the numerical analysis we assume the heavy CP even Higgs boson in the two Higgs doublet model type-II, the signatures are relevant for many other scenarios. Furthermore, the results can be straightforwardly generalized for cases of mixing with the first or second family.
Similar signatures in the lepton sector, cascade decays of heavy Higgs bosons through vectorlike leptons, were studied in refs. [2][3][4][5][6]. If decay modes through both vectorlike quarks and leptons are kinematically open, the decays through quarks are expected to dominate because of the color factor. On the other hand, decay modes through leptons provide several very clean signatures [4,7] that might compensate for smaller rates. Alternatively, in the same model, if vectorlike quarks or leptons are heavier, they can decay through heavy Higgs bosons, including the charged Higgs boson, leading to very rare final states. The corresponding signatures were recently studied in ref. [1].
Vectorlike quarks and leptons near the electroweak scale provide a very rich phenomenology and often they are introduced to explain various anomalies. Examples include discrepancies in precision Z-pole observables [8][9][10][11] and the muon g-2 anomaly [12][13][14] among many others. They are also considered for a variety of theoretical reasons. Examples include studies of their effects on gauge and Yukawa couplings in the framework of grand unification [15][16][17][18][19][20][21][22][23], on electroweak symmetry breaking [24], and the Higgs boson mass and its decays [14,[25][26][27]. The supersymmetric extension with a complete vectorlike family also provides a possibility to understand the values of all large couplings in the SM from the IR fixed point structure of the renormalization group equations [28].
This paper is organized as follows. In section 2 we briefly summarize the model. Details of the analysis and experimental constraints are discussed in section 3. The main results are presented in section 4. We briefly discuss the search strategies in section 5 and conclude in section 6. The appendix contains formulas for partial decay widths of the heavy Higgs boson.

Model
We consider an extension of the two Higgs doublet model type-II by vectorlike pairs of new quarks: SU(2) doublets Q L,R and SU(2) singlets T L,R and B L,R . The Q L , T R and B R have the same quantum numbers as the SM quark doublet q L and the right-handed quark singlets u R and d R , respectively. The quantum numbers of new quarks, SM quarks and two Higgs doublets, are summarized in table 1. The model is described in detail in ref. [1] and thus we just briefly summarize it here. Table 1. Quantum numbers of the 3rd generation standard model quarks (q L , t R , d R ), extra vectorlike quarks and the two Higgs doublets. The electric charge is given by Q = T 3 + Y , where T 3 is the weak isospin, which is +1/2 for the first component of a doublet and -1/2 for the second component.
As is characteristic for the two Higgs doublet model type-II, we assume that the down sector couples to H d and the up sector couples to H u . This can be achieved by the Z 2 symmetry specified in table 1. The generalization to the whole vectorlike family of new fermions, including the lepton sector introduced in ref. [2], is straightforward. We further assume that, in the basis in which the SM quark Yukawa couplings are diagonal, the new quarks mix only with one family of SM quarks and we consider the mixing with the third family as an example. An arbitrary mixing could be easily accommodated.
The most general renormalizable Lagrangian consistent with our assumptions contains the following Yukawa and mass terms for the SM and vectorlike quarks: where the first term is the bottom Yukawa coupling, followed by Yukawa couplings of vectorlike quarks to H d (denoted by various λs), the top Yukawa coupling, Yukawa couplings of vectorlike quarks to H u (denoted by various κs), and finally by mass terms for vectorlike quarks. Note that the explicit mass terms mixing SM and vectorlike quarks, M qqL Q R , M tTL t R and M bBL b R , can be removed by redefinitions of Q L , T R , B R and the Yukawa couplings. The components of doublets are labeled as follows: We assume that the neutral Higgs components develop real and positive vacuum expectation values, H 0 u = v u and H 0 d = v d , as in the CP conserving two Higgs doublet Plugging the vacuum expectation values to the Lagrangian, we obtain the mass matrices describing the mixing between the third generation and vectorlike quarks: We label the resulting mass eigenstates as t i and b i with i = 3, 4, 5, where t 3 and b 3 represent the top quark and the bottom quark. A complete discussion of mass eigenstates, their couplings to the W , Z, and Higgs bosons, various approximate formulas, and other details can be found in ref. [1]. Formulas for partial decay widths of the heavy Higgs boson are summarized in the appendix.

Parameter space scan and experimental constraints
In the numerical study we scan the parameters of the model in the following ranges: We will also comment on the impact of lowering the upper ranges in Eqs. (3.2) and (3.3). We impose experimental constraints from precision electroweak measurements [29], h → (γγ, 4 ) [30][31][32], and direct searches for pair production of vectorlike quarks at the LHC [33,34]. We do not use searches for single production of VLQ [35,36] since the constraints are not stronger than those from the pair production. We further impose searches for heavy Higgs bosons: H → τ + τ − [37,38] and H → γγ [39,40].

Higgs production cross section and decays
Let us start by discussing the heavy neutral CP-even Higgs boson production cross section.
In figure 2, we show the production cross section dependence on the Higgs mass and tan β for scenarios in which the H → (t 4 t, b 4 b) decays are kinematically open and satisfy all experimental constraints. The lower bound on the Higgs mass in the left panel of figure 2 is thus connected to the limits from direct searches for vectorlike quarks. This scenario is also constrained by H → τ + τ − searches that, for large tan β, extend to larger Higgs masses.
The effective ggH vertex is dominated by top and bottom loops, which give contributions almost identical to the type-II two Higgs doublet model ones. Vectorlike quark loops generate the spread at large Higgs mass and small tan β. The lower bounds on vectorlike quark masses imply that the ggH vertex is significantly affected only for very heavy Higgs masses (in general one expects large effects for m H 2m t 4 ,b 4 ). To understand the tan β dependence, we note that λ H t 4 t 4 ∝ v u cos β ∝ sin 2β and that λ H b 4 b 4 ∝ v d sin β ∝ sin 2β, implying that both are the largest at tan β ∼ 1. The actual cross sections are calculated using SusHi [41] and then rescaled to take into account the impact of the modified ggH vertex.
In figure 3 we show the tan β dependence of Higgs partial decay widths and branching ratios for scenarios with couplings to H u only (all λs = 0), H d only (all κs = 0) and to both H u and H d . The dominant features of these plots can be easily understood from the tan β dependence of the heavy Higgs couplings to t and t 4 : λ H tt ∝ 1/ tan β and λ H t 4 t ∝ cos β, and couplings to b and b 4 : λ H bb ∝ tan β and λ H b 4 b ∝ sin β (see table 2 of Ref. [1]). These tan β dependences directly translate into the dependence of the partial widths in the left panels of figure 3.
In the scenario with couplings to H u only, the H → t 4 t mode is asymptotically smaller than both H → tt (at small tan β) and H → bb (at large tan β) and is relevant only at small-to-medium tan β: we find branching ratios larger than 10% for tan β ∈ [0.5, 10] and they can reach up to 40%. On the other hand, in the scenario with couplings to H d , the H → b 4 b mode is still asymptotically smaller than H → tt at small tan β but is not suppressed with respect to H → bb at large tan β. We find branching ratios larger than 10% for any tan β > 0.8. More importantly, this mode can dominate for tan β ∈ [4,18] and can reach up to 95%. The scenario with couplings to both H u and H d can be understood in a similar way. Note that the maximum Higgs partial widths and branching ratios into vectorlike quarks depend on the ranges of Yukawa couplings that we scan over that are given in Eqs. (3.2) and (3.3). The maximum partial widths scale with the square of the maximum coupling allowed; for example, limiting the upper ranges to 0.5 reduces the maximum widths by a factor of 4. The impact on the branching ratios is less straightforward. Reducing the upper range of the scan to 0.5 implies that the H → t 4 t branching ratio peaks at 15%; the H → b 4 b branching ratio dominates for tan β ∈ [5,10], peaks at about 85% but drops to about 20% for tan β ∼ 50.
Due to different tan β dependence of Higgs production cross section and branching ratios, it is interesting to show the total rate into individual final states. In figure 4 we show the various production rates for m H = 2.5 TeV as functions of tan β. We see that the t 4 t mode is the largest at very small tan β and that, although the H → b 4 b mode can dominate at medium tan β, the σ(pp → H → b 4 b) can still be larger at both small and very large tan β. Rates of the order of 0.1 fb are attainable for H → t 4 t at very small tan β and for H → b 4 b at medium-to-large tan β.
The lightest new quarks from heavy Higgs decays further decay into SM particles. The correlations between the branching ratio of H → t 4 t and individual branching ratios of t 4 are shown in figure 5 and similar correlations for b 4 are shown in figure 6. Main features of these plots can be understood from approximate formulas and the discussion in ref. [1]. We see that the decay modes of t 4 and b 4 into W , Z and h, cluster around the pattern expected from the Goldstone boson equivalence limit corresponding to sending all vectorlike quark masses to infinity. For singlet-like new quarks (red) 1 this leads to 2:1:1 branching ratios into W , Z and h. For doublet-like new quarks (blue) this leads to a one parameter family of branching ratios characterized by an arbitrary branching ratio to W and equal branching ratios to Z and h. For example, for a doublet-like t 4 , the W t 4 t coupling is controlled by λ Q while the corresponding couplings to Z and both Higgs bosons are controlled by κ Q . This results in a difference between the plots on the top (no couplings to H d allowed) and bottom (all couplings allowed) in figure 5 and similarly for the b 4 in figure 6. The main distinction between the corresponding plots in figs. 5 and 6 originates from different tan β dependence of relevant couplings. Note especially that while the branching ratios for t 4 → W b and b 4 → W t extend to 100% in the scenario with couplings to both H u and H d , the former anticorrelates with the H → t 4 t branching ratio as can be seen in the lower-left panel of figure 5.
The mixed scenarios (cyan and purple) interpolate between mostly singlet and mostly doublet cases. Note that these scenarios require careful choices of model parameters especially for b 4 at medium to large tan β, see eq. (2.4), where H → b 4 b is sizable. This is the reason for an empty area in between the mostly singlet and mostly doublet cases in the top plots of figure 6. It is expected that with large statistics the whole area would be populated.
Finally, as discussed in detail in ref. [1], with the general structure of Yukawa matrices that we allow, essentially arbitrary branching ratios of t 4 and b 4 can be achieved. However, going away from the Goldstone boson equivalence limit correlates with diminishing H → t 4 t and H → b 4 b because it requires very small κ Q,T and λ Q,B couplings that are directly related to Ht 4 t and Hb 4 b couplings.
The maximum production rates of individual final states in cascade decays of a heavy Higgs boson as functions of m H and m t 4 or m b 4 are presented in figure 7. We see that, The signatures of cascade decays of a heavy Higgs boson through vectorlike quarks are almost identical to the production of any new resonance (e.g. Z ) decaying to t 4 t or b 4 b. In addition, the final states we consider are very similar to the single production of both top-like and bottom-like vectorlike quarks, therefore all searches for a singly produced vectorlike quark can be reinterpreted as bounds on Higgs cascade decays. Note however that the topology of cascade decays provides more handles. For example, in the t 4 case, there is a top quark in the decay chain, and, in all cases, there is an additional resonance at the heavy Higgs mass. Thus, dedicated searches have a potential to considerably improve the limits found in standard single production studies.
Searches for Z → t 4 t by CMS have been presented in refs. [42,43]. In ref.
[42] a dedicated search for pp → Z → t 4 t → (W b, Zt, ht)t placed bounds in the range 0.01-1 pb in the lepton plus jets final state. In ref. [43] a search for pp → t 4 bj → Ztt with Z → and hadronic top, recasted as the production and decay of a Z , found bounds in the range 0.06-0.13 pb. An important result of our analysis is that the rates for cascade decays through a bottom-like vectorlike quark (b 4 ) can be almost an order of magnitude larger than the rates for cascade decays through the top-like vectorlike quark (t 4 ). However, dedicated searches or recasted analyses for a resonance decaying to b 4 b have not been performed. Searches for the single production of bottom-like vectorlike quark in the W t and hb final states by CMS have been presented in refs. [44,45] where bounds in the 0.1-1 pb were found. ATLAS studies of singly produced top-and bottom-like vectorlike quarks have been presented in refs. [36,46,47] where bounds at the 0.1 pb level were found.
Both the single production of vectorlike quarks and their production in Higgs cascade decays depend on the structure of Yukawa couplings and either process can dominate. This is illustrated in figure 8, where we plot the ratios of rates for these two production mechanisms. For the single production we consider both the W mediated modes, pp → t 4 bj and pp → b 4 tj, and the Z mediated modes, pp → t 4 tj and pp → b 4 bj. Note that t 4 (b 4 ) single production is typically dominated by the W (Z) mode. We see that production rates of vectorlike quarks in Higgs cascade decays can easily be orders of magnitude larger than the single production rates. For the b 4 , this result does not depend on its doublet/singlet nature, while for the t 4 it is more typical for the doublet case. 2 These findings further The cascade decays of a heavy Higgs boson through vectorlike quarks provide an interesting opportunity to discover two new particles simultaneously. Not only these decay modes are sizable or can even dominate, but the usually dominant decay modes, H → tt or bb, are extremely challenging due to huge standard model backgrounds. In addition, for these decay modes, the resonant peak can be destroyed by the interference with the SM background [48].

Conclusions
We studied cascade decays of a heavy neutral Higgs boson through vectorlike quarks, H → t 4 t → W bt, Ztt, htt and H → b 4 b → W tb, Zbb, hbb, where t 4 (b 4 ) is the new up-type (down-type) quark mass eigenstate. Limiting the size of Yukawa couplings of vectorlike fields to one, in the two Higgs doublet model type-II, we found that these decay modes can be significant or can even dominate.
(W b4t) vertex, implying the absence of the W mediated single production. This is clearly visible in the upper-left and lower-right panels of figure 8. In scenarios with couplings to both Hu and H d these W couplings are not constrained to vanish and the blue points in these plots would extend to smaller values.
We found that the branching ratio of H → t 4 t is larger than 10% for tan β ∈ [0. 5,10] and can reach up to 40%. More importantly, the branching ratio of H → b 4 b is larger than 10% for any tan β > 0.8 and this mode can dominate for tan β ∈ [4,18] reaching up to 95%. Multiplying with the Higgs production cross section, we found that σ(pp → H → t 4 t) is the largest at very small tan β and that, while the H → b 4 b mode can dominate at medium tan β, the σ(pp → H → b 4 b) can still be larger at both small and very large tan β.
The lightest new quarks from heavy Higgs decays further decay into SM particles through W , Z or h. We studied the correlations between the branching ratios of H → t 4 t (H → b 4 b) and individual branching ratios of t 4 (b 4 ). We presented the maximum production rates of individual final states in cascade decays of a heavy Higgs boson as functions of m H and m t 4 or m b 4 . We found that, for Higgs cascade decays through a t 4 , rates of 0.1 fb extend up to m H 2 TeV and m t 4 1.4 TeV. The rates above 1 ab can be achieved for m H 3.5 TeV or m t 4 2.5 TeV. For Higgs cascade decays through a b 4 , rates of individual final states larger than 0.1 fb extend up to m H 2.5 TeV and m b 4 1.8 TeV and can be even larger than 1 fb for m H 1.6 TeV and m b 4 1.2 TeV. The rates above 1 ab can be achieved for m H 4 TeV or m b 4 3 TeV.
The signatures of cascade decays of a heavy Higgs boson through vectorlike quarks are almost identical to the production of any new resonance decaying to t 4 t or b 4 b. They are also very similar to single productions of vectorlike quarks. However the topology of cascade decays provides more handles on the final states and thus dedicated searches have a potential to considerably improve the limits found in standard single production studies. So far, only searches for a resonance decaying to t 4 t have been performed. We have found that the rates for cascade decays through b 4 can be almost an order of magnitude larger than the rates for cascade decays through t 4 which motivates similar searches for a resonance decaying to b 4 b.
We have also found that the rates for these processes can be much larger, even by orders of magnitude, than the rates for single productions of vectorlike quarks. The final states have significantly lower standard model backgrounds not only compared to single productions of vectorlike quarks but especially compared to the usually dominant decay modes of heavy Higgses, H → tt or bb, searches for which are extremely challenging. Thus, the cascade decays of a heavy Higgs boson through vectorlike quarks provide the best opportunities for the discovery of a new Higgs boson and vectorlike quarks.
Physics (MITP) of the DFG Cluster of Excellence PRISMA + (Project ID 39083149), for its hospitality and support.