Cascade decays of heavy Higgs bosons through vectorlike quarks in two Higgs doublet models

We study cascade decays of heavy neutral Higgs bosons through vectorlike quarks. We focus on scenarios where decay modes into pairs of vectorlike quarks are not kinematically open which extends the sensitivity of the LHC to larger masses. Assuming only mixing with the third family of standard model quarks the new decay modes of heavy Higgs bosons are: H → t4t → Wbt, Ztt, htt and H → b4b → Wtb, Zbb, hbb, where t4 (b4) is the new up-type (down-type) quark mass eigenstate. In the numerical analysis we assume the CP even Higgs boson in the two Higgs doublet model type-II but the signatures are relevant for many other scenarios. We identify the region of the parameter space where these decay modes are significant or can even dominate, and thus they provide the best opportunities for the simultaneous discovery of a new Higgs boson and vectorlike quarks. We further explore the reach of the High Luminosity LHC for two representative decay modes, t4→ Zt → ℓℓt and b4→ Zb → ℓℓb, and found that cross sections at a 0.1 fb level can be probed with simple cut based analyses. We also find that the rates for Higgs cascade decays can be much larger than the rates for a single production of vectorlike quarks. Furthermore, the reach for vectorlike quarks in Higgs cascade decays and pair production extends to comparable masses.

In addition, a heavy neutral Higgs boson could also decay into pairs of vectorlike quarks. However, we focus on the range of masses where H → t 4t4 , b 4b4 are not kinematically open which extends the sensitivity of the LHC to larger masses of vectorlike quarks (moreover, the final states would be the same as for the pair production of vectorlike quarks). We also find that the rates for processes (1.1) and (1.2) can be much larger than the rates for a single production of vectorlike quarks. Thus, in the identified regions, they provide the best opportunities for the discovery of a new Higgs boson and vectorlike quarks. Although in the numerical analysis we assume the heavy CP even Higgs boson in the two Higgs doublet model type-II, the signatures are relevant for many other scenarios.
Similar signatures in the lepton sector, cascade decays of heavy Higgs bosons through vectorlike leptons, were studied in refs. [2][3][4][5][6]. If decay modes through both vectorlike quarks and leptons are kinematically open, the decays through quarks are expected to dominate because of the color factor. On the other hand, decay modes through leptons provide several very clean signatures [4,7] that might compensate for smaller rates. Alternatively, in the same model, if vectorlike quarks or leptons are heavier, they can decay through heavy Higgs bosons, including the charged Higgs boson, leading to very rare final states. The corresponding signatures were recently studied in ref. [1].
Vectorlike quarks and leptons near the electroweak scale provide a very rich phenomenology and often they are introduced to explain various anomalies. Examples include discrepancies in precision Z-pole observables [8][9][10][11] and the muon g-2 anomaly [12,13] among many others. They are also considered for a variety of theoretical reasons. Examples include studies of their effects on gauge and Yukawa couplings in the framework of grand unification [14][15][16][17][18][19][20][21][22], on electroweak symmetry breaking [23], and the Higgs boson mass and its decays [24][25][26][27][28][29][30][31]. The supersymmetric extension with a complete vectorlike family also provides a possibility to understand the values of all large couplings in the SM from the IR fixed point structure of the renormalization group equations [32]. The possibility of JHEP03(2020)029 Table 1. Quantum numbers of the 3rd generation standard model quarks (q L , t R , d R ), extra vectorlike quarks and the two Higgs doublets. The electric charge is given by Q = T 3 + Y , where T 3 is the weak isospin, which is +1/2 for the first component of a doublet and -1/2 for the second component.
a simple embedding into grand unified theories is one of the main motivations for considering complete copies of SM families. We should note, however, that vectorlike quarks with other quantum numbers have been considered in the literature (see, for instance, ref. [33]). Vectorlike fermions have also been studied in composite Higgs models [34][35][36][37][38][39][40][41][42][43][44][45][46][47], little Higgs models [48,49] and models with a gauged flavor group [50][51][52][53]. Most of these scenarios focus on mixing between the new vectorlike fermions and the third generation of SM quarks and lead to some generic signatures (see, for instance, refs. [33,[54][55][56][57]). This paper is organized as follows. In section 2 we briefly summarize the model. Details of the analysis and experimental constraints are discussed in section 3. The main results are presented in section 4. We briefly discuss the search strategies in section 5 and conclude in section 6. The appendix contains formulas for partial decay widths of the heavy Higgs boson.

Model
We consider an extension of the two Higgs doublet model type-II by vectorlike pairs of new quarks: SU(2) doublets Q L,R and SU(2) singlets T L,R and B L,R . The Q L , T R and B R have the same quantum numbers as the SM quark doublet q L and the right-handed quark singlets u R and d R , respectively. The quantum numbers of new quarks, SM quarks and two Higgs doublets, are summarized in table 1. The model is described in detail in ref. [1] and thus we just briefly summarize it here.
As is characteristic for the two Higgs doublet model type-II, we assume that the down sector couples to H d and the up sector couples to H u . This can be achieved by the Z 2 symmetry specified in table 1. The generalization to the whole vectorlike family of new fermions, including the lepton sector introduced in ref. [2], is straightforward. We further assume that, in the basis in which the SM quark Yukawa couplings are diagonal, the new quarks mix only with one family of SM quarks and we consider the mixing with the third family as an example. An arbitrary mixing could be easily accommodated.
The most general renormalizable Lagrangian consistent with our assumptions contains the following Yukawa and mass terms for the SM and vectorlike quarks:

JHEP03(2020)029
where the first term is the bottom Yukawa coupling, followed by Yukawa couplings of vectorlike quarks to H d (denoted by various λs), the top Yukawa coupling, Yukawa couplings of vectorlike quarks to H u (denoted by various κs), and finally by mass terms for vectorlike quarks. Note that the explicit mass terms mixing SM and vectorlike quarks, M qqL Q R , M tTL t R and M bBL b R , can be removed by redefinitions of Q L , T R , B R and the Yukawa couplings. The components of doublets are labeled as follows: We assume that the neutral Higgs components develop real and positive vacuum expectation values, H 0 u = v u and H 0 d = v d , as in the CP conserving two Higgs doublet model with v 2 u + v 2 d = v = 174 GeV and we define tan β ≡ v u /v d . Plugging the vacuum expectation values to the Lagrangian, we obtain the mass matrices describing the mixing between the third generation and vectorlike quarks: We label the resulting mass eigenstates as t i and b i with i = 3, 4, 5, where t 3 and b 3 represent the top quark and the bottom quark. A complete discussion of mass eigenstates, their couplings to the W , Z, and Higgs bosons, various approximate formulas, and other details can be found in ref. [1]. Formulas for partial decay widths of the heavy Higgs boson are summarized in the appendix. In the following section, we perform the numerical analysis assuming the alignment limit in which the light Higgs coupling to gauge bosons are identical to those in the SM. This is most easily, but not necessarily, attained in the decoupling limit where all the non-SM Higgs bosons are heavy [58][59][60].

Parameter space scan and experimental constraints
In the numerical study we scan the parameters of the model in the following ranges: and we will also comment on the impact of lowering the upper ranges in eqs. Due to current constraints on vectorlike quark masses, all indirect constraints have a minimal impact on our scan. We impose experimental constraints from oblique corrections and R b [61] following the results presented in section II of ref. [62] and section IIIB of ref. [63] (see also refs. [41,64]). We further include bounds from h → (γγ, 4 ) [65][66][67] (note that in the calculation of the effective hgg coupling we include effects of vectorlike quarks, that can be found, for instance, in refs. [68,69]), and direct searches for pair production of vectorlike quarks at the LHC [70,71] (for each scan point we calculate the t 4 and b 4 branching ratios into W , Z and h and compare with the constraint). We do not use searches for single production of VLQ [72,73] since the constraints are not stronger than those from the pair production. We finally impose searches for heavy Higgs bosons: [74,75], H → γγ [76,77]. In the alignment limit, the width H → hh is given in eq. (A.7) which impacts the H → hh branching ratio at the O(0.1%) level; hence H → hh constraints [78] are negligible.

Higgs production cross section and decays
Let us start by discussing the heavy neutral CP-even Higgs boson production cross section. Note that the results for the case of a CP-odd Higgs (A) would be very similar. In fact, the bbH and bbA production cross sections, which dominate at large tan β, are essentially identical, and the gluon fusion production of A is only slightly larger than the H one for large Higgs masses (see, for instance, figure 3.36 of ref. [68]). Also, in the alignment limit, the H and A branching ratios are almost identical.
In figure 2, we show the production cross section dependence on the Higgs mass and tan β for scenarios in which the H → (t 4 t, b 4 b) decays are kinematically open and satisfy all experimental constraints. The lower bound on the Higgs mass in the left panel of figure 2 is thus connected to the limits from direct searches for vectorlike quarks. This scenario is also constrained by H → τ + τ − searches that, for large tan β, extend to larger Higgs masses.
The effective ggH vertex is dominated by top and bottom loops, which give contributions almost identical to the type-II two Higgs doublet model ones. Vectorlike quark loops generate the spread at large Higgs mass and small tan β. The lower bounds on vectorlike quark masses imply that the ggH vertex is significantly affected only for very heavy Higgs masses (in general one expects the largest effects for m H ∼ 2m t 4 ,b 4 , see, for instance, ref. [69]). To understand the tan β dependence, we note that λ H , implying that both are the largest at tan β ∼ 1. The actual cross sections are calculated using SusHi [79] and then rescaled to take into account the impact of the modified ggH vertex.
In figure 3 we show the tan β dependence of Higgs partial decay widths and branching ratios for scenarios with couplings to H u only (all λs = 0), H d only (all κs = 0) and to both H u and H d . The dominant features of these plots can be easily understood from the JHEP03(2020)029 tan β dependence of the heavy Higgs couplings to t and t 4 : λ H tt ∝ 1/ tan β and λ H t 4 t ∝ cos β, and couplings to b and b 4 : λ H bb ∝ tan β and λ H b 4 b ∝ sin β (see table 2 of ref. [1]). These tan β dependences directly translate into the dependence of the partial widths in the left panels of figure 3.
In the scenario with couplings to H u only, the H → t 4 t mode is asymptotically smaller than both H → tt (at small tan β) and H → bb (at large tan β) and is relevant only at small-to-medium tan β: we find branching ratios larger than 10% for tan β ∈ [0.5, 10] and they can reach up to 40%. On the other hand, in the scenario with couplings to H d , the H → b 4 b mode is still asymptotically smaller than H → tt at small tan β but is not suppressed with respect to H → bb at large tan β. We find branching ratios larger than 10% for any tan β > 0.8. More importantly, this mode can dominate for tan β ∈ [4,18] and can reach up to 95%. The scenario with couplings to both H u and H d can be understood in a similar way. 1 Note that the maximum Higgs partial widths and branching ratios into vectorlike quarks depend on the ranges of Yukawa couplings that we scan over that are given in eqs. (3.2) and (3.3). The maximum partial widths scale with the square of the maximum coupling allowed; for example, limiting the upper ranges to 0.5 reduces the maximum widths by a factor of 4. The impact on the branching ratios is less straightforward. Reducing the upper range of the scan to 0.5 implies that the H → t 4 t branching ratio peaks at 15%; the H → b 4 b branching ratio dominates for tan β ∈ [5,10], peaks at about 85% but drops to about 20% for tan β ∼ 50.
Due to different tan β dependence of Higgs production cross section and branching ratios, it is interesting to show the total rate into individual final states. In figure 4 we show the various production rates for m H = 2.5 TeV as functions of tan β. We see that the t 4 t mode is the largest at very small tan β and that, although the H → b 4 b mode can dominate at medium tan β, the σ(pp → H → b 4 b) can still be larger at both small and JHEP03(2020)029  figure 6. Main features of these plots can be understood from approximate formulas and the discussion in ref. [1]. We see that the decay modes of t 4 and b 4 into W , Z and h, cluster around the pattern expected from the Goldstone boson equivalence limit corresponding to sending all vectorlike quark JHEP03(2020)029 masses to infinity. For singlet-like new quarks (red) 2 this leads to 2:1:1 branching ratios into W , Z and h. For doublet-like new quarks (blue) this leads to a one parameter family of branching ratios characterized by an arbitrary branching ratio to W and equal branching ratios to Z and h. For example, for a doublet-like t 4 , the W t 4 t coupling is controlled by λ Q while the corresponding couplings to Z and both Higgs bosons are controlled by κ Q . This results in a difference between the plots on the top (no couplings to H d allowed) and bottom (all couplings allowed) in figure 5 and similarly for the b 4 in figure 6. The main distinction between the corresponding plots in figures 5 and 6 originates from different tan β dependence of relevant couplings. Note especially that while the branching ratios for t 4 → W b and b 4 → W t extend to 100% in the scenario with couplings to both H u and H d , the former anticorrelates with the H → t 4 t branching ratio as can be seen in the lower-left panel of figure 5.
The mixed scenarios (cyan and purple) interpolate between mostly singlet and mostly doublet cases. Note that these scenarios require careful choices of model parameters especially for b 4 at medium to large tan β, see eq. (2.4), where H → b 4 b is sizable. This is the reason for an empty area in between the mostly singlet and mostly doublet cases in the top plots of figure 6. It is expected that with large statistics the whole area would be populated.
Finally, as discussed in detail in ref. [1], with the general structure of Yukawa matrices that we allow, essentially arbitrary branching ratios of t 4 and b 4 can be achieved. However, going away from the Goldstone boson equivalence limit correlates with diminishing H → t 4 t and H → b 4 b because it requires very small κ Q,T and λ Q,B couplings that are directly related to Ht 4 t and Hb 4 b couplings.
JHEP03(2020)029   The signatures of cascade decays of a heavy Higgs boson through vectorlike quarks are almost identical to the production of any new resonance (e.g. Z ) decaying to t 4 t or b 4 b. 3 In addition, the final states we consider are very similar to the single production of both top-like and bottom-like vectorlike quarks, therefore all searches for a singly produced vectorlike quark can be reinterpreted as bounds on Higgs cascade decays. Note however that the topology of cascade decays provides more handles. For example, in the t 4 case, there is a top quark in the decay chain, and, in all cases, there is an additional resonance at the heavy Higgs mass. Thus, dedicated searches have a potential to considerably improve the limits found in standard single production studies.
Searches for Z → t 4 t by CMS have been presented in refs. [83,84] using 35.9 fb −1 of integrated luminosity. In ref. [83] a search for pp → t 4 bj → Ztt with Z → and hadronic top, recasted as the production and decay of a Z , found bounds in the range 0.06-0.13 pb. In ref. [84] a dedicated search for pp → Z → t 4 t → (W b, Zt, ht)t placed bounds in the range 10-100 fb in the lepton plus jets final state for m t 4 < 3 TeV and m Z < 4 TeV. An important result of our analysis is that the rates for cascade decays through a bottom-like vectorlike quark (b 4 ) can be almost an order of magnitude larger than the rates for cascade decays through the top-like vectorlike quark (t 4 ). However, dedicated searches or recasted analyses for a resonance decaying to b 4 b have not been performed. Searches for the single production of bottom-like vectorlike quark in the W t and hb final states by CMS have been presented in refs. [85,86] where bounds in the 0.1-1 pb were found. ATLAS studies of singly produced top-and bottom-like vectorlike quarks have been presented in refs. [73,87,88] where bounds at the 0.1 pb level were found.
Estimating the reach of searches at the High Luminosity LHC (HL-LHC) with 3 ab −1 of integrated luminosity is challenging because of the difficulty to reproduce multi-variate analyses and to envision improvements in future analysis techniques. Nevertheless, we can obtain some conservative estimates. A naive rescaling of the analysis in ref. [84] by the square root of the ratio of future to present integrated luminosities yields expected bounds at the 1-10 fb level. Future improvements in the analysis will certainly lead to lower upper bounds.
With more data, other decay modes become competitive or even yield a stronger sensitivity. Examples include the channels H → t 4 t → Ztt → + −t t ( = e, µ) with hadronic tops and H → b 4 b → Zbb → + −b b ( = e, µ). In this case there is no missing energy (as opposite to the analysis in ref. [84]) and all invariant masses can be reconstructed. We implemented the new physics model in FeynRules [89]. We generated parton level events with MadGraph5 [90] and used Pythia8 [91,92] and Delphes [93] for shower, hadronization and detector simulation. As a representative example we choose m H = 2 TeV and m t 4 ,b 4 = 1.5 TeV. In the t 4 t case, we require H T > 1 TeV, two leptons with total transverse momentum P T ( ) > 450 GeV, two fat-jets (as reconstructed by Delphes in its default setting) and the invariant mass of the di-lepton pair plus the hardest fat-jet m j ∈ [1.3, 1.6] TeV. Similarly, in the b 4 b case, we require H T > 1 TeV, two leptons JHEP03(2020)029 with total transverse momentum P T ( ) > 550 GeV, two jets out of which at least one is b-tagged and m j ∈ [1.2, 1.6] TeV, where j is the hardest jet (whether b-tagged or not).
The resulting m j distributions are displayed in figure 8 where the signals are normalized to 1 fb (prior to the Z decay). From these results we estimate that the HL-LHC with an integrated luminosity of 3 ab −1 is sensitive to cross sections: σ(pp → H → t 4 t → Ztt) 0.19 fb and σ(pp → H → b 4 b → Zbb) 0.16 fb. We stress that this is a naive cutbased analysis which focuses on vectorlike quarks decaying to leptonic Z. We reasonably expect that, with a more sophisticated analysis and by considering/combining more decay channels, significantly smaller cross sections could be probed.
Finally, we would like to compare the production rates of vectorlike quarks in Higgs cascade decays and single production. Both mechanisms depend on the structure of Yukawa couplings and either process can dominate. This is illustrated in figure 9, where we plot the ratios of rates for these two production mechanisms. For the single production we consider both the W mediated modes, pp → t 4 bj and pp → b 4 tj, and the Z mediated modes, pp → t 4 tj and pp → b 4 bj. Note that t 4 (b 4 ) single production is typically dominated by the W (Z) mode. We see that production rates of vectorlike quarks in Higgs cascade decays can easily be orders of magnitude larger than the single production rates. For the b 4 , this result does not depend on its doublet/singlet nature, while for the t 4 it is more typical JHEP03(2020)029 for the doublet case. 4 These findings further motivate dedicated analyses for resonances decaying into t 4 t or b 4 b.
In contrast to Higgs cascade decays and single production mechanisms, the pair production cross section of vectorlike quarks is model independent. Projections for the reach of the HL-LHC can be found in section 6.3.2 of ref. [6] where it is shown that vectorlike quarks with masses up to about 1350 GeV can be discovered at the 5σ level. The corresponding 2σ sensitivity would extend to only somewhat larger masses because of the steep decrease of the pair production cross section for increasing vectorlike quark mass. We see that the experimental sensitivities of the HL-LHC to vectorlike quarks in heavy Higgs cascade decays and pair production are comparable.
The cascade decays of a heavy Higgs boson through vectorlike quarks provide an interesting opportunity to discover two new particles simultaneously. Not only these decay modes are sizable or can even dominate, but the usually dominant decay modes, H → tt or bb, are extremely challenging due to huge standard model backgrounds. In addition, for these decay modes, the resonant peak can be destroyed by the interference with the SM background [94].

Conclusions
We studied cascade decays of a heavy neutral Higgs boson through vectorlike quarks, H → t 4 t → W bt, Ztt, htt and H → b 4 b → W tb, Zbb, hbb, where t 4 (b 4 ) is the new up-type (down-type) quark mass eigenstate. Limiting the size of Yukawa couplings of vectorlike fields to one, in the two Higgs doublet model type-II, we found that these decay modes can be significant or can even dominate.
We found that the branching ratio of H → t 4 t is larger than 10% for tan β ∈ [0.5, 10] and can reach up to 40%. More importantly, the branching ratio of H → b 4 b is larger than 10% for any tan β > 0.8 and this mode can dominate for tan β ∈ [4,18] reaching up to 95%. Multiplying with the Higgs production cross section, we found that σ(pp → H → t 4 t) is the largest at very small tan β and that, while the H → b 4 b mode can dominate at medium tan β, the σ(pp → H → b 4 b) can still be larger at both small and very large tan β.
The lightest new quarks from heavy Higgs decays further decay into SM particles through W , Z or h. We studied the correlations between the branching ratios of H → The signatures of cascade decays of a heavy Higgs boson through vectorlike quarks are almost identical to the production of any new resonance decaying to t 4 t or b 4 b. They are also very similar to single productions of vectorlike quarks. However the topology of cascade decays provides more handles on the final states and thus dedicated searches have a potential to considerably improve the limits found in standard single production studies. So far, only searches for a resonance decaying to t 4 t have been performed. We have found that the rates for cascade decays through b 4 can be almost an order of magnitude larger than the rates for cascade decays through t 4 which motivates similar searches for a resonance decaying to b 4 b.
We explored the reach of decay modes in which there is no missing energy and all invariant masses can be reconstructed. As examples we considered the channels H → t 4 t → Ztt → + −t t ( = e, µ) with hadronic tops and H → b 4 b → Zbb → + −b b ( = e, µ). We have shown that even a very simple cut based analysis allows the HL-LHC with 3 ab −1 of integrated luminosity to probe cross sections σ(pp → H → t 4 t → Ztt) 0.19 fb and σ(pp → H → b 4 b → Zbb) 0.16 fb. We expect that, with more sophisticated analyses and by considering/combining more decay channels, significantly smaller cross sections could be tested.
We have also found that the rates for these processes can be much larger, even by orders of magnitude, than the rates for single productions of vectorlike quarks. The final states have significantly lower standard model backgrounds not only compared to single productions of vectorlike quarks but especially compared to the usually dominant decay modes of heavy Higgses, H → tt or bb, searches for which are extremely challenging. Thus, the cascade decays of a heavy Higgs boson through vectorlike quarks provide the best opportunities for the discovery of a new Higgs boson and vectorlike quarks.

JHEP03(2020)029
The decay width for H → γγ can be obtained from eq. (2.23) of ref. [68] supplemented by the presecription discussed in refs. [96,97] and is given by: Finally, the decay width for H → hh, where h is the SM-like Higgs boson, is given by: Open Access. This article is distributed under the terms of the Creative Commons Attribution License (CC-BY 4.0), which permits any use, distribution and reproduction in any medium, provided the original author(s) and source are credited.