Tau-jet signatures of vectorlike quark decays to heavy charged and neutral Higgs bosons

We study $4b+2\tau$ and $4b+1\tau$ signatures of heavy neutral and charged Higgs bosons originating from cascade decays of pair-produced new quarks. Decays of vectorlike quarks through heavy Higgses can easily dominate in the two Higgs doublet model of type-II, and the studied signatures are common to many possible decay chains. We design search strategies for these final states and discuss the mass ranges of heavy Higgs bosons and new quarks that can be explored at the Large Hadron Collider as functions of branching ratios in a model independent way. We further combine the results with a similar study focusing on decays which lead to a $6b$ final state and interpret the sensitivity to charged and neutral Higgs bosons and vectorlike quarks in the type-II two Higgs doublet model. We find that the LHC reach for their masses extends to well above 2 TeV in the case of an SU(2) doublet quark and to at least 1.8 TeV for a bottom-like SU(2) singlet quark in the whole range of $\tan\beta$ between 1 and 50.


Introduction
In two Higgs doublet models (2HDMs) with vectorlike quarks, the decays of vectorlike quarks are often dominated by cascade decays through charged or neutral Higgs bosons leading to signatures with 6 standard model (SM) fermions [1]. The heavy Higgs bosons are effectively pair produced with QCD-size cross sections, while the final states with a large number of 3rd generation fermions, resulting from their dominant decay modes, have very small irreducible SM background. This presents a great opportunity for the Large Hadron Collider (LHC). However, depending on hierarchies in masses of new quarks and Higgs bosons and other model parameters, many decay chains are possible. Moreover, the rates for several of them are often comparable, diluting individual final states. Thus designing optimal searches for specific possibilities is not practical, and focusing on the common signature of a whole class of decay chains might be more beneficial.
In ref. [2] we investigated final states with combinations of 6 top and bottom quarks in an extension of the type-II 2HDM. In that case, since the top quark also decays into a bottom quark, a common signature for many possible decay chains is 6 bottom quarks. Although the search strategies were tailored to the final state consisting of exactly 6 bottom quarks, they were found to be very effective for all the other processes. Even with current data, new quarks and Higgs bosons with masses of about 1.5 TeV could be seen irrespective of tan β which is striking in comparison with the reach for heavy Higgs bosons in 2HDMs without vectorlike matter. On the other hand, the current limits for heavy neutral CP-even and CP-odd Higgs bosons, H and A, are only about 400 GeV for medium tan β [3,4] and are significantly weaker for the charged Higgs boson, H ± [5][6][7][8][9].
In this paper we present analyses based on final states where one of the heavy neutral or charged Higgs bosons decays to tau leptons, see figure 1. Although in the type-II 2HDM the decay modes H(A) → τ τ and H ± → τ ν are subleading, they have the advantage of competing with significantly smaller background. Actually, in the type-II 2HDM, H(A) → ττ leads to stronger limits [3,4] than H(A) → bb [10,11]. We present search strategies for 4b + 2τ and 4b + 1τ signatures which are common to several possible decay chains involving both top and bottom quarks in final states and discuss the mass ranges of heavy Higgs bosons and new quarks that can be explored at the LHC as functions of branching ratios (BRs) in a model independent way. We find that the reach of the LHC for these decay modes is comparable to the reach for the similar processes with dominant decay modes of heavy Higgses, discussed in [2]. We further combine these analyses to study the ultimate reach of the high luminosity (HL) LHC and interpret the sensitivity to charged and neutral Higgs bosons and vectorlike quarks in the type-II 2HDM. We find that the LHC reach for their masses extends to well above 2 TeV in the case of an SU(2) doublet quark and to at  The same or very similar signatures do not rely on type-II 2HDM and could be present in many models and frameworks with extended Higgs sectors and vectorlike quarks. For scalars that couple to leptons more than in the type-II 2HDM the rates for the processes we consider could be further enhanced. These signatures are also relevant for models with Z or W with the neutral (charged) Higgs boson in figure 1 replaced by Z (W ). For example, a Z with couplings to b and a vectorlike quark was suggested in connection with the Z-pole anomalies [12,13] or tensions in rare B decays [14,15].
Related signatures of cascade decays of heavy neutral Higgs bosons through vectorlike and SM quarks were recently studied in ref. [16]. These decay modes, depending on the size and the structure of Yukawa couplings of vectorlike quarks, are relevant when vectorlike quarks are lighter than the new Higgs bosons.
This paper is organized as follows. In Sec. 2 we describe signal topologies and briefly review the model and expected pattern of branching ratios. In Section 3 we discuss search strategies for final states which contain two bottom quarks, and one or two tau leptons, and estimate the reach of the LHC. We summarize the main results in Sec. 4. Our conclusions are drawn in section 5. Supplementary details of the analysis are provided in the appendix.

Signal topologies and expected rates
The 4b + 2τ and 4b + 1τ final states can be produced with large rates from cascade decays of new quarks through heavy Higgs bosons. The processes shown in figure 1 are expected to be the most sizable in the type-II 2HDM extended by SU (2) doublet and singlet quarks with the same quantum numbers as SM quarks and corresponding vectorlike partners. We label the lightest new mass eigenstates of bottom-like and top-like vectorlike quarks as b 4 and t 4 . We assume that new quarks mix dominantly with the third generation of SM quarks, although the Yukawa couplings can be very small. We work in the alignment limit of the 2HDM so that the light CP-even Higgs boson, h, is SM-like. Details of the model have been presented in refs. [1,16]. The pair production cross section of new quarks depends only on their mass, see figure 2, while the typical BRs of both new quarks and heavy Higgs bosons depend only on tan β. Depending on the doublet/singlet nature of new quarks there are ranges of tan β where their decays through heavy Higgses dominate the decays through W , Z, and h. Branching ratios of vectorlike quarks were studied in detail in ref. [1]. The typical pattern of BRs, reproduced from table 6 of ref. [1], is shown in figure 3 for singlet-like (left) and doublet-like (right) b 4 and t 4 . The plots assume that a decay through only one heavy Higgs boson is kinematically open and the remaining BRs are to W , Z, and h. The BRs to A are the same as to H (in this model b 4 has almost identical couplings to H and A, see, for instance, appendix A.4 of ref. [1] and footnote 3 of ref. [2]). We see that a singlet-like b 4 can dominantly decay to both heavy neutral and charged Higgses at moderate to large tan β, while singlet-like t 4 decays to heavy Higgses can be sizable only for small tan β. On the other hand, note that doublet-like b 4 and t 4 can have large BRs to one of the neutral or charged Higgs bosons at any tan β.
If decays through multiple heavy Higgs bosons are simultaneously kinematically open, the corresponding BRs are appropriately rescaled. Assuming the same masses of heavy Higgses, the typical ratios of partial widths of a singlet-like b 4 and t 4 to H, A, and H ± are 1/4 : 1/4 : 1/2. Doublet-like b 4 and t 4 dominantly decay to the charged Higgs in a different region of tan β than to neutral Higgses. For more details, see the discussion in ref. [2]. Subsequent decays of heavy Higgses follow the usual pattern of type-II 2HDM. Large BRs of heavy neutral Higgses to bottom quarks require medium to large tan β (see e.g. figure 3 of ref. [16]). Moreover, branching ratios of both heavy neutral and charged Higgses into quarks and leptons are strongly correlated. For projected sensitivities at the HL-LHC  tanβ tanβ  we take BR(H/A → bb) = BR(H + → tb) = 90% and BR(H/A → τ + τ − ) = BR(H + → τ + ν τ ) = 10% motivated by type-II 2HDM at large tan β, however the sensitivities can be straightforwardly reinterpreted for any other pattern of BRs.
The combination of tan β dependences of BRs of new quarks and heavy Higgses identifies processes shown in figure 1 as the most promising within the type-II 2HDM. However, the same or very similar signatures can arise in many other models with new quarks and scalars or new gauge bosons, Z or W . The search strategies discussed in following sections are thus not limited to the 2HDM.

Event generation and selection
All cascade decays in figure 1 yield final states with multiple b-jets and one or two τ leptons. Accordingly, we present two strategies in which we consider events with exactly one or two τ -tagged jets. For each signal, events are generated using a FeynRules [54] and corresponding UFO model file based on the model described in detail in ref. [1]. Both signal and background events are generated using MadGraph5 [52] and are passed to Pythia8 [55,56] for showering and hadronization. Delphes [57] is used with the standard settings for the detector simulation; in particular for determining the origin of the τ -and b-tagged jets. We note that the standard CMS simulation cards in Delphes require |η| < 2.5 in selecting jets that are either b-or τ -tagged. However, the total efficiency for tagging b-jets is dependent on the p T (in units of GeV) of the observed jet, b = tanh(400/p T )85/(4 + 0.252p T ), while that of a τ -jet is fixed with respect to p T , τ = 0.6.
At the detector level, an advantage of search strategies based on τ -jets is in avoiding large irreducible QCD backgrounds. As previously discussed [58], QCD processes involving final states with multiple b-jets can be easily faked by gluons splitting to a pair of b-quarks. τ -jet final states avoid the need to include these large irreducible backgrounds allowing for simple cut-based strategies and confidence in the simulated background samples. In addition, the analyses we present here rely largely on the ability to reconstruct decays of the top quark in order to suppress tt backgrounds which requires kinematic information of both τ -and b-jets in the samples. Alternative strategies could be explored utilizing the leptonic decay modes of the τ . However, these modes contribute only about 35% of the total width and lack the features in the final state allowing to efficiently suppress the backgrounds originating from tt.
The production cross sections pp → (b 4b4 , t 4t4 ) depend only on the vectorlike quark mass and are presented in figure 2 where we show the LO and NLO pair production cross section for the LHC running at 14 TeV. The NLO cross section has been obtained using the public code Top++ [53] with standard settings. In the inset, we present the relative enhancement of the NLO cross section compared to the LO result. In the next two subsections we present the details of the 2τ and 1τ analyses.

2b + 2τ analysis
For cascade decays leading to final states with two τ leptons, the initial event selection we require for this analysis is: • Exactly two τ -tagged jets with p T > 50 GeV among the six highest p T jets; • At least four additional jets, any two of which are b-tagged, with p T > (200, 100, 100, 50) GeV;  Figure 4. Distribution of the total hadronic energy in the 2b + 2τ analysis before (left) and after (right) the anti-top cuts on m τ1bi and m τ2bi . Signal distributions for m b4 = 1.5 and 2.5 TeV are shown in blue and cyan respectively. In order to make evident the position of the peak in the H T distributions we have rescaled the later by a factor of 10 2 (left) and 10 (right), respectively. The red, green, magenta (×10), and gray regions show the background distributions from tt + jj, tt, bb + W W + jj, and bb + τ τ respectively.
• In order to suppress backgrounds from Standard Model Higgs and Z boson, the invariant mass of the two τ -tagged jets is required to be larger than 200 GeV.
These initial cuts allow for two dominant sources of backgrounds: bbW + W − (jj) and bbτ + τ − (jj), where we include backgrounds with two additional jets denoted by (jj), i.e. bbτ + τ − (jj) denotes the sum of bbτ + τ − and bbτ + τ − + jj backgrounds. We split the former in two parts corresponding to resonant tt(jj) where the b-W ± pairs are produced from the decay of the top and the non-resonant remainder without tops in the decay chain. In figure 4 (left), we show the transverse hadronic energy (H T ) distribution which we obtain after the initial selection described above; we show the relevant backgrounds and two signals corresponding to m b 4 = 1.5 and 2.5 TeV (the signal cross sections correspond to maximal branching ratios and the latter has been rescaled by a factor 10 2 (left)). We see that even a moderate cut of H T > 1500 GeV alone can reduce most background events by roughly an order of magnitude compared to signal events even with lower masses. This kinematic advantage improves for heavier vectorlike quark masses. It is clear that the most formidable background is tt(jj). The top quark decays can be reconstructed by forming the minimum invariant mass between a τ -tagged jet and a b-jet in the event. We begin by first forming the minimum of the invariant masses between the leading τ -jet and any of the b-jets in the event m τ 1 b i followed by the same for the next-to-leading τ -jet and the remaining b-jets: where b i min is the b-jet selected when constructing m τ 1 b . For tt(jj) backgrounds, the distributions of these variables peak around the top quark mass, see figure 5 (left). By contrast, since the τ -jets in signal events do not form a well-defined invariant mass with any b-jet in the event, the corresponding distribution shows no such correlation, see figure 5 (right) and this allows to efficiently distinguish the top decays. Thus, tt(jj) can easily be cut away using an anti-top cut requiring: The distributions we obtain after applying the anti-top cut are shown in figure 4 (right), where we have rescaled the m b 4 = 2.5 TeV signal instead by a factor of 10 to avoid overlap of the peak in the distribution with the legend. Note that while τ -b pairs in the 2b+2τ background do not reconstruct the mass of the top quark, we find that the m τ 1,2 b invariant masses typically cluster even lower. Thus, events in this background in figure 4 (right) are highly suppressed compared to the scale of the figure. 1 Finally we apply a cut on H T which is optimized to the vectorlike fermion masses we consider: • H T > [500, 3000] GeV, depending on m b 4 ,t 4 .
1 Note that there are similar approaches using the information of top quark mass to subtract the tt background, e.g., in Ref [59]. We believe adding these methods using the neural network would provide extra power in searching for our signals.

2b + 1τ analysis
For cascade decays leading to final states with a τ and ν τ pair, the initial event selection we require for this analysis is: • Exactly one τ -tagged jet with p T > 50 GeV among the five highest p T jets; • At least four additional jets, any two of which are b-tagged, with p T > (200, 100, 100, 50) GeV; • A lower bound on the total missing transverse energy / E T > 350 GeV to suppress leptonic decays of the W boson in the backgrounds; • In order to suppress backgrounds from tops, the minimum invariant mass, m τ b , between the τ -tagged jet and any of the two b-tagged jets is required to be larger than 300 GeV.
Since we have only one τ -tagged jet we remove part of the tt(jj) background by strengthening the cut on m τ b . These initial cuts allow for backgrounds from resonant tt(jj), nonresonant bbW + W − (j) and bbτ ν τ . In the two panels of figure 6 we present the H T distribution of signal and backgrounds before (left) and after (right) the anti-top cut m τ b > 300 GeV. The comparison highlights the impact of this cut on the resonant tt(jj) background. The suppression of the bbτ ν τ background is due to a similar effect as for 2b+2τ in the previous analysis. The 2b + 1τ analysis is aimed at constraining signal events in which the τ originates from the charged Higgs decay H ± → τ ± ν τ , implying the presence of significant missing energy. Apart from a cut on / E T , it is expected that the correlation between the p T of the τ -tagged jet with the / E T in the events will differ in the signal compared to the background due to the heavy intermediate particle. We take advantage of this feature of the signal by considering the transverse mass of the τ − / E T system:

Model-independent sensitivities of 2b + 2τ and 2b + 1τ analyses
We begin by assuming that only one heavy Higgs channel is kinematically open and proceed to find sensitivities of the HL-LHC to b 4 → Hb, b 4 → H ± t, and t 4 → H ± b. The b 4 → Hb can be constrained by the 2b + 2τ analysis and the HL-LHC sensitivity is given by the solid orange curve in  In the case that b 4 is an SU (2) singlet, the rates for decays through neutral Higgses, charged Higgses or mixed decays through both charged and neutral Higgses (see figure 1 bottom left) can be simultaneously large. In this scenario the 2b + 2τ analysis is sensitive to the top-left and bottom-left diagrams in figure 1, while the 2b + 1τ to the top-right and bottom diagrams. To set a limit on the overall decay of b 4 to heavy Higgses, we assume that H, A and H ± are degenerate in mass and that the BRs of b 4 into them are typical, 1/4 : 1/4 : 1/2, as discussed in section 2. The HL-LHC sensitivity to the combined branching ratio into heavy Higgses resulting from a statistical combination of the 2b + 2τ and 2b + 1τ analyses is given by the dot-dashed orange curve. The procedure we adopt for the statistical combination of the two analyses is based on Poisson statistics and is summarized in appendix A. Since the 2b + 2τ analysis has superior reach than the 2b + 1τ one, these bounds are in between those we obtained from analyses in which decays into only one Higgs are kinematically allowed. We see that we achieve sensitivities close to 2 TeV.
For comparison, in green we also show the projected limits on BR(b 4 → Hb) using the 5b analysis in ref. [2] updated with the NLO production cross section and assuming BR(H → bb) = 0.9. We see that despite an order of magnitude smaller branching ratio, the H → τ + τ − leads to a similar reach in masses of new quarks and heavy Higgses at the HL-LHC. Furthermore, in case that the signal is observed in either of the final state, searching for the complementary signal in other decay modes would be crucial to uncover the nature of new physics.

Reach of HL-LHC for new quarks and heavy Higgses in type-II 2HDM
We recast model independent sensitivities obtained above as functions of vectorlike quark masses and tan β for the typical BRs of vectorlike quarks in type-II 2HDM discussed in section 2 and the usual BRs of heavy Higgses in this model. In figure 9 we show the reach of the HL-LHC for singlet-like and doublet-like t 4 and b 4 . The sensitivities to various decay modes based on 2b + 2τ and 2b + 1τ analyses are shown in orange and purple.
For comparison, we also show the corresponding sensitivities obtained in ref. [2] in the green regions. To obtain the sensitivity to BR(b 4 → Hb) we have extended the analysis to account for cascade decays through BR(H → tt). Thus, pair production of b 4 decaying to neutral Higgses can result in 6b, 4b2t, and 2b4t final states covering large, medium, and small tan β regions respectively. We show the envelope of the maximum reach among these three individual channels with the thin green line in figure 9 (right). It should be noted that all of these decays would simultaneously contribute to the 6b final state studied in ref. [2] (where we use the 5b analysis based on tagging at least 5 b-jets and top quarks are identified as b-jets through their decays to W ± b). Thus, including the regions of parameters where the events from each channel are combined results in the sensitivity to BR(b 4 → Hb) for masses of b 4 and H up to 2.3 and 2.1 TeV, which is largely independent of tan β (solid green) except for small tan β 4 region. Similarly, the sensitivities to decays of t 4 and b 4 through the charged Higgs extends to vectorlike quark (Higgs) masses up to 2.2 (2) TeV which is also largely independent of tan β.
When t 4 and b 4 are doublet-like, it is expected that they have almost the same masses and would be produced simultaneously, effectively doubling the production cross section of the signal. Allowing for all possible decay modes of the heavy Higgses, we see that the BR(b 4 → Hb) and BR(t 4 → H ± b) decays provide the strongest reach. The reach of the signal when these channels are combined is shown in the light blue region and denoted by the mixed scenario in the legend. Thus, for doublet-like t 4 and b 4 the HL-LHC will have sensitivity to vectorlike quark (Higgs) masses up to 2.4 (2.2) TeV for tan β 4. However, even for tan β 1 this search strategy has sensitivity to vectorlike quark    GeV. It should also be noted that while search strategies using decays of heavy Higgses to τ leptons are generally weaker, due to dramatically smaller branching ratios BR(H/A → τ + τ − ) 10%, the reach of the 2b + 2τ and 2b + 1τ analyses at tan β = 50 in the mixed scenario would be similar to the tan β = 1 curve in Fig. 10. A striking result shown in Fig. 10 is that even with currently available data, the discussed searches could lead to very strong limits on (or possibly the evidence of) vectorlike quarks and especially heavy Higgses for any tan β.

Conclusions
In this paper, we presented search strategies for new quarks decaying through charged and neutral heavy Higgs bosons to final states with τ leptons, and multiple b-jets. In extensions of type-II 2HDM, the decays of vectorlike quarks to heavy Higgses can dominate over those to SM bosons, and thus dramatically alter the projected limits on the masses of new quarks. Likewise, the production of heavy Higgses from decays of vectorlike quarks leads to QCDsize rates for signals of both heavy charged and neutral Higgs bosons. We motivate, in particular, final states with two or one τ tagged jets based on the pattern of branching ratios of heavy Higgses in type-II 2HDM. This strategy is simultaneously sensitive to multiple possible decay chains such as b 4 → Hb, b 4 → H ± t, and t 4 → H ± b. Decay modes of heavy Higgses to τ leptons typically offer much cleaner signatures at the LHC despite much weaker rates compared to, e.g. H(A) → bb. As our analyses are also based on tagging multiple b-jets, QCD-driven events, such as tt(jj), comprise a formidable source of SM background. However, the kinematic advantage of the heavy resonance in the decay chain allows to suppress the dominant backgrounds due to markedly different correlations between τ -and b-tagged jets. These search strategies would also be applicable to many models of new physics in which new quarks are pair produced at the LHC decaying to heavy bosons with couplings to third-generation leptons. We therefore encourage our experimental colleagues to search for the τ signatures with high H T .
The model-independent upper limits we find on the b 4 → Hb, b 4 → H ± t, and t 4 → H ± b decays show that searches focusing on τ leptons can probe vectorlike quark masses up to 1.8 TeV and 2 TeV at the HL-LHC when decaying to charged and neutral heavy Higgses, respectively. These limits imply sensitivity to charged and neutral Higgs masses to 1.6 TeV and 1.8 TeV. We find that these channels offer only comparable reach with respect to searches based only on b-tagging [2]. However, note that the utility of independent search channels would offer further insight to the heavy Higgs sector in the event of a discovery. In particular, while our model-independent limits are motivated by type-II 2HDM scenarios where heavy Higgses decay mostly to quarks, other 2HDM variants (or scenarios with heavy charged or neutral vector bosons) would include different hierarchies of branching ratios. The observation of vectorlike quarks in the absence of either τ -or only b-tagged channels could be used to disentangle the underlying dynamics of the Higgs sector itself.
Among our results, we recast the model independent bounds explored in this paper to sensitivities of vectorlike singlet and doublet quark masses in type-II 2HDM at the HL-LHC. We also present the corresponding sensitivity of the 5b analyses presented in ref [2] based on comprehensive searches of a 6b final state. Further, we extend the 5b analyses to include all sizable decay modes of charged and neutral heavy Higgses. For singlet-like quarks we find that the combined sensitivity to vectorlike quark and heavy Higgs masses extends up to 2.2 TeV and 2 TeV, respectively. The limits are improved when new quarks are doublet-like as both top-and bottom-like quarks are expected to have similar masses which would effectively double the rates we consider. In this case, we find sensitivity of the HL-LHC up to 2.4 (2.2) TeV for vectorlike quark (Higgs) masses.
We remark that the limits we obtain for new quark and heavy Higgs masses in the type-II 2HDM, both for singlet-and doublet-like b 4 quarks, are largely independent of tan β, a striking difference of the projected limits compared to those of searches for heavy Higgses without vectorlike quarks. This is mirrored also in the limits for doublet-like t 4 . Further, when the production of doublet-like t 4 and b 4 are combined, even the weakest sensitivity we obtain, M Q 2.1 TeV for tan β = 1, is still quite strong compared to existing searches. This implies robust sensitivities in searches for new quarks and heavy Higgs bosons, even with current data. TRIUMF receives federal funding via a contribution agreement with the National Research Council of Canada. SS acknowledges support from the National Research Foundation of Korea (NRF-2020R1I1A3072747).

A Poisson statistics
In this appendix we review our methodology to obtain upper limits on signal cross sections based on Poisson statistics. For the analyses based on a single final state, we derive an upper limit on the number of signal events when the expected number of background events, b, is known via [60] s up (α, n, b) = 1 2 F −1 χ 2 [p, 2(n + 1)] − b (A.1) where n is the total number of events observed, F χ 2 is the χ 2 cumulative distribution, and the confidence level of the resulting upper bound is given by (1 − α). In the absence of signal events, the median of upper limits can be calculated by sampling over the background distributions The upper limit on the signal cross section is then given by where L is the integrated luminosity and ε sig is the signal efficiency, optimized over the chosen kinematic cuts. We compare the derived σ up sig to our signal cross sections which are fixed for a given mass of t 4 or b 4 and corresponding branching ratios.
In our main results, we have calculated the statistical combination of vectorlike quark cascade decays contributing independently to the 2b + 2τ and 2b + 1τ final states. In general, assuming X independent final states (e.g. resulting from multiple decay channels of some particle(s)) the total probability of observing N = X k=1 n k events, where n k is the number of observed events in the k-th channel, given an expected number of background events b k for each final state is given by where s k is the expected number of signal events in the k-th channel. Following the same reasoning as in the case of a single channel, the upper limit on the signal cross section can be derived by calculating the confidence interval for X independent poisson distributions [61] ( where b k = σ B k L, s k = k σL and k is the signal efficiency in the k-th channel. We compute the upper limit on the cross section, σ up sig , numerically and compare this to the signal for a given vectorlike quark mass and cascade branching ratios through heavy Higgses.