New decay modes of heavy Higgs bosons in a two Higgs doublet model with vectorlike leptons

In models with extended Higgs sector and additional matter fields, the decay modes of heavy Higgs bosons can be dominated by cascade decays through the new fermions rendering present search strategies ineffective. We investigate new decay topologies of heavy neutral Higgses in two Higgs doublet model with vectorlike leptons. We also discus constraints from existing searches and discovery prospects. Among the most interesting signatures are monojet, mono Z, mono Higgs, and Z and Higgs bosons produced with a pair of charged leptons.


Introduction
In models with extended Higgs sector and additional matter fields, the decay patterns of heavy Higgs bosons can be dramatically altered limiting the potential of present search strategies or rendering them nugatory. However, new search strategies can be designed that could lead to a simultaneous discovery of heavy Higgs bosons and matter particles, and that are more potent than separate searches for such particles. Such a situation occurs already in a two Higgs doublet model with vectorlike leptons. We consider an extension of the two Higgs doublet model type-II by vectorlike pairs of new leptons introduced in ref. [1]. In this model, the new leptons mix only with one family of standard model (SM) leptons and we will use the second family as an example. As a result of the mixing of new vectorlike leptons with leptons in the SM the flavor changing couplings of W , Z and Higgs bosons between heavy and light leptons are generated. These couplings allow new decay modes for heavy CP even (or CP odd) Higgs boson: H → ν 4 ν µ and H → e 4 µ, where e 4 and ν 4 are the lightest new charged and neutral leptons. These decay modes can be very large when the mass of the heavy Higgs boson is below the tt threshold and the light Higgs boson (h) is SM-like so that H → ZZ, W W are suppressed or not present. In this case, flavor changing decays H → ν 4 ν µ or H → e 4 µ compete only with H → bb and for sufficiently heavy H also with H → hh. Subsequent decay modes of e 4 and ν 4 : e 4 → W ν µ , e 4 → Zµ, e 4 → hµ and ν 4 → W µ, ν 4 → Zν µ , ν 4 → hν µ lead to the following 6 decay chains of the heavy Higgs boson: which are also depicted in figure 1. In addition, H could also decay into pairs of vectorlike leptons. This is however limited to smaller ranges for masses in which these decays are kinematically open. Moreover, the final states are the same as in pair production of vectorlike leptons. We will not consider these possibilities here. Finally, although we focus on the second family of SM leptons in final states, the modification for a different family of leptons or quarks is straightforward. We show that in a large range of the parameter space branching ratios for the decay modes (1.1) and (1.2) can be sizable or even dominant while satisfying constraints from searches for heavy Higgs bosons, pair production of vectorlike leptons [2] obtained from searches for anomalous production of multilepton events and constraints from precision electroweak (EW) observables [3]. Since the Higgs production cross section can be very large, for example the cross section for a 200 GeV Higgs boson at 8 TeV (13 TeV) LHC for tan β = 1 is 7pb (18pb) [4], the final states above can be produced in large numbers. Thus searching for these processes could lead to the simultaneous discovery of a new Higgs boson and a new lepton if they exist. Some of the decay modes in figure 1 also allow for full reconstruction of the masses of both new particles in the decay chain.
The final states of the processes (1.1) and (1.2) are the same as final states of pp → W W, ZZ, Zh production or H → W W, ZZ decays with one of the gauge bosons decaying into second generation of leptons. Since searching for leptons in final states is typically advantageous, our processes contribute to a variety of existing searches. Even searches for processes with fairly large cross sections can be significantly affected. For example, the contribution of pp → H → ν 4 ν µ → W µν µ to pp → W W can be close to current limits while satisfying the constraints from H → W W . This has been recently studied in ref. [1] JHEP05(2016)148 in the two Higgs doublet model we consider here, and also in a more model independent way in ref. [5]. However, the processes with tiny SM rates would be the best place to look for this scenario and here we will focus on such signatures. Examples of almost background free processes include H → hν µ ν µ and H → hµµ with h → γγ.
Vectorlike quarks and leptons near the electroweak scale provide a very rich phenomenology. For example, similar processes to (1.1) and (1.2) involving SM-like Higgs boson decaying into 2 2ν or 4 through a new lepton were previously studied in ref. [6] and the 4 case also in ref. [7]. An explanation of the muon g−2 anomaly with vectorlike leptons was studied in [8,9]. Vectorlike quarks and possibly Z offer possibilities to explain anomalies in Z-pole observables [10][11][12][13]. Extensions of the SM with complete vectorlike families were considered to provide an understanding of values of gauge couplings from IR fixed point behavior and threshold effects of vectorlike fermions [14,15]. Vectorlike quarks and leptons were also considered in supersymmetric framework, see for example refs. [16][17][18][19][20][21]. Further discussion and references can be found in a recent review [22]. This paper is organized as follows. In section 2 we briefly summarize the model, discuss constraints and present result for branching ratios of the heavy Higgs boson and new leptons. In section 3 we discuss relevant existing searches and the most promising search strategies for each of the six processes. We summarize and present concluding remarks in section 4.

Two Higgs doublet model with vectorlike leptons
In ref. [1] we introduced an explicit model consisting of a type-II two Higgs doublet model augmented by vectorlike pairs of new leptons: SU(2) doublets L L,R , SU(2) singlets E L,R and SM singlets N L,R , where the L L and E R have the same hypercharges as leptons in the SM. In order to avoid dangerous rates of lepton flavor changing transitions between the light leptons we further assume that the new leptons mix only with one family of SM leptons and we consider the mixing with the second family as an example. This can be achieved by requiring that the individual lepton number is an approximate symmetry (violated only by light neutrino masses). With these assumptions, one can write the most general renormalizable Lagrangian containing Yukawa and mass terms for the second generation of SM leptons and new vectorlike leptons.
After spontaneous symmetry breaking, , the model can be summarized by mass matrices in the charged lepton sector, with left-handed fields (μ L ,L − L ,Ē L ) on the left and right-handed fields (µ R , L − R , E R ) T on the right [9], handed fields (ν R = 0, L 0 R , N R ) T on the right [1], The superscripts on vectorlike fields represent the charged and the neutral components (we inserted ν R = 0 for the right-handed neutrino which is absent in our framework in order to keep the mass matrix 3 × 3 in complete analogy with the charged sector). The usual SM Yukawa coupling of the muon is denoted by y µ , the Yukawa couplings to H d are denoted by various λs, the Yukawa couplings to H u are denoted by various κs and finally the explicit mass terms for vectorlike leptons are given by M L,E,N . Note that explicit mass terms between SM and vectorlike fields (i.e.μ L L R andĒ L µ R ) can be rotated away. These mass matrices can be diagonalized by bi-unitary transformations and we label the two new charged and neutral mass eigenstates by e 4 , e 5 and ν 4 , ν 5 respectively: Since SU(2) singlets mix with SU(2) doublets, the couplings of all involved particles to the Z, W and Higgs bosons are in general modified. The flavor conserving couplings receive corrections and flavor changing couplings between the muon (or muon neutrino) and heavy leptons are generated. The relevant formulas for these couplings in terms of diagonalization matrices defined above can be found in refs. [1,9]. In the limit of small mixing, approximate analytic expressions for diagonalization matrices can be obtained which are often useful for understanding of numerical results. These are also given in refs. [1,9].

Branching ratios of the heavy Higgs boson
In order to focus on decay modes (1.1) or (1.2) we either allow mixing only in the neutral sector, κ couplings, or only in the charged sector, λ couplings. We further assume that the relevant lighter new lepton, ν 4 or e 4 , is heavier that m H /2 to avoid decays into pairs of new leptons and the heavier new lepton, ν 5 or e 5 , is heavier than H. Finally, we work in the limit with the light Higgs boson being fully SM-like and thus the heavy CP even Higgs H has no direct couplings to gauge bosons. We apply these requirements on randomly generated points in the parameter space specified by following ranges: where we focus on m H < 2m t in order to avoid H → tt and on the small tan β region where the heavy Higgs production cross section is the largest.

JHEP05(2016)148
We impose constraints from precision EW data related to the muon and muon neutrino: muon lifetime, Z-pole observables (Z partial width to µ + µ − , the invisible width, forwardbackward asymmetry, left-right asymmetry) and the W partial width; constraints from oblique corrections, namely from S and T parameters; and the LEP limit on the mass of a new charged lepton, 105 GeV. These constraints are obtained from ref. [3]. We also impose constraints on pair production of vectorlike leptons [2] obtained from searches for anomalous production of multilepton events. 1 In addition to constraints on new leptons we also impose constraints from searches for new Higgses: H → W W [24,25] and H → γγ [26]. Although the processes in (1.1) or (1.2) do not contribute to H → W W directly, they contribute to the same final states as obtained from decays of W bosons. In applying the constraints from H → W W we follow the analyses presented in refs. [5,27]. We implement the cut-based analysis using the data for eµν e ν µ final state in [24,25] but the results using µµν µ ν µ would be similar. Since our H has no direct coupling to the W boson, it is only the top quark and new charged leptons in loops that contribute to H → γγ. The top quark contribution to H → γγ scales as cot 2 β and thus this process is highly constraining the parameter space at very small tan β. In the usual two Higgs doublet model this was studied in ref. [28]. However, new charged leptons can enhance or partially cancel the contribution from the top quark depending on the signs of the new Yukawa couplings. Thus the allowed range of tan β is expected to extend to lower values compared to the usual two Higgs doublet model. Since new leptons also couple to the SM-like Higgs boson, the constraints from h → γγ have to be also satisfied. These at present still allow for a sizable new physics contribution [29].
Results of the scan in the case of mixing in the neutral lepton sector, allowing for H → ν 4 ν µ , are depicted in figure 2 in various planes of relevant branching ratios and tan β. The dark colored points satisfy all the limits summarized above. The gray points satisfy all the limits on new leptons but are excluded by H → γγ, W W and h → γγ. Finally, the light colors represent the points which are phenomenologically viable and satisfy H → γγ and h → γγ limits only if mixing in the charged lepton sector is simultaneously allowed which partially cancels the contribution from the top quark. In this case, for simplicity, we allow L and E to mix but not with the muon and we conservatively extend the ranges for λ andλ to [−1, 1]. We clearly see the lower bound on tan β 0.55. Without the mixing in the charged sector the lower limit on tan β moves to about 0.7.
Similarly, results of the scan in the case of mixing only in the charged lepton sector, allowing for H → e 4 µ, are given in figure 3 in the same planes and color scheme. In this case, the tan β dependence is not so significant because the new Yukawa coupling inducing H → e 4 µ scales with tan β in the same way as the bb coupling. The lowest possible value of tan β is 0.7 as it was in the H → ν 4 ν µ case. Even extending the ranges for λ andλ to [−1, 1], as indicated by light colors, only allows tan β 0.6.
For completeness we plot the same points in the plane of Higgs branching ratios for H → ν 4 ν µ and H → e 4 µ versus the Higgs mass in figure 4. Finally, although we focussed on the CP even Higgs, the results for CP odd Higgs would be qualitatively similar.

Branching ratios of the lightest neutral and charged leptons
The branching ratios of new lightest neutral and charged leptons decaying through Z and W bosons are shown in figure 5. Although not explicitly shown, the remaining branching ratio of the decay through the SM Higgs boson can be easily read out since only three decay modes are possible. In the left panels only constraints from EW precision data and direct searches are imposed. In the right panels we include the impact of constraints from searches for anomalous production of multilepton events [2]. In these plots, the colors indicate the doublet fractions of ν 4 or e 4 . The blue, cyan, magenta, and red points have doublet fractions in the ranges [95,100]%, [50,95]%, [5,50]%, and [0, 5]% respectively. The doublet fraction the ν a (a = 4, 5) is defined as and the doublet fraction of e a (a = 4, 5) is obtained by replacing the V L,R matrices by U L,R matrices in the formula above. Singlet fractions are given by 1 -(doublet fraction). We see that the multilepton searches are very constraining for doublet-like new leptons and do not even allow a doublet-like charged lepton in the mass range considered. Note however that in the case of charged leptons with large BR(e 4 → W ν) the constraints come from the pair production of ν 4 that accompanies doublet-like e 4 or from e 4 ν 4 production; the e 4 e 4 pair production is not directly constrained by multilepton searches. Without mixing in the neutral sector, BR(ν 4 → W µ) = 1 and this decay mode is highly constrained [2]. Allowing simultaneously full mixing in both charged and neutral sectors would relax this constraint somewhat.

JHEP05(2016)148
The main features of plots in figure 5 can be understood from analytic formulas for couplings that can be obtained in the limit of small mixing [1]. For singlet-like e 4 or ν 4 , the flavor changing couplings to W and Z have, in the leading order, the same dependence on parameters controlling the mixing. Thus this dependence disappears in the ratio and we find Γ(e 4 → W ν µ )/Γ(e 4 → Zµ) to be approximately 2:1 leading to the red bands with this slope.  andκ will result in an ellipse in the BR(ν 4 → Zν µ ) − BR(ν 4 → W µ) plane. The major axis of these ellipses is almost horizontal for M L v; for smaller M L the ellipses collapse onto the diagonal, which corresponds to BR(ν 4 → hν µ ) ∼ 0. This behavior can be seen in figures 3, 4 and 7 of ref. [1] for various choices of m ν 4 .
Finally, for the doublet-like e 4 , there are two couplings, λ L and λ E , connecting the muon to vectorlike fermions and thus controlling the overall strengths of flavor changing couplings to W , Z, and h. In result, there is significantly more freedom and generating couplings to W , Z, and h are less correlated.
In order to illuminate more subtle features of the scenario, we plot the same points in the planes of branching ratios versus the masses of ν 4 and e 4 in figure 6.

Signatures
In this section we discuss each of the novel heavy Higgs decay modes with details of the main features of each channel, of existing experimental searches to which these new process contribute, and of possible new searches. Considering the variety of possible final states, detailed Monte Carlo studies of these signatures are beyond the scope of this paper.
For estimates of rates of various processes we will use, as a reference point, the production cross section of a 200 GeV Higgs boson at 8 TeV LHC for tan β = 1 which is 7pb. The corresponding cross section at 13 TeV LHC is 18pb, and for different values of tan β these numbers should be divided by tan 2 β. Furthermore, we will assume BR(H → ν 4 ν) or BR(H → e 4 µ) to be 50% and branching ratios of ν 4 or e 4 relevant for a given process to be 100%. These branching ratios are close to the upper possible values allowed as we saw in the previous section. Note however that after imposing all constraints e 4 → Zµ branching ratios above 40%, while possible, are difficult to achieve.

H → W µν µ
The diagrams in figures 1a and 1d yield the W µν µ final state. A detailed study of these topologies with W → ν ( = e, µ) has been presented in refs. [1,5], where with focus on contributions to the existing pp → W W and pp → H → W W measurements. A crucial feature of the process in figure 1a is that the intermediate ν 4 → W µ decay, with a hadronically decaying W , allows the kinematic reconstruction of the neutral fermion ν 4 . Experimental studies of this mode can be affected by searches for semileptonic decays of a heavy Higgs (H → W W → νjj) which have been presented in refs. [30][31][32][33][34]. In these searches, the assumption that the observed missing transverse momentum is caused by a neutrino emitted in the W decay, is used to reconstruct the complete four-momentum of the neutrino; thus allowing the reconstruction of the Higgs mass (m 2 H = (p + p ν + p jj ) 2 ). The efficiency loss due to this reconstruction procedure is about 50%. However, in our case, the neutrino does not originate from a W and this procedure does not reconstruct the correct Higgs mass. An alternative approach is to consider the transverse mass m T = ( / E 2 T − / p 2 T ) 1/2 that is expected to have an edge at m H . Moreover the p T distribution of the neutrino is expected to be different because the latter does not originate from a W . Finally, for Higgs masses above 200 GeV our signal is potentially enhanced, with respect to the one studied in ref. [30], by the ratio of BR(H → ν 4 ν → W µν → jjµν) 0.5 × 1 × 0.  W 's decays). For W → τ ν our process is H → τ µνν and is constrained by searches for H → τ τ (with a leptonic tau) and H → τ µ.

H → Zν µνµ
The diagram in figure 1b leads to the Zνν final state. For Z → νν our process contributes to the jet plus missing E T signature, that is common to monojet searches for dark matter pair production or for invisible Higgs boson decays. For instance, in refs. [35,36] the JHEP05(2016)148 ATLAS and CMS collaborations placed upper limits on the visible cross section defined as the product of cross section, Monte Carlo acceptances and detector efficiencies, i.e. the observed number of events divided by the integrated luminosity. Requiring / E T > 150 GeV, the ATLAS limit is 726 fb which is larger than the typical values that we can obtain. For our reference point, even before requiring an extra high-p T jet and including acceptances and detector efficiencies, the total cross section is For leptonic Z decays, Z → , our signal contributes to the pp → ZZ → νν measurement [37] and to searches for H → ZZ → νν [38], mono-Z with missing energy, Z + / E T [39,40] and Zh(H) → + / E T [41]. For instance, the CMS search for Z + / E T [40] finds a limit on the visible cross section (for various / E T cuts) at the 1-2 fb level; in our case the m H = 200 GeV total cross section is up to σ(pp → H) × BR(H → ν 4 ν µ → Zν µ ν µ → ν µ ν µ ) (7 pb) × 0.5 × 1 × 0.06 ∼ 210 fb, before acceptances and efficiencies. Therefore, this channel is likely to offer the strongest constraint.
The hadronic Z → jj mode is less clean and has been studied in the context of the H → ZZ → jjνν presented in ref. [38].

H → Zµµ
The diagram in figure 1e leads to a Zµµ final state with two resonances corresponding to the Higgs, P H = p Z + p µ 1 + p µ 2 , and charged vectorlike lepton, P e 4 = p Z + p µ 1 (up to a dilution due to picking the wrong muon).
The most promising channel is H → Zµµ → µµ. A recent ATLAS search for e 4 → Z → 3 with the e 4 produced with another vectorlike lepton in a Drell-Yan process is presented in ref. [42] where the visible cross section into 4 , 3 + jj and 3 final states is found to be smaller than 1 fb. For our reference point, we obtain σ(pp → H) × BR(H → e 4 µ → Zµµ → µµ) (7 pb) × 0.5 × 1 × 0.06 ∼ 210 fb. Since our signal has a different underlying process and kinematics with respect to vectorlike leptons pair production, we expect our acceptance for the ATLAS search to be somewhat smaller than the quoted (20-50)% acceptance. In addition, one could additionally search for a fourth lepton and require one lepton pair to form a Z while imposing a Z veto on the second pair thus suppressing ZZ backgrounds; the four leptons invariant mass is also expected to peak at the heavy Higgs mass.
In ref. [43], CMS presented a study of a heavy SM-like Higgs decaying to ZZ and found that for Higgs masses smaller than about 500 GeV the 95% upper limit on the total cross section into four leptons is 0.1 × σ SM (pp → H → ZZ → 4 ). For m H = 200 GeV this corresponds to about 0.1 × (7 pb) × 0.255 × (3 × 0.03 2 ) ∼ 0.48 fb (the factor of 3 takes into account that only the 4µ and 2e2µ final states should be included), which has to be compared to our reference cross section of about 210 fb discussed above. Since these searches require two on-shell Z bosons compared to the single Z in our signal, we expect the Monte Carlo level acceptances for this process to be very small. A similar argument applies to pp → ZZ → 4 [44]. However, one can again impose a Z veto on two of the charged leptons, almost completely removing the SM background.

JHEP05(2016)148
The Z → jj mode is also interesting but it yields experimental bounds that are roughly an order of magnitude weaker than in the di-lepton case [43]. The invisible Z channel is problematic because it leads to a dimuon plus missing energy final state in which the two muons do not reconstruct a Z. This final state would also contribute to a W W search.

H → hµµ
This mode stems from the diagram in figure 1f and leads to large contributions to several very promising final states: (γγ) h µµ, (ZZ * ) h µµ → 6 , (W W * ) h µµ → 4 + MET and (bb) h µµ (the subscript indicates the particles whose invariant mass reconstruct the SM Higgs). Standard Model backgrounds to the first three modes are essentially absent making them golden channels to discover this process. In fact, the dominant SM backgrounds are pp → h(γ * , Z); moreover, the latter can be further suppressed by requiring the Z to be virtual by vetoing di-leptons with invariant mass close to m Z .

H → hνν
The SM Higgs plus missing energy signal, depicted in figure 1c, is also very interesting because it overlaps with dark matter searches.
The clear golden mode is the (γγ) h + / E T final state. A search for this signal has been performed in ref. [45] where a small excess of 3 events has been observed over (essentially) no background. In fact, the pp → Zh → ννγγ process has a total cross section of about 0.2 fb and, with a 10% acceptance, leads to about 0.5 events at 20 fb −1 . The cross section for our reference point is about σ(pp → H) × BR(H → ν 4 ν µ → hν µ ν µ → γγν µ ν µ ) (7 pb) × 0.5 × 1 × 3 · 10 −3 ∼ 10.5 fb. Assuming that our signal acceptance is identical to the acceptance used in this search (10%), we expect about 21 events with 20 fb −1 of integrated luminosity. Future updates of this search will certainly place interesting constraints on our model. In ref. [46] the pp → (Z → bb)(h → invisible) → bb + / E T was studied. Interestingly a small excess of 20 events is observed for m bb ∼ 125 GeV that would be compatible with our signal with h → bb. For our reference point and considering the h → bb decay of the SM Higgs we get a cross section of up to σ(pp → H) × BR(H → ν 4 ν µ → hν µ ν µ → bbν µ ν µ ) (7 pb) × 0.5 × 1 × 0.57 ∼ 2.0 pb. This is further reduced by the b tagging efficiency (0.7 2 = 0.5) and by the acceptance. In ref. [47], CMS presents a similar study but the di-jet invariant mass distribution after b-tagging is not presented; hence we do not know whether an excess at m bb = 125 GeV is seen.
In refs. [48] ATLAS performed a dedicated search for pp → (h → bb)Z/W . Events are classified according to the number of leptons in the final state: Z → νν (0-leptons), W → ν (1-lepton), Z → (2-leptons). The 0-lepton channel shares the final state with the search presented in ref. [46] (that we discussed above) and a small excess compatible

JHEP05(2016)148
with the one observed in that search has also been observed. A similar search has been performed by CMS in ref. [49].
A search for h → bb produced in association with dark matter has been presented in ref. [50] where, unfortunately, the bb invariant mass distribution is not shown.
Other interesting final states are pp → h + / E T → (µµ, τ τ, W W * , ZZ * ) + / E T . These modes are similar to the corresponding SM pp → h → (µµ, τ τ, W W * , ZZ * ) ones, albeit with sizable extra missing energy that further reduces all backgrounds.

Conclusions
In two Higgs doublet model type II with vectorlike leptons, the decay modes of heavy Higgs After applying constraints from precision electroweak observables, searches for heavy Higgs bosons and constraints on pair production of vectorlike leptons obtained from searches for anomalous production of multilepton events we found that branching ratios of H → ν 4 ν µ and H → e 4 µ, where e 4 and ν 4 are the lightest new charged and neutral leptons can be as large as 50%. These decay modes are especially relevant below the tt threshold and when the light Higgs boson (h) is SM-like so that H → ZZ, W W are suppressed or not present, competing only with H → bb and for sufficiently heavy H also with H → hh.
Furthermore, we found that each of the subsequent decay modes of e 4 and ν 4 : e 4 → W ν µ , e 4 → Zµ, e 4 → hµ and ν 4 → W µ, ν 4 → Zν µ , ν 4 → hν µ can be close to 100% providing many possible search opportunities. Among the most interesting signatures are monojet, mono Z, mono Higgs, and Z and Higgs bosons produced with a pair of charged leptons. Some of these signatures are almost background free. Combining this with potentially large production cross section for these processes presents great discover prospects at the LHC.