Probing exotic triple Higgs couplings at the LHC

In extended Higgs sectors that exhibit alignment without decoupling, the additional scalars are allowed to have large couplings to the Standard Model Higgs. We show that current nonresonant di-Higgs searches can be straightforwardly adapted to look for additional Higgses in these scenarios, where pair production of non-SM Higgses can be enhanced. For concreteness, we study pair production of exotic Higgses in the context of an almost inert two Higgs doublet model, where alignment is explained through an approximate ℤ2 symmetry under which the additional scalars are odd. In this context, the smallness of the ℤ2 violating parameter suppresses single production of exotic Higgses, but it does not prevent a sizeable trilinear coupling hHH between the SM Higgs (h) and the additional states (H). We study the process pp → h* → HH in the final states bb¯bb¯,bb¯γγ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ b\overline{b}b\overline{b},b\overline{b}\gamma \gamma $$\end{document}, and multi-leptons. We find that at the HL-LHC these modes could be sensitive to masses of the additional neutral scalars in the range 130 GeV ≲ mH ≲ 290 GeV.


JHEP05(2020)087
1 Introduction The properties of the observed 125 GeV Higgs boson are in close agreement with the Standard Model (SM) predictions. Therefore, any extension of the SM Higgs sector needs to reproduce the SM-like Higgs couplings to the weak gauge bosons and fermions. In multi-Higgs sectors, one possible realization of this agreement is near the so-called "decoupling limit", where the physical masses of the additional Higgses are sent to infinity while keeping the remaining dimensionless self-couplings fixed [1,2]. Another possibility, the so-called "alignment limit", is that the vacuum expectation value (VEV) of one of the CP-even states is closely aligned with the direction of the VEVs in the multi-Higgs space [2][3][4][5]. In both limits the properties of one of the Higgses approach those of the SM Higgs. The decoupling limit implies the alignment limit, but alignment can also be realized without requiring the additional scalars to be much heavier than the electroweak scale. In this kind of scenario we focus on in this work.

JHEP05(2020)087
The agreement of the properties of the observed 125 GeV Higgs with the SM predictions can be easily satisfied in a setup where the observed Higgs boson comes mainly from a SM doublet H 1 plus a small deviation from it: (1.1) JHEP05(2020)087 Z 2 breaking terms will introduce a VEV H 2 ∼ v and induce O( ) suppressed couplings for interactions between one Z 2 -odd field and SM fields. The smallness of the Z 2 -violating couplings will not prevent a sizeable triple Higgs coupling g hHH , since the latter is dominated by the Z 2 -preserving quartics. Consequently, as mentioned before, the production mode of interest is di-Higgs production, pp → h * → HH, where H = A, φ, H ± . Since the coupling between H and SM fermions is suppressed by , the box diagram contributing to this process, dominated by a top loop, can be safely ignored. This leaves di-Higgs production via off-shell SM Higgs as the only important diagram, unlike for SM-Higgs pair production, where the interference between the box and the triangle diagrams is important. There are many CMS and ATLAS searches [25] targeting SM di-Higgs production via the final states bbbb [26,27], bbγγ [28][29][30], bbW W [31, 32], and bbτ τ [33,34], thus constraining the SM triple Higgs coupling g hhh . However, an enhanced g hHH coupling could also be constrained by these searches. Here we study the sensitivity of these analyses to the non-SM triple Higgs couplings g hHH .
Another reason to look for di-Higgs production via off-shell SM Higgs is that this production mode can be a complementary probe to electroweak pair production (EWPP) in searches looking for almost inert Higgs bosons [24]. Exotic Higgses produced through EWPP (via W or Z) are always distinct pairs of physical states: φH ± , AH ± , φA or H + H − . Therefore, if the neutral scalar φ is the lightest approximately Z 2 -odd state, pairs of φ cannot be produced directly via electroweak processes. In this case, pp → h * → φφ is kinematically favored and could become the most important production channel for exotic Higgses. Since pairs of carged Higgses H + H − can be produced through EWPP, in this work we focus on pair production of the neutral states (φ and A).
The paper is organized in the following way. In section 2 we present the model, that corresponds to a Type I 2HDM with small Z 2 -breaking terms, and discuss the production and decay modes of the approximately inert Higgses. In section 3 and 4 we find current constraints on g hφφ and g hAA coming from SM di-Higgs searches. Section 5 contains our conclusions and outlook.

Almost inert Higgs bosons
In this section we present the model and introduce the relevant parameters for the phenomenology. Consider two Higgs doublets that have the following Z 2 symmetry: The most general Z 2 -conserving potential is We assume that m 2 1 < 0 and m 2 2 > 0, such that in the exact Z 2 -symmetry limit, only H 1 gets a VEV. It is instructive to trade the parameters in the Higgs potential with the JHEP05(2020)087 phenomenological parameters. Assuming all the parameters are real (so there is no explicit CP violation), we trade the parameters in the potential for the following parameters: where v and m h are the VEV and mass of the SM Higgs (which resides in H 1 ), while m φ , m A , and m H + are the masses of the additional neutral scalar, pseudoscalar and charged scalar, all of which reside in the inert doublet H 2 . This vacuum configuration does not break the symmetry H 2 → −H 2 , so the lightest Z 2 -odd particles are stable. This is also known as the Inert Doublet Model (IDM). A large mass splitting between m H ± and m A would result in large violations of the SU(2) custodial symmetry, which is very well constrained by electroweak precision tests. Therefore, in this work we assume m A = m H ± .

Di-Higgs production at the LHC
We investigate the non-SM di-Higgs production cross section, i.e. gg → h * → φφ(AA), in the exact Z 2 -symmetry limit (IDM). First, we extract the triple-Higgs coupling g hHH , where H = φ, A, H + . As discussed in the introduction, these couplings arise from Z 2preserving quartic terms in the Higgs potential: where the factor of 2 in g hAA and g hφφ will compensate the symmetry factor in the relevant terms of the Lagrangian, where identical particles appear. 1 To see what range of values λ 3 , λ 4 , and λ 5 can take, it is instructive to rewrite them as: (2.5) Therefore, in the IDM we have: where m 2 2 > 0. In the case where m 2 2 = 0, the masses of the exotic Higgses are solely generated by the Z 2 -preserving quartic couplings λ 3,4,5 upon EWSB. In particular, λ 3 scales with the mass squared of the additional scalars. We can see that even a mild mass hierarchy m φ,A,H ± m h produces λ 3 ∼ O(1) λ 1 , and consequently a sizeable di-Higgs production rate. This will partially compensate for the drop in the gluon parton distribution function in the large mass regime. As an example, figure 1 shows the cross section of gg → h * → φφ for m 2 2 = 0 (red solid) and m 2 2 = (100 GeV) 2 (blue solid). One can see that for different choices of m 2 2 , the cross section can change for orders of magnitude at a given mass. However, the coupling soon reaches 4π when m φ 400 GeV. Therefore, the theory reaches a non-perturbative regime and a leading order estimate of the cross section is no longer reliable. For this reason, we restrict ourselves to 125 GeV < m φ < 400 GeV. (2.7) For comparison, in figure 1 we also plot the cross section with the coupling set to be a constant equal to the SM triple Higgs coupling (black dashed). Its steep fall is due to the fast drop in the gluon parton distribution function.

Decay of approximately inert Higgs
In the IDM, the lightest Z 2 -odd neutral state can only appear as missing energy at the LHC. However, if the symmetry is not exact, it can decay within the detector, which is what we investigate in this section. Starting with the Higgs potential, eq. (2.2), one can introduce Z 2 breaking via including odd powers of H 1 : where the Z 2 -breaking terms are all of O( ) 1. Now the 'inert' doublet H 2 gets a VEV of O( ):

JHEP05(2020)087
The ratio of the VEVs of the two doublets defines the conventionally adopted parameter tan β in 2HDM. In the approximate Z 2 limit, it is given by v/ṽ 2 , and tend to be large. 2 The fields can be parametrized around their VEVs as . (2.10) All the physical fields receive an O( ) correction with respecto to their Z 2 -symmetric limit. The neutral pseudoscalar and charged components of the doublets mix to generate the physical fields A and H ± : , Note that A is related to the usual tan β parameter through A ≈ −1/ tan β. The physical scalar fields are where α is another conventionally adopted parameter in 2HDM. From the kinetic terms, we find how Z 2 -odd fields couple to SM fields: The limit V → 0 corresponds to the so-called alignment limit in 2HDM [2][3][4][5].
The Z 2 -breaking terms in the Higgs potential also induce O( ) Z 2 -violating terms in the Yukawa sector. In the exact Z 2 -symmetry limit, only H 1 can have Yukawa couplings, therefore, this corresponds to a Type I 2HDM. After Z 2 breaking, the approximate Z 2 -odd scalars couple to SM fermions via an O( ) VEV and/or mass mixing: (2.17) In summary, the following set of phenomenological parameters can be used to discuss the approximate Z 2 Higgses: Branching Ratio  Branching Ratio where we have left out λ 2 and ∆λ . Both terms have more than two powers of H 2 . Since H 2 =ṽ 2 ∼ O( ), these terms are not important at the tree level. Figure 2 shows examples of φ branching ratios. It can be seen that adjusting the relative size of V to A radically changes φ's dominant decays.
A special case: φ → γγ. If φ is the lightest approximate Z 2 -odd Higgs, in the limit that V → 0, A → 0, we expect that φ will never decay inside the detector. However, in this scenario, where the tree-level decays of φ are suppressed, one can no longer ignore loop JHEP05(2020)087 induced processes. In particular, the quartic interaction ∆λ |H 2 | 2 (H 1 H † 2 +h.c.) contributes a Z 2 -violating coupling g φH + H − ∼ ∆λ v. In the limit that ∆λ V ∼ A → 0, the dominant decay of φ can become γγ/γZ induced by an H + loop, with one caveat. The inclusion of a term like ∆λ |H 2 | 2 (H 1 H † 2 + h.c.) necessarily introduces the Z 2 -breaking term ∆m 2 H † 1 H 2 back at one loop. If Λ is the UV cutoff of the model, ∆m 2 can be estimated to be ∼ ∆λ 16π 2 Λ 2 . From eqs. (2.9) and (2.12), we can see that the limit ∆λ Figure 2d shows an example of the branching ratios of φ in this region of the parameter space. In the subplot, we also show the total width of φ. Therefore, even in the limit that V ∼ A → 0, φ can still decay within the detector. More importantly, it decays with a spectacular signal involving multiple photons, which have much smaller SM backgrounds.

Constraining hφφ coupling
In this section, we begin detailed benchmark studies under the assumption of different schemes of Z 2 -breaking parameters, as studied in section 2.2. Even though all the Z 2breaking parameters are assumed to be small, the main decay modes of φ can change drastically depending on the relative size between them (cf. figure 2). Since there are no a priori assumptions on the origin of the Z 2 breaking, we have to study all possible qualitatively-different decays of φ at the LHC. Figure 2 shows the main decay modes for different regions of the parameter space. Each of the following subsections focus on a scenario where φ dominantly decays to only one or two types of SM particles, based on which, limits of g hφφ as a function of mass are drawn. If the LHC is insensitive to such ideal scenarios, we may conclude that it is not going to be sensitive to g hφφ in a more general scheme.
For all the following studies, we generate the parton level events for signal with Mad-Graph5 [36], followed by hadronization and showering by Pythia8 [37], and detector simulation with Delphes3 [38]. Each subsection assumes a distinct relative size between V and A and discusses possible constraints on g hφφ arising from the dominant decays.

3.1
A V : φφ → (bb)(bb) As shown in figure 2b, just like the SM Higgs, φ can dominantly decay to a pair of b quarks. The signature of interest here is four b jets. Since both ATLAS and CMS have performed SM di-Higgs searches in this channel [26,39], our goal is to see whether these standard searches can be sensitive to the almost inert heavy Higgses. We follow the selection and cuts from the ATLAS di-Higgs search at 36 fb −1 [26], excepting one cut involving the btagging score. The interested reader can find the details of these cuts in appendix A.1. The right panel of figure 3 shows the efficiencies of both signal reconstruction and final selection, where reconstruction simply requires 4 b-tagged jets with p T > 40 GeV.
The SM backgrounds for this channel are multijets, hadronic tt and semi-leptonic tt. Both CMS and ATLAS use mainly data-driven methods to model them. Therefore, to draw limits on coupling vs. mass for higher luminosities, we simply extrapolate the backgrounds at 36 fb −1 from ref. [26]. This is a conservative estimate because a real analysis is expected JHEP05(2020)087 4b, 300/fb (95% CL) 4b, 3000/fb (95% CL) to be more efficient reducing backgrounds for higher values of m φ . It is clear from figure 3 that the 4b channel of the SM di-Higgs searches can be important in constraining g hφφ , even though they are not optimised to target an exotic Higgs.

3.2
As shown in section 2.2, the coupling of φ to SM fermions, g φff , is proportional to the difference between V and A . Since these two parameters are independent, there is a scenario (figure 2a) where they cancel each other out, leaving φ mainly decaying to W + W − . Out of all the possibilities for the 4W decays, we focus on the final state with 3 leptons and 2 jets. This is a relatively clean signature, where the additional jets allow triggering. However, one cannot easily reconstruct the decay chain, therefore this search reduces to lepton counting, which can be potentially covered by several multi-lepton searches at ATLAS and CMS, for example [40,41]. Our goal here is to investigate whether these standard multi-lepton searches can be sensitive to the almost inert Higgs φ. We follow the 36 fb −1 three lepton search adopted in [42], for which details of the cuts are spelled out in appendix A.2.
For this channel, the SM backgrounds are mainly non-prompt leptons, diboson and ttV . Again, to draw 95% CL contours on coupling vs. mass for higher luminosities, we simply extrapolate the backgrounds at 36 fb −1 from ref. [42]. As can be seen in figure 4, this search starts being sensitive to g hφφ only at the high luminosity LHC.

∆λ
In the limit where ∆λ A ∼ V , tree level decays of φ are suppressed, so loop-mediated decays become important. The interaction ∆λ |H 2 | 2 (H † 1 H 2 + h.c.) contributes to the Z 2violating vertex g φH + H − ∼ v∆λ , allowing the decays φ → γγ and φ → Zγ. Therefore, the final states (γγ)(γγ) and (γγ)(Zγ) could be sensitive in this scenario, but the analysis would require an estimation of the backgrounds due to fake photons from multijets, which at the LHC with 300 fb −1 and HL-LHC (3000 fb −1 ). We also include the value of g hφφ /v in aIDM for m 2 2 = 0, which is given by 4m 2 φ /v 2 . Right panel shows the cut efficiency for the signals.
is beyond the scope of this work. Additionally, if A is small, but not zero, we could also have a sizeable decay to bb, as shown in figure 2d. In this scenario, di-Higgs searches in the bbγγ final state [28, 43] could be relevant. On the other hand, if A ∼ V → 0, depending on the value of ∆λ , the exotic Higgs could be long lived. In the latter case, delayed photon searches [44,45] could be sensitive, but these analyses cannot be straightforwardly recast, and a dedicated analysis, beyond the scope of this work, would be required. Therefore, for this scenario we focus on the prompt bbγγ final state.
In particular, we study the potential sensitivity to g hφφ of current ATLAS and CMS di-Higgs searches, in the bbγγ final state [28,43]. For concreteness, we consider the scenario depicted in figure 2d, where ∆λ = 0.1, V = 0 and A = 10 −4 .
We perform a Monte Carlo simulation of the signal, for different masses of the neutral scalar φ, and apply the cuts from ref. [43], excepting for a modification of the invariant mass cuts. Contrary to the cuts for the bbbb final state in the previous section, these cuts are optimized for the mass of the SM Higgs, so the efficiency drops rapidly as we depart from the SM Higgs mass (as shown in the right panel of figure 5). Therefore, we use the following invariant mass cuts: which reduce to the cuts from ref. [43] for m φ = 125 GeV. The right panel in figure 5 shows the total signal efficiency as a function of m φ , for m φ = 125 GeV (SM di-Higgs cuts) and for variable m φ (modified cuts). The efficiencies obtained from our Monte Carlo simulation have been scaled to match the efficiency obtained in ref. [43] for the point m φ = 125 GeV (≈ 2.89%).
Using the signal efficiency with modified cuts, and the branching fractions shown in figure 2d, we find 95% CL limits on g hφφ . To find these limits, we scale the backgrounds from the SM di-Higgs search from ref. [43]. This is a conservative estimate because the continuum JHEP05(2020)087 bbγγ, 300/fb (95% CL) bbγγ, 3000/fb (95% CL) bbγγ Signal Efficiency Figure 5. Left panel shows the upper limits on g hφφ /v from pp → h → φφ → (bb)(γγ) at the LHC with 300 fb −1 and HL-LHC (3000 fb −1 ). We also include the value of g hφφ /v in aIDM for m 2 2 = 0, which is given by 4m 2 φ /v 2 . Right panel shows the signal efficiency with the cuts from ref. [43] and using m φ -dependent invariant mass cuts. diphoton background is expected to be much smaller for higher invariant mass regions. We find that current di-Higgs searches in the bbγγ final state can be straightforwardly adapted to be sensitive to the exotic triple Higgs coupling g hφφ for m φ 250 GeV (300 fb −1 ) and m φ 300 GeV (3000 fb −1 ).

Constraining hAA coupling
The lightest almost inert neutral state can also be the pseudoscalar A. How A decays largely depends on the relative size between the Z 2 -breaking parameters A and V . For instance, as shown in figure 2c, for V ≈ A , the decay of A is dominated by bb for m A < 125 GeV and by Zh for m A > 125 GeV. Therefore, the 4b channel previously studied (section 3.1) also constrains g hAA . The new ingredient here is that for high masses of A, the pair produced As can undergo cascade decays to two Z bosons and two SM hs. If all of these particles are produced on shell, it is possible to reconstruct the entire decay chain.
Each subsection below assumes a distinct relative size between V and A , and discusses possible constraints on g hAA arising from the dominant decays.

4.1
A V : AA → (bb)(bb) As mentioned before, for a wide region of the parameter space, A → bb is one of the main decay modes of A. In general, this is the dominant decay mode (almost 100%) for A V . Also, for A ≈ V , this mode dominates for m A 215 GeV, as shown in figure 2c. Therefore, the limits obtained in figure 3 for g hφφ in the (bb)(bb) final state are also valid for g hAA .

4.2
A V : AA → (Zh)(Zh) The final states coming from a pair of cascade decays can be rich and complex. In order to reconstruct the decay chain, we require h → bb for both SM Higgses and at least one Z JHEP05(2020)087  Table 1. Signal cross section is given by where g hAA is taken to be 4m 2 A /v in the signal benchmarks. j = u, c, d, s, b, g.
boson undergoing leptonic decay. Depending on how the other Z boson decays, there arise multiple decay channels that can be potentially interesting. The cleanest channel is the one with both Z bosons decaying leptonically, but it suffers from a tiny signal rate. The second cleanest channel is when ZZ → ( ¯ )(νν), which has a same final state as the dileptonic decay of tth, where h → bb. There exist dedicated search on tth from CMS [46], where many of the search strategies can be migrated here. However, a crucial difference between tth and our model is that the former requires a Z veto which is undesirable in the latter case. In order to constrain the coupling from this channel, we estimate the SM backgrounds based on our own simulation. We also considered ZZ → ( ¯ )(jj), but the SM backgrounds such as Z+jets are difficult to reduce. Therefore, we focus on the final state ( ¯ bb)(ννbb), coming from the cascade decays, A → Zh, of pair produced As at the 13 TeV LHC. The dominating backgrounds include tt, single top and Z+jets, as well as other SM rare processes (cf. table 1). We first perform a detailed benchmark study, for which we generate the number of background events that is equivalent to an integrated luminosity of 300 fb −1 . Table 1 shows the cutflow for two signal benchmarks and the main SM backgrounds. For the pre-selection, we require 2 leptons and 4 b-tagged jets with standard p T and η cuts. To suppress Z+jets, we require / E T > 120 GeV. To suppress leptonically-decayed top backgrounds, we further require that the lepton pair has an invariant mass within the 15 GeV window around the Z boson mass (on-Z). Furthermore, we investigate M T 2 type of kinematic variables [47,48]. In particular, M T 2W , formed by two leptons plus / E T , and M T 2t , formed by two leptons and two leading b-tagged jets plus / E T , can be used to discriminate the di-leptonic tt. Even though the efficiency for pre-selection is 5% for signals, the cut efficiency is 50% in spite of different mass values.
From figure 6, we see that the HL-LHC will be sensitive in the region around 215 − 230 GeV. As well as for the other modes, this reach could be improved with a more sophisticated analysis, therefore this remains an interesting final state to look at. . We also include the value of g hAA /v in the aIDM for m 2 2 = 0, which is given by 4m 2 A /v 2 . Right panel shows the cut efficiency for the signals.

Results and conclusions
In this work, we assume a simple extended Higgs sector (2HDM) that exhibits alignment without the requirement for the additional scalars to be much heavier than the electroweak scale. A mild hierarchy (m φ,A,H ± m h ) allows the additional states to have large couplings to the Standard Model Higgs. We investigate the potential of the LHC for constraining the exotic triple Higgs couplings hHH, where h represents the SM Higgs and H the lightest exotic neutral Higgs. In particular, the exotic Higgs sector is assumed to be odd under an approximate Z 2 symmetry, such that single production of the additional states is highly suppressed, leaving pair production as the main production mechanism. Motivated by the non-decoupling feature exhibited by this scenario, where the triple Higgs coupling g hHH scales roughly as the mass squared of H, we look at the pair production of H via an off-shell SM Higgs: pp → h * → HH. This cross section does not drop as fast as the square of the gluon parton distribution function, because of the enhancement of the coupling in the high mass regime.
We identify the main decay modes of pair produced Hs by varying the size of the small Z 2 -breaking parameters V and A , that control the tree-level couplings of H to SM vector bosons and fermions, respectively. Furthermore, when both V and A become vanishingly small, the loop-induced di-photon decay becomes dominant for the lightest exotic neutral Higgs.
We find that the HL-LHC can be efficient in constraining g hHH in a non-decoupling extended Higgs sector. In particular, the most sensitive channel is 4b-jets when A > V , and 3 when A < V , both of which are covered by existing ATLAS and CMS searches. Figure 7 shows a summary of the regions, in the g hφφ vs. m φ plane, excluded by the different final states we analyzed. For bbbb and bbγγ, we extrapolated the backgrounds from the SM di-Higgs searches, that are not optimized for the heavy Higgses. Consequently, the limit JHEP05(2020)087 on the coupling obtained in these analyses is rather conservative, and has the potential to be considerably improved by a full optimized study.
In this work we have analyzed the scenario where the exotic Higgses decay promptly inside the detector. However, this is no longer valid if all the Z 2 -breaking parameters become O(10 −4 ), where searches based on displaced jets, leptons, or delayed photons become important. We leave this non-prompt scenario for future work, since it features a very distinctive phenomenology and requires different search strategies.