Higgs boson pair productions in the Georgi-Machacek model at the LHC

Higgs bosons pair production is well known for its sensitivity to probing the sign and size of Higgs boson self coupling, providing a way to determine whether there is an extended Higgs sector. The Georgi-Machacek (GM) model extends the Standard Model (SM) with an $SU(2)_L$ triplet scalar field that has one real and one complex components. The Higgs self coupling now has a wider range than that in the SM, with even the possibility of a sign flip. The new heavy singlet Higgs boson $H^{0}_{1}$ can contribute to s-channel production of the $hh$ pairs. In this work, we study non-resonant/resonant Higgs boson pair productions $p p \rightarrow hh$ and $p p \rightarrow H^{0}_{1} \rightarrow hh$, focusing exclusively on the contribution of $H^0_1$. We show the sensitivity for Higgs boson pair production searches at the 13-TeV LHC with the luminosities of $3.2,\ 30$ and $100$~fb$^{-1}$.


I. INTRODUCTION
After the discovery of Higgs boson at the LHC [1,2], couplings of the Higgs boson to certain other Standard Model (SM) particles have been measured and the best fit is performed with the result very close to the SM expectation [3]. However, the Higgs boson self coupling, a key parameter to test the structure of Higgs potential and electroweak symmetry breaking, has not yet been measured. At the LHC, Higgs boson pair production is known to be the primary process where one can use to determine this coupling [4][5][6][7][8][9].
Nonetheless, it is expected to be a challenging measurement due to its low production cross section predicted in the SM, σ(pp → hh) SM ∼ 40 fb at the 14-TeV LHC [10][11][12][13]. In the SM, tree-level Higgs trilinear and quartic self couplings are given as where m h is the Higgs boson mass, and are related by a factor of the vacuum expectation value (VEV) v = 246 GeV.
Physics beyond the SM (BSM) can easily affect the Higgs pair production cross section at the LHC through either modification in the top Yukawa coupling and/or new colored particles running in the triangle and box loops (non-resonance effects), or the existence of new heavy scalars decaying into Higgs pairs (resonance effect). The enhancement in production cross section can reach a few orders of magnitude in some cases [14][15][16][17][18][19]. Currently, the ATLAS and CMS Collaborations have imposed upper limits on the production cross section (bbγγ) and production cross section times branching ratios (4b, γγW W * and τ τ bb) with various categories of signal final states in Higgs pair searches at the 13-TeV LHC [20][21][22][23][24][25][26]: 3.9 pb, 330 fb, 25 pb and 508 fb for the γγbb, 4b, γγW W * and τ τ bb channels, respectively.
The Georgi-Machacek (GM) model, proposed in the mid 1980s [27,28], provides a good way to generate Majorana mass for neutrinos through the type-II seesaw mechanism while preserving the custodial symmetry at tree level. In addition to the SM-like Higgs boson h, the extended Higgs sector has another three neutral scalars, among which two are CP-even (H 0 1 and H 0 5 ) while the other is CP-odd (H 0 3 ), where the subscripts denotes their representations under SU (2) L . One distinctive feature of this model is that the couplings between h and the SM weak gauge bosons, g hV V , can be larger than their SM values. Phenomenology of this and similar models, including their supersymmetric and dark matter extensions, at both hadron and lepton colliders have been extensively studied [19,.
With the GM scalars also in the Higgs potential, the SM-like Higgs trilinear coupling and its couplings to the SM fermions are modified, with the possibility of enhancing the non-resonant Higgs boson pair production cross section. Furthermore, H 0 1 can also mediate the Higgs boson pair production, and virtually the gg → H 0 1 → hh channel dominates at the LHC when H 0 1 can be produced on shell. Constraints on the GM model have already been studied from unitarity of scalar field scattering amplitudes, tree-level stability of the Higgs potential, and Higgs boson precision measurements [29][30][31][32]. The most stringent constraint allows only a small window in the interaction between the Higgs boson and weak gauge bosons κ V ≡ g hW W g SM hW W = 0.94 +0.11 −0.12 [52]. Ref. [31] studied the constraints on the α-v ∆ plane using a χ 2 fit to the data of Higgs boson production at LHC Run-I, including both gluon-gluon fusion (GGF) and vector boson fusion processes with the tree-dominated bb, τ + τ − , ZZ and W W decay channels. Within the 2σ contour, the mixing angle α and the VEV of the Higgs triplet field v ∆ are found to roughly fall within the following ranges: −50 ○ ≲ α ≲ 40 ○ and 0 ≤ v ∆ ≲ 50 GeV, as shown explicitly in Fig. 1 of Ref. [31]. In this work, we will focus on the 125-GeV Higgs boson pair production via the non-resonant pp → hh channel and the resonant pp → H 0 1 → hh channel in GM model. The rest of this paper is organized as follows. In the Section II, we review the GM model and show the relevant couplings. The pair production of Higgs bosons in the model is discussed in Section III. Section IV shows our numerical results and direct search constraints from the 13-TeV LHC. Finally, we give a summary of our work in Section V.

II. GEORGI-MACHACEK MODEL
In the GM model, two SU (2) L triplet scalar fields, χ with hypercharge Y = 1 and ξ with Y = 0, are introduced to the Higgs sector in addition to the SU (2) L doublet Φ with Y = 1 2 already in the SM. In this paper, we use the convention that Q = T 3 + Y with Q and T 3 being the electric charge and the third component of the weak isospin, respectively. Writing in an where we use the following phase convention for the scalar field components: As in the SM, due to the instability of the Higgs potential, the neutral component of Φ spontaneously develops a VEV to break the electroweak symmetry and to induce VEVs for the neutral components of ∆. We can parameterise these neutral fields as where v φ , v χ and v ξ denote the VEVs of φ, χ and ξ, respectively. In the case of vacuum More explicitly, the Higgs potential in the GM model is given by where σ's and T 's are the 2 × 2 and 3 × 3 matrix representations of the SU (2) generators, After the SU (2) L × SU (2) R symmetry is broken down to the diagonal SU (2) L , the scalar fields in the GM model can be classified into different representations under the custodial symmetry transformation: Φ is decomposed into a 3-plet and a singlet and ∆ into a 5-plet, a 3-plet and a singlet. Among the neutral fields, we have two CP-even singlets H 1 Φ = φ r and H 1 ∆ = 1 3ξ r + 2 3χ r that mix through a mixing angle α to render two physical Higgs bosons: and one CP-even H 0 5 given by Here, we take h to be the SM-like Higgs boson of mass 125 GeV. The two CP-odd 3-plet fields mix via a mixing angle β to produce a physical H 0 3 = − cos βφ i +sin βχ i and a Goldstone boson that becomes the longitudinal component of the Z boson. Because of the custodial symmetry, the different charged states within each representation are almost degenerate in mass, subject to small mass splitting ∼ O(100) MeV due to electromagnetic corrections. In the following, we will ignore such small mass differences and denote the Higgs masses by m H 5 , m H 3 , m H 1 , and m h for the physical 5-plet, 3-plet, heavy singlet, and SM-like Higgs boson.
The five dimensionless scalar couplings λ 1 − λ 5 in the GM model can be expressed in terms of the physical Higgs masses and the mixing angles α and β as where c θ and s θ are abbreviations for cos θ and sin θ for θ = α, β, respectively, and M 1 and M 2 are defined as The Higgs boson trilinear self coupling in the model is therefore modified approximately as where g SM hhh denotes the SM Higgs triple coupling shown in Eq. (1). On the other hand, the coupling between one H 0 1 and two h is

Couplings of neutral Higgs bosons to fermions and gauge bosons relevant to this analysis
are expressed in terms of the corresponding SM values as:

III. HIGGS BOSON PAIR PRODUCTION
As shown in Fig. 1, SM-like Higgs boson pair production in the GM model at the LHC receives contributions from both non-resonant process (plot (a)), mainly through top and bottom quark loops, and resonant process through the heavy H 0 1 decay (plot (b)). The differential cross section for the process g( where and G are given in Appendix A.1 of Ref. [13]. More explicitly, In the following, we will focus in the scenario where m H 0 1 > 2m h and a pair of SM-like Higgs bosons can be produced via the production and decay of H 0 1 . In this case, we divide the total cross section into resonant and nonresonant contributions. For the resonant production of the Higgs boson pair, we employ the narrow width approximation and calculate the production cross section of H 0 1 , σ(gg → H 0 1 ), times its decay branching ratio to two Higgs bosons, BR(H 0 1 → hh). Consider the dominant H 0 1 production by GGF at the LHC 1 . Since the production of H 0 1 takes the same form as the SM Higgs boson production, the production cross section can be obtained by rescaling the result of SM Higgs boson with the modified Yukawa couplings and different masses. We then have the resonant production of Higgs boson pairs as In view of the scaling of couplings in different parts of Eq. (13), the nonresonant production cross section of a pair of Higgs boson can be parameterized as where we have removed the H 0 1 resonant production channel from the above expression to avoid double counting with Eq. (14). The coefficients c 1 = 0.263, c 2 = −1.310, c 3 = 2.047, and c 4 = −0.001 for √ s = 13 TeV. We also take a good approximation thatc 2 = c 2 when the production is off the resonance. Our estimates of resonant production cross section to be given in the next section are scaled from the GGF single Higgs boson production cross section calculated at NNLO+NNLL QCD+NLO EW [11]. The SM Higgs boson pair production appearing in Eq. (15) is calculated at NLO [12].
In this work, we use GMCALC [53] to calculate the Higgs mass spectrum, couplings and branching ratios in the GM model. Both theoretical and experimental constraints are taken into account, including tree-level unitarity, stability of Higgs potential, check of electroweak vacuum, and data of b → sγ and B 0 s → µ + µ − decays. We have scanned 140,000 points in the parameter space of −90 ○ < α < 90 ○ , 0 < v ∆ < 60 GeV and m H 0 1 ≲ 1000 GeV. We find that in a restricted region in the α-v ∆ plane m H 0 1 can be as heavy as 1 TeV, while most other space allows a maximum of around 700 GeV. It is a general feature that as H 0 1 becomes heavier, the range of BR(H 0 1 → hh) becomes narrower and closer to 1, meaning that a heavy H 0 1 preferentially decays to a pair of SM-like Higgs bosons.      , are larger in magnitude for benchmark points D, E, F and G. Combined with the sizeable decay branching ratio of H 0 1 → hh, the resonant production of SM-like Higgs boson pair can be significant. In the close-to-decoupling limit, (α, v ∆ ) = (−1, 1), the pair production of h becomes virtually the same as the SM prediction.
In addition to the couplings that are fixed by the chosen values of (α, v ∆ ) shown in Table I, the scalar self-couplings are also crucial for the production of hh pairs. We show in Before presenting our simulations, let us summarize the current situation of the search for Higgs boson pairs at the LHC. Here we only focus on the bbγγ and 4b final states since these two channels impose stronger constraints and are complementary when a resonance H 0 1 exists. The bbγγ channel serves as a good search channel in the lower mass regime as it has a cleaner signature, particularly for the non-resonant Higgs boson pair production in the SM. In the case of resonant production via a heavy resonance (M X ≳ 500 GeV), its efficiency becomes lower than the 4b channel. This is because the photon pair coming from the more boosted Higgs boson decay will be very collinear. Experimentally, separating the two photons in this case significantly lowers the efficiency.
At ATLAS, the search for a light H 0 1 with mass 275 GeV ≤ m H 0 1 ≤ 400 GeV is constrained by the bbγγ channel [5,22]. The efficiencies for signal events to pass the selection criteria are about 5 − 8%, depending on the mass of H 0 1 . It is shown that the distribution of invariant mass of the h pair, M hh , in the SM peaks around 400 GeV at the LHC [7], and the peak position does not shift much as the collision energy varies from 8 TeV to 100 TeV. Therefore, a light resonant can contribute to the h pair production rate through both interference effect and on-shell production.
The 4b search channel used by the ATLAS Collaboration [6,23], on the other hand, gives a cross section upper limit for a heavy scalar resonance in the mass range of 500 GeV ≤ m H 0 1 ≤ 1000 GeV using the resolved analysis, and 1000 GeV ≤ m H 0 1 ≤ 3000 GeV using the boosted analysis. The event selection efficiencies in the resolved analysis, where different cuts are applied for different masses of heavy resonance, are given by Mass (GeV) 500 600 700 800 900 1000 Efficiency [23] 0.95% 1.91% 2.55% 2.86% 3.14% 3.45% Here the calculation of efficiency assumes a 100% branching ratio for the heavy scalar resonance to a pair of SM-like Higgs bosons and a fixed total decay width of 1 GeV.
In our simulations, events of Higgs boson pair production are generated with the loopinduced mode in Madgraph5 aMC@NLO [55] with m h = 125 GeV. The model file is adopted from the model database of FeynRules [56,57]. The decays of Higgs boson into bb and γγ are performed with MadSpin [58]. The events are then passed to Pythia8 [59] for parton showering and hadronization, and the fast detector simulation in Delphes3 (ATLAS settings) [60] is used to include the detector effects. Finally, events are analyzed with MadAnalysis5 [61].
In the case of light H 0 1 in the mass range 250 GeV ≤ m H 0 1 ≤ 500 GeV, we follow the cuts used in the ATLAS bbγγ channel analysis [22]: Here and the following, N p refers to the number of particle p, P T (h) is the transverse momentum of particle or system h, the superscripts "lead" and "subl" denote respectively the leading and subleading jets, and M xx (x = b, γ) is the invariant mass of the system. The transverse momentum of the decaying h. Since the production of Higgs boson pair via a lighter resonance generally has less boosted h, the opening angle of the Higgs decay products tends to be wider in this case, as seen in both plots (b) and (c) of Fig. 5. It is also noted that the reason for the SM background to have smaller ∆R in these two plots is because the Higgs pair production mainly comes from the non-resonance production (i.e., the box diagram) that produces more Higgs bosons with larger p T .
In the case of heavy H 0 1 with mass larger than 500 GeV, the ATLAS 4b search using the resolved analysis is employed. We take benchmark point G as an example to show the distribution in the invariant mass M bbbb and that in ∆R of the second and third energetic b jets. The curves in the plots are the results after imposing the preselection cuts used by ATLAS for the 4b channel analysis: We observe that as m H 0 1 becomes heavier, the peak in the distribution of M bbbb becomes broader as its total width gets bigger. The ∆R distribution also moves to smaller values, as expected. In order to make a comparison with experimental constraints measured by the ATLAS Collaboration, we further follow their analysis to impose the additional massdependent cuts in our numerical simulations: The efficiencies for different masses of H 0 1 and the decay branching ratio to hh for benchmark points E and G are listed in Table II the resonant Higgs pair production rate via GGF (and thus the branching ratio of H 0 1 → hh), whose value is also given in the table. The efficiency for the bbγγ channel in the SM is also given for a comparison. The efficiency for our cases depends on both the mass of H 0 1 , its production rate, and its branching ratio to a pair of SM-like Higgs bosons. For the bbγγ channel in the lower mass regime, the experimental cuts are designed to be optimal for the non-resonant production that is peaked around 400 GeV. Therefore, we find that the efficiency in benchmark point E reduces as m H 0 1 becomes smaller. For the 4b channel in the higher mass regime, on the other hand, the cuts are designed for resonant production and will cut away non-resonant events if m H 0 1 is sufficiently large. For each benchmark point set, we have scanned 3000 points 2 . Most of the parameter space in benchmark points D, E, F, and G predict larger cross sections at the level of a few picobarns, in comparison with the other benchmark points. This is because the Higgs boson trilinear coupling g hhh in these four benchmark points can go negative, resulting in a constructive interference between the box and triangle Feynman diagrams in Fig. 1. It is noted that at the same time in these benchmark points, g H 0 1 hh is also negative, resulting in destructive interference to cancel part of the aforementioned constructive interference. The left plot shows scattered points for all the benchmark points in the mass range of 250 GeV 2 Note that if we sample more points, the cross section ranges may only go slightly wider. ≤ m H 0 1 ≤ 500 GeV. The right plot shows scattered points for benchmark points C and G in the mass range of 500 GeV ≤ m H 0 1 ≤ 1 TeV as only they allow larger m H 0 1 among the benchmark points considered here.
We also show the current constraints (red solid curves) on the searches for H 0 1 from the γγbb channel [22] and the 4b channel [23] done by the ATLAS Collaboration using the 3.2 fb −1 dataset at the 13-TeV LHC. As a comparison, we also show the constraints (blue curves) of the corresponding searches from LHC Run-I [20] after taking into account the acceptances and rescaling of the parton luminosity. It is seen that benchmark point E is close to the constraint of the γγbb channel. The parameter space of 500 GeV ≲ M H 0 1 ≲ 650 GeV for benchmark point G is already excluded by the 4b channel search. We also estimate the projected exclusion limits (red dashed curves for an integrated luminosity of 30 fb −1 and red dotted curves for 100 fb −1 ) when more data are collected. With 30 fb −1 , the LHC has the sensitivity to most of the parameter space with the H 0 1 mass heavier than twice the Higgs boson mass for benchmark points D, E, F and G. The parameter space of heavier H 0 1 with mass larger than 500 GeV for benchmark point C can be probed as well.
We note that the ATLAS γγbb and 4b constraints are rescaled with the efficiencies for benchmark points E and G, respectively (see Table II). Different benchmark points would have slightly different efficiencies. In addition to the current luminosity of 3.2 fb −1 (drawn in red solid curves), we also plot those for 30 fb −1 (red dashed curves) and 100 fb −1 (red dotted curves). Among the eight scenarios considered here, benchmark points E and G predict largest cross sections in the lower and higher mass regimes, respectively, and benchmark points C and G allow wider mass ranges for H 0 1 . The pink scattered points for benchmark point H have production rates approaching the SM prediction.

V. CONCLUSION
In this paper, we have studied in the Georgi-Machacek (GM) model the SM-like Higgs boson pair production through the gluon-gluon fusion (GGF) process at the 13-TeV LHC. We find that under various theory and experimental constraints, the Higgs boson couplings (self and with other SM particles) can have some deviations from the SM values. In particular, the model and current data even allow an interesting possibility that the Higgs boson selfcoupling g hhh can flip its sign from the SM value. In addition, the existence of the heavier Higgs singlet H 0 1 in the model gives an additional contribution to the di-Higgs production cross section through its mixing with the SM-like Higgs boson. The mass of H 0 1 can in some cases be as heavy as 1 TeV, especially in some parameter region with a negative mixing angle α.
When H 0 1 is sufficiently heavy to decay into a pair of SM-like Higgs bosons, the production rate can be significantly enhanced, particularly when the Higgs trilinear coupling g hhh becomes negative as constructive interference would occur. We also note that at the same time the other Higgs trilinear coupling g H 0 1 hh is also negative to result in a smaller destructive interference. For illustration purposes, we select eight benchmark points and perform a detailed numerical study. The Higgs boson pair production rate is estimated and compared with current and projected search bounds given by the ATLAS Collaboration. A couple of scenarios considered here can be probed or ruled out by the LHC experiments in the near future.