Properties and searches of the exotic neutral Higgs bosons in the Georgi-Machacek model

The Georgi-Machacek model predicts the existence of four neutral Higgs bosons, one of which can be identified as the 125-GeV Higgs boson. The latest Higgs data favor the parameter space of small mixing angle alpha between the two custodial singlets of the model. The other two neutral Higgs bosons belong respectively to the custodial triplet and quintet. We study the general decay and production properties of these particles in the small-alpha scenario. Constraints on the SU(2)_L triplet vacuum expectation value are obtained as a function of the exotic Higgs boson masses using latest ATLAS data of various search channels for additional neutral Higgs bosons.


I. INTRODUCTION
For almost half a century since its introduction to particle physics, the standard model (SM) has survived many precision tests. Over the years, we have attained to knowing sufficient details of its structure and parameters from experimental data. Despite its success in explaining countless empirical observations, the SM cannot be a UV complete theory. From a theoretical point of view, the model is unsatisfactory for various reasons. For example, the Higgs boson mass is found to be at the electroweak scale and its radiative correction poses a fine-tuning issue. There are also unanswered questions such as why fermions of different flavors have such a mass hierarchy and what is the origin of CP violation in the quark sector. Experimentally, we have found the phenomena of neutrino oscillations and dark matter/energy in the Universe that the SM falls short of.
The discovery of the SM-like Higgs boson at the mass of about 125 GeV not only completes the particle spectrum in the SM, but also stimulates studies about detailed properties of the particle and pursuits of an extended Higgs family. Among many new physics models with a larger Higgs sector, the Georgi-Machacek (GM) model [1,2] has received much attention in view of its intriguing features. The Higgs sector of GM model houses a complex triplet of hypercharge Y = 1 and a real triplet of Y = 0 under the SM SU (2) L × U (1) Y gauge symmetry, in addition to a SM-like doublet.
With vacuum alignment between the complex and real triplets, the model preserves the custodial symmetry at tree level, granting the possibility of a triplet vacuum expectation value (VEV) as large as up to a few tens of GeV. Naturalness associated with divergent one-loop corrections to the electroweak ρ parameter and certain mixings among the Higgs bosons have been studied and found to be similar to the SM Higgs mass [3]. The model predicts the existence of many Higgs bosons, forming two singlets, one triplet, and one quintet under the custodial symmetry [4]. There have been extensive phenomenological studies of the exotic Higgs bosons in the literature [5][6][7][8][9][10][11][12][13][14], including their effects in enhancing the strength of phase transition in electroweak baryogenesis [15].
Due to mixing between the doublet and the triplet fields, the coupling between the SM-like Higgs boson and the weak gauge bosons can be greater than the SM value [11,[16][17][18], which is impossible for models with only extra SU (2) L singlet and/or doublet fields. 1 Besides, the model has a flavoruniversal correction to the Yukawa couplings of the SM-like Higgs boson. Although some of the above-mentioned features are also shared by the septet extension model [19], the latter suffers from an accidental U (1) symmetry that could lead to an undesirable Nambu-Goldstone (NG) boson if it is broken spontaneously. There are also supersymmetric [20,21] and little Higgs extensions [22,23] of the GM model.
In this paper, we concentrate on the productions and decays of the neutral Higgs bosons in the GM model at the CERN Large Hadron Collider (LHC). After the discovery of the 125-GeV Higgs boson, it is of great interest to know whether there exists another neutral Higgs boson of higher or even lower mass. Therefore, continued efforts are made to search for such Higgs-like resonances within the full mass range explorable at the LHC, utilizing various major production and decay channels. Such searches offer the exclusion limit in each channel as a function of the Higgs mass.
In the GM model, there are totally four neutral Higgs bosons. We use h, H 0 1 to denote respectively the SM-like Higgs boson discovered at the LHC and the other Higgs singlet. Both of them are mixtures of the CP-even custodial singlets. The other two are the CP-odd H 0 3 belonging to the triplet and the CP-even H 0 5 in the quintet. Since h and H 0 1 are mixed states of the two singlets, they involve essentially the same production and decay processes. For definiteness in our analysis, we ignore the possibility of ϕ → ϕ ′ ϕ ′ , with ϕ denoting any of the other neutral Higgs bosons, because of the uncertainties in the triple Higgs couplings. Due to its odd CP property, the H 0 3 boson does not couple to the weak gauge bosons at tree level. Therefore, it can only be produced via the gluon-gluon fusion (GGF) and top associated production channels. Its decay modes do not include the weak boson pairs. For sufficiently high mass, in particular, it can also decay into hZ.
On the other hand, the H 0 5 boson comes purely from the weak isospin triplet fields and does not couple to the SM fermions at tree level. Consequently, it can only be produced through the vector boson fusion (VBF) and Higgs-strahlung production processes, and decay only to the final states of W W , ZZ, and γγ. It is our objectives in this work to study in detail the above-mentioned features of the neutral Higgs bosons in the GM model and to constrain the model using the available search data.
This paper is organized as follows. Section II briefly reviews the GM model and, in particular, provides the couplings of the neutral Higgs bosons with the SM fermions and weak gauge bosons.
We show how such couplings vary from their SM values as one adjusts the triplet VEV and the mixing angle α between the two singlets. In view of good agreement between the measured Higgs data and the SM predictions, we will focus on the region of small α. In Section III, we discuss the decay branching ratios and the signal strengths of different production channels for each of the neutral Higgs bosons in the model. Section IV discusses the constraints on the GM model based upon experimental searches of another SM-like Higgs bosons in various channels. This complements the other constraints from doubly-charged Higgs boson searches and indirectly from B physics and electroweak precision data. We summarize our analysis in Section V.

II. GEORGI-MACHACEK MODEL
The Higgs sector in the GM model comprises an isospin doublet field φ with Y = 1/2, a complex triplet field χ with Y = 1, and a real triplet field ξ with Y = 0, where the electric charge Q is given by Q = T 3 + Y with T 3 being the third isospin generator. Organized in an SU (2) L × SU (2) R covariant form, one has: where the phase convention for the component scalar fields is such that Moreover, the neutral components after electroweak symmetry breaking are parameterized as where v φ , v χ and v ξ denote the VEV's of φ, χ and ξ, respectively. The most general Higgs potential consistent with the SU (2) L × SU (2) R × U (1) Y symmetry has four dimensionful and five dimensionless parameters. Its explicit form can be found, for example, in Ref. [8].
When the VEV's of the two triplet fields are aligned: v χ = v ξ ≡ v 3 , the original SU (2) L ×SU (2) R symmetry in the Higgs potential of the model breaks down to the custodial SU (2) V symmetry. In this case, the masses of the W and Z bosons have exactly the same form as in the SM: M 2 W = g 2 v 2 /4 and M 2 Z = g 2 v 2 /(4 cos 2 θ W ), where θ W is the weak mixing angle and v 2 ≡ v 2 φ + 8v 2 3 = (246 GeV) 2 . Therefore, the electroweak ρ parameter keeps unity at tree level. Following the convention in Ref. [24], we define the ratio of the VEV's as Note that tan β is the reciprocal of tan θ H used in most other works and goes to infinity in the SM limit.
The component scalar fields can be classified into irreducible representations of SU (2) V multiplets: the Φ field is decomposed as 2 ⊗ 2 → 3 ⊕ 1, and the ∆ field as 3 ⊗ 3 → 5 ⊕ 3 ⊕ 1 after the SU (2) L ⊗ SU (2) R → SU (2) V symmetry breaking. As a result, we have one 5-plet, two 3-plet and two singlet representations under the custodial SU (2) V symmetry. The 5-plet, denoted by H 5 = (H ±± 5 , H ± 5 , H 0 5 ) T , arises within the ∆ field. The two 3-plet fields mix through the angle β to render a physical CP-odd Higgs 3-plet, denoted by H 3 = (H ± 3 , H 0 3 ) T , and another NG 3-plet, (G ± , G 0 ) T , to become the longitudinal components of the weak gauge bosons. The two CP-even singlet fields further mix by an angle α, determined by the quartic coupling constants in the Higgs potential, to produce the SM-like Higgs boson h and another physical singlet denoted by H 0 1 . Due to the custodial symmetry, Higgs bosons belonging to the same SU (2) V multiplet are degenerate in mass. Therefore, we will simply use M H 1,3,5 to denote the masses of H 0 1,3,5 , respectively. For later uses, we write out the physical neutral Higgs fields in terms of the original component scalar fields as follows: Here we identify the quantum of the h field as the 125-GeV Higgs boson found at the LHC. Note that, the H 0 3 does not couple to the weak gauge bosons at tree level, whereas the H 0 5 does not couple to the SM fermions.
Denote the couplings of a neutral Higgs boson ϕ (= h, H 0 1 , H 0 3 or H 0 5 ) to a SM fermion pair and a weak gauge boson in the model by g ϕf f and g ϕV V , respectively. Define the scaling factors κ F and κ V as where g SM hf f and g SM hV V refer to the corresponding values in the SM. The scaling factors for the neutral Higgs bosons in the GM model are summarized in Table I. Since H 3 is CP-odd, its coupling with fermions involves a γ 5 factor that is not reflected in the scaling factor given in the table.
As only the two custodial singlets in the model can possibly couple to both SM fermions and weak bosons simultaneously, we plot in Fig. 1 the contours of different pairs of scaling factors for the two custodial singlets. In Figs. 1(a) and 1(b), contours are shown on the κ V -κ F plane for h and H 0 1 , respectively, for various representative values of v 3 and α. In fact, the blue solid contours in both plots are identical as the scaling factors of h for the phase α are the same as those of H 0 1 for the phase α + π/2. This also explains that the fixed-α contours (black dotted ones) have the α ↔ α + π/2 correspondence between the two plots. Similarly, Fig. 1(c) shows contours on v 3 values in the two plots depict a series of circles concentric at the origin, of radii 1/ sin β and 1 + (5/3) cos 2 β, respectively. The only difference is that the black dotted contours of fixed α values are along the radial direction in the former, but spiral out in the latter.
Note that in the limit α = 0, meaning that the singlet from the Φ field does not mix with that from the ∆ field, the scaling factors (κ F , κ V ) = (1/ sin β, sin β) for h and (0, 8/3 cos β) for H 0 1 . Therefore, the two scaling factors of h are reciprocal to each other with κ F ≥ 1 and κ V ≤ 1, as indicated by the α = 0 contours in Fig. 1(a). The H 0 1 boson becomes fermiophobic but still allows a nonzero coupling with the weak gauge bosons, depending solely upon the value of v 3 , as shown by the horizontal dotted line in Fig. 1(b). In contrast, the couplings of H 0 5 to the weak bosons are purely proportional to v 3 , independent of α. H 0 3 also has couplings with the SM fermions as a result of mixing with the custodial triplet in the Φ field.
On the other hand, sin β → 1 in the limit of vanishing triplet VEV, v 3 = 0. In this case, (κ F , κ V ) = (cos α, cos α) for h and (sin α, sin α) for H 0 1 , shown as the two blue diagonal lines in Figs. 1(a) and 1(b), while both H 0 3 and H 0 5 totally decouple from the SM fermions and weak bosons.

III. BRANCHING RATIOS AND SIGNAL STRENGTHS
We discuss in this section patterns of the branching ratios and signal strengths for each of the neutral Higgs bosons in the model. In this paper, we define the signal strength of a particular Higgs production channel at LHC as the production rate in the GM model normalized to the SM production rate for a fictitious Higgs with the same mass: where X is some decay final state of ϕ, σ and B denote respectively the cross section and branching ratio, and the narrow width approximation for ϕ has been assumed. The black dotted curves are for α = −π/6, −π/12, −π/24, 0, π/24, π/12, and π/6 with varying β.

A. The h boson
Since the SM-like Higgs boson h generally has different couplings with SM particles from the SM expectation, the signal strengths of various search channels can be different from unity. Therefore, one could use the measured signal strengths to constrain the parameters α and β, as have been done in Ref. [11]. Instead of performing another global fit as in Ref. [11], here we want to study the effects of individual production channels. In Fig. 2, we plot the signal strength contours for the h boson in the model. Plot (a) shows those for the channels of γγ, ff , and V V * via the GGF and top associated (ttH) productions, and plot (b) for the same channels via the VBF and Higgs-strahlung (VH) productions. In both plots, we depict the contours for each signal strength µ = 1 by solid curves, as well as µ = 0.8 and 1.2 by dotted curves. We note in passing that we have ignored the effects of charged Higgs bosons in the h → γγ decays. This is justified in view of the current overall agreement of signal strengths with the SM expectations. Because of higher statistics, the signal strengths of ZZ * (→ 4ℓ, 2ℓ 2ν) and γγ channels in the GGF production are measured more accurately than in the VBF production. The latter production channels should have a much better precision at the high-luminosity LHC (HL-LHC). Due to better detector sensitivity, these bosonic channels are superior to the fermionic decay channels (bb and τ τ ). The τ τ mode in the VBF production has a relatively better sensitivity among the fermionic decays.
As shown in Fig. 2, the signal strengths of the bosonic channels for non-negligible v 3 have a strong dependence on α. Therefore, one can easily use their current values to exclude the parameter space of α +0.5. Moreover, the region for each 0.8 ≤ µ ≤ 1.2 have two branches at large v 3 . Thanks to the different shapes in different channels in both GGF and VBF productions, the region with α −0.5 in one of the branches can also be excluded. The remaining parameter space can be constrained effectively once we are able to measure the f f decay channels in the GGF process with sufficiently good precision, particularly the τ τ channel in the 1-jet category. In addition, if the signal strengths of the γγ and V V channels in the VBF and GGF processes can be separately measured to a good accuracy, it is also possible to constrain the remaining parameter space, especially in the large v 3 region.
In summary, the current Higgs data point us to the region of small mixing angle α. Therefore, we will focus our attention to such cases in the following analyses. More explicitly, we will consider the examples of α = −π/24 and −π/12 with v 3 = 10, 30, and 70 GeV. We also note that the H 0 1 → hh decay mode is not included in the calculation, because the relevant coupling, λ hhH 1 , is not determined without explicitly specifying the Higgs sector parameters.
Qualitatively, a nonzero λ hhH 1 coupling will result in a suppression of the regular search channels when M H 1 250 GeV, and may lead to more production of four-body final states. Owing to its small rate, the γZ decay channel is totally neglected here for simplicity. We now consider major production channels for the H 0 1 boson. We define the signal strength for each channel by normalizing its production rate in the GM to that for the Higgs boson in the SM but with a mass equal to M H 1 . Therefore, the signal strengths of the X channel from the GGF/ttH (referred to simply as GGF) and VBF/VH (referred to simply as VBF) productions can be respectively expressed as where B SM V (M H 1 ) and B SM F (M H 1 ) denote the inclusive branching ratios of a SM Higgs boson of mass M H 1 decaying into vector boson pairs and fermion pairs, respectively, and all the other modes (e.g., multi-particle and Zγ final states) are neglected in the last expressions.
The signal strengths of H 0 1 → ff, V V , and γγ are drawn respectively in the left, middle, and right columns of Fig. 4. For each channel, we further divide the plots according to the production mechanism to the GGF process (upper row) and the VBF process (lower row).
For small α, such as −π/24 and −π/12 in our examples, the scaling factor of Yukawa couplings with H 0 1 is proportional to sin α ∼ α. In this scenario, the denominators on the right-hand side of Eqs. (7) and (8) can be dominated by the inclusive bosonic branching ratio. This is the situation when v 3 is sufficiently large, as seen in Fig. 3. This leads to µ VBF factor is always an enhancement factor that is large when M H 1 2M W and the bosonic decay modes are subdominant, and gets close to unity when M H 1 2M W . This is seen in Fig. 4 by the decline in the curves for v 3 = 30 and 70 GeV. In contrast, the signal strengths through the GGF process µ GGF , which is smaller for larger v 3 values. Therefore, one observes this general trend in the plots except when v 3 = 10 GeV.
In the M H 1 2M W region, the predicted signal strengths are relatively flat until the tt channel opens up. This reflects the fact that the production rates vary in the same way as the SM Higgs boson in the higher mass region. Finally, it should be noted that the possibility of having the H 0 1 → hh decay is not considered here to avoid the uncertainty in the λ hhH 1 coupling. As we showed in Fig. 3, the γγ decay mode is important only in low mass region. In the right column, we find enhancements in the signal strength of γγ decay for both the VBF and GGF production channels. This enhancement is caused by the kinematical (off-shell) reduction in the V V decay channels, suppressed fermionic decays by the v 3 /v φ factor, and less destructive interference between the W and top loops. The enhancement is substantial for the VBF process, as the GGF process is suppressed by the fermionic scaling factor κ  Since we are mainly interested in |α| 0.5, the fermionic decay channels have less restricting power in this part of the parameter space. The bosonic decay modes in the GGF process are also insensitive to parameter variation within the region. Considering the region α > −0.5, the ZZ and W W decay channels in the VBF process have a potential to constrain the parameter space with v 3 40 GeV if the signal strength is found to be less than 0.05. The decay branching ratio of H 1 → γγ is smaller than 10 −3 , a rate that is difficult to probe in the VBF production. Note that the scaling factor for the couplings of H 0 3 to SM fermions is universally ± cot β, where the sign is determined by the sign of the third isospin number of the fermion. Being a CP-odd particle, its fermion pair decay width is enhanced from that of a CP-even Higgs boson by the factor Besides, the H 0 3 boson does not couple to the weak bosons at tree level and therefore has no weak boson pair decay modes. Moreover, its gg and γγ decays are mediated only by fermions in the loop. The particle has, however, a tree-level coupling with the SM-like Higgs boson and the Z boson. In fact, the hZ mode becomes dominant once the decay is on mass shell, until the tt decay is also on shell. The H 0 3 -h-Z coupling has dependences on both α and β (or v 3 ). The patterns of branching ratios as functions of the H 0 3 mass M H 3 are shown in Fig. 6. As shown in Fig. 6, the bb mode remains most dominant in the low-mass region up to the hZ decay threshold. When the tt decay is kinematically allowed, it becomes the most dominant mode.
The hZ decay mode is important once again in the further higher mass region. Incidentally, the branching ratio curve of the γγ mode has exactly the same behavior as the gg curve, but down by the factor of (Q t α EM /α S ) 2 ≃ 0.002, where α S and α EM denote respectively the strong and electromagnetic couplings and Q t = +2/3.
Since the H 0 3 does not couple to the weak gauge bosons, one can only make use of the ff modes through the GGF production mechanism. In such cases, the signal strengths for the on-shell decays in the Born approximation are where F S 1/2 (M ) and F A 1/2 (M ) are the fermionic loop functions for CP-even and -odd scalar bosons [26], respectively, and (κ H 3 F ) 2 = cot 2 β, which goes to zero in the small v 3 limit. Even though the above formula is only approximate, some general properties can be extracted. Since Since the H 0 5 boson does not couple to the SM fermions at tree level, the only search channels are the V V and γγ modes via the VBF production mechanism. Note that as given in Table I, both its couplings with the W and Z bosons scale with cos β (or equivalently v 3 ) without any dependence on α. But the latter is larger than the former by a factor of 2. The pattern of branching ratios is independent of both α and β (or v 3 ) and plotted as functions of the H 0 5 mass M H 5 in Fig. 8. The signal strengths of the ZZ, W W , and γγ modes through the VBF mechanism are plotted in Fig. 9. Note that there is no α dependence in the curves, and the v 3 dependence comes from the production part as the decay branching ratios are independent of both α and β. The VBF cross section given by the Higgs Cross Section Working Group [25] is the sum of W W fusion and ZZ fusion cross sections. In order to reflect their respective modifications in the production part, we have evaluated separately the W W fusion and ZZ fusion cross sections using MadGraph5 aMC@NLO [27]. The ratio is found to be about 3 : 1 for a wide range of M H 5 . As expected from the prediction of branching ratios, the signal strengths for V V and γγ decay channels are both enhanced substantially in the low mass region, M H 5 2M W . In analyzing the experimental data, we find that the ATLAS results are particularly convenient to use because they are all taken from the 8-TeV run with integrated luminosities about 20 fb −1 , whereas the CMS results mix measurements in both 7-TeV and 8-TeV runs. Therefore, we will use exclusively the ATLAS results in the following analysis. Up to now, the H → ZZ → 4ℓ mode is analyzed separately in both GGF and VBF+VH production processes [28], and the H → W W → eνµν mode in both GGF and VBF production processes [29]. The H → γγ mode is divided into low-mass (65-110 GeV) and high-mass (110-600 GeV) regions [30]. than the GGF process. This is consistent with the earlier observation that the H 0 1 boson is fermiophobic in the small α scenario considered here. Comparing Fig. 10(b) and Fig. 10(d), one notices that the ZZ channel is generally more constraining than the W W channel except for the region 375 M H 1 450 GeV, in which the former (latter) has a slightly worse (better) sensitivity experimentally.
In Fig. 11, we plot the upper limit of the triplet VEV as a function of M H 1 from the γγ decay mode. In order to compare with the experimental data, we need to have an efficiency factor C X [30], which is contained in the calculation of the fiducial cross section. The mass dependence of C X is not presented in Ref. [30]; only the range of 0.56 to 0.71 for the C X factor is mentioned. We here take C X = 0.56 to calculate a conservative limit on the triplet VEV. As shown in Fig. 11, the data are more (less) constraining for smaller α in the lower (higher) mass regime.
Finally, Fig. 12 shows the upper limit of the triplet VEV as a function of M H 5 from the ZZ → 4ℓ decay mode via the VBF mechanism. Clearly, this constraint from the search of H 0 5 is weaker than those presented in Figs. 10 and 11, where we focus on the small mixing scenario (−π/12 ≤ α ≤ 0). This is related to the fact that the signal strength is mainly enhanced in the low-mass region only.
We note in passing that no useful constraint can be obtained from the W W mode. Similar to H 0 1 → γγ, one can also extract constraints from the H 0 5 → γγ mode, as shown in Fig. 13. The parameters using the SM-like Higgs data, one can use the data of searching for additional neutral Higgs bosons at other masses to put constraints. Taking the cue from current SM-like Higgs data, we focus on the parameter space of small mixing angle α and study the decay branching ratios and signal strengths of these Higgs bosons at the CERN LHC. We have examined in this work how they vary with α, the triplet VEV v 3 , and the corresponding Higgs mass.
As a result of mixing with h, H 0 1 share all the decay modes of h. Moreover, in the small α scenario considered in this work, the weak boson decay modes are dominant in the high-mass region for large triplet VEV v 3 . Due to its odd CP property, the H 0 3 does not have the weak boson channels. Its dominant decay mode from the low-mass region up to about 600 GeV is bb, hZ, and tt. In the case of H 0 5 , it mainly decays to the ZZ, W W , and γγ states. In the high-mass region, the branching ratio of ZZ is about twice that of W W .
Using the latest search data of extra Higgs boson from ATLAS, we have put constraints on v 3 as a function of the Higgs mass for the H 0 1 and H 0 5 bosons. For H 0 1 , we have employed the ZZ → 4ℓ, W W → eνµν, and γγ modes through the gluon-gluon fusion (GGF) and vector boson fusion (VBF) productions. The constraints from the VBF mechanism are stronger than those from the GGF mechanism, and the results have dependence on the α angle. For H 0 5 , we have found that only the ZZ → 4ℓ mode is useful and renders a weaker constraint than those of H 0 1 . Nevertheless, this constraint is independent of α.
We have found that the fermionic channels of H 0 3 are enhanced a lot in the mass range between 2M W and 2M t . Therefore, a search of such channels in this regime can readily discover the particle or put stringent constraints on the model parameters.