Two Higgs doublet models augmented by a scalar color octet

The LHC is now studying in detail the couplings of the Higgs boson in order to determine if there is new physics. Many recent studies have examined the available fits to Higgs couplings from the perspective of constraining two Higgs doublet models (2HDM). In this paper we extend those studies to include constraints on the one loop couplings of the Higgs to gluons and photons. These couplings are particularly sensitive to the existence of new colored particles that are hard to detect otherwise and we use them to constrain a 2HDM augmented with a color-octet scalar, a possibility motivated by minimal flavor violation. We first study theoretical constraints on this model and then compare them with LHC measurements.


I. INTRODUCTION
Following up on their discovery of the Higgs boson with mass near 125 GeV [1,2], the ATLAS and CMS collaborations continue the detailed study of its properties. For example, the Higgs couplings to top, bottom and tau have been measured to be in agreement with the standard model (SM) although the errors are still large. Couplings to W W , ZZ as well as effective one-loop couplings h → gg and h → γγ are also well described by the SM [3]. However, present day uncertainties still allow for a variety of new physics possibilities. For example, when compared with two Higgs doublet models (2HDM), these measurements constrain the parameter space but do not exclude the possibility of additional scalars below 1 TeV [4][5][6][7][8][9][10][11][12][13][14]. Two Higgs doublet models can also be confronted with h → gg and h → γγ fits and this comparison restricts the allowed parameter space. Manohar and Wise (MW) [15] introduced a model consisting of the SM augmented by a colour octet electroweak doublet of scalars. The addition was motivated by minimal flavour violation: assuming that the scalars transform trivially under the flavour group, only electroweak doublets which are colour singlets or octets are allowed. These coloured scalars are very weakly constrained by direct searches at LHC but they can affect the loop induced Higgs couplings by factors of two. The model has been constrained theoretically and also using the h → gg and h → γγ fits with comparable results, and there are many phenomenological studies in the literature .
In this paper we combine these two extensions of the SM and consider a two Higgs doublet model with an additional scalar octet as in MW. The motivation for studying this model is that this is a simple extension of the SM that can satisfy minimal flavour violation. More complicated models exist that contain both of these ingredients [39,40], but our approach here is purely phenomenological. Our main goal is to explore the one-loop effective couplings h → gg and h → γγ of the SMlike Higgs in two Higgs doublet models in the presence of the additional scalar S transforming as (8, 2, 1/2) under the SM gauge group SU (3) C × SU (2) L × U (1) Y .

II. THE MODEL
The model we discuss in this paper is an extension of the type I and type II two Higgs doublet models. In this extension we add a colour octet electroweak doublet of scalars as in the MW [15] extension of the SM. The scalar content is chosen to satisfy desirable properties: minimal flavour violation which naturally suppresses flavour changing neutral currents and custodial symmetry which naturally preserves the relation ρ ≈ 1. As observed in Ref. [15], the only possible extensions of the scalar sector that do not transform under the flavour group and that satisfy minimal flavour violation are electroweak doublets that are colour singlets or colour octets and this motivates our choice for this model.
The scalar content of the model consists of two SU (2) scalars (Φ 1 , Φ 2 ) and one colour-octet scalar S. The general potential for (Φ 1 , Φ 2 ) is well known from the literature [71,72]. Our starting point will be more modest, consisting of the CP conserving, two Higgs doublet model with a discrete symmetry Φ 1 → −Φ 1 that is only violated softly by dimension two terms 1 To this starting block we can add the most general, renormalizable potential that describes the couplings of the colour octet S to the two colour singlets (Φ 1 , Φ 2 ) as well as the self interactions of the colour octet. This potential can be easily constructed by analogy with Ref. [15], changing the notation for couplings to accommodate the standard use in Eq. 1. The octet self interactions do not change, but we use µ 1−6 instead of λ 6−11 to label them, The interactions between each one of the two colour singlets and the colour octet also follow Ref. [15] but using ν 1−5 for Φ 1 or ω 1−5 for Φ 2 in place of λ 1−5 , Some of the couplings ν 3,4,5 and ω 3,4,5 can be complex and violate CP, but we will restrict our study to the CP conserving case. Finally, we have terms that involve both Φ 1 and Φ 2 as well as S, 2 in all cases we have explicitly shown the SU (2) indices i, j, S i = T A S A i , and the trace is taken over colour indices. The complete potential is thus, After symmetry breaking, this potential implies the following relations between couplings and scalar masses where λ 345 = λ 3 + λ 4 + λ 5 , and v 2 = v 2 1 + v 2 2 with v 1,2 the vevs of Φ 1,2 respectively. Similarly, for the colour octet sector we obtain The Yukawa couplings in this model consist of two types of terms that we can write as corresponding to the usual two Higgs doublet model couplings plus the interactions of the fermions with the colour octet. In the flavour eigenstate basis, they are where we have defined as usualH i = ε ij H * j for all three scalar doublets H = Φ 1,2 , S, S = T A S A , and α, β are flavour indices.

A. Minimal flavour Violation
To suppress flavour changing neutral currents in two Higgs doublet models, it is conventional to introduce discrete symmetries. For the Type I model, g D,U 1 = 0, while in the Type II model, g U 1 = g D 2 = 0. In the Yukawa terms, the type I model can be enforced with the discrete symmetry φ 1 → −φ 1 , whereas the type II model can be enforced with the discrete symmetry φ 1 → −φ 1 , d R → −d R [71]. We will instead follow Ref. [15] and enforce MFV, requiring that there be only two flavour symmetry breaking matrices G U transforming as (3 U ,3 Q ) under the flavour group and G D transforming as (3 D ,3 Q ) under the flavour group. The matrices appearing in Eq. 9 must satisfy where η D,U i , i = 1, 2, 3, are complex scalars. The two types of two Higgs doublet model under consideration are then defined by Requiring MFV instead of a discrete symmetry to define the models allows quartic terms in the scalar potential that are odd in either of the doublets. This justifies including the terms with coefficients ν 4,5 , ω 4,5 and κ 1,2,3 in Eqs. 3 and 4. One should note that in general, this also allows the additional terms in Eq. 1, We will not include these two terms in our numerical studies for ease in comparing with the usual definitions of these two types of 2HDM, and because our main new ingredient is the colour octet sector.

B. Custodial symmetry
To impose custodial symmetry conveniently, we follow the matrix formulation of Ref. [44] in which the scalar doublets are written as follows, and the custodial symmetry is imposed by writing the scalar potential directly in terms of O(4) invariants such as Φ †i Tr S † S . There are two methods proposed in the literature, • Case 1. Construction using only M 11 and M 22 . This yields the following constraints on the couplings of Eqs. 1-4: all the λ i are real and • Case 2. Construction using only M 12 yielding instead the constraints For the vacuum to be invariant as well one needs v 1 = v 2 .
An immediate consequence of custodial symmetry is that ∆ρ = 0 holds. The change induced in ∆ρ by the colour octet scalars is [15], Upon substitution of Eqs. 14 and 15 we find both sets of constraints result in ∆ρ = 0 as expected.
As is known, both cases also in mass degeneracies m H ± = m A and from Eqs. 6,7 they also result in m S ± = m S 0 I . The constrain v 1 = v 2 is too restrictive so we will only use the first method, Eq. 14 for our numerical study.
It has been pointed out before that it is also possible to satisfy ∆ρ = 0 with m H ± = m H [73,74] and with m S ± = m S 0 R [18], and that this follows from 'twisted' custodial symmetry.

III. UNITARITY AND STABILITY CONSTRAINTS
In this section we consider high energy two-to-two scalar scattering to constrain the strength of the self interactions with the requirement of perturbative unitarity. The potential is renormalizable and the tree-level scattering amplitudes approach a constant value at high energy proportional to the quartic couplings. Perturbative unitarity then constrains their size as it does for the Higgs boson mass [46]. These constraints have been previously applied to two Higgs doublet models [47][48][49][50], and to the Manohar-Wise model [31]. We extend them here to the combined model as described in the previous section, considering only the neutral, colour singlet amplitudes. We begin by defining the two particle state basis for the calculation of amplitudes, The unitarity constraints for the 2HDM without the coloured scalars are known from Ref. [47,49]. The two-to-two scattering matrix is a 14 × 14 matrix that can be diagonalized exactly producing the following eigenvalues (which we have simplified by setting λ 5 = λ 4 as per custodial symmetry), and have used the numbers in square brackets to denote the degeneracy of each particular eigenvalue. Unitarity constraints are obtained from the J = 0 partial waves of these two-to-two scattering amplitudes, by requiring that |a 0 | ≤ 1/2. This is equivalent to requiring that the largest eigenvalue in Eq. 18 be less than 8π.
In addition to the unitarity constraint, we also impose the known conditions for having a positive definite Higgs potential with a Z 2 symmetry [81], For phenomenological studies one prefers to control the scalar masses instead of the λ i couplings as input parameters via the relations Eq. 6. We will always identify the lightest neutral scalar h with the 125.6 GeV state found at LHC [1,2]. The other masses will be allowed to vary in ranges discussed later on, but we will always use λ s that ensure all the squared masses are positive and larger than around (400 GeV) 2 .
When we add the colour octet, the two-to-two scattering matrix becomes an 18 × 18 matrix which we diagonalize numerically. Unitarity constraints are obtained again from the J = 0 partial wave as in the case of the 2HDM. Approximate results in the custodial symmetry limit from 4 × 4 submatrices are, The couplings that affect only octet self-interactions at tree level, those in V (S) Eq. 2, have identical constraints as already found in Ref. [31]. In particular Eq. 3.9 of that reference (translated to the notation of this paper) is reproduced in our numerical diagonalization of the 18 × 18 matrix. Additional constraints obtained in Ref. [31] by studying unitarity in the colour octet channel are imposed on our entries and we quote them here for convenience, We illustrate the constraints resulting from perturbative unitarity in several figures to be described below.

IV. EXISTING LHC CONSTRAINTS
A. Tree-level Higgs decay The tree-level Higgs couplings to fermion pairs, in particular tt, bb and τ + τ − as well as the couplings to W and Z already constrain the parameter space of the 2HDM requiring it to be close to the SM. Allowed regions of parameter space under different scenarios have been presented recently for example in Ref. [6,7,14,68,70] and we do not repeat this exercise. The reader interested in the results of that global fit is referred to Figure 1 in Ref. [70], for example.
There are a few relevant comments to be made that are not apparent from the global fit. To this end we consider the results of the seven parameter fit to the Higgs couplings as per the ATLAS-CMS combination of data. We further consider their second scenario, in which contributions from BSM particles are allowed both in the loops and in the Higgs decay but κ V ≤ 1 is assumed. Those results, as listed on Table 14 of [3] are: Recalling that in 2HDM-I one sees that the b and t couplings to the Higgs from Eq. 23 are in tension within the 2HDM-1, being a bit more than 3σ away if one adds the two errors in quadrature.
To connect with the usual plot presented in the literature [14,68,70], we can do a simple fit to the 5 couplings in Eq.23, which we show in Figure 1. The left panel illustrates the same point as the best fit is closer to κ b and so is the 68% c.l. region enclosing the best fit point. The second dashed-green region is closer to κ t and one needs to go to a 95% c.l. to obtain a connected region which covers most of the parameter space. The addition of the colour octet cannot help address this problem as it does not affect the fermion Yukawa couplings at tree-level.
On the right panel we repeat the comparison for the type-II 2HDM. In this case there is a much smaller allowed region of parameter space but the goodness of the fit (as measured by χ 2 min ) is better than that for 2HDM-I. The blue contour is similar, but not identical, to that obtained in the literature from a direct global fit to LHC measurements. The slight shift of this region towards larger values of cos(β − α) is due to the small value of κ b and its small error in Eq. 23.
The values of κ Z = 1.00 −0.08 and κ W = 0.90 ± 0.09 in Eq. 23 prefer the region cos(β − α) near one, the so called alignment limit. In addition there are constraints from the non-observation of the additional Higgs bosons that are shown in Ref. [14], for example, and that we do not reproduce here. The constraints shown Figure 1 are not affected by the additional coloured scalars and should be identical to those obtained in the 2HDM if the same constraints are used. For this reason, they are not directly the concern of this paper.

B. Direct bounds on the colour octet
One would expect that the LHC can place stringent constraints on the existence of the additional colour scalars from their non-observation. It turns out however that the existing bounds are not very restrictive for this model, depending on the values of the couplings in the scalar potential the masses. The main reason is that the cross-sections for production of one or two such scalars are below current LHC sensitivity as can be ascertained by a quick glance at theoretical predictions [16,24] compared to those for coloured scalars that are currently constrained [82] and visa-vis LHC results [83,84]. Indirect constraints allow masses as low as ∼ 100 GeV [18].
The most important decays of the neutral scalars for example, would be into two jets or a tt pair. CMS limits on a colour-octet scalar S 0 from dijet final state quote M S < 3.1 TeV [83]. However, this is a gross overestimate for the MW model where the S 0 production cross-section is a few thousand times smaller than the model used by CMS. Similarly, bounds on Z resonances decaying to tt pairs [84] can be interpreted as posing no significant constraint for these scalars where σ S B(S → tt) ∼ 50 − 100 fb since their best sensitivity is to σ S B(S → tt) > ∼ 200 fb for the mass range studied (up to 2 TeV for narrow resonances and 3 TeV for wide resonances).
As already mentioned in Ref. [15,16] the cross sections for producing pairs of coloured scalars are larger than those for single scalar production for much of the parameter space. In this case the relevant constraints would arise from searches for dijet pairs and four top-quarks. Again the relevant quantity σ S Br 2 for this model is measured in fb whereas the published constraints are above this. Nonetheless, the dijet pair channel appears to be the most promising one to constrain this model and a detailed study will be forthcoming.
For our numerical study we will use two examples, one in which M S ± is set at 1 TeV and another one at 800 GeV. The couplings in the potential affecting Eq. 7 are constrained so that 725 ≤ M S 0 R ≤ 1200 GeV, and the custodial symmetry will ensure that M S 0 I = M S ± .

V. ONE-LOOP DECAYS OF NEUTRAL COLOUR-SINGLET SCALARS TO gg AND γγ
Finally we discuss the loop induced Higgs couplings where the colour-octet can play its most important role. Fits to the LHC Higgs data already exist in the literature and we use Ref. [80] for our discussion. It is standard to parameterize the one-loop results with effective operators for hgg and hγγ A general parametrization for couplings to the Higgs of different kinds of new particles such as a complex scalar S, a Dirac fermion f , and a charged and colourless vector V µ are They contribute to the effective Higgs coupling to gluons and to photons at one-loop as [85][86][87][88] where δc i = c i − c i,SM , C 2 (r) is the quadratic Casimir of the colour representation r, and N (r) is the number of colours of the representation r. A i (i = S, f, V , standing for scalar boson loop, fermion loop and vector boson loop, respectively) are loop functions, In terms of these general results and using we can write the effective one loop couplings. We begin quoting, for completeness, the amplitudes for these two processes within the SM [88], Similarly the one-loop γγ and gg couplings for the 2HDM neutral scalars are given by where t 1 = 1, t 2 = 0 for Type-I and t 1 = 0, t 2 = 1 for Type-II and H ± −λ 1 sin α sin 2 β cos β + λ 2 cos α sin β cos 2 β + λ 3 (cos α sin 3 β − sin α cos 3 β) − 2λ 4 cos(α + β) sin β cos β g HH ± = v 2 m 2 H ± λ 1 cos α sin 2 β cos β + λ 2 sin α sin β cos 2 β + λ 3 (cos α cos 3 β + sin α sin 3 β) − 2λ 4 sin(α + β) sin β cos β The top-quark and W -boson contributions to M (h → gg) and M (h → γγ) in the above expressions for the 2HDM, reduce to the SM in the limit β − α = π 2 . The colour octet scalars contribute the additional terms where where we have shown our results in the custodial SU (2) limit, and the total contributions for the models in this work are M 2HDM + M S .

VI. NUMERICAL STUDY
The model contains a large number of free parameters so we begin by presenting numbers for special values of masses to get a simple picture. We assume the lighter neutral CP-even Higgs h is the one discovered at LHC, and then compare the branching ratios to gg and γγ to the fit of Ref. [80]. We first set β − α = π 2 , m H ± = 600 GeV, m A = 500 GeV, m S ± = 800 GeV, ω 1,2 = 0, and use the Type II 2HDM. Ref. [14] provides a convenient form for scanning over input parameters for the 2HDM, which we adopt in this numerical study, we use input parameters Z 5,7 in place of m A and m 2 12 given by For this set of parameters we obtain the following constraints from unitarity, In addition the parameters ν 1 and ν 2 as well as ν 1 and κ 1 exhibit the correlated unitarity constraint shown in Figure 2. The allowed parameter region for this example in the tan β − Z 7 plane is shown in the left panel in Figure 2. From one-loop Higgs decays at 1σ we find |κ 1 | < ∼ 12.4 as well as the blue dotted areas in Figure 2. To illustrate the tree-level unitarity constraints implied by Eq. 18 and the constraints from the LHC data fit more generally, we randomly scanned the parameter space of the 2HDM (and its colour-octet extension) to find a set of allowed points. To produce these figures we have used the custodial symmetry results by Method I as in Eq. 14, including m H ± = m A . We have scanned over the range 600 ≤ M H ≤ 900 GeV. Our plots reproduce those of Ref. [14] for m H = 300, 600 GeV and we also find that the allowed region is reduced as m H increases. We further scan Z 5,7 over the ranges −10 ≤ Z 5 ≤ 2.5, −10 ≤ Z 7 ≤ 10. The upper bound on Z 5 arises from the requirement of m A being larger than about 400 GeV [89] 3 , and the lower bound keeps m A below around 1300 GeV. tan β is scanned over the range 0.2, 50 and cos(β − α) is scanned over (−0.5, 0.5). The charged Higgs mass is equal to m A and as calculated from Eq. 6, is found to lie in the range (400, 1200) for these parameter values. The independent parameters that involve the colour octet scalars in the SU (2) C limit are allowed to vary in the range −5π ≤ ν 1,2 , ω 1,2 , κ 1,2 ≤ 5π, to cover the region implied by Eq. 20c. The parameters that affect only colour-octet self interactions at tree-level, µ i are constrained by Eq. 21 (which we reproduce numerically by first setting a slightly larger range) and Eq. 22 which also constrains ν 4,5 , ω 4,5 which do not affect two-to-two scattering in the colour singlet zeroth partial wave. Finally, the mass M S ± is set to 1 TeV, which combined with the other parameters implies 725 ≤ M S 0 R ≤ 1200 GeV.

A. Two Higgs doublet model parameters
We reproduce the known shape of the region allowed by unitarity in the tan β − cos(β − α) plane [14] 4 : it is very narrow for tan β larger than about 10 as can be seen in Figure 3 and it gets smaller as M H increases, so that the red region shown is mostly determined by the value m H = 600 GeV, the lowest in our range. The same 3 Taking at face value the constraint from B → X s γ: m H ± ≥ 380 GeV. 4 We use the condition a 0 0 ≤ 1 2 instead of |a 0 0 | ≤ 1 figure shows that there is a small overlap between the regions allowed by unitarity (red) and those allowed by the effective loop decays of the Higgs (blue) in both type-I and type-II 2HDM but this overlap region is enlarged with the addition of the colour octet (green). However, the colour octet tends to populate regions that are not allowed by the tree-level unitarity constraints. Next, we illustrate in Figure 4 the two dimensional projections of the multidimensional region allowed by the tree-level unitarity constraints in the parameters of the 2HDM. The more significant correlation found is that between λ 3 and λ 4 . The darker regions in the plots reflect the concentration of points in the narrow region allowed in the tan β − cos(β − α) plane. We considered the question of overlap between the allowed regions in Figure 4 and additional constraints arising from the one-loop Higgs decays, and found that tree-level unitarity is more restrictive in all cases. We show in Figure 5 the region most constrained by h → gg and h → γγ. The two dimensional projections of the region allowed by tree-level unitarity for this sector are shown in Figure 6. The figures show approximate correlations of the form |2ν 1 + ν 2 | < ∼ 14, |2ω 1 + ω 2 | < ∼ 15 and |2κ 1 + κ 2 | < ∼ 11. In the same manner

C. Loop-induced Higgs decay
Now we present the points allowed by tree-level unitarity in a h → gg-h → γγ plot in Figure 8. The black contours are taken from Ref. [80] 5 and are respectively the 1σ and 2σ allowed regions, with the cross being the best fit point. The SM point is, of course, (1,1). On these contours we have overlaid the blue regions which consist of the points allowed by unitarity for the 2HDM parameter space, and the red regions corresponding to those allowed by unitarity for the 2HDM augmented by the colour-octet. The colour-octet extends the region which can be explained with a 2HDM mostly in the direction of a larger BR(h → gg). This figure does not give any insight into the values of different parameters in various regions of the plot. We have studied this issue by looking at all the possible correlations between pairs of parameters and the value of the (h → gg, h → γγ) point in Figure 8, but found no notable correlations beyond those already shown in Figure 7. Given the complexity of Eq. 34 this is not too surprising. One could also constrain the points illustrated in this figure by requiring them to lie within the 95% confidence level region of Figure 1. Since this is only an approximation to the global fit, it is easier to require instead that they satisfy −0.04 ≤ cos(β − α) ≤ 0.08 and 0.1 ≤ tan β ≤ 5, roughly mapping the region shown in Figure 1 of Ref. [70] for 2HDM-II. The result is indistinguishable from the red region already in Figure 8. These results illustrate how the loop induced Higgs decays are at present the best channels to constrain a Manohar-Wise type colour-octet.
We can consider the effect of the additional parameters from the colour-octet sector as follows. For each of the points in parameter space that satisfies the treelevel unitarity constraints we can compute two different points (h → gg, h → γγ). The first one would use the results of the 2HDM ignoring the additional contributions from the colour octet. These points are shown in blue in Figure 9. The second point (in red) is the one corresponding to the calculation in the full model, already shown in Figure 8. The region allowed by both tree-level unitarity and Higgs decays at one-loop can be used to predict the loop-induced decays of the heavier neutral scalars. As an example we show in Figure 10 the decay rates for the heavy neutral scalar of the 2HDM, H 0 , into two photons and two gluons.

VII. SUMMARY AND CONCLUSIONS
We have constructed an extension of 2HDM in which a colour-octet electroweakdoublet (MW) is added. Starting from the most general renormalizable scalar potential we have reduced the number of allowed terms with the usual theoretical requirements of minimal flavour violation and custodial symmetry. We have scanned the remaining parameter space to find the region which satisfies perturbative unitarity and have presented two dimensional projections of this region. The high energy two-to-two scattering matrix elements imply that correlations exist between certain pairs of the new couplings which are observed in these projections.
We have then confronted the model with available LHC results in the form of fitted couplings of the Higgs boson which we identify with the lightest scalar in the 2HDM. After collecting constraints on the parameters of the 2HDM from tree-level Higgs couplings we constrain the new sector couplings to the colour-octet using a current fit on the one loop h → γγ and h → gg couplings.
Addition of the colour-octet affects most the one loop h → γγ and h → gg modes where it enlarges the allowed region of parameter space in the tan β − cos(β − α) plane, but not notably in the overlap zone with tree-level unitarity constraints as seen in Figure 3. Of course, introducing a new colour-octet scalar doesn't populate more points in the unitarity allowed region when projected to the 2HDM parameter space.
The colour-octet also enlarges the region of overlap with the 1σ bounds h → γγ and h → gg, but the branching ratio of h → gg tends to increase more significantly than that of h → γγ as can be seen in Figure 8.
Finally we predict the one loop couplings of the heavier neutral scalar H → γγ and H → gg using the points in parameter space that satisfy all our constraints.