Stability and symmetry breaking in the general three-Higgs-doublet model

Stability, electroweak symmetry breaking, and the stationarity equations of the general three-Higgs-doublet model (3HDM) where all doublets carry the same hypercharge are discussed in detail. Employing the bilinear formalism the study of the 3HDM potential turns out to be straightforward. For the case that the potential leads to the physically relevant electroweak symmetry breaking we present explicit formulae for the masses of the physical Higgs bosons.


INTRODUCTION
T.D. Lee has shown decades ago that in the general two-Higgs-doublet model (THDM) CP violation is possible in the Higgs sector [1]. Meanwhile a lot of effort has been spent to investigate the THDM; see for instance the review [2] and references therein. In particular, some progress could be made employing the bilinear approach. The bilinears appear naturally in the Higgs potential in any n-Higgs doublet model (nHDM), since only the gauge-invariant scalar products of the Higgsboson doublet fields may appear in the potential. The bilinear formalism was developed in detail in [3,4] and independently in [5].
Initiated by these works, many aspects of the THDM and the general nHDM were considered within this formalism. For instance, CP-violation properties of the THDM were presented in [5,6]. Different symmetries of the THDM and the general nHDM were considered in some detail employing bilinears; see for instance [7][8][9][10][11][12] In this work we will focus on the three-Higgs-doublet model (3HDM). Many of the properties of this model are direct generalizations of the THDM, but there appear also new aspects. As we will see in detail, the space of Higgs-boson doublets does, in terms of bilinears, not correspond to the forward light cone space, as in case of the THDM [4], but to a certain subspace; see [5,6,13].
In an analogous way to the study of the THDM in [4] we will discuss stability, electroweak symmetry breaking, and the stationarity points of the potential for any 3HDM in the following.

BILINEARS
We consider the tree-level Higgs potential of models with three Higgs-boson doublets satisfying SU (2) L × U (1) Y electroweak gauge symmetry. This is a generalization of the case of two Higgs-boson doublets which were discussed in detail in [4]. * E-mail: MManiatis@ubiobio.cl † E-mail: O.Nachtmann@thphys.uni-heidelberg. de We assume that all doublets carry hypercharge y = +1/2 and denote the complex doublet fields by ; i = 1, 2, 3. (2.1) In the most general SU (2) L × U (1) Y gauge invariant Higgs potential the Higgs-boson doublets enter solely via products of the following form: It is convenient to discuss the properties of the Higgs potential such as its stability and its stationary points in terms of gauge invariant bilinears.
First we introduce the 3 × 2 matrix of the Higgs-boson fields in the following way, We arrange all possible SU (2) L × U (1) Y invariant scalar products into the hermitian 3×3 matrix A basis for the 3 × 3 matrices is given by and λ a , a = 1, . . . , 8, are the Gell-Mann matrices. Here and in the following greek indices (α, β, . . .) run from 0 to 8 and latin indices (a, b, . . .) from 1 to 8. We have tr(λ α λ β ) = 2δ αβ , The decomposition of K (2.4) reads now where the real coefficients K α are given by With the matrix K, as defined in terms of the doublets in (2.4), as well as the decomposition (2.8), (2.9), we immediately express the scalar products in terms of the bilinears, (2.10) The matrix K (2.4) is positive semidefinite which follows directly from its definition. This in turn gives The hermitian matrix K (2.4) is constructed from the Higgs field matrix, K = φφ † . Therefore, the nine coefficients K α of its decomposition (2.8) are completely fixed given the Higgs-boson fields.
Since the 3 × 2 matrix φ has trivially rank smaller or equal 2, this holds also for the matrix K. On the other hand, any hermitian 3 × 3 matrix with rank equal or smaller than 2 which clearly has then vanishing determinant, det(K) = 0, determines the Higgs-boson fields ϕ i , i = 1, 2, 3 uniquely, up to a gauge transformation. This was shown in detail in [4] in their theorem 5 for the general case of n-Higgs-boson doublets. In appendix A we show that the gauge orbits of the three Higgs fields (2.1) are characterised by the following set in the 9-dimensional space of (K 0 , . . . , K 8 ): (2.12) Here G αβγ are completely symmetric constants defined in (A25), (A26). That is, to every gauge orbit of the Higgs-boson fields corresponds exactly one vector (K α ) satisfying (2.12) and vice versa. The first two relations of (2.12) are analogous to the light cone conditions of the THDM; see (36) of [4]. The determinant relation, trilinear in the K α , is specific for the 3HDM. Based on the bilinears we shall in the following discuss the potential, basis transformations, stability, minimization, and electroweak symmetry breaking of the general 3HDM.

THE 3HDM POTENTIAL AND BASIS TRANSFORMATIONS
In terms of the bilinear coefficients, K 0 , K a , a = 1, ..., 8 we can write the general 3HDM potential in the form where the 54 parameters ξ 0 , ξ a , η 00 , η a and η ab = η ba are real. The potential (3.1) contains all possible linear and quadratic terms of the bilinears -corresponding to all gauge invariant quadratic and quartic terms of the Higgs-boson doublets. Terms higher than quadratic in the bilinears should not appear in the potential with view of renormalizability. Any constant term in the potential can be dropped and therefore (3.1) is the most general 3HDM potential. We also introduce the notation Let us now consider a change of basis of the Higgsboson fields, and for the bilinears Here R ab (U ) is defined by The matrix R(U ) has the properties For the bilinears a pure phase transformation, U = exp(iα)½ 3 , plays no role. We shall, therefore, consider here only transformations (3.2) with U ∈ SU (3). In the transformation of the bilinears (3.4) R ab (U ) is then the 8 × 8 matrix corresponding to U in the adjoint representation of SU (3).
The Higgs potential (3.1) remains unchanged under the replacement (3.4) if we perform an appropriate transformation of the parameters Let us remark on the basis transformation with respect to the 3-Higgs-doublet model. In a realistic model we have to consider, besides the Higgs potential, kinetic terms for the Higgs-boson doublet fields as well as Yukawa terms which provide couplings of the Higgsboson doublets to fermions. Under a basis transformation, that is, a transformation of the Higgs-boson doublets of the form (3.2), or equivalently, in terms of the bilinears, a transformation of the form (3.4), the kinetic terms of the Higgs doublets will remain invariant. However, we note that, in general, the Yukawa couplings are not invariant under such a change of basis.
In order to illustrate the use of the bilinears we will consider a simple illustrative example of an explicit 3HDM Higgs potential, Here µ 2 > 0 is a parameter of dimension mass squared and λ > 0 is dimensionless. Employing (2.10) we write this potential in terms of the bilinears as This corresponds to the general form (3.1) with parameters, (3.10)

STABILITY OF THE 3HDM
Let us now analyse stability of the general 3HDM potential (3.1), given in terms of the bilinears K 0 and K on the domain determined by (2.12). This can be done in an analogous way to the THDM; see [4]. The case 3/2K 0 = ϕ † 1 ϕ 1 + ϕ † 2 ϕ 2 + ϕ † 3 ϕ 3 = 0 corresponds to vanishing Higgs-boson fields and V = 0. For K 0 > 0 we define Due to (2.12) we have for k the domain D k : The domain boundary, ∂D k , is characterised by From (3.1) and (4.1) we obtain, for where we introduce the functions J 2 (k) and J 4 (k) on the domain (4.2). A stable potential means that it is bounded from below. The stability is determined by the behaviour of V in the limit K 0 → ∞, that is, by the signs of J 4 (k) and J 2 (k) in (4.4), (4.5). For a model to be at least marginally stable, the conditions for all k ∈ D k , that is, all k satisfying (4.2) are necessary and sufficient, since this is equivalent to V ≥ 0 for K 0 → ∞ in all possible allowed directions k. The more strict stability property V → ∞ for K 0 → ∞ and any allowed k requires V to be stable either in the strong or the weak sense. For strong stability we require for all k ∈ D k ; see (4.2). For stability in the weak sense we require for all k ∈ D k To check that J 4 (k) is positive (semi-)definite, it is sufficient to consider its value for all stationary points on the domain D k . This holds because the global minimum of the continuous function J 4 (k) is reached on the compact domain D k , and the global minimum is among those stationary points.
To obtain the stationary points of J 4 (k) in the interior of the domain D k we add to J 4 (k) the second condition in (4.2) with a Lagrange mulitplier u. The stationary points are then obtained from ∇ k1,...,k8 J 4 (k) − u det( 2/3½ 3 + k a λ a ) = 0, For the stationary points on the boundary ∂D k we have to add the condition (4.3) with a second Lagrange multiplier. We get then (4.10) All stationary points satisfying (4.9) or (4.10) have to fulfill the condition J 4 (k) > 0 for stability in the strong sense. If for all stationary points we have J 4 (k) ≥ 0, then for every solution k with J 4 (k) = 0 we have to have J 2 (k) > 0 for stability in the weak sense, or at least J 2 (k) = 0 for marginal stability. If none of these conditions are fulfilled, that is, if we find at least one stationary direction k with J 4 (k) < 0 or J 4 (k) = 0 but J 2 (k) < 0, the potential is unstable. In our explicit example, V expl , (3.9), the functions J 2 (k) and J 4 (k) read (4.11) Obviously, J 4 (k) is always positive for λ > 0 in any direction k, therefore, the potential is stable in the strong sense. That is, stability is here guarantied by the quartic terms of the potential alone.

ELECTROWEAK SYMMETRY BREAKING OF THE 3HDM
Suppose now that the 3HDM potential is stable, that is, bounded from below. Then the global minimum will be among the stationary points of V . In the following the different types of minima with respect to electroweak symmetry breaking are discussed and the corresponding stationarity equations are presented.
As we have discussed in section 2, the space of the Higgs-boson doublets is determined, up to electroweak gauge transformations, by the space of the hermitian 3 × 3 matrices K with rank smaller or equal 2. Since the rank of the matrix K is equal to the rank of the Higgsboson field matrix φ (2.3) we can distinguish the different types of minima with respect to electroweak symmetry breaking as follows. At the global minimum, that is, the vacuum configuration, we write the 3 × 2 matrix of the Higgs-boson fields as In the case this matrix has rank 2, we cannot, by a SU (2) L × U (1) Y transformation, achieve a form with all charged components v + i , i = 1, 2, 3 vanishing. This means that the full SU (2) L × U (1) Y is broken. In case we have at the minimim a matrix φ with rank one, we can, by a SU (2) L × U (1) Y transformation, achieve a form with all charged components v + i vanishing. The unbroken U (1) gauge group can then be identified with the electromagnetic gauge group. Therefore, a minimum with rank one corresponds to the electroweak-symmetry breaking SU (2) L × U (1) Y → U (1) em . Eventually, a vanishing matrix at the minimum, φ = 0, corresponds to an unbroken electroweak symmetry. Of course, only a minimum with a partially broken electroweak symmetry is physically acceptable.
We study now the matrix K v corresponding to φ (5.1) For an acceptable vacuum φ , K v must have rank 1. From (A14) we see that K v has rank 1 and is positive semidefinite if and only if With (5.4) we get in this basis a particularly simple form for K v respectively K vα : Another possible choice for the vacuum expectation value, obtainable by a suitable transformation (3.2) is Here we get In the cases where φ of (5.1) has rank 2 or rank 0 also the matrix K v , (5.2), has rank 2 or zero, respectively. The corresponding conditions for K v are given explicitly in (A13) and (A15), respectively. We can, therefore, summarise our findings for the vacuum values to a given potential V as follows.
Let φ be the vacuum expectation value of the Higgsboson field matrix to a given, stable, potential V and K v = φ φ † = K vα λ α /2. The gauge symmetry SU (2) L × U (1) Y is fully broken by the vacuum if and only if We have the breaking SU We have no breaking of SU (2) L × U (1) Y if and only if Of course, we always have with G αβγ defined in (A25).

STATIONARY POINTS
Following the study of stability and electroweak symmetry breaking in the last two sections we shall now present the stationarity equations. We suppose again that the potential is stable. Then the global minimum is among the stationary points of V .
We classify the stationary points by the rank of the stationarity matrix K. In the following we use the conditions for K having rank 0, 1, 2, or 3 as given in appendix A; see (A12) -(A15).
All stationarity matrices K = K α λ α /2 of rank 1 are obtained from the following system of equations where u 1 and u 2 are Lagrange multipliers: All stationarity matrices K = K α λ α /2 of rank 2 are obtained from the following system of equations where u is a Lagrange multiplier: The stationarity matrix K = K α λ α /2 with the lowest value of V (K 0 , . . . , K 8 ) gives the global minimum K v of the potential. Note that in general there may be degenerate global minima with the same potential value.
Systems of equations of the kind (6.1), (6.2) can be solved via the Groebner-basis approach or homotopy continuation; see for instance [14,15].
In our example 3HDM Higgs potential, (3.9), we find stationary points for vanishing fields, corresponding to an unbroken EW symmetry, from the set (6.2) we get no solution with K 0 > 0, and from the set (6.1) we get one real solution with with any value for the Lagrange multiplier u 2 = 0. The corresponding potential value is V 0 = −1/4 · (µ 2 ) 2 /λ and is the deepest stationary point and therefore the global minimum. From (5.6) (or (5.7)) we see that the global minimum corresponds to a vacuum expectation value v 0 = µ 2 /λ. Let us eventually mention the procedure to calculate the Higgs-boson masses in the 3HDM. We will assume that the potential is stable and leads to the desired electroweak symmetry breaking. In particular, the global minimum is then a solution of the set of equations (6.1). In this case we can, in the unitary gauge, by an electroweak gauge transformation always achieve the form (5.7) for the vacuum expectation value of the Higgs-field matrix. For the original Higgs fields expressed in terms of the physical fields we get . (6.4) with v 0 real and positive, neutral Obviously, the 3 original complex doublets of any 3HDM, corresponding to 12 real degrees of freedom, yield 5 real fields and 2 complex fields, with the 3 remaining degrees of freedom absorbed via the mechanism of electroweak symmetry breaking. Expressing the bilinears in the parametrization (6.4) via (2.4) and (2.8) we can write the potential in terms of the physical fields (6.5). Selecting all contributions quadratic in these physical fields, we arrive at In this way we can find the mass matrices squared M 2 neutral and M 2 charged in terms of the initial parameters. The physical masses follow from a diagonalization of these squared mass matrices.
In our explicit example (3.9), collecting the quadratic terms of the Higgs potential, we can directly read off the squared mass matrices which are already of diagonal form. In this case we have, besides one Higgs boson with mass 3λv 2 0 − µ 2 = 2µ 2 , four mass-degenerate neutral Higgs bosons and two massdegenerate charged Higgs bosons.

CONCLUSION
The three-Higgs-doublet model has been studied as a generalization of the THDM. Stability, electroweak symmetry breaking, and the types of stationary points of the potential have been investigated. Explicit sets of equations have been presented which allow to determine the stability of any 3HDM and, in case of a stable potential, to find the global minimum or the degenerate global minima in case the potential has such. The use of bilinears turns out to be very helpful: in particular, irrelevant gauge degrees of freedom are avoided and the degree of the polynomial equations which are to be solved is reduced in this formalism. In general, the sets of equations which determine stability and the stationary points are rather involved. However, approaches like the Groebnerbasis approach or homotopy continuation may be applied to solve these systems of equations in an efficient way.
(A10) On the other hand, having the conditions (A10) for a hermitian matrix K fulfilled, employing (A2), the determinant condition requires that at least one κ i vanishes, for instance κ 3 = 0 without loss of generality. Then the second condition requires that another eigenvalue has to vanish, for instance κ 2 = 0. Eventually, the first condition then dictates that the remaining κ 1 > 0. Hence, K has rank 1 and is positive semidefinite.
(A11) Vice versa, starting with the conditions (A11) for a hermitian matrix K, the determinant condition requires that one eigenvalue, for instance κ 3 = 0 has to vanish, the second condition in turn requires that another, say κ 2 = 0, and the first trace condition that also the third κ 1 = 0. This means K = 0. Therefore, we have shown the following theorem. With this theorem we have expressed the properties of the matrix K in terms of the expansion coefficients K α , α = 0, . . . , 8. The conditions concerning K 0 and K a in (A12) to (A15) are of the type of light-cone conditions familiar from the two-Higgs-doublet model; see (36) of [4]. But the determinant condition, trilinear in K α , is specific for the 3HDM.