Families from supergroups and predictions for leptonic CP violation

As was shown in 1984 by Caneschi, Farrar, and Schwimmer, decomposing representations of the supergroup SU(M |N ), can give interesting anomaly-free sets of fermion representations of SU(M ) × SU(N ) × U(1). It is shown here that such groups can be used to construct realistic grand unified models with non-abelian gauged family symmetries. A particularly simple three-family example based on SU(5) × SU(2) × U(1) is studied. The forms of the mass matrices, including that of the right-handed neutrinos, are determined in terms of SU(2) Clebsch coefficients; and the model is able to fit the lepton sector and predict the Dirac CP-violating phase of the neutrinos. Models of this type would have a rich phenomenology if part of the family symmetry is broken near the electroweak scale.


Introduction
One way of finding chiral sets of fermions that are anomaly-free under product gauge groups is to decompose anomaly-free multiplets of larger groups. For example, by decomposing the 10 + 5 of SU(5) under its SU(3) × SU(2) × U(1) subgroup, one finds the anomaly-free set that comprises the fermions of the Standard Model. And by decomposing the 16 of SO(10) under its SU(5) × U(1) subgroup, one finds the anomaly-free set 10 1 + 5 −3 + 1 5 .
It was shown in [1] that interesting sets of fermions that are anomaly-free under groups of the form SU(M ) × SU(N ) × U(1) can be found by decomposing multiplets of the supergroup SU(M |N ) [2]. The idea is based on the fact that the Casimirs of SU(M |N ) only depend on (M − N ). Thus the third-order Casimirs for the groups SU(M + P |P ) are the same for any P , and thus the same as for SU(M ). If one considers, therefore, an irreducible fermion representation that is anomaly-free (i.e. has vanishing third-order Casimir) under SU(M ), the corresponding Young tableaux representation of SU(M +P |P ) will yield anomaly-free sets when decomposed under the bosonic subgroup SU(M +P )×SU(P )×U (1).

JHEP10(2017)128
The anomaly-free sets constructed in [1] are interesting from the point of view of gauged family symmetry. In a theory having the gauge group SU(M ) × SU(N ) × U(1), the first factor could contain the Standard Model group if M ≥ 5, while SU(N ) could be a family group if it has three-dimensional representations, whether irreducible or reducible.
An anomaly-free set of fermions that contains exactly three families of the Standard Model can be obtained by looking at the rank-3 tensors of SU(3 + P |P ). This gives the anomaly free-set of SU(3 + P ) × SU(P ) × U(1) fermion multiplets ( [3], (0)) −P + ( [2], (1)) (P +1) + ( [1], (2)) −(P +2) + ([0], (3)) P +3 , (1.2) An interesting case, which gives a family group SU (3), is obtained by setting P = 3 in eq. (1.2), in which case the group is SU(6) × SU(3) × U(1) and the multiplets are (20, 1) −3 + (15 −3 , 3) 4 + (6, 6) −5 + (1, 10) 6 . This contains in addition to the Standard Model fermions many fermions that are vector-like under the Standard Model group. Under SU (5), it contains in addition to the three families of 10 + 5, three sets of 5 + 5, one of 10 + 10, and sixteen singlets. A more economical case is obtained by setting P = 2 in eq. (1.2). Then one has the following group and fermion multiplets: The only fermions this contains besides those of the Standard Model (SM) are four SMsinglets, which can play the role of the right-handed neutrinos. This is the simplest and most economical case based on the constructions of [1]. We shall therefore study it in detail. As will be seen, models can be constructed for this case in which the family SU(2) gives non-trivial forms for the fermion mass matrices, fits the lepton sector well, and predicts the Dirac CP phase of the neutrinos.

JHEP10(2017)128
Note that an SU(2)-singlet mass for the right-handed neutrinos, i.e. (1, 4) 5 1, 4) 5 (1, 1) −10 H , is forbidden by Fermi statistics, since the symmetric product of two 4-plets of SU(2) does not contain a singlet. (Note that from the fact that the Higgs multiplets in eq. (2.1) can couple to the fermions as in eq. (2.2) it is evident that they also can arise from decomposing multiplets of the supergroup SU(5|2).) As we shall see in detail, the forms of mass matrices of the quarks and leptons that arise from the Yukawa terms in eq. (2.2) are determined by the SU(2) family symmetry, and the Clebsch coefficients of SU (2).
Let us denote the vacuum expectation values (VEVs) of the Higgs multiplets shown in eq. (2.1) as follows Here we have expressed the VEVs of the SU(2) triplets in a "Cartesian basis". But we can also denote them in a "spherical basis", is a complex vector. If we assume that its real and imaginary parts are aligned, then we can choose the basis in SU(2) space so that (t 1 , t 2 , t 3 ) = (0, 0, t). Such alignment happens if a certain quartic self-coupling of (1, 3) −10 H has the right sign. The most general renormalizable potential for this field is of the form If we write t = a + i b, where a and b are real vectors, then V = −µ 2 (a 2 + b 2 ) + (λ + λ )(a 2 + b 2 ) 2 − 4(λ + λ )a 2 b 2 sin 2 θ ab . There are two cases: case I with λ + λ < 0, and Case II with λ + λ > 0. In Case I, the angle θ ab between a and b vanishes. Then t =â(a + ib), where the phase of a + ib can be gauged away. Choosingâ to point in the 3 direction, and defining t ≡ √ a 2 + b 2 , one ends up with the form (t 1 , t 2 , t 3 ) = (0, 0, t). In Case II, θ ab = π/2, so a and b are perpendicular to each other, and one can choose the basis in SU(2) space so that (t 1 , t 2 , t 3 ) = (0, it , t). Moreover, in this case the term with sin 2 θ ab becomes −|λ + λ |a 2 b 2 , meaning that a and b become of equal magnitude, and one has (t 1 , t 2 , t 3 ) = (0, it, t). These two cases give different mass matrices for the right-handed neutrinos and will both be examined below.
In eq. (2.1) we have written several 5-plets of SU(5). These contain altogether six electroweak doublets of scalars. All of them would "naturally" be expected to have superheavy masses. In a non-SUSY SU(5) model, one fine-tuning is done to make the mass-squared matrix of the six electroweak doublets have one small (i.e. electroweak-scale) eigenvalue. The linear combination of electroweak doublets corresponding to this eigenvalue is the Standard Model Higgs doublet; the five orthogonal linear combinations are superheavy. If the Standard Model Higgs doublet is a linear combination of the six doublets in (5, 1) 4 H with all six of the coefficients being non-zero, then all six of the VEVs denoted S, d ↑ , d ↓ , v 1 , v 2 , and v 3 in eq. (2.3) will be non-zero. That JHEP10(2017)128 this can be achieved will be shown in detail in section 5. In SUSY SU(5) models, the situation is similar. There one considers the 6 × 6 mass matrix of the electroweak doublet Higgsinos and arranges, either by fine-tuning or by technically natural mechanism (such as the missing partner mechanism), that one of its eigenvalues is of electorweak scale.
We will define the complex numbers Let us similarly denote the fermion multiplet (5, 3) −4 by (5 1 , 5 2 , 5 3 ) or (5 − , 5 0 , 5 + ) and the fermion multiplet (10, 2) 3 by (10 ↓ , 10 ↑ ). The (10, 1) −2 we will denote simply by 10, without any subscript. Then the 3 × 3 mass matrix of the up quarks mass can be written so that the up quark mass matrix can be written in the form . Since the VEVs that give the fermions mass do not break SU(4) c , one obtains the unrealistic "minimal SU(5)" relation [7] between the down quark and charged lepton mass matrices: M d = M T . These come from the term This gives , and we have used the complex parameters x, z 1 , and z 2 parameters in eq. (2.4). The neutrino mass matrix arises through a Type I see-saw mechanism [3][4][5][6]. There are three left-handed neutrinos in the (5, 3) −4 and four left-handed anti-neutrinos in the (1, 4) 5 . The Dirac neutrino mass matrix comes from

JHEP10(2017)128
where the form is entirely determined by SU(2) Clebsch coefficients. The 4 × 4 Majorana mass matrix of the ν c is also determined by Clebsch coefficients. In Case I, where (t − , t 0 , t + ) = (0, t, 0), one then finds Writing this in the Cartesian basis (ν 1 , ν 2 , ν 3 ), and defining This is not, however, the most general form of the neutrino mass matrix, because another operator can contribute to it, namely the effective dim-5 operator In the Cartesian basis, this just gives the identity matrix. Defining the ratio of the coefficient of this term to µ ν by the complex number y, we have Note that the complex parameter y actually makes a difference for the neutrino mixing angles and mass splittings, despite appearing as the coefficient of the identity matrix. This is so, because M ν is complex and symmetric and thus diagonalized by , This gives the following mass matrix for the right-handed neutrinos: (2.14)

JHEP10(2017)128
After straightforward algebra, this gives the following mass matrix for the three light neutrinos in a Cartesian basis: which is to be compared to eq. (2.13). The model described above, while not fully realistic because of the "minimal SU(5) relation M d = M T [7], can account for many of the qualitative features of the quark and lepton masses and mixing angles. The fact that the overall scales of the up quark masses, down quark and charged lepton masses, and neutrino masses are very different can be explained by the fact that they are determined by the three independent parameers µ u , µ d , and µ (see eqs. (2.6), (2.8), (2.13), and (2.15)), which are in turn determined by the VEVs of different types of Higgs multiplets. Moreover, several features of the inter-family mass ratios can also be accounted for.
The most striking feature of the observed inter-family fermion mass ratios is that they are hierarchical. That can partly be explained in this model by the fact that the three families are distinguished from each other by how they transform under the SU(2) family symmetry. For instance, because of SU(2), three different types of Higgs multiplet contribute to the up quark masses, as one sees from eq. (2.2). If one assumes a hierarchy among the VEVs (or Yukawa coefficients, or both) of those three Higgs multiplets, one can have δ 1, which gives m u m c m t , as is apparent from eq. (2.6). Two types of Higgs multiplets contribute to the down quark (and charged lepton) masses, as shown in eq. (2.2). If one assumes a hierarchy in their VEVs (or Yukawa couplings, or both), one can have η 1. This would explain why the third family of down quarks and charged leptons is heavier than the first two families, as eq. (2.8) shows. However, it would not explain the lightness of the first family compared to the second for the down quarks and charged leptons. As one can see from eq. (2.8), that would require a certain relationship (which will be given later) to hold among the parameters x, z 1 and z 2 .
Each of the parameters x, z 1 and z 2 is defined as a ratio of VEVs of different components of an SU(2) Higgs multiplet. One would therefore naturally expect that these (complex) parameters would have magnitudes of O(1). Thus, the relationship among them that would make m e /m µ 1 and m d /m s 1 would involve a fine-tuning of order 10 −2 . The hierarchies δ 1 and η 1 would also partially explain the smallness of the CKM angles. An examination of eqs. (2.6) and (2.8) shows that V cb and V ub come out to be of order η, while the Cabibbo mixing V us comes out to be O(1) if x, z 1 and z 2 are arbitrary parameters of O(1). The "fine-tuning? required to fit the Cabibbo angle is mild, but a tuning of order 10 −1 is required to explain the smallness of |V ub |. This tuning takes the form of a relation among x, z 1 and z 2 that must be approximately satisfied.
If the parameters x, z 1 and z 2 have magnitudes of O(1), as one would naturally expect, then the forms of the lepton mass matrices given in eq. that the PMNS angles should typically be of O(1) as well, and that the ratio of neutrino masses should not be small. Thus, this model can account in a natural way for most of the qualitative features of the quark and lepton mass ratios and mixing angles. The two exceptions are the smallness of m e /m µ (and m d /m s ) and the smallness of |V ub |, each of which requires a somewhat tuned condition to hold among the complex parameters x, z 1 , and z 2 .
The minimal model described above is very simple, and, as we shall see, the SU(2) family symmetry yields non-trivial predictions for the lepton sector, in particular for the Dirac CP phase of the neutrinos. As noted, however, this minimal model's predictions for the quark sector are not realistic, because the model gives the "minimal SU(5)" relation M d = M T at the GUT scale. This defect can be repaired if some of the quarks and leptons obtain mass from effective higher-dimension Yukawa terms that contain the adjoint Higgs of SU(5) (or whatever Higgs breaks SU(5) down to the Standard Model group). This can be done in such a way that the quark sector is made realistic without changing the minimal model's predictions for the lepton sector. We shall therefore defer to section 4 a discussion of how this can be done, and first derive the lepton-sector predictions of the minimal model in section 3.

Predictions of neutrino properties
Let us now see whether the simple model we have presented can fit the lepton sector, i.e. the masses of the charged leptons and neutrinos, and the PMNS angles.
As noted before, the fact that m e m µ requires a tuning of parameters. As can be seen from an inspection of eq. (2.8), for |z 1 |, |z 2 | and |x| of O(1), and |η| small, the three eigenvalues of M are of order |µ d |, |ηµ d |, and |ηµ d |. To have m e ∼ 10 −2 m µ requires that | det M | ∼ 10 −2 |η 2 µ 3 d |. This yields the condition that It will make no significant difference, and will simply calculations, if in fitting the neutrino properties we simply set this small quantity to zero. In that case, solving a quadratic equation allows one to solve for x in terms of z 1 and z 2 : Suppose the mass matrices M and M ν are diagonalized by the following unitary transformations: . Then, with our conventions, the PMNS matrix is given by U P M N S = U * U T ν . If we ignore effects that are subleading by order |η| 2 , the unitary matrix U depends only on the complex parameters z 1 , z 2 , and x, as can be seen by inspection of the form of M given in eq.
where N 12 ≡ |z 1 | 2 + |z 2 | 2 , N ≡ 1 + N 2 12 = 1 + |z 1 | 2 + |z 2 | 2 , and sin θ 12 cos θ 12 ≡ i z * 2 1 + z * 2 where we have used eq. (3.2) to eliminate the parameter x and write U entirely in terms of z 1 and z 2 . The diagonalization of M ν , given in eq. (2.13) for Case I and eq. (2.15) for Case II, must be done numerically. This requires searching over three complex parameters of O(1), namely z 1 , z 2 , and y. For each choice of these parameters, one can compute the PMNS angles and the ratio of neutrino mass splittings ∆m 2 12 /∆m 2 23 . (The overall scale of the neutrino masses is set by the parameter µ ν .) One might think that one should be able to fit these four experimental numbers with the three complex model parameters z 1 , z 2 , and y. A good fit is not guaranteed to exist, however, as the equations are nonlinear.
For Case I, we have done a numerical search of parameter space and found that there are values of the parameters that give excellent fits to the three PMNS angles, but none of them also gives a small enough value for the ratio of mass splittings ∆m 2 12 /∆m 2 23 . For Case II, we have two found satisfactory solutions for the leptons, one corresponding the minus sign in eq. (3.2), and the other corresponding to the plus sign. We will call these Solutions 1 and 2, respectively. These two solutions give a good fit all three neutrino mixing angles and the ratio of neutrino mass splittings ∆m 2 12 /∆m 2 23 , but give different predictions for the Dirac CP phase of the neutrinos δ CP .
In table 1, we present the fits to the neutrino mixing angles and the predictions of δ CP for the two solutions. These were found in the following way. We searched over the three complex parameters z 1 , z 2 , y and kept only those points which yielded values for the three PMNS angles and for the ratio of neutrino mass splittings that were each within one-sigma of the experimental value. The error bars in the second and third columns of table 1 represent the standard deviation of the values obtained in this way. One notes that the prediction for the Dirac CP phase of the neutrinos δ CP is fairly sharp for each of the two solutions. The fourth column in table 1 gives the 1σ best fit values from the 2014 particle data group [8], and the fifth column gives the best fit values from the 2016 particle data group [9]. In table 2, we give the values of the complex model parameters z 1 , z 2 and y for the two solutions.
This model illustrates the predictive potential of models with non-abelian family groups. The SU(2) family symmetry strongly constrains the forms of the mass matrices. The patterns arising from the SU(2) family symmetry allow the model to account for many qualitative features of the quark and lepton spectrum, as well as yielding very precise predictions for the Dirac CP phase of the neutrinos. We now show that the model can be modified to allow a realistic quark sector, without affecting the predictions for the lepton sector.

Making the quark sector realistic
The quark sector of the minimal model described in previous sections is unrealistic in two ways. First, the down quark masses come out wrong, because of the relation M d = M T . Second, there are not enough free parameters to ensure that the CKM mixing parameters are fit. There are two standard ways to avoid the unrealistic prediction M d = M T in SU(5) GUT models. One is to introduce 45-plets of Higgs fields in addition to the 5-plets so that there are Yukawa couplings of the form 10 · 5 · 45 H in addition to the 10 · 5 · 5 H . Because the VEV of the 45 H couples differently to the quarks and leptons, this would break the d − degeneracy. In our case, this would mean introducing at least (45, 2) H . While it may be possible to have a Higgs sector and Higgs potential that ensures such alignment, it appears to be quite difficult to achieve.
The second way to avoid the bad relation M d = M T is to have some of the quark and lepton masses come from higher-dimension effective Yukawa terms that contain the SU (5) JHEP10(2017)128 adjoint Higgs field (or whatever Higgs field breaks SU(5) down to the Standard Model group at the GUT scale). As we shall see, this altogether avoids the alignment problem mentioned in the previous paragraph if the adjoint is a singlet under the family SU(2), as then its insertion into the Yukawa terms does not affect the family structure of those terms. Using the adjoint Higgs to break the relation M d = M T has the advantage that it does not require introducing any additional Higgs fields beyond those required in the minimal model. On the other hand, there need to be new fermion fields at the GUT scale that when integrated out yield the desired higher-dimension effective Yukawa terms. As we shall see, however, these new fermions can be vector-like pairs consisting of fermions in some of the same representations shown in eq. (1.3) together with their conjugates. So this is quite economical.
We shall now discuss one way to build a realistic extension of the minimal model described in the previous sections using higher-dimension effective Yukawa terms. Let us suppose there is a Z 2 symmetry under which the Higgs multiplets (5,2)   It should be noted that the terms in eq. (4.1) with coefficients denoted b, c, e, and f are each in reality two terms, since the indices in the SU(5) products of Higgs fields can be contracted in two distinct ways, corresponding to 5 × 24 = 5 + 45. So, b, c, e, and f really each represent two Yukawa coefficients, but we have not written this out explicitly in eq. (4.1). Note that the top quark mass still comes from a dimension-4 term, namely (10, 1) −2 (10, 1) −2 (5, 1) 4 H , so that it receives no suppression by a factor of (24, 1) 0 H /M GUT . The terms with coefficients denoted e in eq. (4.1) contribute to the third row of M d and the third column of M in eq. (2.8), which are now no longer equal as in the minimal model, but are multiplied by different factors due to the VEV of (24, 1) 0 H , which gives different contributions to quarks and leptons. We assumed that these are the largest row Similarly the terms with coefficients denoted f in eq. (4.1) cause the first and second rows of M d in eq. (2.8) to be multiplied by a different factor than the first and second columns of M . As a result the degeneracy of m s and m µ at the GUT scale is lifted. However, as the first and second families get multiplied by the same factors, the terms in eq. (4.1) still give the bad prediction that m e /m µ = m d /m s at the GUT scale. We will return to this issue shortly.
While the effective Yukawa terms involving (24, to fit the CKM matrix. In the extension of the minimal model, however, there are new parameters in M d , as we just saw, coming from the dimension-6 operators. These are of order 10 −3 m b and should significantly affect both V ub and V us . This gives in effect four real adjustable parameters for fitting the CKM matrix. Turning to M u , it contains in the minimal model six complex parameters: x, z 1 , z 2 , µ u , , and δ. The first three of these were fixed by fitting the lepton sector. The phase of µ u is irrelevant, and three real parameters are needed to fit the masses of u, c, and t. That leaves two real adjustable parameters in M u in the minimal model that are available to fit the CKM matrix. In the extended model, however, an additional complex parameter appears in M u . The reason for this is that the form of M u is changed when we go to the extended model. As noted before, the term b (10, 1) −2 (10, 2) 3 (5, 2) −1 H (24, 1) 0 H /M GUT in eq. (4.1) actually contains two terms, in which the SU (5)  , one sees that the flavor anti-symmetric part of (10, 2) × (10, 2) is a singlet of SU(2), whereas the product of Higgs fields must be a triplet.) Altogether then, we have four real parameters coming from M u and four from M d that are available to fit the CKM matrix, which is more than sufficient.
We now turn to the question of where the higher-dimension operators in eq. (4.1) come from. They can come from integrating out superheavy vectorlike fermion fields that we denote as follows (the subscript V standing for vector-like): These are all assumed to be odd under Z 2 , while the fermion multiplets that appear in the minimal model are all assumed to be even. These vectorlike fermions have the explicit mass terms M 1 (10, 1) −2 V (10, 1) 2 They also have the following couplings to the Z 2 -even fermions: y 1 (10, 1) −2 (10, 1) 2 which just parallel the forms in eq. (4.1), but without the adjoint Higgs (which is not needed as each term here contains a Z 2 -odd fermion multiplet).

The Higgs sector
There are two issues that must be considered with respect to the Higgs sector: the breaking of the family group SU(2) at a large scale (which we take to be of order the GUT scale), and the electroweak breaking. We will consider them in turn.
One Higgs field that breaks the family group at the large scale is the (1, 3) −10 H in eq. (2.3). It turns out, however, that to get a realistic model there must be additional Higgs fields that transform as (1, 2) 5 and are odd under Z 2 . It turns out that there need to be two of these in order to break the family SU(2) in a realistic way, so we will denote them by (1, 2) 5 HK , K = 1, 2. We will sometimes use the notation (1, 3) −10 = t = (t 1 , t 2 , t 3 ) and (1, 2) 5 HK = s K = (s ↑ , s ↓ ) K , K = 1, 2. In minimizing the Higgs potential for these fields, which will get GUT-scale VEVs, we can ignore the electroweak-breaking Higgs fields. Thus, we may consider the potential V ( t, s K ) = V t ( t)+V s (s K )+V ts ( t, s K ), where the most general renormalizable form is (5.1) We already examined the minimization of V t ( t) in section 2. For (λ + λ ) > 0, it was found that the VEV could be brought to the form t = t √ 2 (0, i, 1) by a suitable choice of axes in SU(2) space. We will now assume that the VEVs of s K are sufficiently small that the terms in V st do not significantly affect the form of the VEV of t. For example, if the quartic couplings in V are of order one, and s K ∼ t/30, then the VEVs of s K would only affect the VEV of t at the 10 −3 level. (Also, the dimension-6 operator (1, 4) 5 (1, 4) 5 [(1, 2) 5 H (1, 2) 5 H ] * /M GUT would only affect the form of the right-handed neutrino mass matrix by order 10 −3 .) These effects would be too small to be significant in the fits of lepton properties in section 3.
The directions of the VEVs of s K are determined by V s + V st . It is straightforward to show that non-trivial values of (s ↑ , s ↓ ) K can be obtained by choosing the coefficients in V s + V st appropriately. This is only the case because there are at least two s multiplets. If there were only one, the most general form of V s + V st would lead to (s ↑ , s ↓ ) being proportional to either (1, 1) or (1, −1). (The reason for this is simple. The VEV t ∝ (0, i, 1)

JHEP10(2017)128
in [1]. Models based on such groups and multiplets can give both grand unification of the Standard Model gauge interactions and gauged non-abelian family groups.
In this paper we have explored one potential of such models, namely that the family symmetry could constrain the form of the quark and lepton mass matrices in such a way as to explain the main qualitative features of the quark and lepton properties. We studied the smallest such model that contains three families, which has the group SU(5)×SU(2)×U(1). In particular, we studied the minimal form of this model, and showed it can account in a simple way for many of the qualitative features of the spectrum of quark and lepton masses and mixing angles, as well as making definite predictions for the lepton sector, specifically the Dirac CP phase of the neutrinos. This predictiveness arises because very definite and non-trivial forms are obtained for the fermion mass matrices (including that of the righthanded neutrinos), determined in large part by the Clebsch coefficients of the SU(2) family group. We showed that the minimal form of the model can be modified to make it realistic for the quark sector without affecting the neutrino predictions.
In addition to their implications for quark and lepton masses and mixing angles, such models in general would have a rich phenomenology if a subgroup of the family gauge groups were broken near the electroweak scale. This phenomenology would include (a) extra Z bosons, whose couplings to the quarks and leptons would be quite distinctive; (b) flavor-changing non-abelian gauge interactions, which would give rare flavor-violating decays of leptons, whose branching ratios would be constrained by family symmetry; and (c) extra vector-like quarks and leptons. Clearly, there are many possibilities that remain to be explored. Moreover, in such models, one would expect the flavor structure to be sufficiently constrained by the family symmetry to give predictions for proton-decay branching ratios.