Leptogenesis, Fermion Masses and Mixings in a SUSY $SU(5)$ GUT with $D_{4}$ Flavor Symmetry

We propose a model of fermion masses and mixings based on $SU(5)$ grand unified theory (GUT) and a $D_{4}$ flavor symmetry. This is a highly predictive 4D $SU(5)$ GUT with a flavor symmetry that does not contain a triplet irreducible representation. The Yukawa matrices of quarks and charged leptons are obtained after integrating out heavy messenger fields from renormalizable superpotentials while neutrino masses originate from the type I seesaw mechanism. The group theoretical factors from 24- and 45-dimensional Higgs fields lead to ratios between the Yukawa couplings in agreement with data, while the dangerous proton decay operators are highly suppressed. By performing a numerical fit, we find that the model captures accurately the mixing angles, the Yukawa couplings and the $CP$ phase of the quark sector at the GUT scale. The neutrino masses are generated at the renormalizable level with the prediction of trimaximal mixing while an additional effective operator is required to account for the baryon asymmetry of the universe (BAU). The model is remarkably predictive because only the normal neutrino mass ordering and the lower octant of the atmospheric angle are allowed while the $CP$ conserving values of the Dirac neutrino phase $\delta_{CP}$ are excluded. Moreover, the predicted values of the effective Majorana mass $m_{\beta \beta}$ can be tested at future neutrinoless double beta decay experiments. An analytical and a numerical study of the BAU via the leptogenesis mechanism is performed. We focused on the regions of parameter space where leptogenesis from the lightest right-handed neutrino is successfully realized. Strong correlations between the parameters of the neutrino sector and the observed BAU are obtained.


Introduction
During the last two decades, neutrino oscillation experiments have presented vigorous measurements of the neutrino mass-squared differences and their mixing angles 1 [1,2], conflicting the zero mass prediction of the standard model (SM) of electroweak interactions. Besides the precise measurement of the oscillation parameters, the CP violation and the flavor pattern in the quark and lepton sectors along with the fermion mass hierarchies are not firmly established within the SM. Therefore, theoretical investigations beyond the SM are urgently needed to explain the fermion flavor structure. The CP violation is of particular interest especially after the T2K collaboration excluded some values of δ CP giving rise to a large improvement of the observed antineutrino oscillation probability at 3σ confidence level [6]. Moreover, CP violation is one of the essential ingredients among the three conditions presented by Sakharov to explain the observed BAU through the baryogenesis mechanism [7]. The other two conditions being the baryon number violation and the deviation from thermal equilibrium. The reason to search for this in the lepton sector is due to the fact that the SM predictions for the CP violation -which is encoded in the CKM phase δ CKM -is insufficient to generate the observed BAU and thus, new sources for CP violation beyond the SM are required. An interesting approach to successfully produce the observed excess of matter over antimatter in the universe is through the leptogenesis mechanism 2 [13], which relies on the right-handed (RH) Majorana neutrinos introduced in the context of type I seesaw mechanism [14][15][16][17][18]. In practical terms, this approach requires lepton number violation which arise naturally in type I seesaw models via the Majorana masses of the RH neutrinos. Then, a lepton asymmetry is generated by the out of-thermal-equilibrium and CP violating decays of these RH neutrinos that is eventually converted into a primordial baryon asymmetry by means of the SM sphaleron processes [19]. As a result, the three Sakharov conditions are satisfied in this scenario, which is remarkable considering that leptogenesis connects high energy scales where the BAU takes place and neutrino oscillations that take place at low energy scales.
Grand unified theories are the most attractive high-energy completions of the SM that can bridge the experimentally accessible low energies with extremely high energy phenomenon while providing the unification of electromagnetic, weak and strong interactions [20][21][22][23][24]. When combined with supersymmetry (SUSY) [25], GUTs provide a more powerful explanation to some of the open questions in the SM as well as a solution to some of the problems that are not addressed in the minimal non-SUSY GUTs [26]. The simplest realization of such a combination is provided by the SUSY SU (5) model where the unification of the gauge couplings occurs at a scale of approximately 2 × 10 16 GeV [27][28][29][30][31]. As a result of unification, the masses of down quarks and charged leptons are generated from a common renormalizable operator leading to Yukawa couplings of same order of magnitude; y e = y d , y µ = y s , and y τ = y b . It is well-known that these equalities are acceptable for the third generation but fails for the remaining ones because of their conflict with the experimental data. The Georgi Jarlskog (GJ) relations m µ /m s = 3 and m e /m d = 1/3 generated from a specific renormalizable operator involving a 45-dimensional Higgs H 45 presented a first example solution to this issue [32]. In contrast to these relations, considering additional Higgs fields in the 24-or 75-dimensional representations of SU (5) gives rise to nontrivial Clebsch-Gordan (CG) factors with new ratios for the first two generations of Yukawa couplings that are preferred phenomenologically; see for instance refs. [33,34] for ratios derived from dimension 5 and dimension 6 operators in the context of SUSY SU (5). On the other hand, neutrinos are massless in SUSY SU (5) model which implies that the oscillation phenomenon can not be explained within its minimal realization. The simplest way to address this issue is by introducing RH singlet fermions to generate neutrino masses via the type I seesaw mechanism, while the mixing angles can be determined by invoking the well-known approach of flavor symmetries. Non-Abelian discrete symmetries 3 are in particular a powerful tool for explaining the mass hierarchies and the mixing of all fermions [40], especially, those with triplet representations. For example, the discrete symmetry A 4 is widely used in SU (5) flavor models to explain the patterns of neutrino masses and their mixing; see for instance refs. [40][41][42][43][44][45][46][47][48]. On the other hand, the discrete groups with doublet representations like S 3 and D 4 are less employed in 4D SU (5) GUTs. In fact, a SUSY SU (5) model based on D 4 symmetry was considered before in ref. [49]; however, the phenomenological implications of both the lepton and quark sectors were lacking. Here, we will show that D 4 can provide good results regarding the neutrinos as well as the charged fermions flavor structures by allowing the three generations of matter to be unified into the representations 1 and 2 instead of 3 4 .
In this work, we build a predictive model based on SUSY SU (5) GUT supplemented by a D 4 flavor symmetry suitable for addressing the above mentioned questions. In particular, we show that our construction leads to results for the pattern of fermion masses and mixings that are consistent with the current experimental data. In fact, this is the first phenomenological analysis of the fermion mass and mixing structures within SUSY SU (5) using the dihedral group D 4 . Besides this discrete group, we have added a U (1) symmetry to engineer the invariance of the superpotentials in the quark and lepton sectors, and also to prevent dangerous operators that mediate rapid proton decay. Apart from the usual SUSY SU (5) superfield spectrum, various superfields are added to the model in order to fulfill different tasks. Namely, many messenger fields denoted as X i and Y i are needed to make the model renormalizable, higher dimensional Higgs fields in the 24 and 45 representations required to obtain realistic Yukawa coupling ratios, gauge singlets superfields -the so-called flavons -needed to break the flavor symmetry and structure the fermions mass matrices, and three right-handed neutrinos N c i=1,2,3 responsible for the tiny neutrino masses as well as the BAU through the leptogenesis mechanism 5 . The introduction of all of the above fields with the requirement to keep the effective superpotentials invariant is highly controlled by the group theoretical structure of the D 4 × U (1) flavor symmetry. In the charged sector, the messenger fields X i and Y i are coupled to the matter fields, the flavon fields and the 24 and 45 Higgs fields. When X i and Y i are integrated out we obtain the effective operators responsible for the quark and lepton Yukawa couplings, which is then followed by the spontaneous breaking of the flavor and gauge symmetries after the flavons and Higgs fields of the SU (5) -that is the 5,5, 24, and 45 dimensional Higgs fields denoted respectively as H 5 , H5, H 24 and H 45 -acquire nonzero vacuum expectation values (VEVs). On the one hand, the specific VEV alignments of the flavons break the D 4 × U (1) symmetry and help shape the fermions mass matrices, leading eventually to the appropriate flavor structure of the quarks and leptons. On the other hand, the CG factors obtained from the VEV structures of H 24 and H 45 lead to the following double ratio of the Yukawa coupling of the first and second generation yµ ys y d ye ≃ 10.12 which is consistent with experimental data [62]. In the chargeless sector, the neutrino masses are generated at the renormalizable level through the type I seesaw mechanism. The obtained neutrino mass matrix m ν is described by only three parameters leading to strong constraints among the physical parameters. Moreover, m ν is invariant under a particular remnant Z 2 symmetry which is commonly referred to as a magic symmetry [63], indicating that m ν is diagonalized by the well-known trimaximal mixing (T M 2 ) matrix which is consistent with the observed neutrino mixing angles [64][65][66][67][68][69][70]. However, the leptogenesis mechanism can not be induced at the renormalizable level given that the neutrino Yukawa coupling matrix is proportional to the identity matrix which leads to a vanishing lepton asymmetry. Therefore, we show that by introducing one effective operator as a correction to the neutrino Yukawa coupling matrix, our model can accommodate successfully the observed BAU via leptogenesis 6 . Our main results in the neutrino sector are: • only the normal hierarchy (NH) for neutrino mass spectrum is allowed, • only the lower octant of the atmospheric angle is allowed, • the CP conserving values of the Dirac CP phase δ CP are excluded, • the predicted values of the effective Majorana mass in neutrinoless double beta decay (0νββ) are testable at future 0νββ searches, and • the correlation between the BAU parameter denoted as Y B and the neutrino sector parameters satisfies the experimental bound of the baryon asymmetry from the Planck collaboration [73].
The rest of the paper is organized as follows. In section 2, we present the particle content of the model as well as their transformation properties under the SU (5)×D 4 ×U (1) symmetry. In section 3, we derive the mass matrices of the charged fermions and give brief comments on the fast proton decay operators within our construction. In section 4, we study the neutrino sector where the analytical expressions of the neutrino masses and mixing parameters are obtained as a function of the model parameters. In section 5, we show that a perfect fit to the fermion masses and mixings can be obtained for all observables. In section 6, we carry out an analytical and a numerical study of the BAU via the leptogenesis mechanism. A conclusion is given in section 6. Appendix A describes the messenger sector of the model. Appendix B shows that the contribution of the charged leptons and the higher dimensional Dirac operators to the lepton asymmetry is highly suppressed to account for the BAU. Appendix C provides some tools on D 4 discrete group. Appendix D describes the realization of the vacuum alignment of D 4 flavon doublets.

Theoretical setup
In this section, we describe the different sectors of our SU (5) × D 4 × U (1) GUT proposal and fix some notations. The chiral sector of the minimal supersymmetric SU (5) model involves matter and Higgs superfields which are both supplemented by extra superfields in the present setup. Apart from the usual SUSY SU (5) superfield spectrum, the building blocks of the present model can be classified into four sets: • (a) a renormalizable messenger sector with messenger fields X i associated to down quarks, charged leptons and neutrinos, and Y i associated to the up quarks; details on this sector is provided in appendix A, • (b) two additional higher dimensional Higgs fields in the 24 and 45 GUT representations required for gauge symmetry breaking and for generating Yukawa coupling ratios compatible with the data, • (c) several flavon superfields carrying quantum numbers under the flavor symmetry D 4 × U (1) needed to break the flavor symmetry and structure the fermions mass matrices, and • (d) three right-handed neutrinos N c i=1,2,3 responsible for generating the tiny neutrino masses via the type I seesaw mechanism as well as the BAU through the leptogenesis mechanism.
Recall that the usual matter superfields denoted as T i = (u c i , e c i , Q iL ) and F i = (d c i , L i )with i = 1, 2, 3 refers to the three generations of matter -fit into the 10 i and5 i representations respectively. Recall also that the low energy Higgs doublets H u and H d of the minimal supersymmetric standard model (MSSM) arise from H 5 and a mixture of H 5 and H 45 respectively. The 45-dimensional Higgs is usually used to produce the GJ relations differentiating between the (2-2) entry of the down quark and charged lepton mass matrices; however, it has been shown in [33,34] that there are many other options which are preferred compared to GJ relations. These alternatives arise from higher-dimensional operators involving essentially higher dimensional Higgs representations. In our proposal, we use H 45 and H 24 to produce the following ratios of the diagonal Yukawa couplings y e /y d = 4/9 and y µ /y s = 9/2 which are in perfect agreement with experimental data [62]. The different steps leading to these ratios is elaborated in the next section. The 45-dimensional Higgs H 45 satisfy the following relations where υ 45 is the VEV of H 45 . As for the adjoint Higgs H 24 which is also responsible for breaking the SU (5) group, it develops its VEV along the direction All the above superfields carry as well quantum numbers under the D 4 × U (1) group as depicted in table (1). In this table, F 2,3 and N c 3,2 notations stand for D 4 doublet assignments (F 2 , F 3 ) T and (N 3 , N 2 ) T respectively. On the other hand, the D 4 × U (1) invariance requires the introduction of several flavon fields in all the sectors of the model. In the up-quark  sector, only the top quark mass arises from a tree level Yukawa coupling, the up and charm quark masses are derived from higher dimensional couplings involving five flavon fields denoted as ξ i=1,...,5 . In the down quark and charged lepton sector, four flavon fields denoted as φ, ϕ, Ω and Φ are needed for D 4 × U (1) invariance. When these flavons acquire their VEVs, they break the D 4 group and lead to appropriate mass matrices of down quarks and charged leptons. In the neutrino sector, five flavons are required for U (1) invariance. Three of them, denoted as ρ 1 , ρ 2 and ρ 3 , are assigned into different D 4 singlets while the remaining two denoted as ̥ and Γ are transforming as D 4 doublets. The flavons ρ 1 and ̥ Flavons ρ 1 ρ 2 ρ 3 ̥ Γ D 4 1 +,+ 1 +,− 1 −,− 2 0,0 2 0,0 U (1) 10 10 10 10 10 Table 3. The D 4 × U (1) quantum numbers of the flavons used in the neutrino sector.
lead to the popular tribimaximal mixing matrix [75], while ρ 2 , ρ 3 and Γ are responsible for the deviation of the neutrino mixing angles from their TBM values. The quantum numbers under D 4 × U (1) of these five flavons is as depicted in table (3).

Charged fermion sector
To derive the Yukawa matrices of the charged fermion sector, we start by the up-type quarks Yukawa matrix which descend from the trilinear interaction terms 10 i .10 j .5 Hu ≡ T i T j H 5 where i, j = 1, 2, 3 . However, the up-type quarks Yukawa matrix is generated within our construction from higher order operators derived from several renormalizable terms involving messenger fields Y i and gauge singlet flavon fields ξ i , see appendix A for more details on Y i and tables (1) and (2) to check the invariance under SU (5) × D 4 × U (1) symmetry. After integrating out these messenger fields 7 we obtain the invariant effective superpotential for the up quarks where y u ij are the Yukawa coupling constants and Λ is the cutoff scale of the model which we take as the GUT scale. The D 4 flavor symmetry is broken by the VEVs of the flavon fields as ξ i = υ ξ i with i = 1, ..., 5 while the electroweak doublet H u contained in H 5 acquire its VEV as usual H 5 = υ u . Assuming that the parameters in W up are all real, the Yukawa matrix of up-type quarks can be written as We will now proceed with the down-type quarks and charged leptons generated from the same Yukawa coupling 10 i .5 j .5 H d ≡ T i F j H 5 where i, j = 1, 2, 3 are the generation indices. The superpotential leading to the Yukawa matrices is obtained from a renormalizable superpotential that contains messenger fields denoted as X i . However, this time we need to add higher-dimensional Higgs representations to differentiate between down quarks and charged lepton masses, in particular we use the adjoint Higgs H 24 and the 45-dimensional Higgs H 45 for this purpose, see eqs. (2.1) and (2.2). Therefore, by using the superfield assignments in tables (1) and (2) and integrate out the messenger fields, we get the following invariant effective superpotential for the down quarks and charged leptons where y d ij are the Yukawa coupling constants associated to the down quarks and charged leptons. To illustrate how the adjoint Higgs contributes to the entries of the Yukawa matrices and leading subsequently to a particular CG factors that distinguish the down quarks Yukawa couplings from those of the charged leptons,let us discuss the y d 11 effective operator in (3.3). This term is achieved by integrating out the heavy messenger fields from the following renormalizable terms 8 After integrating out X 1,2 and X 1,2 from the first three terms in (3.4), we are left with the first operator in (3.3); T 1 F 1 φϕH 5 . On the other hand, the last two terms in eq. (3.4) are responsible for the appearance of the square of the Higgs adjoint VEV H 24 2 in the denominator of the effective operator. Specifically, the masses of the messenger pairs X 1,2 and X 1,2 are achieved when H 24 acquire its VEV 9 H 24 with the group structure given in 8 The coupling constants are omitted in W Ren e,d for clarity. 9 We assume for simplicity that the VEV of the adjoint Higgs is around the GUT scale as well as the cutoff scale; Λ ≡ H24 = MGUT ≃ 2 × 10 16 GeV. eq. (2.2) which is then followed by integrating out X 1 X 1 and X 2 X 2 to obtain eventually the first effective operator in (3.3). According to the group structure of H 24 , the down quark mass is multiplied by the inverse of the CG factors in the first three entries of the adjoint Higgs VEV in eq. (2.2) which is just 1 in this case, while the electron mass is multiplied by the inverse of the fourth and fifth components such that the resulting CG coefficient is The same discussion holds for the second effective operator in (3.3) while for the third effective operator the CG factors arise in the numerator. For completeness, when the flavon fields acquire their VEVs in accordance with the following alignment we end up with the Yukawa matrices of the down-type quarks Y d and charged leptons Y e expressed as These Yukawa matrices imply diagonal Yukawa couplings y d = b 11 , y s = b 22 , y b = b 33 , y e = 4 9 b 11 , y µ = 9 2 b 22 and y τ = b 33 where y d , y s , and y b stand for the eigenvalues of the down quark Yukawa matrix Y d , and y e , y µ , and y τ stand for the eigenvalues of the charged lepton Yukawa matrix Y e . Thus, we find that for the third family Yukawa coupling we have the well-known b − τ unification; y τ = y b which is still compatible with experimental constraints [33], while for the first two families -instead of the GJ relation -we find alternative GUT predictions with modified CG factors given as In general, to test the validity of the GUT Yukawa couplings in a model of low-energy SUSY such as the MSSM, an accurate incorporation of SUSY threshold effects is necessary especially in the case of large or medium tan β [76][77][78][79]. However, there have been some studies showing that the threshold corrections may be ignored when the running of fermion masses to the GUT scale are included, see, e.g., [80,81]. Moreover, to check the validity of the ratios in eq. (3.8), there are two particular constraints for their GUT values developed in reference [62]; these are given by whereη l andη q denote the threshold correction parameters while the numerical values represent the 1σ uncertainties. From these relations, one can derive the following double ratio at the GUT scale independent of the threshold corrections [62] y µ y s As a result, we find in the present model that the ratios between Yukawa couplings of the first two generations given in eq. (3.8) give rise to the relation yµ ys y d ye ≃ 10.12 which is consistent with the double ratio at the GUT scale given in eq. (3.10).

• Comments on proton decay
Before we turn to the neutrino sector, we give brief comments on proton decay which is one of the most important predictions in GUTs. It is well known that, in the framework of the minimal SUSY SU (5), the fast proton decay comes from the contributions of the dimension four T F F and dimension five T T T F baryon number violating operators 10 . These operators lead to proton lifetime lower than the limit provided by the Super-Kamiokande experiment [83]. It has been shown that the d = 4 operators can be prevented by imposing the usual R symmetry like in the case of the MSSM [84]. On the other hand, the d = 5 operators T i T j T k F l are generically induced via the exchange of color triplet Higgsino [85,86]. This issue of Higgsino-mediated proton decay 11 is intimately connected with the so-called doublet-triplet splitting problem-that is the problem of differentiating between the masses of the Higgs triplets and the Higgs doublets contained in the five dimensional Higgs of the SU (5) GUT-The most effective ways proposed in the literature to resolve this splitting problem is provided by the missing partner (MP) and the double missing partner (DMP) mechanisms [90][91][92][93].
In our model, the renormalizable operators T i F j F k are forbidden because they transform nontrivially under the U (1) symmetry and thus, the proton stability at dimension 4 is guaranteed. Moreover, the d = 4 operators that may arise from the coupling with the flavon fields present in the model are also prevented by the U (1) symmetry as can be checked easily from the field assignments in tables (1-3). For the d = 5 operators, recall first that the usual tree-level Yukawa couplings of the first and second generations are prevented by the U (1) symmetry, and thus the usual operators T i T j T k F l are absent in our model. On the other hand, since the masses of quarks and charged leptons are generated from the effective operators in eqs. (3.1) and (3.3) that involve flavon fields, the operator T T T F inducing proton decay can arise -after integrating out the color-triplet Higgs with GUT-scale mass M T -from higher dimension-7 and dimension-8 operators of the form where M stands for the cutoff scale Λ or the VEV of the adjoint Higgs, n = 0, 1 and f i stands for the flavon fields ξ i=1,...,5 , φ, ϕ, Φ, and Ω. It is clear that these higher dimensional operators are consistent with the messenger content given in appendix A considering that they are derived from the effective operators in eqs. (3.1) and (3.3) which are themselves obtained after integrating out a set of messenger fields required to make the model renormalizable. As an illustration, after integrating out the messenger fields Y 1 and Y 1 from the renormalizable superpotential W Ren up ⊃ H 5 T 1 Y 1 + Y 1 T 1 ξ 1 (see appendix A for the complete superpotential), we obtain the first operator in eq. (3.1) given by (1/Λ)T 1 T 1 H 5 ξ 1 . Then, after integrating out the colored Higgs triplet from this resulting operator and the last operator in eq. (3.3), we obtain the dimension-7 operator Since M T and Λ are both expected to be at the GUT scale, it is straightforward to realize that the contribution of the operator (3.12) to proton decay is sufficiently suppressed. The same discussion holds for all the allowed T i T j T k F l operators generated with a highly suppressed factors manifested by the ratios for dimension-8 operators.

Neutrino sector
The fermion sector of the supersymmetric SU (5)-GUT model is extended by three righthanded neutrino superfields N c i=1,2,3 transforming as gauge singlets 1 i , and carrying quantum numbers under the D 4 × U (1) flavor group. Therefore, the light active neutrino masses are generated through the famous type I seesaw mechanism. In our setup with the quantum numbers of matter and Higgs superfields given in table (1), five flavon superfields are required for U (1) invariance in such a way that they all couple only to the Majorana mass term N c i N c j and they all carry the same U (1) charge q U (1) = 10. Three of these flavons denoted as ρ 1 , ρ 2 and ρ 3 are assigned to the D 4 singlets 1 +,+ , 1 +,− and 1 −,− respectively, while the remaining two denoted as Γ and ̥ are assigned to the D 4 doublet 2 0,0 . Thus, by using the quantum numbers in tables (1) and (2), the superpotential invariant under the SU (5) × D 4 × U (1) group is given by where λ i=1,..., 8 are Yukawa coupling constants. The first two terms in W ν are the Dirac Yukawa terms leading to the Dirac mass matrix m D while the remaining couplings give rise to the Majorana mass matrix m M . Our aim here is to achieve a configuration from W ν that is consistent with the well-known trimaximal mixing matrix which allows naturally for nonzero reactor angle θ 13 , nonmaximal atmospheric angle θ 23 and for sin 2 θ 12 = 1/3. The T M 2 matrix is known to preserve the second column of the famous TBM matrix which is ruled out by the data from reactor neutrino experiments; nevertheless, since it is congruous with the solar and atmospheric angles, it can still be used as a good zeroth-order approximation. Before we develop our neutrino mass matrix m ν , let us recall briefly some of the properties of the mass matrix acquired by TBM and T M 2 . For the TBM matrix, the mass matrix m ν must respects the well-known µ − τ symmetry referring to the invariance of m ν after the interchange of the µ and τ indices [51][52][53][54][55], and the following condition among the entries of m ν : (m ν ) 11 + (m ν ) 12 = (m ν ) 22 + (m ν ) 23 . The deviation from TBM is realized by adding small perturbations to m ν in such a way that the µ − τ symmetry gets broken. There are in particular two matrix perturbations that give rise to a mass matrix with magic symmetry known to be consistent with T M 2 [63]; these two matrices are given as follows Now, let us use these properties in our superpotential W ν and derive the mass matrices of Dirac and Majorana neutrinos to calculate the total neutrino mass matrix using the type I seesaw formula The Higgs doublet develops its VEV as usual H u = υ u while we assume that the VEVs of the D 4 breaking flavon fields point in the following directions 3) The study of the potential which gives rise to the alignment of the flavon doublets is discussed in appendix D. By using the tensor product of D 4 irreducible representations given in eqs. (C.2) and (C.3), we find that the Dirac and Majorana mass matrices have the following forms The Majorana mass matrix is decomposed in terms of two matrices to show that the TBM conditions and its deviation to the T M 2 are obtained from m M 1 and m M 2 , respectively. Accordingly, the µ − τ symmetry and the condition (m ν ) 11 + (m ν ) 12 = (m ν ) 22 + (m ν ) 23 require the imposition of the following assumptions on m D and m M 1 while the deviation from TBM to T M 2 requires a mass matrix with the magic symmetry which is conceivable by the imposition of the following assumption on m M 2 leading to the form of the matrix perturbation δm 1 ν in (4.2). The plausibility of these assumptions is discussed in appendix D. To simplify the parametrization of the total neutrino mass matrix and later the expressions of neutrino masses as well as the mixing angles, we parametrize the Majorana mass matrix as follows where M R is the mass scale of the heavy RH Majorana neutrinos and a = It is clear to verify that in the limit where k → 0, this matrix obeys the µ − τ symmetry, and thus it is diagonalized by the TBM matrix which is CP conserving and predicts θ 13 = 0 and θ 23 = π/4. Therefore, the presence of k is necessary to break the µ − τ symmetry and produce a small deviation from the TBM pattern as mentioned above. Without loss of generality, only the parameter k is taken to be complexk → |k| e iφ k where φ k is a CP violating phase -which is sufficient to ensure CP violation in the lepton sector. On the other hand, the matrix m ν enjoys the magic symmetry property which refers to the equality of the sum of each row and the sum of each column in the neutrino mass matrix. This property implies that the neutrino matrix is diagonalized by the well-known trimaximal mixing matrix U T M 2 so that m diag The full mixing matrix is given by 2 , e i α 31 2 ) is a diagonal matrix that contains the Majorana phases α 21 and α 31 . The parameters θ and σ are respectively an arbitrary angle and a phase which will be related to the neutrino oscillation parameters; the observed neutrino mixing angles θ ij and the Dirac CP phase δ CP . The diagonalization of the neutrino matrix 4.8 by U T M 2 induces relations between our model parameters and the trimaximal mixing parameters σ and θ, we find As a result, the eigenmasses of m ν are as follows where the denominators of these masses corresponds to ratios of the right-handed neutrino masses and their mass scale M R Regarding the mixing angles, it is well-known that the total lepton mixing matrix is derived from the product between two matrices; U P M N S = U † l U ν where U l is the matrix that diagonalizes the charged lepton mass matrix while U T M 2 is as described above. In the present model however, it is easy to check that the charged lepton mixing angles θ l ij derived from the diagonalization of m l = υ d Y e -where Y e is given in eq. (3.7)-are all equals to zero; thus, they do not affect the neutrino mixing angles derived from U ν . Therefore, by using the PDG standard parametrization of the PMNS matrix [94], the reactor, solar and atmospheric angles are expressed as Before we perform a numerical analysis of neutrino masses and model parameters, we should notice that the above masses and mixing parameters are valid at the GUT scale. Therefore, to match these parameters with the experimental values (of the mixing angles, the CP phase and the mass squared differences), their evolution from the GUT scale to low energy must be carried out. However, although the final values are model dependent, it was illustrated in ref. [82] that for SUSY models, if tan β is small, the RG-induced effects on the above parameters are controllable and can be safely neglected. Accordingly, since the type I seesaw mechanism is related to physics at very high energy scales, we can work in a scenario with small neutrino Yukawa couplings in such a way that their contribution to the RG evolution can be neglected [62].

Numerical analysis and results
In this section, we carry out a detailed numerical analysis for both charged fermion and neutrino sectors. For the charged fermion sector, we fix the values of the model parameters in order to reproduce the observed fermion Yukawa couplings and the CKM mixing parameters at the GUT scale within 1σ ranges. As for the neutrino sector, we constrain our model parameters using the 3σ allowed range of the neutrino oscillation parameters. We also use constraints from non-oscillatory experiments to make predictions concerning the physical observables m ββ , m β , and m i .

Numerical fits for charged fermion sector
Our model predicts the Yukawa couplings and mixing parameters at the GUT scale which we assumed to be also the flavor symmetry breaking scale. To compare the obtained spectrum of our model with the data extrapolated at the GUT scale, the experimental values must run up to the GUT scale taking into account the SUSY parameters; tan β and SUSY threshold correction effects. Such an analysis has been performed in ref. [62], where the extracted CKM parameters and all Yukawa couplings at the GUT scale for tan β = 5 and tan β = 10 with M SU SY = 1 TeV and η b = −0.2437 are given in table (3) of ref. [41]. In our model, recall that the mass matrices of the charged fermions are generally expressed as  and tan β = 5. For tan β = 10, the numerical values are reported in table (4). The values reported in this table are fixed respecting the fact that the magnitude of the flavon VEVs are smaller than the flavor symmetry breaking scale; υ f lavons < M GU T , while we fix the phase ǫ to the value π 3 which yields the correct experimental fit of the CP -violating Dirac phase of the quark sector.The above estimates concerning the input parameters produce the values of the physical quantities -namely the quark mixing angles, the Yukawa couplings and the CP phase -at the GUT scale; these numerical values are as reported in table (5). We repeat the same numerical fit for tan β = 5, where the input parameters and the output for the physical parameters are reported in tables (6) and (7), respectively. This fit has been performed using the Mixing Parameter Tools package [82]. The obtained values are in a good agreement with the GUT scale data for both tan β = 5 and tan β = 10 [41,62].  Table 7. The predictions for the Yukawa eigenvalues, the mixing angles and the CP phase of the quark sector for tan β = 5

Neutrino phenomenology
The fact that the neutrino mass ordering remains unknown requires the investigation of the two possible options: either ∆m 2 31 > 0 referred to as normal mass hierarchy or ∆m 2 32 < 0 known as the inverted mass Hierarchy (IH). In the latter case that implies m 3 < m 2 < m 1 , it is easy to deduce from the first relation in eq. (4.13) as well as the 3σ region of the reactor angle from ref. [3] that the parameter θ lies in the interval 0.1763 ≤ θ ≤ 0.1920. On the other hand, by requiring the values of the mass-squared differences ∆m 2 ij within their 3σ experimental ranges and using the eigenmasses in eq. (4.11) as well as the constraint on the sum of neutrino masses from cosmological observations m i < 0.12 eV [73], we find that θ lies in the interval 0.398 θ 0.579 which implies that both sin 2 θ 13 and sin 2 θ 12 fall far outside their 3σ experimental range. For this reason, the IH scheme is excluded in our model.
As regards to the NH scheme, we rewrite the masses m 2 and m 3 in terms of the lightest neutrino mass m 1 and the mass squared differences as m 2 = m 2 1 + ∆m 2 21 and m 3 = m 2 1 + ∆m 2 31 . Moreover, by using eqs. Notice by the way that the parameter k is responsible for the deviation from the TBM values of the mixing angles. This deviation is encoded in the parameter θ which is easily seen when we set θ → 0 in eq. (4.13) resulting to restore the TBM values. From the top right panel of figure (1), we find that the range of θ is also restricted to 0.17548 θ 0.19129 while the range of the reactor angle remains almost unchanged compared to its 3σ allowed range.
In figure (1), the bottom panel shows the correlation between the parameters a and b with the color code showing the phase φ k . By taking into account the 3σ experimental ranges of ∆m 2 ij , sin θ ij and δ CP from the most recent global fit by NuFIT collaboration [3], and the current cosmological upper bound on the sum of the three light neutrino masses given by m i < 0.12 eV, we find that the range of the phase φ k gets more restricted compared to its input range; 0.55275 φ k 2.56095. Regarding the Dirac CP phase δ CP , the results reported by the T2K long-baseline experiment showed strong hints for CP violation in neutrino oscillations while CP conservation is disfavored at 2σ level [6]. One approach to estimate the magnitude of δ CP is by means of the Jarlskog invariant parameter defined as J CP = Im(U e1 U * µ1 U µ2 U * e2 ). In the PDG standard parametrization, this parameter is exhibited in terms of the three mixing angles and the Dirac CP phase as follows [94] J CP = 1 8 sin 2θ 12 sin 2θ 13 sin 2θ 23 cos θ 13 sin δ CP (5.4) while in the case of the trimaximal mixing, it takes a simpler form given byJ T M CP = 1/6 √ 3 sin 2θ sin σ. By matching J T M CP with eq. (5.4), we find a correlation between the Dirac CP phase, the arbitrary phase σ, and the atmospheric angle sin 2θ 23 sin δ CP = sin σ (5.5) Taking into account the fact that atmospheric angle is well determined experimentally as well as the fact that the range of σ given in eq. (5.3) excludes the exact value of nπ with n can be any integer, it is easy to deduce analytically that the CP conserving values of δ CP are not allowed which implies that the present model admits only the CP violating values of δ CP .

• Neutrino masses from non-oscillatory experiments
Constraining the absolute neutrino mass scale is one of the most important purposes of the forthcoming neutrino experiments. This scale can be probed by various non-oscillatory neutrino experiments. Cosmological observations are in particular a powerful tool to probe the total sum of neutrino masses. Indeed, in the framework of ΛCDM model with three massive active neutrinos, the latest Planck data combined with baryon acoustic oscillations (BAO) measurements provided an upper bound on the sum of neutrino masses of m i < 0.12 eV [73]; see also ref. [74] for a comprehensive analysis of the changes in the upper bounds of m i after taking into account neutrino oscillation data. Another way to probe this scale is through direct neutrino mass determination where the study of the electron energy spectrum near its endpoint region is up to date the most sensitive method to determine the electron antineutrino mass. The effective electron neutrino mass is defined in terms of the three neutrino mass eigenvalues m i and the flavor mixing parameters . Currently, the most valid bounds on m β are presented by the KATRIN experiment which provides an upper limit on the electron antineutrino mass of 1.1 eV [95] and eventually aims at a sensitivity of 0.2 eV [96]. Using the upper limit m i < 0.12 eV and the neutrino oscillation parameters (θ ij and ∆m 2 ij ) within their currently allowed 3σ ranges as well as the restricted interval of our model parameters given in the previous subsection, we show in the left panel of figure (2)    As a result, the predicted values of m i around the lower bound ∼ 0.064759 eV are consistent with normal mass hierarchy which requires 12 m i (eV) 0.065431.This lower bound of m i may be achieved in the forthcoming experiments with further cosmological data such as CORE+BAO aiming to reach a 0.062 eV sensitivity on the sum of the three active neutrino masses [97]. In the right panel of figure (2), we show the correlation between m β and the lightest neutrino mass m 1 . The orange region is achieved by varying all the input parameters (∆m 2 ij , a, b, |k| and φ k ) in their 3σ ranges while the red points stands for our model prediction. We find that the effective electron neutrino mass is given by 0.0100158 m β (eV) 0.023765 (5.7) It is clear that our predictions for m β are too small when compared to the anticipated future β-decay experiments sensitivities such as KATRIN (∼ 0.2 eV) [96], HOLMES (∼ 0.1 eV) [98], and Project 8 (∼ 0.04 eV) [99]. If the actual electron neutrino mass would be measured by one of these experiments the neutrino sector of the present model will be ruled out. Otherwise, the obtained values could be probed by new experimental projects that must aim to reach improved sensitivities around 0.01 eV.
Another possible portal to probe the scale of neutrino masses comes from experiments exploring the nature of neutrinos which is also one of the present objectives in the field of neutrino physics. Up to now, the probe of the Majorana nature of neutrinos is available only through 0νββ decay. This is a process that violates lepton number L by two units, and since there are no SM interactions that violates L, the discovery of 0νββ would have 12 This bound is obtained by taking the best fit values of the mass squared differences ∆m 2 21 and ∆m 2 31 from ref. [3] with any value of the lightest neutrino mass m1 obtained in eq. (5.6).
interesting implications for model building beyond the SM such as the existence of a new mechanism for mass generation compared to the charged fermions obtaining their masses via the Higgs mechanism. The 0νββ decay amplitude is proportional to the effective Majorana mass |m ββ | defined as |m ββ | = i U 2 ei m i , and may be expressed in terms of our model parameters and the parameters of the U ν mixing matrix  The predictions for |m ββ | are far from the current sensitivities mentioned above, on the other hand, the next-generation experiments such as GERDA Phase II, CUPID, nEXO and SNO+-II will cover the values of |m ββ | in eq. (5.9) as they aim for sensitivities around |m ββ | ∼ (0.01 − 0.02) eV [104], |m ββ | ∼ (0.006 − 0.017) eV [105], |m ββ | ∼ (0.008 − 0.022) eV [106] and |m ββ | ∼ (0.02 − 0.07) eV [107] respectively.

Leptogenesis
In this section, we investigate the generation of the baryon asymmetry of the universe within our SUSY SU (5) × D 4 × U (1) model in the case of normal mass hierarchy. In this scenario, the presence of three RH neutrinos as the key ingredients for small neutrino masses can also produce the BAU through the leptogenesis mechanism. In this case, a lepton asymmetry Y L (equally B − L asymmetry Y B−L ) is generated through the out-ofequilibrium CP violating decays of RH neutrinos N c i (and their supersymmetric partners in SUSY models) in the early universe. This lepton asymmetry is then partially converted into the baryon asymmetry of the universe Y B via (B + L) violating sphaleron transitions [19].
The excess of baryons over anti-baryons is evaluated through the baryon asymmetry Y B relative to the entropy density s or the baryon asymmetry η B relative to the density of photons n γ , defined respectively as where n B and n B are the number densities of baryons and anti-baryons. The experimental values of these parameters obtained from the latest data from the Planck satellite are given by Y B = (8.72 ± 0.08) × 10 −11 and η B = (6.13 ± 0.04) × 10 −10 [73]. In order to perform an approximate estimation of Y B , we use the following two approaches: • It is well-known that when the right-handed neutrino mass spectrum is hierarchical, the contribution to the lepton asymmetry can be created only by the decay of the lightest RH neutrino [13,108,109]. Since only the NH is allowed in our model, it is clear from eqs. (4.11) and (4.12) as well as from figure (4)  • Since all Majorana masses are above T = 10 12 (1 + tan 2 β) for tan β = 5 and tan β = 10-used in the charged fermion sector to fit the experimental data-we perform our study in the one flavor approximation where all charged leptons are out-of-equilibrium and there is no difference between them at the time leptogenesis takes place.
Taking this two points into consideration, the magnitude of B − L asymmetry generated by N 3 can be parameterized as follows [110] is the CP asymmetry produced in the decay of N c 3 (Ñ c 3 ), η 33 is the efficiency factor 13 related to the washout of the CP asymmetry ε N 3 (εÑ Normal Hierarchy t a n β > 5 t a n β < 5 tanβ=5 and Y eq N 3 is the number density of N 3 over the entropy density (n N 3 /s) defined as [110] Y eq where ζ(3) denotes the Riemann zeta function and g * is the number of spin-degrees of freedom in thermal equilibrium; in MSSM g * = 228.75.
The first component to consider in our calculation is the source of the CP asymmetry given by the CP violating parameters ε N i and εÑ i in N c i andÑ c i decays, averaged over the different decay channels N c i → LH u ,LH u andÑ c i →LH u , LH u respectively. These RH neutrinos and their superpartners decay, with decay rates that reads respectively as ) while the CP violating parameters are given by In SUSY models, the effects from the superparticles produce relatively small corrections to the BAU [111]. Therefore, by ignoring supersymmetry breaking 14 -as a result of which the RH neutrinos and their superpartners have equal masses M N i = MÑ i , equal decay rates Γ N i = ΓÑ i and equal CP asymmetries ε N i = εÑ i [110]-we can factorize by the CP asymmetry in eq. (6.2) as Likewise, when the equilibrium densities for leptons and sleptons are equal Y eq N i ≈ Y eq N i , we find that the B − L asymmetry parameter Y B−L is enhanced by a factor of 2. Bringing together all these effects, the CP asymmetry can be explicitly expressed in the one flavor approximation as 14 For cases where SUSY can not be ignored see [112,113].
and Y ν is the neutrino Yukawa coupling matrix in the basis where the Majorana mass matrix m M and the Yukawa matrix of the charged leptons Y e are both diagonal. However, as explained in the appendix B, the contribution of the mixing matrix that diagonalizes Y e leads to CP asymmetry of order |ε N 3i | ∼ O(10 −12 −10 −10 ) which suppress the value of the baryon asymmetry Y B . Therefore, in order to meet the requirements of a successful leptogenesis that produces the experimental values of Y B , we add a correction to the leading order Dirac Yukawa matrix in eq. (4.7).
To account for this correction, we introduce a new flavon field ω which transforms as 1 +− under D 4 with zero U (1) charge, we have where λ 9 is a complex coupling constant λ 9 = |λ 9 | e iφω . This effective coupling is obtained from the following renormalizable superpotential where X 5 is a messenger field that transforms as SU (5) quintet, D 4 singlet 1 +− and has a U (1) charge equals to −8. The contribution of δW D is small and will not provide any considerable effect in the obtained neutrino masses and mixing. When the flavon field ω acquires its VEV as ω = υ ω , we end up with the total Yukawa mass matrix 15 where κ = |λ 9 |υω Λ is a free parameter which should be small (κ << 1) in order to produce the correct BAU. Taking into account this correction, the total Yukawa neutrino mass matrix is defined as Y ν = U † ν Y D . Thus, after calculating the product Y ν Y † ν in the basis where the Majorana mass matrix is diagonal, the CP asymmetry parameter ε N 3 corresponding to the lightest RH neutrino N 3 is given approximately by wherem i are the washout mass parameters expressed asm The second component to address in this computation is the efficiency factor η 33 . A good approximation is to consider the region of RH neutrino masses smaller than 10 14 GeV, preventing possible washout effects from ∆L = 2 scattering processes. In this case, 15 Notice that the total light neutrino mass matrix involving the small correction δYD is almost similar to the one in eq. (4.1) and yields approximately to the same neutrino phenomenology. the efficiency factor η 33 can be expressed approximately as a function of the washout mass parameterm 3 as [110] Notice here that the smallness of the parameter κ << λ 1 implies that the washout mass parameters become approximately identicalm i ≈ m i and hencem 3 ≈ m 3 . Moreover, since the neutrino mass m 3 has values close to 0.5 × 10 −1 eV as given in eq. (5.6), then the efficiency factor η 33 in our model is roughly η 33 ≈ 0.5 × 10 −2 . Let us now derive the expression of the baryon asymmetry parameter Y B . This parameter is related to lepton asymmetry Y B−L given in eq. (6.2) through sphaleron transitions, we have [114]  Furthermore, it is clear from the CP asymmetry parameter ε N 3 in eq. (6.10) that the source of CP violation in the lepton sector could arise from the interplay between the low energy CP phases (Dirac and Majorana phases δ CP , α 31 and α 21 ) and the high energy CP phase φ ω originated from the complex coupling constant λ 9 in the Dirac mass matrix; see eq. (6.7). Therefore, we plot in figure (6), the baryon asymmetry parameter Y B as a function of the low energy CP phases (α 31 , α 21 and δ CP ) and the high energy CP phase φ ω which is the key ingredient for generating the observed range of Y B . We observe that the ranges of the Majorana phases (top panels) and the Dirac phase (bottom left panel) are not constrained compared to their inserted intervals, nevertheless, the scattered pointsincluding the CP conserving values of the Majorana phases α 31 , α 21 = 0, π-are consistent with the Planck limit on Y B . However, even in the case of these CP conserving values, CP violation is guaranteed by the high energy CP phase φ ω . For this reason, we plot in the bottom right panel of figure (6) the correlation between Y B and φ ω where we find that φ ω vary within the range 0 φ ω 6.279, while the CP conserving values φ ω = π 2 and φ ω = 3π 2 as well as the regions around them are excluded (the sections of the blue line without any points). Therefore, this source of CP violation plays a crucial role in generating the baryon asymmetry in the present model.

Summary and conclusion
In this work, we have presented a model with a D 4 family symmetry to explain the fermion flavor structures in the framework of supersymmetric SU (5) grand unified theory. Besides the SU (5) × D 4 model proposed in ref. [49] -which was merely an implementation of the D 4 in SU (5) -this is the first comprehensive study of a four-dimensional SU (5) GUT with a flavor symmetry that does not include triplet irreducible representations. To establish a thorough analysis of this model, we have enlarged the field content of the usual scalar and matter sectors of SUSY SU (5) GUT. Explicitly, we have added three RH neutrinos to generate neutrino masses via the type I seesaw mechanism, heavy messenger fields to make the model renormalizable at the GUT scale, higher dimensional Higgs multiplets to produce realistic quark-lepton Yukawa coupling ratios, and gauge singlet flavon fields to give rise to the observed fermion mass spectrum and mixing through the spontaneous symmetry breaking of the flavor group. Moreover, after adding these fields, an additional U (1) symmetry is imposed to control the invariance of the superpotentials in the quark and lepton sectors, and also the dangerous d = 4 and d = 5 proton decay operators.
Integrating out the heavy messenger fields from the renormalizable superpotentials gives rise to higher-dimensional effective operators responsible for the fermion flavor structures. Moreover, to go beyond the minimal SU (5) relation Y T e = Y d as well as the popular GJ relations which are disfavored by the experimental results, we have considered the CG factors y e /y d = 4/9 and y µ /y s = 9/2 which are realized through the coupling of messenger fields with higher 24-and 45-dimensional Higgs fields and the flavon fields. This has led to the double ratio yµ ys y d ye ≃ 10.12 which is in good agreement with the phenomenological value at GUT scale. We have performed a numerical analysis in the down and charged lepton Yukawa sector where we have fixed our model parameters -the free parameters in the entries of the Yukawa matrices -and provided an accurate fit to the mixing angles, the Yukawa couplings and the Dirac CP phase of the quark sector at the GUT scale.
The small neutrino masses are generated via the type I seesaw mechanism where the Dirac and Majorana mass matrices arise from renormalizable terms. The resulting neutrino mass matrix is of the trimaximal mixing form which is compatible with current neutrino data. By using the 3σ experimental range of sin 2 θ 13 for both neutrino mass hierarchies we derived the range of the trimaximal mixing parameter θ where we found that only the normal mass hierarchy is allowed. Therefore, we have carried out our numerical study in this regime where we found that our model allows for θ 13 = 0 and θ 23 < π/4 as well as excludes the conserving values of the Dirac neutrino CP phase δ CP . We have explored the neutrino parameter space and showed numerically the predicted ranges of the non-oscillatory observables m β , m ββ and Σm i that fit the 3σ experimental range of the mixing angles and the mass squared splittings. In particular, the predicted values of m ββ are testable at future neutrinoless double beta decay experiments.
Since the low energy CP violation which manifest itself in the mixing matrix in the form of the Dirac and Majorana phases is not sufficient to describe the BAU, we have added an extra effective operator in the neutrino sector to produce the observed BAU via the leptogenesis mechanism. This operator which involves a new flavon field ω is obtained, as in the quark sector, by integrating out heavy messenger fields. Its contribution serves as a correction that perturbs the structure of the Dirac mass matrix while the high energy CP phase φ ω that arises from the complex coupling constant in this operator is a new source of CP violation. Therefore, we have performed a numerical study to estimate the values of the CP asymmetry parameter ε N 3i that are consistent with the baryon asymmetry parameter Y B . We have focused on the unflavored leptogenesis approximation scenario under which the range of the lightest RH neutrino mass is given by M 3 (GeV) ∈ 2.6 × 10 13 → 10 14 . We found that the CP asymmetry parameter ε N 3 is mainly related to the high energy CP phase φ ω . Therefore, we showed through scatter plots that the CP conserving values φ ω = π 2 and φ ω = 3π 2 as well as the regions around them are excluded, while the lepton asymmetry parameter ε N 3i must be of order |ε N 3i | ∼ O(10 −5 ) to satisfy the Planck limit on Y B .

A Messenger sector
In this Appendix we discuss the renormalizable superpotentials of all the fermions including their Feynman diagrams to obtain the higher dimensional operators relevant for the Yukawa mass matrices. The complete list of the messenger field content including their SU (5) and D 4 representations as well as their U (1) charges is given in table (8). To be precise, the messenger fields Y i are relevant for the up quark sector while X i are involved in the down quark, the charged lepton and the Dirac neutrino sectors. The renormalizable superpotential invariant under D 4 × U (1) associated to the up quarks reads as where we have omitted the coupling constants from all terms for simplicity. The couplings in this superpotential are illustrated by the Feynman diagrams provided in figure (8). After integrating out the messenger fields Y i andȲ i from W Ren up we obtain the effective superpotential responsible for the masses of the up quarks given in eq. (3.1). As for the down-type quark and charged lepton sector, the renormalizable superpotential involving the five-plets messenger fields X i and X i is given by while the renormalizable terms relevant for the effective operator responsible for generating a successful BAU is given as follows Once more, integrating out the these heavy messenger fields give rise to the effective superpotentials of the down quarks, the charged leptons and the Dirac neutrino; see eqs. (3.3) and (6.7). The mass terms of the messenger fields takes the form  B CP asymmetry from the charged lepton mixing and d = 6 Dirac operators In this appendix, we show that the contribution of the charged lepton mixing matrix to the Yukawa mass matrix in eq. (4.7), before adding the correction δY D , cannot accommodate the observed value of the BAU. Starting with the CP asymmetry formula in eq. (6.6) which can be explicitly expressed in the one flavor approximation as For our calculations, the values of the free parameters b ij are fixed by their values in the case of tan β = 5 as reported in table (6). As a result, by inserting the expression of Y ν in eq. (B.1), we find that the CP asymmetry parameter ε N c 3 depends on the coupling constant λ 1 , the trimaximal mixing parameters θ and σ, the Majorana CP phases α 31 and α 21 as well as the light neutrino masses m i=1,2,3 . Approximately, ε N c 3 is given as Assuming that the coupling constant λ 1 is of order one and taking into account the obtained regions of the parameters θ and σ, the Majorana CP phases α 31 and α 21 as well as the neutrino masses m i=1,2,3 , the baryon asymmetry parameter ε N c 3 is up to order O(10 −12 − 10 −10 ). However, as discussed in section 6, to generate the observed baryon asymmetry, the parameter ε N c 3 must be of order O(10 −5 ). Therefore, the charged lepton contribution to the CP asymmetry parameter ε N c 3 is too small and subsequently the baryon asymmetry parameter Y B is strongly suppressed.
Before we close this appendix, we discuss the possibility of producing a successful leptogenesis using higher dimensional Yukawa operators as an alternative to the additional coupling in eq. (6.7). Using the charge assignments of D 4 and U (1) symmetries, we find that there are three invariant six dimensional operators that can be used as a correction to the leading order Dirac Yukawa matrix Since our model is renormalizable, these operators must be derived from renormalizable Yukawa couplings involving the existing messenger fields listed in table (8). For example, generating the operator 1 Λ 2 N c 3,2 F 2,3 H 5 ρ 2 ξ 2 calls for a new messenger field X 6 which transforms as SU (5) quintet, D 4 singlet 1 ++ and has a U (1) charge equals to −18. Nevertheless, the absence of this messenger field in our model forbids the existence of this operator. On the other hand, even if we add messenger fields to allow the operators in eq. (B.3), we end up with a highly suppressed contribution to the CP asymmetry parameter ε N c 3 . As a verification, we use the same example as above where the renormalizable superpotential which induces the effective coupling λ 10 Λ 2 N c 3,2 F 2,3 H 5 ρ 2 ξ 2 is given by W ren D = N c 3,2 F 2,3 X 5 + X 5 ρ 2 X 6 + X 6 H 5 ξ 2 , (B.4) where the coupling constants are omitted for simplicity. Subsequently, the total Yukawa mass matrix reads as where λ 10 is a complex coupling constant λ 10 = |λ 10 | e iφ H . The CP asymmetry parameter ε N 3 corresponding to the lightest RH neutrino N 3 is given approximately by The obtained CP asymmetry parameter ε N 3 is proportional to the factor ε N 3 ∼ |λ 10 |υρ 2 υ ξ 2 Λ 2 2 which involves the flavon VEV υρ 2 Λ from the neutrino sector as well as υ ξ 2 Λ from the up-quark sector. According to the numerical analysis we have performed in the two sectors, we derive the interval of the ratio In order to get an estimate on the obtained CP asymmetry parameter ε N 3 in eq. (B.6), we assume, as is reasonable to do, that the coupling constants λ 7 , λ u 12 and |λ 10 | are of order one and we allow the phase φ H to vary in the interval [0, 2π]. Therefore, we find that the CP asymmetry parameter ε N c 3 is up to order O(10 −12 − 10 −7 ) which is too small to account for a successful leptogenesis. As a result, the baryon asymmetry parameter Y B is strongly suppressed when addressing leptogenesis through the six dimensional operator λ 10 Λ 2 N c 3,2 F 2,3 H 5 ρ 2 ξ 2 . The same discussion holds for the other two operators 1 Λ 2 N c 3,2 F 2,3 H 5 ρ 1 ξ 2 and 1 Λ 2 N c 3,2 F 2,3 H 5 ρ 3 ξ 2 .
C Some aspects of the dihedral group D 4 The dihedral group D 4 is a finite group that is generated by the reflection t and the 45 • rotation s satisfying s 4 = t 2 = I and tst = s −1 . A rotation followed by a reflection is different than a reflection followed by a rotation which means that the two generators s and t do not commute with each other. This non-Abelian group has 5 irreducible representations R i=1,...,5 : one doublet denoted as 2 0,0 , and four singlets denoted as 1 +,+ (the trivial singlet), 17 Notice that the value of the flavon VEV λ u 12 υ ξ 2 Λ = a12 ≃ 0.15866 × 10 −3 in the case of tan β = 5 is much smaller and therefore the estimate on the CP asymmetry parameter εN 3 becomes much suppressed. 1 +,− 1 −,+ and 1 −,− . The indices of these representations represent their characters under the two generators t and s as in the following table The squares of the dimensions of these irreducible representations are related to the order 8 of the D 4 group through the formula 8 = 1 2 +,+ + 1 2 +,− + 1 2 −,+ + 1 2 −,− + 2 0,0 . Let us now turn to the tensor products among the irreducible representations of D 4 . The tensor product between two D 4 doublets is decomposed into a sum of the four singlet representations of D 4 as x 1 x 2 2 0,0 ⊗ y 1 y 2 2 0,0 = (x 1 y 2 + x 2 y 1 ) 1 +,+ ⊕ (x 1 y 1 + x 2 y 2 ) 1 +,− ⊕ (x 1 y 2 − x 2 y 1 ) 1 −,+ while the tensor products among the singlet representations can be expressed as 1 d,e ⊗ 1 f,g = 1 df,eg with d, e, f, g = ± (C.3) For more details on the D 4 group, see, e.g., [36].

D Vacuum alignment of D 4 flavon doublets
Establishing an origin of the VEV directions is an essential part when using non-Abelian discrete flavor symmetries to build models of fermion masses and mixing. In our model, the VEVs of the D 4 doublet flavons pointing in the directions given in eqs. (3.5) and (4.3) were assumed in order to produce the charged fermions and neutrino masses consistent with the experimental data. One of the well-known approaches to check if these VEV directions are a solution of the scalar potential is by introducing a set of alignment fields called driving fields and a continuous U (1) R symmetry. Under such a symmetry, the matter superfields including right-handed neutrinos carry charge +1, flavons and Higgs fields are uncharged while the driving fields have charge +2 [115]. As a result of the these U (1) R charge assignments, the driving fields couple only to flavons and appear linearly in the superpotential, while the vacuum alignment is obtained by setting their F-terms to zero. In general, the alignment through F-terms provide also relations between flavons VEVs. Here, we introduce two driving fields denoted as χ q and χ ν transforming under D 4 × U (1) as The renormalizable terms involving these driving fields invariant under the flavor symmetry D 4 × U (1) are given by W d = c 1 χ q (ΩΦ) 1 −,− + c 2 χ ν (Γ̥) 1 −,+ + c 3 χ ν (̥̥) 1 −,+ + c 4 χ ν (ΓΓ) 1 −,+ + c 5 χ ν ρ 2 ρ 3 (D.2) In the SUSY limit where the F-terms of χ q and χ ν vanish, the condition for the minima are Clearly, the first equation admits three different solutions given by which we have used to produce the Majorana mass matrix provided the following relation between the involved VEVs holds According to the assumptions we have adopted to obtain the total neutrino mass matrixsee eqs. (4.5) and (4.6) -it follows that the set of flavon VEVs {υ ρ 1 ,υ ̥ } and {υ ρ 2 , υ ρ 3 , υ Γ } are of the same order of magnitude. Moreover, the flavon VEVs υ ρ 2 and υ ̥ are related in eq. (D.5) through the couplings c 2 and c 5 which we assume that they are of the same order. As a result, we deduce that all the flavons used in the neutrino sector are comparable to each other which is in agreement with the numerical analysis performed in Section 5. On the contrary, the first equation in (D.3) responsible for aligning the flavon doublets Ω and Φ does not induce any relation between their VEVs υ Ω and υ Φ . This is clearly reasonable since they are not of the same order of magnitude as discussed numerically in section 5. These two flavon VEVs contribute respectively to the second and the third generations of down quarks (charged leptons) which are strongly hierarchical.