Fermion masses and mixings in the 3-3-1 model with right-handed neutrinos based on the $S_3$ flavor symmetry

We propose a 3-3-1 model where the $SU(3)_{C}\otimes SU(3)_{L}\otimes U(1)_{X}$ symmetry is extended by $S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ and the scalar spectrum is enlarged by extra $% SU(3)_{L}$ singlet scalar fields. The model successfully describes the observed SM fermion mass and mixing pattern. In this framework, the light active neutrino masses arise via an inverse seesaw mechanism and the observed charged fermion mass and quark mixing hierarchy is a consequence of the $Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ symmetry breaking at very high energy. The obtained physical observables for both quark and lepton sectors are compatible with their experimental values. The model predicts the effective Majorana neutrino mass parameter of neutrinoless double beta decay to be $m_{\beta \beta }=$ 4 and 48 meV for the normal and the inverted neutrino spectra, respectively. Furthermore, we found a leptonic Dirac CP violating phase close to $\frac{\pi }{2}$ and a Jarlskog invariant close to about $3\times 10^{-2}$ for both normal and inverted neutrino mass hierarchy.


I. INTRODUCTION
The ATLAS and CMS experiments at the CERN Large Hadron Collider (LHC) have found a 126 GeV Higgs boson [1][2][3][4], increasing our knowledge of the Electroweak Symmetry Breaking (EWSB) sector and opening a new era in particle physics. Now the priority of the LHC experiments will be to measure precisely the couplings of the new particle to standard model fermions and gauge bosons and to establish its quantum numbers. It also remains to look for further new states associated with the EWSB mechanism which will allow to discriminate among the different theoretical models addressed to explain EWSB. Despite its great success, the Standard Model (SM) based on the SU (3) C ⊗ SU (2) L ⊗ U (1) Y gauge symmetry is unlikely to be a truly fundamental theory due to having internal problems and unexplained features [5,12]. Most of them are linked to the mechanism responsible for the stabilization of the weak scale, the origin of fermion masses and mixing and the existence of three generations of fermions. Because of these reasons, many people consider the Standard Model to be an effective framework of a more fundamental theory that one expects should have a dynamical explanation for the fermion masses and mixing. The flavour structure of Yukawa interactions is not restricted by gauge invariance, consequently fermion masses and mixing cannot be entirely determined in a gauge theory. In the SM there are in fact arbitrary parameters that must be measured. The lack of predictivity of the fermion masses and mixing in the SM has motivated many models based on extended symmetries, leading to specific textures for the Yukawa couplings. There are models with Multi-Higgs sectors, Grand Unification, Extradimensions and Superstrings, [9][10][11]. In particular, discrete flavour symmetries may play an important role in models of fermion mixings and many models based on flavour symmetries have been proposed in order to provide an explanation for the current pattern of fermion mixing, for recent reviews see Refs. [6][7][8]. The understanding of the discrete flavor symmetries hidden in such textures may be useful in the knowledge of the underlying dynamics responsible for quark mass generation and CP violation. One clear and outstanding feature in the pattern of quark masses is that they increase from one generation to the next spreading over a range of five orders of magnitude, and that the mixings from the first to the second and to the third family are in decreasing order [12][13][14][15]. From the phenomenological point of view, it is possible to describe some features of the mass hierarchy by assuming zero-texture Yukawa matrices [16]. Models with spontaneously broken flavor symmetries may also produce hierarchical mass structures. These horizontal symmetries can be continuous and Abelian, as the original Froggatt-Nielsen model [17], or non-Abelian as for example SU (3)

II. THE FERMION AND SCALAR SECTOR.
We consider the 3-3-1 model where the electric charge is defined by: with T 3 = 1 2 Diag(1, −1, 0) and T 8 = ( 1 2 √ 3 )Diag (1, 1, −2). In order to avoid chiral anomalies, the model introduces in the fermionic sector the following (SU (3) C , SU (3) L , U (1) X ) left-and right-handed representations: where U i L and D i L for i = 1, 2, 3 are three up-and down-type quark components in the flavor basis, while ν i L and e i L are the neutral and charged lepton families. The right-handed sector transforms as singlets under SU (3) L with U (1) X quantum numbers equal to the electric charges. In addition, we see that the model introduces heavy fermions with the following properties: a single flavor quark T 1 with electric charge 2/3, two flavor quarks J 2,3 with charge −1/3, three neutral Majorana leptons (ν 1,2,3 ) c L and three right-handed Majorana leptons N 1,2,3 R (recently, a discussion about neutrino masses via double and inverse see-saw mechanism was perform in ref. [35]). On the other hand, the scalar sector introduces one triplet field with VEV χ 0 = υ χ , which provides the masses to the new heavy fermions, and two triplets with VEVs ρ 0 = υ ρ and η 0 = υ η , which give masses to the SM-fermions at the electroweak scale. The [SU (3) L , U (1) X ] group structure of the scalar fields are: The EWSB follows the scheme SU ( Furthermore, we impose the S 3 flavor symmetry for fermions and scalars so that the full symmetry of our model is extended to be SU ( The inclusion of the discrete S 3 symmetry will allow to reduce the number of parameters in the Yukawa and scalar sector of the SU (3) C ⊗ SU (3) L ⊗ U (1) X model making it more predictive. We choose the S 3 discrete symmetry since it is the smallest non-Abelian discrete symmetry group that contains a two-dimensional irreducible representation which can connect two maximally mixed generations. Besides facilitating maximal mixing through its doublet representation, the S 3 discrete group provide two inequivalent singlet representations which play a crucial role in reproducing fermion masses and mixing [21]. The scalar fields are grouped into doublet and singlet representions of S 3 as follows: Notice that the scalar triplet ρ cannot be assigned to a trivial S 3 singlet 1 since in that assignment, the S 3 symmetry will forbid a trilinear term in the scalar potential crucial to keep the non Standard Model like Higges heavy. Furthermore, the absence of the trilinear term will imply a breaking of the S 3 symmetry in order to have the Φ vacuum consistent with the minimization conditions of the scalar potential.
Regarding the quark sector, it is assumed that all quarks are assigned to S 3 doublets excepting the Q 1 L , Q 2 L , Q 3 L and D 1 R fields, which are assigned to S 3 singlets: With the above spectrum, we obtain the following SU (3) C ⊗ SU (3) L ⊗ U (1) X ⊗ S 3 renormalizable Yukawa part of the model Lagrangian for the quark sector: where n = 2, 3 is the index that label the second and third quark triplet shown in Eq. (1), and h (f ) φij (f = U, D, T, J) are the i, j components of non-diagonal matrices in the flavor space associated with each scalar triplet φ : η, ρ, χ. The interactions among the scalar fields are contained in the following most general potential that we can construct with three scalar triplets: where The S 3 group has three irreducible representations: 1, 1 ′ and 2. The multiplication rules of the S 3 group for the case of real representations are given by [8]: Besides that, the scalar potential can be written in terms of the three scalar triplets as follows: As shown in detail in the Appendix, the Φ vacuum is consistent with the minimization conditions of the scalar potential when the following relation is fullfilled: After the symmetry breaking, as shown in detail in the Appendix, it is found that the mass eigenstates are related to the weak states in the scalar sector by: A 0 where the mixing angles are given by: while the masses of the physical scalar fields are: where we have taken into account that f ≃ v χ ≫ v η , v ρ . Notice that after the spontaneous breaking of the gauge symmetry SU (3) L ⊗ U (1) X and rotations into mass eigenstates, the model contains 4 massive charged Higgs (H ± 1 , H ± 2 ), one CP-odd Higgs (A 0 ) and 4 neutral CP-even Higgs (h 0 , H 0 1 , H 0 2 , H 0 3 ) bosons. Here we identify the scalar h 0 with the SM-like 126 GeV Higgs boson observed at the LHC. We recall that the neutral Goldstone bosons G 0 1 , G 0 2 and G 0 3 correspond to the Z, K 0 and Z ′ gauge bosons, respectively. Furthermore, the charged Goldstone bosons G ± 1 and G ± 2 correspond to the W ± and K ± gauge bosons, respectively [25][26][27][28]. Using the multiplication rules of the S 3 group, it follows that the Yukawa part of the model Lagrangian for the quark sector takes the following form: The Yukawa part of the model Lagrangian for the quark sector can be rewritten as: correspond to the quark mass terms, while −L φQQ includes the interactions of the quarks with the neutral and charged Higgses and Goldstone bosons. These terms are given by: III. TREE LEVEL QUARK MASS TEXTURES.
From Eq. (24) it follows that the mass matrices for the up and down type quarks are given by: where: In sake of simplicity we assume that the Yukawa couplings h Φ are real. Then, the matrix M U satisfies the following relation: where: Now we identify the top and bottom quarks as U 1 and D 1 , respectively, while the remaining SM quarks are identified Then, it follows that the masses of the up-type quarks are given by: Therefore, the up and charm quarks are massless at tree level, while the top quark and exotic T quark get their masses from the η and χ triplets, respectively. Notice that the masslessness of the up and charm quarks at tree level, is a consequence of the absence of mixings among them and between them and the heavy t, T quarks.
Regarding the down type quark sector, we have that the matrix M D can be rewritten as: where: In the down type quark sector we assume that the Yukawa coupling h where: The matrix S D is written in the basis D 1 , D 2 , D 3 = (b, d, s). Furthermore, the matrix S D can be written in the basis d, s, b as follows: The matrices Z D and Y D can be diagonalized by the rotation matrices R D and P D according to: where: Furthermore, the masses of the down-type quarks are given by: Therefore, the hierarchy m d << m s << m b can be explained by the inequality h , with m = 1, 2. Therefore, considering v ρ ∼ v η , we obtain that the triplet ρ couples more strongly to the bottom quark than the triplets η and χ to the exotic down type quarks. This assumption implies that the hierarchy m b << m t can be explained by one of the inequalities, i.e, h . Furthermore, the bottom quark gets its mass from the triplet ρ, while the exotic down type quarks J 2 and J 3 , get their masses from the triplet χ. Unlike the up type quark sector, the light down type quarks d and s get tree level masses from the triplet η. Besides that, we get a strong hierarchy between the masses of the exotic quarks J 2 , J 3 and T , i.e, m T >> m J 3 >> m J 2 .

IV. ON THE EFFECTS OF LOOP CORRECTIONS AND NON RENORMALIZABLE OPERATORS ON
THE UP-TYPE QUARK MASS TEXTURE.
As it can be seen from Eqs. (25) and (A8), the vanishing entries of the up type quark mass matrix do not receive one loop level corrections. In the up type quark sector, only some of the non vanishing entries of the corresponding mass matrix receive radiative corrections at one loop level due to virtual neutral scalars and heavy exotic T quark. Notice that charged scalars do not contribute radiatively to the entries of the up type quark mass matrix since J Using the multiplication rules of the S 3 group, it follows that the aforementioned non renormalizable Yukawa part of the model Lagrangian can be rewritten as follows: After the scalars triplets get vacuum expectation values, the previous non renormalizable interaction yields the following contributions to the entries of the up type quark mass matrix: Therefore, the up type quark mass matrix takes the following form: where: We introduced the function [34]: Notice that the dimensionless couplings in Eq. (46) satisfy the following hierarchy: Then, the mass matrix for the up-type quarks satisfies the following relation: From Eqs. (52) and (53), it follows that the dimensionless couplings in Eq. (53) satisfy the following hierarchy: For the sake of simplicity we assume that the dimensionless couplings ε sn , with s = 1, 2 and n = 2, 3 are approximatelly equal. This assumption implies ζ l ≃ ζ, ω i ≃ ω with l = 1, 2 and i = 1, 2, 3, 4. In order to reduce the parameter space in our model, we choose ω j = ω (j = 1, 2, 3), ζ 1 = (1 + ϑ) ζ, ζ 2 = (1 − ϑ) ζ with ϑ << 1. Notice that the parameter ϑ controls the breaking of universality in the dimensionless couplings of the six dimensional operators. Then, after the aforementioned choice is made, the expression given by Eq. (53) becomes: where: The matrix S U can be diagonalized by a rotation matrix X U , according to: where: And the masses of the SM up type quarks are given by: Notice that the six dimensional operators are crucial to give masses the up and charm quarks. However if universality in the dimensionless couplings of the six dimensional operators were exact, the up quark would be massless. Therefore, the smallness of the up quark mass can be explained by the suppression of universality violation in the dimensionless couplings of six dimensional operators. On the other hand, in the u, c, t basis, the matrix X U can be written as: V. QUARK MIXINGS.
With the rotation matrices R U and R D for the up and down type quarks, respectively, we find the CKM mixing matrix: where the CKM matrix elements are given by: while the Jarlskog invariant which measures the amount of CP violation takes the following form: Varying the parameters ̟, θ, ψ and φ we have fitted the magnitudes of the CKM matrix elements and the Jarlskog invariant J to the experimental values shown in Tables I, II. The experimental values of CKM magnitudes and Jarlskog invariant are taken from [12]. These tables also show the obtained values of CKM magnitudes and Jarlskog invariant. The values of the parameters ̟, θ, ψ and φ that successfully reproduce the CKM magnitudes and the value of the Jarlskog invariant are given by:  The obtained magnitudes of the CKM matrix elements and Jarlskog invariant are in excellent agreement with the experimental data. Notice that we have assumed that the Yukawa couplings for the up type quarks as well as the couplings of higher dimensional operators are real while the Yukawa couplings for down type quarks are complex with the exception of h ρ . It is worth mentioning that with only four effective parameters, i.e, the complex phase ̟ and the mixing angles θ, φ and ψ, we obtain a realistic pattern of SM quark mixings. Furthermore, the complex phase ̟ in the mass matrix for down type quarks is the only one responsible for CP violation in the quark sector.

VI. CONCLUSIONS
In this work we proposed a model based on the extended group SU (3) C ⊗ SU (3) L ⊗ U (1) X ⊗ S 3 . By assuming specific particle assignments into S 3 doublets and S 3 singlets for scalars and quarks, we obtained a predictive model, where quark masses and mixing can successfully be reproduced. Taking into account the multiplication rules of the S 3 discrete group, we obtained the complete mass terms of the scalar and quark sector. In particular, the model reproduces the mass structures of the down sector at tree level, where the light d-and s-quarks acquire masses through one scalar triplet (η) and the b-quark obtains mass from the other triplet (ρ), while the two heavy quarks J 1 and J 2 get large masses from the third scalar triplet (χ). Furthemore, by assuming that the light Standard Model down-type quarks possess smaller Yukawa couplings than the couplings of the heavy quarks, we may explain the hierarchy m d << m s . The hierarchy m b >> m s can be explained by assuming that the Yukawa coupling of the bottom quark is much larger than the Yukawa couplings of the exotic down type quarks, considering the vaccuum expectation values of the triplets (ρ) and (η) of the same order. Regarding the up sector, the model predicts at tree level two massless quarks (the u-and c-quarks), one massive quark at the electroweak scale (the t-quark), and a very heavy T-quark. Due to the symmetries of the model, the massless quarks do not get masses from radiative corrections. However, these quarks get masses from non renormalizable operators. In particular, if universality of the non renormalizable couplings is assumed, the u-quark remains massless. Only if the above universality is violated, the lightest quark gets mass. Thus, the smallness of the u-quark can be understood as a consequence of a small deviation in the universality of non renormalizable effective couplings. We have reproduced with only three mixing angles and one complex phase, the magnitudes of the CKM matrix elements and the Jarlskog invariant, which turn out to be in excellent agreement with the experimental data. The complex phase responsible for CP violation in the quark sector has been assumed to come from the down type quark mass matrix.

Acknowledgments
AECH thanks Universidad Nacional de Colombia for hospitality where this work was finished.
Appendix A: Scalar Potential and mass spectrum for the neutral and charged scalar fields.
The interactions among the scalar fields are contained in the following most general potential invariant under SU (3) C ⊗ SU (3) L ⊗ U (1) X ⊗ S 3 that we can construct with three scalar triplets: From the previous expressions and from the scalar potential minimization conditions, the following relations are obtained: From which it follows that: Then, the following relation is obtained: Furthermore, from the scalar potential it follows that the quartic interactions relevant for the computation of radiative corrections to the up-and down-type quark mass matrices are: where ξ ρ = v ρ + ξ ρ , ξ η = v η + ξ η and ξ χ = v χ + ξ χ . By expanding the scalar potential up to quadratic terms of the neutral scalar fields and using the relations given by Eqs. (A5)-(A6), we obtain the following mass terms for the neutral scalar fields: Therefore, the squared mass matrix for the neutral scalar fields in the basis ξ ρ , ξ η , ξ χ , η 0 3 , χ 0 1 , ζ ρ , ζ η , ζ χ is given by: where the squared mass matrices M 2 N a , M 2 N b and M 2 N c are given by: Here we assume that the mass parameter of the cubic term in the scalar potential satisfies f ≃ v χ . Since v χ ≫ v η , v ρ , the squared mass matrix M 2 N a can be block-diagonalized through the rotation matrix W , according to: where the relation λ 2 = −λ 4 given by Eq. (A7) has been used. From the condition of the vanishing of the off-diagonal submatrices in the previous expression, we obtain at leading order in B the following relations: where a and b are given by: (A19) By using the method of recursive expansion taking into account the hierarchy m 2 H 0 3 << b n1 << a nm , (m, n = 1, 2) we find that the submatrix B is approximatelly given by: where we have used Eq. (A7) and the fact that f ≃ v χ ≫ v η , v ρ .
so that the physical neutral scalar mass eigenstates contained in the matrix M 2 N b are given by: where Besides that, the following relation is fulfilled: so that the physical neutral scalar mass eigenstates contained in the matrix M 2 N c are given by: