Fermion Hierarchies in $SU(5)$ Grand Unification from $\Gamma_6^\prime$ Modular Flavor Symmetry

We construct a model in which the hierarchies of the quark and lepton masses and mixing are explained by the $\Gamma_6^\prime$ modular flavor symmetry. The hierarchies are realized by the Froggatt-Nielsen-like mechanism due to the residual $Z^T_6$ symmetry, approximately unbroken at $\tau \sim i\infty.$ We argue that the $\Gamma_6^{(\prime)}$ symmetry is the minimal possibility to realize the up-type quark mass hierarchies, since the Yukawa matrix is symmetric. We find a combination of the representations and modular weights and then show numerical values of $\mathcal{O}(1)$ coefficients for the realistic fermion hierarchies.


Introduction
The Grand Unified Theory (GUT) is an attractive framework to understand the gauge structure of the Standard Model (SM) [1].In the SU (5) GUT, the quarks and leptons in a generation are unified into 5 and 10 representations.As a result, for instance, the exactly opposite charges of the electron and the proton are manifestly explained.The SU (5) GUT predicts the unification of the three gauge coupling constants which is consistent with the Minimal Supersymmetric SM (MSSM) [2][3][4][5][6][7][8][9].Due to the unification of the quarks and leptons, the Yukawa couplings are also constrained to be consistent with the SU (5) gauge symmetry, as we shall study in this paper.
There are the hierarchies in the masses of the SM fermions and in the Cabibbo-Kobayashi-Maskawa (CKM) matrix.The residual Z N symmetry of the modular symmetry induces hierarchies of Yukawa couplings if the modulus τ has a value near a fixed point.For instance, if τ ∼ i∞, the Z N symmetry remains in the modular Γ N symmetry, and then the Yukawa couplings are suppressed by powers of q N := e −2πImτ /N 1 depending on the charge under the Z N symmetry.This is a realization of the Froggatt-Nielson (FN) mechanism, where the powers of the flavon field are replaced by those of q N .Recently, this idea is applied for the quark sector based on the modular A 4 [72,73], A 4 × A 4 × A 4 [74], S 4 [75] and Γ 6 [76] symmetries, and the lepton sector is also studied in Refs.[77,78].In Ref. [79], the present authors constructed a model for both quark and lepton sectors based on Γ 4 S 4 symmetry, where Γ N is the double covering of Γ N symmetry.
In this work, we construct a model to explain the hierarchies of the quarks and leptons in the SU (5) GUT.It will be turned out that the residual symmetry should be Z N with N ≥ 5, to realize the hierarchy of the up-type quark masses because the up Yukawa matrix is a symmetric matrix.Since the Γ ( ) 5 symmetry does not have proper representation to realize the hierarchy [14,68] as explained later, we shall consider the Γ 6 symmetry as the minimal possibility.We consider Γ 6 , double covering of Γ 6 , to construct a model with non-singlet representations.We also study the neutrino sector assuming the type-I seesaw mechanism by introducing three SU (5) singlets.
This paper is organized as follows.In Sec. 2, we briefly review the flavor structures of the SU (5) GUT and the modular flavor symmetry, and then discuss how to explain the hierarchies of the masses and mixing.The explicit model is studied and our results of numerical calculations for the charged fermions are shown in Sec. 3, and then the neutrino sector is discussed in Sec. 4. Section 5 concludes.The details of the modular Γ 6 symmetry are shown in App. A.

SU (5) GUT
The superpotential at the dimension-4 is given by where A, B, • • • = 1, 2, 3, 4, 5 are the SU (5) indices, and i, j = 1, 2, 3 are the flavor indices.The MSSM fields are in the SU (5) multiplets as where a, b, c = 1, 2, 3 and α, β = 1, 2 are the SU (3) C and SU (2) L indices, respectively.In these expressions, 's with the indices are the complete anti-symmetric tensors.The Higgs doublets are included in the (anti-)fundamental representations H (H) as We assume that the triplets are so heavy that it will not induce too fast proton decay by a certain mechanism [80][81][82][83] 1 .Since the Yukawa matrices of the down quarks and charged leptons are given by the common Yukawa matrix h ij , the dimension-4 Yukawa couplings in Eq. (2.1) can not explain the realistic fermion masses.
In this work, we assume that the SU (5) gauge symmetry is broken by an adjoint field Σ B A whose VEV is given by so that the GUT symmetry is broken down to the SM gauge symmetry.We shall consider the dimension-5 interaction involving the adjoint field, which splits the Yukawa couplings of the down quarks and the charged leptons.Here Λ is a cutoff scale.The dimension-5 operators inserting the adjoint field to other places will not change the flavor structure, and thus we do not write them explicitly.The Yukawa couplings for the MSSM fields are given by where the Yukawa matrices are given by where R Σ := v Σ /Λ.Thus the O (R Σ ) splitting of the down-type Yukawa couplings is induced via the VEV of the adjoint scalar field v Σ .

Modular flavor symmetry
We consider the so-called principal congruence subgroups Γ(N ), N ∈ N, defined as where Γ := SL(2, Z) = Γ(1) is the special linear group of 2 × 2 matrices of integers.The modulus τ is transformed by the group Γ as (2.9) The finite modular group Γ N is defined as a quotient group Γ N := Γ/Γ(N ), which can be generated by the generators, One can consider the quotient group Γ N := Γ/Γ(N ), where Γ := Γ/Z R 2 with Z R 2 being the Z 2 symmetry generated by R. Γ N is the double covering of the group Γ N .For N < 6, the generators of Γ N satisfy and those for Γ N are given by taking R = 1.At N = 6, the generators should also satisfy in addition to those in Eq. (2.11).Under Γ N , a modular form, a holomorphic function of τ , Y (k) r with representation r and modular weight k transforms as where ρ(r) is the representation matrix of r.We assume that the chiral superfield Φ with representation r Φ and weight −k Φ transform as If the modulus is stabilized near a fixed point τ ∼ i, w := e 2πi/3 or i∞, there remains a residual symmetry Z S 4 , Z ST 3 or Z T N , respectively.Here, the superscript represents a generator associated with the residual symmetry.
The modular invariant Kähler potential for the kinetic term is given by and hence the canonically normalized chiral superfield is given by The normalization factors from √ 2 Imτ will be important for τ ∼ i∞, as we shall consider in this work.

Assignments of representations
The hierarchical Yukawa matrices can be realized by the FN mechanism with a Z N symmetry [85,86] if the modulus field is stabilized near a fixed point.We shall consider the FN-like mechanism by Z T N symmetry, and define := √ 3e 2πiτ /6 in Eq. (A.24) throughout this paper.See Appendix A for more details.Since the up-type Yukawa matrix is symmetric in the SU (5) GUT, the texture of Y u is, in general, given by where 1 and n, m ∈ N. The singular values are read as ( 2n , 2m , 1), and hence the minimal possibility to obtain the hierarchal up quark masses is (m, n) = (1, 2), so that the singular values are given by ( 4 , 2 , 1).Thus N > 4 is required to realize the hierarchy by a Z N symmetry.
The Γ ( ) 6 symmetry is the minimal possibility for the up quark mass hierarchy.N > 4 can be realized only from the residual T symmetry in the modular Γ ( ) N symmetry with N > 4, where Z T N symmetry remains unbroken at τ ∼ i∞.In the case of N = 5, however, there are no such representations whose T -charges are (0, −1, −2) ≡ (0, 4, 3) mod N = 5 [14,68].Therefore, N = 6 is the minimal possibility to realize the hierarchal up Yukawa couplings.
As shown in detail in Appendix A, the irreducible representations of Γ 6 are [87] where s = 0, 1 and b = 0, 1, 2. Since Γ 6 is isomorphic to Γ 2 × Γ 3 S 3 × A 4 , upper (lower) indices can be considered as S 3 (A 4 ) indices.For the double covering Γ 6 , there are additional representations, The charges under the residual Z T 6 symmetry of the representations less than three dimensions are given by (2.20) modulo 6.
At N = 6, the texture can be realized for, The Z T 6 -charges are (2, 1, 3).Since the doublet representation 2 1 2 exists only for Γ 6 , we should consider the double covering of Γ 6 .We can replace 2 1  2 to 1 1 2 ⊕ 1 1 0 which exist in Γ 6 , but we do not consider this case because the model is less predictive and may need larger modular weights to have certain modular forms [76].
Since the doublet lepton i is in 5 i , the hierarchal structures of the neutrino masses and the PMNS matrix are determined by the representation of 5 i .As ∼ (m u /m t ) 1/4 ∼ O (0.05) for the charged fermion hierarchies, i should have a common Z T 6 charge, so that there is no hierarchy in the neutrino sector.Thus we assign with s = 0, 1 and b = 0, 1, 2. The values of (s, b) will be determined to have the modular forms for a given modular weights.In this case, the singular values of the down quarks and charged leptons are given by q ( 2 , , 1), where q = 0, 1, 2, 3 depends on (s, b).The textures of the CKM and PMNS matrices are given by and there is no hierarchy in the neutrino masses originated from .We discuss the neutrino sector explicitly in Sec. 4. We also note that the textures will also be modified by powers of √ 2Imτ as in Eq. (2.16) depending on modular weights, as will be discussed in the next section in which we will take √ 2Imτ ∼ 2.5.

Yukawa couplings
We denote the modular weights of 10 1,2 and 5 1,2,3 as The modular invariant superpotential of the Yukawa couplings are given by where the contractions of the SU (5) indices are implicit.It is assumed that Σ, H and H are trivial singlets with modular weight k = 0.The symbol (• • • ) 1 indicates that the trivial singlet combinations inside the parenthesis.Here α i , β 1i and β 2i are O (1) coefficients.The Yukawa matrices for the MSSM quarks and leptons are read as ) where Here, [Y

Assigning modular weights
The minimal choice to obtain the realistic Yukawa matrices is and we take (s, b) = (1, 0) for 5 i = 1 s b , so that the Yukawa matrices are given by ) There are two modular forms Y (6) 3 0 ,i , i = 1, 2, and there is no 1 1 2 at k = 4.The Yukawa matrices are rescaled by the canonical normalization of the kinetic terms as in Eq. (2.15).Since the top Yukawa coupling is O (1), we fix the overall factor of the Yukawa couplings, so that the factor ( √ 2Imτ ) 6 is compensated.Note that the normalization of the modular form is not fixed from the bottom-up approach2 .We absorb this effect into the coefficients, rather than the modular forms, by defining αi := (2Imτ ) With the assignments of the modular weights, the texture of the Yukawa matrices are given by where η := 1/(2Imτ ) is the factor from the canonical normalization.The hierarchical structures of the quark masses and the CKM matrix is given by , η 1/2 , 2 ∼ (0.2, 0.03, 0.005) , where = 0.067 and η = 0.16 are used for the numerical estimations.Here, s Q ij is the mixing angles in the standard parametrization of the CKM matrix.The mass hierarchies of the charged leptons are the same as those for the down-type quarks.These values well fit to the data shown in Table 1 at the benchmark points discussed in the next section, except for y u and y e which are about 10 larger than the experimental values.These differences will be explained by the numerical factors in the modular forms and the O (1) coefficients.It is interesting that the CKM angles in our models fit to the data after taking account the powers of η from the canonical normalization, so that s

Benchmark points
We find the numerical values of the parameters by numerical optimization.We restrict the parameter space to be tan β ∈ (5, 60), η b ∈ (−0.Here, tan β includes the threshold correction to the tau lepton, and η b is that for the bottom quark as defined in Ref. [88].In this work, we treat η b as a parameter which will be determined from the soft SUSY breaking parameters, see for threshold corrections Refs.[89,90].The threshold corrections to the light flavors, η q and η , are assumed to be zero for simplicity.We take R Σ = 0.1 and restrict |γ ij | < 1, so that the contribution from the dimension-5 operator is sub-dominant.We found the following two benchmark points.At the first benchmark point (BP1), the inputs are given by tan β = 11.4643,η b = 0.187818 τ = 0.0592 + 3.1033i, and at the second point (BP2), tan β = 11.4303,η b = 0.598145, τ = 0.0661 + 3.0791i, These points realize the quark and charged lepton masses, and the CKM angles as shown in Table 1.At both points, all of the observables are within 2σ range, and the largest discrepancy is 0.61σ (1.93σ) at the BP1 (BP2) for the charm (down) quark mass.The values of the parameters are similar at both points, but η b is relatively small (large) at the BP1 (BP2).The absolute values of the coefficients are in the range of [0.19, 2, 2] and [0.20, 1.7] at BP1 and BP2, respectively.Thus, the O (1) coefficients can explain the hierarchies with the good accuracy.

Neutrino sector
We shall consider the neutrino sector in this section.We assume that the neutrinos are Majorana, so the neutrino masses are given by the Weinberg operator W ( H u ) 2 at low-energies.
With our choice of the representations and the weights of ∈ 5, the masses of the neutrinos and the mixing angles in the PMNS matrix are predicted to be and thus the neutrino observables will have the texture independently to the UV completion of the Weinberg operator.Hence, the angle s 2 13 is naturally explained, while the ratio of the mass squared differences R 21  32 and s 2 23 are predicted to be about an order of magnitude smaller than the observed values.We will discuss how these discrepancies are explained in an explicit model based on the type-I seesaw mechanism.
For illustration, we assume the type-I seesaw mechanism to realize the tiny neutrino masses by introducing the three generations of singlets N i .The superpotential is given by We choose the representations and modular weights of the right-handed neutrinos as The modular invariant superpotential is given by ) The neutrino mass matrix is given by where Ŷn and MN are after the canonical normalization.For the neutrino observables, the elements suppressed by ∼ O (0.05) are irrelevant, and hence the matrices are approximately given by MN Since the neutrino mass matrix M ν is rank-2, the lightest neutrino mass appears only at O (3 ) 3 .
The ratio of the heavier two neutrinos are given by at the leading order in η.Because of the hierarchy in the neutrino masses, the model predicts the normal ordering [91].The ratio is enhanced by (B 2 11 + B 2 22 )/(B 23 /4) 2 ∼ O (10) coming from the Dirac Yukawa matrix Y n for all A, B = O(1), and thus the ratio of the mass squared difference R 12  23 is enhanced by O (100), consistent with the observed value ∼ 0.03.While the mild discrepancy of s 2 23 will be explained simply by O (5) ratios of the coefficients.We find the values to explain the neutrino observables at the benchmark points.At the and at BP2, M 0 = 1.7775 × 10 15 GeV, At these points, the CP phase in the PMNS matrix is originated from that in the charged lepton Yukawa matrix, which is common to the CKM matrix.We see that the values of the coefficients are close to 1 and the ratio of the coefficients are at most 3.9 (5.7) at the BP1 (BP2).

Summary
In this work, we build a model to realize the fermion hierarchy in SU (5) GUT utilizing the modular Γ 6 symmetry.The residual symmetry associated with the generator T , namely Z T 6 symmetry, controls the powers of the small parameter e −2πImτ /6 .We argue that the Γ 6 is the minimal possibility to realize the hierarchical masses of the up-type quarks, because the Yukawa matrix for the up-type quarks is symmetric in the SU (5) GUT.We assign the representations of 10 as 1 0 1 ⊕ 2 0 2 and 5 as We have to consider the double covering Γ 6 to have the representations 2 s b .The CKM matrix is hierarchical because the T charges of Q ∈ 10 are different, while the PMNS matrix is not hierarchical because the those of ∈ 5 are the same.The representations of r and (s, b) are chosen such that the realistic Yukawa matrices are realized for the modular weights smaller than 6.We assigned the modular weights, so that certain modular forms exists and the Yukawa matrices are rank 3 up to O ( 6 ).In the model, we show that the O (1) coefficients without hierarchies explain the observed quark and lepton masses and the CKM elements.
We assume the type-I seesaw mechanism to explain the tiny neutrino masses.The singlet right-handed neutrinos are assigned to be 1 1 0 ⊕ 2 0 and their modular weights are chosen to be 0⊕2.With these assignments, one of the three neutrinos is suppressed by O ( 3 ) compared with the other two heavier neutrinos.The mass ratio of the heavier two neutrinos is not suppressed by , but is suppressed by (Imτ /2) 2 ∼ O (10) due to the canonical normalization, see Eq. (4.9).We showed that the O (1) coefficients can explain the neutrino observables in the SU (5) GUT model.(A.11) The products of the doublets are given by .
The products of the doublet and triplet are , and the product of the triplets is given by The product rules involving the higher dimensional representations are shown in Ref. [87].
Table 3: The representations of modular forms at k ≤ 6. A.

Modular forms
The modular forms at k = 1 are given by [87] where the functions are defined as The modular forms with higher weights can be constructed by taking the direct products.

r
] i denotes the i-th component of the modular form Y (k) r .As shown in Eq. (2.7), Y d = Y T e up to the splitting at O (R Σ ) because d ∈ 5 and Q ∈ 10 while e ∈ 10 and ∈ 5. Note that the number of coefficients may change depending on the number of modular forms for a given modular weight.

at k = 2 .
A i and B ij are the O (1) coefficients, and M 0 is the overall scale of the Majorana mass term.(N 2 N 2 ) 2 0 takes the combination of the representation 2 0 .Here, we take Y The Majorana mass matrix M N and the neutrino Yukawa matrix Y n are given by

Table 1 :
[88]values of the Yukawa couplings of the SM fermions and the mixing angles in the CKM matrix at the BP1 and BP2 in the left and right panels, respectively.The second column is the value at the benchmark points, and the third (fourth) column is the experimental value (and its error).The experimental values are given at the GUT scale calculated in Ref.[88].

Table 2 :
The values of the neutrino observables at the benchmark points.