Fermion masses and mixings in the 3-3-1 model with right-handed neutrinos based on the $S_3$ flavor symmetry

Abstract

We propose a 3-3-1 model where the ${SU}(3)_{C}\otimes {SU}(3)_{L}\otimes U(1)_{X}$ symmetry is extended by $S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ and the scalar spectrum is enlarged by extra ${SU}(3)_{L}$ singlet scalar fields. The model successfully describes the observed SM fermion mass and mixing pattern. In this framework, the light active neutrino masses arise via an inverse seesaw mechanism and the observed charged fermion mass and quark mixing hierarchy is a consequence of the $Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ symmetry breaking at very high energy. The obtained physical observables for both quark and lepton sectors are compatible with their experimental values. The model predicts the effective Majorana neutrino mass parameter of neutrinoless double beta decay to be $m_{\beta \beta }=$ 4 and 48 meV for the normal and the inverted neutrino spectra, respectively. Furthermore, we found a leptonic Dirac CP-violating phase close to $\frac{\pi }{2}$ and a Jarlskog invariant close to about $3\times 10^{-2}$ for both normal and inverted neutrino mass hierarchy.

1 Introduction

After the discovery of the 126 GeV Higgs boson by ATLAS and CMS collaborations at CERN Large Hadron Collider (LHC) [1, 2], the vacancy of the Higgs boson needed for the completion of the Standard Model (SM) at the Fermi scale has been filled and the weak gauge bosons mass generation mechanism has also been confirmed. Despite LHC experiments indicating that the decay modes of the new scalar state are SM like, there is still room for new extra scalar states, whose search are an essential task of the LHC experiments. Furthermore, despite the great consistency of the SM predictions with the experimental data, there are several aspects that the SM does not explain, some of them are the observed hierarchy among charged fermion masses and quark mixing angles, the tiny neutrino masses and the smallness of the quark mixing angles, which contrast with the sizable leptonic mixing ones. The global fits of the available data from the Daya Bay [3], T2K [4], MINOS [5], Double CHOOZ [6] and RENO [7] neutrino oscillation experiments, constrain the neutrino mass squared splittings and mixing parameters [8]. It is a well-established experimental fact that the observed hierarchy of charged fermion masses goes over a range of five orders of magnitude in the quark sector and that there are six orders of magnitude between the neutrino mass scale and the electron mass. Accommodating the charged fermion masses in the SM requires an unnatural tunning among its different Yukawa couplings. Furthermore, experiments with solar, atmospheric and reactor neutrinos [3,4,5,6,7, 9] have brought about evidence of neutrino oscillations caused by nonzero masses. All these unexplained issues strongly indicate that new physics has to be invoked to address the fermion puzzle of the SM (Table 1).

Table 1 Range for experimental values of neutrino mass squared splittings and leptonic mixing parameters, taken from Ref. [8], for the case of normal hierarchy

Table 2 Range for experimental values of neutrino mass squared splittings and leptonic mixing parameters, taken from Ref. [8], for the case of inverted hierarchy

The aforementioned flavor puzzle, not understood in the context of the SM, motivates extensions of the Standard Model that explain the fermion mass and mixing patterns. From the phenomenological point of view, it is possible to describe some features of the mass hierarchy by assuming Yukawa matrices with texture zeros [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38]. A very promising approach is the use of discrete flavor groups, which have been considered in several models to explain the fermion masses and mixing (see Refs. [39,40,41,42] for recent reviews on flavor symmetries). Models with spontaneously broken flavor symmetries may also produce hierarchical mass structures. Recently, discrete groups such as $A_{4}$ [43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62], $S_{3}$ [63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83], $S_{4}$ [84,85,86,87,88,89,90,91,92,93,94], $D_4$ [95,96,97,98,99,100,101,102,103,104], $Q_6$ [105,106,107,108], $T_7$ [109,110,111,112,113,114,115,116,117,118], $T_{13}$ [119,120,121,122], $T^{\prime }$ [123,124,125,126,127,128,129], $\Delta (27)$ [130,131,132,133,134,135,136,137,138,139,140,141,142,143,144] and $A_5$ [145,146,147,148,149,150,151,152,153,154,155] have been considered to explain the observed pattern of fermion masses and mixings. In particular the $S_{3}$ flavor symmetry is a very good candidate for explaining the prevailing pattern of fermion masses and mixing. The $S_{3}$ discrete symmetry is the smallest non-Abelian discrete symmetry group having three irreducible representations (irreps), explicitly two singlets and one doublet irreps. The $S_3$ discrete symmetry was used as a flavor symmetry for the first time in Ref. [156]. The different models based on discrete flavor symmetries have as a common issue the breaking of the flavor symmetry so that the observed data be naturally produced. The breaking of the flavor symmetry takes place when the scalar fields acquire vacuum expectation values (Table 2).

Besides that, another of the greatest mysteries in particle physics is the existence of three fermion families at low energies. The origin of the family structure of the fermions can be addressed in family dependent models where a symmetry distinguish fermions of different families. One explanation to this issue can be provided by the models based on the gauge symmetry ${SU}(3)_{c}\otimes {SU}(3)_{L}\otimes U(1)_{X}$, also called 3-3-1 models, which introduce a family non-universal $U(1)_{X}$ symmetry [26, 61, 62, 78, 79, 115, 117, 157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202]. These models have a number of phenomenological advantages. First of all, the three family structure in the fermion sector can be understood in the 3-3-1 models from the cancellation of chiral anomalies and asymptotic freedom in QCD. Second, the fact that the third family is treated under a different representation can explain the large mass difference between the heaviest quark family and the two lighter ones. Third, these models contain a natural Peccei–Quinn symmetry, necessary to solve the strong-CP problem [203,204,205,206]. Finally, 3-3-1 models including heavy sterile neutrinos have cold dark matter candidates as weakly interacting massive particles (WIMPs) [179, 207,208,209]. Besides that, the 3-3-1 models can explain the 2 TeV diboson excess found by ATLAS [210]. When the electric charge in the 3-3-1 models is defined in the linear combination of the ${SU}(3)_{L}$ generators $T_{3}$ and $T_{8}$, it is a free parameter, independent of the anomalies ($\beta$). The choice of this parameter defines the charge of the exotic particles. Choosing $\beta =- \frac{1}{\sqrt{3}}$, the third component of the weak lepton triplet is a neutral field $\nu _{R}^{C}$, which allows one to build the Dirac matrix with the usual field $\nu _{L}$ of the weak doublet. If one introduces a sterile neutrino $N_{R}$ in the model, then it is possible to generate light neutrino masses via inverse seesaw mechanism. The 3-3-1 models with $\beta =- \frac{1}{\sqrt{3}}$ have the advantage of providing an alternative framework to generate neutrino masses, where the neutrino spectrum includes the light active sub-eV scale neutrinos as well as sterile neutrinos which could be dark matter candidates if they are light enough or candidates for detection at the LHC, if their masses are at the TeV scale. This interesting feature makes the 3-3-1 models very interesting since if the TeV scale sterile neutrinos are found at the LHC, these models can be very strong candidates for unraveling the mechanism responsible for electroweak symmetry breaking.

In the 3-3-1 models, one heavy triplet field with a Vacuum Expectation Value (VEV) at high energy scale $\nu _{\chi }$, breaks the symmetry ${SU}(3)_{L}\otimes U(1)_{X}$ into the SM electroweak group ${SU}(2)_{L}\otimes U(1)_{Y}$, while the another two lighter triplets with VEVs at the electroweak scale $\upsilon _{\rho }$ and $\upsilon _{\eta }$, trigger the electroweak symmetry breaking [26]. Besides that, the 3-3-1 model could possibly explain the excess of events in the $h\rightarrow \gamma \gamma$ decay, recently observed at the LHC, since the heavy exotic quarks, the charged Higges, and the heavy charged gauge bosons contribute to this process. On the other hand, the 3-3-1 model reproduces a specialized Two Higgs Doublet Model type III (2HDM-III) in the low energy limit, where both electroweak triplets $\rho$ and $\eta$ are decomposed into two hypercharge-one ${SU}(2)_{L}$ doublets plus charged and neutral singlets. Thus, like the 2HDM-III, the 3-3-1 model can predict huge flavor changing neutral currents (FCNC) and CP-violating effects, which are severely suppressed by experimental data at electroweak scales. In the 2HDM-III, for each quark type, up or down, there are two Yukawa couplings. One of the Yukawa couplings is for generating the quark masses, and the other one produces the flavor changing couplings at tree level. One way to remove both the huge FCNC and CP-violating effects is by imposing discrete symmetries, obtaining two types of 3-3-1 models (type I and II models), which exhibit the same Yukawa interactions as the 2HDM type I and II at low energy where each fermion is coupled at most to one Higgs doublet. In the 3-3-1 model type I, one Higgs electroweak triplet (for example, $\rho$) provide masses to the phenomenological up- and down-type quarks, simultaneously. In the type II, one Higgs triplet ($\eta$) gives masses to the up-type quarks and the other triplet ($\rho$) to the down-type quarks [26].

It is noteworthy the $S_3$ flavor symmetry was implemented for the first time in the 3-3-1 model of Ref. [69]. That model introduces a new ${U}(1)_{\mathcal {L}}$ lepton global symmetry, responsible for lepton number and lepton parity. That lepton parity symmetry suppresses the mixing between ordinary quarks and exotic quarks. Furthermore, the ${U}(1)_{\mathcal {L}}$ new lepton global symmetry enforces to have different scalar fields in the Yukawa interactions for charged lepton, neutrino and quark sectors. The scalar sector of that model includes six ${SU}(3)_L$ scalar triplets and four ${SU}(3)_L$ scalar antisextets. The $\mathrm {SU}(3)_{C}\otimes \mathrm {SU}(3)_{L}\otimes {U}(1)_{X}\otimes {U}(1)_{\mathcal {L}}\otimes S_3$ assignments of the fermion sector of the aforementioned model, require that these 6 ${SU}(3)_L$ scalar triplets be distributed as follows: 3 for the quark sector, 2 for the charged lepton sector and 1 for the neutrino sector. Furthermore the 4 ${SU}(3)_L$ scalar antisextets are needed to implement a type II seesaw mechanism. In that model, light active neutrino masses are generated from type-I and type-II seesaw mechanisms, mediated by three heavy right-handed Majorana neutrinos and four ${SU}(3)_{L}$ scalar antisextets, respectively. Since the Yukawa terms of that model are renormalizable, to explain the SM charged fermion mass pattern, one needs to impose a strong hierarchy among the charged fermion Yukawa couplings of the model. Furthermore, the work described in Ref. [69] is mainly focused on the lepton sector, while in the quark sector, the obtained quark mass matrices are diagonal and the quark mixing matrix is trivial.

Recently two of us proposed a ${SU}(3)_{C}\times {SU}(3)_{L}\times U(1)_{X}\otimes S_{3}\otimes Z_{2}\otimes Z_{4}\otimes Z_{12}$ model [78], with a scalar sector composed of three ${SU}(3)_{L}$ scalar triplets and seven ${SU}(3)_{L}$ scalar singlets, that successfully accounts for quark masses and mixings. In that model, all observables in the quark sector are in excellent agreement with the experimental data, excepting $\bigl |V_{td}\bigr |$, which turns out to be larger by a factor ${\sim }1.3$ than its corresponding experimental value, and naively deviated 8 sigma away from it. That model has the following drawbacks: $\bigl |V_{td} \bigr |$ is deviated 8 sigma away from its experimental value, a $S_{3}$ soft breaking term has to be introduced by hand in the low energy scalar potential in order to fullfill its minimization equations, the top quark mass arises from a five dimensional Yukawa term and lepton masses and mixings are not addressed.

It is interesting to find an alternative and better explanation for the SM fermion mass and mixing hierarchy than the ones considered in Refs. [69, 78]. To this end we propose a multiscalar singlet extension of the ${SU}(3)_{C}\times {SU}(3)_{L}\times U(1)_{X}$ model with right-handed neutrinos, where $\beta =- \frac{1}{\sqrt{3}}$ and an extra $S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ discrete group, that extends the symmetry of the model and 15 very heavy ${SU}(3)_{L}$ singlet scalar fields are added with the aim to generate viable textures for the fermion sector, which successfully describe the observed SM fermion mass and mixing pattern. Let us note that whereas the scalar sector of our model only has three ${SU}(3)_{L}$ scalar triplets and 15 ${SU}(3)_{L}$ scalar singlets, the scalar sector of the $S_{3}$ flavor 3-3-1 model of Ref. [69] has six ${SU}(3)_{L}$ scalar triplets and four ${SU}(3)_{L}$ scalar antisextets. Whereas in the model of Ref. [69], the quark mixing matrix is equal to the identity, in our model the quark mixing matrix is in excellent agreement with the low energy quark flavor data. In our model, the obtained physical observables in the quark and lepton sector are consistent with the experimental data. Our model at low energies reduces to the 3-3-1 model with right-handed neutrinos, where $\beta =-\frac{1}{\sqrt{3}}$. Furthermore, our current model does not include the ${U}(1)_{\mathcal {L}}$ new lepton global symmetry presented in the $S_{3}$ flavor 3-3-1 model of Ref. [69]. Unlike the $S_{3}$ flavor 3-3-1 model of Ref. [69], in our current 3-3-1 model, the charged fermion mass and quark mixing pattern can successfully be accounted for, by having all Yukawa couplings of order unity and arises from the breaking of the $Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ discrete group at very high energy, triggered by ${SU}(3)_{L}$ scalar singlets acquiring vacuum expectation values much larger than the TeV scale. Despite our current model has more ${SU}(3)_{L}$ scalar singlets than the model that two of us have recently proposed in Ref. [78], our current model addresses both the quark and lepton sectors and does not have the aforementioned drawbacks of the model of Ref. [78]. Because of the aforementioned reasons, our current model represents an important improvement over the previously studied scenarios [69, 78]. The particular role of each additional scalar field and the corresponding particle assignments under the symmetry group of the model under consideration are explained in detail in Sect. 2. The model we are building with the aforementioned discrete symmetries, preserves the content of particles of the 3-3-1 model with $\beta =-\frac{1}{\sqrt{3}}$, but we add 15 additional very heavy ${SU}(3)_{L}$ singlet scalar fields, with quantum numbers that allow to build Yukawa terms invariant under the local and discrete groups. This generates the right textures that successfully account for SM fermion masses and mixings. We assume that the Majorana neutrinos have very small masses, implying that the small active neutrino masses are generated via an inverse seesaw mechanism. This mechanism for the generation of the light active neutrino masses differs from the one implemented in the $S_{3}$ flavor 3-3-1 model of Ref. [69], where the light active neutrinos get their masses from type I and type II seesaw mechanisms.

The paper is outlined as follows. In Sect. 2 we explain some theoretical aspects of the 3-3-1 model with $\beta =-\frac{1}{\sqrt{3}}$ and its particle content, as well as the particle assignments under doublet and singlet $S_{3}$ representations, in particular in the fermionic and scalar sector. The low energy scalar potential of our model is discussed in Sect. 2.2. In Sect. 3 we focus on the discussion of lepton masses and mixing and give our corresponding results. In Sect. 4, we present our results in terms of quark masses and mixing, which is followed by a numerical analysis. Conclusions are given Sect. 5. In the appendices we present several technical details: Appendix A gives a brief description of the $S_{3}$ group; Appendix B shows a discussion of the stability conditions of the low energy scalar potential.

2 The model

2.1 Particle content

The first 3-3-1 model with right-handed Majorana neutrinos in the ${SU}(3)_{L}$ lepton triplet was considered in [160]. However, that model cannot describe the observed pattern of SM fermion masses and mixings, due to the unexplained hierarchy among the large number of Yukawa couplings in the model. Below we consider a ${SU}(3)_{C}\otimes {SU}(3)_{L}\otimes U(1)_{X}\otimes S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ multiscalar singlet extension of the 3-3-1 model with right-handed neutrinos, which successfully describes the SM fermion mass and mixing pattern. In our model the full symmetry $\mathcal {G}$ is spontaneously broken in three steps as follows:

$$\begin{aligned} \mathcal {G}= & {} SU(3)_{C}\otimes SU\left( 3\right) _{L}\otimes U\left( 1\right) _{X}\otimes S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16} \nonumber \\&{\xrightarrow {\Lambda _{int}}}SU(3)_{C}\otimes SU\left( 3\right) _{L}\otimes U\left( 1\right) _{X}\otimes Z_{3}\nonumber \\&{\xrightarrow {v_{\chi }}}SU(3)_{C}\otimes SU\left( 2\right) _{L}\otimes U\left( 1\right) _{Y}{\xrightarrow {v_{\eta },v_{\rho }}} \nonumber \\&SU(3)_{C}\otimes U( 1) _{Q}, \end{aligned}$$

(1)

where the hierarchy $v_{\eta },v_{\rho }\ll v_{\chi }\ll \Lambda _{int}$ among the symmetry breaking scales is fulfilled.

The electric charge in our 3-3-1 model is defined as [92]

$$\begin{aligned} Q=T_{3}-\frac{1}{\sqrt{3}}T_{8}+XI, \end{aligned}$$

(2)

where $T_3$ and $T_8$ are the ${SU}(3)_L$ diagonal generators, I is the $3\times 3$ identity matrix and X the $U(1)_X$ charge.

Two families of quarks are grouped in a $3^{*}$ irreducible representations (irreps), as required from the ${SU}(3)_{L}$ anomaly cancellation. Furthermore, from the quark colors, it follows that the number of $3^{*}$ irreducible representations is six. The other family of quarks is grouped in a 3 irreducible representation. Moreover, there are six 3 irreps taking into account the three families of leptons. Consequently, the ${SU}(3)_{L}$ representations are vector like and do not contain anomalies. The quantum numbers for the fermion families are assigned in such a way that the combination of the $U(1)_{X}$ representations with other gauge sectors is anomaly free. Therefore, the anomaly cancellation requirement implies that quarks are unified in the following $({SU}(3)_{C},{SU}(3)_{L},U(1)_{X})$ left- and right-handed representations:

$$\begin{aligned}&Q_{L}^{1,2} = \begin{pmatrix} D^{1,2} \\ -U^{1,2} \\ J^{1,2} \\ \end{pmatrix} _{L}:(3,3^{*},0),Q_{L}^{3}= \begin{pmatrix} U^{3} \\ D^{3} \\ T \\ \end{pmatrix} _{L}:(3,3,1/3), \end{aligned}$$

(3)

$$\begin{aligned}&\begin{array}{c} D_{R}^{1,2,3}:(3,1,-1/3),\;\;\\ J_{R}^{1,2}:(3,1,-1/3),\;\; \end{array} \begin{array}{l} U_{R}^{1,2,3}:(3,1,2/3),\\ T_{R}:(3,1,2/3). \end{array} \end{aligned}$$

(4)

Here $U_{L}^{i}$ and $D_{L}^{i}$ ($i=1,2,3$) are the left-handed up- and down-type quarks in the flavor basis. The right-handed SM quarks $U_{R}^{i}$ and $D_{R}^{i}$ ($i=1,2,3$) and right-handed exotic quarks $T_{R}$ and $J_{R}^{1,2}$ are assigned into ${SU}(3)_{L}$ singlets representations, so that their $U(1)_{X}$ quantum numbers correspond to their electric charges.

Furthermore, cancellation of anomalies implies that leptons are grouped in the following $({SU}(3)_{C},{SU}(3)_{L},U(1)_{X})$ left- and right-handed representations:

$$\begin{aligned}&L_{L}^{1,2,3} = \begin{pmatrix} \nu ^{1,2,3} \\ e^{1,2,3} \\ (\nu ^{1,2,3})^{c} \\ \end{pmatrix} _{L}:(1,3,-1/3), \end{aligned}$$

(5)

$$\begin{aligned}&\begin{array}{c} e_{R}:(1,1,-1),\;\;\\ N_{R}^{1}:(1,1,0),\;\; \\ \end{array} \begin{array}{l} \mu _{R}:(1,1,-1),\;\; \\ N_{R}^{2}:(1,1,0),\;\; \\ \end{array}\quad \begin{array}{l} \tau _{R}:(1,1,-1),\\ N_{R}^{3}:(1,1,0). \end{array} \end{aligned}$$

(6)

where $\nu _{L}^{i}$ and $e_{L}^{i}$ ($e_{L},\mu _{L},\tau _{L}$) are the neutral and charged lepton families, respectively. Let us note that we assign the right-handed leptons as ${SU}(3)_{L}$ singlets, which implies that their $U(1)_{X}$ quantum numbers correspond to their electric charges. The exotic leptons of the model are three neutral Majorana leptons $(\nu ^{1,2,3})_{L}^{c}$ and three right-handed Majorana leptons $N_{R}^{1,2,3}$ (A recent discussion of double and inverse seesaw neutrino mass generation mechanisms in the context of 3-3-1 models can be found in Ref. [182]).

The scalar sector the 3-3-1 models includes: three 3’s irreps of ${SU}(3)_{L}$, where one triplet $\chi$ gets a TeV scale vaccuum expectation value (VEV) $v_{\chi }$, which breaks the ${SU}(3)_{L}\otimes U(1)_{X}$ symmetry down to ${SU}(2)_{L}\otimes U(1)_{Y}$, thus generating the masses of non-SM fermions and non-SM gauge bosons; and two light triplets $\eta$ and $\rho$ acquiring electroweak scale VEVs $v_{\eta }$ and $v_{\rho }$, respectively, and thus providing masses for the fermions and gauge bosons of the SM.

Regarding the scalar sector of the minimal 331 model, we assign the scalar fields in the following $[{SU}(3)_{L},U(1)_{X}]$ representations:

$$\begin{aligned} \chi&= \begin{pmatrix} \chi _{1}^{0} \\ \chi _{2}^{-} \\ \frac{1}{\sqrt{2}}(\upsilon _{\chi }+\xi _{\chi }\pm i\zeta _{\chi }) \\ \end{pmatrix} :(3,-1/3),\nonumber \\&\quad \rho = \begin{pmatrix} \rho _{1}^{+} \\ \frac{1}{\sqrt{2}}(\upsilon _{\rho }+\xi _{\rho }\pm i\zeta _{\rho }) \\ \rho _{3}^{+} \\ \end{pmatrix} :(3,2/3), \nonumber \\&\quad \eta = \begin{pmatrix} \frac{1}{\sqrt{2}}(\upsilon _{\eta }+\xi _{\eta }\pm i\zeta _{\eta }) \\ \eta _{2}^{-} \\ \eta _{3}^{0} \end{pmatrix} :(3,-1/3). \end{aligned}$$

(7)

We extend the scalar sector of the minimal 331 model by adding the following 15 very heavy ${SU}(3)_{L}$ scalar singlets:

$$\begin{aligned} \begin{aligned}&\sigma \sim (1,0),\quad \phi :(1,0),\quad \zeta :(1,0), \\&\varphi _{j} :(1,0),\quad \xi _{j}:(1,0), \\&\tau _{j} :(1,0),\quad \Delta _{j}:(1,0),\quad j=1,2,\\&\Sigma _{k} :(1,0),\quad k=1,2,3,4. \end{aligned} \end{aligned}$$

(8)

We assign the scalars into $S_{3}$ doublet, and $S_{3}$ singlet representations. The $S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ assignments of the scalar fields are

$$\begin{aligned}&\eta \sim \left( \mathbf {1},e^{\frac{2\pi i}{3}},1,1,1\right) ,\quad \rho \sim \left( \mathbf {1,}e^{-\frac{2\pi i}{3}},1,1,1\right) , \\&\chi \sim \left( \mathbf {1,}1,1,1,1\right) ,\quad \sigma \sim \left( \mathbf {1}^{\prime }\mathbf {,}1,1,1,e^{-\frac{\pi i}{8}}\right) \\&\quad \phi \sim \left( \mathbf {1}^{\prime }\mathbf {,}1,1,-i,1\right) ,\quad \zeta \sim \left( \mathbf {1}^{\prime }\mathbf {,}1,1,1,1\right) , \\&\xi \sim \left( \mathbf {2,}1,1,-1,1\right) ,\quad \tau \sim \left( \mathbf {2,}1,1,i^{\frac{1}{2}},1\right) , \\&\varphi _{1} \sim \left( \mathbf {1,}e^{-\frac{2\pi i}{3}},1,-i,1\right) ,\quad \varphi _{2}\sim \left( \mathbf {1}^{\prime }\mathbf {,}e^{-\frac{ 2\pi i}{3}},1,-i,1\right) , \\&\Delta \sim \left( \mathbf {2,}e^{-\frac{2\pi i}{3}},1,-i,1\right) ,\quad \Sigma _{1}\sim \left( \mathbf {1},1,e^{\frac{2\pi i}{3}},-1,e^{ \frac{3i\pi }{8}}\right) , \\&\Sigma _{2} \sim \left( \mathbf {1},1,e^{\frac{2\pi i}{3}},-1,e^{\frac{ 2i\pi }{8}}\right) ,\quad \Sigma _{3}\sim \left( \mathbf {1}^{\prime },1,e^{\frac{2\pi i}{3}},-1,e^{-\frac{i\pi }{7}}\right) , \\&\quad \Sigma _{4} \sim \left( \mathbf {1},1,e^{-\frac{2\pi i}{3}},-1,1\right) . \end{aligned}$$

It has been shown in Ref. [81] that the minimization equations for the scalar potential involving the $S_{3}$ scalar doublet, imply that the $S_{3}$ scalar doublets $\xi$, $\tau$ and $\Delta$ can acquire the following VEV pattern:

$$\begin{aligned} \left\langle \xi \right\rangle =v_{\xi }\left( 1,0\right) ,\quad \left\langle \tau \right\rangle =v_{\tau }\left( 1,1\right) ,\quad \left\langle \Delta \right\rangle =v_{\Delta }\left( 1,0\right) . \end{aligned}$$

(9)

The vacuum configuration of a $S_{3}$ scalar doublet, pointing either in the $\left( 1,0\right)$ or in the $\left( 1,1\right)$ $S_{3}$ directions, has been considered in several $S_{3}$ flavor models (see for instance Refs. [78, 81, 211]). In our model we assume the hierarchy $v_{\Delta }<<v_{\tau }<<v_{\xi }$, between the VEVs of the $S_{3}$ scalar doublets in order to neglect the mixings between these fields and to treat their scalar potentials independently. Let us note that mixing angles between those fields are suppressed by the ratios of their VEVs, as follows from the method of recursive expansion of Ref. [212].

In the lepton sector, we have the following $S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ assignments:

$$\begin{aligned}&L_{L}^{1} \sim \left( \mathbf {1,}e^{\frac{2\pi i}{3}},1,i^{\frac{1}{2} },1\right) ,\nonumber \\&\quad L_{L}=\left( L_{L}^{2},L_{L}^{3}\right) \sim \left( \mathbf {2,}e^{\frac{2\pi i}{3}},1,i^{\frac{1}{2}},1\right) \nonumber \\&e_{R} \sim \left( \mathbf {1}^{\prime }\mathbf {,}e^{-\frac{2\pi i}{3}},1,i^{ \frac{1}{2}},-1\right) ,\quad \mu _{R}\sim \left( \mathbf {1}^{\prime } \mathbf {,}e^{-\frac{2\pi i}{3}},1,1,e^{\frac{\pi i}{4}}\right) ,\nonumber \\&\quad \tau _{R} \sim \left( \mathbf {1}^{\prime }\mathbf {,}e^{-\frac{2\pi i}{3} },1,1,1\right) N_{R}^{1}\sim \left( \mathbf {1,}e^{\frac{2\pi i}{3 }},1,i^{\frac{1}{2}},1\right) \nonumber \\&N_{R} =\left( N_{R}^{2},N_{R}^{3}\right) \sim \left( \mathbf {2,}e^{\frac{ 2\pi i}{3}},1,i^{\frac{1}{2}},1\right) , \end{aligned}$$

(10)

while the $S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ assignments for the quark sector are

$$\begin{aligned}&Q_{L} =\left( Q_{1L},Q_{2L}\right) \sim \left( \mathbf {2},1,1,-1,e^{-\frac{ i\pi }{8}}\right) ,\nonumber \\&\quad Q_{L}^{3}\sim \left( \mathbf {1,} 1,1,1,1\right) , \nonumber \\&U_{R}^{1} \sim \left( \mathbf {1,}e^{-\frac{2\pi i}{3}},e^{\frac{2\pi i}{3} },1,e^{\frac{6i\pi }{8}}\right) ,\nonumber \\&\quad U_{R}^{2}\sim \left( \mathbf {1} ^{\prime }\mathbf {,}e^{-\frac{2\pi i}{3}},e^{\frac{2\pi i}{3}},1,e^{\frac{ 2i\pi }{8}}\right) ,\nonumber \\&\quad U_{R}^{2}\sim \left( \mathbf {1,}e^{-\frac{ 2\pi i}{3}},1,1,1\right) , \nonumber \\&D_{R}^{1} \sim \left( \mathbf {1},e^{-\frac{2\pi i}{3}},1,1,e^{\frac{5i\pi }{8}}\right) ,\nonumber \\&\quad D_{R}^{2}\sim \left( \mathbf {1},e^{-\frac{2\pi i }{3}},e^{-\frac{2\pi i}{3}},-1,e^{\frac{3i\pi }{8}}\right) ,\nonumber \\&\quad D_{R}^{3}\sim \left( \mathbf {1}^{\prime },e^{-\frac{2\pi i}{3} },1,-1,1\right) , \nonumber \\&T_{R} \sim \left( \mathbf {1}^{\prime }\mathbf {,}1,1,1,1\right) ,\quad J_{R}^{1}\sim \left( \mathbf {1}^{\prime }\mathbf {,}1,1,1,-1\right) ,\nonumber \\&\quad J_{R}^{2}\sim \left( \mathbf {1}^{\prime }\mathbf {,}1,1,1,-i\right) . \end{aligned}$$

(11)

In the following we explain the role each discrete group factors of our model. The $S_{3}$, $Z_{3}$, $Z_{3}^{\prime }$, and $Z_{8}$ discrete groups reduce the number of the ${SU}(3)_{C}\otimes {SU}(3)_{L}\otimes U(1)_{X}$ model parameters. This allow us to get viable textures for the fermion sector that successfully describe the prevailing pattern of fermion masses and mixings, as we will show in Sects. 3 and . Let us note that we use the $S_{3}$ discrete group since it is the smallest non-Abelian group that has been considerably studied in the literature. It is worth mentioning that the ${SU}(3)_{L}$ scalar triplets are assigned to a $S_{3}$ trivial singlet representation, whereas the ${SU}(3)_{L}$ scalar singlets are accommodated into three $S_{3}$ doublets, three $S_{3}$ trivial singlets and three $S_{3}$ non-trivial singlets. The $Z_{3}$ and $Z_{8}$ symmetries determines the allowed entries of the charged lepton mass matrix. Furthermore, the $Z_{3}$ symmetry distinguishes the right-handed exotic quarks, being neutral under $Z_{3}$ from the right-handed SM quarks, charged under this symmetry. Note that SM right-handed quarks are the only quark fields transforming non trivially under the $Z_{3}$ symmetry. This results in the absence of mixing between SM quarks and exotic quarks. Consequently, the $Z_{3}$ symmetry is crucial for decoupling the SM quarks from the exotic quarks. Besides that, the $Z_{3}^{\prime }$ symmetry selects the allowed entries of the SM quark mass matrices. Besides that, the $Z_{8}$ symmetry separates the $S_{3}$ scalar doublets participating in the quark Yukawa interactions from those ones participating in the charged lepton and neutrino Yukawa interactions. The $Z_{16}$ symmetry generates the hierarchy among charged fermion masses and quark mixing angles that yields the observed charged fermion mass and quark mixing pattern. It is worth mentioning that the properties of the $Z_{N}$ groups imply that the $Z_{16}$ symmetry is the smallest cyclic symmetry that allows one to build the Yukawa term $\overline{L}_{L}^{1}\rho e_{R}\frac{\sigma ^{8}}{\Lambda ^{8}}$ of dimension 12 from a $\frac{\sigma ^{8}}{\Lambda ^{8}}$ insertion on the $\overline{L}_{L}^{1}\rho e_{R}$ operator, crucial to get the required $\lambda ^{8}$ suppression (where $\lambda =0.225$ is one of the Wolfenstein parameters) needed to naturally explain the smallness of the electron mass.

Now let us briefly comment on a possible large discrete symmetry group that could be used to embed the $S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ discrete symmetry of our model. Considering that the discrete group $\Delta \left( 6N^{2}\right)$ is isomorphic to$\ (Z_{N}\times Z_{N}^{\prime })\rtimes S_{3}$ [39] and the fact the $Z_{24}$ discrete group is the smallest cyclic group that contains the $Z_{3}$ and $Z_{8}$ symmetries and the $Z_{3}^{\prime }$ symmetry is contained in the $Z_{24}^{\prime }$ group, it follows that the $S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ discrete group of our model can be embedded in the $\Delta \left( 6N^{2}\right) =\Delta$ $\left( 3456\right)$ discrete group (where $N=24$). It would be interesting to implement the $\Delta \left( 6N^{2}\right)$ discrete symmetry in the 331 model and to study its implications on fermion masses and mixings. This requires careful studies that are beyond the scope of the present paper and will be done elsewhere.

With the aforementioned field content of our model, the relevant quark and lepton Yukawa terms invariant under the group $\mathcal {G}$, take the form

$$\begin{aligned} \mathcal {L}_{Y}^{\left( Q\right) }= & {} y_{33}^{\left( U\right) }\overline{Q} _{L}^{3}\eta U_{R}^{3}+y_{23}^{\left( U\right) }\overline{Q}_{L}^{2}\rho ^{*}U_{R}^{3}\frac{\xi \sigma }{\Lambda ^{2}}\nonumber \\&+y_{22}^{\left( U\right) } \overline{Q}_{L}^{2}\rho ^{*}U_{R}\frac{\xi \sigma ^{3}}{\Lambda ^{4}}\nonumber \\&+\,y_{11}^{\left( U\right) }\overline{Q}_{L}^{1}\rho ^{*}U_{R}\frac{\xi \sigma ^{7}}{\Lambda ^{8}} \nonumber \\&+\,y_{33}^{\left( D\right) }\overline{Q}_{L}^{3}\rho D_{R}^{3}\frac{\sigma ^{2}\Sigma _{2}}{\Lambda ^{3}}+y_{22}^{\left( D\right) }\overline{Q}_{L}\eta ^{*}D_{R}^{2}\frac{\xi \Sigma _{3}\sigma ^{3}}{\Lambda ^{5}}\nonumber \\&+\,y_{12}^{\left( D\right) }\overline{Q}_{L}\eta ^{*}D_{R}^{2}\frac{\xi \Sigma _{4}\sigma ^{4}}{\Lambda ^{6}} \nonumber \\&+\,y_{13}^{\left( D\right) }\overline{Q}_{L}\eta ^{*}D_{R}^{3}\frac{\xi \sigma ^{4}\Sigma _{1}}{\Lambda ^{6}}+y_{11}^{\left( D\right) }\overline{Q} _{L}\eta ^{*}D_{R}^{1}\frac{\xi \sigma ^{6}}{\Lambda ^{7}} \nonumber \\&+\,y^{\left( T\right) }\overline{Q}_{L}^{3}\chi T_{R}+y_{1}^{\left( J\right) }\overline{Q}_{L}^{1}\chi ^{*}J_{R}^{1}+y_{2}^{\left( J\right) } \overline{Q}_{L}^{2}\chi ^{*}J_{R}^{2}+H.c.,\nonumber \\ \end{aligned}$$

(12)

$$\begin{aligned} -\mathcal {L}_{Y}^{\left( L\right) }= & {} h_{1\rho e}^{\left( L\right) } \overline{L}_{L}^{1}\rho e_{R}\frac{\sigma ^{8}}{\Lambda ^{8}}+h_{1\rho \mu }^{\left( L\right) }\left( \overline{L}_{L}\rho \tau \right) _{\mathbf { \mathbf {1}}}\mu _{R}\frac{\sigma ^{2}}{\Lambda ^{3}}\nonumber \\&+\,h_{2\rho \mu }^{\left( L\right) }\left( \overline{L}_{L}\rho \tau \right) _{\mathbf {1}^{\prime }}\mu _{R}\frac{\sigma ^{2}\zeta }{\Lambda ^{4}} \nonumber \\&+\,h_{1\rho \tau }^{\left( L\right) }\left( \overline{L}_{L}\rho \tau \right) _{\mathbf {1}}\tau _{R}\frac{1}{\Lambda }+h_{2\rho \tau }^{\left( L\right) }\left( \overline{L}_{L}\rho \tau \right) _{\mathbf {1}^{\prime }}\tau _{R}\frac{\zeta }{\Lambda ^{2}}\nonumber \\&+\,h_{1\chi }^{\left( L\right) }\left( \overline{L}_{L}\chi N_{R}\right) _{\mathbf {\mathbf {1}}}+h_{3\chi }^{\left( L\right) }\overline{L}_{L}^{1}\chi N_{R}^{1} \nonumber \\&+\,\frac{1}{2}h_{1N}\left( \overline{N}_{R}N_{R}^{C}\right) _{\mathbf {1} }\varphi _{1}\nonumber \\&+\,\frac{1}{2}h_{2N}\overline{N}_{R}^{1}N_{R}^{1}\varphi _{1}+ \frac{1}{2}h_{3N}\left( \overline{N}_{R}N_{R}^{C}\right) _{\mathbf {1} ^{\prime }}\varphi _{2} \nonumber \\&+\,h_{\rho }^{\left( 1\right) }\varepsilon _{abc}\left( \overline{L} _{L}^{a}\left( L_{L}^{C}\right) ^{b}\right) _{\mathbf {1}^{\prime }}\left( \rho ^{*}\right) ^{c}\frac{\phi }{\Lambda }\nonumber \\&+\,h_{\rho }^{\left( 2\right) }\varepsilon _{abc}\left( \overline{L}_{L}^{a}\left( L_{L}^{1C}\right) ^{b}\left( \rho ^{*}\right) ^{c}\Delta \right) _{\mathbf {1}}\frac{1}{ \Lambda } \nonumber \\&+\,h_{\rho }^{\left( 3\right) }\varepsilon _{abc}\left( \left( \overline{L} _{L}^{1}\right) ^{a}\left( L_{L}^{C}\right) ^{b}\left( \rho ^{*}\right) ^{c}\Delta \right) _{\mathbf {1}}\frac{1}{\Lambda }+H.c., \nonumber \\ \end{aligned}$$

(13)

where the dimensionless couplings in Eqs. (12) and (13) are $\mathcal {O}(1)$ parameters.

Considering that the charged fermion mass and quark mixing pattern arises from the breaking of the $Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ discrete group, we set the VEVs of the ${SU}(3)_{L}$ singlet scalars $\sigma$, $\zeta$, $\phi$, $\tau _{j}$, $\Delta _{j}$, $\xi _{j}$ ($j=1,2$) and $\Sigma _{k}$ ($k=1,2,3,4$) scalar singlets, as follows:

$$\begin{aligned}&v_{\phi }\sim v_{\Delta }\sim \lambda ^{5}\Lambda \ll v_{\tau }\Lambda \nonumber \\&\quad =\,\lambda ^{3}\ll v_{\sigma }=v_{\zeta }=v_{\xi }=v_{\Sigma _{k}}=\Lambda _{int}=\lambda \Lambda ,\nonumber \\&\quad k=1,2,3,4, \end{aligned}$$

(14)

where $\lambda =0.225$ is one of the parameters of the Wolfenstein parametrization and $\Lambda$ the cutoff of our model. Let us note that the ${SU}(3)_{L}$ singlet scalar fields $\sigma$, $\zeta$, $\xi _{j}$ ($j=1,2$) and $\Sigma _{k}$ ($k=1,2,3,4$) having the VEVs of the same order of magnitude are the ones that appear in the SM charged fermion Yukawa terms, thus playing an important role in generating the SM charged fermion masses and quark mixing angles. Regarding the ${SU}(3)_{L}$ singlet scalar fields $\tau _{j}$ ($j=1,2$), which participates in the charged lepton Yukawa interactions, we assume that it acquires a VEV, much smaller than $\lambda \Lambda$ (we set its VEV as $\lambda ^{3}\Lambda$) in order to suppress its mixing with the $S_{3}$ scalar doublet $\xi$, which allows us to treat their scalar potentials independently. Because of the reason mentioned above, we have also assumed that the $S_{3}$ scalar doublet $\Delta$, which appears in the Dirac neutrino Yukawa terms, acquires a VEV much smaller than $\lambda ^{3}\Lambda$, which we set close to $\lambda ^{5}\Lambda$. Furthermore, in order to have the neutrino sector model parameters of the same order, we have assumed that $v_{\phi }\sim v_{\Delta }$. As previously mentioned, the aforementioned hierarchy between the VEVs of the $S_{3}$ scalar doublets $\xi$, $\tau$ and $\Delta$ allows us to treat their scalar potentials independently, thus providing a more natural justification for their chosen VEV patterns given in Eq. (9) as natural solutions of the scalar potential minimization equations for the whole region of parameter space.

As we will explain in the following, we are going to implement an inverse seesaw mechanism for the generation of the light active neutrino masses. To implement an inverse seesaw mechanism, we need very light right-handed Majorana neutrinos, which implies that the ${SU}(3)_{L}$ singlet scalars having Yukawa interactions with those neutrinos should acquire very small vacuum expectation values, much smaller than the scale of breaking of the SM electroweak symmetry. Because of this reason, we further assume that the ${SU}(3)_{L}$ scalar singlets $\varphi _{j}$ ($j=1,2$) giving masses to the right handed Majorana neutrinos have VEVs much smaller than the electroweak symmetry breaking scale, then providing small masses to these Majorana neutrinos, and thus giving rise to an inverse seesaw mechanism of active neutrino masses. Therefore, we have the following hierarchy among the VEVs of the scalar fields in our model:

$$\begin{aligned} v_{\varphi _{1}}\sim v_{\varphi _{2}}\ll v_{\rho }\sim v_{\eta } \sim v \ll v_{\chi }\ll v_{\phi }\sim v_{\Delta }\ll v_{\tau }\ll \Lambda _{int}.\nonumber \\ \end{aligned}$$

(15)

In summary, for the reasons mentioned above and considering a very high model cutoff $\Lambda \gg v_{\chi }$, we set the vacuum expectation values (VEVs) of the ${SU}(3)_{L}$ scalar singlets at a very high energy, much larger than $v_{\chi }\approx \mathcal {O}(1)$ TeV, with the exception of the VEVs of $\varphi _{j}$, $\Delta _{j}$ ($j=1,2$), taken to be much smaller than the electroweak symmetry breaking scale $v=246$ GeV. It is noteworthy that the ${SU}(3)_{C}\otimes {SU}(3)_{L}\otimes U(1)_{X}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ symmetry is broken down to ${SU}(3)_{C}\otimes {SU}(3)_{L}\otimes U(1)_{X}\otimes Z_{3}$, at the scale $\Lambda _{int}$, by the vacuum expectation values of the ${SU}(3)_{L}$ singlet scalar fields $\sigma$, $\zeta$, $\xi _{j}$ and $\Sigma _{k}$ ($k=1,2,3,4$).

It is worth mentioning that in order that the small VEVs of the ${SU}(3)_{L}$ scalar singlets $\varphi _{j}$ ($j=1,2$) be stable under radiative corrections, a Veltmann condition that connects a combination of the quartic couplings of the scalar potential that involve a pair of these scalar fields with the remaining ones and the combination of the Yukawa couplings of these scalar singlets with the right-handed Majorana neutrinos, has to be fulfilled. That Veltmann condition will arise by requiring the cancellation of the quadratically divergent scalar and fermionic contributions, contributions that interfere destructively. The aforementioned Veltmann condition will constrain the quartic scalar couplings of the scalar interactions involving a pair of the scalar fields that acquire very small VEVs. The resulting constraints on these quartic scalar couplings will not affect neither the fermions masses and mixings nor the flavor changing top quark decays. Having the VEVs of the scalar fields of our model stable under radiative corrections in the whole region of parameter space, will require one to embed our model in a warped five dimensional framework or to implement supersymmetry. This requires careful studies which are left beyond the scope of the present paper.

2.2 Low energy scalar potential

The renormalizable low energy scalar potential of the model takes the form

$$\begin{aligned} V_{H}= & {} \mu _{\chi }^{2}(\chi ^{\dagger }\chi )+\mu _{\eta }^{2}(\eta ^{\dagger }\eta )+\mu _{\rho }^{2}(\rho ^{\dagger }\rho )\nonumber \\&+\,f\left( \eta _{i}\chi _{j}\rho _{k}\varepsilon ^{ijk}+H.c.\right) +\lambda _{1}(\chi ^{\dagger }\chi )(\chi ^{\dagger }\chi ) \nonumber \\&+\,\lambda _{2}(\rho ^{\dagger }\rho )(\rho ^{\dagger }\rho )+\lambda _{3}(\eta ^{\dagger }\eta )(\eta ^{\dagger }\eta )+\lambda _{4}(\chi ^{\dagger }\chi )(\rho ^{\dagger }\rho )\nonumber \\&+\,\lambda _{5}(\chi ^{\dagger }\chi )(\eta ^{\dagger }\eta ) \nonumber \\&+\,\lambda _{6}(\rho ^{\dagger }\rho )(\eta ^{\dagger }\eta )+\lambda _{7}(\chi ^{\dagger }\eta )(\eta ^{\dagger }\chi )+\lambda _{8}(\chi ^{\dagger }\rho )(\rho ^{\dagger }\chi )\nonumber \\&+\,\lambda _{9}(\rho ^{\dagger }\eta )(\eta ^{\dagger }\rho ). \end{aligned}$$

(16)

After the symmetry breaking, it is found that the scalar mass eigenstates are connected with the weak scalar states by the following relations:

$$\begin{aligned} \begin{pmatrix} G_{1}^{\pm } \\ H_{1}^{\pm } \\ \end{pmatrix}= & {} R_{\beta _{T}} \begin{pmatrix} \rho _{1}^{\pm } \\ \eta _{2}^{\pm } \\ \end{pmatrix} ,\quad \begin{pmatrix} G_{1}^{0} \\ A_{1}^{0} \\ \end{pmatrix} =R_{\beta _{T}} \begin{pmatrix} \zeta _{\rho } \\ \zeta _{\eta } \\ \end{pmatrix},\quad \begin{pmatrix} H_{1}^{0} \\ h^{0} \\ \end{pmatrix} =R_{\alpha _{T}} \begin{pmatrix} \xi _{\rho } \\ \xi _{\eta } \\ \end{pmatrix} , \nonumber \\ \end{aligned}$$

(17)

$$\begin{aligned} \begin{pmatrix} G_{2}^{0} \\ H_{2}^{0} \\ \end{pmatrix}= & {} R \begin{pmatrix} \chi _{1}^{0} \\ \eta _{3}^{0} \\ \end{pmatrix} ,\quad \begin{pmatrix} G_{3}^{0} \\ H_{3}^{0} \\ \end{pmatrix} =R \begin{pmatrix} \zeta _{\chi } \\ \xi _{\chi } \\ \end{pmatrix}, \quad \begin{pmatrix} G_{2}^{\pm } \\ H_{2}^{\pm } \\ \end{pmatrix} =R \begin{pmatrix} \chi _{2}^{\pm } \\ \rho _{3}^{\pm } \\ \end{pmatrix} , \nonumber \\ \end{aligned}$$

(18)

with

$$\begin{aligned} R_{\alpha _{T}(\beta _{T})}=\left( \begin{array}{cc} \cos \alpha _{T}(\beta _{T}) &{} \sin \alpha _{T}(\beta _{T}) \\ -\sin \alpha _{T}(\beta _{T}) &{} \cos \alpha _{T}(\beta _{T}) \end{array} \right) ,\quad R=\left( \begin{array}{cc} -1 &{} 0 \\ 0 &{} 1 \end{array} \right) , \end{aligned}$$

(19)

where $\tan \beta _{T}=v_{\eta }/v_{\rho }$, and $\tan 2\alpha _{T}=M_{1}^{2}/(M_{2}^{2}-M_{3}^{2})$ with:

$$\begin{aligned} M_{1}^{2}= & {} 4\lambda _{6}v_{\eta }v_{\rho }+2\sqrt{2}fv_{\chi }, \nonumber \\ M_{2}^{2}= & {} 4\lambda _{2}v_{\rho }^{2}-\sqrt{2}fv_{\chi }\tan \beta _{T}, \\ M_{3}^{2}= & {} 4\lambda _{3}v_{\eta }^{2}-\sqrt{2}fv_{\chi }/\tan \beta _{T}.\nonumber \end{aligned}$$

(20)

The low energy physical scalar spectrum of our model includes: four massive charged Higgs ($H_{1}^{\pm }$, $H_{2}^{\pm }$), one CP-odd Higgs ($A_{1}^{0}$), three neutral CP-even Higgs ($h^{0},H_{1}^{0},H_{3}^{0}$) and two neutral Higgs ($H_{2}^{0},\overline{H}_{2}^{0}$) bosons. The scalar $h^{0}$ is identified with the SM-like 126 GeV Higgs boson found at the LHC. It it noteworthy that the neutral Goldstone bosons $G_{1}^{0}$, $G_{3}^{0}$, $G_{2}^{0}$, $\overline{G}_{2}^{0}$ are associated to the longitudinal components of the Z, $Z^{\prime }$, $K^{0}$, and $\overline{K}^{0}$gauge bosons, respectively. Furthermore, the charged Goldstone bosons $G_{1}^{\pm }$ and $G_{2}^{\pm }$ are associated to the longitudinal components of the $W^{\pm }$ and $K^{\pm }$ gauge bosons, respectively.

3 Lepton masses and mixings

From Eqs. (9), (13), and (14) and using the product rules of the $S_{3}$ group given in Appendix A, it follows that the mass matrix for charged leptons is

$$\begin{aligned} M_{l}=\left( \begin{array}{ccc} a_{11}^{\left( l\right) }\lambda ^{8} &{}\quad 0 &{}\quad 0 \\ 0 &{} a_{22}^{\left( l\right) }\lambda ^{5} &{} a_{23}^{\left( l\right) }\lambda ^{3} \\ 0 &{} a_{32}^{\left( l\right) }\lambda ^{5} &{} a_{33}^{\left( l\right) }\lambda ^{3} \end{array} \right) \frac{v}{\sqrt{2}}. \end{aligned}$$

(21)

Since the charged lepton mass hierarchy arises from the breaking of the $Z_{3}\otimes Z_{8}\otimes Z_{16}$ discrete group and in order to simplify the analysis, we consider a scenario of approximate universality in the dimensionless SM charged lepton Yukawa couplings, as follows:

$$\begin{aligned} a_{32}^{\left( l\right) }=a_{4}^{\left( l\right) }, a_{23}^{\left( l\right) }=a_{4}^{\left( l\right) }e^{-i\alpha } \end{aligned}$$

(22)

where $a_{11}^{\left( l\right) }$, $a_{22}^{\left( l\right) }$, $a_{33}^{\left( l\right) }$ and $a_{4}^{\left( l\right) }$ are assumed to be real $\mathcal {O}(1)$ parameters.

The matrix $M_{l}M_{l}^{\dagger }$ is diagonalized by a rotation matrix $R_{l}$ according to

$$\begin{aligned} R_{l}^{\dagger }M_{l}M_{l}^{\dagger }R_{l}= & {} diag\left( m_{e},m_{\mu },m_{\tau }\right) ,\nonumber \\ R_{l}= & {} \left( \begin{array}{ccc} 1 &{} 0 &{} 0 \\ 0 &{} \cos \theta _{l} &{} -\sin \theta _{l}e^{-i\alpha } \\ 0 &{} \sin \theta _{l}e^{i\alpha } &{} \cos \theta _{l} \end{array} \right) , \nonumber \\ \tan \theta _{l}\simeq & {} -\frac{a_{4}^{\left( l\right) }}{a_{33}^{\left( l\right) }},\quad \cos \theta _{l}\simeq \frac{a_{33}^{\left( l\right) }}{\sqrt{\left( a_{33}^{\left( l\right) }\right) ^{2}+\left( a_{4}^{\left( l\right) }\right) ^{2}}},\nonumber \\ \sin \theta _{l}\simeq & {} -\frac{ a_{4}^{\left( l\right) }}{\sqrt{\left( a_{33}^{\left( l\right) }\right) ^{2}+\left( a_{4}^{\left( l\right) }\right) ^{2}}}, \end{aligned}$$

(23)

where from Eq. (21) it follows that the charged lepton masses are approximately given by

$$\begin{aligned} m_{e}= & {} a_{11}^{\left( l\right) }\lambda ^{8}\frac{v}{\sqrt{2}},\quad m_{\mu }\simeq \frac{\left| a_{22}^{\left( l\right) }a_{33}^{\left( l\right) }-\left( a_{4}^{\left( l\right) }\right) ^{2}\right| }{\sqrt{ \left( a_{33}^{\left( l\right) }\right) ^{2}+\left( a_{4}^{\left( l\right) }\right) ^{2}}}\lambda ^{5}\frac{v}{\sqrt{2}},\nonumber \\&m_{\tau }\simeq \sqrt{\left( a_{33}^{\left( l\right) }\right) ^{2}+\left( a_{4}^{\left( l\right) }\right) ^{2}}\lambda ^{3}\frac{v}{\sqrt{2}}. \end{aligned}$$

(24)

Note that the charged lepton masses are connected with the electroweak symmetry breaking scale $v=246$ GeV by their scalings with powers of the Wolfenstein parameter $\lambda =0.225$, with $\mathcal {O}(1)$ coefficients. This is consistent with our previous assumption made in Eq. (14) regarding the size of the VEVs for the ${SU}(3)_{L}$ singlet scalars appearing in the charged fermion Yukawa terms. Furthermore, it is noteworthy that the mixing angle $\theta _{l}$ in the charged lepton sector is large, which gives rise to an important contribution to the leptonic mixing matrix, coming from the mixing of charged leptons.

In the concerning to the neutrino sector, the following neutrino mass terms arise:

$$\begin{aligned} -\mathcal {L}_{mass}^{\left( \nu \right) }=\frac{1}{2}\left( \begin{array}{ccc} \overline{\nu _{L}^{C}}&\overline{\nu _{R}}&\overline{N_{R}} \end{array} \right) M_{\nu }\left( \begin{array}{c} \nu _{L} \\ \nu _{R}^{C} \\ N_{R}^{C} \end{array} \right) +H.c, \end{aligned}$$

(25)

where the $S_{3}$ discrete flavor group constrains the neutrino mass matrix to be of the form

$$\begin{aligned}&M_{\nu } =\left( \begin{array}{ccc} 0_{3\times 3} &{} M_{D} &{} 0_{3\times 3} \\ M_{D}^{T} &{} 0_{3\times 3} &{} M_{\chi } \\ 0_{3\times 3} &{} M_{\chi }^{T} &{} M_{R} \end{array}\quad \right) ,\quad M_{D}=\frac{v_{\xi }v_{\phi }v_{\rho }}{\sqrt{2} \Lambda ^{2}}\left( \begin{array}{ccc} 0 &{} a &{} 0 \\ -a &{} 0 &{} b \\ 0 &{} -b &{} 0 \end{array} \right) , \nonumber \\&M_{\chi }=h_{1\chi }^{\left( L\right) }\frac{v_{\chi }}{\sqrt{2}}\left( \begin{array}{ccc} x &{} 0 &{} 0 \\ 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} 1 \end{array} \right) , \quad M_{R}=h_{1N}v_{\varphi _{1}}\left( \begin{array}{ccc} 1 &{} 0 &{} 0 \\ 0 &{} y &{} z \\ 0 &{} z &{} y \end{array} \right) , \end{aligned}$$

$$\begin{aligned}&a =h_{\rho }^{\left( 2\right) }-h_{\rho }^{\left( 3\right) },\quad b=2h_{\rho }^{\left( 1\right) }, \quad x=\frac{h_{2\chi }^{\left( L\right) }}{h_{1\chi }^{\left( L\right) }},\nonumber \\&y=\frac{h_{2N}}{ h_{1N}},\quad z=\frac{h_{3N}v_{\varphi _{2}}}{h_{1N}v_{\varphi _{1}}}. \end{aligned}$$

(26)

Since the ${SU}(3)_{L}$ scalar singlets $\varphi _{j}$ ($j=1,2$) having Yukawa interactions with the right-handed Majorana neutrinos acquire VEVs much smaller than the electroweak symmetry breaking scale, these Majorana neutrinos are very light, so that the active neutrinos get small masses via the inverse seesaw mechanism.

As shown in detail in Ref. [182], the full rotation matrix is approximately given by

$$\begin{aligned} \mathbb {U}= \begin{pmatrix} V_{\nu } &{} B_{3}U_{\chi } &{} B_{2}U_{R} \\ -\frac{(B_{2}^{\dagger }+B_{3}^{\dagger })}{\sqrt{2}}V_{\nu } &{} \frac{(1-S)}{ \sqrt{2}}U_{\chi } &{} \frac{(1+S)}{\sqrt{2}}U_{R} \\ -\frac{(B_{2}^{\dagger }-B_{3}^{\dagger })}{\sqrt{2}}V_{\nu } &{} \frac{(-1-S) }{\sqrt{2}}U_{\chi } &{} \frac{(1-S)}{\sqrt{2}}U_{R} \end{pmatrix} , \end{aligned}$$

(27)

where

$$\begin{aligned} S=-\frac{1}{2\sqrt{2}h_{\chi }^{\left( L\right) }v_{\chi }}M_{R},\quad B_{2}\simeq B_{3}\simeq \frac{1}{h_{\chi }^{\left( L\right) }v_{\chi }}M_{D}^{*}, \end{aligned}$$

(28)

and the physical neutrino mass matrices are

$$\begin{aligned}&M_{\nu }^{\left( 1\right) } =M_{D}\left( M_{\chi }^{T}\right) ^{-1}M_{R}M_{\chi }^{-1}M_{D}^{T}, \end{aligned}$$

(29)

$$\begin{aligned}&\quad M_{\nu }^{\left( 2\right) } =-M_{\chi }^{T}+\frac{1}{2}M_{R},\quad M_{\nu }^{\left( 3\right) }=M_{\chi }^{T}+\frac{1}{2}M_{R}, \end{aligned}$$

(30)

where $M_{\nu }^{\left( 1\right) }$ corresponds to the active neutrino mass matrix whereas $M_{\nu }^{\left( 2\right) }$ and $M_{\nu }^{\left( 3\right) }$ are the exotic Dirac neutrino mass matrices. Note that the physical eigenstates include three active neutrinos and six exotic neutrinos. The exotic neutrinos are pseudo-Dirac, with masses $\sim \pm M_{\chi }^{T}$ and a small splitting $M_{R}$. Furthermore, $V_{\nu }$, $U_{R}$ and $U_{\chi }$ are the rotation matrices which diagonalize $M_{\nu }^{\left( 1\right) }$, $M_{\nu }^{\left( 2\right) }$ and $M_{\nu }^{\left( 3\right) }$, respectively.

Furthermore, as follows from Eq. (27), we can connect the neutrino fields $\nu _{L}=\left( \nu _{1L},\nu _{2L},\nu _{3L}\right) ^{T}$, $\nu _{R}^{C}=\left( \nu _{1R}^{C},\nu _{2R}^{C},\nu _{3R}^{C}\right)$ and $N_{R}^{C}=\left( N_{1R}^{C},N_{2R}^{C},N_{3R}^{C}\right)$ with the neutrino mass eigenstates by the following approximate relations:

$$\begin{aligned} \left( \begin{array}{ll} \nu _{L} \\ \nu _{R}^{C} \\ N_{R}^{C} \end{array} \right)\simeq & {} \left( \begin{array}{c} V_{\nu }\xi _{L}^{\left( 1\right) } \\ \frac{1}{\sqrt{2}}U_{\chi }\xi _{L}^{\left( 2\right) }+\frac{1}{\sqrt{2}} U_{R}\xi _{L}^{\left( 3\right) } \\ -\frac{1}{\sqrt{2}}U_{\chi }\xi _{L}^{\left( 2\right) }+\frac{1}{\sqrt{2}} U_{R}\xi _{L}^{\left( 3\right) } \end{array} \right) ,\nonumber \\ \xi _{L}^{\left( j\right) }= & {} \left( \begin{array}{c} \xi _{1L}^{\left( j\right) } \\ \xi _{2L}^{\left( j\right) } \\ \xi _{3L}^{\left( j\right) } \end{array} \right) ,\quad j=1,2,3, \end{aligned}$$

(31)

where $\xi _{kL}^{\left( 1\right) }$, $\xi _{kL}^{\left( 2\right) }$ and $\xi _{kL}^{\left( 3\right) }$ ($k=1,2,3$) are the three active neutrinos and six exotic neutrinos, respectively.

From Eq. (29) it follows that the light active neutrino mass matrix is given by

$$\begin{aligned} M_{\nu }^{\left( 1\right) }= & {} m_{\nu }\left( \begin{array}{ccc} a^{2} &{} \kappa ab &{} -ab \\ \kappa ab &{} c^{2} &{} -\kappa b^{2} \\ -ab &{} \quad -\kappa b^{2} &{} b^{2} \end{array} \right) , \nonumber \\ m_{\nu }= & {} \frac{h_{1N}v_{\varphi _{1}}v_{\xi }^{2}v_{\phi }^{2}v_{\rho }^{2}}{\left( h_{1\chi }^{\left( L\right) }\right) ^{2}v_{\chi }^{2}\Lambda ^{4}},\kappa =\frac{z}{y}, \nonumber \\ c^{2}= & {} b^{2}+\frac{a^{2}}{x^{2}y}. \end{aligned}$$

(32)

Let us note that the smallness of the active neutrino masses arises from their scaling with inverse powers of the high energy cutoff $\Lambda$ as well as from their linear dependence on the very small VEVs of the ${SU}(3)_{L}$ singlets $\varphi _{j}$ ($j=1,2$), assumed to be of the same order of magnitude.

Considering that the orders of magnitude of the SM particles and new physics yield the constraints $v_{\chi }\gtrsim 1$ TeV and $v_{\eta }^{2}+v_{\rho }^{2}=v^{2}$ and taking into account our assumption that the dimensionless lepton Yukawa couplings are $\mathcal {O}(1)$ parameters, from Eq. (32) and the relations $v_{\xi }=\lambda \Lambda$, $v_{\phi }\sim \lambda ^{5}\Lambda$, $v_{\rho }\sim 100$ GeV, $v_{\chi }\sim 1$ TeV, we see that the mass scale for the light active neutrinos satisfies $m_{\nu }\sim 10^{-10}v_{\varphi }$. Consequently, taking $m_{\nu }\sim 50$ meV, we find for the VEV $v_{\varphi _{1}}$ of the singlet scalar $\varphi _{1}$ the estimate

$$\begin{aligned} v_{\varphi _{1}}\sim 0.5\;\text {GeV}. \end{aligned}$$

(33)

Fig. 1

Correlations between $\sin ^2\theta _{23}$ and $\sin ^2\theta _{13}$, $\sin ^2\theta _{12}$ and $\sin ^2\theta _{13}$, $\Delta m_{21}^{2}$ and $\Delta m_{31}^{2}$ for the case of normal hierarchy. The horizonal and vertical lines are the minimum and maximum values of the leptonic mixing parameters and neutrino mass squared splittings inside the $3 \sigma$ experimentally allowed range

Fig. 2

Correlations between $\sin ^2\theta _{23}$ and $\sin ^2\theta _{13}$, $\sin ^2\theta _{12}$ and $\sin ^2\theta _{13}$, $\Delta m_{21}^{2}$ and $\Delta m_{31}^{2}$ for the case of inverted hierarchy. The horizontal and vertical lines are the minimum and maximum values of the leptonic mixing parameters and neutrino mass squared splittings inside the $3\sigma$ experimentally allowed range

In the following we proceed to fit the lepton sector model parameters $m_{\nu }$, $a_{11}^{\left( l\right) }$, $a_{22}^{\left( l\right) }$, $a_{33}^{\left( l\right) }$, $a_{4}^{\left( l\right) }$, a, b, c and $\kappa$ to reproduce the experimental values for the physical observables of the lepton sector, i.e., the three charged lepton masses, the two neutrino mass squared splittings and the three leptonic mixing angles. To this end, we fix $m_{\nu }=50$ meV and we vary the parameters $a_{11}^{\left( l\right) }$, $a_{22}^{\left( l\right) }$, $a_{33}^{\left( l\right) }$, $a_{4}^{\left( l\right) }$, a, b, c and $\kappa$ to fit the charged lepton masses, the neutrino mass squared splittings $\Delta m_{21}^{2}$, $\Delta m_{31}^{2}$ (note that we define $\Delta m_{ij}^{2}=m_{i}^{2}-m_{j}^{2}$) and the leptonic mixing angles $\sin ^{2}\theta _{12}$, $\sin ^{2}\theta _{13}$, and $\sin ^{2}\theta _{23}$ to their experimental values for normal (NH) and Inverted (IH) neutrino mass hierarchy. The results shown in Table 3 correspond to the following best-fit values:

$$\begin{aligned} a_{11}^{\left( l\right) }\simeq & {} 0.42,\quad a_{22}^{\left( l\right) }\simeq 1.88,\quad a_{33}^{\left( l\right) }\simeq 0.67, \nonumber \\ \quad a_{4}^{\left( l\right) }\simeq & {} 0.58,\quad a\simeq 0.28, \quad b\simeq 0.39 \nonumber \\ c\simeq & {} -0.97,\quad \kappa \simeq 1.11 \nonumber \\ \theta _{l}\simeq & {} -41.69^{\circ },\quad \alpha \simeq -85.99^{\circ },\quad \text{ for } \text{ NH }, \end{aligned}$$

(34)

$$\begin{aligned} a_{11}^{\left( l\right) }\simeq & {} 0.42,\quad a_{22}^{\left( l\right) }\simeq 2.33,\quad a_{33}^{\left( l\right) }\simeq 0.57, \nonumber \\ \qquad a_{4}^{\left( l\right) }\simeq & {} -0.67,\quad a\simeq 0.98, \qquad b\simeq 0.15 \nonumber \\ c\simeq & {} -0.99,\quad \kappa \simeq -0.05 \nonumber \\ \qquad \theta _{l}\simeq & {} 49.20^{\circ },\quad \alpha \simeq -93.60^{\circ },\quad \text{ for } \text{ IH }. \end{aligned}$$

(35)

Table 3 Model and experimental values of the charged lepton masses, neutrino mass squared splittings, and leptonic mixing parameters for the normal (NH) and inverted (IH) mass hierarchies. Model values for Jarlskog invariant and CP-violating phase

Using the best-fit values given above, we get, for NH and IH, respectively, the following neutrino masses:

$$\begin{aligned}&m_{1}=0,\quad m_{2}\approx 8.72\;\text{ meV },\quad m_{3}\approx 49.80\;\text{ meV },\quad \text{ for } \text{ NH },\end{aligned}$$

(36)

$$\begin{aligned} \nonumber \\\\&m_{1}\approx 49.56\;\text{ meV },\quad m_{2}\approx 48.79\;\text{ meV },\quad m_{3}=0,\quad \text{ for } \text{ IH }.\nonumber \\ \end{aligned}$$

(37)

The obtained and experimental values of the observables in the lepton sector are shown in Table 3. The experimental values of the charged lepton masses, which are given at the $M_{Z}$ scale, have been taken from Ref. [213] (which are similar to those in [214]), whereas the experimental values of the neutrino mass squared splittings and leptonic mixing angles for both normal (NH) and inverted (IH) mass hierarchies, are taken from Ref. [8]. The obtained charged lepton masses, neutrino mass squared splittings, and lepton mixing angles are in excellent agreement with the experimental data for both normal and inverted neutrino mass hierarchies. We found a leptonic Dirac CP-violating phase close to $\frac{\pi }{2}$ and a Jarlskog invariant close to about $3\times 10^{-2}$ for both normal and inverted neutrino mass hierarchy.

In order to study the sensitivity of the obtained values for the neutrino mass squared splittings, under small variations around the best-fit values (maximum variation of $+0.2$, minimum of $-0.2$), we show in Figs. 1 and 2 the correlations between $\sin ^2\theta _{23}$ and $\sin ^2\theta _{13}$, $\sin ^2\theta _{12}$ and $\sin ^2\theta _{13}$, $\Delta m_{21}^{2}$, and $\Delta m_{31}^{2}$ for the case of normal and inverted neutrino mass hierarchies, respectively. These figures show that a slight variation from the best-fit values yields for several points of the parameter space an important deviation in the values of the neutrino mass squared splittings and leptonic mixing parameters, thus making it difficult to reproduce their experimental values, especially for the case of inverted neutrino mass hierarchy. Thus, the solution corresponding to the best-fit point is fine-tuned in the case of inverted neutrino mass hierarchy. Addressing this problem requires one to consider a discrete flavor group having a triplet irreducible representation, such as, for example $A_4$, $S_4$ and $T^{\prime }$. This will yield more predictive textures for the lepton sector thus solving the fine tuning problem. Addressing this issue requires a careful investigation beyond the scope of the present paper and which is is left for future studies.

Now we determine the effective Majorana neutrino mass parameter, which is proportional to the neutrinoless double beta ($0\nu \beta \beta$) decay amplitude. The effective Majorana neutrino mass parameter is given by

$$\begin{aligned} m_{\beta \beta }=\left| \sum _{j}U_{ek}^{2}m_{\nu _{k}}\right| , \end{aligned}$$

(38)

where $U_{ej}^{2}$ and $m_{\nu _{k}}$ are the PMNS mixing matrix elements and the Majorana neutrino masses, respectively.

We predict the effective Majorana neutrino mass parameter for both normal and inverted hierarchies:

$$\begin{aligned} m_{\beta \beta }=\left\{ \begin{array}{ll} 4\ \text{ meV }&{}\quad \text {for NH} ,\\ 48\ \text{ meV }&{} \quad \text {for IH}. \\ \end{array} \right. \end{aligned}$$

(39)

Our obtained value $m_{\beta \beta }\approx 4\ \text{ meV }$ for the effective Majorana neutrino mass parameter in the case of normal hierarchy, is beyond the reach of the present and forthcoming $0\nu \beta \beta$ decay experiments. Concerning the inverted neutrino mass hierarchy, we get the value $m_{\beta \beta }\approx 48$ for the Majorana neutrino mass parameter, which is within the declared reach of the next-generation bolometric CUORE experiment [215] or, more realistically, of the next-to-next-generation ton-scale $0\nu \beta \beta$-decay experiments. The current best upper bound on the effective neutrino mass is $m_{\beta \beta }\le 160$ meV, which corresponds to $T_{1/2}^{0\nu \beta \beta }(^{136}\mathrm {Xe})\ge 1.6\times 10^{25}$ year at 90% C.L, as indicated by the EXO-200 experiment [216]. This bound will be improved within a not too far future. The GERDA “phase-II” experiment [217, 218] is expected to reach $T^{0\nu \beta \beta }_{1/2}(^{76}\mathrm{Ge}) \ge 2\times 10^{26}$ year, which corresponds to $m_{\beta \beta }\le 100$ meV. A bolometric CUORE experiment, using ${}^{130}\mathrm{Te}$ [215], is currently under construction and has an estimated sensitivity of about $T_{1/2}^{0\nu \beta \beta }(^{130}\mathrm {Te})\sim 10^{26}$ years, which corresponds to $m_{\beta \beta }\le 50$ meV. Furthermore, there are proposals for ton-scale next-to-next-generation $0\nu \beta \beta$ experiments with $^{136}$Xe [219, 220] and $^{76}$Ge [217, 221] claiming sensitivities over $T_{1/2}^{0\nu \beta \beta }\sim 10^{27}$ years, which corresponds to $m_{\beta \beta }\sim 12{\text {--}}30$ meV. For a recent review, see for example Ref. [222]. Consequently, as follows from Eq. (39), our model predicts $T_{1/2}^{0\nu \beta \beta }$ at the level of sensitivities of the next-generation or next-to-next-generation $0\nu \beta \beta$ experiments.

4 Quark masses and mixing

From Eq. (12) and taking into account that the VEV pattern of the ${SU}\left( 3\right) _{L}$ singlet scalar fields is described by Eq. (9), with the nonvanishing VEVs set to be equal to $\lambda \Lambda$ (being $\Lambda$ the cutoff of our model) as shown in Eq. (14), it follows that the SM quark mass matrices have the form

$$\begin{aligned} M_{U}= & {} \left( \begin{array}{ccc} a_{1}^{\left( U\right) }\lambda ^{8} &{} 0 &{} 0 \\ 0 &{} a_{2}^{\left( U\right) }\lambda ^{4} &{} a_{4}^{\left( U\right) }\lambda ^{2} \\ 0 &{} 0 &{} a_{3}^{\left( U\right) } \end{array} \right) \frac{v}{\sqrt{2}}, \nonumber \\ M_{D}= & {} \left( \begin{array}{ccc} a_{1}^{\left( D\right) }\lambda ^{7} &{} a_{4}^{\left( D\right) }\lambda ^{6} &{} a_{5}^{\left( D\right) }\lambda ^{6} \\ 0 &{} a_{2}^{\left( D\right) }\lambda ^{5} &{} 0 \\ 0 &{} 0 &{} a_{3}^{\left( D\right) }\lambda ^{3} \end{array} \right) \frac{v}{\sqrt{2}}, \end{aligned}$$

(40)

where $\lambda =0.225$ is one of the Wolfenstein parameters, $v=246$ GeV the scale of electroweak symmetry breaking and $a_{i}^{\left( U\right) }$ ($i=1,2,3,4$) and $a_{j}^{\left( D\right) }$ ($j=1,2,3,4$) are $\mathcal {O}(1)$ parameters. From the SM quark mass textures given above, it follows that the Cabibbo mixing as well as the mixing in the 1–3 planes emerges from the down-type quark sector, whereas the up-type quark sector generates the quark mixing angle in the 2–3 planes. Besides that, the low energy quark flavor data indicates that the CP-violating phase in the quark sector is associated with the quark mixing angle in the 1–3 planes, as follows from the standard parametrization of the quark mixing matrix. Consequently, in order to get quark mixing angles and a CP-violating phase consistent with the experimental data, we assume that all dimensionless parameters given in Eq. (40) are real, except for $a_{5}^{\left( D\right) }$, taken to be complex.

Furthermore, the exotic quark masses read

$$\begin{aligned} m_{T}= & {} y^{\left( T\right) }\frac{v_{\chi }}{\sqrt{2}},\quad m_{J^{1}}=y_{1}^{\left( J\right) }\frac{v_{\chi }}{\sqrt{2}}=\frac{ y_{1}^{\left( J\right) }}{y^{\left( T\right) }}m_{T},\nonumber \\ \quad m_{J^{2}}= & {} y_{2}^{\left( J\right) }\frac{v_{\chi }}{\sqrt{2}}=\frac{ y_{2}^{\left( J\right) }}{y^{\left( T\right) }}m_{T}. \end{aligned}$$

(41)

Since the charged fermion mass and quark mixing pattern arises from the breaking of the $Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ discrete group, and in order to simplify the analysis, we adopt a benchmark where we set $a_{4}^{\left( D\right) }=a_{1}^{\left( D\right) }$ as well as $a_{1}^{\left( U\right) }=a_{3}^{\left( U\right) }=1$ and $a_{3}^{\left( D\right) }=a_{2}^{\left( U\right) }$, motivated by naturalness arguments and by the relation $m_{c}\sim m_{b}$, respectively. Then we proceed to fit the six parameters $a_{2}^{\left( U\right) }$, $a_{4}^{\left( U\right) }$, $a_{1}^{\left( D\right) }$, $a_{2}^{\left( D\right) }$, $a_{5}^{\left( D\right) }$ and the phase $\tau$, to reproduce the 10 physical observables of the quark sector, i.e., the six quark masses, the three mixing angles, and the CP-violating phase. The obtained values for the quark masses, the three quark mixing angles, and the CP-violating phase $\delta$ in Table correspond to the best-fit values:

$$\begin{aligned} a_{2}^{\left( U\right) }\simeq & {} 1.43,\quad a_{4}^{\left( U\right) }\simeq 0.80,\quad a_{1}^{\left( D\right) }\simeq 0.58, \nonumber \\ \quad a_{2}^{\left( D\right) }\simeq & {} 0.57, \quad \left| a_{5}^{\left( D\right) }\right| \simeq 0.44,\quad \tau =68^{\circ }. \end{aligned}$$

(42)

Table 4 Model and experimental values of the quark masses and CKM parameters

The obtained quark masses, quark mixing angles and CP-violating phase are consistent with the experimental data. Let us note that despite the aforementioned simplifying assumptions that allow us to eliminate some of the free parameters, a good fit with the low energy quark flavor data is obtained, showing that our model is indeed capable of a very good fit to the experimental data of the physical observables for the quark sector. The obtained and experimental values for the physical observables of the quark sector are reported in Table 4. We use the experimental values of the quark masses at the $M_{Z}$ scale, from Ref. [213] (which are similar to those in [214]), whereas the experimental values of the CKM parameters are taken from Ref. [9]. We have numerically checked that a slight deviation from the best-fit values, keeps all the obtained SM quark masses, with the exception of the bottom quark mass, inside the $3\sigma$ experimentally allowed range. We checked that small variations around the best-fit values, keep most of the resulting values of the bottom quark mass inside the $3\sigma$ experimentally allowed range. The values outside the $3\sigma$ experimentally allowed range are close to the lower and upper experimental bounds of the bottom quark mass. Consequently, our model is very predictive for the quark sector.

On the other hand, from the SM quark textures, it follows that in order to obtain realistic SM quark masses and mixing angles without requiring a strong hierarchy among the Yukawa couplings, one should have $v_{\rho }\sim v_{\eta }$, which implies that $\tan \beta \sim \mathcal {O}(1)$. Furthermore, as the $h^{0}b\bar{b}$ coupling is proportional to $\frac{\sin \alpha }{\cos \beta }$, in order to get a $h^{0}b\bar{b}$ coupling close to the SM expectation, we have $\alpha \sim \beta \pm \frac{\pi }{2}$. In the following we briefly comment on the phenomenological implications of our model in the concerning to the flavor changing processes involving quarks. As previously mentioned, the different $Z_{3}$ charge assignments for SM and exotic right-handed quark fields imply the absence of mixing between them. The absence of mixings between the SM and exotic quarks will imply that the exotic fermions will not exhibit flavor changing decays into SM quarks and gauge (or Higgs) bosons. After being pair produced they will decay into the SM quarks and the intermediate states of heavy gauge bosons, which in turn decay into the pairs of the SM fermions; see e.g. [228]. The precise signature of the decays of the exotic quarks depends on details of the spectrum and other parameters of the model. The present lower bounds from the LHC on the masses of the $Z^{\prime }$ gauge bosons in the 3-3-1 models reach around 2.5 TeV [229]. One can translate these bounds on the order of magnitude of the scale $v_{\chi }$ of breaking of the ${SU}(3)_{C}\otimes {SU}\left( 3\right) _{L}\otimes U\left( 1\right) _{X}\otimes Z_{3}$ symmetry. These exotic quarks can be produced at the LHC via Drell–Yan processes mediated by charged gauge bosons, where the final states will include the exotic T quark with a SM down-type quark as well as any of the exotic $J^{1}$ or $J^{2}$ quarks with a SM up-type quark. It would be interesting to perform a detailed study of the exotic quark production at the LHC, the exotic quark decay modes and the flavor changing top quark decays. This is beyond the scope of this work and is left for future studies.

5 Conclusions

We have constructed an extension of the 3-3-1 model with $\beta =-\frac{1}{ \sqrt{3}}$, based on the extended ${SU}(3)_{C}\otimes {SU}(3)_{L}\otimes U(1)_{X}\otimes S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ symmetry. Our $S_{3}$ flavor 3-3-1 model, which at low energies reduces to the 3-3-1 model with right-handed neutrinos, where $\beta =-\frac{1}{\sqrt{3}}$, is in agreement with the current data on SM fermion masses and mixing. The $S_{3}$, $Z_{3}$, $Z_{3}^{\prime }$ and $Z_{8}$ discrete groups reduce the number of the model parameters. Specifically, the $Z_{3}$ and $Z_{8}$ symmetries determine the allowed entries of the charged lepton mass matrix. Furthermore, the $Z_{3}$ symmetry decouples the SM quarks from the exotic quarks. The $Z_{3}^{\prime }$ symmetry selects the allowed entries of the SM quark mass matrices. The $Z_{16}$ symmetry generates the hierarchy among charged fermion masses and quark mixing angles that yields the observed charged fermion mass and quark mixing pattern. We assumed that the ${SU}(3)_{L}$ scalar singlets having Yukawa interactions with the right-handed Majorana neutrinos acquire VEVs much smaller than the electroweak symmetry breaking scale, then providing very small masses to these Majorana neutrinos, and thus giving rise to an inverse seesaw mechanism of active neutrino masses. The smallness of the active neutrino masses is attributed to their scaling with inverse powers of the high energy cutoff $\Lambda$ as well as well as by their linear dependence on the very small VEVs of the ${SU}(3)_{L}$ singlets $\varphi _{j}$ ($j=1,2$), assumed to be of the same order of magnitude. We found for these VEVs the estimate $v_{\varphi }\sim 0.5$ GeV. The observed hierarchy of the SM charged fermion masses and quark mixing matrix elements arises from the breaking of the $Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ discrete group at very high energy. Furthermore, the model features a leptonic Dirac CP-violating phase close to $\frac{\pi }{2}$ and a Jarlskog invariant close to about $3\times 10^{-2}$ for both normal and inverted neutrino mass hierarchy. In addition, under the assumption that the exotic T, $J^{1}$ and $J^{2}$ quarks are lighter than the $H_{2}^{0}$ and $\overline{H}_{2}^{0}$ neutral Higgs bosons, our model predicts the absence of the flavor changing neutral exotic quark decays, which implies that they can be searched at the LHC via their decay into the SM quarks and the intermediate states of heavy gauge bosons, which in turn decay into the pairs of the SM fermions; see e.g. [228]. Possible directions for future work along these lines would be to study the constraints on the heavy charged gauge boson masses in our model arising from the upper bound on the branching fraction for the flavor changing top quark decays, the oblique parameters, the $Zb\overline{b}$ vertex and the Higgs diphoton signal strength. The heavy exotic quark production at the LHC may be useful to study. Finally we briefly comment on a possible large discrete symmetry group that could be used to embed the $S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ discrete symmetry of our model. Considering that the discrete group $\Delta \left( 6N^{2}\right)$ is isomorphic to$\ (Z_{N}\times Z_{N}^{\prime })\rtimes S_{3}$ [39] and the fact the $Z_{24}$ discrete group is the smallest cyclic group that contains both the $Z_{3}$ and $Z_{8}$ symmetries, it follows that the $S_{3}\otimes Z_{3}\otimes Z_{3}^{\prime }\otimes Z_{8}\otimes Z_{16}$ discrete group of our model can be embedded in the $\Delta \left( 6N^{2}\right) =\Delta$ $\left( 3456\right)$ discrete group (where $N=24$). It would be interesting to implement the $\Delta \left( 6N^{2}\right)$ discrete symmetry in the 331 model and to study its implications on fermion masses and mixings. All these studies require carefull investigations that we left outside the scope of this work.

Appendices

The product rules for $S_{3}$.

The $S_{3}$ group has three irreducible representations: $\mathbf {1}$, $\mathbf {1}^{\prime }$ and $\mathbf {2}$. Denoting the basis vectors for two $S_{3}$ doublets as $\left( x_{1},x_{2}\right) ^{T}$ and $\left( y_{1},y_{2}\right) ^{T}$ and $y^{\prime }$ a non-trivial $S_{3}$ singlet, the $S_{3}$ multiplication rules are [39]

$$\begin{aligned} \left( \begin{array}{c} x_{1} \\ x_{2} \end{array} \right) _{\mathbf {2}}\otimes \left( \begin{array}{c} y_{1} \\ y_{2} \end{array} \right) _{\mathbf {2}}= & {} \left( x_{1}y_{1}+x_{2}y_{2}\right) _{\mathbf {1} }+\left( x_{1}y_{2}-x_{2}y_{1}\right) _{\mathbf {1}^{\prime }}\nonumber \\&+\,\left( \begin{array}{c} x_{2}y_{2}-x_{1}y_{1} \\ x_{1}y_{2}+x_{2}y_{1} \end{array} \right) _{\mathbf {2}}, \end{aligned}$$

(A1)

$$\begin{aligned} \left( \begin{array}{c} x_{1} \\ x_{2} \end{array} \right) _{\mathbf {2}}\otimes \left( y^{\prime }\right) _{\mathbf {1}^{\prime }}= & {} \left( \begin{array}{c} -x_{2}y^{\prime } \\ x_{1}y^{\prime } \end{array} \right) _{\mathbf {2}},\quad \left( x^{\prime }\right) _{ \mathbf {1}^{\prime }}\otimes \left( y^{\prime }\right) _{\mathbf {1}^{\prime }}=\left( x^{\prime }y^{\prime }\right) _{\mathbf {1}}. \nonumber \\ \end{aligned}$$

(A2)

With these multiplication rules we have to assign to the scalar fields in the $S_{3}$ irreps and build the corresponding scalar potential invariant under the symmetry group.

Stability conditions of the low energy scalar potential.

In the following we are going to determine the conditions required to have a stable scalar potential by following the method described in Ref. [230]. The gauge invariant and renormalizable low energy scalar potential as a function of the fields $\phi _{1}=\chi$, $\phi _{2}=\rho$ and $\phi _{3}=\eta$ is a linear hermitian combination of the following terms:

$$\begin{aligned} \phi _{i}\phi _{j},\quad \phi _{i}\phi _{j}\phi _{k}\phi _{l} \end{aligned}$$

(B1)

where $i,j,k,l=\phi _{1}$, $\phi _{2}$ and $\phi _{3}$. To discuss the stability of the potential, its minimum, and its gauge invariance one can make the following arrangement of the scalar fields by using $2\times 2$ hermitian matrices:

$$\begin{aligned} \widetilde{K}_{(\phi _{i}\phi _{j})}= & {} \left( \begin{array}{cc} \phi _{i}^{\dagger }\phi _{i} &{} \phi _{i}^{\dagger }\phi _{j} \\ \phi _{j}^{\dagger }\phi _{i} &{} \phi _{j}^{\dagger }\phi _{j} \end{array} \right) , \nonumber \\= & {} \frac{1}{2}\left( K_{0\left( \phi _{i}\phi _{j}\right) }1_{2\times 2}+K_{a\left( \phi _{i}\phi _{j}\right) }\sigma ^{a}\right) \end{aligned}$$

(B2)

where $(\phi _{i}\phi _{j})=\rho \eta ,\rho \chi ,\eta \chi$, $\sigma ^{a}$ ($a=1,2,3$) are the Pauli matrices and $1_{2\times 2}$ is the identity matrix. From the previous expressions one can build the following bilinear terms as functions of the scalar fields:

$$\begin{aligned} K_{0\left( \phi _{i}\phi _{j}\right) }= & {} \phi _{i}^{\dagger }\phi _{i}+\phi _{j}^{\dagger }\phi _{j}, \nonumber \\ K_{a\left( \phi _{i}\phi _{j}\right) }= & {} \sum _{i,j}\left( \phi _{i}^{\dagger }\phi _{j}\right) \sigma _{ij}^{a}. \end{aligned}$$

(B3)

The properties of the potential can be analyzed in terms of $K_{0\left( \phi _{i}\phi _{j}\right) }$ and $\mathbf {K}_{\left( \phi _{i}\phi _{j}\right) }$ with $\phi _{i}\phi _{j}=\rho \eta ,\rho \chi ,\eta \chi$ in the domain $K_{0}\ge 0$ y $K_{0}^{2}\ge \mathbf {K}^{2}$. Defining $\mathbf {\kappa }=\mathbf {K}/K_{0}$ the potential can be written as

$$\begin{aligned} V= & {} V_{2}+V_{4}, \nonumber \\ V_{2}= & {} \sum _{(\phi _{i}\phi _{j})}K_{0(\phi _{i}\phi _{j})}\mathbf {J}_{2(\phi _{i}\phi _{j})}(\mathbf {\kappa }),\nonumber \\ \mathbf {J}_{2{(\phi _{i}\phi _{j})}}(\mathbf {\kappa })= & {} \xi _{0(\phi _{i}\phi _{j})}+\mathbf {\xi }_{(\phi _{i}\phi _{j})}^{T}\mathbf {\kappa }_{(\phi _{i}\phi _{j})}, \nonumber \\ V_{4}= & {} \sum _{(\phi _{i}\phi _{j})}K_{0(\phi _{i}\phi _{j})}^{2}\mathbf {J} _{4(\phi _{i}\phi _{j})}(\mathbf {\kappa }),\;\;\; \\ \mathbf {J}_{4{(\phi _{i}\phi _{j})}}(\mathbf {\kappa })= & {} \eta _{00(\phi _{i}\phi _{j})}+2\mathbf {\eta }_{(\phi _{i}\phi _{j})}^{T}\mathbf {\kappa }_{(\phi _{i}\phi _{j})} \nonumber \\&+\,\mathbf {\kappa }_{(\phi _{i}\phi _{j})}^{T}E_{(\phi _{i}\phi _{j})}\mathbf {\kappa }_{(\phi _{i}\phi _{j})}, \nonumber \end{aligned}$$

(B4)

where $E_{(\phi _{i}\phi _{j})}$ is a $3\times 3$ matrix and the functions $J_{2{(\phi _{i}\phi _{j})}}(\mathbf {\kappa })$ and $J_{4{(\phi _{i}\phi _{j})}}( \mathbf {\kappa })$ are defined in the domain $|\mathbf {\kappa }|\le 1$. The stability of the scalar potential requires that it has to be bounded from below. The stability is determined from the behavior of V in the limit $K_{0}\rightarrow \infty$, i.e.,

$$\begin{aligned} J_{4{(\phi _{i}\phi _{j})}}(\mathbf {\kappa })\ge 0, \end{aligned}$$

(B5)

for all $|\mathbf {\kappa }|\le 1$. To impose $J_{4{(\phi _{i}\phi _{j})}}(\mathbf { \kappa })$ to be positively defined it is enough to consider the values of all stationary points in the domain $|\kappa |<1$ and $|\kappa |=1$. This results in a bound for $\eta _{00{(\phi _{i}\phi _{j})}}$, $\mathbf {\eta }_{0{ (\phi _{i}\phi _{j})}}$ and $E{(\phi _{i}\phi _{j})}$, which parametrize the quartic terms of the potential included in $V_{4}$.

For $|\mathbf {\kappa }|<1$ the stationary points should satisfy

$$\begin{aligned} E\mathbf {\kappa }_{(\phi _{i}\phi _{j})}=-\mathbf {\eta }_{(\phi _{i}\phi _{j})},\quad |\mathbf {\kappa }|<1. \end{aligned}$$

(B6)

For the case where $\det E\ne 0$, the following relation is obtained:

$$\begin{aligned} \;J_{4(\phi _{i}\phi _{j})}(\mathbf {\kappa })|_{est}=\eta _{00{(\phi _{i}\phi _{j})}}-\mathbf {\eta }_{(\phi _{i}\phi _{j})}^{T}E^{-1}\mathbf {\eta }_{(\phi _{i}\phi _{j})}. \end{aligned}$$

(B7)

For $|\mathbf {\kappa }|=1$ the stationary points are obtained from the function:

$$\begin{aligned} F_{4(\phi _{i}\phi _{j})}(\mathbf {\kappa })=J_{4(\phi _{i}\phi _{j})}(\kappa )+u(1-\mathbf {\kappa }^{2}), \end{aligned}$$

(B8)

where u is a Lagrange multiplier that satisfies the following condition:

$$\begin{aligned} (E_{(\phi _{i}\phi _{j})}-u)\mathbf {\kappa }= & {} -\mathbf {\eta }_{(\phi _{i}\phi _{j})}, \nonumber \\ J_{4(\phi _{i}\phi _{j})}(\mathbf {\kappa })|_{est}= & {} u+\eta _{00{(\phi _{i}\phi _{j})}} \\&-\mathbf {\eta }_{(\phi _{i}\phi _{j})}^{T}(E_{(\phi _{i}\phi _{j})}-u)^{-1} \mathbf {\eta }_{(\phi _{i}\phi _{j})}. \nonumber \end{aligned}$$

(B9)

The stationary points of $J_{4(\phi _{i}\phi _{j})}(\kappa )$ for $|\kappa |\le 1$ can be obtained from

$$\begin{aligned} f_{(\phi _{i}\phi _{j})}(u)= & {} J_{4{(\phi _{i}\phi _{j})}}(\mathbf {\kappa } )|_{est}>0, \nonumber \\ f_{(\phi _{i}\phi _{j})}^{\prime }\left( u\right)> & {} 0. \end{aligned}$$

(B10)

Considering that the quartic terms of the scalar potential are dominant when the vacuum expectation values of the scalar fields take large values, these terms will be the most relevant to analyze the stability of the scalar potential. Following the method described in Ref. [230], we proceed to rewrite the quartic terms of the scalar potential in terms of bilinear combinations of the scalar fields. To this end, the bilinear combinations of the scalar fields are included in the following matrices:

$$\begin{aligned} \widetilde{K}_{\rho \eta }= & {} \left( \begin{array}{cc} \rho ^{\dagger }\rho &{} \eta ^{\dagger }\rho \\ \rho ^{\dagger }\eta &{} \eta ^{\dagger }\eta \end{array} \right) =\frac{1}{2}\left( K_{0\left( \rho \eta \right) }1_{2\times 2}+K_{a\left( \rho \eta \right) }\sigma ^{a}\right) , \nonumber \\ \widetilde{K}_{\rho \chi }= & {} \left( \begin{array}{cc} \rho ^{\dagger }\rho &{} \chi ^{\dagger }\rho \\ \rho ^{\dagger }\chi &{} \chi ^{\dagger }\chi \end{array} \right) =\frac{1}{2}\left( K_{0\left( \rho \chi \right) }1_{2\times 2}+K_{a\left( \rho \chi \right) }\sigma ^{a}\right) , \nonumber \\ \widetilde{K}_{\eta \chi }= & {} \left( \begin{array}{cc} \eta ^{\dagger }\eta &{} \chi ^{\dagger }\eta \\ \eta ^{\dagger }\chi &{} \chi ^{\dagger }\chi \end{array} \right) =\frac{1}{2}\left( K_{0\left( \eta \chi \right) }1_{2\times 2}+K_{a\left( \eta \chi \right) }\sigma ^{a}\right) , \nonumber \\&\end{aligned}$$

(B11)

where $\sigma ^{a}$ ($a=1,2,3$) are the Pauli matrices and $1_{2\times 2}$ is the $2\times 2$ identity matrix. From the previous expressions, we find that the bilinear combinations of the scalar fields appearing in Eq. (B11) are given by

$$\begin{aligned} K_{0\left( \rho \eta \right) }= & {} \rho ^{\dagger }\rho +\eta ^{\dagger }\eta ,\quad K_{0\left( \rho \chi \right) }=\rho ^{\dagger }\rho +\chi ^{\dagger }\chi ,\nonumber \\ K_{0\left( \eta \chi \right) }= & {} \eta ^{\dagger }\eta +\chi ^{\dagger }\chi , \\ K_{a\left( \rho \eta \right) }= & {} \left( \rho ^{\dagger }\rho \right) \sigma _{11}^{a}+\left( \eta ^{\dagger }\eta \right) \sigma _{22}^{a}+\left( \rho ^{\dagger }\eta \right) \sigma _{12}^{a}+\left( \eta ^{\dagger }\rho \right) \sigma _{21}^{a}, \nonumber \\ K_{a\left( \rho \chi \right) }= & {} \left( \rho ^{\dagger }\rho \right) \sigma _{11}^{a}+\left( \chi ^{\dagger }\chi \right) \sigma _{22}^{a}+\left( \rho ^{\dagger }\chi \right) \sigma _{12}^{a}+\left( \chi ^{\dagger }\rho \right) \sigma _{21}^{a}, \nonumber \\ K_{a\left( \eta \chi \right) }= & {} \left( \eta ^{\dagger }\eta \right) \sigma _{11}^{a}+\left( \chi ^{\dagger }\chi \right) \sigma _{22}^{a}+\left( \eta ^{\dagger }\chi \right) \sigma _{12}^{a}+\left( \chi ^{\dagger }\eta \right) \sigma _{21}^{a}. \nonumber \end{aligned}$$

(B12)

Since the stability of the scalar potential is determined from its quartic terms, the stationary solutions consistent with a stable scalar potential are described by the following functions:

$$\begin{aligned} f_{\rho \eta }\left( u\right)= & {} u+E_{00\left( \rho \eta \right) }-E_{a\left( \rho \eta \right) }\left( E_{\left( \rho \eta \right) }-u1_{3\times 3}\right) _{ab}^{-1}E_{b\left( \rho \eta \right) }, \nonumber \\ f_{\rho \chi }\left( u\right)= & {} u+E_{00\left( \rho \chi \right) }-E_{a\left( \rho \chi \right) }\left( E_{\left( \rho \chi \right) }-u1_{3\times 3}\right) _{ab}^{-1}E_{b\left( \rho \chi \right) }, \nonumber \\ f_{\eta \chi }\left( u\right)= & {} u+E_{00\left( \eta \chi \right) }-E_{a\left( \eta \chi \right) }\left( E_{\left( \eta \chi \right) }-u1_{3\times 3}\right) _{ab}^{-1}E_{b\left( \eta \chi \right) }, \nonumber \\&\end{aligned}$$

(B13)

where, for the $\rho$ and $\eta$ fields, we have

$$\begin{aligned} E_{00\left( \rho \eta \right) }= & {} \frac{\lambda _{2}+\lambda _{3}+\lambda _{6}}{4}, \nonumber \\ E_{a\left( \rho \eta \right) }= & {} \frac{\lambda _{2}-\lambda _{3}}{4}\delta _{a3}, \nonumber \\ E_{\left( \rho \eta \right) }= & {} \frac{1}{4}\left( \begin{array}{ccc} \lambda _{9} &{} 0 &{} 0 \\ 0 &{} \lambda _{9} &{} 0 \\ 0 &{} 0 &{} \lambda _{2}+\lambda _{3}-\lambda _{6} \end{array} \right) . \end{aligned}$$

(B14)

In the same manner, for the multiplets $\rho$ and $\chi$, the expressions are

$$\begin{aligned} E_{00\left( \rho \chi \right) }= & {} \frac{\lambda _{1}+\lambda _{2}+\lambda _{4}}{4}, \nonumber \\ E_{a\left( \rho \chi \right) }= & {} \frac{\lambda _{1}-\lambda _{2}}{4}\delta _{a3}, \nonumber \\ E_{\left( \rho \chi \right) }= & {} \frac{1}{4}\left( \begin{array}{ccc} \lambda _{8} &{} 0 &{} 0 \\ 0 &{} \lambda _{8} &{} 0 \\ 0 &{} 0 &{} \lambda _{1}+\lambda _{2}-\lambda _{4} \end{array} \right) . \end{aligned}$$

(B15)

Similarly, for the $\eta$ and $\chi$ fields, we find

$$\begin{aligned} E_{00\left( \eta \chi \right) }= & {} \frac{\lambda _{1}+\lambda _{3}+\lambda _{5}}{4},E_{a\left( \eta \chi \right) }=\frac{\lambda _{1}-\lambda _{3}}{4}\delta _{a3},\nonumber \\ E_{\left( \eta \chi \right) }= & {} \frac{1}{4}\left( \begin{array}{ccc} \lambda _{7} &{} 0 &{} 0 \\ 0 &{} \lambda _{7} &{} 0 \\ 0 &{} 0 &{} \lambda _{1}+\lambda _{3}-\lambda _{5} \end{array} \right) . \end{aligned}$$

(B16)

Following Ref. [230], we determine the stability of the scalar potential from the conditions:

$$\begin{aligned} f_{\rho \eta }\left( u\right)>0,\quad f_{\rho \chi }\left( u\right)>0,\quad f_{\eta \chi }\left( u\right) >0. \end{aligned}$$

(B17)

We use the theorem of stability of the scalar potential of Ref. [230] to determine the stability conditions of the scalar potential. To this end, the condition $f_{\rho \eta }\left( u\right) >0$ is analyzed for the set of values of u which include the 0, (since $f {\acute{}} _{\rho \eta }\left( 0\right) >0$) the roots $u_{\rho \eta }^{\left( 1\right) }$ and $u_{\rho \eta }^{\left( 2\right) }$ of the equation $f {\acute{}} _{\rho \eta }\left( u\right) =0$ and the eigenvalues $\widetilde{E}_{\left( \rho \eta \right) }^{\left( a\right) }$ of the matrix $E_{\left( \rho \eta \right) }$ where $f_{\rho \eta }( \widetilde{E}_{( \rho \eta ) }^{( a) })$ is finite and $f {\acute{}} _{\rho \eta }( \widetilde{E}_{( \rho \eta ) }^{( a) }) \ge 0$. We proceed in a similar way when analyzing the conditions $f_{\rho \chi }\left( u\right) >0$ and $f_{\eta \chi }\left( u\right) >0$.

Therefore, the scalar potential is stable when the following conditions are fulfilled:

$$\begin{aligned} \lambda _{1}> & {} 0,\quad \lambda _{2}>0,\quad \lambda _{3}>0,\nonumber \\ \lambda _{4}+\lambda _{8}\gtrless & {} 2\sqrt{\lambda _{1}\lambda _{2}},\quad \lambda _{4}+\lambda _{8}\gtrless \lambda _{1}+\lambda _{2},\nonumber \\&2\sqrt{\lambda _{1}\lambda _{2}}\gtrless \lambda _{4},\quad \lambda _{1}+\lambda _{2}\gtrless \lambda _{4},\nonumber \\ \lambda _{5}+\lambda _{7}\gtrless & {} 2\sqrt{\lambda _{1}\lambda _{3}},\quad \lambda _{5}+\lambda _{7}\gtrless \lambda _{1}+\lambda _{3},\nonumber \\&2\sqrt{\lambda _{1}\lambda _{3}}\gtrless \lambda _{5},\quad \lambda _{1}+\lambda _{3}\gtrless \lambda _{5},\nonumber \\ \lambda _{6}+\lambda _{9}\gtrless & {} 2\sqrt{\lambda _{2}\lambda _{3}},\quad \lambda _{6}+\lambda _{9}\gtrless \lambda _{2}+\lambda _{3},\nonumber \\&2\sqrt{\lambda _{2}\lambda _{3}}\gtrless \lambda _{6},\lambda _{2}+\lambda _{3}\gtrless \lambda _{6}. \end{aligned}$$

(B18)

Furthermore, having masses $m_{H_{1}^{\pm }}^{2}$, $m_{H_{1}^{0}}^{2}$ and $m_{H_{3}^{0}}^{2}$ positively defined requires the following condition:

$$\begin{aligned} f>0. \end{aligned}$$

(B19)