1 Introduction

Despite its great success, the standard model (SM) does not address several fundamental issues such as, for example, the number of fermion families and the observed pattern of fermion masses and mixing. As is well known, in the quark sector the mixing is small while in the lepton sector two of the mixing angles are large. The three neutrino flavors mix with each other and at least two of the neutrinos have non-vanishing masses, which according to neutrino oscillation experimental data must be smaller than the SM charged fermion masses by many orders of magnitude. This so called “flavor puzzle” motivates extensions of the SM with larger scalar and/or fermion sector and with extended gauge groups, supplemented with discrete flavor symmetries, so that the resulting fermion mass matrix textures would explain the observed pattern of fermion masses and mixing. The implementation of these discrete flavor symmetries in several extensions of the SM is expected to provide an elegant solution of the “flavor puzzle” (for recent reviews on discrete flavor groups see, for instance, Refs.  [1,2,3,4]). In fact, considerable attention has already been paid in the literature to the \(S_{3}\) [5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25] \(A_{4}\) [26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54], \(S_{4}\) [55,56,57,58,59,60,61,62,63], \(T_{7}\) [64,65,66,67,68,69,70,71,72] and \(\Delta (27)\) [73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91] flavor groups. Several models with discrete flavor symmetries, which bring about radiative seesaw mechanisms of fermion mass generation, have also been discussed in the literature [92,93,94,95,96,97,98,99]. A typical flaw of these models is the large number of free parameters and consequent limited predictive power, since they lack a mechanism underlying the observed hierarchy of quark mixing angles and masses. The experimental data suggest an empirical relation of the Cabbibo mixing angle to the first- and second-generation down-type quark masses \(\sin \theta _C\sim \sqrt{\frac{m_\mathrm{d} }{m_\mathrm{s}}}\) [100,101,102], which seems to imply a radiative seesaw mechanism of fermion mass generation, where the Cabbibo mixing arises from the down-type quark sector, whereas the up quark sector contributes to the remaining mixing angles. Inspired with this observation we propose an extension of the inert two Higgs doublet model (2HDM) realizing this idea. In our model a mismatch between the down- and up-type quark mass matrix textures is introduced, by distinguishing the two Higgs doublets with respect to the \( Z_{2}\) flavor symmetry, preserved at all scales. One of these doublets—the SM Higgs—is \(Z_{2}\)-even, while another one is \(Z_{2}\)-odd. In order to maintain the \(Z_{2}\) symmetry intact, we assume the latter to be VEV-less. The described setup is similar to the well-known inert 2HDM [103], but with the difference that here some of the SM fermions are \(Z_{2}\)-odd. Furthermore, compared to Ref. [103], the discrete symmetry in our model is extended from \(Z_{2}\) to \(S_{3}\otimes Z_{2}\otimes Z_{12}\), and the field content is enlarged by extra scalars, heavy charged exotic fermions and two heavy right-handed Majorana neutrinos. Note that only the \( Z_{2}\) is preserved at low energies, whereas the \(S_{3}\otimes Z_{12}\) symmetry is spontaneously broken. The latter allows us to explain the observed SM fermion mass and mixing pattern. In our modified inert 2HDM, the SM fermion mass and mixing pattern is due to a combination of tree and one-loop-level effects. At tree level only the third-generation charged fermions acquire masses and there is no quark mixing, while the first- and second-generation charged fermion masses and the quark mixing arise from one-loop-level radiative seesaw-type mechanism, triggered by virtual \(Z_{2}\)-charged scalar fields and electrically charged exotic fermions running inside the loops. Light active neutrino masses are generated from a one-loop-level radiative seesaw mechanism. Due to the preserved \(Z_{2}\) symmetry, our model features natural dark matter candidates.

The following comparison of our model with similar models in the literature could be in order. Despite the similar quality of the data description, our model is more predictive than the model of Ref. [104], since the latter, focused only on the quark sector, has a total of 12 free parameters, whereas the quark sector of our model is described by 9 free effective parameters, which are adjusted to reproduce the 10 physical observables of the quark sector. The models of Refs.  [21, 105, 106], Refs. [13, 15, 20, 63, 107], Refs. [76, 108] and Ref. [109] possess in the quark sector 9, 10, 12, and 13 free parameters, respectively. For the lepton sector, our model is less predictive than in the quark sector, however, under some reasonable assumptions, the number of effective parameters in this sector can be reduced. As shown in detail in Sect. 3, our model can successfully accommodate the eight physical observables in the lepton sector, with only five effective free parameters for the case of normal neutrino mass hierarchy. With respect to the inverted neutrino mass hierarchy, more parameters are needed to successfully reproduce the masses and leptonic mixing angles.

The content of this paper goes as follows. In Sect. 2 we introduce the model setup. Section 3 deals with the derivation of fermion masses and mixings and provides our corresponding results. Our conclusions are stated in Sect. 6. Appendix A gives a brief description of the \(S_{3}\) group. Appendix B shows the analytical expressions for the dimensionless parameters of the SM fermion mass matrices generated at one loop level.

2 The model

We consider a modified inert 2HDM, with the Standard Model gauge symmetry supplemented with the \(S_{3}\otimes Z_{2}\otimes Z_{12}\) discrete group, and a scalar sector composed of 2 scalar SM doublets, i.e., \(\phi _{1}\) and \( \phi _{2}\) plus 4 scalar SM singlets. Out of these four SM scalar singlets, two scalar fields are grouped in a \(S_{3}\) doublet, whereas the remaining ones are assigned to be one \(S_{3}\) trivial singlet and one \(S_{3}\) non-trivial singlet. The \(Z_{2}\) symmetry is assumed to be preserved whereas the \(S_{3}\otimes Z_{12}\) discrete group is broken at certain scale \(\Lambda _{\mathrm{int}}\). The fermion sector of the SM is extended to include four SM gauge singlet charged leptons \(E_{1L}\), \(E_{2L}\), \(E_{1R}\) and \(E_{2R}\), two right-handed neutrinos \(N_{1R}\), \(N_{2R}\) and four \(SU\left( 2\right) _{L}\) singlet heavy quarks \(T_{nL}\), \(T_{nR}\), \(B_{nL}\), \(B_{nR}\) (\(n=1,2\)). The reasons for these specific choices will be explained below. It is assumed that the heavy exotic \(T_{n}\) and \(B_{n}\) quarks have electric charges equal to \(\frac{2}{3}\) and \(-\frac{1}{3}\), respectively. The \(S_{3}\otimes Z_{2}\otimes Z_{12}\) assignments of the scalar fields are:

$$\begin{aligned}&\phi _{1} \sim \left( \mathbf {1},1,1\right) ,\quad \phi _{2}\sim \left( \mathbf {1,}-1,1\right) , \nonumber \\&\xi \sim \left( \mathbf {2},1,1\right) ,\quad \eta \sim \left( \mathbf {1},-1,1\right) ,\nonumber \\&\tau \sim \Big ( \mathbf {1}^{\prime },1,e^{-\frac{i\pi }{6}}\Big ). \end{aligned}$$
(1)

The requirement of unbroken \(Z_{2}\) symmetry implies that the \(Z_{2}\) charged scalar fields \(\phi _{2}\) and \(\eta \) do not acquire vacuum expectation values. The remaining scalar fields, i.e., \(\phi _{1}\), \(\xi \) and \(\tau \), which are neutral under the \(Z_{2}\) symmetry, have non-vanishing vacuum expectation values. Let us give a motivation for the above presented extension (1) of the scalar sector of the original inert 2HDM [103]. One \(Z_{2}\) odd SM singlet scalar \(\eta \) (apart from the \(Z_{2}\) odd scalar doublet \(\phi _{2}\)) is needed to implement the radiative seesaw mechanism of fermion mass generation. It yields a non-trivial quark mixing, provides masses for the first- and second-generation charged fermions and contributes to the light active neutrino masses. Furthermore, one \(Z_{12}\)-charged SM singlet scalar (\(\tau \)) is needed to generate the SM charged fermion mass and quark mixing pattern compatible with the observations. We also need an \(S_{3}\)-doublet scalar (\(\xi \)) to non-trivially couple it to the first- and second-generation right-handed SM down-type quarks, which we unify into an \(S_{3}\)-doublet.

We decompose the Higgs doublets \(\phi _{1}\) and \(\phi _{2}\) in the standard way as

$$\begin{aligned} \phi _{l}= & {} \left( \begin{array}{c} \varphi _{l}^{+} \\ \frac{1}{\sqrt{2}}\left( v_{l}+\mathop {\mathrm{Re}}\nolimits \phi _{l}^{0}+i\mathop {\mathrm{Im}}\nolimits \phi _{l}^{0}\right) \end{array} \right) , \nonumber \\ v_{l}= & {} v\delta _{l1},\quad l=1,2. \end{aligned}$$
(2)

Here v is a Higgs VEV breaking the electroweak symmetry.

The \(S_{3}\) symmetry is broken by the VEV of the \(S_{3}\) scalar doublet \(\xi \). We choose the VEV alignment

$$\begin{aligned} \left\langle \xi \right\rangle =v_{\xi }\left( 1,0\right) , \end{aligned}$$
(3)

compatible with the scalar potential minimization condition as demonstrated in Ref. [20]. In the literature there have been constructed several \(S_{3}\) flavor models with the similar VEV alignment of an \(S_{3}\) scalar doublet (see for instance Refs. [13, 15, 20, 21]).

The \(S_{3}\otimes Z_{2}\otimes Z_{12}\) assignments of the fermions of the model are as follows:

$$\begin{aligned}&q_{1L} \sim \left( \mathbf {1}^{\prime }\mathbf {,}1,1\right) ,\quad q_{2L}\sim \left( \mathbf {1},1,1\right) ,\quad q_{3L}\sim \left( \mathbf {1},1,1\right) ,\nonumber \\&u_{3R} \sim \left( \mathbf {1},1,1\right) ,\quad u_{1R}\sim \left( \mathbf {1}^{\prime },-1,-1\right) ,\nonumber \\&u_{2R}\sim \left( \mathbf {1},-1,e^{\frac{i\pi }{3}}\right) \nonumber \\&D_{R} =\left( d_{1R},d_{2R}\right) \sim \left( \mathbf {2},-1,e^{\frac{i\pi }{3}}\right) ,\quad d_{3R}\sim \left( \mathbf {1}^{\prime },1,i\right) , \nonumber \\&l_{1L} \sim \left( \mathbf {1},1,1\right) ,\quad l_{2L}\sim \left( \mathbf {1},1,1\right) ,\quad l_{3L}\sim \left( \mathbf {1},1,1\right) , \nonumber \\&l_{1R} \sim \left( \mathbf {1},-1,-1\right) ,\quad l_{2R}\sim \left( \mathbf {1}^{\prime },-1,i\right) ,\nonumber \\&\quad l_{3R}\sim \left( \mathbf {1} ^{\prime },1,i\right) , \nonumber \\&T_{1L} \sim \left( \mathbf {1},-1,1\right) ,\quad T_{1R}\sim \left( \mathbf {1}^{\prime },-1,1\right) ,\nonumber \\&\quad T_{2L}\sim \left( \mathbf {1},-1,1\right) ,\quad T_{2R}\sim \left( \mathbf {1},-1,1\right) , \nonumber \\&B_{1L} \sim \left( \mathbf {1},1,1\right) ,\quad B_{1R}\sim \left( \mathbf {1}^{\prime }\mathbf {,}-1,1\right) ,\nonumber \\&\quad B_{2L}\sim \left( \mathbf {1}^{\prime },1,1\right) ,\quad B_{2R}\sim \left( \mathbf {1,} -1,1\right) , \nonumber \\&E_{1L} \sim \left( \mathbf {1},-1,1\right) ,\quad E_{1R}\sim \left( \mathbf {1,-}1,1\right) ,\nonumber \\&\quad E_{2L}\sim \left( \mathbf {1},-1,1\right) ,\quad E_{2R}\sim \left( \mathbf {1,-}1,1\right) , \nonumber \\&N_{1R} \sim \left( \mathbf {1},-1,1\right) ,\quad N_{2R}\sim \left( \mathbf {1},-1,1\right) . \end{aligned}$$
(4)

Now it is timely to comment on the reasons for the introduction of the extended symmetry \(S_{3}\otimes Z_{2}\otimes Z_{12}\). The \(S_{3}\) symmetry reduces the number of parameters in the Yukawa sector of the model, improving its predictivity. The \(Z_{12}\) symmetry shapes the hierarchical structure of the fermion mass matrices that gives rise to the observed charged fermion mass and quark mixing pattern. The preserved \(Z_{2}\) symmetry selects the allowed entries in the quark mass matrices, so that the down-type quark sector contributes to the Cabbibo mixing, whereas the up-type quark sector contributes to the remaining mixing angles. The \(Z_{2}\) symmetry allows also the implementation of a radiative seesaw-type mechanism (induced by the \(Z_{2}\) charged scalar fields and the exotic charged fermions running in the internal lines of the loop), which gives rise to a non-trivial quark mixing and generates the masses for the first- and second-generation charged fermions. Besides that, due to the unbroken \(Z_{2}\) symmetry, light active neutrino masses receive contributions from a one loop radiative seesaw mechanism (induced by the \(\eta \) and \(\phi _{2}\) scalar fields and the heavy Majorana neutrinos \(N_{1R}\) and \(N_{2R}\) running in the internal lines of the loops). Let us note that the masses of the light active neutrinos are generated from a one-loop-level radiative seesaw mechanism. It is worth mentioning that in order to be compatible with the neutrino oscillation data, we need at least two light massive active neutrinos. Having only one right-handed Majorana neutrino, will lead to two massless active neutrinos, which is in clear contradiction with the experimental data on neutrino oscillation experiments. That is why we introduced in the model two massive right-handed neutrinos \(N_{1R}\), \(N_{2R}\) , which is the minimal number necessary for this purpose, both for normal and inverted neutrino mass hierarchy. For similar reasons we introduced in our model four \(SU\left( 2\right) _{L}\) singlet heavy quarks \(T_{nL}\), \( T_{nR}\), \(B_{nL}\), \(B_{nR}\) (\(n=1,2\)), necessary to avoid the appearance of massless charged SM fermions.

With the above particle content, the following quark and lepton Yukawa terms, invariant under the symmetries of the model, arise:

$$\begin{aligned} \mathcal {L}_{Y}^{q}= & {} m_{T_{1}}\overline{T}_{1L}T_{1R}+m_{T_{2}}\overline{T }_{2L}T_{2R}+y_{3}^{\left( u\right) }\overline{q}_{3L}\widetilde{\phi } _{1}u_{3R} \nonumber \\&+\,y_{1}^{\left( u\right) }\overline{q}_{1L}\widetilde{\phi } _{2}T_{1R}+x_{1}^{\left( u\right) }\overline{T}_{1L}\eta u_{3R} \nonumber \\&+\,y_{2}^{\left( u\right) }\overline{q}_{2L}\widetilde{\phi } _{2}T_{2R}+x_{2}^{\left( u\right) }\overline{T}_{2L}\eta u_{3R} \nonumber \\&+\,x_{3}^{\left( u\right) }\overline{T}_{1L}\eta u_{1R}\frac{\tau ^{6}}{ \Lambda ^{6}}+x_{4}^{\left( u\right) }\overline{T}_{2L}\eta u_{2R}\frac{\tau ^{2}}{\Lambda ^{2}} \nonumber \\&+\,m_{B_{1}}\overline{B}_{1L}B_{1R}+m_{B_{2}}\overline{B}_{2L}B_{2R}+y_{3}^{ \left( d\right) }\overline{q}_{3L}\phi _{1}d_{3R}\frac{\tau ^{3}}{\Lambda ^{3}} \nonumber \\&+\,x_{2}^{\left( d\right) }\overline{B}_{1L}\eta D_{R}\frac{\xi \tau ^{2}}{ \Lambda ^{3}}+x_{4}^{\left( d\right) }\overline{B}_{2L}\eta D_{R}\frac{\xi \tau ^{2}}{\Lambda ^{3}} \nonumber \\&+\,y_{1}^{\left( d\right) }\overline{q}_{1L}\phi _{2}B_{1R}+x_{1}^{\left( d\right) }\overline{B}_{1L}\eta D_{R}\frac{\xi \xi \tau ^{2}}{\Lambda ^{4}} \nonumber \\&+\,y_{2}^{\left( d\right) }\overline{q}_{2L}\phi _{2}B_{2R}+x_{3}^{\left( d\right) }\overline{B}_{2L}\eta D_{R}\frac{\xi \xi \tau ^{2}}{\Lambda ^{4}} +h.c., \end{aligned}$$
(5)
Fig. 1
figure 1

One loop Feynman diagrams contributions to the fermion mass matrices. Here \(n=1,2\), \(m=1,2\) and \(l=1,2\)

$$\begin{aligned} \mathcal {L}_{Y}^{l}= & {} m_{E_{1}}\overline{E}_{1L}E_{1R}+m_{E_{2}}\overline{E }_{2L}E_{2R}+y_{3}^{\left( l\right) }\overline{l}_{3L}\phi _{1}l_{3R}\frac{ \tau ^{3}}{\Lambda ^{3}} \nonumber \\&+\,x_{1}^{\left( l\right) }\overline{l}_{1L}\phi _{2}E_{1R}+y_{1}^{\left( l\right) }\overline{E}_{1L}\eta l_{1R}\frac{\tau ^{6}}{\Lambda ^{6}} \nonumber \\&+\,x_{2}^{\left( l\right) }\overline{l}_{2L}\phi _{2}E_{1R}+y_{2}^{\left( l\right) }\overline{E}_{1L}\eta l_{2R}\frac{\tau ^{3}}{\Lambda ^{3}} \nonumber \\&+\,x_{3}^{\left( l\right) }\overline{l}_{1L}\phi _{2}E_{2R}+y_{3}^{\left( l\right) }\overline{E}_{2L}\eta l_{1R}\frac{\tau ^{6}}{\Lambda ^{6}} \nonumber \\&+\,x_{4}^{\left( l\right) }\overline{l}_{2L}\phi _{2}E_{2R}+y_{4}^{\left( l\right) }\overline{E}_{2L}\eta l_{2R}\frac{\tau ^{3}}{\Lambda ^{3}} \nonumber \\&+\,y_{11}^{\left( \nu \right) }\overline{l}_{1L}\widetilde{\phi } _{2}N_{1R}+y_{21}^{\left( \nu \right) }\overline{l}_{2L}\widetilde{\phi } _{2}N_{1R} \nonumber \\&+\,y_{31}^{\left( \nu \right) }\overline{l}_{3L}\widetilde{\phi } _{2}N_{1R}+y_{12}^{\left( \nu \right) }\overline{l}_{1L}\widetilde{\phi } _{2}N_{2R} \nonumber \\&+\,y_{22}^{\left( \nu \right) }\overline{l}_{2L}\widetilde{\phi } _{2}N_{2R}+y_{32}^{\left( \nu \right) }\overline{l}_{3L}\widetilde{\phi } _{2}N_{2R} \nonumber \\&+\,m_{1N}\overline{N}_{1R}N_{1R}^{c}+m_{2N}\overline{N}_{2R}N_{2R}^{c}+h.c. \end{aligned}$$
(6)

Here we considered \(S_{3}\otimes Z_{2}\) soft breaking mass terms for the charged exotic fermions. In addition, we have neglected the mixings between the \(T_{1}\) (\(B_{1}\)) and \(T_{2}\) (\(B_{2}\)) exotic quarks as well as the mixings between the \(E_{1}\) and \(E_{2}\) charged exotic leptons, by considering these charged exotic fermions as physical eigenstates. After the spontaneous breaking of the electroweak and \(S_{3}\otimes Z_{12}\) discrete symmetries, these interactions generate at tree- and one-loop-levels the quark and lepton mass matrices. The one loop Feynman diagrams contributions to the fermion mass matrices are shown in Fig. 1.

The hierarchy of charged fermion masses and quark mixing angles arises in our model from the breaking of the \(Z_{12}\) discrete group. In order to relate the quark masses with the quark mixing parameters, the VEVs of the SM scalar singlets \(\xi \) and \(\tau \) are set as follows:

$$\begin{aligned} v_{\xi }\sim \Lambda _{\mathrm{int}}\sim v_{\tau }=\lambda \Lambda , \end{aligned}$$
(7)

where \(\lambda =0.225\) is one of the Wolfenstein parameters and \(\Lambda \) corresponds to the model cutoff.

From the quark Yukawa interactions it follows that the heavy exotic \(T_{n}\) ( \(B_{n}\)) quarks (\(n=1,2\)) will have a dominant decay mode into a SM up- (down-) type quark and a heavy CP even or CP odd neutral Higgs boson, which is identified as missing energy, due to the preserved \(Z_{2}\) symmetry. Furthermore, from the lepton Yukawa interactions it follows that the charged exotic leptons \(E_{n}\) (\(n=1,2\)) will decay dominantly into a SM charged lepton and a heavy CP even or CP odd neutral Higgs boson. The exotic \(T_{n}\) and \(B_{n}\) quarks are produced in pairs at the LHC via gluon fusion and the Drell–Yan mechanism, and the charged exotic leptons \(E_{n}\) (\(n=1,2\)) are also produced in pairs but only via the Drell–Yan mechanism. Thus, observing an excess of events with respect to the SM background in the dijet and opposite sign dileptons final states can be a signal in support of this model at the LHC. On the other hand, it is worth mentioning that a heavy CP even Higgs can be produced at the LHC in association with a CP odd Higgs boson via Drell–Yan annihilation, with a cross section of about 0.1 fb for heavy CP even and CP odd scalar masses of 600 GeV and LHC center of mass energy of \(\sqrt{s}=13\) TeV. To conclude this section, let us comment on the \(h\rightarrow \gamma \gamma \) decay in our model, where h is the 126 GeV Higgs boson. In the standard model, this decay is dominated by W loop diagrams, which can interfere destructively with the subdominant top quark loop; whereas In the 2HDM, the \(h\rightarrow \gamma \gamma \) decay receives additional contributions from loops with charged scalars \(H^{\pm }\), proportional to \(\sin 2\beta \) (for the case where the Higgs doublets have different \(Z_{N}\) charges), where \(\tan \beta =v_{2}/v_{1}\) (cf. Ref. [20]). In our model \(\tan \beta =0\) since \(v_{2}=0\), and thus the charged Higgs boson contribution to the 126 GeV Higgs diphoton decay is absent. From the explicit form of the \(h\rightarrow \gamma \gamma \) decay rate given in [20] for the 2HDM, it is easy to see that the Higgs diphoton decay rate in our model coincides with the SM expectation, since the light and heavy CP even neutral Higgs bosons do not mix, as a result of the preserved \(Z_{2}\) symmetry.

3 Quark masses and mixings

From the quark sector Yukawa terms (5) we find the quark mass matrices

$$\begin{aligned} M_{U}= & {} \left( \begin{array}{ccc} \varepsilon _{11}^{\left( u\right) }\lambda ^{6} &{} 0 &{} \varepsilon _{13}^{\left( u\right) } \\ 0 &{} \varepsilon _{22}^{\left( u\right) }\lambda ^{2} &{} \varepsilon _{23}^{\left( u\right) } \\ 0 &{} 0 &{} y_{3}^{\left( u\right) } \end{array} \right) \frac{v}{\sqrt{2}}, \end{aligned}$$
(8)
$$\begin{aligned} M_{D}= & {} \left( \begin{array}{ccc} \varepsilon _{11}^{\left( d\right) }\lambda ^{4} &{} \varepsilon _{12}^{\left( d\right) }\lambda ^{3} &{} 0 \\ \varepsilon _{21}^{\left( d\right) }\lambda ^{4} &{} \varepsilon _{22}^{\left( d\right) }\lambda ^{3} &{} 0 \\ 0 &{} 0 &{} y_{3}^{\left( d\right) }\lambda ^{3} \end{array} \right) \frac{v}{\sqrt{2}}, \end{aligned}$$
(9)

where \(\varepsilon _{ij}^{\left( u,d\right) }\) are dimensionless parameters generated at one loop level whose corresponding expressions are given in Appendix B. In addition, \(y_{3}^{\left( u\right) }\) and \(y_{3}^{\left( d\right) }\) are \(\mathcal {O}(1)\) dimensionless couplings generated at tree level from renormalizable and nonrenormalizable Yukawa terms, respectively.

Fig. 2
figure 2

Charm and strange quark masses randomly generated. The horizonal lines are the minimum and maximum values of the SM quark masses inside the \(3 \sigma \) experimentally allowed range

Fig. 3
figure 3

Correlations between the first- and second-generation SM quark masses. The horizonal and vertical lines are the minimum and maximum values of the second- and first-generation quark masses, respectively, inside the \(3\sigma \) experimentally allowed range

In order to show that the quark textures given above can fit the experimental data, and considering that the parameters \(\varepsilon _{ij}^{\left( u,d\right) }\) are generated at one loop level, we choose a benchmark scenario where we set:

$$\begin{aligned} \varepsilon _{11}^{\left( u\right) }= & {} a_{11}^{\left( u\right) }\lambda ^{2},\quad \varepsilon _{22}^{\left( u\right) }=a_{22}^{\left( u\right) }\lambda ^{2},\nonumber \\ \varepsilon _{13}^{\left( u\right) }= & {} a_{13}^{\left( u\right) }\lambda ^{3},\quad \varepsilon _{23}^{\left( u\right) }=a_{23}^{\left( u\right) }\lambda ^{2}, \nonumber \\ \varepsilon _{11}^{\left( d\right) }= & {} a_{11}^{\left( d\right) }\lambda ^{3},\quad \varepsilon _{21}^{\left( d\right) }=a_{21}^{\left( d\right) }\lambda ^{2},\nonumber \\ \varepsilon _{12}^{\left( d\right) }= & {} a_{12}^{\left( d\right) }\lambda ^{3},\quad \varepsilon _{22}^{\left( d\right) }=a_{22}^{\left( d\right) }\lambda ^{2}, \end{aligned}$$
(10)

where \(a_{nn}^{\left( u\right) }\), \(a_{n3}^{\left( u\right) }\), \(a_{mn}^{\left( d\right) }\) (\(m,n=1,2\)) are \(\mathcal {O}(1)\) parameters. Let us note that from the quark mass matrices given above, it follows that the Cabbibo mixing arises from the down-type quark sector, whereas the up-type quark sector contributes to the remaining mixing angles. Besides that, the low energy quark flavor data indicates that the CP violating phase in the quark sector is associated with the quark mixing angle in the 1-3 plane, as follows from the Standard parametrization of the quark mixing matrix. Consequently, in order to get quark mixing angles and a CP violating phase consistent with the experimental data, we assume that all dimensionless parameters given in Eqs. (8) and (9) are real, except \(a_{13}^{\left( u\right) }\), which is taken to be complex. Since the observed pattern of charged fermion masses and quark mixing angles is generated from the \(S_{3}\otimes Z_{12}\) symmetry breaking, and in order to have the right value of the Cabbibo mixing, we need \(a_{21}^{\left( d\right) }\approx a_{22}^{\left( d\right) }\). In addition we set \(y_{3}^{\left( u\right) }\approx 1\), as suggested by naturalness arguments. Then the quark sector of our model contains nine effective free parameters, i.e., \(a_{11}^{\left( u\right) }\), \(a_{22}^{\left( u\right) }\), \(|a_{13}^{\left( u\right) }|\), \(a_{23}^{\left( u\right) }\), \(a_{11}^{\left( d\right) }\), \(a_{12}^{\left( d\right) }\), \(a_{22}^{\left( d\right) }\), \(y_{3}^{\left( d\right) }\) and the phase \(\arg \left( a^{\left( u\right) }_{13}\right) \), which are fitted to reproduce the ten physical observables of the quark sector, i.e., the six quark masses, the three mixing angles and the CP violating phase. By varying these parameters, we find the quark masses, the three quark mixing angles, and the CP violating phase \(\delta \) reported in Table 1, which correspond to the best-fit values:

$$\begin{aligned} a_{23}^{\left( u\right) }\simeq & {} 0.81,\quad |a_{13}^{\left( u\right) }|\simeq 0.3,\quad \arg \left( a^{\left( u\right) }_{13}\right) =-113^{\circ },\nonumber \\ a_{22}^{\left( u\right) }\simeq & {} 1.43,\quad a_{11}^{\left( u\right) }\simeq 1.27, \nonumber \\ a_{11}^{\left( d\right) }\simeq & {} 0.84,\quad a_{12}^{\left( d\right) }\simeq 0.4,\quad a_{22}^{\left( d\right) }\simeq 0.57,\nonumber \\ y_{3}^{\left( d\right) }\simeq & {} 1.42. \end{aligned}$$
(11)
Table 1 Model and experimental [110] values of the quark masses and CKM parameters

It is worth mentioning, as follows from Eq. (B1) given in Appendix B, that the functions \(\varepsilon _{11}^{\left( u\right) }\), \(\varepsilon _{22}^{\left( u\right) }\), \(\varepsilon _{13}^{\left( u\right) }\) and \(\varepsilon _{23}^{\left( u\right) }\) depend on the dimensionless parameters \(x_{1}^{\left( u\right) }\), \(x_{2}^{\left( u\right) }\), \(x_{3}^{\left( u\right) }\), \(x_{4}^{\left( u\right) }\), \(y_{1}^{\left( u\right) }\), \(y_{2}^{\left( u\right) }\), on the masses \(m_{T_{1}}\), \( m_{T_{2}}\), \(m_{\mathop {\mathrm{Re}}\nolimits \phi _{2}^{0}}\), \(m_{\mathop {\mathrm{Re}}\nolimits \eta }\), \(m_{\mathop {\mathrm{Im}}\nolimits \phi _{2}^{0}}\) and \(m_{\mathop {\mathrm{Im}}\nolimits \eta }\) as well as on the trilinear scalar coupling \(C_{\phi _{2}\phi _{1}\eta }\). Furthermore, the functions \( \varepsilon _{11}^{\left( d\right) }\), \(\varepsilon _{22}^{\left( d\right) }\), \(\varepsilon _{12}^{\left( d\right) }\) and \(\varepsilon _{21}^{\left( d\right) }\) depend on the dimensionless parameters \(x_{1}^{\left( d\right) }\), \(x_{2}^{\left( d\right) }\), \(x_{3}^{\left( d\right) }\), \(x_{4}^{\left( d\right) }\), \(y_{1}^{\left( d\right) }\), \(y_{2}^{\left( d\right) }\), on the masses \(m_{B_{1}}\), \(m_{B_{2}}\), \(m_{\mathop {\mathrm{Re}}\nolimits \phi _{2}^{0}}\), \(m_{\mathop {\mathrm{Re}}\nolimits \eta }\), \(m_{\mathop {\mathrm{Im}}\nolimits \phi _{2}^{0}}\) and \(m_{\mathop {\mathrm{Im}}\nolimits \eta }\) as well as on the trilinear scalar coupling \(C_{\phi _{2}\phi _{1}\eta }\). Consequently, there is a good amount of parametric freedom to reproduce the obtained values for the \(\varepsilon _{ij}^{\left( u,d\right) }\) (\(i,j=1,2,3\)) functions that successfully reproduce the physical observables of the quark sector. For instance, the best-fit values given above imply that \( \varepsilon _{11}^{\left( u\right) }\simeq 6.4\times 10^{-2}\), which can be obtained by setting \(m_{T_{1}}=1\) TeV, \(m_{\mathop {\mathrm{Re}}\nolimits \phi _{2}^{0}}=420\) GeV, \(m_{\mathop {\mathrm{Re}}\nolimits \eta }=560\) GeV, \(m_{\mathop {\mathrm{Im}}\nolimits \phi _{2}^{0}}=600\) GeV, \(m_{ \mathop {\mathrm{Im}}\nolimits \eta }=800\) GeV, \(C_{\phi _{2}\phi _{1}\eta }=3\) TeV, \( x_{3}^{\left( u\right) }=y_{1}^{\left( u\right) }=3\). Similar considerations apply for the down-type quark and lepton sector.

In order to study the sensitivity of the obtained values for the SM quark masses under small variations around the best-fit values (maximum variation of \(+0.2\), minimum of \(-0.2\)), we show in Fig. 2 the predicted charm and strange masses as functions of the iteration. We find that, for a slight deviation from the best-fit values, the obtained charm and strange masses stay inside the \(3\sigma \) experimentally allowed range. We also have numerically checked that the remaining SM quark masses are kept inside the \(3\sigma \) experimentally allowed limits when we perform small variations around the best-fit values, with the exception of the down and bottom quark masses. The sensitivity of the down-type quark mass under small variations around the best-fit values is due to the fact that all entries in the upper left block of the SM down-type quark mass matrix are different from zero, thus implying that there is an important region of parameter space where the determinant of the down-type quark mass matrix takes low values. With respect to the bottom quark mass, we find that most of the points are inside the \(3\sigma \) experimentally allowed range. Those outside the \(3\sigma \) experimentally allowed range correspond to values close to the lower and upper experimental bounds of the bottom quark mass. Consequently, our model is very predictive for the quark sector. Correlations between the first- and second-generation SM quark masses are shown in Fig. 3. The horizontal and vertical lines are the minimum and maximum values of the second- and first-generation quark masses, respectively, inside the \(3\sigma \) experimentally allowed range.

4 Lepton masses and mixings

From the lepton Yukawa terms (6) we derive the neutrino mass matrices

$$\begin{aligned} M_{l}= & {} \left( \begin{array}{ccc} \varepsilon _{11}^{\left( l\right) }\lambda ^{6} &{} \varepsilon _{12}^{\left( l\right) }\lambda ^{3} &{} 0 \\ \varepsilon _{21}^{\left( l\right) }\lambda ^{6} &{} \varepsilon _{22}^{\left( l\right) }\lambda ^{3} &{} 0 \\ 0 &{} 0 &{} y_{3}^{\left( l\right) }\lambda ^{3} \end{array} \right) \frac{v}{\sqrt{2}}, \end{aligned}$$
(12)
$$\begin{aligned} M_{\nu }= & {} \left( \begin{array}{ccc} W_{1}^{2} &{} W_{1}W_{2}\cos \varphi &{} W_{1}W_{3}\cos \left( \varphi -\varrho \right) \\ W_{1}W_{2}\cos \varphi &{} W_{2}^{2} &{} W_{2}W_{3}\cos \varrho \\ W_{1}W_{3}\cos \left( \varphi -\varrho \right) &{} W_{2}W_{3}\cos \varrho &{} W_{3}^{2} \end{array} \right) , \nonumber \\ \end{aligned}$$
(13)

where \(\varepsilon _{nm}^{\left( l\right) }\) (\(n,m=1,2\)) are dimensionless parameters generated at one loop level whose corresponding expressions are given in Appendix B. The parameters of the neutrino mass matrix of Eq. (13) take the form

$$\begin{aligned}&\overrightarrow{W_{j}} =\left( \frac{A_{j1}\sqrt{m_{N_{1}}f_{1}^{\left( \nu \right) }}}{64\pi ^{3}\Lambda },\frac{A_{j2}\sqrt{m_{N_{2}}f_{2}^{\left( \nu \right) }}}{64\pi ^{3}\Lambda }\right) ,\quad j=1,2,3 \nonumber \\&\cos \varphi =\frac{\overrightarrow{W_{1}}\cdot \overrightarrow{W_{2}}}{ \left| \overrightarrow{W_{1}}\right| \left| \overrightarrow{W_{2} }\right| },\quad \cos \left( \varphi -\varrho \right) =\frac{ \overrightarrow{W}_{1}\cdot \overrightarrow{W_{3}}}{\left| \overrightarrow{W}_{1}\right| \left| \overrightarrow{W_{3}} \right| },\quad W_{j}=\left| \overrightarrow{W_{j}} \right| , \nonumber \\&\cos \varrho =\frac{\overrightarrow{W_{2}}\cdot \overrightarrow{W_{3}}}{ \left| \overrightarrow{W_{2}}\right| \left| \overrightarrow{W_{3} }\right| },\quad A_{js}=y_{js}^{\left( \nu \right) }\frac{v}{ \sqrt{2}}\quad s=1,2, \nonumber \\&f_{s} =\frac{m_{R}^{2}}{m_{R}^{2}-m_{N_{s}}^{2}}\ln \left( \frac{m_{R}^{2} }{m_{N_{s}}^{2}}\right) -\frac{m_{I}^{2}}{m_{I}^{2}-m_{N_{s}}^{2}}\ln \left( \frac{m_{I}^{2}}{m_{N_{s}}^{2}}\right) , \nonumber \\&m_{R} =m_{\mathop {\mathrm{Re}}\nolimits \phi _{2}^{0}},\quad m_{I}=m_{\mathop {\mathrm{Im}}\nolimits \phi _{2}^{0}}. \end{aligned}$$
(14)

In order to show that the lepton textures given above can fit the experimental data, and considering that the parameters \(\varepsilon _{ij}^{\left( l\right) }\) are generated at one loop level, we choose a benchmark scenario where we set:

$$\begin{aligned} \varepsilon _{11}^{\left( l\right) }= & {} a_{11}^{\left( l\right) }\lambda ^{2}, \quad \varepsilon _{21}^{\left( l\right) }=a_{21}^{\left( l\right) }\lambda ^{2},\nonumber \\ \varepsilon _{12}^{\left( l\right) }= & {} a_{12}^{\left( l\right) }\lambda ^{2},\quad \varepsilon _{22}^{\left( l\right) }=a_{22}^{\left( l\right) }\lambda ^{2}, \end{aligned}$$
(15)

where \(a_{nm}^{\left( l\right) }\) (\(n,m=1,2,3\)) are \(\mathcal {O}(1)\) parameters.

Then it follows that the charged lepton mass matrix takes the following form:

$$\begin{aligned} M_{l}=\frac{v}{\sqrt{2}}\left( \begin{array}{ccc} a_{11}^{\left( l\right) }\lambda ^{8} &{} a_{12}^{\left( l\right) }\lambda ^{5} &{} 0 \\ a_{21}^{\left( l\right) }\lambda ^{8} &{} a_{22}^{\left( l\right) }\lambda ^{5} &{} 0 \\ 0 &{} 0 &{} y_{3}^{\left( l\right) }\lambda ^{3} \end{array} \right) , \end{aligned}$$
(16)

which implies that the following relation is fulfilled:

$$\begin{aligned} M_{l}M_{l}^{T}=\frac{v^{2}}{2}\left( \begin{array}{ccc} \left( a_{12}^{\left( l\right) }\right) ^{2}\lambda ^{10}+\left( a_{11}^{\left( l\right) }\right) ^{2}\lambda ^{16} &{} a_{12}^{\left( l\right) }a_{22}^{\left( l\right) }\lambda ^{10}+a_{11}^{\left( l\right) }a_{21}^{\left( l\right) }\lambda ^{16} &{} 0 \\ a_{12}^{\left( l\right) }a_{22}^{\left( l\right) }\lambda ^{10}+a_{11}^{\left( l\right) }a_{21}^{\left( l\right) }\lambda ^{16} &{} \left( a_{22}^{\left( l\right) }\right) ^{2}\lambda ^{10}+\left( a_{21}^{\left( l\right) }\right) ^{2}\lambda ^{16} &{} 0\\ 0 &{} 0 &{} \left( y_{3}^{\left( l\right) }\right) ^{2}\lambda ^{6} \end{array} \right) . \end{aligned}$$
(17)

The mass matrix \(M_{l}M_{l}^{T}\) can be diagonalized by a rotation matrix \( R_{l}\) according to

$$\begin{aligned} R_{l}^{T}M_{l}M_{l}^{T}R_{l}= & {} \left( \begin{array}{ccc} m_{e}^{2} &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad m_{\mu }^{2} &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad m_{\tau }^{2} \end{array} \right) ,\nonumber \\ R_{l}= & {} \left( \begin{array}{ccc} \cos \theta _{l} &{}\quad \sin \theta _{l} &{} 0 \\ -\sin \theta _{l} &{}\quad \cos \theta _{l} &{} 0 \\ 0 &{}\quad 0 &{}\quad 1 \end{array} \right) ,\quad \tan \theta _{l}\simeq \frac{ a_{12}^{\left( l\right) }}{a_{22}^{\left( l\right) }}, \nonumber \\ \end{aligned}$$
(18)

where the charged lepton masses are given by

$$\begin{aligned} m_{e}\simeq & {} \lambda ^{8}\cdot \frac{\left| a_{12}^{\left( l\right) } a_{21}^{\left( l\right) } - a_{11}^{\left( l\right) } a_{22}^{\left( l\right) }\right| }{\sqrt{\left( a_{12}^{\left( l\right) }\right) ^{2}+\left( a_{22}^{\left( l\right) }\right) ^{2}}} \frac{v}{\sqrt{2}}, \end{aligned}$$
(19)
$$\begin{aligned} m_{\mu }\simeq & {} \lambda ^{5}\cdot \sqrt{\left( a_{12}^{\left( l\right) }\right) ^{2}+\left( a_{22}^{\left( l\right) }\right) ^{2}} \frac{v}{\sqrt{2}}, \end{aligned}$$
(20)
$$\begin{aligned} m_{\tau }= & {} \lambda ^{3}\cdot y_{3}^{\left( l\right) }\frac{v}{\sqrt{2}}. \end{aligned}$$
(21)

Thus we correctly reproduce the charged lepton mass hierarchy from the symmetry structure of the model.

To simplify further the analysis, we set \(\varphi =\varrho \), obtaining that the light neutrino mass matrix is given by

$$\begin{aligned} M_{\nu }=\left( \begin{array}{ccc} W_{1}^{2} &{} \kappa W_{1}W_{2} &{} W_{1}W_{3} \\ \kappa W_{1}W_{2} &{} W_{2}^{2} &{} \kappa W_{2}W_{3} \\ W_{1}W_{3} &{} \kappa W_{2}W_{3} &{} W_{3}^{2} \end{array} \right) ,\quad \kappa =\cos \varphi . \end{aligned}$$
(22)

Assuming that the neutrino Yukawa couplings are real, we find that for the normal (NH) and inverted (IH) mass hierarchies, the light neutrino mass matrix is diagonalized by a rotation matrix \(R_{\nu }\), according to

$$\begin{aligned}&R_{\nu }^{T}M_{\nu }R_{\nu } =\left( \begin{array}{ccc} 0 &{} 0 &{} 0 \\ 0 &{} m_{\nu _{2}} &{} 0 \\ 0 &{} 0 &{} m_{\nu _{3}} \end{array} \right) , \nonumber \\&R_{\nu }=\left( \begin{array}{ccc} -\frac{W_{3}}{\sqrt{W_{1}^{2}+W_{3}^{2}}} &{} \frac{W_{1}}{\sqrt{ W_{1}^{2}+W_{3}^{2}}}\sin \theta _{\nu } &{} \frac{W_{1}}{\sqrt{ W_{1}^{2}+W_{3}^{2}}}\cos \theta _{\nu } \\ 0 &{} \cos \theta _{\nu } &{} -\sin \theta _{\nu } \\ \frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}}} &{} \frac{W_{3}}{\sqrt{ W_{1}^{2}+W_{3}^{2}}}\sin \theta _{\nu } &{} \frac{W_{3}}{\sqrt{ W_{1}^{2}+W_{3}^{2}}}\cos \theta _{\nu } \end{array}\right) ,\text{ for } \text{ NH, } \nonumber \\ \end{aligned}$$
(23)
$$\begin{aligned}&\tan \theta _{\nu } =-\sqrt{\frac{W_{2}^{2}-m_{\nu _{2}}}{m_{\nu _{3}}-W_{2}^{2}}},\end{aligned}$$
(24)
$$\begin{aligned}&m_{\nu _{1}}=0,\quad m_{\nu _{2,3}}=\frac{W_{1}^{2}+W_{2}^{2}+W_{3}^{2}}{2}\nonumber \\&\qquad \qquad \mp \frac{\sqrt{\left( W_{1}^{2}-W_{2}^{2}+W_{3}^{2}\right) ^{2}+4\kappa ^{2}W_{2}^{2}\left( W_{1}^{2}+W_{3}^{2}\right) }}{2},\end{aligned}$$
(25)
$$\begin{aligned}&R_{\nu }^{T}M_{\nu }R_{\nu } =\left( \begin{array}{ccc} m_{\nu _{1}} &{} 0 &{} 0 \\ 0 &{} m_{\nu _{2}} &{} 0 \\ 0 &{} 0 &{} 0 \end{array} \right) ,\nonumber \\&R_{\nu }=\left( \begin{array}{ccc} \frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\sin \theta _{\nu } &{} \frac{W_{1}}{ \sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{\nu } &{} -\frac{W_{3}}{\sqrt{W_{1}^{2}+W_{3}^{2}}} \\ \cos \theta _{\nu } &{} -\sin \theta _{\nu } &{} 0 \\ \frac{W_{3}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\sin \theta _{\nu } &{} \frac{W_{3}}{ \sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{\nu } &{} \frac{W_{1}}{\sqrt{ W_{1}^{2}+W_{3}^{2}}} \end{array} \right) ,\quad \text{ for } \text{ IH }\nonumber \\ \end{aligned}$$
(26)
$$\begin{aligned}&\tan \theta _{\nu } =-\sqrt{\frac{W_{2}^{2}-m_{\nu _{1}}}{m_{\nu _{2}}-W_{2}^{2}}}\end{aligned}$$
(27)
$$\begin{aligned}&m_{\nu _{1,2}}=\frac{W_{1}^{2}+W_{2}^{2}+W_{3}^{2}}{2}\nonumber \\&\qquad \qquad \mp \frac{1}{2}\sqrt{\left( W_{1}^{2}\!-\!W_{2}^{2}\!+\!W_{3}^{2}\right) ^{2}\!+\!4\kappa ^{2}W_{2}^{2} \left( W_{1}^{2}\!+\!W_{3}^{2}\right) },\quad m_{\nu _{3}}=0.\nonumber \\ \end{aligned}$$
(28)

With the rotation matrices in the charged lepton sector \(R_{l}\), given by Eq. (18), and in the neutrino sector \(R_{\nu }\), given by Eqs. (23) and (26) for NH and IH, respectively, we find the PMNS mixing matrix:

$$\begin{aligned} U= & {} R_{l}^{T}R_{\nu }, =\left\{ \begin{array}{l} \left( \begin{array}{ccc} -\frac{W_{3}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{l} &{} \frac{W_{1}}{ \sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{l}\sin \theta _{\nu }-\cos \theta _{\nu }\sin \theta _{l} &{} \sin \theta _{l}\sin \theta _{\nu }+\frac{W_{1}}{ \sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{l}\cos \theta _{\nu } \\ -\frac{W_{3}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\sin \theta _{l} &{} \cos \theta _{l}\cos \theta _{\nu }+\frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\sin \theta _{\nu }\sin \theta _{l} &{} \frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\sin \theta _{l}\cos \theta _{\nu }-\cos \theta _{l}\sin \theta _{\nu } \\ \frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}}} &{} \frac{W_{3}}{\sqrt{ W_{1}^{2}+W_{3}^{2}}}\sin \theta _{\nu } &{} \frac{W_{3}}{\sqrt{ W_{1}^{2}+W_{3}^{2}}}\cos \theta _{\nu } \end{array} \right) \quad \text{ for } \text{ NH },\\ \left( \begin{array}{ccc} \frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{l}\sin \theta _{\nu }-\cos \theta _{\nu }\sin \theta _{l} &{} \sin \theta _{l}\sin \theta _{\nu }+ \frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{l}\cos \theta _{\nu } &{} -\frac{W_{3}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{l} \\ \cos \theta _{l}\cos \theta _{\nu }+\frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}}} \sin \theta _{l}\sin \theta _{\nu } &{} \frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}} }\cos \theta _{\nu }\sin \theta _{l}-\cos \theta _{l}\sin \theta _{\nu } &{} - \frac{W_{3}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\sin \theta _{l} \\ \frac{W_{3}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\sin \theta _{\nu } &{} \frac{W_{3}}{ \sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{\nu } &{} \frac{W_{1}}{\sqrt{ W_{1}^{2}+W_{3}^{2}}} \end{array} \right) \quad \text{ for } \text{ IH }. \end{array} \right. \end{aligned}$$
(29)

From the standard parametrization of the leptonic mixing matrix, it follows that the lepton mixing angles for NH and IH, respectively, are

$$\begin{aligned} \sin ^{2}\theta _{12}= & {} \frac{\left( \frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}} }\cos \theta _{l}\sin \theta _{\nu }-\cos \theta _{\nu }\sin \theta _{l}\right) ^{2}}{1-\left( \sin \theta _{l}\sin \theta _{\nu }+\frac{W_{1}}{ \sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{l}\cos \theta _{\nu }\right) ^{2}},\nonumber \\ \sin ^{2}\theta _{13}= & {} \left( \sin \theta _{l}\sin \theta _{\nu }+ \frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{l}\cos \theta _{\nu }\right) ^{2}, \nonumber \\ \sin ^{2}\theta _{23}= & {} \frac{\left( \frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}} }\sin \theta _{l}\cos \theta _{\nu }-\cos \theta _{l}\sin \theta _{\nu }\right) ^{2}}{1-\left( \sin \theta _{l}\sin \theta _{\nu }+\frac{W_{1}}{ \sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{l}\cos \theta _{\nu }\right) ^{2}}\, \qquad \text{ for } \text{ NH } \nonumber \\\end{aligned}$$
(30)
$$\begin{aligned} \sin ^{2}\theta _{12}= & {} \frac{\left( \sin \theta _{l}\sin \theta _{\nu }+ \frac{W_{1}}{\sqrt{W_{1}^{2}+W_{3}^{2}}}\cos \theta _{l}\cos \theta _{\nu }\right) ^{2}}{1-\frac{W_{3}^{2}\cos ^{2}\theta _{l}}{W_{1}^{2}+W_{3}^{2}}}, \nonumber \\ \sin ^{2}\theta _{13}= & {} \frac{W_{3}^{2}\cos ^{2}\theta _{l}}{ W_{1}^{2}+W_{3}^{2}}, \nonumber \\ \sin ^{2}\theta _{23}= & {} \frac{\frac{W_{3}^{2}}{W_{1}^{2}+W_{3}^{2}}\sin ^{2}\theta _{l}}{1-\frac{W_{3}^{2}\cos ^{2}\theta _{l}}{W_{1}^{2}+W_{3}^{2}}}\ ,\quad \text{ for } \text{ IH. } \end{aligned}$$
(31)

In the charged lepton sector the model has five free parameters \(a_{11}^{(l)}, a_{12}^{(l)}, a_{21}^{(l)}, a_{22}^{(l)}, y_{3 }^{(l)}\). Fitting Eqs. (19)–(21) for \(m_{e, \mu , \tau }\) to the corresponding experimental values [111], we found \(\mathcal {O}(1)\) solutions for these parameters. Therefore, we conclude that the model correctly predicts the charged lepton mass hierarchy according to Eqs. (19)–(21). On the other hand, it does not predict the particular values of the charged lepton masses. In fact, within the hierarchical structure incorporated in Eqs. (19)–(21), the five free \(\mathcal {O}(1)\) parameters \(a_{11}^{(l)}, a_{12}^{(l)}, a_{21}^{(l)}, a_{22}^{(l)}, y_{3 }^{(l)}\) allow one to reproduce the experimental values of \(m_{e,\mu ,\tau }\) with arbitrary precision limited only by the experimental errors.

Fig. 4
figure 4

Correlations between \(\Delta m_{21}^{2}\) and \(\Delta m_{31}^{2}\) for the cases of normal (left plot) and inverted (right plot) neutrino mass hierarchy. The horizonal and vertical lines are the minimum and maximum values of the neutrino mass squared splittings inside the \(3\sigma \) experimentally allowed range

The charged lepton sector “contributes” to the neutrino sector with a single free parameter \(\theta _{l}\) defined in Eq. (18). It remains unrestricted by the charged lepton masses. Thus, in the neutrino sector we have five free parameters \(W_{1,2,3},\kappa ,\theta _{l}\) and five observables: two neutrino mass squared splittings \(\Delta m_{21}^{2}\), \(\Delta m_{31}^{2}\) (we define \(\Delta m_{ij}^{2}=m_{i}^{2}-m_{j}^{2}\)) and three mixing angles \(\sin ^{2}\theta _{12}\), \(\sin ^{2}\theta _{13}\), \(\sin ^{2}\theta _{23}\) for both NH and IH. We solve Eqs. (25), (28) and (30), (31) with respect to the model parameters for the central values of the corresponding observables shown in Table 2. The unique solutions for NH and IH are

$$\begin{aligned}&\text{ for } \text{ NH: } \kappa \simeq 0.7,\quad W_{1}\simeq 0.09\,\text{ eV }^{ \frac{1}{2}},\nonumber \\&\quad W_{2}\simeq 0.16\text{ eV }^{\frac{1}{2}},\ \ W_{3}\simeq 0.15\,\text{ eV }^{\frac{1}{2}},\quad \theta _{l}\simeq 18.95^{\circ } \end{aligned}$$
(32)
$$\begin{aligned}&\text{ for }\ \ \text{ IH: } \kappa \simeq 6.11\times 10^{-3},\ \ W_{1}\simeq 0.14\,\text{ eV }^{\frac{1}{2}},\nonumber \\&\quad W_{2}\simeq 0.22\,\text{ eV }^{\frac{1}{2} },\ \ W_{3}\simeq 0.17\,\text{ eV }^{\frac{1}{2}},\ \ \theta _{l}\simeq -78.30^{\circ }. \end{aligned}$$
(33)

Figure 4 shows the correlation between \(\Delta m_{21}^{2}\) and \(\Delta m_{31}^{2}\) for the cases of normal and inverted neutrino mass hierarchies, respectively. We find that a slight variation from the best-fit values yields, for several points of the parameter space, an important deviation in the values of the neutrino mass squared splittings and leptonic mixing parameters, especially for the case of inverted neutrino mass hierarchy.

Table 2 The symmetry inspired benchmark scenarios (i) and (ii) versus the experimental values of the charged lepton masses [111], neutrino oscillation observables for the normal (NH) and inverted (IH) mass hierarchies [113]. Scenario (i): the four model parameters (35) fit the five neutrino oscillation observables for the NH. Scenario (ii)’: from the experimental input value \(m_{e}=0.487\) MeV we determine a model parameter \(a_{21}^{(l)}=0.52\) and then calculate \(m_{\mu }=113.07\) MeV; the neutrino oscillation observables for the NH are fitted with three parameters (36). Scenario (ii)’: the same as (ii), but fitting jointly the four model parameters (37) to the seven observables

Nevertheless the model in its present version is not predictive in the lepton sector except for the charged lepton mass hierarchy Eqs. (19)–(21). This is because the number of free parameters equals the number of observables.

However, considering the model parameters for NH in Eq. (32) we note that their numerical values \(W_{2}\simeq W_{3}\), \(\kappa \approx 1\) may point to an approximate underlying \(\mu \)\(\tau \) symmetry. In fact for \(W_{2}=W_{3}\), \(\kappa =1\) the light active neutrino mass matrix (22) coincides in structure with the \(\mu \)\(\tau \) symmetric Fukuyama–Nishiura texture [114]:

$$\begin{aligned} M_{\nu }=\left( \begin{array}{ccc} A &{} B &{} B \\ B &{} C &{} D \\ B &{} D &{} C \end{array} \right) . \end{aligned}$$
(34)

With this observation let us study a \(\mu \)\(\tau \) symmetry inspired benchmark scenario in our model. Incorporation of the \(\mu \)\(\tau \) symmetry in our model implies \(W_{2}=W_{3}\) and \(\kappa =1\) and, therefore, \(C=D\) in (34) leading to two vanishing eigenvalues, which is in clear contradiction with the neutrino oscillation data. After all, this symmetry cannot be exact, since it is explicitly broken in the charged lepton sector (for more details of the phenomenological aspects of the \(\mu \)\(\tau \) interchange symmetry see, for instance, Refs. [115,116,117,118,119,120,121,122,123,124,125]). However, the \(\mu \)\(\tau \) symmetry can be incorporated in our model as a “minimally” broken symmetry. Its maximal breaking corresponds to the violation of both \(W_{2}=W_{3}\) and \(\kappa =1\) conditions. In this sense the minimal breaking is introduced by the violation of only one of these conditions. As can easily be checked the case \(W_{2}\ne W_{3}\) with \(\kappa =1\) is an inappropriate one since it has two zero eigenvalues of the neutrino mass matrix Eq. (22). Thus, we study the benchmark scenario (i) with \(W_{2}= W_{3}\) and \(\kappa \) being a free \(\mu \)\(\tau \) symmetry breaking parameter. In this case there is only one zero neutrino mass matrix eigenvalue. Now we have four free model parameters \(W_{1}\), \(W_{2}\), \(\kappa \) and \(\theta _{l}\) versus the five physical observables in the neutrino sector, i.e., the two neutrino mass squared splittings and the three leptonic mixing parameters. Thus, the model is able to predict one of these observables using the experimental values of the other four. Instead of doing this we conventionally fit the model parameters to the experimental values of all the five observables within the corresponding experimental errors. The best-fit results are shown in Table 2 for

$$\begin{aligned}&\kappa \simeq 0.70,\quad W_{1}\simeq 0.09\text{ eV }^{\frac{1}{2}},\nonumber \\&\quad W_{2}\simeq 0.16\text{ eV }^{\frac{1}{2}},\quad \theta _{l}\simeq 19.81^{\circ }, \end{aligned}$$
(35)

corresponding to the normal neutrino hierarchy. Here the parameter \(\kappa \) is not too far from its \(\mu \)\(\tau \) symmetric value \(\kappa =1\). In the case of the inverted hierarchy it is two–three orders of magnitude away from this value and the \(\mu \)\(\tau \) symmetry does not show up as an approximate symmetry of the neutrino sector. A more detailed study of this observation and its possible implementation in the model Lagrangian will be addressed elsewhere.

Let us examine even more restrictive benchmark scenario assuming additionally to \(W_{3}=W_{2}\) also the relation for the model parameters \(a_{11}^{\left( l\right) }=a_{22}^{\left( l\right) }=a_{21}^{\left( l\right) }=1\) in the sector of charged leptons \(e, \mu \). So that there is only one free parameter \(a_{12}^{\left( l\right) }\) to fit \(m_{e}\) and \(m_{\mu }\) using Eqs. (19), (20). This scenario is motivated by the fact we already discussed above [see the paragraph after Eqs. (31)] that our model correctly reproduces the charged lepton mass hierarchy with all the parameters \(a_{11}^{(l)}, a_{12}^{(l)}, a_{21}^{(l)}, a_{22}^{(l)}\) of the order \(\mathcal {O}(1)\). Setting all these parameters to be \(a_{ij}^{(l)} = 1\) leads to a rather rough estimate of \(m_{e}\) and \(m_{\mu }\). Moreover, in this case \(\tan \theta _{l} =1\), which is incompatible with the neutrino oscillation data as seen from (35). Therefore, we release \(a_{12}^{(l)}\) assuming it to be the only free parameter in the \(e-\mu \)-sector of charged leptons. Thus we consider a benchmark scenario (ii) with \(a_{11}^{\left( l\right) }=a_{22}^{\left( l\right) }=a_{21}^{\left( l\right) }=1\), \(W_{2}=W_{3}\). There are only four free parameters are \(W_{1}, W_{2}, \kappa \) and \(a_{12}^{(l)}\) to reproduce seven observable: \(m_{e}, m_{\mu }\) and five neutrino oscillation parameters. Note that the charged lepton \(e-\mu \)-sector depends only on one of these parameters, namely, \(a_{12}^{(l)}\). It is tempting to try to make a prediction for \(m_{\mu }\) starting from the best measured \(m_{e}\). This also restricts the number of the free parameters in the neutrino sector down to \(W_{1}, W_{2}, \kappa \). In this way we solve Eq. (19) with respect to \(a_{12}^{(l)}\) for the central experimental value of \(m_{e}\) and predict \(m_{\mu }\) using (20). The result is shown in Table 2 in the column (ii)’ for \(a_{12}^{(l)}=0.52\) obtained from \(m_{e}\). In the neutrino sector, having three parameters, the model is able to predict any two of the five observables using as an input the experimental values of the other three. However, as in the scenario (i) we fit all the neutrino oscillation observables, \(\Delta m_{21}^{2}\), \(\Delta m_{31}^{2}\), \(\sin ^{2}\theta _{12}\), \(\sin ^{2}\theta _{13}\), \(\sin ^{2}\theta _{23}\) with the model parameters \(W_{1}, W_{2}, \kappa \). The best-fit values shown in the column (ii)’ of Table 2 correspond to

$$\begin{aligned}&\kappa =0.71,\quad W_{1}=0.05\text{ eV }^{\frac{1}{2}},\quad W_{2}=-0.17\text{ eV }^{\frac{1}{2}}. \end{aligned}$$
(36)

Now, instead of fixing \(a_{12}^{(l)}\) from \(m_{e}\), we fit all the model parameters \(W_{1}, W_{2}, \kappa , a_{12}^{(l)}\) to seven observables \(m_{e}, m_{\mu }\) and \(\Delta m_{21}^{2}\), \(\Delta m_{31}^{2}\), \(\sin ^{2}\theta _{12}\), \(\sin ^{2}\theta _{13}\), \(\sin ^{2}\theta _{23}\). The best-fit values are shown in the column (ii) of Table 2 and correspond to the following values of the model parameters:

$$\begin{aligned}&\kappa \simeq 0.71,\quad W_{1}\simeq 0.06\text{ eV }^{\frac{1}{2}}, \nonumber \\&\quad W_{2}\simeq - 0.17\text{ eV }^{\frac{1}{2}},\quad a_{12}^{\left( l\right) }\simeq 0.56. \end{aligned}$$
(37)

As seen from Table 3 in the scenario (ii) the neutrino oscillation parameters \(\sin ^{2}\theta _{12}\), \(\sin ^{2}\theta _{23}\), \(\Delta m_{21}^{2}\), \(\Delta m_{31}^{2}\) are inside the \(1\sigma \) experimentally allowed range. The reactor mixing parameter \(\sin ^{2}\theta _{13}\) is inside the \(2\sigma \) range.

Now we evaluate in our model the effective Majorana neutrino mass parameter of neutrinoless double beta (\(0\nu \beta \beta \)) decay

$$\begin{aligned} m_{\beta \beta }=\left| \sum _{j}U_{ek}^{2}m_{\nu _{k}}\right| , \end{aligned}$$
(38)

where \(U_{ej}^{2}\) and \(m_{\nu _{k}}\) are the PMNS mixing matrix elements and the Majorana neutrino masses, respectively.

Table 3 Range for experimental values of neutrino mass squared splittings and leptonic mixing parameters, taken from Ref. [113], for the case of normal hierarchy

As we already mentioned the model parameters in Eq. (13) may have arbitrary complex phases. Therefore there may occur cancellation between the terms in Eq. (38), which we are not able to control in the model. Thus we can predict only upper bounds for \(m_{\beta \beta }\). For the parameters (32), (33) we find

$$\begin{aligned} m_{\beta \beta }\le \left\{ \begin{array}{l}\ \, 4\ \text{ meV }\quad \text{ for } \text{ NH } ,\\ 49\ \text{ meV }\quad \text{ for } \text{ IH } .\\ \end{array} \right. \end{aligned}$$
(39)

Our obtained value \(m_{\beta \beta }\approx 4\ \text{ meV }\) for the effective Majorana neutrino mass parameter is beyond the reach of the present and forthcoming \(0\nu \beta \beta \) decay experiments. The current best upper bound on the effective neutrino mass is \(m_{\beta \beta }\le 160\) meV, which corresponds to \(T_{1/2}^{0\nu \beta \beta }(^{136}\mathrm {Xe})\ge 1.1\times 10^{26}\) year at 90% C.L, as indicated by the KamLAND-Zen experiment [126]. This bound will be improved within a not too far future. The GERDA “phase-II”experiment [127, 128] is expected to reach \(T^{0\nu \beta \beta }_{1/2}(^{76}\mathrm{Ge}) \ge 2\times 10^{26}\) yr, which corresponds to \(m_{\beta \beta }\le 100\) meV. A bolometric CUORE experiment, using \({}^{130}Te\) [129], is currently under construction and has an estimated sensitivity of about \( T_{1/2}^{0\nu \beta \beta }(^{130}\mathrm {Te})\sim 10^{26}\) year, which corresponds to \(m_{\beta \beta }\le 50\) meV. Furthermore, there are proposals for ton-scale next-to-next generation \(0\nu \beta \beta \) experiments with \(^{136}\)Xe [130, 131] and \( ^{76}\)Ge [127, 132], claiming sensitivities over \( T_{1/2}^{0\nu \beta \beta }\sim 10^{27}\) yr, which corresponds to \(m_{\beta \beta }\sim \)12–30 meV. For a recent review, see for example Ref. [133]. Consequently, as follows from Eq. (39), our model predicts \(T_{1/2}^{0\nu \beta \beta }\) at the level of sensitivities of the next generation or next-to-next generation \(0\nu \beta \beta \) experiments.

5 Dark matter candidate

It is worth mentioning that there is a viable dark matter (DM) candidate in our model. If we consider a specific scenario where only the extra scalar doublet \(\phi _{2}\) acquires mass at low energy, while the rest of extra scalars and exotic fermions live at high energies, an approach to the well-known inert Higgs doublet model could be done. This model, originally proposed in [103] and extensively studied in a number of recent works [134, 135], introduces an additional doublet, namely denoted in the literature as \(H_{2}\), odd under an additional \(Z_{2}\) symmetry. The lightest inert \(Z_{2}\)-odd particle turns out to be stable and hence a suitable DM candidate. Our additional scalar doublet \( \phi _{2}\), analogous to \(H_{2}\), is odd under \(Z_{2}\), which guarantees that it does not have direct couplings with SM fermion pairs and then its stability. The inert Higgs dark matter, in particular, could annihilate into \(WW^*\), \(ZZ^*\), \(hh^* \) and \(t\overline{t}^*\). In the region restricted by the relic density constraint, i.e: \(\Omega _{\mathrm{DM}} h^2=0.1181 \pm 0.0012\) the annihilation on the DM into \(W^+\) \(W^-\) and ZZ is very effective, much more than annihilation into \(hh^*\) and \(t\overline{t}^*\). The DM constraints can only be satisfied for restricted values of \(m_{\phi _{2}}\). In [135], three viable regions of DM are pointed out: a small regime with \(3\le m_{H_{2}} \le 50\) GeV, an intermediate regime with \(60\le m_{H_{2}} \le 100\) GeV and a large regime \(3 m_{H_{2}} \ge 550\) GeV. Therefore, if we expect \(\phi _{2}\) to be a DM candidate, its mass must fall in some of the three allowed regions. The limits on \(m_{\phi _{2}}\) can be translated into some restrictions on the other model parameters, for example, \(\epsilon _{ij}^{f}\) (\(f=u,d,\nu \)), whose precise values are fixed by the requirement of having a realistic spectrum of SM fermion masses and mixing angles. The resulting constraints on the \(\epsilon _{ij}^{l}\) parameters will yield bounds on the exotic fermion masses, thus setting limits on the total production cross sections of the non-SM particles at the LHC. Derivation of these constraints requires a dedicated study beyond the scope of the present paper and is left for future studies.

6 Conclusions

We have constructed an extension of the inert 2HDM, based on the extended \(S_{3}\otimes Z_{2}\otimes Z_{12}\) symmetry that successfully describes the current pattern of SM fermion masses and mixings. In our model the \(Z_{2}\) is preserved, whereas the \(S_{3}\) and \(Z_{12}\) symmetries are broken, giving rise to the observed pattern of charged fermion masses and mixing angles. The preserved \(Z_{2}\) symmetry allows the implementation of a one loop level radiative seesaw mechanism, which generates the masses for the first- and second-generation charged fermions, as well as a non-trivial quark mixing.

In our modified inert 2HDM, the SM fermion masses and mixing pattern arise from a combination of tree and one-loop-level effects. At tree level only the third-generation charged fermions acquire masses and there is no quark mixing, while the first- and second-generation charged fermion masses and the quark mixing arise from one loop level radiative seesaw-type mechanisms, triggered by virtual \(Z_{2}\)-charged scalar fields and electrically charged exotic fermions running inside the loops. Light active neutrino masses are generated from a one-loop-level radiative seesaw mechanism.

As follows from the expressions for the loop functions given in Eq. (B2), the entries of the SM charged fermion and light active neutrino mass matrices generated at one-loop-level are monotonically decreasing functions of the masses of the other particles in the loops shown in Fig. 1. The condition that the mass matrices be compatible with the observed masses and mixing of the SM fermions sets constraints on the masses of the \(Z_{2}\)-odd scalars and the non-SM fermions. Derivation of these constraints requires a dedicated study because of the complexity of the model parameter space.

The preserved \(Z_{2}\) symmetry of our model allows not only implementation of the radiative seesaw-type mechanism, but also provides natural dark matter candidates, stable due to this symmetry. They are the right-handed Majorana neutrinos \(N_{1R}\), \(N_{2R}\) and/or the lightest of the \(Z_{2}\) odd scalars \(\mathop {\mathrm{Re}}\nolimits \eta \), \(\mathop {\mathrm{Im}}\nolimits \eta \), \( \mathop {\mathrm{Re}}\nolimits \phi _{2}^{0}\), \(\mathop {\mathrm{Im}}\nolimits \phi _{2}^{0}\). Their masses are constrained by the dark matter relic density. The constraints on the masses of the \(Z_{2}\) odd scalars states, will yield bounds on the total production cross sections of these particles at the LHC. The implications of our model in collider physics and dark matter requires careful studies that we left outside the scope of this paper and defer for a future publication.