Probing neutrino and Higgs sectors in $\text{ SU(2) }_1 \times \text{ SU(2) }_2 \times \text{ U(1) }_Y$ model with lepton-flavor non-universality

Abstract

The neutrino and Higgs sectors in the $\text{ SU(2) }_1 \times \text{ SU(2) }_2 \times \text{ U(1) }_Y$ model with lepton-flavor non-universality are discussed. We show that active neutrinos can get Majorana masses from radiative corrections, after adding only new singly charged Higgs bosons. The mechanism for the generation of neutrino masses is the same as in the Zee models. This also gives a hint to solving the dark matter problem based on similar ways discussed recently in many radiative neutrino mass models with dark matter. Except the active neutrinos, the appearance of singly charged Higgs bosons and dark matter does not affect significantly the physical spectrum of all particles in the original model. We indicate this point by investigating the Higgs sector in both cases before and after singly charged scalars are added into it. Many interesting properties of physical Higgs bosons, which were not shown previously, are explored. In particular, the mass matrices of charged and CP-odd Higgs fields are proportional to the coefficient of triple Higgs coupling $\mu$. The mass eigenstates and eigenvalues in the CP-even Higgs sector are also presented. All couplings of the SM-like Higgs boson to normal fermions and gauge bosons are different from the SM predictions by a factor $c_h$, which must satisfy the recent global fit of experimental data, namely $0.995<|c_h|<1$. We have analyzed a more general diagonalization of gauge boson mass matrices, then we show that the ratio of the tangents of the W–$W'$ and Z–$Z'$ mixing angles is exactly the cosine of the Weinberg angle, implying that number of parameters is reduced by 1. Signals of new physics from decays of new heavy fermions and Higgs bosons at LHC and constraints of their masses are also discussed.

1 Introduction

One of the most important purposes of the LHC is to search for manifestations of new physics (NP). It seems that some clues have appeared with massive neutrinos and recent observations of lepton-flavor non-universality (LNU). Recall that the lepton family replication is assumed in the Standard Model (SM). Therefore, the lepton-flavor is universal in the latter. For the recent two decades, neutrino and Higgs physics are hot topics in Particle Physics. With increasing luminosity and beam energy, the LHC becomes a powerful tool for searching for NP. With larger masses, the third generation seems to be more interesting, in the sense of the sensitivity to NP. Nowadays, there are two kinds of anomalies in the semileptonic B meson decays which are captivating for the LNU. The first one is the class of the following ratios of branching fractions:

$$\begin{aligned}&R_{D^*} = \frac{\Gamma (\bar{B} \rightarrow D^*\, \tau \, \tilde{\nu } )}{\Gamma (\bar{B} \rightarrow D^*\, l\, \tilde{\nu } )} = 0.310 \pm 0.015 \pm 0.008, \nonumber \\&R_D = \frac{\Gamma (\bar{B} \rightarrow D\, \tau \, \tilde{\nu } )}{\Gamma (\bar{B} \rightarrow D\, l\, \tilde{\nu } )} = 0.403 \pm 0.040\pm 0.024, \nonumber \\&\quad l = e, \mu , \end{aligned}$$

(1)

which show $3.5 \sigma$ deviations from the corresponding SM predictions [1],

$$\begin{aligned} R_{D^*} = 0.252 \pm 0.004\quad , R_D=0.305 \pm 0.012. \end{aligned}$$

(2)

The above results provide hints for violation of the lepton flavor universality (LFU).

The second kind of anomalies¹ is the interesting LNU ratio reported recently by LHCb [2], namely,

$$\begin{aligned} R_K = \frac{\Gamma (B \rightarrow K\, \mu ^+\, \mu ^- )}{\Gamma (B \rightarrow K\, e^+\, e^- ))} = 0.745^{+0.090}_{-0.074} \pm 0.036, \end{aligned}$$

(3)

which has $2.6 \sigma$ deviation from the SM value $R_K = 1.0003 \pm 0.0001$ in the dilepton mass squared bin $(1 \le q^2 \le 6) \, \text {GeV}^2$.

From the physical point of view, the mass of a particle plays a quite important role in its characteristic properties. To justify this, let us mention some well-known examples. The first is that the proton and neutron have a tiny mass difference (940 vs. 938) MeV, but the proton is long-lived while the neutron is unstable with its mean lifetime of just under 15 min ($881.5 \pm 1.5$ s). The second example is the situation with the electron and the muon. Both particles are leptons with just a mass difference (0.511 vs. 105.6) MeV, the electron is stable while the muon is unstable with the mean lifetime of 2.2 $\upmu$s. So one may expect that the third generation of quarks and leptons where particles are heavier, has to be different from the first two ones. Within this context, the above data showing the LNU look quite understandable. In other words it is quite natural to expect that the third fermion generation is more strongly coupled to some New Physics than the first two ones. Recently the $R_D$ and $R_{D^*}$ were subjects of intensive studies mostly in scalar leptoquark models [4,5,6,7,8,9,10,11].

One of the beyond the SM models satisfying the recent experimental data of LNU is the model based on the $\mathrm {SU(2)}_1 \times \mathrm {SU(2)}_2 \times \mathrm {U(1)}_Y$ (G221) gauge group [12, 13] (more kinds of G221 models can be found in Ref. [14]). In Refs. [12, 13] the authors have mainly concentrated on explanation of LNU in the lepton sector. But at present, any theoretical model in particle physics has to deal with neutrino masses, the baryon asymmetry of the universe (BAU) and the dark matter (DM).

The aim of this work is to study further details in the gauge, Higgs and neutrino sectors of the model presented in Refs. [12, 13]. We will show that the problems of the active neutrino mass and DM in this model can be solved without any changes of results of allowed parameter regions satisfying all constraints of the flavor physics, tau decays, electroweak precision data, and recent anomalies in B decays, which were indicated in Refs. [12, 13]. In particular, the active neutrinos get Majorana masses from radiative corrections, where new lepton-number violating interactions have to be introduced. The simplest way is the Zee method [15, 16], where a pair of singly charged scalars transformed as singlets under both SU(2) gauge groups is introduced. Like in the Zee models, where a second $\mathrm{SU(2)}_L$ Higgs doublet is necessary for creating a nonzero triple coupling of two Higgs doublets and a singly charged Higgs singlet, the $\mathrm{SU(2)}_1$ Higgs doublet $\phi '$ in this G221 model plays the role of the second $\mathrm{SU(2)}_L$ Higgs doublet. Hence, no new breaking scales need to appear, implying that there are no new mass terms contributing to the fermion and gauge boson sectors. This explains why all results investigated in Refs. [12, 13] are unchanged, therefore we can use them to study the coupling properties of the Higgs and gauge bosons with fermions. In addition, it suggests that the ways of generating active neutrino masses in many recent radiative neutrino mass models can be applied to the G221 model. Many of these models have DM candidates that are neutral fermion singlets and have odd charges under a new $Z_2$ symmetry. To avoid complicate Higgs sectors, where just new charged Higgs bosons are included, we will not pay much attention to models solving the DM problems in this work. We will discuss in detail the mechanism of generating neutrino masses from the Zee mechanism, and the Higgs potential with the appearance of two singly charged Higgs singlets. In the gauge boson sector, we will apply general method to diagonalize neutral and charged gauge boson sectors, and from this we get a consequence that the tangents of the mixing angles in two sectors are proportional. This will reduce the number of the model parameters by 1. In the Higgs sector, the physical Higgs spectrum is presented. Then the SM-like Higgs boson and its couplings to other SM-like particles are identified and compared with the SM predictions. A comparison between properties of the Higgs spectrum in the G221 model and the minimal supersymmetric model (MSSM) and two Higgs doublet models (THDM) will also be discussed in this work. Based on these properties and the constraints of parameters given in [12, 13], we will discuss the bounds of new Higgs boson masses as well as promoting decay channels of the Higgs bosons and fermions that can be searched for at modern colliders such as the LHC.

This paper is organized as follows. In Sect. 2, after a brief review of the model, we present a more careful consideration of charged lepton masses and the Zee method for the generation of neutrino masses. In Sect. 2.2 we suggest two possibilities of appearance of DM candidates in the G221 model. The first is based on a radiative neutrino mass model introduced previously. This way will not change the results of parameter constraints in Refs. [12, 13]. The second way is different, because a new scalar $\mathrm{SU(2)}_2$ triplet is included. It contains a new neutral component with nonzero vacuum expectation value (VEV), leading to a new mass scale contributing to gauge boson masses. But this model may predict some active neutrinos playing the role of DM candidates. A more careful diagonalization of squared mass matrices and mixing parameters of gauge bosons is presented in Sect. 2.3. In this section, the relation between the tangents of the W–$W'$ and Z–$Z'$ mixing angles, is derived. Then a validation of $\rho$ parameter under recent experimental constraint will be shown at TeV scale of $\mathrm{SU(2)}_1$ breaking scale. Section 3 is devoted to charged and neutral currents in the model. Here we notice their difference from the SM ones. For the further discussion of the NP searches at colliders, the couplings of Z and W gauge bosons with fermions are given. In Sect. 4 a detailed analysis of the Higgs sector is presented. This section covers both versions of the Higgs sector content without and with the mentioned charged scalars. And the SM-like Higgs boson is identified. Interesting properties of singly charged Higgs bosons are also discussed. In Sect. 5, we review briefly the allowed regions of parameters given in [12, 13], which resulted from a specific numerical illustration in the limit of two vector-like fermion generations and simple textures of Yukawa couplings. Following the searches for new heavy particles at the LHC, we use these allowed regions to investigate lower bounds of masses and promoting decay channels of new fermions and Higgs bosons predicted by this model. Conclusions are given in the last Sect. 6.

2 Brief review of the model

The model is based on the gauge group $\mathrm {SU(2)}_1 \times \mathrm {SU(2)}_2 \times \mathrm {U(1)}_Y$ with the following gauge couplings, fields and generators [12, 13]:

$$\begin{aligned}&\mathrm {SU(2)}_1 : g_1, \quad W_i^1,\quad T_i^1, \nonumber \\&\mathrm {SU(2)}_2 : g_2, \quad W_i^2, \quad T_i^2,\\&\mathrm{{U(1)}}_Y : g' , \quad B, \quad Y, \nonumber \end{aligned}$$

(4)

where $i= 1, 2, 3$ is the $\mathrm {SU(2)}$ index. All the chiral fermions transform as

$$\begin{aligned} q_L\sim & {} \left( \mathbf 3,1,2 ,\frac{1}{6}\right) ,\quad \ell _L \sim \left( \mathbf 1, 1, 2 , -\frac{1}{2}\right) ,\nonumber \\ u_R\sim & {} \left( \mathbf 3, 1, 1 , \frac{2}{3}\right) ,\quad e_R \sim \left( \mathbf 1, 1, 1 , -1\right) ,\nonumber \\ d_R\sim & {} \left( \mathbf 3, 1, 1 , -\frac{1}{3}\right) , \end{aligned}$$

(5)

where the numbers in brackets refer to $\mathrm {SU(3)}_C$, $\mathrm {SU(2)}_1$, $\mathrm {SU(2)}_2$, and the hypercharge. The electric charge operator is determined in the form

$$\begin{aligned} Q = (T_3^1 + T_3^2) + Y. \end{aligned}$$

(6)

For the subgroup $\mathrm {SU(2)}_1$ there are $n_{VL}$ generations of vector-like fermions which are transformed as its doublets, while they are singlets for the $\mathrm {SU(2)}_2$,

$$\begin{aligned}&Q_{L,R} \equiv \left( \begin{array}{c} U \\ D\ \end{array}\, \right) _{L,R} \sim \left( \mathbf 3, 2, 1 , \frac{1}{6} \right) ; \quad L_{L,R} \equiv \left( \begin{array}{c} N \\ E\ \end{array}\, \right) _{L,R} \nonumber \\&\sim \left( \mathbf 1, 2, 1 , -\frac{1}{2} \right) . \end{aligned}$$

(7)

The vector-like fermion generation number is greater than one in order to explain successfully the LNU, and it was fixed by $n_{VL}=2$ for simplicity in numerical illustration [12, 13].

The Higgs sector consists of two doublets $\phi$ and $\phi '$ and one self-dual bidoublet $\Phi$ (i.e., $\Phi = \sigma _2 \Phi ^* \sigma _2$ where $\sigma _2$ is the usual Pauli matrix)

$$\begin{aligned} \phi \sim \left( \mathbf 1,1,2 , \frac{1}{2} \right) , \quad \phi ' \sim \left( \mathbf 1,2,1 , \frac{1}{2} \right) , \quad \Phi \sim (\mathbf 1, 2 , \tilde{\mathbf{2}},0), \nonumber \\ \end{aligned}$$

(8)

with components as

$$\begin{aligned} \phi = \left( \begin{array}{c} \varphi ^+ \\ \varphi ^0\ \end{array}\, \right) , \quad \phi ' = \left( \begin{array}{c} \varphi '^+ \\ \varphi '^0\ \end{array}\, \right) , \quad \Phi = \frac{1}{\sqrt{2}} \left( \begin{array}{cc} \Phi ^0 &{} \Phi ^+ \\ -\Phi ^- &{} \tilde{\Phi }^0\\ \end{array}\, \right) , \nonumber \\ \end{aligned}$$

(9)

with $\tilde{\Phi }^0 = (\Phi ^0)^*$. The scalar fields develop VEVs

$$\begin{aligned}&\langle \phi \rangle = \frac{1}{\sqrt{2}} \left( \begin{array}{c} 0\\ v_\phi \ \end{array}\, \right) , \quad \langle \phi ' \rangle =\frac{1}{\sqrt{2}} \left( \begin{array}{c} 0 \\ v_{\phi '}\ \end{array}\, \right) , \nonumber \\&\langle \Phi \rangle = \frac{1}{2} \left( \begin{array}{cc} u &{} 0\\ 0 &{} u\\ \end{array}\, \right) . \end{aligned}$$

(10)

The spontaneous symmetry breaking (SSB) of the model follows the pattern

$$\begin{aligned}&\mathrm {SU(2)}_1 \times \mathrm {SU(2)}_2 \times \mathrm{{U(1)}}_Y \mathop {\longrightarrow }\limits ^{u}\mathrm {SU(2)}_L \times \mathrm{{U(1)}}_Y\nonumber \\&\quad \mathop {\longrightarrow }\limits ^{ v_\phi ,\, v_{\phi '} }\mathrm{U(1)}_Q. \end{aligned}$$

(11)

The main phenomenology of the model concerned B-decay anomalies and the lepton-flavor non-universality has been presented in [12, 13]. However, the current physical model has to satisfy Higgs and neutrino physics as well as DM candidate.

With the above breaking chain, the VEVs are assumed to satisfy the relation

$$\begin{aligned} u \gg v_\phi , v_{\phi '}. \end{aligned}$$

(12)

Yukawa Lagrangian, fermion mass matrices, and diagonalization steps to construct physical states and masses of fermions were presented in detail in [12, 13]. Hence, we will summarize here only important results and focus on new features of generating active neutrino masses from loop corrections.

2.1 Charged fermion masses

The chiral fermions couple to the SM Higgs-like $\phi$ doublet

$$\begin{aligned} -\mathcal{L}_\phi = {\bar{q}_L}\, y_d \phi \, d_R + {\bar{q}_L}\, y_u \, \tilde{\phi }\, u_R + {\bar{\ell }_L}\, y_\ell \, \phi \, e_R\, + \mathrm {H.c.}, \nonumber \\ \end{aligned}$$

(13)

where $\tilde{\phi } \equiv i\sigma _2 \phi ^*$. The matrices $y_d, y_u, y_\ell$ are $3 \times 3$ matrices. The vector-like fermions can have gauge-invariant Dirac mass terms

$$\begin{aligned} -\mathcal{L}_M = {\bar{Q}_L}\, M_Q \, Q_R + {\bar{L}_L}\, M_L L_R + \mathrm {H.c.} \end{aligned}$$

(14)

Other contributions are

$$\begin{aligned}&-\mathcal{L}_\Phi = {\bar{Q}_R}\, \lambda ^\dag _q \, \Phi \, q_L + {\bar{L}_R}\, \lambda ^\dag _\ell \,\Phi \, \ell _L \, + \mathrm {H.c.} , \end{aligned}$$

(15)

$$\begin{aligned}&-\mathcal{L}_{\phi '} = {\bar{Q}_L}\, {\tilde{y}}_d \phi '\, d_R + {\bar{Q}_L}\, {\tilde{y}}_u \, \tilde{\phi '}\, u_R + {\bar{L}_L}\, {\tilde{y}}_\ell \, \phi ' \, e_R\, +\mathrm { H.c.} , \nonumber \\ \end{aligned}$$

(16)

where $\lambda ^\dagger _{q,\, \ell }$ and ${\tilde{y}}_{u, \, d,\,\ell }$ are $n_{VL} \times 3$ Yukawa matrices. After the SSB, the above couplings will induce mixing between the vector-like and the SM chiral fermions. This is crucial for the phenomenology of the model.

For the sake of simplicity one can assume a softly broken discrete $\mathcal{Z}_2$ symmetry under which only $\phi '$ is odd, making unnecessary Yukawa couplings vanish, i.e., ${\tilde{y}}_{u, \, d,\,\ell } \simeq 0$ [12, 13]. There is another charge assignment that also forbids Lagrangian in (16), while keeps $\phi '$ even: only $Q_L$ and $L_L$ are odd. This is necessary for generating active neutrino masses by the Zee method considered in this work.

We combine the chiral and vector-like fermions as

$$\begin{aligned} \mathcal{U}^I_{L,R}\equiv & {} (u^i_{L,R}, U^k_{L,R})^T, \, \mathcal{D}^I_{L,R} \equiv (d^i_{L,R}, D^k_{L,R})^T, \mathcal{E}^I_{L,R}\nonumber \\\equiv & {} (e^i_{L,R}, E^k_{L,R})^T, \end{aligned}$$

(17)

where $i=1, 2, 3$, $k=1, \ldots , n_{VL}$ and $I = 1, \ldots , 3+n_{VL}$. After the SSB, the fermion mass Lagrangian has the form

$$\begin{aligned} - \mathcal{L}_\mathrm{mass}^f = {\bar{\mathcal{U}}_L} \mathcal{M_U} \mathcal{U}_R + {\bar{\mathcal{D}}_L} \mathcal{M_D} \mathcal{D}_R + {\bar{\mathcal{E}}_L} \mathcal{M_E} \mathcal{E}_R + \mathrm {H.c.} \end{aligned}$$

(18)

Here, all above mass matrices are $(3+n_{VL}) \times (3+n_{VL})$ and have the form

$$\begin{aligned} \mathcal{M_U}= & {} \left( \begin{array}{cc} \frac{1}{\sqrt{2}}y_u v_\phi &{} \frac{1}{2} \lambda _q u \\ \frac{1}{\sqrt{2}}{\tilde{y}}_u v_{\phi '} &{} M_Q\\ \end{array}\, \right) , \, \mathcal{M_D} = \left( \begin{array}{cc} \frac{1}{\sqrt{2}}y_d v_\phi &{} \frac{1}{2} \lambda _q u \\ \frac{1}{\sqrt{2}}{\tilde{y}}_d v_{\phi '} &{} M_Q\\ \end{array}\, \right) , \nonumber \\ \mathcal{M_E}= & {} \left( \begin{array}{cc} \frac{1}{\sqrt{2}}y_{\ell } v_\phi &{} \frac{1}{2} \lambda _\ell u \\ \frac{1}{\sqrt{2}}{\tilde{y}}_{\ell } v_{\phi '} &{} M_L\\ \end{array}\, \right) . \end{aligned}$$

(19)

In the limit $\epsilon =v/u\ll 1$, these matrices are blocked-diagonalized perturbatively via two steps. After that, the SM parts are separated from the total. The transformations of fermion states are: $\mathcal{U}_L\rightarrow V^{\dagger }_QV^{\dagger }_u\mathcal{U}_L$, $\mathcal{D}_L\rightarrow V^{\dagger }_QV^{\dagger }_d\mathcal{D}_L$, $\mathcal{E}_L\rightarrow V^{\dagger }_LV^{\dagger }_e\mathcal{E}_L$, $\mathcal{U}_R\rightarrow W^{\dagger }_u\mathcal{U}_R$, $\mathcal{D}_R\rightarrow W^{\dagger }_d\mathcal{D}_R$, and $\mathcal{E}_R\rightarrow W^{\dagger }_e\mathcal{E}_R$, where $V_{F}$ ($F=Q,L$), $V_{f}$ ($f=e,u,d$), and $W_f$ are $(3+n_{V_L})\times (3+n_{V_L})$ unitary matrices [12, 13]. At the first step where $v=0$, every $\mathcal{M_F}$ ($\mathcal{F = U, D, E}$) is diagonalized by an exact $V_{F}$ depending on $u,\, M_{F}$ and $\lambda _{\ell ,q}$. At the second step, transformations $V_{f}$ and $W_{f}$ are expanded in terms of power series of $\epsilon$, $V_f= 1+i \epsilon ^2 H^f_V+ \cdots$ and $W_f= 1+i\epsilon H^f_W+1/2 (i\epsilon H^f_W)^2+\cdots$ They were listed precisely in [12, 13]. After the two steps, all original mass matrices in (19) will be transformed into block-diagonal forms $\hat{\mathcal{M}}_{ \mathcal{F}}=V_fV_{ F}\mathcal{M}_{ \mathcal{F}}W^{\dagger }_f$. One of the blocks in every $\hat{\mathcal{M}}_{ \mathcal{F}}$ is identified with a SM fermion block, which is diagonalized by $3\times 3$ unitary transformations: $f_L\rightarrow S^{\dagger }_f f_L$ and $f_R\rightarrow U^{\dagger }_f f_R$. Only the CKM matrix, $V_{\mathrm {CKM}}=S_uS^{\dagger }_d$, appears in the gauge couplings [12, 13]. We can fix $S_e=U_e=I_3$.

For studying Higgs boson phenomenology satisfying the allowed regions of parameters given in [12, 13], which resulted from a specific assumption of two new lepton families and textures of Yukawa couplings $\lambda _{q,\ell }$, we will present more detailed masses and eigenstates of charged leptons. The quark sector can be derived similarly. In the flavor basis $\mathcal {E}$ of charged leptons, the mass matrix $\mathcal{M_E}$ in (19) is $5\times 5$. Following Refs. [12, 13], a simple texture of $\lambda _\ell$ is chosen as

$$\begin{aligned} \lambda _{\ell }= 2 c_{\beta '} \left( \begin{array}{cc} \widetilde{M}_{L_1} &{} 0 \\ 0 &{} \widetilde{M}_{L_2}\Delta _{\mu } \\ 0 &{} \widetilde{M}_{L_2}\Delta _{\tau } \\ \end{array} \right) , \end{aligned}$$

(20)

where new parameters $\Delta _{\mu }$ and $\Delta _{\tau }$ will be considered as free parameters; while $\widetilde{M}_{L_1}, \widetilde{M}_{L_2}$ are “reduced” masses of new charged leptons, $m_{E_k}\simeq u\widetilde{M}_{L_k}$ [12, 13],

$$\begin{aligned} \widetilde{M}_L= \mathrm {diag} \left( \widetilde{M}_{L_1},\; \widetilde{M}_{L_2}\right) = \sqrt{\frac{M^{\dagger }_L M_L}{u^2}+ \frac{\lambda ^{\dagger }_{\ell }\lambda _{\ell }}{4}}. \end{aligned}$$

(21)

We recall here important properties of charged lepton parameters used in constructing radiative active neutrino masses. According to [12, 13], physical masses $(m_{e_i}, m_{E_k})$ related to $\mathcal {M}_{\mathcal {E}}$ in (19) by $V_eV_{L} \mathcal {M}_{\mathcal {E}}W_{e}^{\dagger }= \mathrm {diag}(m_{e_i},\; m_{E_k})$, the mass bases of left-and right-handed leptons $\mathcal {E}^{(\mathrm {d})I}_{L,R}\equiv (e^{(\mathrm {d})i}_{L,R},\; E^{(\mathrm {d})k}_{L,R})^T$ are defined as $\mathcal {E}_{L}=V^{\dagger }_LV^{\dagger }_e \mathcal {E}^{(\mathrm {d})}_{L}$ and $\mathcal {E}_{R}=W^{\dagger }_e \mathcal {E}^{(\mathrm {d})}_{R}$. The product $V_eV_{L}$ can be found from the relation $V_eV_{L} \mathcal {M}_{\mathcal {E}}\mathcal {M}_{\mathcal {E}}^{\dagger } (V_eV_{L})^{\dagger }=\mathrm {diag}(m^2_{e_i},\; m^2_{E_k})$. Non-diagonal elements of $V_L$ may be large because those of $\mathcal {M}_{\mathcal {E}}$ are at the $\mathrm{SU(2)}_1$ scale. In contrast, those of $V_e$ and $W_e$ are at least one order of $\frac{v_{\phi }}{u}$, because these elements of $V_{L} \mathcal {M}_{\mathcal {E}}$ are order of the electroweak scale. Hence, $V_e$ and $W_e$ are nearly identical when $u \gg v_{\phi }$. They only play the role of generating light charged lepton masses of $e,\;\mu$, and $\tau$. Hence, in many cases we can use the approximations $\mathcal {E}_{L}=V^{\dagger }_L\mathcal {E}^{(\mathrm {d})}_{L}$ and $\mathcal {E}^{I}_{R}= \mathcal {E}^{(\mathrm {d})I}_{R}$. We can see that the $V_L$ is exactly the mixing matrix of neutrinos if they are all considered as the pure Dirac particles. Formula of $V_L$ is written in the block form, namely [12, 13]

$$\begin{aligned} V_L=\left( \begin{array}{l|l} V^{11}_L=\sqrt{I_3-\frac{1}{4}\lambda _{\ell }\widetilde{M}_L^{-2}\lambda ^{\dagger }_{\ell }} \;\; &{} V^{12}_L=-\frac{u}{2}V^{11}_L\lambda _{\ell }M^{-1}_L \\ V^{21}_L=\frac{1}{2} \widetilde{M}_L^{-1}\lambda ^{\dagger }_{\ell } &{} V^{22}_L=\frac{1}{u} \widetilde{M}_L^{-1}M_L^{\dagger }\\ \end{array} \right) , \nonumber \\ \end{aligned}$$

(22)

where analytic expression of $V_L^{ij}$, with $i,j=1,2$, corresponding to $\lambda _\ell$ in Eq. (20) are given in Appendix A. The Yukawa coupling matrix $y_{\ell }$ (13) is also mentioned, with a requirement that the SM block of the charged leptons is diagonal after the block-diagonalization. It does not affect results obtained in Refs. [12, 13], which depend mainly on the gauge couplings.

Hereafter, many calculations to discuss on phenomenology of Higgs bosons will ignore small mixing between different flavor quarks. We will apply the same results of the charged lepton sector to the quarks. The equivalences between notations are: $V^{ij}_L, \lambda _{\ell }, \widetilde{M}_{L_{1,2}}, \Delta _{\mu ,\tau }\rightarrow V^{ij}_{Q},\lambda _{q}, \widetilde{M}_{Q_{1,2}}, \Delta _{b,s}$, which were given in [12, 13].

Next, we will discuss another possibility that neutrinos can get Majorana mass terms.

2.2 Neutral lepton masses

Unlike charged leptons, where the SM-like charged leptons have their own right-handed partners, the SM-like neutrinos do not. In addition, the neutral leptons may inherit Majorana mass terms, for example $\frac{1}{2}\overline{(\nu _L)^c}m_{\nu }\nu _L$ for active neutrinos. Hence, it is more convenient to write the mass matrix of neutral leptons in the form discussed in the seesaw models [17], which is different from [12, 13]. At the beginning $\nu _L$, $N_R$ and $N_L$ will be considered as independent fields, where the left- and right-handed bases are $\mathcal {N}'^I_L=(\nu _L,\; (N_R)^c,\; N_L)^T$ and $(\mathcal {N}'^{I}_L)^c=((\nu _L)^c,\; N_R,\; (N_L)^c)^T$, respectively. The mass term in the Lagrangian is now $-\mathcal {L}^{N}_{\mathrm {mass}}= \frac{1}{2} \overline{\mathcal {N}'_L} \mathcal {M}_N (\mathcal {N}'_L)^c + \mathrm {h.c.}$ For $\mathrm {n_{VL}}=2$, the mass matrix of the neutral leptons is a $7\times 7$ symmetric matrix having the following form:

$$\begin{aligned} \mathcal {M}^I_N= \left( \begin{array}{ccc} 0 &{} m_D &{}0 \\ m_D^T &{} 0 &{} M^T_L \\ 0 &{} M_L &{} 0 \\ \end{array} \right) , \end{aligned}$$

(23)

where $m_D\equiv \frac{1}{2} \lambda _\ell u$ and $M_L$ are $3\times 2$ and $2\times 2$ matrices, respectively. Similarly to seesaw models, $(N_L)^c$ and $N_R$ are additional right-handed neutrinos. The matrix (23) can be generally diagonalized through the transformation $\Omega ^T \mathcal {M}^I_N \Omega =\mathrm {diag}(\widehat{m},\; \widehat{M})$, where $\Omega$ is an unitary $7\times 7$ matrix; $\widehat{m}=\mathrm {diag}(m_{\nu _1},m_{\nu _2},m_{\nu _3})$ are light Majorana neutrino masses and $\widehat{M}$ gives Dirac masses for heavy neutrinos. Unfortunately, all light neutrinos are massless. This can be proved as follows. The neutral neutrino masses are $m_I=\sqrt{x}$ where the values of x are roots of the equation, $\mathrm {det}[x\times I_7- \mathcal {M}^{I\dagger }_N \mathcal {M}^{I}_N]=0$. For arbitrary forms of the $m_D$ and $M_L$, there are always three massless values of x. The matrices $M_L=\mathrm {diag}(M_{L_1},\;M_{L_2})$ and $\lambda _\ell$ in (20) result in four other solutions: $x_4=x_5=u^2 \widetilde{M}^2_{L_1}$ and $x_6=x_7=u^2 \widetilde{M}^2_{L_2}$. A pair of two degenerate values corresponds to one Dirac mass of a heavy Dirac neutrino, the same as that mentioned in [12, 13]. The mixing matrix of neutrinos is derived from (22) as follows:

$$\begin{aligned} \Omega ^T\equiv \left( \begin{array}{ccc} V^{11}_L&{} 0 &{} V^{12}_L \\ 0 &{} 1 &{} 0 \\ V^{21}_L &{} 0 &{} V^{22}_L \\ \end{array} \right) \rightarrow \Omega ^T \mathcal {M}^I_N \Omega =\left( \begin{array}{ccc} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} u \widetilde{M}_{L} \\ 0 &{} u \widetilde{M}_{L}&{} 0 \\ \end{array} \right) , \nonumber \\ \end{aligned}$$

(24)

where new neutrino masses are pure Dirac. In addition, new lepton masses in each family are nearly degenerate. Equation (24) gives the relations between the original and mass bases $\mathcal {N}_{L}= V^{\dagger }_{L}\mathcal {N}^{(d)}_L$ and $\mathcal {N}_{R}=\mathcal {N}^{(d)}_R$, which are the same as those of the charged leptons.

To keep the lepton spectrum being unchanged and looking for a solution of active neutrino mass problem, the mass terms of active neutrinos must come from the effective Majorana terms $\frac{1}{2}\overline{(\nu _L)^c}\; m_{\nu }\nu _L + \mathrm {h.c}.$ Because the active neutrino masses are tiny, their effect on the mixing parameters with heavy neutrinos is negligible. Based on the mechanism of the neutrino mass generation in the Zee model [15, 16], in this model only one pair of new singly charged Higgs bosons, denoted as $\delta ^{\pm }\sim (1,1,1)_{\pm 1}$ carrying even $\mathcal {Z}_2$ charges, is introduced. New couplings for generating one-loop radiative neutrino masses are

$$\begin{aligned} -\Delta \mathcal {L}= & {} f_{ij}\overline{(\ell _{L_i})^c}(i\sigma _2) \ell _{L_j}\delta ^+ +\sqrt{2}\lambda _{\delta }(i\sigma _2\phi ')^T\Phi \phi \delta ^-\nonumber \\&+\, f^{\prime }_{kl}\overline{(L_{L_k})^c}(i\sigma _2) L_{L_l}\delta ^+ +\mathrm {H.c.}, \end{aligned}$$

(25)

where $i,j=1,2,3$; and $k,l=1,2$. In the general case, $k,l=1,2,\ldots ,n_{V_L}$. We stress that all terms in (25) are simultaneously survival only when both $\phi '$ and $\delta ^{\pm }$ are even under $\mathcal {Z}_2$ symmetry.

The terms in the first line of (25) violate the lepton numbers, exactly in the same way as in the Zee model, where $\phi '$ plays a similar role as the second Higgs doublet. Similarly to the Zee model, the trilinear coupling is $\lambda _{\delta }u$ after the first step of the spontaneous breaking. An one-loop diagram generating active neutrino masses is shown in Fig. 1.

Fig. 1

One loop correction to active neutrino masses

Following [15, 18, 19], the effective mass matrix of light neutrinos is derived in Appendix B, where $\varphi ^\pm$ and $\delta ^\pm$ are assumed to be the physical Higgs bosons. But in the model under consideration, $\varphi ^{\pm }$ and $\delta ^\pm$ are not mass eigenstates. As we will discuss later, the physical fields in the Higgs sector are $h^\pm _{1,2}$, and there are some useful relations: $\varphi ^\pm \sim c_{\zeta }h^\pm _1$, $\Phi ^\pm \sim s_{\zeta }c_{\xi }h^\pm _1 -s_{\xi } h^\pm _2$, and $\delta ^\pm \sim s_{\zeta }s_{\xi }h^\pm _1+c_{\xi } h^\pm _2$. The parameters $c_{\xi },s_{\xi },c_{\zeta }$, and $s_{\zeta }$ involve with mixing parameters $\xi$ and $\zeta$ of the Higgs bosons, defined in Eqs. (60) and (76), as we will present below. The Higgs coupling in (25) can be rewritten as follows:

$$\begin{aligned}&-\frac{1}{2}\lambda _{\delta } vc_{\beta }\left[ 2s_{\zeta }s_{\xi }\left( uc_{\zeta } +vs_{\beta } s_{\zeta } s_{\xi }\right) h^{+}_1h^-_1 -2s_{\xi }c_{\xi }v h^{+}_2h^-_2 \right. \nonumber \\&\quad +\, \left. \left( uc_{\zeta }c_{\xi } +v s_{\beta } s_{\zeta }(c^2_{\xi }-s^2_{\xi }) \right) \left( h^{+}_2h^-_1+\mathrm {H.c.} \right) \right] \nonumber \\&\quad \subset -\sqrt{2}\lambda _{\delta }(i\sigma _2\phi ')^T\Phi \phi \delta ^- +\mathrm {H.c.}, \end{aligned}$$

(26)

where $v_{\phi }=v s_{\beta }$, $v_{\phi '}=vc_{\beta }$ and $t_{\beta }\equiv \tan \beta =s_{\beta }/c_{\beta }$, which are defined in [12, 13].

Charged leptons $e_c$ in the loop will be considered as mass eigenstates with masses $m_{e_c}.$ Therefore, the Yukawa terms should be written in terms of physical charged lepton states, and light neutrinos are massless states after the rotation $V_L$. For simplicity, we will assume that $V^{11}_L$ is real and $V_e,W_e\simeq I.$ In addition, we ignore one-loop contributions to heavy neutrino masses because they are extremely smaller than the tree level masses. Then the one-loop corrections are mainly from light leptons, namely

$$\begin{aligned} f_{ac}\overline{e_{L_c}}(\nu _{L_a})^c= & {} f_{ac}(V_L)_{Ih}(V^T_L)_{cJ}\overline{\mathcal {E}^{(\mathrm {d})I}_{L}} (\mathcal {N}^J_{L})^c\nonumber \\\rightarrow & {} f_{gh}(V_L)_{ch}(V^T_L)_{ga}\overline{e_{L_c}}(\nu _{L_a})^c\nonumber \\= & {} (V^{11}_LfV^{11}_L)_{ac}\overline{e_{L_c}}(\nu _{L_a})^c, \end{aligned}$$

and $f^{\prime }_{kl}\overline{L_{L_k}}(N_{L_l})^c\rightarrow (V^{12}_Lf^{\prime }V^{12T}_L)_{ac}\overline{e_{L_c}}(\nu _{L_a})^c$. Here $a,c,g,h=1,2,3$; $I,J=1,2,\ldots ,7$; f and $f'$ are $3\times 3$ and $2\times 2$ antisymmetric matrices, respectively. Similarly, we have

$$\begin{aligned} (y_{\ell })_{bc}\overline{\nu _{L_b}}e_{R_c}= & {} \frac{\sqrt{2}}{vs_{\beta }}\left( \mathcal {M}_{\mathcal {E}}\right) _{bc} (V_L)_{Jb} \overline{\mathcal {N}^J_{L}}e_{R_c}\\\simeq & {} \frac{\sqrt{2}}{vs_{\beta }}\left( V^{\dagger }_L \mathrm {diag}(m_{\mathcal {E}_I})\right) _{bc} (V_L)_{Jb} \overline{\mathcal {N}^J_{L}}e_{R_c}, \\\rightarrow & {} \frac{\sqrt{2}}{vs_{\beta }}(V^{T}_L)_{dc}(V_L)_{bd}m_{e_c} \overline{\nu _{L_b}}e_{R_c} \\= & {} \frac{\sqrt{2}}{vs_{\beta }}(V_LV^{T}_L)_{bc}m_{e_c} \overline{\nu _{L_b}}e_{R_c}. \end{aligned}$$

The effective mass matrix $m_{\nu }$ of active neutrinos is derived based on (B3),

$$\begin{aligned} (m_{\nu })_{ba}= & {} \frac{\lambda _{\delta }\sqrt{2}}{16\pi ^2t_{\beta }}\times \frac{u }{m^2_{h^\pm _1}} \sum _{c=1}^{3}m^2_{e_c}\nonumber \\&\left[ (V^{11}_LfV^{11}_L+ V^{12}_Lf^{\prime }V^{12T}_L)_{ac}(V^{11}_LV^{11T}_L)_{bc} \right. \nonumber \\&\left. +\,(V^{11}_LfV^{11}_L+ V^{12}_Lf^{\prime }V^{12T}_L)_{bc} (V^{11}_LV^{11T}_L)_{ac}\right] \nonumber \\&\times \,\left( \left[ c_{\zeta } c_{\xi }+\frac{v}{u}s_{\beta }s_{\zeta }(c^2_{\xi }-s^2_{\xi })\right] \frac{m^2_{h^\pm _1}}{m^2_{h^\pm _1}-m^2_{h^\pm _2}} \right. \nonumber \\&\left. \ln \left[ \frac{m^2_{h^\pm _2}}{m^2_{h^\pm _1}}\right] - 2s_{\zeta }s_{\xi }\left[ c_{\zeta }+\frac{v}{u}s_{\beta }s_{\zeta }s_{\xi }\right] \right) \nonumber \\\equiv & {} \frac{1}{m_0}\left\{ \left( \left[ V^{11}_LfV^{11}_L+V^{12}_Lf^{\prime }V^{12T}_L\right] M^2_eV^{11T}_LV^{11}_L\right) _{ab} \right. \nonumber \\&+\, \left. \left( \left[ V^{11}_LfV^{11}_L+V^{12}_Lf^{\prime }V^{12T}_L\right] M^2_eV^{11T}_LV^{11}_L\right) _{ba}\right\} , \nonumber \\ \end{aligned}$$

(27)

where $M_e\equiv \mathrm {diag}(m_{e},\;m_{\mu },\;m_{\tau })$.

Including loop contributions (27), the SM block of (24) will be changed from zero into $m_{\nu }$: $\Omega ^T \mathcal {M}^I_N \Omega \supset m_{\nu }=U^*_{\mathrm {PMNS}}\hat{m}_{\nu } U^\dag _{\mathrm {PMNS}}$, where $\hat{m}_{\nu }=\mathrm {diag}(m_{\nu _1},\,m_{\nu _3},\,m_{\nu _3})$ consisting of three active neutrino masses, and $U_{\mathrm {PMNS}}$ is the well-known neutrino mixing matrix. If $V^{11}_L=I_3$ and $V^{12}_L=0$, Eq. (27) is the same as (B3). Like in the Zee models, the parameters arising from the Higgs sector affect the order of the neutrino masses only. But the masses and mixing angles of active neutrinos depend on unknown parameters in $f, f^{\prime }$, and $V^{11}_L$. As a result, the model under consideration is less restrictive in fitting the neutrino data than the Zee models. Because these models are still valid [20, 21], the neutrino sector mentioned here is realistic. In general, fitting recent neutrino data needs at least five free parameters, in agreement with three mixing angles and two squared mass differences. Because two of four parameters, namely $m_0$ and three $f_{ab}$, determine the order of the lightest neutrino mass, there are two free parameters left. When $n_{V_L}\ge 3$, there are at least three additional parameters $f^{\prime }_{kl}$, enough for fitting neutrino data without constraints on $V^{11}_L$. Interestingly, the neutrino fitting results in [20, 21] would be applied to the model under consideration if $L_L$ carries even $\mathcal {Z}_2$ charge which will survive the lepton coupling matrix $\tilde{y}_{\ell }$ in (16).

Regarding $n_{V_L}=2$, there is only one parameter $f'_{12}=-f'_{21}$. Therefore two parameters in $V^{11}_L$ may be involved with fitting neutrino data. Our numerical investigation showed that the allowed regions in Refs. [12, 13], controlled by the texture $\lambda _{\ell }$ (20), seems much more constrained. Note that the $m_{\nu }$ in (27) keep only main contributions from loops containing light charged leptons, where mixing terms with order $\mathcal {O}(\epsilon )$ are ignored. With u around 1 TeV and light new charged leptons, contributions from these lepton mediations to $m_{\nu }$ will be significant, implying that their masses can be free parameters for fitting neutrinos data without much changes of $\Delta _{\mu ,\tau }$. Finding exact allowed regions should be done elsewhere.

When the neutrino data is fitted, the results in Refs. [12, 13] for B-decay anomalies are still unchanged because the analysis considered here addresses only effects of tree contributions from heavy gauge bosons, where other contributions from the light lepton masses are suppressed. The unique changes may come from the gauge couplings of active neutrinos with charged gauge bosons. Following [12, 13], after the block-diagonalization these gauge couplings are proportional to $W^{\mu }_l\overline{ \nu _L}\gamma _{\mu }e_L$ and $W^{\mu }_h\overline{ \nu _L}\gamma _{\mu }\Delta ^{\ell }e_L$, where $W_l$ and $W_h$ are light and heavy charged gauge bosons. In the neutrino mass basis they become $W^{\mu }_l\overline{ \nu _L}U^{\dagger }_{\mathrm {PMNS}}\gamma _{\mu }e_L$ and $W^{\mu }_h\overline{ \nu _L}U^{\dagger }_{\mathrm {PMNS}}\Delta ^{\ell }\gamma _{\mu }e_L$, resulting in the same factor $(U^{\dagger }_{\mathrm {PMNS}})_{ii}$ for coupling $\overline{\nu _i}e_{i}$ with a diagonal $\Delta ^{\ell }$ obtained from the texture of $\lambda _{\ell }$ in (20). This factor will not appear in final results presenting the ratios of B-decay anomalies, as given in [12, 13].

In general, active neutrino mass generation from radiative corrections mentioned above affects only the lepton sector. Furthermore, it does not affect mixing parameters controlling the $\lambda _{\ell }$ structure at the first breaking step, hence suggests that the orders of numerical values in allowed regions will not change after neutrino data is fitted.

The above discussion just refers to a simple extension that can generate active neutrino masses through radiative corrections. The problems of neutrino masses and DM can be solved by models with more charged Higgs bosons and singlet right-handed neutral leptons, such as [22,23,24]. Following the structures of these models, apart from $\delta ^\pm$, at least one pair of singly charged Higgs bosons $S^{\pm }$ and a neutral lepton $\mathcal {F}_R\sim (1,1,1)_{0}$ have to be introduced, where $S^+\sim (1,1,1)_{1}$. In addition, only $S^{\pm }$ and $\mathcal {\mathcal {F}}_R$ are odd under a new $Z_2$ discrete symmetry, $\{ S^\pm ,\; \mathcal {F}_R\}\rightarrow \{ -S^\pm ,\; -\mathcal {F}_R\}$. Therefore, $\mathcal {F}_R$ can play the role of DM. It has a Majorana mass term of the form $\frac{1}{2} \overline{(\mathcal {F}_R)^c}m_{\mathcal {F}}\mathcal {F}_R$. Active neutrinos get mass from loop corrections, which arise from a new Yukawa term, $-\Delta \mathcal {L}_Y=f_{ij}\overline{(\ell _{L_i})^c }(i\sigma _2)\ell _{L_j}\delta ^+ +g_{i}\overline{(\mathcal {F}_R)^c }e_{R_i}\delta ^+ +\mathrm {H.c.}$, and a coupling of charged Higgs bosons, $\frac{1}{4}\lambda _{\delta S} (\delta ^+)^2 (S^-)^2+\mathrm { H.c.}$. This kind of models seems to be less interesting because the origin of neutrino masses is not related to the new leptons.

New ingredients for generating radiative corrections to active neutrinos do not change both results of the gauge sector and LNU discussed in [12, 13], because no new breaking scale contributes to the masses of gauge Higgs bosons. If we add a new $\mathrm{SU(2)}_1$ triplet, denoted as $\Delta \sim (1,3,1)_{1}$, creating an Yukawa term like $-Y_{\Delta }\overline{(L_L)^c}i\sigma _2\Delta L_L+ \mathrm {H.c.}$, a neutral component of this triplet will develop a non-zero VEV $v_{\Delta }$, which contributes a new mass term of the form $\frac{1}{2}\mu _X \overline{N_L} (N_L)^c+ \mathrm {H.c.}$ to the neutrino mass matrix (23). This matrix has the same form shown in the inverse seesaw models [17, 25,26,27]. Hence the active neutrino masses will be non-zero. In addition, some new neutrinos may get light masses and play the role of DM [28,29,30]. These models seem interesting because they may give connections between the $\mathrm{SU(2)}_1$ leptons with neutrino masses and DM. But the appearance of the new vev $v_{\Delta }$ will contribute to masses and mixing parameters of the Higgs and gauge bosons, consequently it will affect the results shown in [12, 13]. This extension is beyond our scope, and should be thoroughly studied in another work.

Now we turn to one of the most important elements: gauge bosons.

2.3 Gauge boson masses

Gauge boson masses arise from the piece

$$\begin{aligned}&\mathcal {L}_{\mathrm {gauge\; boson\; mass}} = (D_\mu \langle \phi \rangle )^\dag D^\mu \langle \phi \rangle + (D_\mu \langle \phi ' \rangle )^\dag D^\mu \langle \phi ' \rangle \nonumber \\&\quad ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ \mathrm{Tr}[(D_\mu \langle \Phi \rangle )^\dag D^\mu \langle \Phi \rangle ], \end{aligned}$$

(28)

where the covariant derivative of $\Phi$ is determined as

$$\begin{aligned} \left( D_\mu \Phi \right) _\alpha ^\beta= & {} \partial _\mu \Phi _\alpha ^\beta - \frac{i}{2} g_1 W_{i \mu }^1 (\sigma _i)_\alpha ^\gamma (\Phi )_\gamma ^\beta \nonumber \\&+\, \frac{i}{2} g_2 (\Phi )_\alpha ^\gamma W_{i \mu }^2 (\sigma _i)_\gamma ^\beta \, . \end{aligned}$$

(29)

With the help of the notation

$$\begin{aligned}&W_\mu ^i \equiv \frac{1}{2} \sum _{\alpha =1}^3 W_{\alpha \mu }^i \sigma _\alpha = \frac{1}{2} \left( \begin{array}{cc} W_3^i &{} \sqrt{2}\, W_{i}^+ \\ \sqrt{2}\, W_{i}^- &{} -W_3^i \\ \end{array}\, \right) _\mu , \nonumber \\&W_i^{ \pm } \equiv \frac{1}{\sqrt{2}}(W_1^i \mp i W_2^i), \quad i = 1,2, \end{aligned}$$

(30)

contributions to masses of gauge bosons are

$$\begin{aligned}&\mathrm{Tr}[(D_\mu \langle \Phi \rangle )^\dag D^\mu \langle \Phi \rangle ]\\&\quad = \frac{u^2}{16}\left[ 2 (g_1 W_3^1 - g_2 W_3^2)^2+ 4 (g_1 W_{1}^+ - g_2 W_{2}^+)_\mu \right. \\&\qquad \left. \times \, ( g_1 W_{1}^- - g_2 W_{2}^-)^\mu \right] , \\&(D_\mu \langle \phi \rangle )^\dag D^\mu \langle \phi \rangle = \frac{v_\phi ^2}{4} \left[ g_2^2 W_2^{ +}W_2^{ -} + \frac{1}{2} (g_2 W_2^3 - g' B)^2\right] , \\&(D_\mu \langle \phi ' \rangle )^\dag D^\mu \langle \phi ' \rangle = \frac{v_{\phi '}^2}{4} \left[ g_1^2 W_1^{ +}W_1^{ -} + \frac{1}{2} (g_1 W_1^3 - g' B)^2\right] . \end{aligned}$$

From this, masses and eigenstates of gauge bosons can be found in agreement with those presented in Refs. [12, 13]. We will review important aspect then discuss some new properties when masses and mixing angles are calculated up to order of $\mathcal {O}(\epsilon ^2)$.

2.4 Neutral gauge bosons

In the basis $(W_3^1, W_3^2, B)$ the squared mass matrix of neutral gauge boson is $M^2_{nb}$. At the first step, where $v_{\phi },v_{\phi '}\rightarrow 0$, only two states $W_3^1$ and $W_3^2$ are rotated through a rotation $C_1$ so that $C_1 M^2_{nb} C_1^T|_{v_{\phi },v_{\phi '}=0} =\mathrm {Diag}(0,0,\frac{1}{4}(g_1^2+g_2^2)u^2)$. Elements of $C_1$ depend on a mixing angle $\beta '$ defined by

$$\begin{aligned} \tan \beta '\equiv \frac{g_1}{g_2}, \; c_{\beta '}=\cos \beta ' =g_2/n_1,\;s_{\beta '}=\sin \beta '=g_1/n_1, \nonumber \\ \end{aligned}$$

(31)

where $n_1=\sqrt{n_1^2+n_2^2}$ was used already in [12, 13].

The first breaking step implies the following transformation of the neutral gauge bosons: $(W^1_3, W^2_3, B)\xrightarrow {u}( B, W_3, Z_h)$, where $(B, W_3)$ are the SM gauge bosons. We have $(W^1_3, W^2_3, B)^T=C_1^T (B, W_3,Z_h )^T$, i.e. $W_3= c_{\beta '} W^1_3 +s_{\beta '} W^2_3$ and $Z_h= s_{\beta '} W^1_3 -c_{\beta '} W^2_3$, where $g=g_1g_2/n_1$ and $g'$ are identified as the SM gauge couplings; $v_{\phi }= vs_{\beta }$, $v_{\phi '}=vc_{\beta }$ [12, 13]. Note that $v\simeq 246$ GeV, $g'=gs_W/c_W$, and $s_W$ is the sine of the Weinberg angle. From now on, $n_1, g_1,g_2$ and $n_2$ will be written as

$$\begin{aligned} n_1= \frac{g}{c_{\beta '}s_{\beta '}}, n_2=\frac{g}{c_W}, g_1= \frac{g}{c_{\beta '}}, g_2= \frac{g}{s_{\beta '}}. \nonumber \\ \end{aligned}$$

(32)

At the second step, the mixing matrix $C_2$ is the SM rotation of only B and $W^3$, giving a new basis $(A,Z_l,Z_h)^T=C_2 (Z_h, W_3, B)^T= C_2C_1(W^1_3, W^2_3, B)^T$, where A and $Z_l$ are the photon and SM gauge boson. The respective matrix $M'^2_{nb}$ is

$$\begin{aligned} M'^2_{nb}= & {} C_2 C_1 M^2_{nb}(C_2C_1)^T\nonumber \\= & {} \frac{g^2}{4} \left( \begin{array}{ccc} 0 &{} 0 &{} 0 \\ 0 &{}\frac{v^2}{c^2_W} &{}\frac{\left( c_{2\beta }-c_{2\beta '}\right) v^2}{c_Ws_{2\beta '}} \\ 0 &{}\frac{\left( c_{2\beta }-c_{2\beta '}\right) v^2}{c_Ws_{2\beta '}} &{} \frac{4 u^2+\left( 1-2c_{2\beta }c_{2\beta '}+ c^2_{2\beta '}\right) v^2}{s^2_{2\beta '}} \\ \end{array}\, \right) . \nonumber \\ \end{aligned}$$

(33)

The mass eigenstates $(Z,Z')$ relates with the $Z_l$–$Z_h$ mixing angle defined as

$$\begin{aligned} t_{2Z}\equiv & {} \tan (2Z) = \frac{-2 \left( M'^2_{nb}\right) _{23}}{\left( M'^2_{nb}\right) _{33} -\left( M'^2_{nb}\right) _{22}}\nonumber \\= & {} \frac{2 s_{2\beta '}\left( c_{2\beta '}-c_{2\beta }\right) \frac{\epsilon ^2}{c_W}}{4+\left( 1-2c_{2\beta }c_{2\beta '} +c^2_{2\beta '}-\frac{s^2_{2\beta '}}{c^2_W}\right) \epsilon ^2}, \end{aligned}$$

(34)

where $\epsilon \equiv \frac{v}{u}$. The $Z_l$–$Z_h$ mixing vanishes when $\beta '=\beta$, where $\tan \beta =s_{\beta }/c_{\beta }$.

The masses of the physical eigenstates $(Z,Z')$ are

$$\begin{aligned} M_Z^2= & {} \frac{g^2}{4}\left[ \frac{v^2}{c^2_W}c^2_{Z}+\left( \frac{4 u^2+\left( 1-2c_{2\beta }c_{2\beta '} + c^2_{2\beta '}\right) v^2}{s^2_{2\beta '}}\right) s^2_{Z}\right. \\&\left. +\, 2\left( \frac{\left( c_{2\beta }-c_{2\beta '}\right) v^2}{c_W s_{2\beta '}} \right) s_{2Z} \right] , \\ M_{Z'}^2= & {} \frac{g^2}{4}\left[ \frac{v^2}{c^2_W}s^2_{Z} +\left( \frac{4 u^2+\left( 1-2c_{2\beta }c_{2\beta '}+ c^2_{2\beta '}\right) v^2}{s^2_{2\beta '}}\right) c^2_{Z}\right. \\&\left. -\, 2\left( \frac{\left( c_{2\beta }-c_{2\beta '}\right) v^2}{c_W s_{2\beta '}} \right) s_{2Z} \right] . \end{aligned}$$

The relation between the two bases $(W_1,W_2, B)$ and $(A,Z,Z')$ is

$$\begin{aligned} \left( \begin{array}{c} W_1\\ W_2\\ B\\ \end{array}\, \right)= & {} \left( \begin{array}{ccc} s_Wc_{\beta '} &{}c_Wc_{\beta '} &{}s_{\beta '} \\ s_Ws_{\beta '}&{}c_Ws _{\beta '} &{}-c_{\beta '} \\ c_W &{}- s_W &{} 0\\ \end{array}\, \right) \left( \begin{array}{ccc} 1 &{} 0 &{} 0 \\ 0 &{} c_Z &{} -s_Z \\ 0 &{} s_Z &{} c_Z \\ \end{array} \right) \left( \begin{array}{c} A\\ Z\\ Z'\\ \end{array}\, \right) , \nonumber \\ \end{aligned}$$

(35)

where $(C_2C_1)^T$ is the first matrix in the right hand side of (35).

Using the new notations of (32), the parameter $\zeta$ in [12, 13] can be expressed as $\zeta \equiv s^2_{\beta }-\frac{g_1^2}{g_2^2}c^2_{\beta }= \frac{1}{2c^2_{\beta }} ( c_{2\beta '}- c_{2\beta })$. In addition, from $m^2_{Z}\simeq \frac{g^2}{4 c^2_W}v^2\simeq \frac{g^2}{4 c^2_W} u^2 \epsilon ^2$, $m^2_{Z'}\simeq \frac{g^2}{4 c^2_{\beta '}s^2_{\beta '}}u^2= \frac{g^2}{ s^2_{2\beta '}}u^2$ and $\frac{g}{n_2}\frac{g_2}{g_1}=\frac{c_{\beta '}c_W}{s_{\beta '}}$, we can deduce an approximate form $\xi _Z\simeq 1/2\tan 2\xi _Z$ in the limit $\epsilon \ll 1$, consistent with the expression of $\tan 2\xi _Z$ shown in (34).

2.5 Charged gauge bosons

In the basis $(W_1^+, W_2^+)$ the squared mass matrix of charged gauge bosons was given in Refs. [12, 13]. Setting $v=0$, we can define a new basis: $W^+_l = ( c_{\beta '} W^+_1 + s_{\beta '} W_2^+)$ and $W^+_h = (s_{\beta '} W^+_1 - c_{\beta '} W_2^+)$, where the corresponding squared mass matrix is

$$\begin{aligned} M_c^2 = \frac{g^2}{4} \left( \begin{array}{cc} v^2 &{} \frac{\left( c_{2\beta }-c_{2\beta '}\right) v^2}{s_{2\beta '}} \\ \frac{\left( c_{2\beta }-c_{2\beta '}\right) v^2}{s_{2\beta '}} &{} \frac{4 u^2+\left( 1-2c_{2\beta }c_{2\beta '}+ c^2_{2\beta '}\right) v^2}{s^2_{2\beta '}} \\ \end{array}\, \right) . \end{aligned}$$

(36)

The SM-like boson $W^\pm$ is identified with $W^\pm \equiv W^\pm _l$ with mass $m_{W_l}= gv/2$. The mixing $W^+_l-W^+_h$ is defined through the mixing angle $\xi _W$ satisfying

$$\begin{aligned} t_{2\xi _W}\equiv & {} \tan (2\xi _W) =\frac{-2 \left( M^2_{c}\right) _{12}}{\left( M^2_{c}\right) _{22} -\left( M^2_{c}\right) _{11}} \nonumber \\= & {} \frac{2 s_{2\beta '}\left( c_{2\beta '}-c_{2\beta }\right) \epsilon ^2}{4+\left( 1-2c_{2\beta }c_{2\beta '} + c_{4\beta '}\right) \epsilon ^2} = c_W t_{2Z}. \end{aligned}$$

(37)

From (37), it follows that the ratio of the tangents of W–$W'$ and Z–$Z'$ mixing angles is $c_W$. This will reduce the number of parameters in the model by 1.

The physical mass eigenstates $(W^\pm , W'^\pm )$ are given by

$$\begin{aligned} \left( \begin{array}{c} W^\pm \\ W'^\pm \ \end{array}\, \right) = \left( \begin{array}{cc} c_{\xi _W} &{} s_{\xi _W} \\ -s_{\xi _W}&{} c_{\xi _W}\\ \end{array}\, \right) \left( \begin{array}{c} W^\pm _l\\ W^\pm _h\ \end{array}\, \right) \end{aligned}$$

(38)

with $c_{\xi _{W}}\equiv \cos \xi _W$, $s_{\xi _{W}}\equiv \sin \xi _W$, and masses

$$\begin{aligned} M_W^2= & {} \frac{g^2}{4}\left[ v^2c^2_{\xi _W}+\left( \frac{4 u^2+\left( 1-2c_{2\beta }c_{2\beta '} + c^2_{2\beta '}\right) v^2}{s^2_{2\beta '}}\right) s^2_{\xi _W}\right. \nonumber \\&\left. \phantom {\left[ v^2c^2_{\xi _W}+\left( \frac{4 u^2+\left( 1-2c_{2\beta }c_{2\beta '} + c^2_{2\beta '}\right) v^2}{s^2_{2\beta '}}\right) s^2_{\xi _W}\right. }+\, 2\left( \frac{\left( c_{2\beta }-c_{2\beta '}\right) v^2}{s_{2\beta '}} \right) s_{2\xi _W} \right] , \nonumber \\ M_{W'}^2= & {} \frac{g^2}{4}\left[ v^2s^2_{\xi _W}+\left( \frac{4 u^2+\left( 1-2c_{2\beta }c_{2\beta '} + c^2_{2\beta '}\right) v^2}{s^2_{2\beta '}}\right) c^2_{\xi _W}\right. \nonumber \\&\left. \phantom {\left[ v^2s^2_{\xi _W}+\left( \frac{4 u^2+\left( 1-2c_{2\beta }c_{2\beta '} + c^2_{2\beta '}\right) v^2}{s^2_{2\beta '}}\right) c^2_{\xi _W}\right. }-\, 2\left( \frac{\left( c_{2\beta }-c_{2\beta '}\right) v^2}{s_{2\beta '}} \right) s_{2\xi _W} \right] . \end{aligned}$$

(39)

Note that Z and W are the SM-like gauge bosons.

We will derive the approximate formulas for the mixing angles and masses of the SM-like gauge boson up to the order of $v^2\times \mathcal {O}(\epsilon ^2)$ because the corrections at this order to the masses may contribute significantly to precision tests such as the $\rho$ parameter. Because $t_{2Z}, t_{2\xi _W}\sim \epsilon ^2$, we have $s_{Z}\simeq t_{2Z}/2$, $s_{\xi _W}\simeq t_{2\xi _W}/2$. From (34) and (37), we get

$$\begin{aligned} s_Z \simeq t_Z\simeq \frac{s_{2\beta '} \left( c_{2\beta }- c_{2\beta } \right) }{4 c_W} \epsilon ^2, \; s_{\xi _W} \simeq s_Z c_W. \end{aligned}$$

(40)

This means that $s_Z^2,s^2_{\xi _W}\sim \epsilon ^4$, hence $c_Z^2=1-s_Z^2=1- \mathcal {O}(\epsilon ^4)=1, \; c^2_{\xi _W}=1$. For this reason, the masses of the gauge bosons in (35) and (39) can be written as

$$\begin{aligned} M^2_{Z}\simeq & {} \frac{g^2v^2}{4 c^2_W} \left[ 1+ \frac{(c_{2\beta }-c_{2\beta '})^2 \epsilon ^2}{4} \right] ,\nonumber \\ M^2_{W}\simeq & {} \frac{g^2v^2}{4} \left[ 1+ \frac{(c_{2\beta }-c_{2\beta '})^2 \epsilon ^2}{4} \right] , \nonumber \\ M_{Z'}^2\simeq & {} M_{W'}^2 \simeq \frac{g^2}{4} \frac{4 u^2+\left( 1-2c_{2\beta }c_{2\beta '} + c^2_{2\beta '}\right) v^2}{s^2_{2\beta '}}. \end{aligned}$$

(41)

Then we have $\frac{M^2_W}{M^2_{W'}}\simeq \frac{s^2_{2\beta '}\epsilon ^2}{4 }\simeq c_W^2\frac{M_Z^2}{M^2_{Z'}}$. In addition, at the tree level the $\rho$ parameter satisfies $\rho =\frac{M^2_W}{c^2_W M^2_Z}=1+\mathcal {O}(\frac{v^4}{u^4})$. Hence, a TeV scale of u gives a contribution to $(\rho -1)= \mathcal {O}(\frac{v^4}{u^4})\sim 10^{-4}$, in agreement with the recent experimental constraint on $\rho$ parameter [31].

3 Currents

The Lagrangian $L_{\mathrm {fermion}} = i \sum _f {\bar{f}} \gamma ^\mu D_\mu f$ contains interactions of the gauge bosons with the fermions. Let us firstly consider neutral currents. From Eq. (35), one gets

$$\begin{aligned} L_{NC}= & {} \sum _f {\bar{f}} \gamma _\mu A^\mu \left\{ g' c_W Y + \frac{g_1 g_2 s_W}{n_1}(T_3^1 + T_3^2)\right\} f \nonumber \\\end{aligned}$$

(42)

$$\begin{aligned}&+\, \sum _f {\bar{f}} \gamma _\mu Z_l^\mu \left\{ -g' s_W Y + \frac{g_1 g_2 c_W}{n_1}(T_3^1 + T_3^2)\right\} f \nonumber \\\end{aligned}$$

(43)

$$\begin{aligned}&+\, \sum _f {\bar{f}} \gamma _\mu Z_h^\mu \left( \frac{g_1^2 }{n_1}T_3^1 - \frac{g_2^2 }{n_1} T_3^2\right) f. \end{aligned}$$

(44)

Using $e = g' c_W = g s_W$, etc., the expression in (42) gives the well-known electromagnetic current $L_{em} = A_\mu J_{em}^\mu = e A_\mu \sum _f {\bar{f}} \gamma ^\mu Q f$, where Q is the electric charge operator defined in (6).

Neutral currents are defined as $L_{NC}^{Z_l,Z_h} = Z_l^\mu J_\mu (Z_l)+ Z_h^\mu J^\mu (Z_h)$, where $J_\mu (Z_l)$ and $J^\mu (Z_h)$ can be found from (43) and (44). Recall that physical neutral gauge bosons are Z and $Z'$ defined from the $Z_l$–$Z_h$ mixing angle (34), leading to the respective neutral currents

$$\begin{aligned} J^\mu (Z)= & {} \frac{ c_Z g}{c_W} \sum _f {\bar{f}} \gamma ^\mu \left( T_3^1+ T_3^2 - s^2_W Q \right) f \nonumber \\&+\, \frac{ s_Z g}{t_{\beta '}} \sum _f {\bar{f}} \gamma ^\mu \left( t^2_{\beta '} T_3^1 - T_3^2)\right) f \end{aligned}$$

(45)

and

$$\begin{aligned} J^\mu (Z')= & {} \frac{ c_Z g}{t_{\beta '}} \sum _f {\bar{f}} \gamma ^\mu \left( t^2_{\beta '}T_3^1 - T_3^2)\right) f \nonumber \\&-\, \frac{ s_Z g}{c_W} \sum _f {\bar{f}} \gamma ^\mu \left( T_3^1 + T_3^2 - s^2_W Q \right) f. \end{aligned}$$

(46)

The second term in (45) is the NP contribution.

Table 1 Couplings of Z boson with fermions, $\ell =e,\mu ,\tau$ and $\Delta _e=\Delta _d=1$

Let us write explicitly the neutral current of the Z boson

$$\begin{aligned} J^\mu (Z)= & {} \frac{ c_Z g}{c_W} \sum _{\psi = q,\ell } {\bar{\psi }} \gamma _\mu \left( T_3^2 - s^2_W Q \right) \psi \text {(as SM)}\nonumber \\&-\, \frac{g s_Z }{2t_{\beta '}} \left( {\bar{\nu }}_L \gamma _\mu \nu _L + {\bar{u}}_L \gamma _\mu u_L - {\bar{l}}_L \gamma _\mu l_L - {\bar{d}}_L \gamma _\mu d_L\right) \nonumber \\&+ \, \frac{ c_Z g}{ c_W} \left[ \frac{1}{2} {\bar{N}} \gamma _\mu N + {\bar{E}} \gamma _\mu \left( -\frac{1}{2} + s^2_W \right) E\right. \nonumber \\&+\, \left. {\bar{U}} \gamma _\mu \left( \frac{1}{2} - \frac{2}{3} s^2_W \right) U + {\bar{D}} \gamma _\mu \left( -\frac{1}{2} + \frac{1}{3} s^2_W \right) D\right] \nonumber \\&+\, \frac{g s_Z t_{\beta '}}{2}\left( {\bar{N}} \gamma _\mu N - {\bar{E}} \gamma _\mu E + {\bar{U}} \gamma _\mu U - {\bar{D}} \gamma _\mu D\right) ,\nonumber \\ \end{aligned}$$

(47)

where $T^1_3=0$ and $T^2_3=\frac{1}{2}\sigma _3$ for the SM fermion doublets, and $T^1_3=\frac{1}{2}\sigma _3, T^2_3=0$ for the extra fermion doublets. Only interactions in the first line of (47) are the SM ones. The remaining provides NP effects. Note that interactions of new vector-like fermions include both $P_L$ and $P_R$ parts (as vector).

Let us write the couplings of the Z boson with physical fermion states in the form

$$\begin{aligned} L^{NC}(Z,f) = \frac{ c_Z g}{c_W} Z_\mu {\bar{f}}\gamma ^\mu (g_L P_L + g_R P_R) f, \end{aligned}$$

(48)

where $P_{L,R} = (1\mp \gamma _5)/2$. The couplings $g_L$ and $g_R$ are listed in Table 1, where we denote

$$\begin{aligned} \rho _{\mu \tau }\equiv & {} \sqrt{1-c^2_{\beta '}(\Delta ^2_{\mu }+\Delta ^2_{\tau })},\nonumber \\ \rho _{sb}\equiv & {} \sqrt{1-c^2_{\beta '}(\Delta ^2_{s}+\Delta ^2_{b})}. \end{aligned}$$

(49)

Here we keep only significant contributions to $g_R$, in which they contain both factors of heavy masses and $\epsilon$, as shown in the two last lines of Table 1. We can see that although new fermions are all vector-like in the flavor bases, they are not vector-like in the mass bases because they are mixed with the chiral $\mathrm{SU(2)}_2$ leptons through Yukawa interactions (16).

In contrast to [12, 13], in our work the neutral currents are written in the basis of physical neutral gauge bosons, SM Z and extra $Z'$, from which their decays can easily be studied.

3.1 Charged currents

The Lagrangian of charged currents is

$$\begin{aligned} \mathcal {L}_{CC}= & {} \frac{1}{\sqrt{2}}\left\{ {\bar{\nu }}_L\gamma ^\mu g_2 W_{2 \mu }^+ l_L + {\bar{N}}_{L,R}\gamma ^\mu g_1 W_{1 \mu }^+ E_{L,R}\right. \nonumber \\&+ \, \left. {\bar{u}}_L\gamma ^\mu g_2 W_{2 \mu }^+ d_L + {\bar{U}}_{L,R}\gamma ^\mu g_1 W_{1 \mu }^+ D_{L,R} \right\} + \mathrm {H.c.}. \nonumber \\ \end{aligned}$$

(50)

In the physical states of charged gauge bosons, it is

$$\begin{aligned} \mathcal {L}_{CC}= & {} \frac{g c_{\xi _W}}{\sqrt{2}}\left( 1- \frac{t_{\xi _W}}{t_{\beta '}}\right) W_{\mu }^+\left( {\bar{\nu }}_L\gamma ^\mu l_L + {\bar{u}}_L\gamma ^\mu d_L\right) \nonumber \\&+\, \frac{g c_{\xi _W}}{\sqrt{2}}\left( 1- \frac{t_{\xi _W}}{t_{\beta '}}\right) W_{\mu }^+( {\bar{N}}\gamma ^\mu E + {\bar{U}}\gamma ^\mu D )\nonumber \\&-\, \frac{g c_{\xi _W}}{\sqrt{2}}\left( \frac{1}{t_{\beta '}} + t_{\xi _W} \right) W_{\mu }^{\prime +}\left( {\bar{\nu }}_L\gamma ^\mu l_L + {\bar{u}}_L\gamma ^\mu d_L\right) \nonumber \\\end{aligned}$$

(51)

$$\begin{aligned}&+\, \frac{g c_{\xi _W}}{\sqrt{2}}\left( \frac{1}{t_{\beta '}} + t_{\xi _W} \right) W_{\mu }^{\prime +}({\bar{N}}\gamma ^\mu E + {\bar{U}}\gamma ^\mu D ) + \mathrm {H.c}. \nonumber \\ \end{aligned}$$

(52)

If the W boson part of the Lagrangian is written as $\mathcal {L} = \frac{g c_{\xi _W}}{\sqrt{2}} W_{\mu }\overline{f}\gamma ^{\mu } \left( g_LP_L+g_RP_R\right) f' +\mathrm {H.c.}$, the couplings of the W boson with physical fermions are shown in Table 2.

Table 2 Couplings of W boson with fermions

New-physics interactions are in (51). Within the experimental data on the W decay width, ones can get constraints on the mixing angles. That was discussed in detail in Refs. [12, 13].

4 Higgs sector

From

$$\begin{aligned} \Phi = \frac{1}{\sqrt{2}} \left( \begin{array}{cc} \Phi ^0 &{} \Phi ^+ \\ -\Phi ^- &{} \tilde{\Phi }^0\\ \end{array}\, \right) \rightarrow \Phi ^\dag = \frac{1}{\sqrt{2}} \left( \begin{array}{cc} \tilde{\Phi }^0 &{} - \Phi ^+ \\ \Phi ^- &{} \Phi ^0\\ \end{array}\, \right) ,\nonumber \\ \end{aligned}$$

(53)

the potential is given as

$$\begin{aligned} V= & {} \mu _\phi ^2 \phi ^\dag \phi + \mu _{\phi '}^2 \phi '^\dag \phi ' + \mu _\Phi ^2 \mathrm{Tr}(\Phi ^\dag \Phi ) \nonumber \\&+\, \frac{\lambda _1}{2} (\phi ^\dag \phi )^2 + \frac{\lambda _2}{2} (\phi '^\dag \phi ')^2 + \frac{\lambda _3}{2} [\mathrm{Tr}(\Phi ^\dag \Phi )]^2\nonumber \\&+\, \lambda _4 (\phi ^\dag \phi )(\phi '^\dag \phi ') + \mathrm{Tr}(\Phi ^\dag \Phi )\nonumber \\&[\lambda _5 (\phi ^\dag \phi ) + \lambda _6 (\phi '^\dag \phi ')] - \mu (\phi '^\dag \Phi \phi + \mathrm {H. c.}). \end{aligned}$$

(54)

Because the $\mu$ parameter is proportional to the squared masses of the charged and CP-odd Higgs bosons, it must be positive with the minus sign before it in the potential (54).

The neutral scalars are expanded as

$$\begin{aligned}&\varphi ^0 = \frac{1}{\sqrt{2}} (v_\phi + S_\phi + i A_\phi ), \varphi '^0 = \frac{1}{\sqrt{2}} (v_{\phi '} + S_{\phi '} + i A_{\phi '}) ,\nonumber \\&\phi ^0 = \frac{1}{\sqrt{2}} (u+ S_\Phi + i A_\Phi ). \end{aligned}$$

(55)

At the tree level, the minimum conditions of the Higgs potential are similar to the ones in Refs. [12, 13], except the opposite signs of $\mu$.

$$\begin{aligned}&\mu _{\phi }^2 +\frac{1}{2}\left( \lambda _1 v^2_\phi + \lambda _4 v_{\phi '}^2 + \lambda _5 u^2 - \frac{u\mu }{t_{\beta }}\right) = 0,\nonumber \\&\mu _{\phi '}^2 + \frac{1}{2}\left( \lambda _2 v^2_{\phi '} + \lambda _4 v_{\phi }^2 + \lambda _6 u^2 -u \mu t_{\beta } \right) = 0, \\&\mu _{\Phi }^2 + \frac{1}{2}\left( \lambda _5 v^2_\phi + \lambda _6 v_{\phi '}^2 + \lambda _3 u^2 -\frac{\mu }{u}v_{\phi }v_{\phi '} \right) = 0\, .\nonumber \end{aligned}$$

(56)

Based on the minimum conditions, the parameter $\mu ^2_{\phi ,\phi ',\Phi }$ can be expressed as a function of the Higgs-self couplings u, v and $\beta$. Next, the masses, mass eigenstates, and couplings of Higgs bosons will be calculated by inserting these functions into the Higgs potential (54).

4.1 Squared mass matrices of the Higgs bosons

In the original bases of singly charged and CP-odd neutral Higgs bosons $\phi ^{\pm }=(\varphi ^\pm ,\; \varphi '^\pm ,\;\Phi ^\pm )^T$ and $A=(A_\phi ,\; A_{\phi '},\;A_{\Phi } )^T$, the corresponding squared mass matrices are

$$\begin{aligned}&M^2_{h^\pm }= \frac{\mu }{2} \left( \begin{array}{ccc} \frac{uc_{\beta }}{s_{\beta }} &{}-u &{}v c_{\beta } \\ &{} \frac{us_{\beta }}{c_{\beta }} &{}-vs_{\beta } \\ &{} &{} \frac{v^2s_{\beta }c_{\beta }}{u}\\ \end{array} \right) , \nonumber \\&M^2_{A}= \frac{\mu }{2} \left( \begin{array}{ccc} \frac{uc_{\beta }}{s_{\beta }} &{}-u &{}-v c_{\beta } \\ &{} \frac{us_{\beta }}{c_{\beta }} &{}vs_{\beta } \\ &{} &{} \frac{v^2s_{\beta }c_{\beta }}{u}\\ \end{array} \right) . \end{aligned}$$

(57)

In the basis of CP-even Higgs bosons $S=(S_\phi , S_{\phi '} , S_\Phi )^T$ the squared mass matrix $M_S^2$ corresponding the mass term $\frac{1}{2} S^T M_S^2 S$ is

$$\begin{aligned} M^2_S = \left( \begin{array}{ccc} \frac{u\mu c_{\beta }}{2s_{\beta }}+ \lambda _1 v^2 s^2_{\beta } &{}-\frac{1}{2}u\mu + \lambda _4 v^2 s_{\beta }c_{\beta } &{}-\frac{1}{2}v\mu c_{\beta }+ \lambda _5 u vs_{\beta } \\ &{}\frac{u\mu s_{\beta }}{2c_{\beta }} + \lambda _2 v^2 c^2_{\beta } &{} -\frac{1}{2}v\mu s_{\beta }+ \lambda _6 u v c_{\beta }\\ &{} &{}\frac{v^2 \mu s_{\beta }c_{\beta }}{2u}+ \lambda _3 u^2\\ \end{array}\, \right) . \nonumber \\ \end{aligned}$$

(58)

The above matrices are consistent with those given in [12, 13] after using the relations (56).

4.2 Physical spectrum of Higgs bosons and their couplings

We will find the Higgs bosons masses in two steps. At the first step, where $v\rightarrow 0$, all the three squared mass matrices are diagonalized through the same transformation

$$\begin{aligned} C_1 = \left( \begin{array}{ccc} s_{\beta } &{} c_{\beta } &{} 0 \\ 0&{} 0&{} 1 \\ c_{\beta }&{}-s_{\beta } &{} 0 \\ \end{array} \right) . \end{aligned}$$

(59)

In the second step, it is easy to determine rotations diagonalizing the squared mass matrices of charged and CP-odd neutral Higgs bosons. By defining the mixing angle $\zeta$ as

$$\begin{aligned} \sin \zeta \equiv \frac{v s_{\beta }c_{\beta }}{\sqrt{u^2+(v s_{\beta }c_{\beta })^2}} , \cos \zeta \equiv \frac{u}{\sqrt{u^2+(v s_{\beta }c_{\beta })^2}}, \nonumber \\ \end{aligned}$$

(60)

the total mixing matrices used to diagonalize mass matrices in (57) are

$$\begin{aligned} C_{h^\pm } =\left( \begin{array}{ccc} -c_{\beta }s_{\zeta } &{} s_{\beta }s_{\zeta } &{} c_{\zeta } \\ s_{\beta } &{} c_{\beta } &{} 0 \\ c_{\beta }c_{\zeta } &{} - s_{\beta }c_{\zeta } &{} s_{\zeta } \\ \end{array} \right) , C_{A} =\left( \begin{array}{ccc} c_{\beta }s_{\zeta } &{}- s_{\beta }s_{\zeta } &{} c_{\zeta } \\ s_{\beta } &{} c_{\beta } &{} 0 \\ c_{\beta }c_{\zeta } &{} -s_{\beta }c_{\zeta } &{} -s_{\zeta } \\ \end{array} \right) . \nonumber \\ \end{aligned}$$

(61)

Mass eigenstates of the charged and CP-odd Higgs bosons, denoted as $H^{\pm }=(G^\pm _{1},\; G^\pm _{2}, h^{\pm } )^T$ and $H_A=(G_{Z_1},\;G_{Z_2},h_{a})^T$, relate with the original states through the following equations:

$$\begin{aligned} \phi ^{\pm }= C^T_{h^\pm } H^{\pm }, \mathrm {and}\; A= C^T_{A} H_A. \end{aligned}$$

(62)

Two linear combinations of $G^\pm _{1}$ and $G^\pm _{2}$ are Goldstone bosons eaten up by the $W'^\pm$ and $W^\pm$ gauge bosons. Similarly, linear combinations of $G_{Z_1}$ and $G_{Z_2}$ are eaten up by Z and $Z'$. There are two physical charged Higgs bosons $h^{\pm }$ and one physical CP-odd neutral Higgs boson $h_a$ with masses $m^2_{h^\pm }$ and $m^2_{A}$, respectively. They satisfy $C_{h^\pm } M^2_{h^\pm } C_{h^\pm }^T=\mathrm {diag}(0,0,m^2_{h^\pm })$ and $C_{A} M^2_{A} C_{A}^T=\mathrm {diag}(0,0,m^2_{A})$, where

$$\begin{aligned} m^2_{h^\pm }= m^2_{A}= \frac{\mu \left( u^2+ v^2s^2_{\beta }c^2_{\beta }\right) }{2 us_{\beta }c_{\beta }}. \end{aligned}$$

(63)

Regarding the CP-even neutral Higgs bosons, after the rotation (59), the squared mass matrix is $M'^2_S =C_1M^2_SC_1^T$, which is a $3\times 3$ matrix with the following elements:

$$\begin{aligned} \left( M'^2_S\right) _{11}= & {} v^2\left( \lambda _1 s^4_{\beta } +\lambda _2 c^4_{\beta } +2\lambda _4 s^2_{\beta }c^2_{\beta } \right) , \nonumber \\ \left( M'^2_S\right) _{12}= & {} \left( M'^2_S\right) _{21}= v\left[ u\left( \lambda _5 s^2_{\beta } +\lambda _6 c^2_{\beta }\right) -\mu s_{\beta }c_{\beta }\right] , \nonumber \\ \left( M'^2_S\right) _{13}= & {} \left( M'^2_S\right) _{31}= v^2 s_{\beta }c_{\beta }\left[ \left( \lambda _4-\lambda _1\right) s^2_{\beta } +\left( \lambda _2-\lambda _4\right) c^2_{\beta } \right] , \nonumber \\ \left( M'^2_S\right) _{22}= & {} \lambda _3 u^2+ \frac{\mu v^2 s_{\beta } c_{\beta }}{2u}, \nonumber \\ \left( M'^2_S\right) _{23}= & {} \left( M'^2_S\right) _{32}=\frac{v}{2}\left[ \left( c^2_{\beta } -s^2_{\beta }\right) \mu +\left( \lambda _6- \lambda _5 \right) u 2s_{\beta }c_{\beta } \right] , \nonumber \\ \left( M'^2_S\right) _{33}= & {} \frac{\mu u}{2s_{\beta }c_{\beta }}+ \left( \lambda _1+ \lambda _2- 2\lambda _4 \right) v^2 s^2_{\beta }c^2_{\beta }. \end{aligned}$$

(64)

In general, $M'^2_S$ is complicated and it cannot be diagonalized exactly. Instead, using the parameter $\epsilon \equiv v/u\ll 1$, we will find approximate solutions for mass eigenvalues, keep terms up to the order of the electroweak scale. This is reasonable because the SM-like Higgs boson mass was found to be 125 GeV. Approximate solutions was used earlier to find consistent masses of the lightest CP-even neutral Higgs bosons in supersymmetric models [40, 41]. The mixing matrix will also be determined approximately, corresponding to the mass eigenvalues.

Table 3 $h^0_1$ couplings , where $f_i=e_i,u_i,d_i$; $q_i=u_i,d_i$ ($i=1,2,3)$; $q_1=u,d$; $Q_1=U_1,D_1$; $F_1=E_1,U_1,D_1$

We start from finding the eigenvalues of the matrix $M'^2_S$ by solving the equation $\mathrm { Det}\left( M'^2_S-\lambda I_3 \right) =0$, where $\lambda$ is expanded as $\lambda = u^2 (\lambda _0+\lambda _1 \epsilon ^2)$ to keep it up to the order of electroweak scale v. We assume that $\lambda _0,\lambda _1\sim \mathcal {O}(1)$. Using $v=u \epsilon$, we can write $\mathrm { Det}\left( M'^2_S-\lambda I_3 \right) =a_0 +a_1\epsilon ^2+ \mathcal {O}(\epsilon ^4)=0$ where $a_0=a_0 (\lambda _0)$ and $a_1=a_1(\lambda _0,\lambda _1)$. We will consider only the two following equations:

$$\begin{aligned}&a_0(\lambda _0) = -\lambda _0\left( \lambda _0-\frac{\mu u}{2s_{\beta }c_{\beta }} \right) \left( \lambda _0- \lambda _3 u^2\right) , \nonumber \\&a_1(\lambda _0,\lambda _1) \left. \frac{}{}\right| _{\lambda _0=0} \sim u^2\left[ \left( \lambda _1 s^4_{\beta } +\lambda _2 c^4_{\beta } +2\lambda _4 s^2_{\beta }c^2_{\beta }\right) \right. \nonumber \\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\left. -\, \frac{1}{\lambda _3}\left( \lambda _5 s^2_{\beta } +\lambda _6 c^2_{\beta } - \frac{\mu }{u} s_{\beta }c_{\beta } \right) ^2 \right] . \nonumber \\ \end{aligned}$$

(65)

The first equation in (65) shows that the largest contributions to Higgs masses are the solutions of $a_0(\lambda _0)=0$, giving one zero and two non-zero values, $\lambda _0= \frac{\mu u}{2s_{\beta }c_{\beta }}$ and $\lambda _0= \lambda _3 u^2$. Hence there are two heavy CP-even neutral Higgs bosons with the corresponding masses $m^2_{h^0_2}= \lambda _3 u^2 + \mathcal {O}(v^2)$ and $m^2_{h^0_3} = \frac{\mu u}{2s_{\beta }c_{\beta }} + \mathcal {O}(v^2)$, which equal the largest contributions of the two last diagonal entries of $M'^2_S$ shown in (64). A light CP-even neutral Higgs boson corresponds to $\lambda _0=0$. Its mass comes from the second equation of (65):

$$\begin{aligned} m^2_{h^0_1}\simeq & {} v^2 \left[ \left( \lambda _1 s^4_{\beta } +\lambda _2 c^4_{\beta } +2\lambda _4 s^2_{\beta }c^2_{\beta }\right) \right. \nonumber \\&\left. - \frac{1}{\lambda _3}\left( \lambda _5 s^2_{\beta } +\lambda _6 c^2_{\beta } - \frac{\mu }{u} s_{\beta }c_{\beta } \right) ^2\right] . \end{aligned}$$

(66)

We stress that $m^2_{h^0_1}\ne \left( M'^2_S\right) _{11}$, i.e. contributions from non-diagonal entries of $M'^2_S$ to $m^2_{h^0_1}$ cannot be ignored. The mixing matrix in this case can be found based on a mixing angle defined by $c_h\equiv \cos \xi _h$ and $s_h\equiv \sin \xi _h$ satisfying

$$\begin{aligned} t_{2h}= \tan 2\xi _h=-\frac{2\left( M'^2_S\right) _{12 }}{\left( M'^2_S\right) _{22 } -\left( M'^2_S\right) _{11 }}\sim \frac{v}{u}=\epsilon . \end{aligned}$$

(67)

It can be checked that after this rotation the light Higgs boson mass is consistent with (66). Therefore, the mixing matrix relating two original and physical bases S and $H_0=( h^0_1,h^0_2,h^0_3)^T$ is $S=C_{h}^T H_0$, where

$$\begin{aligned} C_{h}\simeq C_1 C_{h2}= \left( \begin{array}{ccc} s_{\beta } c_h &{}c_{\beta } c_h &{} s_h \\ -s_{\beta }s_h &{}-c_{\beta } s_h &{} c_h \\ c_{\beta } &{} - s_{\beta } &{} 0 \\ \end{array} \right) . \end{aligned}$$

(68)

The light Higgs boson $h^0_1$ is identified with the SM-like Higgs boson found by the LHC. The recent experimental data shows that the SM predictions perfectly agree with the observation within 1 sigma [32]. Hence, couplings of $h^0_1$ with other SM particles must be consistent with this data. The relevant couplings of $h^0_1$ are shown in Table 3, including couplings with $h^\pm _2$ needed to generate active neutrino masses. We can see easily that all couplings with the SM-like particles are different from the SM predictions by a common factor $c_h$. So, $|c_h|$ should be close to unity, i.e., $|s_h|$ should be small. Its upper bound can be found as follows. Consider the $h^0_1$ productions at LHC, new heavy quarks can play the roles of the top quark in gluon–gluon fusion mechanism, where their couplings are proportional to $s_h$ or $\epsilon$. A significant contribution related to $\epsilon$ may come from the quarks $U_2$ where the couplings contain a factor $(\Delta _b m_t)^2\epsilon /v^2$. But the constraint from [12, 13] gives $(\Delta _b m_t)^2\epsilon /v^2\sim 10^{-4}\epsilon$, which is suppressed. Now, the production of $h^0_1$ through gluon–gluon fusion at lowest order is [42, 43]

$$\begin{aligned} \sigma ^0_{h^0_1}=\frac{G_F \alpha _s^2 }{288\sqrt{2}\pi }\left| \frac{3}{4}\sum _{q}\frac{g_{hqq}v}{m_q} A_{1/2}(t_{q})\right| ^2, \end{aligned}$$

(69)

where $t_{q}=\frac{m^2_{h^0_1}}{4 m_q^2}$, $m_q$ is a quark mass; $g_{hqq}=c_h \frac{m_q}{v}$, $s_h c^2_{\beta '}\widetilde{M}_{Q_1}$, and $s_h c^2_{\beta '}\left( \Delta ^2_s +\Delta ^2_b\right) \widetilde{M}_{Q_2}$ for SM-like quarks; new quarks $U_1,D_1$; and new quarks $U_2,D_2$, respectively. The form factor $A_{1/2}(t)$ is determined as

$$\begin{aligned} A_{1/2}(t)=2\left[ t+(t-1)f(t)\right] t^{-2} , \end{aligned}$$

(70)

where

$$\begin{aligned} f(t)\equiv \left\{ \begin{matrix} \arcsin ^2\left( \sqrt{t}\right) &{} \ \mathrm {for}\ t\le 1\\ -\frac{1}{4}\left( -i\pi +\ln \left[ \frac{1+\sqrt{1-t^{-1}}}{1-\sqrt{1-t^{-1}}}\right] \right) ^2 &{} \mathrm {for}\ t> 1.\\ \end{matrix}\right. \end{aligned}$$

Using $m_{Q_{1,2}}$, $m_{Q_{1,2}}\simeq \widetilde{ M}_{Q_{1,2}}u$, as given in [12, 13], Eq. (69) is written as

$$\begin{aligned} \sigma ^0_{h^0_1}= & {} \frac{G_F \alpha _s^2 }{288\sqrt{2}\pi }\left| \frac{3}{4} c_h A_{1/2}(t_{0}) + \frac{3}{2} c^2_{\beta '}s_h \epsilon \right. \nonumber \\&\left. \left[ A_{1/2}(t_{1})+(\Delta _{s}^2+\Delta ^2_b)A_{1/2}(t_{2})\right] \right| ^2, \end{aligned}$$

(71)

where $t_0=\frac{m_{h^0_1}}{4m^2_t}$ and $t_{1,2}=\frac{m_{h^0_1}}{4m^2_{Q_{1,2}}}$. The condition $m_t, m_{Q_{1,2}}> m_{h^0_1}=125.09$ GeV gives the limit $A_{1/2}(t)\rightarrow 4/3$ for all $t=0,1,2$. The respective signal strength of Higgs production is

$$\begin{aligned} \mu _{ggF}= & {} \frac{\sigma (pp\rightarrow h^0_1)_{\mathrm {221}}}{\sigma (pp\rightarrow h)_{\mathrm {SM}}}\nonumber \\= & {} \left| c_h +2c^2_{\beta '}s_h\epsilon \left[ 1+\Delta _{s}^2+\Delta ^2_b\right] \right| ^2, \end{aligned}$$

(72)

where we follow the notations of signal strengths defined in [32]. Similarly, the partial decay width of the channel $h^0_1\rightarrow gg$ is determined as

$$\begin{aligned} \Gamma (h^0_1\rightarrow gg)_{221}= \mu ^{hgg}\times \Gamma (h\rightarrow gg)_{\mathrm {SM}}, \end{aligned}$$

(73)

where $\mu ^{hgg}=\mu _{ggF}$. Because $s_h \epsilon =\mathcal {O}(\epsilon ^2)$ and the branching ratio of this decay is smaller than $9\%$, we will use the naive approximation $\mu _{ggF}\simeq c^2_h$ to find a lower bound of $|c_h|$.

For all remaining decay channels of the SM-like Higgs boson into SM particles, the tree-level couplings are always different from the SM prediction the factor $c_h$, therefore $\mu ^f=c_h^2$ for all main decays $f=f\bar{f},\;WW^*,\;ZZ^*$. The global signal strength defined in [32] can be formulated approximately as follows:

$$\begin{aligned} \mu ^f_i(\mathrm {global})= & {} \mu _i\times \mu ^f\simeq |c_h|^4=1.09\pm 0.11, \rightarrow 0.98\\\le & {} |c_h|^4\le 1. \end{aligned}$$

This gives the constraint

$$\begin{aligned} 0.995\le |c_h|\le 1,\mathrm {and}\; |s_h|\le 0.10. \end{aligned}$$

(74)

If we use the constraints of $\Delta _{b,s}$ and $c_{\beta '}$ given in [12, 13], the values of $|s_h|$ satisfying (74) are reasonable for the approximation we have just discussed. In addition, the constraint of $s_h$ results in small couplings of $h^0_1$ with the heavy fermions and $W'$ gauge boson, giving their suppressed contributions to the decay rate $h^0_1\rightarrow \gamma \gamma$. Couplings $h^0_1h^{\pm }_{1,2}h^{\pm }_{1,2}$ depend on many unknown Higgs-self couplings, implying that charged Higgs masses are not constrained from the experimental data of the decay $h^0_1\rightarrow \gamma \gamma$, so we will not consider this decay further.

4.3 Singly charged Higgs bosons with additional $\delta ^\pm$

In this section we consider the model including new singly charged Higgs bosons $\delta ^\pm$ discussed in the neutral lepton sector. Apart from the second term in (25), the Higgs potential has new terms,

$$\begin{aligned} \Delta V_h= & {} \mu ^2_4(\delta ^+\delta ^-)+ \lambda '_{0}(\delta ^+\delta ^-)^2 + (\delta ^+\delta ^-)\nonumber \\&\times \left[ \lambda '_1 \mathrm {Tr}(\Phi ^\dagger \Phi ) + \lambda '_2\phi ^\dagger \phi + \lambda '_3\phi '^\dagger \phi ' \right] . \end{aligned}$$

(75)

The appearance of $\delta ^\pm$ does not change the allowed regions of parameters discussed in [12, 13]. Also, the results derived for Higgs bosons are unchanged, except the singly charged Higgs sector. In the basis $(\varphi ^\pm ,\; \varphi '^\pm ,\Phi ^\pm ,\delta ^\pm )^T$, the squared mass matrix is denoted as $\mathcal {M}^2_{h^\pm }$. We can find a matrix $C'_{h^\pm }$ so that $\mathcal {M}'^2_{h^\pm }=C'_{h^\pm }\mathcal {M}'^2_{h^\pm }C'_{h^\pm T}$ has only the following non-zero elements: $\left( \mathcal {M}'^2_{h^\pm }\right) _{34}=\left( \mathcal {M}'^2_{h^\pm }\right) _{43}=\frac{1}{2}\lambda _{\delta } v\sqrt{u^2+(vs_{\beta }s_{\beta })^2}$, $\left( \mathcal {M}'^2_{h^\pm }\right) _{33}=\frac{\mu \left[ u^2+(vs_{\beta }c_{\beta })^2\right] }{2us_{\beta }c_{\beta }}$ and $\left( \mathcal {M}'^2_{h^\pm }\right) _{44}=\mu ^2_4+\frac{1}{2}v^2\left( \lambda '_1 s^2_{\beta }+ \lambda '_2 c^2_{\beta } \right) + \frac{1}{2}\lambda '_3 u^2$. This matrix is diagonalized by a transformation relating with a mixing angle $\xi$ satisfying

$$\begin{aligned} t_{2\xi }\equiv \tan 2\xi =-\frac{2 \left( \mathcal {M}'^2_{h^\pm }\right) _{34}}{\left( \mathcal {M}'^2_{h^\pm }\right) _{44} -\left( \mathcal {M}'^2_{h^\pm }\right) _{33}}. \end{aligned}$$

(76)

Then the total transformation can be found to be

$$\begin{aligned} C_{\pm }= \left( \begin{array}{cccc} -c_{\beta }s_{\zeta }&{} s_{\beta } s_{\zeta }&{}c_{\zeta } &{} 0\\ s_{\beta } &{}c_{\beta } &{}0 &{}0 \\ c_{\beta }c_{\zeta }c_{\xi } &{} -s_{\beta }c_{\zeta }c_{\xi } &{}s_{\zeta }c_{\xi } &{} -s_{\xi } \\ c_{\beta }c_{\zeta }s_{\xi } &{} -s_{\beta }c_{\zeta }s_{\xi } &{} -s_{\zeta }s_{\xi } &{} c_{\xi }\\ \end{array} \right) , \end{aligned}$$

which changes the original basis into the mass eigenstate basis $( G^\pm _1, \; G^\pm _2,\;h^\pm _1,\; h^\pm _2)^T$, namely $(\varphi ^\pm ,\; \varphi '^\pm ,\; \Phi ^\pm ,\delta ^\pm )^T=C^T_{\pm }( G^\pm _1,\; G^\pm _2,\; h^\pm _1,\; h^\pm _2)^T$. We note that the Goldstone bosons $G^\pm _{1,2}$ defined in (62) are not affected by the presence of $\delta ^\pm$.

The masses $m^2_{h^\pm _{1,2}}$ of $h^{\pm }_{1,2}$ are solutions of the equation $\left[ x-\left( \mathcal {M}'^2_{h^\pm }\right) _{33} \right] \left[ x- \left( \mathcal {M}'^2_{h^\pm }\right) _{44} \right] + \left( \mathcal {M}'^2_{h^\pm }\right) _{34}^2=0$. If $\left( \mathcal {M}'^2_{h^\pm }\right) _{34}\ll \left( \mathcal {M}'^2_{h^\pm }\right) _{33},\left( \mathcal {M}'^2_{h^\pm }\right) _{44}$, we have $h^\pm _1\equiv h^\pm$ given in (62) and $h^\pm _2\equiv \delta ^\pm$, which is used for simple approximations because Eq. (76) means $t_{2\xi }\sim \epsilon \ll 1$. The relevant couplings of the charged Higgs bosons to fermions are collected in Table 4.

Table 4 Couplings of charged Higgs boson $h^+_1$. Couplings of $h^{\pm }_2$ are only different from those of $h^\pm _1$ by replacements: $c_{\xi }h^\pm _1 \rightarrow s_{\xi } h^\pm _2$ and $s_{\xi } h^\pm _1\rightarrow -c_{\xi } h^\pm _2$

Couplings of the charged Higgs bosons with gauge bosons are shown in Table 5. Only couplings of $h^\pm _1$ are shown because the couplings of $h^\pm _2$ can be derived by the following replacements: $c_{\xi }h^\pm _1 \rightarrow s_{\xi } h^\pm _2$ and $s_{\xi } h^\pm _1\rightarrow -c_{\xi } h^\pm _2$. We consider here only the case of $\xi \rightarrow 0$.

Some important properties of $h^\pm _1$ are as follows. Differences between couplings of $h^\pm _1$ and the SM-like Higgs bosons to normal fermions are $g_{h^\pm _1\ell \nu _{\ell }}/g_{h^0_1\ell \ell }=\frac{\sqrt{2}c_{\xi }c_{\zeta }}{t_{\beta }}$ and $g_{h^\pm _1u_i d_i}/g_{h^0_1u_iu_i}=-\frac{\sqrt{2}c_{\xi }c_{\zeta }}{t_{\beta }}( P_L-\frac{m_{u_i}}{m_{d_i}}P_R)$. From Table 5, the couplings of $h^\pm _1$ to the SM-like bosons, namely $h^\pm _1 Z W$ and $h^\pm _1 h^0_1 W$, are extremely small because they contain factors $s_Z\epsilon ^2\sim \mathcal {O}(\epsilon ^4)$, $s_Z\sim \mathcal {O}(\epsilon ^2)$, and another mixing smaller than 0.02. Other couplings to light fermions are also small because $s_{\xi }\rightarrow 0$. With $c_{\zeta }, c_{\xi }\rightarrow 1$, the main decays of $h^\pm _1$ into light particles are $h^+_1\rightarrow t\overline{b}$. If $m_{h^{\pm }_1}> m_{W'}, m_{Z'}$, there will appear two additional large decay modes $h^+_1\rightarrow Z'W^+, ZW'^+$.

In contrast to $h^\pm _1$, the charged Higgs bosons $h^\pm _2$ only couple strongly with leptons and Higgs bosons. Therefore, the main decay modes are $h^+_2\rightarrow \overline{(\nu _{e_i})^c}e_j$ with $e_{j}=e,\mu ,\tau$. The main processes for $h^{\pm }_2$ production at colliders are $\overline{f}f\rightarrow \gamma ^*,h^{0*}_{1,2,3},h^*_a\rightarrow h^+_2h^-_2$, $f=e,u,d$.

We would like to compare the above singly charged Higgs bosons with the ones predicted by Zee models, where the charged Higgs sector was investigated thoroughly in Ref. [33]. The equivalent notations are $v_1, v_2, \tan {\beta }=v_2/v_1\leftrightarrow v_{\phi },v_{\phi '}, 1/t_{\beta }=v_{\phi '}/v_{\phi }$, and $\chi \leftrightarrow \xi$. With $\zeta =\mathcal {O}(\epsilon )\rightarrow 0$, the charged components of $\Phi$ are Goldstone bosons of $W'^\pm$. Meanwhile those of $\phi$ and $\phi '$ create two Goldstone bosons of $W^\pm$, and two other freedoms that mix with the singlet ones to generate physical states. Hence, the model under consideration and the Zee models predict very similar properties of the charged Higgs couplings to SM-like particles. But the predictions for charged Higgs boson production at colliders like the LHC are different, because of the appearance of new particles, such as new heavy quarks and Higgs bosons, and the constraints from allowed regions of parameters indicated in [12, 13]. We will review these regions before discussing the signal of new particles at colliders.

5 Phenomenology

5.1 Properties of masses and mixing parameters of new particles

In this work, the results of parameter constraints reported in Refs. [12, 13] are still valid. They will be used to discuss the Higgs phenomenology. These allowed regions are

$$\begin{aligned} \Delta _s\in & {} [-1.16,-0.97],\Delta _b \in [0.003,0.007], \nonumber \\&\Delta _{\mu } \in [0.94, 0.99], \Delta _{\tau } \in [0,0.11], \nonumber \\ M_{Z'}\in & {} [500, 1710]\; \mathrm {GeV}, \frac{g}{s_{\beta '}} \in [1.2,3.5], \nonumber \\&\zeta '\equiv (s^2_{\beta }-t^2_{\beta '}c^2_{\beta }) \in [0,0.02], \end{aligned}$$

(77)

where $g=\frac{2m_W}{v}\simeq 0.651$ is the SM gauge coupling, and $\zeta '$ satisfies $t_{\xi _W}=c_W t_{Z}\simeq c^3_{\beta '}s_{\beta '}\zeta '\epsilon ^2$ [12, 13]. This gives the constraints

$$\begin{aligned}&s_{\beta '}\in [0.186,0.542],t_{\beta }\in [0.24,0.654], \nonumber \\&\mathrm {and}\; 0\le t_{Z}=\frac{t_{\xi _W}}{c_W}<8\times 10^{-3}\epsilon ^2. \end{aligned}$$

(78)

Table 5 Couplings of $h^+_1$ with bosons. Momenta of $h^+_1$ and $h^0_1$ are $p_+$ and $p_0$, respectively

We can see that the allowed values of $\zeta$ give very small values of $t_{Z,\xi _W}$, even with large $\epsilon <1$. For simplicity, we will also use the following simple approximations:

$$\begin{aligned}&|\Delta _{\mu ,s}|\simeq 1, \Delta _{\tau ,b},\frac{\Delta _{\tau ,b} m_{\tau ,t}}{v}\ll 1, \nonumber \\&\Delta ^2_{\mu ,s}+\Delta ^2_{\tau ,b} \simeq 1, \rho _{\mu \tau ,bs}\simeq s_{\beta '}. \end{aligned}$$

(79)

The simple texture of $\lambda _\ell$ in (20) gives $M_{L_1}= u \widetilde{M}_{L_1} s_{\beta '}$ and $M_{L_2}= u \widetilde{M}_{L_2}\rho _{\mu \tau }$. For the quark sector, $M_{Q_1}=u \widetilde{M}_{Q_1} s_{\beta '}$ and $M_{Q_2}= u \widetilde{M}_{Q_2}\rho _{sb}$. Now the masses of the heavy particles are

$$\begin{aligned}&m_{E_1}=M_{N_1} + \frac{m^2_{e}(s_{\beta }-\frac{1}{2})}{ M_{N_1} s^2_{\beta }t^2_{\beta '}}, \nonumber \\&m_{U_1,D_1}= u\widetilde{M}_{Q_1} +\frac{m^2_{u,d}(s_{\beta }-\frac{1}{2})}{ u\widetilde{M}_{Q_1} s^2_{\beta }t^2_{\beta '}}, \nonumber \\&m_{E_2}=M_{N_2} - \frac{c^2_{\beta '}(\Delta ^2_{\mu }m^2_{\mu }+ \Delta ^2_{\tau }m^2_{\tau })(s_{\beta } -\frac{1}{2})}{ M_{N_2} s^2_{\beta }\rho ^2_{\mu \tau }}, \nonumber \\&m_{U_2,D_2}= u\widetilde{M}_{Q_2} -\frac{c^2_{\beta '}(\Delta ^2_{s}m^2_{c,s} + \Delta ^2_{b}m^2_{t,b})(s_{\beta }-\frac{1}{2})}{ u\widetilde{M}_{Q_2} s^2_{\beta }\rho ^2_{sb}}, \nonumber \\&m_{W'},m_{Z'}\simeq \frac{g u}{s_{2\beta '}}+ \frac{s^2_{2\beta }+ (c_{2\beta }-c_{2\beta '})^2}{8} m_W \times \epsilon + \mathcal {O}(\epsilon ^3), \nonumber \\&m_{h^0_3},\;m_A,\; m_{h^\pm _1}\simeq \sqrt{\frac{u\mu }{s_{2\beta }}} + v\times \mathcal {O}\left( \frac{vs_{2\beta }}{\sqrt{u\mu }}\right) , \end{aligned}$$

(80)

and the mixing parameters

$$\begin{aligned} s_{Z}\simeq & {} \frac{s_{\xi _W}}{c_W}\simeq \frac{t_{2\xi _W}}{2c_W} \simeq \frac{1}{4} s_{2\beta '} \left( c_{2\beta }-c_{2\beta '}\right) \epsilon ^2 , \nonumber \\ s_{\zeta }\simeq & {} s_{\beta }c_{\beta }\epsilon , c_{\zeta } \simeq 1- \frac{1}{2} (s_{\beta }c_{\beta })^2\epsilon ^2, \nonumber \\ s_{h}\simeq & {} \frac{\lambda _5 s^2_{\beta } +\lambda _6 c^2_{\beta } -\frac{\mu }{u}s_{\beta }c_{\beta }}{\lambda _3}\times \epsilon , c_h \simeq 1 -\frac{s^2_h}{2}, \nonumber \\ s_{\xi }\simeq & {} \frac{\lambda _{\delta }}{2\mu ^2_4/u^2+\frac{\mu }{(us_{\beta }c_{\beta })}+\lambda '_{3}}\times \epsilon , c_{\xi }\simeq 1-\frac{s^2_{\xi }}{2}. \end{aligned}$$

(81)

Inserting formula of $s_h$ into Eq. (66), we have a simple expression for $m_{h^0_1}$ as follows

$$\begin{aligned} m^2_{h^0_1}\simeq & {} v^2 \left( \lambda _1 s^4_{\beta } +\lambda _2 c^4_{\beta } +2\lambda _4 s^2_{\beta }c^2_{\beta }\right) - \lambda _3 u^2s_h^2. \end{aligned}$$

(82)

In addition, there are two other Higgs bosons with the masses

$$\begin{aligned}&m_{h^0_2}\simeq \sqrt{\lambda _3}u, \mathrm {and}\nonumber \\&m^2_{h^\pm _2}\simeq \mu ^2_{4} +\frac{v^2}{2}\left( \lambda '_1s^2_{\beta }+ \lambda '_2c^2_{\beta }\right) + \frac{\lambda '_3}{2}u^2. \end{aligned}$$

(83)

The important property is that the model predicts several groups of new heavy particles having same spins and degenerate masses, therefore forbid many decay modes. The new Yukawa couplings generating heavy mass terms are always the same for both up and down components of the $\mathrm{SU(2)}_1$ fermion doublets in the same families. Mass differences come only from the Yukawa terms of the electroweak sector (13), because of the large $V_{Q,L}$. As shown in (80), a difference between a pair of new up and down fermions is $\mathcal {O}(m_f)$. Hence, the top quark may give the largest difference if $\Delta _{b}m_t<1.4$ is not considered. Hence, the mass differences are always smaller than $m_W$. Because of the kinetic condition and the fermion number conservation, a three-body decay of a new fermion must decay to at least a light fermion, namely a SM fermion plus a boson. If a fermion is the lightest among the new particles, it will decay only to a light fermion and a SM boson like W, Z or $h^0_1$, leading to large branching ratios, which can be searched by recent colliders.

5.2 Searches for new fermions at colliders

From the above discussion, if the new fermions are lighter than all new bosons, including $W'^\pm , Z', h^0_{2,3}$, $h^\pm _{1,2}$, they have only the following three-body decays:

• For the first family of new fermions:

  $$\begin{aligned}&N_{1}\rightarrow \nu _e h^0_1,\; \nu _e Z,\; e^\pm W^{\mp }; \nonumber \\&E^\pm _{1}\rightarrow e^{\pm } h^0_1,\; e^{\pm } Z,\; \nu _e W^{\mp }; \nonumber \\&U_{1}\rightarrow u h^0_1,\; u Z,\; d W^{+};D_{1}\rightarrow d h^0_1,\; d Z,\; u W^{-}, \nonumber \\ \end{aligned}$$

  (84)

• For the second family of new fermions:

  $$\begin{aligned} N_{2}\rightarrow & {} \nu _{\mu ,\tau } h^0_1,\; \nu _{\mu , \tau } Z,\; \mu ^\pm W^{\mp }\;\tau W^{\mp };\nonumber \\ E^\pm _{2}\rightarrow & {} \mu ^{\pm } h^0_1,\;\tau ^{\pm } h^0_1,\; \mu ^{\pm } Z,\; \tau ^{\pm }Z,\; \nu _{\mu ,\tau } W^{\mp }; \nonumber \\ U_{2}\rightarrow & {} c h^0_1,\;t h^0_1, \; c Z,\; t Z,\; b W^{+} ,\; s W^{+};\nonumber \\ D_{2}\rightarrow & {} b h^0_1,\; s h^0_1,\; b Z,\; s Z,\; c W^{-},\; t W^{-}. \end{aligned}$$

  (85)

  Because of the suppressed $\Delta _{\tau ,b}$, the main decay modes are $F_2\rightarrow f_2h^0_1,f_2W,f_2Z$.

The partial decay widths of decays $F\rightarrow fh^0_1, f W, fZ$ are

$$\begin{aligned} \Gamma (F\rightarrow h^0_1f)= & {} \frac{m_F}{8\pi }\left| Y_{Ffh^0_1}\right| ^2\left( 1-\frac{m^2_{h^0_1}}{m^2_F}\right) ^2, \nonumber \\ \Gamma (F\rightarrow Vf)= & {} \frac{m^3_F}{32\pi m_V^2}\left| g_{FfV}\right| ^2\left( 1-\frac{m^2_{V}}{m^2_F}\right) ^2, \end{aligned}$$

(86)

where $V=W,Z$, $g_{FfV}$ and $Y_{Ffh^0_1}$ are, respectively, couplings of fermions with gauge and Higgs bosons given in Tables 1, 2 and 3. Additional factors 3 are included for quark decays. The decays listed in (84) and (85) always have $g_{FfV}\sim \zeta '\epsilon ^2$ and $Y_{Ffh^0_1}\sim s_h \sim \epsilon$, leading to the consequence that $\Gamma (F\rightarrow Vf)/\Gamma (F\rightarrow h^0_1f) \sim \zeta '^2\le \mathcal {O}(10^{-4})$, with $\zeta '$ satisfying (77). Hence, every heavy fermion will decay mainly into a light fermion and a SM-like Higgs boson.

Heavy fermions have been being searched for at the LHC recently, for example the heavy lepton decays into pairs of light leptons and the SM-like gauge bosons [34], and the null result is consistent with this investigation. Other heavy quark decays listed in (85) are $U_2\rightarrow h^0_1 t$ [35], $U_2\rightarrow Wb$ [36,37,38], and $U_2, D_2\rightarrow Zt,Zb$ [39]. But the promoting channels predicted from this discussion are only $U_2\rightarrow ch^0_1$ and $D_2\rightarrow b h^0_1$.

Table 6 Possible large couplings of $h^0_{2,3}$ with fermions and charged Higgs bosons

Table 7 Couplings of CP-odd neutral Higgs bosons, where $p_a,\; p_+$ and $p_0$ are respective incoming momenta of $h_a$, $h^+_{1,2}$, and $h^0_1$

In conclusion, we have indicated that the allowed regions of parameters given in [12, 13] predict following main fermion decays: $E_1\rightarrow h^0_1 e$, $U_1,D_1\rightarrow h^0_1u, h^0_1d$, $E_2\rightarrow h^0_1 \mu$, and $U_2, D_2\rightarrow h^0_1 c,h^0_1 b$. According to our knowledge, these decay channels were not treated experimentally. We emphasize that this discussion is valid for heavy fermions lighter than all other new bosons. Any fermions that are heavier than a heavy gauge boson or a Higgs boson will decay mainly into light fermions and this boson.

5.3 Searches for new Higgs bosons at colliders

At the LHC, the promoting possibility of detecting $h^0_2$ coupling strongly with heavy fermions was indicated in [12, 13]. These large couplings are shown in Table 6, where only large couplings of neutral CP-even Higgs bosons are shown for investigating Higgs productions.

We will focus on the remaining new Higgs bosons, including $h^0_3$, $h_a$, $h^\pm _1$ and $h^\pm _2$. It turns out that they inherit many properties of the new Higgs bosons predicted in THDMs and the MSSM, except $h^\pm _2$. The Higgs sector of the Zee models can be regarded as the one of a THDM plus a pair of singly charged Higgs bosons, as investigated thoroughly in Ref. [33]. And the complete investigation of the Higgs phenomenology of the MSSM was presented in [42, 43] including a brief comparison with Higgs sector in THDMs. The Higgs bosons $h^0_3$, $h_a$, and $h^\pm _1$ have degenerate masses containing the factor of the trilinear Higgs self-coupling $\mu$. This property is the same as that of the MSSM, but completely different from THDMs. Both Refs. [33, 42, 43] considered the Yukawa part of the THDM type II, where up and down right-handed singlets of light fermions couple with different Higgs doublets. In contrast in the model under consideration, all right-handed fermions couple with the same Higgs doublet $\phi$. This explains why the couplings of the neutral $h^0_3$ and $h^\pm _1$ with all quarks always contain the same factor $\frac{1}{t_{\beta }}$, and couplings of $h_a$ with all SM-like fermions contain the same factor $1/_{t_{\beta }}$, as shown in Table 7. While, the couplings of up and down quarks in the MSSM and THDM type II have different factors of $1/t_{\beta }$ and $t_{\beta }$, respectively.

The notation $\beta$ in this work is equivalent to $1/t_{\beta }$ defined in [33, 42, 43], where the allowed $t_{\beta }$ is consistent with the constraint (78).

Now the recent searches for Higgs bosons in THDMs and MSSM will be used for predictions of detecting new Higgs bosons discussed in this work. We consider only Higgs bosons heavier than the top quark. Possible main decays are

$$\begin{aligned} \Gamma (h\rightarrow \overline{ f_1}f_2)= & {} \frac{m_h}{8\pi }K(x_1,x_2)\left[ 1-(\sqrt{x_1}+\sqrt{x_2})^2\right] \nonumber \\&\times \,\left| Y_{hff}\right| ^2, \nonumber \\ \Gamma (h\rightarrow V_1V_2)= & {} \frac{m_{h}}{8 \pi } K(x_1,x_2) \left[ 8x_1x_2+(1-x_1-x_2)^2 \right] \nonumber \\&\times \, \frac{m^2_{h}\left| g_{hVV}\right| ^2}{8m^2_{V_1}m^2_{V_2}},\nonumber \\ \Gamma (h_1\rightarrow h_2 V)= & {} \frac{m_{h_1}}{8 \pi } K(x_1,x_2)\nonumber \\&\times \left[ (1-x_1)^2 -x_2(2+2x_1-x_2)\right] \nonumber \\&\times \, \frac{m^2_{h_1}\left| g_{hhV}\right| ^2}{2m^2_{V}}, \nonumber \\ \Gamma (h\rightarrow h_1 h_2)= & {} \frac{m_h}{8 \pi }K(x_1,x_2) \times \left| \frac{\lambda _{hhh}}{ m_{h}}\right| ^2 , \end{aligned}$$

(87)

where $K(x_1,x_2)=\left[ (1-x_1-x_2)^2 -4 x_1x_2\right] ^{\frac{1}{2}}$; $x_{1,2}=\frac{m_{1,2}^2}{m_0^2}$; $m_{0}$ and $m_{1,2}$ denote the masses of the initial and final states, respectively. The factor 1 / 2 is applied if the two final state particles are identical. A final massless state gives $K(x_1,0)=(1-x_1)^{\frac{1}{2}}$. Expressions for couplings Yukawa $Y_{hff}$, gauge–Higgs–Higgs $g_{hVV}$, the Higgs–Higgs–gauge $g_{hhV}$, and $\lambda _{hhh}$ were listed in the above Tables. The correlations between the different partial decay widths of a Higgs boson depend only on the last factors of formulas in (87). Hence, they will be used to estimate the largest partial decay widths.

The main decay channels of $h^\pm _1$ are $h^+_1\rightarrow \overline{t}b, Z'W,ZW'$ have relative factors as follows:

$$\begin{aligned}&\left| Y_{h^\pm _1 tb}\right| ^2=\left| \frac{\sqrt{2m_t}}{vt_{\beta }}\right| ^2\simeq \frac{1}{t^2_{\beta }}, \frac{m^2_{h^\pm _1}\left| g_{h^\pm _1ZW'}\right| ^2}{8m^2_{Z}m^2_{W'}};\;\\&\frac{m^2_{h^\pm _1}\left| g_{h^\pm _1Z'W}\right| ^2}{8m^2_{Z'}m^2_{W}} \simeq \frac{g^2}{8} \times \frac{m^2_{h^\pm _1}}{m^2_{W'}}, \end{aligned}$$

where the allowed values of $t_{\beta }$ are given in (78). Hence, if $m_{h^\pm _1}$ is not too larger than the heavy gauge boson masses, the main decay is $h^+_1\rightarrow \overline{t}b$, where the $h^\pm _1 tb$ coupling is the same as in the MSSM. The LHC has searched for this decay recently [44, 45], through the production channel $pp\rightarrow tbh^\pm$, giving the lower bound of 1 TeV for $m_{h^\pm _1}$.

6 Conclusion

Recently, the G221 model has been introduced in Refs. [12, 13] with the main purpose to explain all experimental data in flavor physics, tau decays, electroweak precision data, and LNU phenomenology from the anomalies in B decays. But there are still to crucial questions to this model, namely, how to generate active neutrino masses and DM? This work indicated that these problems can be solved based on the mechanisms of generating the active neutrino masses by radiative corrections. In particular, the simplest way to generate the active neutrino masses based on the Zee models was shown in detail. The model predicts the existence of a new pair of singly charged Higgs bosons that have large couplings only with light leptons and Higgs bosons. The DM problem can be solved by applying similar mechanisms shown in many radiative neutrino mass models with DM that were widely investigated previously.

In this work we have analyzed a more general diagonalization of gauge boson mass matrices. We have found that the ratio of the tangents of the Z–$Z'$ and W–$W'$ mixing angles is the cosine of the Weinberg angle, $\cos \theta _W$. This leads to the consequence that the number of the model parameters is reduced by 1.

The most important results of this work were obtained in the Higgs sector where new interesting properties of physical Higgs bosons were explored. First, using the minimal conditions of the Higgs potential to cancel all mutually dependent parameters in the potential, we found that the two squared mass matrices of singly charged and neutral CP-odd Higgs bosons are proportional to the coefficient of the triple Higgs couplings $\mu$. Second, the masses and physical states of all Higgs bosons, as well as their mixing matrices, were presented clearly so that all couplings of the Higgs bosons with the remaining particles can be determined. From this, the SM-like Higgs boson and its couplings were easily identified and compared with experimental data, leading to the important constraint on the mixing parameter $c_h$, namely $0.995<|c_h|<1$. Regarding the new Higgs bosons, three Higgs bosons $h^0_1$, $h_a$ and $h^\pm _1$ have degenerate masses. Namely, the analytic expression for the squared mass is $\frac{\mu u }{s_{2\beta }}+ v^2 s_{\beta }c_{\beta }$, where the main contribution has the same form as the new Higgs boson masses in the MSSM. In addition, their coupling properties are the same as in the THDM of type I. Hence, their behaviors can be predicted based on well-known studies of the THDM as well as of the MSSM.

We combined the above results and the allowed regions of parameters indicated in Refs. [12, 13] to predict some promoting decay channels of new fermions and Higgs bosons. We found that the decays of new heavy particles to SM-like gauge bosons are very suppressed, due to the very small mixing of heavy and SM gauge bosons. The main decays of heavy fermions into two SM-like particles are the decays $F_{1,2}\rightarrow h^0_{1} f_{1,2}$. Decays into SM-like fermions in the third family are very suppressed because the allowed regions contain the tiny coefficient $\Delta _{\tau ,b}$. The main decay of $h^\pm _1$ is $h^\pm \rightarrow tb$. The latest searches for this decay channel give a 1 TeV lower bound for the charged Higgs boson mass.

The LHC have searched for many decay channels of new fermions into SM-like fermions of the third family. So the model will be checked by experiments in coming years. If these decay channels are detected, the model must be extended. For example, the third family of new vector-like fermions should be added to release the allowed regions of parameters.

Appendices

Appendix A: Masses and mixing parameters of charged leptons

If $\lambda _{\ell }$ has the form given in Eq. (20), the precise formula of $V^{11}_L$ defined in (22) is

$$\begin{aligned} V^{11}_L= & {} \sqrt{I_3-\frac{1}{4}\lambda _{\ell }\widetilde{ M}^{-2}_L\lambda ^{\dagger }_{\ell }}\nonumber \\= & {} \left( \begin{array}{ccc} s_{\beta '} &{} 0 &{} 0 \\ 0 &{} 1-\frac{c^2_{\beta '}\Delta ^2_{\mu }}{1+ \rho _{\mu \tau }} &{}-\frac{c^2_{\beta '}\Delta _{\mu }\Delta _{\tau }}{1+\rho _{\mu \tau }} \\ 0 &{} -\frac{c^2_{\beta '}\Delta _{\mu }\Delta _{\tau }}{1+\rho _{\mu \tau }} &{} 1-\frac{c^2_{\beta '}\Delta ^2_{\tau }}{1+\rho _{\mu \tau }} \\ \end{array} \right) , \end{aligned}$$

(A1)

where $\rho _{\mu \tau }=\sqrt{1-c^2_{\beta '}\left( \Delta ^2_{\mu }+\Delta ^2_{\tau }\right) }$. Other submatrices contained in $V_L$ are

$$\begin{aligned} V^{12}_L= & {} \left( \begin{array}{cc} -\frac{s^2_{\beta }\widetilde{M}_{L_1}u}{M_{L_1}}&{} 0 \\ 0 &{}-\frac{\Delta _{\mu }s_{\beta '}\widetilde{M}_{L_2}u\rho _{\mu \tau }}{M_{L_2}} \\ 0 &{} -\frac{\Delta _{\tau }s_{\beta '}\widetilde{M}_{L_2}u\rho _{\mu \tau }}{M_{L_2}}\\ \end{array} \right) , \\ V^{21}_L= & {} \left( \begin{array}{ccc} s_{\beta '} &{} 0 &{} 0 \\ 0 &{} s_{\beta '}\Delta _{\mu } &{} s_{\beta '}\Delta _{\tau } \\ \end{array} \right) , \nonumber \\ V^{22}_L= & {} \mathrm {diag}\left( \frac{M_{L_1}}{\widetilde{M}_{L_1}u},\; \frac{M_{L_2}}{\widetilde{M}_{L_2}u}\right) . \end{aligned}$$

After the block-diagonalization, the SM blocks of fermions matrices must satisfy the experimental constraints. In general, the SM block of the charged lepton mass matrix $V_eV_L\mathcal {M}_{\mathcal {E}}W^{\dagger }_e=\mathcal {M}'_{\mathcal {E}}$ will not be diagonal if the matrix $y_{\ell }$ in (13) is assumed to be diagonal for simplicity. Instead of, $y_{\ell }$ is chosen so that only mixing on $\mu -\tau$ sector is non-zero, the corresponding SM block of $\mathcal {M}'_{\mathcal {E}}$ is

$$\begin{aligned} M_{\ell } \simeq \frac{v s_{\beta }}{\sqrt{2}}\left( \begin{array}{ccc} y_{e}s_{\beta '} &{}0 &{} 0 \\ 0 &{} y_{\mu }\left[ 1-\frac{c^2_{\beta '}\Delta _{\mu }(\Delta _{\mu }+ \Delta _{\tau }y_{\tau \mu }/y_{\mu })}{1+\rho _{\mu \tau }}\right] &{} y_{\mu \tau }-\frac{c^2_{\beta '}\Delta _{\mu } (\Delta _{\mu }y_{\mu \tau } + \Delta _{\tau }y_{\mu })}{1+\rho _{\mu \tau }} \\ 0 &{} y_{\tau \mu }-\frac{c^2_{\beta '}\Delta _{\tau } (\Delta _{\mu }y_{\tau \mu } + \Delta _{\tau }y_{\tau })}{1+\rho _{\mu \tau }} &{}y_{\tau }\left[ 1-\frac{c^2_{\beta '}\Delta _{\tau }(\Delta _{\tau }+ \Delta _{\mu }y_{\mu \tau }/y_{\tau })}{1+\rho _{\mu \tau }}\right] \\ \end{array} \right) . \end{aligned}$$

(A2)

There exist values of $y_{\mu \tau ,\tau \mu }$ such that the matrix (A2) is diagonal and the result of [12, 13] is unchanged. The diagonal SM block of charged leptons also guarantees that the lepton flavor violating decay $h^0_1\rightarrow \mu \tau$ is suppressed, consistent with experimental constraints. Then $y_{\mu \tau }$ and $y_{\tau \mu }$ are chosen to satisfy the condition $(M_{\ell })_{23}=(M_{\ell })_{32}=0$. Now the elements of the Yukawa coupling matrix $y_\ell$ can be expressed as

$$\begin{aligned}&y_{e}=\frac{\sqrt{2}m_e}{vs_{\beta }s_{\beta '}}, \; y_{\mu , \tau }=\frac{\sqrt{2}m_{\mu ,\tau }}{vs_{\beta }}\nonumber \\&~~~~~~~~~~~~ \times \, \frac{1}{\Delta ^2_{\tau }+\Delta ^2_{\mu }} \left[ \Delta ^2_{\tau ,\mu } + \frac{\Delta ^2_{\mu ,\tau }}{\rho _{\mu \tau }}\right] , \nonumber \\&y_{\mu \tau ,\tau \mu } =- \frac{\sqrt{2}m_{\tau ,\mu }}{vs_{\beta }} \times \frac{\Delta _{\mu }\Delta _{\tau }}{\Delta ^2_{\tau }+\Delta ^2_{\mu }} \left[ 1- \frac{1}{\rho _{\mu \tau }}\right] . \end{aligned}$$

(A3)

Appendix B: Neutrino masses from one-loop corrections

First, we consider the simplest case where $\varphi ^\pm$, $\delta ^\pm$ and charged leptons in Fig. 1 are all mass eigenstates; the one-loop amplitude contributing to the neutrino masses is

$$\begin{aligned}&i \frac{1}{2}(m_{\nu })_{ba}\overline{\nu _{L_b}} (\nu _{L_a})^c\nonumber \\&\quad \equiv \int \frac{\mathrm{{d}}^4p}{(2\pi )^4} \overline{\nu _{L_b}} [-i(y_{\ell })_{db}] \frac{im_{e_d}}{p^2-m^2_{e_d}} [-i2f_{da}](\nu _{L_a})^c \nonumber \\&\quad \times \,\frac{i}{p^2-m^2_{\delta }}\left( -i\lambda _{\delta }\frac{uvc_{\beta }}{2}\right) \frac{i}{p^2-m^2_{\varphi ^\pm }}, \end{aligned}$$

(B1)

where we have used $\langle \Phi ^0\rangle =\frac{u}{\sqrt{2}}$, $\langle \varphi '^0\rangle =\frac{ v c_{\beta }}{\sqrt{2}}$ and masses of the charged leptons $m_{e_d}=(y_{\ell })_{cd}\langle \varphi ^0\rangle$. The right hand side of (B1) is rewritten as

$$\begin{aligned} i(m'_{\nu })_{ba}\equiv & {} 2m_{e_d}f_{da} \lambda _{\delta }c_{\beta }(y_\ell )_{db}uv\\&\times \,\int \frac{\mathrm{{d}}^4p}{(2\pi )^4} \frac{1}{\left( p^2-m^2_{\delta }\right) \left( p^2-m^2_{\varphi ^\pm }\right) \left( p^2-m^2_{e_d}\right) }\\= & {} 2m_{e_d}f_{da} \lambda _{\delta }c_{\beta }(y_\ell )_{db} \frac{uv}{16\pi ^2}\times \frac{1}{m^2_{\varphi ^\pm }-m^2_{\delta }}\nonumber \\&\times \ln \left[ \frac{m^2_{\delta }}{m^2_{\varphi ^\pm }}\right] , \end{aligned}$$

where

$$\begin{aligned}&\left. \int \frac{\mathrm{{d}}^4p}{(2\pi )^4} \frac{1}{\left( p^2-m^2_{\delta }\right) \left( p^2-m^2_{\varphi ^\pm }\right) \left( p^2-m^2_{e_d}\right) }\right| _{m^2_{e_d}\rightarrow 0}\nonumber \\&=\frac{i}{16\pi ^2}\times \frac{1}{m^2_{\varphi ^\pm }-m^2_{\delta }} \ln \left[ \frac{m^2_{\delta }}{m^2_{\varphi ^\pm }}\right] . \end{aligned}$$

(B2)

Because $\overline{\nu _{L_b}} (\nu _{L_a})^c= \overline{\nu _{L_a}} (\nu _{L_b})^c$, the mass matrix $(m_{\nu })_{ab}$ is written in the symmetric form as follows: $(m_{\nu })_{ab}=(m_{\nu })_{ba}=\frac{1}{2}\left[ (m'_{\nu })_{ab}+(m'_{\nu })_{ba}\right]$. In the simple case, where $(y_{\ell })_{db}=\delta _{db}m_{e_b}/\langle \varphi ^0\rangle =\delta _{db}m_{e_b}\sqrt{2}/(vs_{\beta })$, we can use the antisymmetric property $f_{ad}=-f_{da}$ to write the neutrino mass matrix in the following form:

$$\begin{aligned} (m_{\nu })_{ba} = \frac{f_{ba}\lambda _{\delta }\sqrt{2}}{16\pi ^2t_{\beta }}\times \frac{u \left( m^2_{e_b}- m^2_{e_a}\right) }{m^2_{\varphi ^\pm }-m^2_{\delta }} \ln \left[ \frac{m^2_{\delta }}{m^2_{\varphi ^\pm }}\right] . \end{aligned}$$

(B3)

If $m_{\delta }=m_{\varphi ^\pm }$, then $\lim _{m_{\delta }\rightarrow m_{\varphi ^\pm }} \frac{1}{m^2_{\varphi ^\pm }- m^2_{\delta }} \ln [\frac{m^2_{\delta }}{m^2_{\varphi ^\pm }}] = -\frac{1}{m^2_{\varphi ^\pm }},$ leading to a simpler form of (B3).