Physics implication from higher weak isospin decomposition

Abstract

The $SU(3)_L\otimes U(1)_X$ symmetry actually studied is directly broken to the electroweak symmetry $SU(2)_L\otimes U(1)_Y$ by a Higgs triplet, predicting a relevant new physics at TeV scale. This work argues, by contrast, that the higher weak isospin $SU(3)_L$ might be broken at a high energy scale, much beyond 1 TeV, by a Higgs octet to an intermediate symmetry $SU(2)_L\otimes U(1)_{T_8}$ at TeV, before the latter $U(1)_{T_8}$ recombined with $U(1)_X$ defines (i.e., broken to) $U(1)_Y$ by a Higgs singlet. The new physics coupled to $SU(3)_L$ breaking phase is decoupled, whereas what remains is a novel family-nonuniversal abelian model, $U(1)_{T_8}\otimes U(1)_X$, significantly overhauling the standard model as well as yielding consistent results for neutrino mass, dark matter, W-mass anomaly, and FCNC, differently from the usual 3-3-1 model.

1 Introduction

The experimental evidences of neutrino oscillation [1, 2] and dark matter existence [3,4,5] indicate that the standard model must be extended. Recently, the W-boson mass anomaly measured by CDF collaboration with high precision at 7$\sigma$ also requires an extension for the standard model [6]. On the theoretical side, the standard model cannot explain the quantization of electric charge and the existence of just three families of fermions.

Among compelling attempts that address the above questions, the $SU(3)_C\otimes SU(3)_L\otimes U(1)_X$ (called 3-3-1) model is well-motivated due to its ability to predict the number of families to match that of colors by anomaly cancellation condition [7,8,9,10,11]. Further, the electric charge quantization in the 3-3-1 model naturally arises from specific fermion contents under gauge symmetry principles [12,13,14,15,16]. The 3-3-1 model can provide both appropriate neutrino masses by implementing seesaw mechanisms [17,18,19,20,21,22,23,24,25,26,27,28,29] and dark matter stability by supplementing global or residual gauge symmetry [30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]. Recently, the 3-3-1 model has been shown to give a suitable answer for the W-mass deviation [48].

The strong interest of the 3-3-1 model is perhaps associated with spontaneous symmetry breaking sector, which leads to distinct kinds of new physics. The minimal, but nontrivial, representations of Higgs fields under 3-3-1 symmetry must contain a triplet, a sextet, and an octet, as well as their complex conjugates if any. The triplet and sextet are eventually studied, since they directly couple to fermions, providing relevant fermion masses, besides responsible for breaking the gauge symmetry. Hence, in the literature, the symmetry breaking scheme that breaks 3-3-1 symmetry directly down to the standard model due to such Higgs fields is uniquely recognized. By contrast, the octet – which possesses a representation as of the adjoint gauge field – does not couple to fermions in Yukawa Lagrangian.¹ Hence, such octet has eventually been suppressed.

This work considers all of the Higgs effects including the octet, indicating that there might be two new physics phases for the 3-3-1 model, which has not been observed/recognized before. Indeed, the octet breaks the 3-3-1 symmetry down to an intermediate symmetry that contains family-dependent abelian charges, whereas the triplet/sextet has a singlet component, breaking the intermediate symmetry to the standard model symmetry. Interestingly, given that the high symmetry phase broken by the octet is decoupled, the intermediate abelian theory supplies interesting physical results for neutrino mass and dark matter, as well as recent physical issues, which are not presented in the usual 3-3-1 model.

The rest of this work is organized as follows. In Sect. 2, we reconsider the 3-3-1 model when including the effect of a Higgs octet as well as imposing a matter parity that governs this setup. In Sect. 3, we investigate the family-dependent abelian theory that results after the 3-3-1 new physics is decoupled, where the W-mass deviation and FCNC are discussed. The fermion mass including that of neutrinos and dark matter stability are investigated in the subsequent sections, Sects. 4 and 5, respectively. We summarize the results and make conclusion as well as outlook in the last section, Sect. 6.

2 Revisiting the 3-3-1 model

We first provide the necessary features of the 3-3-1 model, gauge symmetry and fermion content. We then revisit the 3-3-1 symmetry breaking with the presence of a Higgs octet. Lastly, we discuss $B-L$ symmetry and its resultant matter parity.

2.1 Gauge symmetry and fermion content

In the standard model, the weak isospin $SU(2)_L$ forms left-handed fermions as isodoublets, $l_L=(\nu _L,e_L)$ and $q_L=(u_L,d_L)$, while putting right-handed partners, $e_R$, $u_R$, and $d_R$, as isosinglets. Additionally, the $[SU(2)_L]^3$ anomaly always vanishes, since ${\textrm{Tr}}[\{T_i,T_j\}T_k]=0$ for every $SU(2)_L$ charge, $T_i$ $(i=1,2,3)$. Enlarging $SU(2)_L$ to a higher weak isospin, say $SU(3)_L$, the relevant $[SU(3)_L]^3$ anomaly does not vanish, i.e. ${\textrm{Tr}}[\{T_m,T_n\}T_p]\ne 0$, for complex representation, where $T_m$ $(m=1,2,\ldots ,8)$ denotes $SU(3)_L$ charge. This subsequently constrains fermion content by anomaly cancelation [49,50,51,52]. Notice that the new symmetry must span $SU(3)_C\otimes SU(3)_L$ by imposing the color charge.

The $SU(3)_L$ group has two fundamental representations, $3=2\oplus 1$ and $3^*=2^*\oplus 1$, as decomposed under $SU(2)_L$. Hence we enlarge known fermion isodoublets $(f_1,f_2)$ to respective triplets/antitriplets, i.e. $(f_1,f_2,f_s)\sim 3$ or $(f_2,-f_1,f_s)\sim 3^*$, where $(f_2,-f_1)\sim 2^*$ given that $(f_1,f_2)\sim 2$, isosinglets $f_s\sim 1$ necessarily added, while other known fermion isosinglets presumably transform as $SU(3)_L$ singlets by themselves. Obviously, the $[SU(3)_L]^3$ anomaly is nonzero for each family and cancelled over families if the number of triplets equals that of antitriplets, since the anomaly contributions of 3 and $3^*$ are opposite, and note that the color number must be appropriately counted. It follows that family number is a multiple of color number, 3. Further, QCD asymptotic freedom limits family number less than or equal to 5, since each exotic quark isosinglet is added to a family that also contributes to the strong running coupling. Therefore, the family number matches 3 as observed. That said, the fermion content is arranged under $SU(3)_L$ as

$$\begin{aligned} \psi _{aL}=\begin{pmatrix} \nu _{aL}\\ e_{aL}\\ N_{aL}\end{pmatrix}\sim 3 \end{aligned}$$

(1)

for three lepton families $a=1,2,3$,

$$\begin{aligned} Q_{\alpha L}=\begin{pmatrix} d_{\alpha L}\\ -u_{\alpha L}\\ J_{\alpha L}\end{pmatrix} \sim 3^* \end{aligned}$$

(2)

for first two quark families $\alpha =1,2$, and

$$\begin{aligned} Q_{3L}=\begin{pmatrix} u_{3L}\\ d_{3L}\\ J_{3L}\end{pmatrix}\sim 3 \end{aligned}$$

(3)

for third quark family, whereas relevant right-handed fermions $e_{aR}$, $N_{aR}$, $u_{aR}$, $d_{aR}$, and $J_{aR}$ are all put in $SU(3)_L$ singlets. Notice that minimal 3-3-1 versions assume either $N_L=(e_R)^c$ or $N_L=(\nu _R)^c$, while $N_R$ is suppressed.

Theoretically, N is an arbitrary new lepton. Let the electric charge of it be q, i.e. $Q(N)=q$. We have $Q={\textrm{diag}}(0,-1,q)$ for a lepton triplet, implying an algebra

$$\begin{aligned} \left[ Q, T_1\pm i T_2\right]= & {} \pm (T_1\pm i T_2),\end{aligned}$$

(4)

$$\begin{aligned} \left[ Q, T_4\pm i T_5\right]= & {} \mp q(T_4\pm i T_5),\end{aligned}$$

(5)

$$\begin{aligned} \left[ Q, T_6\pm i T_7\right]= & {} \mp (1+q)(T_6\pm i T_7). \end{aligned}$$

(6)

As the standard model, Q does not commute with $SU(3)_L$ charge. With the aid of commutation relations for $T_{3,8}$ with $T_4\pm i T_5$ and $T_6\pm i T_7$, that is

$$\begin{aligned}{} & {} \left[ T_3,T_1\pm i T_2\right] =\pm (T_1\pm i T_2),\ \left[ T_8,T_1\pm i T_2\right] =0, \end{aligned}$$

(7)

$$\begin{aligned}{} & {} \left[ T_3,T_4\pm i T_5\right] =\pm 1/2 (T_4\pm i T_5),\nonumber \\{} & {} \left[ T_8,T_4\pm i T_5\right] =\pm \sqrt{3}/2(T_4\pm i T_5), \end{aligned}$$

(8)

$$\begin{aligned}{} & {} \left[ T_3,T_6\pm i T_7\right] =\mp 1/2(T_6\pm i T_7),\nonumber \\{} & {} \left[ T_8,T_6\pm i T_7\right] =\pm \sqrt{3}/2(T_6\pm i T_7), \end{aligned}$$

(9)

one finds a new charge,

$$\begin{aligned} X=Q-T_3-\beta T_8, \end{aligned}$$

(10)

that commutes with every $T_i$ charge, i.e. $[X,T_i]=0$, given that

$$\begin{aligned} \beta =-\frac{1+2q}{\sqrt{3}}. \end{aligned}$$

(11)

On the other hand, Q does not algebraically close with $SU(3)_L$ charge, since otherwise algebraic closure requires Q to be a linear composition of $T_i$, i.e. $Q=x_i T_i$, leading to ${\textrm{Tr}}(Q)=0$, which is incorrect for present fermion representations. It follows that X is an independent abelian charge, defining a symmetry,

$$\begin{aligned} SU(3)_C\otimes SU(3)_L \otimes U(1)_X, \end{aligned}$$

(12)

where the color group is also imposed, called 3-3-1. Comparing to the standard model, i.e. $Y=Q-T_3$, the hypercharge is identical to

$$\begin{aligned} Y=\beta T_8+X. \end{aligned}$$

(13)

It is easy to find electric charge of new quarks, $Q(J_3)= 2/3+q$ and $Q(J_{1,2})=-1/3-q$. The fermion content transforms under the 3-3-1 symmetry as

$$\begin{aligned}{} & {} \psi _{aL}\sim \left( 1,3,\frac{q-1}{3}\right) ,\ e_{aR}\sim (1,1,-1),\nonumber \\{} & {} N_{aR}\sim (1,1,q),\end{aligned}$$

(14)

$$\begin{aligned}{} & {} Q_{\alpha L}\sim \left( 3,3^*,-\frac{q}{3}\right) ,\ Q_{3L}\sim \left( 3,3,\frac{q+1}{3}\right) ,\end{aligned}$$

(15)

$$\begin{aligned}{} & {} u_{aR}\sim (3,1,2/3),\ d_{aR}\sim (3,1,-1/3),\end{aligned}$$

(16)

$$\begin{aligned}{} & {} J_{\alpha R} \sim (3,1,-1/3-q),\ J_{3R}\sim (3,1,2/3+q). \end{aligned}$$

(17)

In the literature, there exists an alternative fermion content that is anomaly free, achieved by switching the current representations with conjugated representations, such as $(\nu _{aL},e_{aL},N^q_{aL})\rightarrow (e_{aL},-\nu _{aL},N^q_{aL})$, $(d_{\alpha L}, -u_{\alpha L}, J^{-1/3-q}_{\alpha L})\rightarrow (u_{\alpha L},d_{\alpha L}, J^{-1/3-q}_{\alpha L} )$, and $(u_{3\,L}, d_{3\,L},J^{2/3+q}_{3\,L}) \rightarrow (d_{3\,L},-u_{3\,L},J^{2/3+q}_{3\,L})$, whereas the right-handed fermion partners retain the same. This correspondingly changes $\beta \rightarrow -\beta$ and leads to a model with rather similar phenomena, which is not considered in this work. Surely, our results apply for both types of fermion content. In what follows we consider 3-3-1 breaking.

2.2 Higher weak isospin decomposition and hypercharge realization

In the literature, the 3-3-1 symmetry is broken by a Higgs triplet,

$$\begin{aligned} \chi =\begin{pmatrix} \chi ^{-q}_1\\ \chi ^{-1-q}_2\\ \chi ^0_3\end{pmatrix}\sim \left( 1,3,\frac{-1-2q}{3}\right) , \end{aligned}$$

(18)

with vacuum expectation value (VEV),

$$\begin{aligned} \langle \chi \rangle =\begin{pmatrix} 0\\ 0 \\ w/\sqrt{2}\end{pmatrix}, \end{aligned}$$

(19)

directly down to the standard model, that is

This is because the new charges $T_{4,5,6,7,8}$ do not annihilate the vacuum, i.e. $T_{4,5,6,7,8}\langle \chi \rangle \ne 0$, and are thus broken, while the weak isospin conserves the vacuum, i.e. $T_{1,2,3}\langle \chi \rangle =0$. Additionally, $X\langle \chi \rangle =-\frac{1+2q}{3}\langle \chi \rangle \ne 0$ for $q\ne -1/2$ yields that X is broken, but Y conserves the vacuum similar to the weak isospin, since $Y\langle \chi \rangle =0$. Note that for $q=-1/2$, i.e. $\beta =0$, one has $X=Y$, hence the above statement is valid for every q, i.e. every 3-3-1 model. It is easily verified that a Higgs sextet designed for 3-3-1 breaking takes the same situation. As a matter of fact, the symmetry breaking by a Higgs triplet or sextet has been extensively studied, leading to a new physics consistently bounded at TeV scale.²

While such a new physics has not been confirmed, we would like to argue in this work that the 3-3-1 symmetry may be broken at a very high energy scale, much beyond TeV scale, but imprinted at TeV is an abelian theory entirely different from the usual 3-3-1 model. That said, the 3-3-1 symmetry is broken at high energy by a Higgs octet,

$$\begin{aligned} S= & {} \sqrt{2} T_m S_m = \begin{pmatrix}\frac{S_3}{\sqrt{2}}+\frac{S_8}{\sqrt{6}} &{} S^+_{12} &{} S^{-q}_{13}\\ S^-_{12} &{} -\frac{S_3}{\sqrt{2}}+\frac{S_8}{\sqrt{6}} &{} S^{-1-q}_{23}\\ S^q_{13} &{} S^{1+q}_{23} &{} -\frac{2S_8}{\sqrt{6}} \end{pmatrix}\sim (1,8,0), \nonumber \\ \end{aligned}$$

(20)

which transforms the same with the $SU(3)_L$ gauge field in adjoint representation. Notice that in this case the $SU(3)_L$ generators are related to the Gell-Mann matrices, such as $T_m=\lambda _m/2$ $(m=1,2,3,\ldots ,8),$ while the non-Hermitian scalars

$$\begin{aligned} S^\pm _{12}=\frac{S_1\mp i S_2}{\sqrt{2}},\ S^{\mp q}_{13}=\frac{S_4 \mp i S_5}{\sqrt{2}},\ S^{\mp (1+q)}_{23}=\frac{S_6\mp i S_7}{\sqrt{2}} \nonumber \\ \end{aligned}$$

(21)

are associated with the weight-raising and lowering operators, $(T_1\pm i T_2)/\sqrt{2}$, $(T_4\pm i T_5)/\sqrt{2}$, and $(T_6\pm i T_7)/\sqrt{2}$, respectively. Notice also that each complex field $S_{12}$, $S_{13}$, and $S_{23}$ contains two independent degrees of freedom as its real and imaginary parts originating from $S_m$; or in other words, as appeared in (20), $S^+_{12}$ and $S^-_{12}\equiv (S^+_{12})^*$ and so forth for pairs of $S_{13}$ and $S_{23}$ are independent degrees of freedom. When $S_8$ develops a VEV, i.e. $\langle S_8 \rangle =\Lambda$, the Higgs octet has a VEV structure,

$$\begin{aligned} \langle S \rangle = \frac{1}{\sqrt{6}}\begin{pmatrix} \Lambda &{} 0 &{} 0 \\ 0 &{} \Lambda &{} 0\\ 0 &{} 0 &{} -2 \Lambda \end{pmatrix}. \end{aligned}$$

(22)

Such a S breaks the 3-3-1 symmetry to an intermediate symmetry,

Table 1 Isopartner of normal/known isomultiplet

Indeed, only $T_{4,5,6,7}$ are broken due to $[T_{4,5,6,7},\langle S\rangle ]\ne 0$, whereas $T_{1,2,3}$, $T_8$, and X are all conserved by $\Lambda$, because of

$$\begin{aligned}{}[T_{1,2,3,8},\langle S \rangle ]=0=[X,\langle S \rangle ], \end{aligned}$$

(23)

where note that $T_m=\lambda _m/2$, as mentioned. That said, the octet breaks $SU(3)_L\rightarrow SU(2)_L\otimes U(1)_{T_8}$, while conserving $U(1)_X$, which differs from the cases of Higgs triplet and sextet. Interestingly, this kind of breaking leads to a decomposition of every particle multiplet of $SU(3)_L$ into those of $SU(2)_L\otimes U(1)_{T_8}$ below the breaking energy scale, for instance, the isodoublet-isosinglet splitting in $3=2\oplus 1$ or in $3^*=2^*\oplus 1$ in which $T_8$ values of component 2 or $2^*$ and 1 are appropriately taken. Generically, a $SU(3)_L$ multiplet separates into a normal isomultiplet – a weak isospin multiplet that has normal $B-L$ number as of the standard model – simply called isomultiplet and an abnormal isomultiplet – a weak isospin multiplet that has abnormal $B-L$ number as of dark sector – called isopartner (see below for imposing $B-L$ and matter parity to this model).

Similar to the Higgs octet, the $SU(3)_L$ gauge field $A=\sqrt{2}T_m A_m$ can be written in terms of non-Hermitian gauge fields, $W^\pm =(A_1\mp i A_2)/\sqrt{2}$, $U^{\mp q}=(A_4\mp i A_5)/\sqrt{2}$, and $V^{\mp (1+q)}=(A_6\mp i A_7)/\sqrt{2}$, coupled to the weight-raising and lowering operators, $(T_1\mp i T_2)/\sqrt{2}$, $(T_4\pm i T_5)/\sqrt{2}$, and $(T_6\pm i T_7)/\sqrt{2}$, besides neutral gauge fields, $A_3$ and $A_8$, coupled to $T_3$ and $T_8$, respectively. Given that the 3-3-1 breaking scale is very high, say $\Lambda \gg 1$ TeV, the new gauge bosons $U^{\mp q}$ and $V^{\mp (1+q)}$ associated with broken charges $T_{4,5,6,7}$ obtain a large mass proportional to $\Lambda$, which can be integrated out. Correspondingly, this gauges away the massless Goldstone bosons $S^{\mp q}_{13}$ and $S^{\mp (1+q)}_{23}$ as eaten by $U^{\mp q}$ and $V^{\mp (1+q)}$, respectively. The remaining components of S all obtain a heavy mass at $\Lambda$ scale, which must be integrated out too. What imprints at TeV is a novel abelian theory, $U(1)_{T_8}\otimes U(1)_X$, besides the usual group $SU(3)_C\otimes SU(2)_L$, as in the above breaking scheme due to the Higgs octet. Hence, in this 3-3-1 breaking phase, due to $SU(2)_L \otimes U(1)_{T_8}$ decomposition the new fields N, J, $\chi _3$, $A_8$, and $U(1)_X$ gauge field, called B, coexist but essentially separated from the standard model fields. Also, although unified at $\Lambda$ scale, the $SU(2)_L$ and $U(1)_{T_8}$ couplings at TeV would be different due to the running coupling. As we will be seen, this new abelian theory is very predictive than the usual 3-3-1 model at TeV. Last, but not least, the isosinglet $\chi _3$ has both $T_8$ and X charges. Hence, it breaks these charges down to the hypercharge, i.e.

where $Y=\beta T_8+X$ is as given, while $w=\sqrt{2}\langle \chi _3\rangle$ is at TeV scale.

In summary, there may be two new physics phases for the 3-3-1 model, such as

The first phase is governed by a Higgs octet S, while the second phase close to the standard model is set by a Higgs singlet $\chi _3$. Given that $\Lambda$ is large enough, i.e. $\Lambda \gg w \sim$ TeV, only the second phase with abelian symmetry $U(1)_{T_8}\otimes U(1)_X$ affects the standard model, which must be probed in detail. Notice also that in the second phase every isomultiplet has its isopartner resulting from the higher weak isospin splitting according to $SU(3)_L\rightarrow SU(2)_L\otimes U(1)_{T_8}$, except for original $SU(3)_L$ singlets. For instance, the isopartner of $\chi ^0_3$ is the isodoublet $(\chi ^{-q}_1,\chi ^{-1-q}_2)$, a potential candidate for dark matter since it is not eaten by $(U^{-q},V^{-1-q})$ gauge isodoublet unlike the usual 3-3-1 model. Additionally, normal Higgs isodoublets $(\eta ^0_1,\eta ^-_2)$ and $(\rho ^+_1,\rho ^0_2)$ necessary for breaking $SU(2)_L\otimes U(1)_Y\rightarrow U(1)_Q$ and generating appropriate masses have isopartners as $\eta ^q_3$ and $\rho ^{1+q}_3$ isosinglets, respectively, other potential candidates for dark matter. The new lepton isosinglet $N^q$ is the isopartner of $(\nu ,e)$ isodoublet, possibly contributing to dark matter too, whereas the exotic quark isosinglet J – the isopartner of (u, d) isodoublet – does not. All isomultiplet and corresponding isopartner are collected in Table 1 for convenience in reading.

2.3 $B-L$ symmetry and matter parity

Before proceeding with the $U(1)_{T_8}\otimes U(1)_X$ model, let us examine the symmetry of baryon number minus lepton number $B-L$ and its residual matter parity P, which governs the current setup. By the same argument for Q, we can determine the behavior of $B-L$. Indeed, since the new leptons $N_a$ are arbitrary, let their $B-L$ be n, i.e. $[B-L](N_a)=n$. One has $B-L={\textrm{diag}}(-1,-1,n)$ for lepton triplets $(\nu _{aL},e_{aL},N_{aL})$, which generically neither commutes nor closes with $SU(3)_L$ charges, because of

$$\begin{aligned} \left[ B-L, T_4\pm i T_5\right]= & {} \mp (1+n)(T_4\pm i T_5),\end{aligned}$$

(24)

$$\begin{aligned} \left[ B-L, T_6\pm i T_7\right]= & {} \mp (1+n)(T_6\pm i T_7), \end{aligned}$$

(25)

and ${\textrm{Tr}}(B-L)\ne 0$ for present fermion representations. This $B-L$ assignment is motivated by the traditional 3-3-1 models with $N_L=(\nu _R)^c$ or $(e_R)^c$ for which $B-L={\textrm{diag}}(-1,-1,1)$ for lepton triplets and is also inspired by a TeV-scale seesaw mechanism with $L(N_L)=0$ [54] for which $B-L={\textrm{diag}}(-1,-1,0)$ for relevant lepton triplets. Exceptionally, only if N is a new lepton with $n=-1$, $B-L$ commutes with $SU(3)_L$ and the following trick does not apply, similar to the standard model. It is stressed that the 3-3-1 model also conserves $B+L$ but experiences anomalies when $B+L$ is promoted as a gauge symmetry, in contrast to $B-L$, which is not of interest. Using the given commutation relations, we derive an independent abelian charge

$$\begin{aligned} M=B-L-\beta ' T_8, \end{aligned}$$

(26)

given that $\beta '=-2(1+n)/\sqrt{3}$. Since M and X are linearly independent as $B-L$ and Q are, this defines a symmetry $SU(3)_C\otimes SU(3)_L\otimes U(1)_X\otimes U(1)_M$, called 3-3-1-1 [42, 46, 55]. Acting $B-L=\beta ' T_8+M$ on relevant multiplets, we derive $B-L$ for exotic particles, apart from that for new leptons as given, namely $[B-L](J_\alpha )=-2/3-n$, $[B-L](J_3)=4/3+n$, $[B-L](\chi _{1,2})=[B-L](\eta ^*_3,\rho ^*_3)=[B-L](S_{13},S_{23})=[B-L](U,V)=-1-n$, whereas other new bosons have $B-L=0$ like those of the standard model. Simultaneously, we obtain M-charge for $SU(3)_L$ multiplets, $M(\psi _{aL})=(n-2)/3$, $M(Q_{\alpha L})=-n/3$, $M(Q_{3\,L})=(n+2)/3$, $M(\chi )=-2(1+n)/3$, $M(\eta ,\rho )=(1+n)/3$, and $M(S,A,B)=0$, while for right-handed fermion singlets M coincides with $B-L$.

Since $T_8$ is gauged, M (thus $B-L$) must be gauged due to the relation (26), a result of noncommutative $B-L$. As proved in [42, 46, 55], the $U(1)_M$ physics (i.e., its gauge boson, Higgs field, and $\nu _{aR}$) may be decoupled at GUT scale, or at least above the Higgs octet VEV, leaving the 3-3-1 symmetry besides a residual matter parity conserved, such as

$$\begin{aligned} P=(-1)^{3(B-L)+2s}=(-1)^{3(\beta ' T_8+M)+2s}, \end{aligned}$$

(27)

where s is spin. Note that the S, $\chi$, $\eta$, and $\rho$ vacua always conserve $B-L$ and P. Isomultiplets that possess a normal $B-L$ number (i.e., independent of n), including the standard model fields, $\eta _{1,2}$, $\rho _{1,2}$, $\chi _3$, $A_8$, B, and $S_{1,2,3,8}$, all have $P=1$, whereas their isopartners that have an abnormal $B-L$ number (i.e., dependent on n), including $N_a$, $J_a$, $\chi _{1,2}$, $\eta _3$, $\rho _3$, $S_{13,23}$, and U, V, all possess a nontrivial $P=P^{\pm }\equiv (-1)^{\pm (3n+1)}\ne 1$, which is odd, i.e. $P=-1$, for $n=2k/3=0,\pm 2/3,\pm 4/3,\pm 2,\ldots$, for which the matter parity of isomultiplet and isopartner are indicated in Table 1. Isopartner of normal isomultiplet is P-nontrivial, since their $B-L$’s differ by an amount depending on n ($\beta '$), caused by $T_8$ according to (26).

It is stressed that since P is always conserved, the lightest isopartner – the lightest abnormal $B-L$ particle – is stabilized, responsible for dark matter (cf. [55]). As specified, dark matter candidates include N, $\eta _3$, $\rho _3$, and $\chi _{1,2}$ depending on q since they must be electrically neutral too. Note that since the dark matter candidate must be colorless and stable, this excludes exotic quarks J and heavy fields (U, V) although they are P-nontrivial too. Because the matter parity P is conserved throughout all the phases of symmetry breaking, it may be applied for low energy processes examined below. Additionally, we are investigating the most interesting setup according to $q=0$, thus $\beta =-1/\sqrt{3}$ and $Y=-1/\sqrt{3}T_8+X$, since this kind of the model reveals a neutrino mass generation scheme at TeV, different from the $U(1)_M$ physics at much large scale [42, 46, 55]; the other model according to $q\ne 0$ does not supply such a TeV scheme for neutrino mass. The choice $q=0$ also supplies a variety of types for dark matter, unlike the value $q=-1$ and even no candidate for $q\ne 0,-1$.³ Also, we consider only $n=0$, thus $P=-1$ for isopartners, without loss of generality.

3 $U(1)_{T_8}\otimes U(1)_X$ model: family-dependent abelian phase

In the standard model, the hypercharge Y is family universal; additionally, its mixed anomaly with weak isospin vanishes for each family, i.e. $[SU(2)_L]^2 U(1)_Y\sim 3 Y_{q_L}+Y_{l_L}=0$, where 3 is the color number. On this theoretical ground, the family number may be left arbitrarily, which is opposite to observation; additionally, the matching $3=-Y_{l_L}/Y_{q_L}$ is questionable, because the color charge and the hypercharge have a distinct nature. Interestingly enough, the abelian charges $T_8$ and X do not obey the properties of Y, since the anomaly $[SU(2)_L]^2 U(1)_D \sim 3 D_{q_L}+D_{l_L}\ne 0$ for each family and D is not family universal, for $D=T_8,X$. Even not referring to 3-3-1 symmetry, the $T_8,X$ anomaly cancelation constrains the number of families to be that of colors, similar to a model studied in [56]. Although $T_8$ and X have a nature distinct from Y, their combination $Y=\beta T_8+X$ yields the matching and the universality for Y. In other words, the standard model with three observed families may be induced by a family-dependent abelian phase. In the following, we will specify the particle content and total Lagrangian of the $U(1)_{T_8}\otimes U(1)_X$ model. We then diagonalize the gauge boson sector, determining the W-mass deviation and FCNC.

3.1 Particle content and Lagrangian

Table 2 Field content in family-dependent abelian phase

In the family-dependent abelian phase, the gauge symmetry and its particle content are summarized in Table 2, along with the matter parity P. Each fermion/Higgs isodoublet or isosinglet that possesses $P=1$ has a corresponding isopartner with $P=-1$ due to the mentioned isodoublet-isosinglet splitting. For convenience, we relabel $\eta \equiv (\eta _1,\eta _2)$ and $\rho \equiv (\rho _1,\rho _2)$ for the normal Higgs isodoublets and $\chi \equiv (\chi _1,\chi _2)$ for the isopartner of $\chi _3$, without confusion with those from the higher symmetry. The isopartners N, $\eta _3$, and $\chi _1$ are now electrically neutral, providing a dark matter candidate, whereas $\rho _3$ and $\chi _2$ are electrically charged, unavailable for dark matter. It is clear that six fermion doublets, $l_{aL}$ and $q_{3\,L}$, have a $T_8$ charge opposite to that of other six fermion doublets, $q_{\alpha L}$, where quark colors must be counted. Hence, the anomaly $[SU(2)_L]^2U(1)_{T_8}$ is cancelled over three families that match the color number, despite the fact that each family is anomalous. Since ${\textrm{Tr}}(T_8)=0$ for left-handed quarks (including exotic quarks) as well as for left-handed leptons (including exotic leptons) in each family, the anomalies $[SU(3)_C]^2 U(1)_{T_8}$ and $[{\textrm{gravity}}]^2 U(1)_{T_8}$ vanish. The anomalies $[SU(2)_L]^2U(1)_{X}$, $[SU(3)_C]^2 U(1)_X$, and $[{\textrm{gravity}}]^2 U(1)_{X}$ vanish as a result of substituting $X=1/\sqrt{3}T_8+Y$, for which the relevant anomalies for $T_8$ and Y are obviously cancelled. Finally, it is easily verified that the anomalies $[U(1)_{T_8}]^2 U(1)_X$, $[U(1)_X]^2 U(1)_{T_8}$, $[U(1)_{T_8}]^3$, and $[U(1)_X]^3$ all vanish too.

Up to gauge fixing and ghost terms, the total Lagrangian takes the form,

$$\begin{aligned} {\mathcal {L}}={\mathcal {L}}_{\textrm{kin}}+{\mathcal {L}}_{\textrm{Yuk}}-V, \end{aligned}$$

(28)

where the kinetic term is

$$\begin{aligned} {\mathcal {L}}_{\textrm{kin}}= & {} \sum _\Psi {\bar{\Psi }} i\gamma ^\mu D_{\mu } \Psi + \sum _\Phi (D^\mu \Phi )^\dagger (D_{\mu } \Phi )\nonumber \\ {}{} & {} -\frac{1}{4} G_{m\mu \nu } G^{\mu \nu }_m -\frac{1}{4} A_{i\mu \nu }A^{\mu \nu }_i \nonumber \\{} & {} -\,\frac{1}{4} A_{8\mu \nu } A_8^{\mu \nu } -\frac{1}{4} B_{\mu \nu } B^{\mu \nu }, \end{aligned}$$

(29)

where $\Psi$ and $\Phi$ are summed over every fermion and scalar multiplets, respectively, and the covariant derivative and field strengths are

$$\begin{aligned}{} & {} D_{\mu } = \partial _{\mu } + i g_s t_m G_{m\mu } + i g T_i A_{i\mu }+ i g_1 T_8 A_{8\mu } + i g_2 X B_{\mu }, \\{} & {} G_{m\mu \nu }=\partial _{\mu } G_{m\nu }-\partial _\nu G_{m \mu } - g_s f_{mnp} G_{n\mu }G_{p\nu }, \\{} & {} A_{i\mu \nu }=\partial _{\mu } A_{i\nu }-\partial _\nu A_{i \mu } - g \epsilon _{ijk} A_{j\mu }A_{k\nu }, \\{} & {} A_{8\mu \nu }=\partial _{\mu } A_{8\nu } - \partial _\nu A_{8\mu },\hspace{0.5cm}B_{\mu \nu }=\partial _{\mu } B_\nu -\partial _\nu B_{\mu }, \end{aligned}$$

where $(g_s,g,g_1, g_2)$, $(t_m, T_i, T_8, X)$, and $(G_{m\mu }, A_{i\mu }, A_{8\mu }, B_{\mu })$ correspond to coupling constants, generators, and gauge bosons of $(SU(3)_C, SU(2)_L, U(1)_{T_8}, U(1)_X)$, respectively. The kinetic mixing term between $U(1)_{T_8,X}$ gauge fields is small and suppressed. Indeed, at high energy such term ${\mathcal {L}}_{\textrm{kin}}\supset -\frac{\epsilon }{2}A_{8\mu \nu }B^{\mu \nu }$ is manifestly prevented by the 3-3-1 gauge symmetry. At low energy, $\epsilon$ is radiatively generated due to higher weak-isospin splitting, which can be obtained by generalizing the result in [57] to be $\epsilon =\frac{g_1g_2}{12\pi ^2}\sum _f T_8 (f_L) X(f_L)\ln \frac{m_r}{m_f}$, where f runs over every fermion of complete $SU(3)_L$ multiplets, and $m_r$ is a renormalization scale. Hence, we have

$$\begin{aligned} \epsilon= & {} \frac{g_1g_2}{72\sqrt{3}\pi ^2}\left( \ln \frac{m^2_{J_3}}{m_b m_t}-\ln \frac{m^2_{N_{1L}}}{m_{\nu _e}m_e}\right. \nonumber \\{} & {} \left. -\ln \frac{m^2_{N_{2L}}}{m_{\nu _{\mu }}m_{\mu }}-\ln \frac{m^2_{N_{3L}}}{m_{\nu _{\tau }}m_\tau }\right) \sim 10^{-4}, \end{aligned}$$

(30)

which is radically smaller than the relevant mass mixing discussed below. The kinetic mixing may also come from a GUT effect through effective interactions $\frac{c_1}{\Lambda _{\textrm{GUT}}}S_m A_{m\mu \nu }B^{\mu \nu }+\frac{c_2}{\Lambda ^2_{\textrm{GUT}}}\chi ^\dagger T_m \chi A_{m\mu \nu } B^{\mu \nu }+\frac{c_3}{\Lambda ^2_{\textrm{GUT}}}\eta ^\dagger T_m \eta A_{m\mu \nu } B^{\mu \nu }+\frac{c_4}{\Lambda ^2_{\textrm{GUT}}}\rho ^\dagger T_m \rho A_{m\mu \nu } B^{\mu \nu }$, as coupled to the scalar octet S and triplets $\chi ,\eta ,\rho$, respectively. The octet dominantly contributes to the mixing, such as $\epsilon _{\textrm{GUT}}\sim \frac{\Lambda }{\Lambda _{\textrm{GUT}}}\sim 10^{-11}$, for $\Lambda _{\textrm{GUT}}\sim 10^{16}$ GeV and $\Lambda \sim 100$ TeV (see below). That said, the GUT effect is much smaller than the higher weak-isospin splitting.

The Yukawa term and the scalar potential are given, respectively, by

$$\begin{aligned} {\mathcal {L}}_{\textrm{Yuk}}= & {} h^d_{\alpha b} {\bar{q}}_{\alpha L} {\tilde{\eta }} d_{bR}+h^u_{3b}{\bar{q}}_{3L} \eta u_{bR}+\,x^d_{\alpha b}{\bar{J}}_{\alpha L} \eta ^*_3 d_{bR}\nonumber \\{} & {} +x^u_{3b}{\bar{J}}_{3L}\eta _3 u_{bR}+\,h^u_{\alpha b} {\bar{q}}_{\alpha L} {\tilde{\rho }} u_{bR}+h^d_{3b}{\bar{q}}_{3L} \rho d_{bR}\nonumber \\{} & {} +\,x^u_{\alpha b}{\bar{J}}_{\alpha L} \rho ^{-}_3 u_{bR}+x^d_{3b}{\bar{J}}_{3L}\rho ^+_3 d_{bR} +\,h^J_{\alpha \beta } {\bar{J}}_{\alpha L}\chi ^*_3 J_{\beta R}\nonumber \\{} & {} + h^J_{33} {\bar{J}}_{3L}\chi _3 J_{3R} +\, y^J_{\alpha \beta } {\bar{q}}_{\alpha L} {\tilde{\chi }} J_{\beta R} +y^J_{33} {\bar{q}}_{3L} \chi J_{3R}\nonumber \\{} & {} +\,h^e_{ab}{\bar{l}}_{aL}\rho e_{bR}+z_{ab} {\bar{N}}_{aL} \rho ^+_3 e_{bR} +\,h^N_{ab} {\bar{N}}_{aL} \chi _3 N_{bR}\nonumber \\{} & {} +t_{ab}{\bar{l}}_{aL} \chi N_{bR} -\, \frac{1}{2} \mu _{ab} {\bar{N}}^c_{aR} N_{bR}+H.c. \end{aligned}$$

(31)

$$\begin{aligned} V= & {} \mu ^2_1 \eta ^\dagger \eta + \mu ^2_2 \rho ^\dagger \rho + {\bar{\mu }}^2_3 \chi ^\dagger \chi +{\bar{\mu }}^2_1 \eta ^*_3 \eta _3+\,{\bar{\mu }}^2_2 \rho _3^- \rho _3^+\nonumber \\{} & {} + \mu ^2_3 \chi ^*_3 \chi _3 +\,\lambda _1(\eta ^\dagger \eta )^2+\lambda _2(\rho ^\dagger \rho )^2+{\bar{\lambda }}_3 (\chi ^\dagger \chi )^2\nonumber \\{} & {} + {\bar{\lambda }}_1 (\eta ^*_3 \eta _3)^2 +\,{\bar{\lambda }}_2 (\rho _3^- \rho _3^+)^2 + \lambda _3 (\chi ^*_3 \chi _3)^2\nonumber \\{} & {} +\,\lambda _4(\eta ^\dagger \eta )(\rho ^\dagger \rho ) +\lambda _5 (\eta ^\dagger \eta ) (\chi ^\dagger \chi ) + \lambda _6 (\rho ^\dagger \rho )(\chi ^\dagger \chi )\nonumber \\{} & {} +\,\lambda _{7}(\eta ^\dagger \rho ) (\rho ^\dagger \eta )+\,\lambda _{8}(\eta ^\dagger \chi ) (\chi ^\dagger \eta )+\lambda _{9}(\rho ^\dagger \chi ) (\chi ^\dagger \rho )\nonumber \\{} & {} +\,\frac{1}{2}\lambda _{10}[(\eta ^\dagger \chi )^2+H.c.]+\lambda _{11} (\eta ^*_3 \eta _3)(\rho _3^- \rho _3^+) \nonumber \\{} & {} +\,\lambda _{12} (\eta ^*_3 \eta _3)(\chi _3^* \chi _3 )+\lambda _{13} (\rho _3^- \rho _3^+)(\chi ^*_3 \chi _3)\nonumber \\{} & {} +\,\frac{1}{2} \lambda _{14}[(\eta ^*_3\chi _3)^2+H.c.]\nonumber \\{} & {} +\,\eta ^*_3\eta _3 (\lambda _{15} \eta ^\dagger \eta +\lambda _{16}\rho ^\dagger \rho +\lambda _{17} \chi ^\dagger \chi )\nonumber \\{} & {} +\,\rho ^+_3\rho ^-_3 (\lambda _{18} \eta ^\dagger \eta +\lambda _{19}\rho ^\dagger \rho +\lambda _{20} \chi ^\dagger \chi )\nonumber \\{} & {} +\, \chi ^*_3\chi _3 (\lambda _{21} \eta ^\dagger \eta +\lambda _{22}\rho ^\dagger \rho \nonumber \\{} & {} +\,\lambda _{23} \chi ^\dagger \chi )+[\eta ^*_3\chi _3(\lambda _{24}\eta ^\dagger \chi +\lambda _{25}\chi ^\dagger \eta )+H.c.]. \nonumber \\ \end{aligned}$$

(32)

Above, h’s, x’s, y’s, z’s, t’s, and $\lambda$’s are dimensionless, while $\mu$’s have a mass dimension, and we define ${\tilde{X}}=i\sigma _2 X^*$ for $X=\eta ,\rho ,\chi$. Particularly, to make sure that (i) the scalar fields have a vacuum structure properly (shown below) and (ii) the potential is bounded from below (vacuum stability), the potential parameters satisfy

$$\begin{aligned} \mu ^2_{1,2,3}<0,\hspace{0.5cm}{\bar{\mu }}^2_{1,2,3}>0,\hspace{0.5cm}\lambda _{1,2,3}>0,\hspace{0.5cm}{\bar{\lambda }}_{1,2,3}>0.\nonumber \\ \end{aligned}$$

(33)

The conditions for scalar self-couplings are obtained by requiring $V>0$ when the relevant scalar fields separately tend to infinity. The vacuum stability also demands that $V>0$ when two or more scalar fields simultaneously tend to infinity, which would supply many other conditions for scalar self-couplings, which will be skipped for simplicity.

The gauge symmetry breaking goes through two stages,

The first stage induced by $\langle \chi ^0_3\rangle =w/\sqrt{2}$ has been given, while the last stage is done by

$$\begin{aligned} \langle \eta \rangle = \begin{pmatrix} \frac{v_1}{\sqrt{2}}\\ 0\end{pmatrix},\hspace{0.5cm}\langle \rho \rangle =\begin{pmatrix}0\\ \frac{v_2}{\sqrt{2}}\end{pmatrix}. \end{aligned}$$

(34)

Note that $\chi ^0_1$ and $\eta ^0_3$ have vanished VEVs, prevented by the matter parity. We demand $w\gg v_{1,2}$ and $v^2_1+v^2_2=(246\ {\textrm{GeV}})^2$ for consistency with the standard model.

3.2 Gauge sector and W-mass deviation

Substituting the VEVs of scalar fields to the kinetic term ${\mathcal {L}}\supset \sum _\Phi (D^\mu \Phi )^\dagger (D_{\mu } \Phi )$, gauge boson masses are given. First note that gluons are massless since the color charge is unbroken. For electroweak sector, the charged gauge boson $W^\pm =(A_1\mp i A_2)/\sqrt{2}$ is a physical field, i.e. a mass eigenstate, by itself with mass,

$$\begin{aligned} m^2_W=\frac{g^2}{4}(v^2_1+v^2_2). \end{aligned}$$

(35)

By contrast, the neutral gauge fields $A_3$, $A_8$, and B mix through a mass matrix,

$$\begin{aligned} {\mathcal {L}}\supset \frac{1}{2} \begin{pmatrix} A_3&A_8&B\end{pmatrix} M^2 \begin{pmatrix} A_3\\ A_8\\ B\end{pmatrix}, \end{aligned}$$

(36)

where

$$\begin{aligned} M^2= & {} \frac{g^2}{4}\begin{pmatrix} v^2_1+v^2_2 &{} \frac{t_1}{\sqrt{3}}(v^2_1-v^2_2)&{} -\frac{2t_2}{3}(v^2_1+2v^2_2)\\ \frac{t_1}{\sqrt{3}}(v^2_1-v^2_2) &{} \frac{t^2_1}{3}(v^2_1+v_2^2+4w^2) &{} -\frac{2t_1 t_2}{3\sqrt{3}}(v^2_1-2v^2_2-2w^2)\\ -\frac{2t_2}{3}(v^2_1+2v^2_2) &{} -\frac{2t_1 t_2}{3\sqrt{3}}(v^2_1-2v^2_2-2w^2) &{} \frac{4t^2_2}{9}(v^2_1+4v^2_2+w^2) \end{pmatrix}, \end{aligned}$$

(37)

where we have defined $t_1=g_1/g$ and $t_2=g_2/g$.

The mass matrix $M^2$ has an exact zero eigenvalue corresponding to the photon mass with an exact mass eigenstate corresponding to the photon field as

$$\begin{aligned} \frac{A}{e}=\frac{A_3}{g}-\frac{A_8}{\sqrt{3}g_1}+\frac{B}{g_2}, \end{aligned}$$

(38)

normalized by

$$\begin{aligned} \frac{1}{e^2}=\frac{1}{g^2}+\frac{1}{3g^2_1}+\frac{1}{g^2_2}. \end{aligned}$$

(39)

Alternatively, since the photon field couples to the electric charge, $Q=T_3+Y=T_3-T_8/\sqrt{3}+X$, the above solution can be derived by replacing every generator in the electric charge relation by corresponding gauge field over coupling, i.e. Q by A/e, $T_3$ by $A_3/g$, $T_8$ by $A_8/g_1$, and X by $B/g_2$. Further, the electromagnetic coupling is related to the sine of the Weinberg’s angle as $e=g s_W$, while the tan of this angle is $t_W=g_Y/g$, where the hypercharge coupling is given by $g_Y=\sqrt{3}g_1 g_2/\sqrt{3g_1^2+g^2_2}$. Additionally, we define $g_Y=-\sqrt{3}g_1 s_{\theta } = g_2 c_{\theta }$, i.e. $t_{\theta }=-g_2/\sqrt{3}g_1$. The photon field is rewritten as

$$\begin{aligned} A=s_W A_3+ c_W (s_{\theta } A_8 + c_{\theta } B). \end{aligned}$$

(40)

The standard model Z field is given orthogonally to A,

$$\begin{aligned} Z=c_W A_3 - s_W (s_{\theta } A_8 + c_{\theta } B), \end{aligned}$$

(41)

as usual, while a new field,

$$\begin{aligned} Z'= c_{\theta } A_8 - s_{\theta } B, \end{aligned}$$

(42)

is identified, orthogonal to the hypercharge field in parentheses in A, Z.

In the new basis with $(A,Z,Z')$, the photon field is decoupled as physical field, while the rest Z and $Z'$ mix through a mass matrix,

$$\begin{aligned} {\mathcal {L}}\supset \frac{1}{2} \begin{pmatrix}Z&Z'\end{pmatrix} \begin{pmatrix} m^2_Z &{} m^2_{ZZ'} \\ m^2_{ZZ'} &{} m^2_{Z'}\end{pmatrix}\begin{pmatrix} Z\\ Z'\end{pmatrix}, \end{aligned}$$

(43)

where

$$\begin{aligned} m^2_Z= & {} \frac{g^2}{4 c^2_W}(v^2_1+v^2_2),\end{aligned}$$

(44)

$$\begin{aligned} m^2_{ZZ'}= & {} \frac{g^2t_W}{6s_{2\theta }c_W}\left[ (1+3s^2_{\theta })v^2_2-(1-3s^2_{\theta })v^2_1\right] ,\end{aligned}$$

(45)

$$\begin{aligned} m^2_{Z'}= & {} \frac{g^2 t^2_W}{9s^2_{2\theta }}\left[ (1+3s^2_{\theta })^2v^2_2+(1-3s^2_{\theta })^2v^2_1+4w^2\right] . \nonumber \\ \end{aligned}$$

(46)

Diagonalizing this mass matrix, we achieve two physical fields,

$$\begin{aligned} Z_1= c_\varphi Z - s_\varphi Z',\hspace{0.5cm}Z_2 = s_\varphi Z + c_\varphi Z', \end{aligned}$$

(47)

where the Z–$Z'$ mixing angle $(\varphi )$ is defined by

$$\begin{aligned} t_{2\varphi }=\frac{2m^2_{ZZ'}}{m^2_{Z'}-m^2_Z}\simeq \frac{3}{4} \frac{s_{2\theta }}{s_W}\frac{\left[ (1+3s^2_{\theta })v^2_2-(1-3s^2_{\theta })v^2_1\right] }{w^2},\nonumber \\ \end{aligned}$$

(48)

and the masses of $Z_1$ and $Z_2$ satisfy

$$\begin{aligned} m^2_{Z_1}= & {} \frac{1}{2} \left[ m^2_Z + m^2_{Z'}-\sqrt{(m^2_Z-m^2_{Z'})^2+ 4 m^4_{ZZ'}}\right] \nonumber \\\simeq & {} m^2_Z-\frac{m^4_{ZZ'}}{m^2_{Z'}},\end{aligned}$$

(49)

$$\begin{aligned} m^2_{Z_2}= & {} \frac{1}{2} \left[ m^2_Z + m^2_{Z'}+\sqrt{(m^2_Z-m^2_{Z'})^2+ 4 m^4_{ZZ'}}\right] \nonumber \\\simeq & {} m^2_{Z'}. \end{aligned}$$

(50)

The Z–$Z'$ mixing angle is small, suppressed by $(v_1,v_2)^2/w^2$. Additionally, the field $Z_1$ possesses a mass approximating that of Z, called the standard model Z-like boson, while the field $Z_2$ is a new heavy gauge field with mass at w scale.

Because of the Z–$Z'$ mixing, the observed $Z_1$ mass is reduced in comparison with the standard model Z mass. Because of this mass reduction, it gives rise to a positive contribution to the T-parameter, such as

$$\begin{aligned} \alpha T= & {} \rho -1=\frac{m^2_W}{c^2_W m^2_{Z_1}}-1\simeq \frac{m^4_{ZZ'}}{m^2_Z m^2_{Z'}}\nonumber \\\simeq & {} \frac{\left[ (1+3s^2_{\theta })v^2_2-(1-3s^2_{\theta })v^2_1 \right] ^2}{4(v^2_1+v^2_2)w^2}. \end{aligned}$$

(51)

Since the $Z_1$ mass is precisely measured, the positive value $\alpha T$ mainly enhances the W-boson mass, such as [58, 59]

$$\begin{aligned} \Delta m^2_W=\frac{c^4_W m^2_Z}{c^2_W-s^2_W}\alpha T \simeq \frac{c^2_W g^2 \left[ (1+3s^2_{\theta })v^2_2-(1-3s^2_{\theta })v^2_1\right] ^2}{16c_{2W}w^2}.\nonumber \\ \end{aligned}$$

(52)

The standard model predicts $m_W|_\textrm{SM}=80.357 \pm 0.006$ GeV, which deviates from the CDF measurement $m_W|_{\textrm{CDF}}=80.4335\pm 0.0094$ GeV at 7$\sigma$ [6]. Additionally, the precision measurements of Z-pole physics at the LEP and SLC make a strong constraint on the Z–$Z'$ mixing angle $\varphi$ to be of the order of $10^{-3}$ [60]. Hence, we contour both $\Delta m^2_W=m^2_W|_{\textrm{CDF}}-m^2_W|_{\textrm{SM}}$ taking the difference of centered values and $|\varphi |=c\times 10^{-3}$ concretely inputting $c=1$ and 3 as functions of $v_2$ and w in Fig. 1. Here, notice that $v_1=\sqrt{(246\ {\textrm{GeV}})^2-v^2_2}$, $s^2_W=0.231$, and $\alpha =1/128$. Additionally, at $SU(3)_L$ breaking scale $\Lambda$, the couplings g and $g_1$ match, i.e. $g=g_1$, as unified. When the energy decreases from $\Lambda$ to w, the $SU(2)_L$ coupling g increases while the $U(1)_{T_8}$ coupling $g_1$ decreases, as generically implied by the RGE. It yields $g>g_1$ at w scale, implying $|s_{\theta }|=\frac{t_W}{\sqrt{3}}\frac{g}{g_1}>\frac{t_W}{\sqrt{3}}=0.316$. Hence, we have taken $|s_{\theta }|=0.35$, 0.5, and 0.7 for the above contours. Obviously, the W-mass deviation is solved if the solid line lies above the dashed line, for each $|s_{\theta }|$. Hence, given that $c = 3$, we obtain viable parameter regime $(v_2>216,w>3075)$, $(v_2>210,w>4045)$, and $(v_2>167,w>4686)$, all in GeV, according to the choice of $|s_{\theta }|=0.35$, 0.5, and 0.7, respectively. Otherwise, if $c=1$, or exactly $|\varphi | < 10^{-3}$, the model cannot explain the W-boson mass anomaly, unless $|s_{\theta }|\rightarrow 1$.

Fig. 1

CDF W-mass deviation (solid lines) and EWPT Z-$Z'$ mixing angle $|\varphi |=c\times 10^{-3}$ for $c=1$ and 3 (dashed lines) contoured as functions of $(v_2,w)$ for fixed values of $|s_{\theta }|$

Last, but not least, this study suppresses a kinetic mixing effect between the two $U(1)_{T_8,X}$ gauge fields – which may be experienced in this family-dependent abelian phase but absolutely does not occur in the 3-3-1 phase due to the symmetry protection – which may modify the current result. However, due to the symmetry protection, such effect that arises at low energy would be small, which may be neglected.

On the other hand, the usual 3-3-1 model also predicts the W-mass shift that corresponds to the case $|s_{\theta }|=0.316$, i.e. $g=g_1$ and $\Lambda$ close to w [48]. The present result according to $c=3$ indicates that the Z-$Z'$ mixing source responsible for the W-mass anomaly in the usual 3-3-1 model would possess a narrow (viable) parameter regime, $v_2\rightarrow 246$ GeV, limited by the EWPT Z-$Z'$ mixing angle and the FCNC below.

3.3 Tree-level FCNC

To extract FCNC, we consider the fermion couplings with neutral gauge bosons,

$$\begin{aligned}{} & {} {\mathcal {L}}\supset -\sum _\Psi {\bar{\Psi }}\gamma ^\mu [g T_3 A_{3\mu }\nonumber \\{} & {} \qquad +g_1 T_8 A_{8\mu }+g_2 (T_8/\sqrt{3}+Y) B_{\mu }]\Psi , \end{aligned}$$

(53)

where $X=T_8/\sqrt{3}+Y$ that couples to B is substituted. Since exotic fermions N, J are matter parity odd, impossibly mixed with ordinary fermions, the charges $T_3$ and Y conserve all flavors. The Lagrangian that leads to FCNC includes only $T_8$ couplings,

$$\begin{aligned} {\mathcal {L}} \supset -\sum _\Psi {\bar{\Psi }}\gamma ^\mu T_8 \Psi [g_1 A_{8\mu }+(g_2/\sqrt{3}) B_{\mu }]. \end{aligned}$$

(54)

Since $T_8$ acts only on left chirality and conserves both lepton flavors and exotic quark flavors, the FCNC is only relevant to ordinary left-handed quarks, i.e. $\Psi \rightarrow q_L$. Additionally, we have $g_1 A_8+(g_2/\sqrt{3}) B= -(gt_W/\sqrt{3}s_{\theta } c_{\theta })Z'$, achieved from (42) with notice that $g_1=-gt_W\sqrt{3}s_{\theta }$ and $g_2=gt_W/c_{\theta }$. Hence, the Lagrangian becomes

$$\begin{aligned} {\mathcal {L}} \supset \frac{gt_W}{\sqrt{3}s_{\theta } c_{\theta }}\sum _{q_L} {\bar{q}}_L\gamma ^\mu T_8 q_L Z', \end{aligned}$$

(55)

Further, because of $T_8(q_{1,2L})=-1/2\sqrt{3}=-T_8(q_{3L})$ and by substituting mass eigenstates of quarks, i.e. $q_{aL}=[V_{qL}]_{ai}q_{iL}$ with $V^\dagger _{qL}V_{qL}=1$, we obtain the FCNC,

$$\begin{aligned} {\mathcal {L}} \supset \frac{gt_W}{3s_{\theta } c_{\theta }} \sum _{i,j}[V^*_{qL}]_{3i} [V_{qL}]_{3j}{\bar{q}}_{iL}\gamma ^\mu q_{jL} Z'_{\mu }, \end{aligned}$$

(56)

which flavor changes for $i\ne j$, where q denotes either up-type or down-type quarks.

Integrating $Z'$ out, we achieve an effective Hamiltonian describing neutral meson mixing,

$$\begin{aligned} {\mathcal {H}}_{\textrm{eff}}=\frac{([V^*_{qL}]_{3i} [V_{qL}]_{3j})^2}{w^2} ({\bar{q}}_{iL}\gamma ^\mu q_{jL})^2, \end{aligned}$$

(57)

where we have used $m^2_{Z'}\simeq (g^2 t^2_W/9s^2_{\theta } c^2_{\theta })w^2$. Notice that due to Z-$Z'$ mixing, the Z boson also contributes to the effective Hamiltonian but being more suppressed than $Z'$, since $t_\varphi \sim m^2_Z/m^2_{Z'}\ll m_Z/m_{Z'}$. The effective Hamiltonian induced is independent of $v_{1,2}$ and $\theta$ due to $w\gg v_{1,2}$. Additionally, it does not depend on the new physics phase associated with $\Lambda$ even for $\Lambda \gg w$ or $\Lambda \sim w$ or even $\Lambda$ not existed, since $\Lambda$ does not contribute to neutral gauge boson masses. In other words, the above result applies for every 3-3-1 model, including the family-dependent abelian model.

The effective interactions (57) contribute to neutral-meson mixing amplitudes, which have been extensively studied. The existing data requires

$$\begin{aligned}{} & {} \frac{([V^*_{dL}]_{31} [V_{dL}]_{32})^2}{w^2}<\left( \frac{1}{10^4\ {\textrm{TeV}}}\right) ^2,\end{aligned}$$

(58)

$$\begin{aligned}{} & {} \frac{([V^*_{dL}]_{31} [V_{dL}]_{33})^2}{w^2}<\left( \frac{1}{500\ {\textrm{TeV}}}\right) ^2,\end{aligned}$$

(59)

$$\begin{aligned}{} & {} \frac{([V^*_{dL}]_{32} [V_{dL}]_{33})^2}{w^2}<\left( \frac{1}{100\ {\textrm{TeV}}}\right) ^2, \end{aligned}$$

(60)

corresponding to $K^0$–${\bar{K}}^0$ mixing $(ij=12),$ $B^0_d$–${\bar{B}}^0_d$ mixing $(ij=13),$ and $B^0_s$–${\bar{B}}^0_s$ mixing $(ij=23),$ respectively [61, 62]. Particularly aligning quark mixing to the down sector, we have $V_{dL}=V_{\textrm{CKM}}$, which is well measured, such as $[V_{dL}]_{31}=0.00886$, $[V_{dL}]_{32}=0.0405$, and $[V_{dL}]_{33}=0.99914$ [63]. Hence, the above constraints yield

$$\begin{aligned} w> 3.6\ {\textrm{TeV}},\hspace{0.5cm}w>4.4\ {\textrm{TeV}},\hspace{0.5cm}w>4\ {\textrm{TeV}}, \end{aligned}$$

(61)

corresponding to the mentioned meson mixings, which are in good agreement with the CDF W-mass and EWPT Z–$Z'$ mixing bounds.

4 Neutrino mass generation: scotogenic setup

The Yukawa Lagrangian in (31) will generate tree-level masses for all relevant fermions, when the scalar fields develop VEVs. The isopartners $\chi ^0_1$ and $\eta ^0_3$ have vanished VEV due to the matter parity conservation. However, $\eta ^0_1$, $\rho ^0_2$, and $\chi ^0_3$ can develop the VEVs, as given, inducing tree-level fermion masses. It is noted that since the matter parity is conserved, usual quarks and exotic quarks, as well as usual leptons and new leptons, do not mix. We first give a comment on charged fermions. The masses of exotic quarks are

$$\begin{aligned}{}[m_J]_{\alpha \beta }=-h^J_{\alpha \beta }\frac{w}{\sqrt{2}},\hspace{0.5cm}[m_J]_{33}=-h^J_{33}\frac{w}{\sqrt{2}}, \end{aligned}$$

(62)

which are large at w scale as appropriate. The masses of up-type quarks are

$$\begin{aligned}{}[m_u]_{\alpha b}=-h^u_{\alpha b}\frac{v_2}{\sqrt{2}},\hspace{0.5cm}[m_u]_{3b}=-h^u_{3b}\frac{v_1}{\sqrt{2}}, \end{aligned}$$

(63)

while those of down-type quarks are

$$\begin{aligned}{}[m_d]_{\alpha b}=h^d_{\alpha b}\frac{v_1}{\sqrt{2}},\hspace{0.5cm}[m_d]_{3b}=-h^d_{3b}\frac{v_2}{\sqrt{2}}, \end{aligned}$$

(64)

which are all at weak scales $v_{1,2}$. Diagonalizing these matrices, we get consistent masses for u, d, c, s, t, and b as well as CKM matrix. Similarly, charged leptons gain suitable masses,

$$\begin{aligned}{}[m_e]_{ab}=-h^e_{ab}\frac{v_2}{\sqrt{2}}, \end{aligned}$$

(65)

analogous to the standard model.

The Lagrangian that generates tree-level masses for neutral fermions contains,

$$\begin{aligned}{} & {} {\mathcal {L}}\supset t_{ab}{\bar{l}}_{aL} \chi N_{bR} +h^N_{ab} {\bar{N}}_{aL} \chi _3 N_{bR} \nonumber \\{} & {} \qquad - \frac{1}{2} \mu _{ab} {\bar{N}}^c_{aR} N_{bR}+H.c. \end{aligned}$$

(66)

The last two terms supply new lepton mass as

$$\begin{aligned} {\mathcal {L}}\supset -\frac{1}{2} \begin{pmatrix} {\bar{N}}_L&{\bar{N}}^c_R \end{pmatrix}\begin{pmatrix}0 &{} m\\ m^T &{} \mu \end{pmatrix}\begin{pmatrix} N^c_L \\ N_R \end{pmatrix}+H.c., \end{aligned}$$

(67)

where $m=-h^N w/\sqrt{2}$, and family indices are suppressed for brevity. The new lepton N has a Dirac mass m naturally given at w scale. Whereas, $N_R$ possesses a bare Majorana mass $\mu$, not protected by symmetry, since it is a singlet and has $B-L=0$. Hence, $\mu$ may be arbitrarily large, which is naturally taken at $\Lambda$ scale. In other words, $\mu \gg w$ is appropriately imposed. The above mass matrix obeys a seesaw form, such that

$$\begin{aligned} m_{N_L}\simeq -m\mu ^{-1} m^T\sim w^2/\mu ,\hspace{0.5cm}m_{N_R}\simeq \mu , \end{aligned}$$

(68)

while the $N_L$–$N_R$ mixing is small and neglected. Since $w\sim$ TeV, we take $\mu \sim \Lambda \sim 100$ TeV, as expected. Therefore, we obtain $m_{N_L}\sim 10$ GeV, while $m_{N_R}\sim 100$ TeV.

The first term in (66) couples lepton doublets to the isopartners which are matter parity odd, an inert scalar doublet $\chi$ and Majorana fermion singlets $N_R$’s. This fully realizes a scotogenic scheme for neutrino mass generation [64]. Indeed, the field $\chi$ couples to normal Higgs doublets and singlet in the potential (32), collected as

$$\begin{aligned} V{} & {} \supset {\bar{\mu }}^2_3 \chi ^\dagger \chi +\lambda _5 (\eta ^\dagger \eta ) (\chi ^\dagger \chi ) + \lambda _6 (\rho ^\dagger \rho )(\chi ^\dagger \chi ) \nonumber \\{} & {} \quad +\lambda _{8}(\eta ^\dagger \chi ) (\chi ^\dagger \eta )+\lambda _{9}(\rho ^\dagger \chi ) (\chi ^\dagger \rho )+\frac{1}{2} \lambda _{10}\nonumber \\{} & {} \quad \times [(\eta ^\dagger \chi )^2+H.c.]+\lambda _{23} (\chi ^*_3\chi _3)(\chi ^\dagger \chi ), \end{aligned}$$

(69)

where the bare mass term of $\chi$, i.e. ${\bar{\mu }}_3$, is concerned as also included. When $\eta ,\rho ,\chi _3$ develop the VEVs $v_{1},v_{2},w$ respectively, the field

$$\begin{aligned} \chi =\begin{pmatrix}\chi ^0_1\\ \chi ^-_2\end{pmatrix}=\begin{pmatrix} \frac{1}{\sqrt{2}}(R_1+ i I_1)\\ \chi ^-_2\end{pmatrix} \end{aligned}$$

(70)

is separated in mass, such as

$$\begin{aligned} m^2_{R_1}= & {} {\bar{\mu }}^2_3+\frac{\lambda _5+\lambda _8+\lambda _{10}}{2}v^2_1+\frac{\lambda _6}{2}v_2^2+\frac{\lambda _{23}}{2}w^2, \end{aligned}$$

(71)

$$\begin{aligned} m^2_{I_1}= & {} {\bar{\mu }}^2_3+\frac{\lambda _5+\lambda _8-\lambda _{10}}{2}v^2_1+\frac{\lambda _6}{2}v_2^2+\frac{\lambda _{23}}{2}w^2, \end{aligned}$$

(72)

$$\begin{aligned} m^2_{\chi _2}= & {} {\bar{\mu }}^2_3+\frac{\lambda _5}{2}v^2_1+\frac{\lambda _6+\lambda _9}{2}v_2^2+\frac{\lambda _{23}}{2}w^2. \end{aligned}$$

(73)

There exist $R_1$–$\Re (\eta _3)$ mixing and $I_1$–$\Im (\eta _3)$ mixing due to $\lambda _{24,25}$ couplings. However, the mixing effects are small, suppressed by $(v_1/w)$, which can be neglected. The physical fields $R_1$, $I_1$, and $\chi _2$ obtain a mass naturally given at w scale. Especially, the $R_1$ and $I_1$ masses are small separated by $\lambda _{10}$ coupling, such as

$$\begin{aligned} m^2_{R_1}-m^2_{I_1}=\lambda _{10}v^2_1\ll m^2_{\chi _1}, \end{aligned}$$

(74)

where $m^2_{\chi _1}\equiv (m^2_{R_1}+m^2_{I_1})/2\simeq {\bar{\mu }}^2_3+(\lambda _{23}/2)w^2$.

That said, the neutrino generation Lagrangian is given by

$$\begin{aligned}{} & {} {\mathcal {L}}\supset \frac{t_{ab}}{\sqrt{2}}{\bar{\nu }}_{aL}(R_1+iI_1)N_{bR}-\frac{1}{2} m_{N_{bR}} N^2_{bR}\nonumber \\{} & {} \qquad -\frac{1}{2} m^2_{R_1} R^2_1-\frac{1}{2} m^2_{I_1} I^2_1, \end{aligned}$$

(75)

where we assume the mass matrix $m_{N_R}={\textrm{diag}}(m_{N_{1R}},m_{N_{2R}}, m_{N_{3R}})$ to be flavor diagonal, i.e. $N_{bR}$ is a physical Majorana field by itself, without loss of generality. The neutrino mass is induced by one-loop diagrams mediated by $N_R$ and $R_1/I_1$, as depicted in Fig. 2. Note that in the gauge basis, the $N_R$ line is attached by Majorana mass term $\mu _{ab}$, while the $R_1/I_1$, i.e. $\chi _1$, line is coupled to external $\eta _1$ field via $\lambda _{10}$ coupling, analogous to that in [64].

Fig. 2

Dark isopartner contribution to neutrino mass

The neutrino mass matrix that is derived in form of ${\mathcal {L}}\supset -\frac{1}{2} {\bar{\nu }}_{aL} [m_\nu ]_{ab} \nu ^c_{bL}+H.c.$ reads

$$\begin{aligned} -i [m_\nu ]_{ab}P_R{} & {} =\int \frac{d^4 p }{(2\pi )^4}\left( i\frac{t_{ak}}{\sqrt{2}} P_R\right) \frac{i}{/\!\!\! p-m_{N_{kR}}} \left( i\frac{t_{bk}}{\sqrt{2}}P_R\right) \nonumber \\{} & {} \quad \times \frac{i}{p^2- m^2_{R_1}}+\int \frac{d^4 p }{(2\pi )^4}\left( -\frac{t_{ak}}{\sqrt{2}} P_R\right) \nonumber \\{} & {} \quad \times \frac{i}{/\!\!\! p-m_{N_{kR}}}\left( -\frac{t_{bk}}{\sqrt{2}} P_R\right) \frac{i}{p^2- m^2_{I_1}}\nonumber \\{} & {} \quad = \frac{P_R}{2}t_{ak}t_{bk} m_{N_{kR}}\int \frac{d^4p}{(2\pi )^4}\nonumber \\{} & {} \qquad \times \frac{m^2_{R_1}-m^2_{I_1}}{(p^2-m^2_{N_{kR}})(p^2-m^2_{R_1})(p^2-m^2_{I_1})}.\nonumber \\ \end{aligned}$$

(76)

Notice that the contributions of $R_1$ and $I_1$ have opposite signs because the product of two neutrino vertices coupled to the real and imaginary parts of $\chi _1$ changes the sign. Hence, the result depends on $R_1,I_1$ mass splitting for which the divergences according to each diagram are cancelled out, matching the convergent integral (76). Given that $x\gg y,z$, one has

$$\begin{aligned}{} & {} \int \frac{d^4 p}{(2\pi )^4}\frac{1}{(p^2-x )(p^2-y)(p^2-z)}\nonumber \\{} & {} \qquad \simeq \frac{i}{16\pi ^2}\frac{y\ln y/x-z \ln z/x}{x (y-z)}. \end{aligned}$$

(77)

Applying to this model for $m_{N_R}\gg m_{R_1,I_1}$, it gives rise to

$$\begin{aligned}{}[m_\nu ]_{ab}\simeq & {} -\frac{1}{32\pi ^2}\frac{t_{ak}t_{bk}}{m_{N_{kR}}}\left( m^2_{R_1}\ln \frac{m^2_{R_1}}{m^2_{N_{kR}}}-m^2_{I_1}\ln \frac{m^2_{I_1}}{m^2_{N_{kR}}}\right) \nonumber \\ {}\simeq & {} \frac{\lambda _{10}}{32\pi ^2} \frac{t_{ak}t_{bk} v^2_1}{m_{N_{kR}}}\left( \ln \frac{m^2_{N_{kR}}}{m^2_{\chi _1}}-1\right) , \end{aligned}$$

(78)

where the last approximation uses the small mass splitting of $R_1,I_1$ in (74). The neutrino mass is naturally small, as suppressed by the loop factor $1/32\pi ^2$, the large scale $m_{N_{R}}$, and the $B-L$ violating coupling $\lambda _{10}$ (notice that $\lambda _{10}$ is also presented in the usual 3-3-1 model and always conserves the matter parity). Indeed, taking $\lambda _{10}\sim 10^{-3}$, $t\sim 10^{-2}$, $m_{N_R}\sim 100$ TeV, $m_{\chi _1}\sim$ TeV, and $v_1\sim 100$ GeV, we have $m_\nu \sim 0.1$ eV, as expected.

Last, but not least, the above mechanism cannot work in the usual 3-3-1 model since $\chi ^0_1$ is a Goldstone boson, eaten by the corresponding gauge boson $U^0$. Since the gauge interaction of scalar field that induces a gauge boson mass always conserves CP, the field like $U^0$ cannot be separated in mass. Thus, the contribution of $U^0$ to neutrino mass via a diagram similar to Fig. 2 by replacing $\chi _1$ by U and $N_R$ by $N_L$ vanish, which is unlike a model in [65]. We call the reader’s attention to recent scotogenic approaches in the 3-3-1-1 model [66,67,68].

5 Dark matter stability: seesaw setup

The present model predicts the lightest state of $N_{aL}$, denoted $N_L$, to be the lightest dark isopartner – the unique candidate for dark matter. Indeed, this candidate is naturally stabilized by the matter parity P and the seesaw mechanism in (67), as given. The other dark isopartners, say $\chi ^0_1$, $\chi ^-_2$, $\eta ^0_3$, $\rho ^+_3$, and $J_a$, all have a mass at (i.e., proportional to) w scale, while $N_{aR}$ like the vector doublet (U, V) and the Higgs octet S all possess a mass at $\Lambda$ scale. The seesaw mechanism makes the $N_L$ mass $\sim w^2/\Lambda$ naturally (far) below w scale, i.e. setting $N_L$ to be the lightest dark isopartner. Hence, contrary to the scotogenic setup, the present model predicts the light seesaw partner $N_L$ of the heavy scotogenic field $N_R$ to be dark matter. Additionally, this setup differs from the usual 3-3-1 model for which several candidates for dark matter are viable.

Fig. 3

Dark matter annihilation to normal matter, where e denotes charged leptons while f indicates usual leptons and quarks (except top quark)

The field $N_L$ annihilates to charged leptons via t-channel diagrams exchanged by $\rho ^\pm _3$ as well as to ordinary fermions (except top quark) via s-channel diagrams through $Z'$ portal, as depicted in Fig. 3, which sets $N_L$ relic density. The t-channel diagrams are governed the Yukawa couplings $z_{ab}{\bar{N}}_{aL}\rho ^+_3 e_{bR}+H.c.$ as in the Yukawa Lagrangian, while the t-channel diagrams are set by the $Z'$ couplings to fermions, such as

$$\begin{aligned} g^{Z'}_V(f)= & {} \frac{s_W}{s_{\theta } c_{\theta }}\left[ s^2_{\theta } T_3(f_L)-\frac{1}{\sqrt{3}}T_8(f_L)-2s^2_{\theta } Q(f)\right] ,\nonumber \\ \end{aligned}$$

(79)

$$\begin{aligned} g^{Z'}_A(f)= & {} \frac{s_W}{s_{\theta } c_{\theta }}\left[ s^2_{\theta } T_3(f_L) -\frac{1}{\sqrt{3}}T_8(f_L)\right] , \end{aligned}$$

(80)

which are explicitly collected in Table 3. Here note that the Z–$Z'$ mixing effect is small and neglected.

Table 3 Couplings of $Z'$ to fermions

If the s-channel diagrams either contribute equivalently to or dominate over the t-channel diagrams, this yields an annihilation cross-section proportional to

$$\begin{aligned} \langle \sigma v\rangle _s \sim \frac{g^4}{16\pi c^4_W}\frac{m^2_{N_L}}{m^4_{Z'}}\simeq \left( \frac{m_{N_L}}{m_{Z'}}\right) ^2\times \left( \frac{1.5\ {\textrm{TeV}}}{m_{Z'}}\right) ^2\times 1~{\textrm{pb}}, \end{aligned}$$

(81)

which is much smaller than the experimental value $\langle \sigma v\rangle \simeq 1$ pb, because of $m_{N_L}\ll m_{Z'}$. Therefore, the expected annihilation cross-section must be dominated by the t-channel diagrams (necessarily enhanced by the z coupling), computed as

$$\begin{aligned} \langle \sigma v\rangle= & {} \frac{\sum _{i,j}|z^*_{1 i} z_{1j}|^2}{8\pi } \frac{m^2_{N_L}}{m^4_{\rho _3}}\simeq \left( \frac{\sum _{i}|z_{1 i}|^2}{4\pi }\right) ^2\nonumber \\{} & {} \times \left( \frac{m_{N_L}}{10\ {\textrm{GeV}}}\right) ^2\times \left( \frac{700~ {\textrm{GeV}}}{m_{\rho _3}}\right) ^4\times 1~ {\textrm{pb}}, \end{aligned}$$

(82)

where the dark matter $N_L$, assumed as $N_{1L}$ for brevity, is heavier than charged leptons $e_{i/j}$, so the charged lepton masses are omitted. The dark matter relic density $\Omega h^2\simeq 0.1~ {\textrm{pb}}/\langle \sigma v\rangle \simeq 0.11$ [63] is explained, given that $\sum _i |z_{1i}|^2/4\pi \sim 1$, $m_{\rho _3}\sim 700$ GeV, with the dark matter mass $m_{N_L}\sim 10$ GeV as implied by the seesaw mechanism. Although the $Z'$ portal that drives the s-channel annihilation does not contribute to the dark matter abundance, it may mark a direct dark matter detection signal, as interpreted hereafter.

Since $N_L$ is a Majorana field, it only scatters with quarks in direct detection experiment through spin-dependent (SD) effective interaction induced by $Z'$ similar to the right diagram in Fig. 3 for $f=q$, such as

$$\begin{aligned} {\mathcal {L}}_{\textrm{eff}}\supset \frac{g^2}{4c^2_Wm^2_{Z'}}[g^{Z'}_A(N)g^{Z'}_A(q)]^2({\bar{N}} \gamma ^\mu \gamma _5 N)({\bar{q}}\gamma _{\mu } \gamma _5q), \end{aligned}$$

(83)

where $N\equiv N_L+N^c_L$ and the couplings $g^{Z'}_A(f)$ for $f=N,q$ have been given in Table 3. We obtain the SD cross-section describing the scatter of N with a target neutron (n), 

$$\begin{aligned} \sigma ^{\textrm{SD}}_{N_L}= & {} \frac{3g^4}{4\pi c^4_W}\frac{m^2_n}{m^4_{Z'}}[g^{Z'}_A(N)]^2\nonumber \\{} & {} \times [g^{Z'}_A(u)\lambda ^n_u +g^{Z'}_A(d)(\lambda ^n_d+\lambda ^n_s)]^2, \end{aligned}$$

(84)

where the fractional quark-spin coefficients for neutron are $\lambda ^n_u=-0.42$, $\lambda ^n_d=0.85$, and $\lambda ^n_s=-0.88$ [69]. The scatter with proton gives a similar bound and is not considered.

Fig. 4

SD cross-section of dark matter $N_L$ with neutron plotted as function of $m_{Z'}$ according to the fixed values of $|s_{\theta }|$

We take $s^2_W=0.231$, $\alpha =g^2\,s^2_W/4\pi =1/128$, and $m_n=1$ GeV, as usual. In Fig. 4, we plot the SD cross-section as a function of $Z'$ mass according to $|s_{\theta }|=0.35$, 0.5, and 0.9, where the first two values have previously been given, while the last one allows us examining a limit. It is clear that this model predicts the dark matter signal strength in direct detection safely below the current bounds $\sim 10^{-41}~\text {cm}^2$ for $m_{N_L}\sim 10$ GeV from various experiments [70,71,72], even for large $|s_{\theta }|$ and $Z'$ mass beyond the weak scale. The SD cross-section reaches the experiment bound when $|s_{\theta }|\rightarrow 1$ (compared to a similar issue concerning the W-mass anomaly and the EWPT Z–$Z'$ mixing discussed above), which requires a large $g,g_1$ coupling splitting such as $g_1=0.316g$, which is not signified as needing a study of the RGE, a task out of the scope of this work.

6 Conclusion and outlook

We have argued that there are two new physics phases associated with the 3-3-1 model, the 3-3-1 phase governed by a Higgs octet and the family-dependent abelian phase set by a Higgs singlet. This argument is valid for the 3-3-1 model with a high Landau pole like the 3-3-1 model with right-handed neutrinos considered in this work, while it is hard applying to the minimal 3-3-1 model, which has a Landau pole around 4–5 TeV.⁴

Given that the Higgs octet VEV is much bigger than that of the Higgs singlet, the 3-3-1 phase is decoupled and imprinted at low energy being only a conserved matter parity. This matter parity may originate from a 3-3-1-1 symmetry as discussed in the paper or an alternative unification, which is left for a further study.⁵ This work simply supposes the matter parity working, neglecting its origin as well as any constraint from the more fundamental theory (if any).

Contrary to the usual 3-3-1 model studied at TeV scale as well as the scotogenic setup, the family-dependent abelian model instead yields compelling results for the W-mass anomaly, neutrino mass, and dark matter. Indeed, the W-mass measured signals a Z–$Z'$ mixing appropriate to the EWPT. The neutrino mass is induced by a scotogenic mechanism, naturally working due to the matter parity and isopartner presence. The model predicts a unique dark matter candidate, the light seesaw partner of the scotogenic Majorana field $N_R$. The family-dependent abelian phase is predicted at TeV scale and the large $SU(2)_L\otimes U(1)_{T_8}$ coupling-splitting is preferred.

Who orders that: the third entries of lepton triplets? The present model reveals a possibility that the third entries of lepton triplets $N_{aL}$ contribute to WIMP dark matter, while their right-handed partners $N_{aR}$ contribute to scotogenic neutrino mass generation, which properly works if these N’s are electrically neutral and possess $B-L=0$. The phenomenology would differ from the current setup if either of the following cases occurs

• 1. $N_{aR}$ are suppressed: the scotogenic scheme is not viable, and needing a mechanism for $N_{aL}$ mass generation and that $N_{aL}$ are not necessarily the lightest dark isopartners;

• 2. $n=2k/3\ne 0$ (still keeping $P=-1$ for N’s and other isopartners): we lack a mechanism for inducing Majorana $N_{aR}$ masses due to $B-L$ conservation, or otherwise the scotogenic setup does not work and N’s are dark Dirac fermions, not necessarily being the lightest dark isopartners. Additionally, $B-L$ gauge portal interaction for dark matter N arises, although small;

• 3. $n=(2k+1)/3$ (now $P=1$ for all fields): the lightest dark isopartner is not stabilized, i.e. the solution for dark matter disappears, besides the issue for Majorana $N_{aR}$ mass generation vs the scotogenic neutrino mass setup similar to the previous case,

• 4. $N_{aL}=(\nu _{aR})^c$ whereas $N_{aR}$ are suppressed ($n=1$): in this case, $P=1$ is trivial for all fields and the scotogenic setup is not viable, i.e. both current schemes for neutrino mass and dark matter are discarded.

In other words, all alternative choices of the third entries of lepton triplets lead to distinct phenomenologies for which the above four cases require different interpretations for neutrino mass and dark matter. Even, this is valid for the 3-3-1 model with $q=1$ in which the third entries contain $(e_{aR})^c$, or new charged leptons $E^+_a$ with $B-L=1$, or a new type of leptons with $B-L=n\ne 1$.