An explanation of experimental data of $(g-2)_{e,\mu }$ in 3-3-1 models with inverse seesaw neutrinos

Abstract

We show that the anomalous magnetic moment experimental data of muon and electron $(g-2)_{\mu ,e}$ can be explained simultaneously in simple extensions of the 3-3-1 models consisting of new heavy neutrinos and a singly charged Higgs boson. The heavy neutrinos generate active neutrino masses and mixing through the general seesaw mechanism. They also have non-zero Yukawa couplings with singly charged Higgs bosons and right-handed charged leptons, which result in large one-loop contributions known as chirally-enhanced ones. Numerical investigation confirms a conclusion indicated previously that these contributions are the key point to explain the large $(g-2)_{\mu ,e}$ data, provided that the inverse seesaw mechanism is necessary to allow both conditions that heavy neutrino masses are above few hundred GeV and non-unitary part of the active neutrino mixing matrix must be large enough.

1 Introduction

Recently, anomalous magnetic moments (AMM) of charged leptons $a_{e_a} \equiv (g-2)_{e_a}/2$ have been studied widely because the recent experimental data shows large deviations from the Standard Model (SM) predictions. The recent improved AMM value of muon $a_{\mu }$ predicted by the SM is accepted widely as [1] $a^{{\mathrm {SM}}}_{\mu }= 116591810(43)\times 10^{-11}$, which is derived from the combination of various contributions using the dispersion approach [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22]. However, this results is inconsistent with the lattice-QCD calculation [23], which is closer to the recent experimental data. In our work, we will use the larger discrepancy between theoretical and experimental results, because it is more interesting for theoretical discussions and the allowed regions of parameter space are still applicable for the smaller deviation reported in Ref. [23]. The latest experimental measurement has been reported from Fermilab [24] and is also in agreement with previous experimental result measured by Brookhaven National Laboratory (BNL) E82 [25]. A combination of these results in the new average value of $a^{\mathrm {exp}}_{\mu }=116592061(41)\times 10^{-11}$, which leads to the improved standard deviation of 4.2 $\sigma$ from the SM prediction, namely

$$\begin{aligned} \Delta a^{\mathrm {NP}}_{\mu }\equiv a^{\mathrm {exp}}_{\mu } -a^{{\mathrm {SM}}}_{\mu } =\left( 2.51\pm 0.59 \right) \times 10^{-9}. \end{aligned}$$

(1)

The recent experimental AMM values of electron $a_e$ were reported from different groups [26,27,28] (for calculation of $a_e$ in the SM, see Refs. [17, 29,30,31,32,33,34]). In our numerical discussion, we adopt the experimental values of $a_e$ corresponding to the following standard deviation of $2.5\sigma$ from the SM one:

$$\begin{aligned} \Delta a^{\mathrm {NP}}_{e}\equiv a^{\mathrm {exp}}_{e} -a^{{\mathrm {SM}}}_{e} = \left( -8.7\pm 3.6\right) \times 10^{-13}. \end{aligned}$$

(2)

Many models beyond the SM (BSM) have been constructed to explain the experimental data of $(g-2)_{\mu ,e}$, such as models adding vector-like lepton multiplets [35,36,37,38,39,40,41,42,43,44,45,46], leptoquarks [47], both neutral and charged Higgs bosons as $SU(2)_L$ singlets [48], $SU(2)_L$ triplets of leptons and scalars [49]. The minimal supersymmetric standard model can explain both $(g-2)_{e,\mu }$ data in the regions of the parameter space with light slepton masses below a few hundred GeVs [50, 51]. Some two Higgs doublet models (THDM) adding new $SU(2)_L$ Higgs doublets can give large two-loop contributions to $\Delta a_{\mu }$ [52,53,54,55,56], provided that the masses of the new neutral and/or charged Higgs bosons must be light at a few hundred GeVs.

In this work, we will focus on the AMM problems predicted by a BSM class called as 3-3-1 models, constructed based on the $SU(3)_C\times SU(3)_L\times U(1)_X$ group [57,58,59,60,61,62,63,64,65,66,67]. It was shown that the early 3-3-1 versions cannot predict large $\Delta a_{\mu }$ given by the experimental data [68,69,70,71,72,73,74]. Extended versions were introduced to solve this problem, such as 3-3-1 models with new vector-like leptons or inert $SU(3)_L$ Higgs triplets [72, 73], the models with new singly charged Higgs couplings with heavy neutrinos generating neutrino masses through the inverse seesaw (ISS) mechanism [75, 76], and 3-3-1 models with discrete symmetries containing a rather large number of new particles needed to explain the hierarchy problems of fermion masses [77, 78]. In Ref. [36], the very precise analytic formulas applicable to calculate general one-loop contributions to AMM in a wide class of BSM were presented. Using these formulas, we can estimate again the previous results available in all current 3-3-1 models. These analytic formulas are consistent with those calculated previously for 3-3-1 models [79, 80]. More importantly, we will show that the 3-3-1 models can give large one-loop contributions to AMM by adding new $SU(3)_L$ singlets such as singly charged Higgs bosons $h^\pm$ and heavy neutrinos, a similar way that was applied to the 3-3-1 model with right-handed (RH) neutrinos (331RHN). On the other hand, Higgs triplets and their Yukawa couplings needed to generate masses for charged leptons, quarks, and neutrinos in many 3-3-1 models may have different features from those in the 331RHN, leading to new predictions of the allowed regions of parameter space satisfying the AMM experimental data predicted by different 3-3-1 models. Heavy neutrinos are needed to generate active neutrino masses and mixing through the general seesaw (GSS) mechanism, and Yukawa couplings of singly charged Higgs boson and RH charged leptons. Hence the new particles result in Yukawa terms like $N\left( \lambda ^L P_L +\lambda ^R P_L\right) e_a H^+$ corresponding to the presence of the so called chirally-enhanced one-loop contributions proportional to $\lambda ^{L*}\lambda ^{R}$, where N and $H^+$ denote two physical states of a neutrino and a singly charged Higgs boson. They are the most important terms that can be large enough to explain the recent AMM data [36]. Other chirally-enhanced one-loop contributions originating from the 3-3-1 models will be also mentioned. For convenience, the 3-3-1 models discussed in our work will be generalized in the form of the 3-3-1 model with an arbitrary parameter $\beta$ (331$\beta$) defining the electric charge operator as follows [64, 67]:

$$\begin{aligned} Q = T_3 + \beta T_8 + X. \end{aligned}$$

(3)

We have introduced the SU(3) generators $T_a$, $a=1,\ldots 8$ and X is the new quantum charge corresponding to the group $U(1)_X$. Thus, the charge operator Q depends on two parameters $\beta$ and X. The 3-3-1 models corresponding to different $\beta$ distinguish each other by new heavy leptons and quarks arranged in the third components of fermion (anti-)triplets, for example, $\beta =-\sqrt{3}, \; \frac{1}{\sqrt{3}},\;-\frac{1}{\sqrt{3}},$ and 0 for the minimal 3-3-1 model [58], the 331RN [62], with heavy singly charged leptons [63], and the simplest 3-3-1 model [65], respectively. These models result in distinguishable consequences for many interesting processes [81,82,83,84,85]. Hence, successful solutions for AMM problems in 3-3-1 models will guarantee their realities.

The explanation of AMM data in Refs. [75, 76] was just valid for the specific 331RHN model corresponding to $\beta =\pm \frac{1}{\sqrt{3}}$, which results in a special case that heavy $SU(3)_L$ leptons in the third components of the lepton (anti) triplets are exotic heavy neutrinos $\psi _{aL}\sim (e_{a}, \nu _{a}, N_{a} )^T_L$. They play roles of right-handed neutrinos $N_{aR}\equiv (N_{aL})^c$, and mix with SM neutrinos through a very special form of the total antisymmetric $3\times 3$ neutrino Dirac mass matrix $m_D$. As a result, strict relations between parameters are necessary to explain simultaneously all neutrino oscillation data, AMM $(g-2)_{\mu }$, and constraints of lepton flavor violating (cLFV) decays $e_b\rightarrow e_a \gamma$ that must be consistent with experiments. In addition, the 331RHN model needs three more neutrino singlets for generating the ISS neutrino mass matrix (331ISS), and a singly charged scalar singlet to give large one-loop contributions to AMM of muon, while the destructive interference among different one-loop contributions gives a small total one loop contribution to every decay amplitude $e_b\rightarrow e_a \gamma$.

The models 331$\beta$ and 331ISS need three $SU(3)_L$ Higgs triplets for generating non-zero masses of all quarks and leptons at the tree level, including active Dirac and Majorana neutrino masses. Two of these Higgs triplets give masses for SM fermions and gauge bosons, therefore they play the same role as the ones well-known in the two Higgs doublet models. They contain two neutral Higgs components with two vacuum expectation values (VEVs) denoted as $v_{1,2}$ satisfying $v_1^2+v_2^2\simeq (246\; \text {GeV})^2$. The important parameter $t_{\beta }\equiv v_2/v_1$, where $v_2$ is always chosen to generate the top quark mass, must have a lower bound $t_{\beta }\ge 0.33$ from the perturbative limit of the top quark Yukawa coupling. In the 331ISS model, charged lepton masses and the neutrino Dirac mass term are originated from $v_1$, therefore large $t_{\beta }>30$ is the necessary condition to give large one-loop contributions to AMM of muon consistent with experimental data [76]. In contrast, the neutrino Dirac mass term comes from the Yukawa couplings of the neutral Higgs component with VEV $v_2$ in the 331$\beta$ model. The same property also happens for the Yukawa couplings between leptons and the singly charged Higgs bosons. In addition, no $SU(3)_L$ neutral leptons are available, therefore the 331$\beta$ model needs six exotic neutrino singlets for the ISS mechanism. Therefore, the allowed regions of the parameter space give large one-loop contributions to $(g-2)_{e,\mu }$ cannot be generalized qualitatively from previous results given in Ref. [76], which requires large $t_{\beta }$. A first derivation may start from the most important property that the Yukawa couplings of singly charged Higgs singlet and $m_D$ relate to the Higgs triplet containing VEV $v_2$ instead of $v_1$. Therefore, the proper values of $t_{\beta }$ may be small enough to explain the experimental AMM data, leading to the existence of an upper bound for $t_{\beta }$. This may be conflict with the perturbative limit $t_{\beta }\ge 0.33$. This problem will be addressed in this work.

Before coming to detailed analysis, we emphasize our works is helpful because of the following reasons. First, we will see that the $331\beta$ model considered in this work explain simultaneously both experimental data of $(g-2)_{e,\mu }$ only when the mixing between $h^\pm$ and singly charged components of the $SU(3)_L$ Higgs triplets is non-zero. This non-zero mixing implies the existence of a non-trivial coupling of $h^\pm$ with two $SU(3)_L$ Higgs triplets, $f_h \left( \rho ^\dagger \eta h^+ +\text {h.c.}\right)$, which was not introduced previously. This may be an indirect link between $SU(3)_L$ Higgs triplets and the ISS neutral lepton singlets, apart from the small ISS mixing among $X_{aR}$ and $\nu _{bL}$. In other words, the existence of the $SU(3)_L$ Higgs triplet components can be detected through their decays to leptons. The second reason, many previous discussions on 3-3-1 models showed clearly that they did not accommodate the $(g-2)_{\mu }$ data unless adding some other new particles, such as vector-like fermions, etc. Our qualitative estimation in this work provides another interesting approach to confirm this conclusion. Finally, the original appearance of the 3-3-1 models solved some interesting questions, such as the answer to the question of three fermion flavors confirmed by experiments, etc. New improved versions of 3-3-1 models have been introduced to explain successfully the latest experimental results. Our model is one of them constructed to explain dark matter data, the hierarchies problems of fermion masses, etc. Many of them contain complicated particle contents including new singly charged Higgs scalars and neutral leptons. Our discussion on AMM data with a very simple Higgs sector will be helpful for further realization solutions for AMM data in these models.

Our work is arranged as follows. In Sect. 2, the 331$\beta$ model will be reviewed, where we pay attention to the leptons, gauge bosons, and Higgs sectors, giving all physical states as well as the couplings that may give large one-loop contributions to AMM. In Sect. 3, the $331\beta$ model with the GSS will be presented along with the two particular frameworks of the minimal seesaw (MSS) and simple ISS. In Sect. 4, analytic formulas for one-loop contributions to AMM are constructed. Numerical discussions for both MSS and ISS will be shown in detail. Our main results are collected in Sect. 5. There are three appendices listing master functions for one-loop contributions to AMM given in Ref. [36], analytic formulas for one-loop contributions from the singly charged Higgs bosons to AMM, and a detailed discussion on the masses and mixing of the singly charged bosons.

2 The 3-3-1 model with arbitrary $\beta$

Let us review the $331\beta$ model. Left-handed leptons are assigned to anti-triplets and RH leptons are singlets:

$$\begin{aligned}&L'_{aL}=\left( \begin{array}{c} e'_a \\ -\nu '_{a} \\ E'_a \\ \end{array} \right) _L \sim \left( 3^*~, -\frac{1}{2}+\frac{\beta }{2\sqrt{3}}\right) ,\quad a=1,2,3,\nonumber \\&e'_{aR}\sim \left( 1~, -1\right) , X_{IR}\sim \left( ~1~, 0\right) , E'_{aR} \sim \left( ~1~, -\frac{1}{2}+\frac{\sqrt{3}\beta }{2}\right) . \nonumber \\ \end{aligned}$$

(4)

The model includes K RH neutrinos $X_{IR}$, $I=1,2,\ldots ,K$, and three exotic leptons $E'^a_{L,R}$ which are much heavier than the normal leptons. The prime denotes flavor states to be distinguished with mass eigenstates introduced later. The numbers in the parentheses are to label the representation of $SU(3)_L\otimes U(1)_X$ group. The quark sector is ignored here because it is irrelevant to our present work. We note that our result will be true for 3-3-1 models consisting left-handed lepton triplets because they are equivalent with the models with lepton sector defined in Eq. (4) through a transformation keeping physical results unchanged [86, 87].

The model has nine electroweak gauge bosons, included in the following covariant derivative

$$\begin{aligned} D_{\mu }\equiv \partial _{\mu }-i g T^a W^a_{\mu }-i g_X X T^9X_{\mu }, \end{aligned}$$

(5)

where $T^9=1/\sqrt{6}$, g and $g_X$ are gauge couplings of the two groups $SU(3)_L$ and $U(1)_X$, respectively. The matrix $W^aT^a$, where $T^a =\lambda _a/2$ corresponding to a triplet representation, is written as

$$\begin{aligned} W^a_{\mu }T^a=\frac{1}{2}\left( \begin{array}{ccc} W^3_{\mu }+\frac{1}{\sqrt{3}} W^8_{\mu }&{} \sqrt{2}W^+_{\mu } &{} \sqrt{2}Y^{A}_{\mu } \\ \sqrt{2}W^-_{\mu } &{} -W^3_{\mu }+\frac{1}{\sqrt{3}} W^8_{\mu } &{} \sqrt{2}V^{B}_{\mu } \\ \sqrt{2}Y^{-A}_{\mu }&{} \sqrt{2}V^{-B}_{\mu } &{}-\frac{2}{\sqrt{3}} W^8_{\mu }\\ \end{array} \right) ,\nonumber \\ \end{aligned}$$

(6)

where we have defined the mass eigenstates of the charged gauge bosons as

$$\begin{aligned} W^{\pm }_{\mu }= & {} \frac{1}{\sqrt{2}}\left( W^1_{\mu }\mp i W^2_{\mu }\right) ,\nonumber \\ Y^{\pm A}_{\mu }= & {} \frac{1}{\sqrt{2}}\left( W^4_{\mu }\mp i W^5_{\mu }\right) ,\nonumber \\ V^{\pm B}_{\mu }= & {} \frac{1}{\sqrt{2}}\left( W^6_{\mu }\mp i W^7_{\mu }\right) . \end{aligned}$$

(7)

From Eq. (3), the electric charges of the gauge bosons are calculated as

$$\begin{aligned} A=\frac{1}{2}+\frac{\sqrt{3}\beta }{2}, \quad B=-\frac{1}{2}+\frac{\sqrt{3}\beta }{2}. \end{aligned}$$

(8)

To generate masses for gauge bosons and fermions, the model has three scalar triplets defined as

$$\begin{aligned}&\chi =\left( \begin{array}{c} \chi _{A} \\ \chi _{B} \\ \chi ^0 \\ \end{array} \right) \sim \left( 3~, \frac{\beta }{\sqrt{3}}\right) , \rho =\left( \begin{array}{c} \rho ^+ \\ \rho ^0 \\ \rho ^{-B} \\ \end{array} \right) \sim \left( 3~, \frac{1}{2}-\frac{\beta }{2\sqrt{3}}\right) \nonumber \\&\eta =\left( \begin{array}{c} \eta ^0 \\ \eta ^- \\ \eta ^{-A} \\ \end{array} \right) \sim \left( 3~, -\frac{1}{2}-\frac{\beta }{2\sqrt{3}}\right) , \; h^+\sim (1,1,1), \end{aligned}$$

(9)

where A, B denote electric charges as defined in Eq. (8); and $h^+$ is a new singly charged Higgs boson needed for giving large one-loop contributions to AMM. These Higgs bosons develop the following non-zero VEVs

$$\begin{aligned} \langle \chi ^0\rangle = \frac{u}{\sqrt{2}},\; \langle \rho ^0 \rangle = \frac{v_2}{\sqrt{2}}, \; \langle \eta ^0 \rangle = \frac{v_1}{\sqrt{2}}. \end{aligned}$$

(10)

This VEV configuration of the $331\beta$ model without $h^\pm$ was shown to be valid in Ref. [88]. This is also true for the $331\beta$ adding new singly charged Higgs boson $h^\pm$ we consider here. For convenience, we will use the following notations:

$$\begin{aligned} t_\beta&\equiv \frac{v_2}{v_1},\; \rightarrow s_{\beta }= \frac{v_2}{v},\; c_{\beta }=\frac{v_1}{v}, \end{aligned}$$

(11)

where $v^2\equiv v_1^2+v_2^2$, and $s^2_{\beta }+c^2_{\beta }=1$.

The symmetry breaking happens in two steps: $SU(3)_L\otimes U(1)_X\xrightarrow {u} SU(2)_L\otimes U(1)_Y\xrightarrow {v_1,v_2} U(1)_Q$, leading to the condition that $u\gg v_1,v_2$. After the first step, five gauge bosons will be massive and the remaining four massless ones can be identified with the before-symmetry-breaking SM gauge bosons, resulting in the following important equation:

$$\begin{aligned} \frac{g_X^2}{g^2} = \frac{6s_W^2}{1-(1+\beta ^2)s_W^2},\quad g=g_2, \end{aligned}$$

(12)

where the weak mixing angle is defined as $t_W = \tan \theta _W = g_1/g_2$, $g_{1,2}$ are the gauge couplings of the SM gauge groups $U(1)_Y$ and $SU(2)_L$, respectively. We denote $s_W = \sin \theta _W$ and $c_W = \cos \theta _W$. Putting in the value of $s_W$, we get approximately

$$\begin{aligned} |\beta | \le \sqrt{3}. \end{aligned}$$

(13)

The masses of the charged gauge bosons are

$$\begin{aligned} m^2_{Y^{\pm A}}= & {} \frac{g^2}{4}(u^2+v^{2}_1),\nonumber \\ m^2_{V^{\pm B}}= & {} \frac{g^2}{4}(u^2+v^2_2),\nonumber \\ m^2_{W^\pm }= & {} \frac{g^2}{4}(v^2_1 + v^{2}_2), \end{aligned}$$

(14)

where the gauge boson $W^\pm$ is identified with the SM one, implying that

$$\begin{aligned} v^2\equiv v_1^2+v_2^2=\frac{4 m_W^2}{g^2} \simeq (246~\text {GeV})^2. \end{aligned}$$

(15)

The above Higgs bosons are enough to generate all SM quark masses and heavy new quark masses [67, 89]. In addition, the Yukawa term $Y^u_{3a}\overline{Q_{3L}}\rho ^*u_{aR}\rightarrow \frac{Y^u_{3a}v_2}{\sqrt{2}} \overline{u_{3L}}u_{aR}$ mainly contributes to the top quark mass, $m_t\simeq \frac{Y^u_{33}v_2}{\sqrt{2}} \le \sqrt{4\pi } v_2/\sqrt{2}$, equivalently $s_{\beta }\ge \sqrt{2}m_t/(\sqrt{4\pi }v) \rightarrow t_{\beta }\ge 0.3$.

In general, the mixing between a SM lepton and a new lepton is allowed if they have the same electric charge in some particular values of $\beta$. This mixing effect will be neglected in the $331\beta$ model under consideration. The Yukawa Lagrangian now is

$$\begin{aligned} -{\mathcal {L}}^\text {yuk}_{\mathrm {lepton}}&= Y^e_{ab} \overline{e'_{aR}} \eta ^TL'_{bL} + Y^E_{ab} \overline{E'_{aR}} \chi ^T L'_{bL} + Y^X_{Ib} \overline{X_{IR}} \rho ^TL'_{bL} \nonumber \\&\quad + \frac{1}{2}M_{N,IJ} \overline{X_{IR}}(X_{JR})^c + Y^{h}_{Ib}\overline{(X_{IR})^c} e'_{bR} h^+ +\text {h.c.}, \end{aligned}$$

(16)

where $a,b=1,2,3$ are family indices, and $I=1,2,3,\ldots K$ are the number of new neutral lepton singlets. The perturbative limit of the $Y^{h}$ is important in this work, which should satisfy $|Y^{h}_{Ia}|<\sqrt{4\pi }$. In fact, the trust values of $|Y^{h}_{Ia}|$ may be smaller [90]. In Lagrangian (16), the neutrino Dirac mass matrix comes from the third term including the Higgs triplet $\rho$ which also generates the top quark mass. This important property is different from that given in Ref. [76], hence the dependence of the Dirac mass term, the Yukawa couplings of singly charged Higgs bosons, and the perturbative condition of the Yukawa couplings with heavy neutrinos $Y^X_{Ib}$ on $t_{\beta }$ will be different between two models 331$\beta$ and 331ISS discussed in Ref. [76]. In later discussions, we will set $K=3$ and $K=6$ for the respective MSS and ISS mechanisms considered in this work. The corresponding mass terms are:

$$\begin{aligned} -{\mathcal {L}}^\text {mass}_{\mathrm {lepton}}&= \frac{Y^e_{ab}v_1}{\sqrt{2}} \overline{e'_{aR}}e'_{bL} + \frac{Y^E_{ab}u}{\sqrt{2}} \overline{E'_{aR}} E'_{bL} \nonumber \\&\quad + \frac{1}{2} \begin{pmatrix} \overline{(\nu '_L)^c}&\overline{X_R} \end{pmatrix} {\mathcal {M}}^{\nu } \begin{pmatrix} \nu '_L \\ (X_R)^c \end{pmatrix} +\text {h.c.}, \nonumber \\ {\mathcal {M}}^{\nu }&= \begin{pmatrix} 0_{3\times 3}&{} M^T_D \\ M_D&{} M_N \end{pmatrix} , \end{aligned}$$

(17)

where $(M_D)_{Ib}\equiv M_{D,Ib}= \frac{-Y^X_{Ib} v_2}{\sqrt{2}}$, $\nu '_L=(\nu '_1, \nu '_2, \nu '_3)_L^T$ and $X_R=(X_1,X_2,\ldots ,X_K)_R^T$. Note that, here, charged lepton masses and $M_D$ come from different Higgs triplets, while these mass terms discussed in Ref. [76] come from the same Higgs triplet. Therefore, the effects on $\Delta a_{\mu ,e}$ relating to the relevant Yukawa couplings in the model under consideration will be different from those discussed in Ref. [76]. At present, the active neutrino masses and mixing are still generated from the GSS mechanism. The total mixing matrix is defined as

$$\begin{aligned} U^{\nu T} {\mathcal {M}}^{\nu } U^{\nu }&= \hat{{\mathcal {M}}}^{\nu }=\text {diag}(m_{n_1},\;m_{n_1},\ldots ,m_{n_{K+3}}), \nonumber \\ \begin{pmatrix} \nu '_L \\ (X_R)^c \end{pmatrix}&= U^{\nu } n_{L},\; \begin{pmatrix} (\nu '_L)^c \\ X_R \end{pmatrix} = U^{\nu *} n_{R}= U^{\nu *} (n_{L})^c, \end{aligned}$$

(18)

where $n_{L,R}=(n_{1},n_{2},\ldots , n_{(K+3)})_{L,R}$ are Majorana neutrino mass eigenstates satisfying $n_{iL,R}=(n_{iR,L})^c$, and the four-component forms are $n_i=( n_{iL}, n_{iR})^T$.

From now on we will work in the basis where the SM charged leptons are in their mass eigenstates, namely $Y^e_{ab} = Y^e_{ab} \delta _{ab}$ and $e'_a= e_a$ in Eqs. (16) and (17). This can always be done without loss of generality. The transformations from the flavor states to mass eigenstates of the heavy lepton $E_a$ are defined as

$$\begin{aligned} E'_{aL} = V^L_{ab} E_{bL}, \quad E'_{aR} = V^R_{ab} E_{bR}, \end{aligned}$$

(19)

where $V^{L,R}$ is a $3\times 3$ unitary mixing matrix for new charged leptons.

For the Higgs sector, the ratios between VEVs are used to define two mixing angles:

$$\begin{aligned} s^2_{iu}=\sin ^2\beta _{iu}= \frac{v_i^2}{u^2+v_i^2},\quad i=1,2. \end{aligned}$$

(20)

We will also use the following notations $t_{iu}=s_{iu}/c_{iu}$. The scalar potential is

$$\begin{aligned} V_{\mathrm {h}}&=\mu _1^2 \eta ^{\dagger }\eta +\mu _2^2\rho ^{\dagger }\rho +\mu _3^2\chi ^{\dagger }\chi +\lambda _1 \left( \eta ^{\dagger }\eta \right) ^2\nonumber \\&\quad +\lambda _2\left( \rho ^{\dagger }\rho \right) ^2 +\lambda _3\left( \chi ^{\dagger }\chi \right) ^2\nonumber \\&\quad +\lambda _{12}(\eta ^{\dagger }\eta )(\rho ^{\dagger }\rho ) +\lambda _{13}(\eta ^{\dagger }\eta )(\chi ^{\dagger }\chi ) +\lambda _{23}(\rho ^{\dagger }\rho )(\chi ^{\dagger }\chi )\nonumber \\&\quad +{\tilde{\lambda }}_{12} (\eta ^{\dagger }\rho )(\rho ^{\dagger }\eta ) +{\tilde{\lambda }}_{13} (\eta ^{\dagger }\chi )(\chi ^{\dagger }\eta ) +{\tilde{\lambda }}_{23} (\rho ^{\dagger }\chi )(\chi ^{\dagger }\rho )\nonumber \\&\quad + \sqrt{2} f\left( \epsilon _{ijk}\eta ^i\rho ^j\chi ^k +\text {h.c.} \right) \nonumber \\&\quad +\mu _4^2 h^+h^- + f_h \left( \rho ^\dagger \eta h^+ +\text {h.c.}\right) \nonumber \\&\quad + (h^+h^-) \left( \lambda ^h_1 \eta ^{\dagger }\eta + \lambda ^h_2\rho ^{\dagger }\rho +\lambda ^h_3 \chi ^{\dagger }\chi \right) + \lambda ^h_4 \left( h^+h^-\right) ^2, \end{aligned}$$

(21)

where the last line includes all terms relating to the singly charged Higgs boson that does not appear in the previous versions [64, 67]. The triple coupling $f_h$ is a very important parameter controlling the mixing between the singly charged Higgs components of the two $SU(3)_L$ Higgs triplets and the Higgs singlet $h^\pm$. It is emphasized that the existence of $f_h$ is a very interesting feature of the 331$\beta$ that did not mention previously, because of the nontrivial property that the coupling $\rho ^\dagger \eta h^+$ always respects $U(1)_X$ symmetry for arbitrary $\beta$.

As we mentioned above, the VEV configuration considered in this work is the same as that chosen in Refs. [88, 91], which was shown to be consistent with the unitarity, perturbativity and bounded-from-below (BFB) constraints. On the other hand, the exact necessary and sufficient BFB constraints are still difficult to determine [92]. They relate to the copositive (conditionally positive) conditions of the quartic term of the Higgs potential to guarantee the existence of local minima, as discussed in Ref. [93]. Determining which local minimum is the global one defining the stability of the Higgs potential corresponding to the VEV structure chosen in this work is more difficult. A method introduced in Ref. [94] can solve this problem, but it is still difficult to apply to BSM models with complicated Higgs sectors such as the 3-3-1 models. The discussions on the VEV structure mentioned in Refs. [88, 91] were not addressed clearly to the vacuum stability issue. In the model under consideration, the global minimum corresponding to the VEV structure mentioned above requires more relations between Higgs couplings. We hope that the large number of Higgs couplings appearing in the Higgs potential (21) will allow the existence of these new relations consistent with the available constraints. They should be discussed in more detail when the Higgs phenomenology is focused. It is not our scope in this work, we therefore will not discuss more.

A detailed calculation to derive masses and mixing matrix of the singly charged Higgs bosons is shown in Appendix C. From this, the relations between the mass and flavor eigenstates of singly charged Higgs bosons are

$$\begin{aligned} \begin{pmatrix} \rho ^{\pm } &{} \\ \eta ^{\pm } &{} \\ h^{\pm } &{} \end{pmatrix}&= \begin{pmatrix} -s_{\beta }&{} c_{\alpha } c_{\beta } &{} s_{\alpha } c_{\beta }\\ c_{\beta }&{} c_{\alpha } s_{\beta } &{} s_{\alpha } s_{\beta }\\ 0 &{} -s_{\alpha } &{} c_{\alpha } \end{pmatrix} \begin{pmatrix} \phi _W^{\pm } &{} \\ H_1^{\pm } &{} \\ H_2^{\pm } &{} \end{pmatrix} , \end{aligned}$$

(22)

where $\phi _W^\pm$ are the Goldstone bosons of $W^\pm$, $H^\pm _{1,2}$ are two physical states with masses $m_{H^\pm _{1,2}}$, and $\alpha$ is a new mixing parameter defined in Eq. (C5). Normally, $m_{H^\pm _{1,2}}$ and $\alpha$ are functions of the potential couplings. For convenience, we will consider $\alpha$, and $m_{H^+_{1,2}}$ as free parameters, while $\mu _4$, f, and $f_h$ are chosen as functions of all free ones, namely:

$$\begin{aligned} \mu ^2_{4}&= c^2_{\alpha } m^2_{H^\pm _2} + s^2_{\alpha } m^2_{H^\pm _1} -\frac{1}{2}\left( \lambda ^h_{1} s^2_{\beta }v^2 + \lambda ^h_{2} c^2_{\beta }v^2 +\lambda ^h_{3} u^2\right) , \nonumber \\ f&=-\frac{c_{\beta } s_{\beta } \left( -{\tilde{\lambda }}_{12} v^2+2 c_{\alpha }^2 m_{H_1^{\pm }}^2+2 s_{\alpha }^2 m_{H_2^{\pm }}^2\right) }{2 u}, \nonumber \\ f_h&=- \frac{\sqrt{2} s_{\alpha }c_{\alpha }\left( m^2_{H^\pm _1} - m^2_{H^\pm _2} \right) }{v}. \end{aligned}$$

(23)

The case of $s_{\alpha }=0$ or $c_{\alpha }=0$ will return to the decouple limit between $h^{\pm }$ and the $SU(3)_L$ Higgs triplets mentioned in Ref. [76], where this limit is allowed. In contrast, we will see that $s_{2\alpha }=2s_{\alpha }c_{\alpha } \ne 0$, equivalently, the triple Higgs coupling $f_h\ne 0$, is one of the necessary condition to give large one-loop contributions to AMM in the $331\beta$.

The relations between the mass and flavor eigenstates of other charged Higgs bosons are:

$$\begin{aligned} \left( \begin{array}{c} \eta ^{\pm A} \\ \chi ^{\pm A} \\ \end{array} \right)&= \left( \begin{array}{cc} s_{1u} &{} c_{1u} \\ -c_{1u} &{} s_{1u} \\ \end{array} \right) \left( \begin{array}{c} \phi ^{ \pm A}_Y \\ H^{\pm A} \\ \end{array} \right) , \nonumber \\ \left( \begin{array}{c} \rho ^{\pm B} \\ \chi ^{\pm B} \\ \end{array} \right)&=\left( \begin{array}{cc} s_{2u} &{} -c_{2u} \\ c_{2u} &{} s_{2u} \\ \end{array} \right) \left( \begin{array}{c} \phi ^{\pm B}_V \\ H^{\pm B} \\ \end{array} \right) , \end{aligned}$$

(24)

where $\phi _W^\pm$, $\phi ^{ \pm A}_Y$ and $\phi ^{\pm B}_V$ are the Goldstone bosons of $W^\pm$, $Y^{\pm A}$ and $V^{\pm B}$, respectively. The masses of the charged Higgs bosons $H^{\pm A}, H^{\pm B}$ are

$$\begin{aligned} m^2_{H^{ A}}= & {} (u^2+v_1^{2})\left( \frac{-fv_2}{v_1u}+\frac{1}{2}{\tilde{\lambda }}_{13}\right) , \nonumber \\ m^2_{H^{B}}= & {} (u^2+v^2_2)\left( \frac{-fv_1}{uv_2}+\frac{1}{2}{\tilde{\lambda }}_{23}\right) . \end{aligned}$$

(25)

Because the neutral Higgs bosons couple to charged lepton through the Yukawa couplings of the form $S^0 \overline{e_a} e_a$, which is the same form as that of the SM-like Higgs boson predicted by the SM, the corresponding one-loop contributions to AMM is very small. Hence, they will be ignored in our calculation from now on. The discussion on the identification of the SM-like Higgs boson can be found in Ref. [89]. In total, there are six charged Higgs bosons, one neutral pseudoscalar Higgs and three neutral scalar Higgs bosons.

3 The minimal seesaw and inverse seesaw mechanisms in the neutral lepton sector

In this section, we will collect important properties of the MSS and ISS mechanisms used in our calculation. In the GSS framework, the neutrino mixing matrix is parameterized in the following form:

$$\begin{aligned} U^{\nu }{=}\left( \begin{array}{cc} \left( I_3-\frac{1}{2}RR^{\dagger } \right) U_{{\mathrm {PMNS}}} &{} RV \\ -R^\dagger U_{{\mathrm {PMNS}}} &{} \left( I_K -\frac{1}{2}R^{\dagger } R\right) V \\ \end{array} \right) {+}{\mathcal {O}}(R^3), \nonumber \\ \end{aligned}$$

(26)

where V is a $K\times K$ unitary matrix; R is a $3\times K$ matrix satisfying $|R_{aI}|<1$ for all $a=1,2,3$, and $I=1,2,\ldots ,K$. The $3\times 3$ unitary matrix $U_{{\mathrm {PMNS}}}$ is the Pontecorvo–Maki–Nakagawa–Sakata (PMNS) matrix [95]. The GSS relations are

$$\begin{aligned} R&\simeq M^{\dagger }_D {M^*_N}^{-1} , \nonumber \\ m_{\nu }&\simeq -M^T_DM_N^{-1}M_D =U^*_{{\mathrm {PMNS}}}{\hat{m}}_{\nu } U^{\dagger }_{{\mathrm {PMNS}}} , \nonumber \\ V^*{\hat{M}}_NV&\simeq M_{N} + \frac{1}{2}R^TR^*M_N +\frac{1}{2}M_N R^{\dagger }R, \end{aligned}$$

(27)

where ${\hat{m}}_{\nu }={\mathrm {diag}}(m_{n_1},m_{n_1},m_{n_3})$ and ${\hat{M}}_N={\mathrm {diag}}(m_{n_{4}}, m_{n_5},\ldots , m_{n_{(K+3)}})$ consist of three active neutrino and K new heavy neutrino masses, respectively. In many formulas discussed below, we will use the equality that $\left( {\hat{m}}_{\nu }\right) _{cc}=m_{n_c}$ with $c=1,2,3$ are active neutrino masses.

The general parameterization of $M_D$ was introduced in Ref. [96]. In the limit of the MSS mechanism with $K=3$, we will use the simplest forms of $M_N$ and $M_D\equiv m_D$ as follows [97, 98],

$$\begin{aligned} M_N =M_0 I_3, \; M_D\equiv m_D=i \sqrt{M_0{\hat{m}}_\nu } U^{\dagger }_{{\mathrm {PMNS}}}. \end{aligned}$$

(28)

The relations in (27) reduce to the following simple form:

$$\begin{aligned}&R = -iU_{{\mathrm {PMNS}}} \left( \frac{{\hat{m}}_{\nu }}{M_0}\right) ^{1/2}, \quad V\simeq I_3,\nonumber \\&{\hat{M}}_N\simeq M_N, \; m_{n_{4,5,6}}\simeq M_0. \end{aligned}$$

(29)

In the ISS mechanism with $K=6$, the total neutrino mixing matrix $U^\nu$ in (26) is $9\times 9$. In the 331$\beta$ model, the well-known ISS form of the total neutrino matrix can be derived from the requirement that the model respects a global $U(1)_{{\mathcal {L}}}$ symmetry called the generalized lepton number, which is defined from the following formula: ${\mathcal {L}}$: $L\equiv -\frac{4}{\sqrt{3}}T^8 +{\mathcal {L}}$, where L is the normal lepton number defined in the SM that $L(\ell )=1$ for all SM leptons $\ell =e,\mu ,\tau ,\nu _{eL},\nu _{\mu L}, \nu _{\tau L}$ and zero for all other SM particles including quarks, gauge, and Higgs bosons [99,100,101]. The specific ${\mathcal {L}}$ assignments for all Higgs bosons and fermions in two 3-3-1 models with right-handed neutrinos and the minimal ones in Refs. [99,100,101] are the same and independent with $\beta$, therefore they are valid for the 331$\beta$ model with ${\mathcal {L}}(L'_{aL})=1/3$, ${\mathcal {L}}(e'_{aR})=1$. In addition, introducing ${\mathcal {L}}(\rho )={\mathcal {L}}(\eta )=2/3$, ${\mathcal {L}}(\chi )=-4/3$, ${\mathcal {L}}(h^+)=0$, and ${\mathcal {L}}(E'_{aR})=-1$ will result in the Lagrangian (16) conserving ${\mathcal {L}}$, except the mass term $\overline{X_{IR}}(X_{JR})^c$, which includes soft-breaking terms. Namely, choosing that ${\mathcal {L}}(X_{IR})=1$ with $I\le 3$ and ${\mathcal {L}}(X_{IR})=-1$ with $I>3$, the conserved Lagrangian (16) implies that $Y^h=\left( {\mathcal {O}}_{3\times 3},\; Y^h_2\right) ^T$ and $Y^X=\left( Y^X_1,\;{\mathcal {O}}_{3\times 3}\right) ^T$, where $Y^h_2$, and $Y^X_1$ are two $3\times 3$ matrices, and ${\mathcal {O}}_{3\times 3}$ is the $3\times 3$ null matrix. This leads to the ISS form of $M_D=(m_D,\; {\mathcal {O}}_{3\times 3})^T$. The $6\times 6$ Majorana mass matrix $M_N$ consists of three parts denoted as $M_R$, $\mu _{X}$, and $\mu '_{X}$. The conserved mass term $\left( M_R\right) _{ab}\equiv M_{N,a(b+3)}$ with $a,b\le 3$ can be arbitrary large, while the soft-breaking term $\left( \mu '_{X}\right) _{ab}\equiv M_{N,ab}$ and $\left( \mu _{X}\right) _{ab}\equiv M_{N,(a+3)(b+3)}$ should be small. Inserting the ISS form of $M_D$ into the GSS relations to derive the active neutrino mass term $m_{\nu }$, we find that small $\mu '_X$ does not affect significantly the final result. Hence, we assume the simplest case of $\mu '_X= {\mathcal {O}}_{3\times 3}$ without loss of generality.

Now, the Dirac and Majorana mass matrices have well-known ISS forms as follows [97, 98]

$$\begin{aligned} M^T_D= (m^T_D, {\mathcal {O}}_{3\times 3}), M_N=\left( \begin{array}{cc} {\mathcal {O}}_{3\times 3}&{} M_R \\ M^T_R &{} \mu _X \\ \end{array} \right) . \end{aligned}$$

(30)

Defining $M=M_R\mu _X^{-1}M_R^T$, the ISS relations now are

$$\begin{aligned} R&= M^{\dagger }_D{M^*_N}^{-1} = \left( -m_D^{\dagger }M^{*-1}, m^\dagger _D\left( M^\dagger _R\right) ^{-1} \right) ,\nonumber \\ m_{\nu }&=-M^T_DM_N^{-1}M_D = m_D^T \left( M_R^T\right) ^{-1}\mu _XM_R^{-1}m_D,\nonumber \\ V^*{\hat{M}}_NV^{\dagger }&\simeq M_{N} + \frac{1}{2}R^TR^*M_N +\frac{1}{2}M_N R^{\dagger }R. \end{aligned}$$

(31)

In the ISS framework, $m_D$ is parameterized in terms of many free parameters, hence it is enough to choose $\mu _X=\mu _X I_3$. The parameter $\mu _X$ is a new scale making the most important difference between the neutrino mixing matrices in the ISS and MSS. We also assume that $M_R={\hat{M}}_R= M_0I_3$. A simple parameterization of $m_D$ is $m_D= {\mathrm {diag}}(\sqrt{M_{11}},\; \sqrt{M_{22}},\; \sqrt{M_{33}})\sqrt{{\hat{m}}_\nu } U^{\dagger }_{{\mathrm {PMNS}}}$ [98], which is completely different from the total antisymmetric $m_D$ given in Ref. [76]. The ISS condition $|{\hat{m}}_{\nu }|\ll |\mu _X|\ll |m_D|\ll M_0$ gives $\frac{\sqrt{\mu _X {\hat{m}}_{\nu }}}{M_0}\simeq 0$. Then we have

$$\begin{aligned} {\hat{M}}_N= \left( \begin{matrix} {\hat{M}}_R &{} {\mathcal {O}}_{3\times 3} \\ {\mathcal {O}}_{3\times 3} &{} {\hat{M}}_R \end{matrix} \right) \simeq M_0I_6 , \; V\simeq \dfrac{1}{\sqrt{2}} \left( \begin{matrix} -iI_3 &{} I_3 \\ iI_3 &{} I_3 \end{matrix} \right) . \end{aligned}$$

(32)

The important results for the ISS mechanism are:

$$\begin{aligned} m_D&= M_0{\hat{x}}_\nu ^{1/2} U^\dagger _{{\mathrm {PMNS}}},\nonumber \\ R&=\left( -U_{{\mathrm {PMNS}}}\frac{\sqrt{\mu _X {\hat{m}}_{\nu }}}{M_0},\; U_{{\mathrm {PMNS}}}{\hat{x}}_\nu ^{1/2} \right) \nonumber \\&\simeq \left( {\mathcal {O}}_{3\times 3},\; U_{{\mathrm {PMNS}}}{\hat{x}}_\nu ^{1/2} \right) , \end{aligned}$$

(33)

where $\; {\hat{x}}_\nu \equiv \frac{{\hat{m}}_\nu }{\mu _X}$ satisfying max$[\left( |{\hat{x}}_\nu |\right) _{ab}]\ll 1$ for all $a,b=1,2,3$.

In numerical discussion, we will use the best-fit values of the neutrino oscillation data [95] corresponding to the normal order (NO) scheme with $m_{n_1}<m_{n_2}<m_{n_3}$, namely

$$\begin{aligned}&s^2_{12}=0.32,\; s^2_{23}= 0.547,\; s^2_{13}= 0.0216 ,\; \delta = 218 \;[{\mathrm {Deg}}] ,\nonumber \\&\Delta m^2_{21}=7.55\times 10^{-5} [{\mathrm {eV}}^2], \; \Delta m^2_{32}=2.424\times 10^{-3} [{\mathrm {eV}}^2]. \end{aligned}$$

(34)

In numerical calculation, we will use the following formulas

$$\begin{aligned} {\hat{m}}_{\nu }&= \left( {\hat{m}}^2_{\nu }\right) ^{1/2}\nonumber \\&={\mathrm {diag}} \left( m_{n_1}, \; \sqrt{ m_{n_1}^2 +\Delta m^2_{21}},\; \sqrt{m_{n_1}^2 +\Delta m^2_{21} +\Delta m^2_{32}} \right) , \nonumber \\ U_{{\mathrm {PMNS}}}&=\left( \begin{array}{ccc} c_{12} c_{13} &{} c_{13} s_{12} &{} s_{13} e^{-i \delta } \\ -c_{23} s_{12}-c_{12} s_{13} s_{23} e^{i \delta } &{} c_{13} c_{23}-s_{12} s_{13} s_{23} e^{i \delta } &{} c_{13} s_{23} \\ s_{12} s_{23}-c_{12} c_{23} s_{13} e^{i \delta } &{} -c_{23} s_{12} e^{i \delta } s_{13}-c_{13} s_{23} &{} c_{13} c_{23} \\ \end{array} \right) \nonumber \\&\simeq \left( \begin{array}{ccc} 0.816 &{} 0.560 &{} 0.147 e^{-i \delta } \\ -0.381-0.09 e^{ i \delta } &{} 0.555-0.062 e^{ i \delta } &{} 0.732 \\ 0.418-0.082 e^{ i \delta } &{} -0.61-0.056 e^{ i \delta } &{} 0.666 \\ \end{array} \right) . \end{aligned}$$

(35)

These neutrino masses satisfy the constraint from Plank 2018 [102] that $\sum _{i=1}^{3}m_{n_a}\le 0.12\; {\mathrm {eV}}$. With the best-fit values of $\Delta m^2_{ij}$ we have $m_{n_1} \le 0.028$ eV.

The other well-known numerical parameters are given in Ref. [95], namely

$$\begin{aligned} g&=0.652,\; \alpha _e=\frac{1}{137}= \frac{e^2}{4\pi },\; s^2_{W}=0.231,\nonumber \\ m_e&=5\times 10^{-4}~\text {GeV},\; m_{\mu }=0.105~\text {GeV}, \; m_W=80.385~\text {GeV}. \end{aligned}$$

(36)

Also, the inverted order (IO) scheme with $m_{n_3}<m_{n_1}<m_{n_2}$ can be considered a similar way, but the qualitative results are the same as those from the NO scheme, so we will not present here.

The non-unitary part of the active neutrino mixing matrix $\left( I_3-\frac{1}{2}RR^{\dagger } \right) U_{{\mathrm {PMNS}}}$ is constrained by other phenomenology such as electroweak precision, lepton flavor violating decays of charged leptons (cLFV) [103,104,105], namely

$$\begin{aligned} \eta \equiv \frac{1}{2}\left| RR^{\dagger }\right| < \begin{pmatrix} 2\times 10^{-3} &{} 3.5\times 10^{-5} &{} 8.\times 10^{-3}\\ 3.5\times 10^{-5}&{}8\times 10^{-4} &{} 5.1\times 10^{-3} \\ 8\times 10^{-3}&{} 5.1\times 10^{-3} &{} 2.7\times 10^{-3} \end{pmatrix}. \end{aligned}$$

(37)

This constraint is consistent with the data used popularly in recent works [48, 106]. The constraint on $\eta$ may be more strict, depending on particular models. For example in the type III general and inverse seesaw models, $|\eta _{aa}|\le {\mathrm {O}}(10^{-4})$ [37, 107]. We will choose the values that $|\eta _{33}|\le 10^{-3}$ in our numerical discussion.

In the next section, we will consider the one-loop contributions to $\Delta a_{\mu ,e}$.

4 Analytical formulas for AMM and numerical discussion

From the above information we obtain all vertices giving one-loop contributions to $e_b \rightarrow e_a \gamma$ decay rates and $a_{e_a}$. They are collected from Lagrangian (16). All relevant couplings are listed in the following Lagrangian

$$\begin{aligned} {\mathcal {L}}&= \frac{g}{\sqrt{2} m_W}\sum _{k=1}^2\sum _{a=1}^3 \sum _{i=1}^{K+3} \overline{n_i} \left[ \lambda ^{L,k}_{ia} P_L +\lambda ^{R,k}_{ia}P_R \right] e_a H^+_k\nonumber \\&\quad -\frac{g}{\sqrt{2} m_Y} \sum _{a,c=1}^3 V^{L*}_{ac}\overline{E_c} \left[ \frac{m_{e_a}}{t_{1u}} P_L+m_{E_c}t_{1u}P_R \right] e_a H^{A}\nonumber \\&\quad +\sum _{a=1}^3\sum _{i=1}^{K+3}\frac{g}{\sqrt{2}} U^{\nu *}_{ai} \overline{n_{i}}\gamma ^{\mu }P_L e_aW^+_{\mu } \nonumber \\&\quad +\sum _{a,c=1}^3 \frac{g}{\sqrt{2}} V^{L*}_{ac} \overline{E_c}\gamma ^{\mu }P_L e_aY^{A}_{\mu } +\text {h.c.}, \end{aligned}$$

(38)

where

$$\begin{aligned} \lambda ^{L,1}_{ia}&= \sum _{I=1}^KM_{D,Ia}t^{-1}_{\beta }c_{\alpha }U^{\nu }_{(I+3)i}\nonumber \\&\simeq t^{-1}_{\beta }c_{\alpha }\times \left[ \begin{array}{cc} -\left( M^T_{D}R^{\dagger }U_{{\mathrm {PMNS}}}\right) _{ai}, &{} \quad i\le 3 \\ \left( M^T_{D} \left( I_K- \frac{1}{2}R^{\dagger }R\right) V\right) _{a(i-3)}, &{} \quad i> 3 \end{array}\right. ,\nonumber \\ \lambda ^{L,2}_{ia}&\simeq \lambda ^{L,1}_{ia}t_{\alpha },\nonumber \\ \lambda ^{R,1}_{ia}&= m_{e_a}t_{\beta } c_{\alpha } U^{\nu *}_{ai} + \sum _{I=1}^K \frac{v}{\sqrt{2}} Y^{h}_{Ia} s_{\alpha }U^{\nu *}_{(I+3)i}\nonumber \\&\simeq \left[ \begin{array}{cc} m_{e_a}t_{\beta } c_{\alpha } \left( \left( I_3 -\frac{1}{2}R^*R^T \right) U^*_{{\mathrm {PMNS}}}\right) _{ai} -\frac{vs_{\alpha }}{\sqrt{2}}\left( Y^{hT}R^{T}U^*_{{\mathrm {PMNS}}}\right) _{ai}, &{} \quad i\le 3 \\ m_{e_a}t_{\beta } c_{\alpha } \left( RV\right) ^*_{a(i-3)}+ \frac{vs_{\alpha }}{\sqrt{2}}\left( Y^{hT} \left( I_K- \frac{1}{2}R^{T}R^*\right) V^*\right) _{a(i-3)} &{} \quad i> 3 \end{array}\right. ,\nonumber \\ \lambda ^{R,2}_{ia}&= m_{e_a}t_{\beta } s_{\alpha } U^{\nu *}_{ai} - \sum _{I=1}^K \frac{v}{\sqrt{2}} Y^{h}_{Ia} c_{\alpha }U^{\nu *}_{(I+3)i}\nonumber \\&\simeq \left[ \begin{array}{cc} m_{e_a}t_{\beta } s_{\alpha } \left( \left( I_3 -\frac{1}{2}R^*R^T \right) U^*_{{\mathrm {PMNS}}}\right) _{ai} -\frac{vc_{\alpha }}{\sqrt{2}}\left( Y^{hT}R^{T}U^*_{{\mathrm {PMNS}}}\right) _{ai}, &{} \quad i\le 3 \\ m_{e_a}t_{\beta } s_{\alpha } \left( RV\right) ^*_{a(i-3)}- \frac{vc_{\alpha }}{\sqrt{2}}\left( Y^{hT} \left( I_K- \frac{1}{2}R^{T}R^*\right) V^*\right) _{a(i-3)} &{} \quad i > 3 \end{array}\right. . \end{aligned}$$

(39)

We can see that $\lambda ^{L,k}_{ia}$ with $k=1,2$ contains a factor $t_{\beta }^{-1}$, which is the inverse value included in $\lambda ^{L,1}_{ia}$ introduced in Refs. [75, 76], where the regions predicting large $(g-2)_{\mu }$ require large $t_{\beta }>40$, consistent with the perturbative constraint $t_{\beta }\ge 0.3$. In contrast, the 331$\beta$ model may support small $t_{\beta }$ for large $(g-2)_{e,\mu }$, which may be excluded if $t_{\beta }<0.3$ is required. Hence, the valid regions satisfying the experimental AMM data must be determined through detailed numerical investigation.

We do not list here the couplings of neutral gauge and Higgs bosons because they give suppressed contributions to $a^{\mathrm {NP}}_{e_a}$. In particularly, the relevant couplings are only with usual charged leptons $s^0\overline{e_a}e_a$ and $V^0_{\mu }\overline{e_a}\gamma ^{\mu }e_a$. The one-loop contribution from $V_0=Z$ is the same as that predicted by the SM. Another one from heavy neutral gauge boson $V_0=Z'$ is suppressed by a factor of $m^2_{Z}/m^2_{Z'}$. The contributions from neutral Higgs bosons are not larger than the one from the SM-like Higgs boson with a suppressed order of ${\mathcal {O}}(10^{-14})$.

The form factors $c^X_{(ab)R}$ relating to new one-loop contributions from exchanging X boson to the $\Delta a_{e_a}$ and cLFV decays were introduced in Ref. [36], see Appendix A. Formulas of $c^X_{(ab)R}$ from $X=H^A,W^\pm , Y^{\pm A}$ are:

$$\begin{aligned} c^{H^A}_{(ab)R}&= \frac{e g^2 m_{e_a}}{32\pi ^2 m_Y^2 m^2_{H^A}} \sum _{c=1}^3 V^{L}_{ac} V^{L*}_{bc} \left\{ \frac{}{} m^2_{E_c} \left[ f_{\Phi } \left( t_{H,c}\right) + B g_\Phi \left( t_{H,c}\right) \right] \right. \nonumber \\&\quad + \left. \left[ m^2_{e_b} t^{-2}_{1u} + m^2_{E_c}t^2_{1u} \right] \left[ {\tilde{f}}_\Phi \left( t_{H,c}\right) + B {\tilde{g}}_\Phi \left( t_{H,c}\right) \right] \right\} , \end{aligned}$$

(40)

$$\begin{aligned} c^W_{(ab)R}&\equiv \frac{e g^2m_{e_b} }{32 \pi ^2m^2_{W} }\sum _{i=1}^{K+3} U^{\nu }_{ai}U^{\nu *}_{bi} {\tilde{f}}_V \left( t_{W,i}\right) , \end{aligned}$$

(41)

$$\begin{aligned} c^Y_{(ab)R}&\equiv \frac{e g^2m_{e_b} }{32 \pi ^2 m^2_{Y}} \sum _{c=1}^3 V^{L}_{ac}V^{L*}_{bc} \left[ {\tilde{f}}_V \left( t_{Y,c}\right) +B {\tilde{g}}_V \left( t_{Y,c}\right) \right] , \end{aligned}$$

(42)

where $t_{H,c}\equiv m^2_{E_c}/m^2_{H^A}$, $t_{W,i}\equiv m^2_{n_i}/m^2_{W}$, and $t_{Y,c}\equiv m^2_{E_c}/m^2_{Y}$.

The particular parameterisations of the MSS and ISS used in this work give the limit $m_{n_i}=0$ with $i=1,2,3$; $m_{n_i}=M_0$ with all $i>3$; $c^X_{(ab)R}=0$ with $a\ne b$ and $X=H^\pm _{1,2},H^A,W,Y$. To avoid large cLFV rates, we also consider the simple limit that $M_{ab}=M_0\delta _{ab}$, $m_{E_1}=m_{E_2}=m_{E_3}\equiv m_E$, and $V^L=I_3$, so that $c^{H^A}_{(ab)R}=0$ and $c^Y_{(ab)R}=0$ for $a\ne b$. Therefore, the cLFV decay rates are much smaller than the current experimental constraints [108, 109]. We will not discuss them from now on.

The one-loop contribution of $X=H^\pm _{1,2},H^A,W,Y$ to AMM of a charged lepton $e_a$ is

$$\begin{aligned} a_{e_a}(X)=- \frac{4m_{e_a}}{e}{\mathrm {Re}} \left[ c^X_{(aa)R}\right] . \end{aligned}$$

(43)

And the deviation from the SM is defined as follows:

$$\begin{aligned} \Delta a_{e_a}&= \sum _{X} a_{e_a}(X) + \Delta a_{e_a}(W), \; \Delta a_{e_a}(W)\nonumber \\&\equiv a_{e_a}(W)-a^{(1){\mathrm {SM}}}_{e_a}(W), \end{aligned}$$

(44)

where $X=H^\pm _{1,2},H^A,Y$, and $a^{(1){\mathrm {SM}}}_{\mu }(W) \simeq 3.83 \times 10^{-9}$ [110]. In the $331\beta$ model, the SM-like Higgs and gauge bosons have the same couplings with usual charged lepton $e_a$ as those predicted by the SM, hence they do not contribute to $\Delta a_{e_a}$. Also, the heavy neutral Higgs and gauge bosons will give one-loop contributions smaller than the ones of the SM-like gauge and Higgs bosons by suppressed factors of $m^2_h/m^2_{H^0}<10^{-1}$ and $m^2_Z/m^2_{Z'}< 6.\times 10^{-4}$. We have used heavy neutral Higgs mass $m_{H^0}>1$ TeV, and $m_{Z'}>3.7$ TeV from the constraints concerned for 3-3-1 models from LHC [111,112,113] and the combination of weak charge data of Cesium and proton [81].

One-loop contributions from heavy charged lepton $E_{a}$ exchanges are

$$\begin{aligned} \Delta a_{\mu }(H^A)&\simeq - \frac{e g^2 m^2_{\mu }}{8\pi ^2 m_W^2}\times \left\{ \frac{m_W^2}{m^2_Y}\left[ t_{H^A}f_{\Phi } \left( t_{H^A}\right) + B t_{H^A} g_\Phi \left( t_{H^A}\right) \right] \right. \nonumber \\&\quad \left. + \left( \frac{m^2_{\mu }}{m^2_{E} c^2_{\beta }} + \frac{m_W^4c^2_{\beta }}{m_Y^4}\right) \right. \nonumber \\&\quad \times \left. \left[ t_{H^A} {\tilde{f}}_\Phi \left( t_{H^A}\right) + B t_{H^A} {\tilde{g}}_\Phi \left( t_{H^A}\right) \right] \right\} , \end{aligned}$$

(45)

$$\begin{aligned} \Delta a_{\mu }(Y)&\simeq - \frac{e g^2 m^2_{\mu }}{8\pi ^2 m_W^2} \times \frac{m_W^2 }{ m^2_{Y}} \left[ {\tilde{f}}_V \left( t_{Y}\right) + B {\tilde{g}}_V \left( t_{Y}\right) \right] , \end{aligned}$$

(46)

where $t_{Y}=m^2_E/m^2_Y$, $t_{H^A}=m^2_E/m^2_{H^A}$. The above formulas are independent from both MSS and ISS mechanisms affecting only the one-loop contributions from singly charged Higgs bosons. $\Delta a_{\mu }(H^A)$ has a chirally-enhanced term but contains a suppressed factor $m^2_W/m^2_Y$.

Firstly, we will show that the one-loop contribution from $W^\pm$ is always close to the SM prediction. Using the approximation that $t_{W,i}=0$ with $i\le 3$ and $t_{W_i}=x_W=m^2_{n_i}/m_W^2$ with $i>3$, we have

$$\begin{aligned} c^W_{(aa)R}&= \frac{e g^2m_{e_a} }{32 \pi ^2m^2_{W} } \left[ {\tilde{f}}_V \left( 0\right) + \left( R^*R^T\right) _{aa}\right. \nonumber \\&\quad \times \left. \left( {\tilde{f}}_V \left( x_W\right) -{\tilde{f}}_V \left( 0\right) \right) \right] , \end{aligned}$$

(47)

leading to the following contribution from W to $a_{e_a}$ with ${\tilde{f}}_V \left( 0\right) = -5/12$:

$$\begin{aligned} a_{e_a}(W)= -\frac{g^2 m^2_{e_a}}{8\pi ^2 m^2_W} \left[ -\frac{5}{12} + \left( R^*R^T\right) _{aa} \times \left( {\tilde{f}}_V \left( x_W\right) + \frac{5}{12}\right) \right] . \end{aligned}$$

(48)

Because $|{\tilde{f}}_V \left( x_W\right) + \frac{5}{12}|\le \frac{5}{12}$ , see the bellow discussion, in the limit $\left( R^*R^T\right) _{aa}\le 10^{-3}\ll 1$ given in (37), $a_{\mu }(W)$ equals to the one-loop contribution predicted by the SM [110]:

$$\begin{aligned}&a^{(1){\mathrm {SM}}}_{\mu }(W) \simeq \frac{g^2 m^2_{\mu }}{8\pi ^2 m^2_W}\times \frac{5}{12}\simeq 383\times 10^{-11},\nonumber \\&\; \frac{g^2 m^2_{\mu }}{8\pi ^2 m^2_W}\simeq 9.19\times 10^{-9}. \end{aligned}$$

(49)

Finally, one-loop contributions from the two singly charged Higgs bosons will be shown precisely in the two frameworks of MSS and ISS. The analytic formulas were collected in Appendix B. Before discussing the total contributions, we just show here the most important part $a_{0,\mu }(H^\pm )$ which can be large enough to reach the allowed ranges consistent with $\Delta a^{\mathrm {NP}}_\mu$:

$$\begin{aligned} a_{\mu }(H^\pm )&= a_{\mu }(H^\pm _1) + a_{\mu }(H^\pm _2) \equiv a_{\mu ,0}(H^\pm ) + \cdots ,\nonumber \\ a_{\mu ,0}(H^\pm )&=-\frac{g^2m_{\mu }\;}{8 \pi ^2 m^2_W} \sum _{k=1}^{2}\sum _{i=3}^{K+3} \left[ \frac{\lambda ^{L,k*}_{ia } \lambda ^{R,k}_{ia }m_{n_i} f_{\Phi }(x_{i,k})}{m^2_{H^\pm _k} }\right] \nonumber \\&\simeq -9.19\times 10^{-9} \left[ \frac{vt^{-1}_{\beta }c_{\alpha }s_{\alpha }}{\sqrt{2} m_{\mu }}\left( \frac{M_D^{\dagger }V^*V^{\dagger }Y^{h}}{M_0} \right) _{22} \right] \nonumber \\&\quad \times \left[ x_1f_{\Phi }(x_{1}) - x_2f_{\Phi }(x_{2}) \right] , \end{aligned}$$

(50)

where $x_k\equiv M_0^2/m^2_{H^\pm _k}$. Note that $a_{\mu }(H^\pm )\ne 0$ requires $s_{2\alpha } =2s_{\alpha }c_{\alpha }\ne 0$ and $x_1\ne x_2$.

In summary, general formulas for one-loop contributions to $a_{e_a}$ used in this work were given in Ref. [36]. They are consistent with those calculated previously for the $331\beta$ models [79, 80]. In the $331\beta$ model under consideration, all the relevant one-loop contributions will be derived in the forms depending on the two classes of the following master functions: $\left\{ f_{\Phi }(x), {\tilde{f}}_{\Phi }(x),\; xf_{\Phi }(x),\; x{\tilde{f}}_{\Phi }(x), xg_{\Phi }(x),\; x{\tilde{g}}_{\Phi }(x)\right\}$ and $\{ {\tilde{f}}_{V}(x),\; {\tilde{g}}_{V}(x)\}$ for charged Higgs and gauge boson exchanges, respectively. The additional factor x originates from the specific properties of the charged Higgs bosons in the $331\beta$ framework. The dependence of these functions on x is shown in Fig. 1, where all allowed ranges are shown precisely.

Fig. 1

The dependence of master formulas as functions of $x=m_E^2/m_X^2$ and $m^2_{n_i}/m_X^2$ with $X=W,Y,H^\pm _{1,2},H^A$

To estimate the one-loop contributions to AMM, it is useful to see the limits for the above master functions as follows:

$$\begin{aligned}&\lim _{x\rightarrow 0} f_{\Phi }(x)= f_{\Phi }(0)= \frac{1}{4}, \quad \lim _{x\rightarrow \infty } f_{\Phi }(x)= f_{\Phi }(\infty )=0, \nonumber \\&\lim _{x\rightarrow 0} {\tilde{f}}_{\Phi }(x)= {\tilde{f}}_{\Phi }(0)=\frac{1}{24}, \quad \lim _{x\rightarrow \infty } {\tilde{f}}_{\Phi }(x)= {\tilde{f}}_{\Phi }(\infty )=0, \nonumber \\&\lim _{x\rightarrow 0} {\tilde{f}}_{V}(x)={\tilde{f}}_{V}(0)= -\frac{5}{12}, \quad \lim _{x\rightarrow \infty } {\tilde{f}}_{V}(x)= {\tilde{f}}_{V}(\infty )= -\frac{1}{6}, \nonumber \\&\lim _{x\rightarrow 0} {\tilde{g}}_{V}(x)= {\tilde{g}}_{V}(0)= -\frac{3}{4}, \quad \lim _{x\rightarrow \infty } {\tilde{g}}_{V}(x)= {\tilde{g}}_{V}(\infty )= -\frac{3}{8},\nonumber \\&\lim _{x\rightarrow 0}\left[ x\times f_{\Phi }(x)\right] =0, \quad \quad \lim _{x\rightarrow \infty }\left[ x\times f_{\Phi }(x)\right] =\frac{1}{4}, \nonumber \\&\lim _{x\rightarrow 0}\left[ x\times {\tilde{f}}_{\Phi }(x)\right] =0, \quad \quad \lim _{x\rightarrow \infty }\left[ x\times {\tilde{f}}_{\Phi }(x)\right] =\frac{1}{12},\nonumber \\&\lim _{x\rightarrow 0}\left[ x\times g_{\Phi }(x)\right] =0, \quad \quad \lim _{x\rightarrow \infty }\left[ x\times g_{\Phi }(x)\right] = \frac{1}{2}, \nonumber \\&\lim _{x\rightarrow 0}\left[ x\times {\tilde{g}}_{\Phi }(x)\right] =0, \quad \lim _{x\rightarrow \infty }\left[ x\times {\tilde{g}}_{\Phi }(x)\right] =\frac{1}{8}. \end{aligned}$$

(51)

Because $|\beta |\le \sqrt{3}$, we have $-1\le B\le 2$. It is easily to show that:

$$\begin{aligned}&\left| xf_{\Phi } \left( x\right) + B x g_\Phi \left( x\right) \right| \le {\mathcal {O}}(1), \nonumber \\&\left| x{\tilde{f}}_{\Phi } \left( x\right) + B x {\tilde{g}}_\Phi \left( x\right) \right| \le {\mathcal {O}}(1), \; \left| {\tilde{f}}_{V} \left( x\right) + B {\tilde{g}}_V \left( x\right) \right| \le \frac{7}{6}, \nonumber \\&0\le {\tilde{f}}_{V}\left( x\right) +\frac{5}{12} \le \frac{5}{12} , \; 0\le xf_{\Phi } \left( x\right) \le \frac{1}{4}. \end{aligned}$$

(52)

First, we consider the one-loop contribution from the SM gauge $W^\pm$ where the deviation from the SM prediction derived from Eq. (48) satisfies:

$$\begin{aligned}&\left| \Delta a_{\mu }(W) \right| \simeq 9.19\times 10^{-9} \left| \left( R^*R^T\right) _{aa} \times \left( {\tilde{f}}_V \left( x_W\right) + \frac{5}{12}\right) \right| \nonumber \\&\quad<2.5\times 10^{-11}< a^{\mathrm {NP}}_{\mu }, \end{aligned}$$

(53)

where the constraint $|\left( R^*R^T\right) _{aa}|\le 2\times 10^{-3}$ consistent with non-unitary condition (37). In Eq. (53), $\Delta a_{\mu }$ is considered as the $1\sigma$ range of the discrepancy between the SM’s prediction and experiments shown in Eq. (1), namely $\left| \Delta a_{\mu }(W) \right| \ll \Delta a_{\mu }\in \left[ 1.92\times 10^{-9},\; 3.1\times 10^{-9}\right]$. Therefore, we will use the following approximation for both frameworks MSS and ISS:

$$\begin{aligned} \Delta a_{\mu }(W) \simeq 0. \end{aligned}$$

(54)

For the recent bound of the $SU(3)_L$ scale, we can use the lower bound $m_Y\ge 1$ TeV, consistent with the recent constraint concerned for 3-3-1 models [81, 111,112,113]. Now the one-loop contributions from $H^A$ and $Y^A$ can be estimated as follows:

$$\begin{aligned}&0<- a_{\mu }(H^A)\le 9.19\times 10^{-9} \nonumber \\&\qquad \times \left[ \frac{m_W^2}{m^2_Y}+ \frac{m^2_{\mu }}{m^2_{E} c^2_{\beta }} + \frac{m_W^4c^2_{\beta }}{m_Y^4} \right]< 6.3 \times 10^{-11}\ll \Delta a^{\mathrm {NP}}_{\mu },\nonumber \\&0<\Delta a_{\mu }(Y)\le 9.19\times 10^{-9} \times \frac{m_W^2}{m^2_Y}\times \frac{7}{6}< 7 \times 10^{-11}\ll \Delta a^{\mathrm {NP}}_{\mu }. \end{aligned}$$

(55)

where a crude lower bound $m_{E}c_{\beta }\ge 5$ GeV was used. We conclude that the two one-loop contributions originated from heavy Higgs $H^A$ and charged gauge boson Y is much smaller than $\Delta a^{\mathrm {NP}}_{\mu }\sim {\mathcal {O}}(10^{-9})$, which is considered as the $1\sigma$ range given in Eq. (1) from now on. This agrees with all previous works, for example for the heavy charged gauge bosons [103]. We will ignore them from now on.

We now discuss on the dominant contributions of the two singly charged Higgs bosons given in Eq. (50), where small $t_{\beta }$ supports large values of these contributions. The reasonable values for a numerical estimation are $t^{-1}_{\beta }s_{\alpha }c_{\alpha }\simeq 0.5$, and $v/(\sqrt{2}m_{\mu })=1.6\times 10^{3}$, max$[| x_1f_{\Phi } \left( x_1\right) - x_2f_{\Phi } \left( x_2\right) |]\simeq 0.25$, we have

$$\begin{aligned} \left| \sum _{k=1}^2\Delta a_{\mu }(H^\pm _1)\right|&\le 1.9\times 10^{-9}\left[ 10^3 \left( M_0^{-1} M_D^{\dagger }Y^{h}\right) _{22} \right] \nonumber \\&\sim \Delta a^{\mathrm {NP}}_{\mu } \left[ 10^3 \left( M_0^{-1} M_D^{\dagger }Y^{h}\right) _{22} \right] . \end{aligned}$$

(56)

In the next discussion for two specific frameworks of MSS and ISS, the allowed values of $\left| \left( M_0^{-1} M_D^{\dagger }Y^{h}\right) _{22}\right|$ will depend strictly on the characteristics of the two models. We will show that the condition (56) will satisfy for only the ISS mechanism, which allows this value to reach the $(g-2)_{\mu }$ data.

For convenience, we will use the following estimation,

$$\begin{aligned} \frac{v}{\sqrt{2}m_{\mu }}&\simeq 1.6\times 10^3; \; | x_1f_{\Phi } \left( x_1\right) - x_2f_{\Phi } \left( x_2\right) | \le 0.25; \nonumber \\&\qquad 0.3\le t_{\beta }\le 10;\nonumber \\ \left| s_{\alpha }c_{\alpha }\right|&= \left| \frac{\sin (2\alpha )}{2}\right| \le 0.5; \; m_{H^\pm _1},\; m_{H^\pm _2} \ge 800~\text {GeV}; \nonumber \\&\quad M_0\ge 100~\text {GeV}. \end{aligned}$$

(57)

After that, other possible values of $m_{H^\pm _K}$, $M_0$, and $t_{\beta }$ will also be discussed.

Now we will derive the specific analytic formulas of one-loop contributions to $\Delta a_{\mu }$ corresponding to the two mechanisms MSS and ISS. We note that all above discussions for $\Delta a_{\mu }$ are applied in the same way to derive to $\Delta a_{e}$, therefore, we just mention to $\Delta a_{\mu }$ in the numerical discussion.

4.1 The MSS mechanism

The MSS relations given in Eqs. (28) and (29) result in that

$$\begin{aligned} M_D^{\dagger }R^{\dagger }= {\hat{m}}_{\nu }, \; R = -iU_{{\mathrm {PMNS}}} \left( \frac{{\hat{m}}_{\nu }}{M_0}\right) ^{1/2}, \quad m_{n_{4,5,6}}\simeq M_0. \end{aligned}$$

(58)

The detailed derivation of the one-loop contributions from singly charged Higgs bosons is given in Appendix B. Using ${\tilde{f}}_{\Phi }(0)= \frac{1}{24}$, the one-loop contribution from $H^\pm _1$ is

$$\begin{aligned}&\Delta a^{\mathrm {MSS}}_{\mu }(H^\pm _1) = - 9.19\times 10^{-9}\nonumber \\&\quad \times {\mathrm {Re}}\left\{ \left[ c^2_{\alpha } \left( \frac{m^2_{n_2} }{M_0^2}\right) ^{1/2} +\frac{vt_{\beta }^{-1}c_{\alpha }s_{\alpha }}{\sqrt{2}m_{\mu }}\sum _{c=1}^{3} \left( \frac{m^2_{n_c} }{M_0^2}\right) ^{1/4}\right. \right. \nonumber \\&\quad \left. \left. \times U_{{\mathrm {PMNS}},2c}\left( -iY^{h}\right) _{c2}\right] \right. \nonumber \\&\quad \times \left. x_1 f_{\Phi }(x_1) \right. +t_{\beta }^{-2} c^2_{\alpha } \sum _{c=1}^3 \left| U_{{\mathrm {PMNS}},2c}\right| ^2 \nonumber \\&\quad \times \left[ \frac{m_{n_c}^2}{m^2_{H^\pm _1}} \left( \frac{1}{24} -{\tilde{f}}_{\Phi }(x_1) \right) + \frac{m_{n_c} }{M_0} x_1{\tilde{f}}_{\Phi }(x_1) \right] \nonumber \\&\quad + \frac{m^2_{\mu } t_{\beta }^2c_{\alpha }^2}{m^2_{H^\pm _1}} \left[ \frac{1}{24} -\sum _{c=1}^{3} \left| U_{{\mathrm {PMNS}},2c}\right| ^2 \frac{m_{n_c} }{M_0} \left( \frac{1}{24} -{\tilde{f}}_{\Phi }(x_1) \right) \right] \nonumber \\&\quad + \frac{v^2s_{\alpha }^2}{2 m^2_{H^\pm _1}} \left[ \sum _{c=1}^{3} \left| Y^{h}_{c2}\right| ^2 \frac{m_{n_c} }{M_0} \left( \frac{1}{24} -{\tilde{f}}_{\Phi }(x_1) \right) \right. \nonumber \\&\quad \left. + \left( Y^{h\dagger }Y^{h}\right) _{22} {\tilde{f}}_{\Phi }(x_1)\right] \nonumber \\&\quad \left. - \frac{vm_{\mu }t_{\beta } s_{2\alpha }}{\sqrt{2} m^2_{H^\pm _1}} \left[ \left( -iU_{{\mathrm {PMNS}}} \left( \frac{{\hat{m}}^2_{\nu } }{M^2_0}\right) ^{1/4}Y^{h}\right) _{22} \right. \right. \nonumber \\&\quad \left. \left. \times \left( \frac{1}{24}- {\tilde{f}}_{\Phi }(x_1) \right) \right] \right\} . \end{aligned}$$

(59)

It can be seen that only two terms proportional to $\left( \frac{ m^2_{n_c}}{M^2_0}\right) ^{1/4} {\mathrm {Re}}[(-iU_{{\mathrm {PMNS}},2c} Y^{h}_{c2})]$ can give contributions having consistent sign with $\Delta a^{\mathrm {NP}}_{\mu }$, and ${\mathrm {Re}}[(-iU_{{\mathrm {PMNS}},2c}Y^{h}_{c2})]$ must be negative (see the first and last lines in the real part of Eq. (59)). We just focus on these two contributions. The remaining terms always give negative contributions to $\Delta a^{\mathrm {NP}}_{\mu }$. The two mentioned terms can be estimated as follows:

$$\begin{aligned} 0&< \left( \frac{m^2_{n_c} }{M_0^2}\right) ^{1/4} \times x_1 f_{\Phi }(x_1) \le \left( \frac{(0.12\; {\mathrm {eV}})^2 }{m_{H^{\pm }_1}^2}\right) ^{1/4} \nonumber \\&\quad \times x_1^{3/4} f_{\Phi }(x_1)<1.1\times 10^{-7}, \nonumber \\ 0&< \left( \frac{m^2_{n_c}}{M_0^2}\right) ^{1/4} \times \left( \frac{1}{24} -{\tilde{f}}_{\Phi }(x_1)\right) \nonumber \\&\le \frac{1}{24} \left( \frac{(0.12\; {\mathrm {eV}})^2 }{m_{H^{\pm }_1}^2}\right) ^{1/4} <10^{-7}, \end{aligned}$$

(60)

where we have used $m_{H^\pm _1}\ge 100$ GeV and max$[x_1^{3/4} f_{\Phi }(x_1)]<0.1$. But in this situation the factor $\left| \frac{v}{\sqrt{2} m_{\mu }} Y^{h}_{c2}t^{-1}_{\beta } \right| \le 10^4$ is still not large enough so that the total can give any significant contributions to $\Delta a^{\mathrm {MSS}}_{\mu }$. In conclusion, the MSS mechanism still fails to explain the experimental AMM data of $\mu$.

4.2 The ISS mechanism

The ISS mechanism will be considered instead of the MSS one. The change is for only singly charged Higgs bosons $H^\pm _k$. Following Eqs. (32) and (33), the results for $H^\pm _{1,2}$ are

$$\begin{aligned} a^{\mathrm {ISS}}_{\mu }(H^\pm _1)&= - 9.19\times 10^{-9} \nonumber \\&\quad \times {\mathrm {Re}}\left\{ \sum _{c=1}^{3} \left[ c^2_{\alpha } \left| U_{{\mathrm {PMNS}},2c} \right| ^2 \frac{m_{n_c}}{\mu _X} \right. \right. \nonumber \\&\quad \left. \left. + \frac{vt_{\beta }^{-1}c_{\alpha }s_{\alpha }}{\sqrt{2}m_{\mu }}U_{{\mathrm {PMNS}},2c} \left( \frac{m_{n_c}}{\mu _X} \right) ^{1/2} \left( Y^{h}_{2}\right) _{c2} \right] x_1f_{\Phi }(x_1) \right. \nonumber \\&\quad + \sum _{c=1}^{3} \left[ \left| U_{{\mathrm {PMNS}},2c} \right| ^2 \left( t_{\beta }^{-2} c^2_{\alpha } \frac{m_{n_c}}{\mu _X} \right) \right] x_1 \tilde{f}_{\Phi }(x_1) \nonumber \\&\quad + \frac{m^2_{\mu } t_{\beta }^2c_{\alpha }^2}{m^2_{H^\pm _1}} \left[ \frac{1}{24} - \sum _{c=1}^{3} \left[ \left| U_{{\mathrm {PMNS}},2c} \right| ^2\frac{m_{n_c}}{\mu _X} \right] \right. \nonumber \\&\qquad \times \left. \left( \frac{1}{24} -\tilde{f}_{\Phi }(x_1) \right) \right] \nonumber \\&\quad + \frac{v^2s_{\alpha }^2}{2 m^2_{H^\pm _1}} \left[ \sum _{c=1}^{3} \left[ \left| \left( Y^{h}_{2}\right) _{c2}\right| ^2 \frac{m_{n_c}}{\mu _X} \right] \left( \frac{1}{24} -\tilde{f}_{\Phi }(x_1) \right) \right. \nonumber \\&\quad \left. + \left( Y^{h\dagger }_2Y^{h}_2\right) _{22} \tilde{f}_{\Phi }(x_1)\right] \nonumber \\&\quad -\left. \frac{vm_{\mu }t_{\beta } s_{2\alpha }}{ \sqrt{2} m^2_{H^\pm _1}} \left( \frac{1}{24}- \tilde{f}_{\Phi }(x_1) \right) \right. \nonumber \\&\quad \left. \sum _{c=1}^{3} \left[ U_{{\mathrm {PMNS}},2c}\left( Y^{h}_{2}\right) _{c2} \left( \frac{m_{n_c}}{\mu _X} \right) ^{1/2}\right] \right\} , \nonumber \\ a^{\mathrm {ISS}}_{\mu }(H^\pm _2)&= a_{\mu }(H^\pm _1) \left[ x_1 \rightarrow \; x_2,\;s_{\alpha }\rightarrow - c_{\alpha }, \; c_{\alpha } \rightarrow s_{\alpha }\right] , \end{aligned}$$

(61)

where we have used the form $Y^h=(O_{3\times 3},\; Y^h_2)^T$ for the ISS framework. Here, the parameter $\mu _X$ appears in the ISS mechanism instead of $M_0$ corresponding to the MSS. The second term in the first line of Eq. (61) is from $H^\pm _1$ emphasized previously in Eq. (50).

Using the constraint (37) for $RR^{\dagger }=U_{{\mathrm {PMNS}}} {\hat{x}}_{\nu }U_{{\mathrm {PMNS}}}^{\dagger }$ we have ${\hat{x}}_{\nu }<{\mathcal {O}}(10^{-3})$. Therefore, we will choose a safe upper bound for the NO scheme as follows

$$\begin{aligned}&{\mathrm {Max}}[\left( {\hat{x}}_{\nu }\right) _{aa}]= \frac{m_{n_3}}{\mu _X}\simeq \left( \frac{\Delta m^2_{32}}{\mu _X^2}\right) ^{1/2} \le 2\times 10^{-3} \nonumber \\&\quad \Rightarrow \mu _X\ge 2.5\times 10^{-8}~\text {GeV}. \end{aligned}$$

(62)

The default value of $\mu _X$ is fixed by $\mu _X= 2.5\times 10^{-8}~\text {GeV}$.

With the allowed range given in Eq. (57), it can be proved that:

$$\begin{aligned} 0&<\sum _{c=1}^3c^2_{\alpha } \left| U_{{\mathrm {PMNS}},2c} \right| ^2 \frac{m_{n_c}}{\mu _X}x_1 f_{\Phi }(x_1)< \left| U_{{\mathrm {PMNS}},23} \right| ^2 \times 5\times 10^{-3}\nonumber \\&\quad \times \frac{1}{12}\simeq 8.9\times 10^{-5}, \nonumber \\ 0&< a^{\mathrm {ISS}}_{\mu ,1}(H_1^\pm )\sim \sum _{c=1}^{3} \left[ \left| U_{{\mathrm {PMNS}},2c} \right| ^2 \left( t_{\beta }^{-2} c^2_{\alpha } \frac{m_{n_c}}{\mu _X} \right) \right] x_1 {\tilde{f}}_{\Phi }(x_1) \nonumber \\&<1.3\times 10^{-3}, \nonumber \\ 0&< \frac{m^2_{\mu } t_{\beta }^2c_{\alpha }^2}{m^2_{H^\pm _1}} \left[ \frac{1}{24} - \sum _{c=1}^{3} \left[ \left| U_{{\mathrm {PMNS}},2c} \right| ^2\frac{m_{n_c}}{\mu _X} \right] \left( \frac{1}{24} -{\tilde{f}}_{\Phi }(x_1) \right) \right]<10^{-7} \nonumber \\ 0&<\frac{v^2s_{\alpha }^2}{2 m^2_{H^\pm _1}}\times \sum _{c=1}^{3} \left[ \left| Y^{h}_{(c+3)2} \right| ^2 \frac{m_{n_c}}{\mu _X} \right] \left( \frac{1}{24} -{\tilde{f}}_{\Phi }(x_1) \right)< 0.32\times 10^{-4}, \nonumber \\ 0&<\frac{vm_{\mu }t_{\beta } s_{2\alpha }}{\sqrt{2} m^2_{H^\pm _1}} \left( \frac{1}{24}- {\tilde{f}}_{\Phi }(x_1) \right) \nonumber \\&\quad \times \sum _{c=1}^{3} \left[ {\mathrm {Re}}[U_{{\mathrm {PMNS}},2c}Y^{h}_{(c+3)2}] \left( \frac{m_{n_c}}{\mu _X} \right) ^{1/2}\right]< 10^{-6}, \nonumber \\ 0&< a^{\mathrm {ISS}}_{\mu ,2}(H_1^\pm )\sim \frac{v^2s_{\alpha }^2}{2 m^2_{H^\pm _1}} \left( Y^{h\dagger }Y^{h}\right) _{22} {\tilde{f}}_{\Phi }(x_1)\nonumber \\&= \frac{v^2s_{\alpha }^2}{2 m^2_{H^\pm _1}} \left( Y^{h\dagger }_2Y^{h}_2\right) _{22} {\tilde{f}}_{\Phi }(x_1)<1.6\times 10^{-2}. \end{aligned}$$

(63)

There are only two contributions $a_{\mu ,1(2)}(H_1^\pm )$ in the second and last lines that may affect significantly $a_{\mu }(H^\pm _1)$ because of the rather large upper bounds $a_{\mu ,1}(H_1^\pm )\le 1.3c^2_{\alpha }\times 10^{-11}$ and $a_{\mu ,2}(H_1^\pm )\le 23s^2_{\alpha }\times 10^{-11}$. But both of them give negative contributions to $\Delta a^{\mathrm {ISS}}_{\mu }$, hence should be small. Ignoring all other contributions smaller $10^{-4}\times \Delta a^{\mathrm {NP}}_{\mu }$, the remaining large contribution in Eq. (61) is the one mentioned in Eq. (50). It has the following form in the ISS framework:

$$\begin{aligned}&a^{\mathrm {ISS}}_{\mu ,0}(H^\pm ) = - 9.19\times 10^{-9}\nonumber \\&\qquad \times {\mathrm {Re}}\left\{ \frac{vt_{\beta }^{-1}c_{\alpha }s_{\alpha }}{\sqrt{2}m_{\mu }} \left( \frac{m_{n_3}}{\mu _X} \right) ^{1/2} Y^{d}_{2} \left[ x_1f_{\Phi }(x_1) -x_2f_{\Phi }(x_2)\right] \right\} \nonumber \\&\quad = -680.58 \times 10^{-9} \nonumber \\&\qquad \times t_{\beta }^{-1}c_{\alpha }s_{\alpha }\left( \frac{m_{n_3}}{\mu _X} \right) ^{1/2} Y^{d}_{2} \left[ x_1f_{\Phi }(x_1) -x_2f_{\Phi }(x_2)\right] , \end{aligned}$$

(64)

where the part relating to $x_k$ is the contribution from $H^{\pm }_k$ exchange, and the new parameter $Y^d_2$ is assumed to relate to Yukawa coupling matrix $Y^h_2$ through the following relation

$$\begin{aligned} U_{{\mathrm {PMNS}}} \left( \frac{{\hat{m}}_{\nu }}{m_{n_3}}\right) ^{1/2} Y^{h}_{2}\equiv {\mathrm {diag}} \left( Y^{d}_{1},\;Y^{d}_{2}, Y^{d}_{3}\right) =Y^d, \end{aligned}$$

(65)

so that the two largest one-loop contributions from $H^{\pm }_{1,2}$ to AMM will allow zero contributions to the cLFV branching ratios Br$(e_b\rightarrow e_a\gamma )\sim \left( |c_{(ab)R}|^2+|c_{(ba)R}|^2\right)$ [36], because $c_{(ab)R}(H^\pm ), c_{(ba)R}(H^\pm ) \sim \left( U_{{\mathrm {PMNS}}} {\hat{m}}_{\nu }^{1/2}Y^{h}_{2}\right) _{ab,ba} (H^\pm ) =0$ for $a\ne b$. Now, the matrix $Y^h_{2}$ is derived through the following relation:

$$\begin{aligned} Y^h_2=\left( \frac{{\hat{m}}_{\nu }}{m_{n_3}}\right) ^{-1/2}U^{\dagger }_{{\mathrm {PMNS}}}Y^d, \end{aligned}$$

(66)

which will be used to check the perturbative limit of all $\left| (Y^{h}_{2})_{ab}\right| <\sqrt{4\pi }\simeq 3.5$ while scanning values of $Y^d_{1,2,3}$. In this case, the factors $\left( \frac{({\hat{m}}_{\nu })_{aa}}{m_{n_3}}\right) ^{-1/2} \ge 1$ in the NO scheme and all active neutrino masses lie in the denominators. Therefore, these masses must be non-zero and large enough to guarantee that all entries of $Y^{h}_2$ satisfy the perturbative limits.

From now on, we will fix $m_{n_1}=0.01$ eV in our numerical discussion. Smaller $m_{n_1}$ will give smaller allowed $Y^d_1$ satisfying the perturbative limit of $Y^h_2$. Using the best-fit points corresponding to the NO scheme of neutrino oscillation data given in Eq. (34), $Y^h_2$ has the following form

$$\begin{aligned} Y^{h}_2=\left( \begin{array}{ccc} 1.84 Y^d_1 &{} (-0.7-0.125 i) Y^d_2 &{} (1.09-0.113 i) Y^d_3 \\ 1.1 Y^d_1 &{} (1.184-0.074 i) Y^d_2 &{} (-1.11-0.068 i) Y^d_3 \\ (-0.116-0.09 i) Y^d_1 &{} 0.732 Y^d_2 &{} 0.666 Y^d_3 \\ \end{array} \right) . \end{aligned}$$

(67)

For small $m_{n_1}$ including the case $m_{n_1}=0$, matrix $Y^h_2$ is chosen as follows:

$$\begin{aligned} Y^h_2 = \left( \begin{array}{ccc} 0 &{} 0 &{} 0 \\ (2.9+0.3 i) Y^d_{1} &{} (0.50-0.32 i)Y^d_{2} &{} (0.61-0.68 i)Y^d_{3} \\ (-1.23-0.21 i)Y^d_{1} &{} (1.15+0.12 i)Y^d_{2} &{} Y^d_{3} \\ \end{array} \right) , \end{aligned}$$

which $Y^d$ defined from Eq. (65) is not diagonal, but it always keeps $(Y^d)_{12}=(Y^d)_{21}=(Y^d)_{13}=0$. In addition, $Y^d_3=0$ gives $(Y^d)_{32}=0$. This will avoid large contributions to the strict constraint of cLFV decay Br$(\mu \rightarrow e\gamma )$. The numerical investigations show that the two choices of $Y^h_2$ mentioned above have the same qualitative results. Numerical illustration will be done with the $Y^h_2$ given in Eq. (67). Therefore, we will fix $Y^d_3=0$ in our numerical investigation on $a^{\mathrm {ISS}}_{e,\mu }(H^\pm )$. We note that $Y^d_3$ also contributes to the AMM of the $\tau$ lepton, which still has weak constraints from recent experiments [114,115,116] and a combination derived from these experimental results [117, 118], see discussions on this topic in Refs. [119, 120], suggesting new experiments to improve measurements.

Similarly, the data of $\Delta a^{\mathrm {NP}}_{e}$ may be explained by the following contribution:

$$\begin{aligned}&a^{\mathrm {ISS}}_{e,0}(H^\pm ) = -9.19 \times 10^{-9} \nonumber \\&\qquad \times \frac{m_e^2}{m^2_{\mu }} {\mathrm {Re}}\left\{ \frac{vt_{\beta }^{-1}c_{\alpha }s_{\alpha }}{\sqrt{2}m_{e}} \left( \frac{m_{n_3}}{\mu _X} \right) ^{1/2} Y^{d}_{1} \left[ x_1f_{\Phi }(x_1) -x_2f_{\Phi }(x_2)\right] \right\} \nonumber \\&\quad = -32409\times 10^{-13} \nonumber \\&\qquad \times t_{\beta }^{-1}c_{\alpha }s_{\alpha } \left( \frac{m_{n_3}}{\mu _X} \right) ^{1/2} Y^{d}_{1} \left[ x_1f_{\Phi }(x_1) -x_2f_{\Phi }(x_2)\right] . \end{aligned}$$

(68)

In the simple forms of the matrices $Y^{h}$ given in Eq. (66) and $M_D$ we assumed here, the main difference between $a^{\mathrm {ISS}}_{e,0}(H^\pm )$ and $a^{\mathrm {ISS}}_{\mu ,0}(H^\pm )$ is that they contain different free factors $Y^{d}_{1}$ and $Y^{d}_{2}$, respectively. The numerical results show that this difference is enough to explain both AMM data of e and $\mu$ at $1\sigma$ discrepancy given in Eqs. (1) and (2). The regions of the parameter space satisfying simultaneously these will be defined as the allowed regions from now on.

The first numerical illustrations are shown in Fig. 2, where free parameters are fixed in the ranges given in (57) and predict valid regions satisfying both the experimental AMM data of muon (two upper panels) and electron (two lower panels). In addition, in the upper left panel of Fig. 2, the numerical values $s_{\alpha }=0.5$, $Y^{d}_{2}=0.21$, and $t_{\beta }=0.5$ safely satisfy perturbative limits of max$|\left( Y^h_2\right) _{ab}|<0.2$. On the other hand, numerical values of free parameters in the upper right panel are somewhat special: $s_{\alpha }=1/\sqrt{2}$ is maximal for $s_{2\alpha }=1$, large $Y^{d}_{2}=2.8$ close the perturbative limit max$|\left( Y^h_2\right) _{ab}|=3.32$, and $t_{\beta }=20 \gg 1$ does not support large $a^{\mathrm {ISS}}_{\mu }(H^\pm )$, which excludes the regions satisfying $0<x_2<0.1$ and all $x_1>0$, for example. All values of $t_{\beta }>30$ are excluded in this case. We conclude that the AMM data will result in a upper bound of $t_{\beta }$. The values of $Y^{d}_{1}$ are chosen so that there exist allowed values of $(x_1,x_2)$ satisfying simultaneously both $1\sigma$ experimental AMM data of muon and electron. Namely, the allowed values of $(x_1,\;x_2)$ in the two left panels are in the ranges $0<x_1\le 20$ and $0.1< x_2$. Similarly, the allowed regions in the two right panels satisfy $0<x_1\le 10$ and $x_2>0.1$. Large $t_{\beta }$ gives strong upper constraint on $M_0<750$ GeV derived from the perturbative limit of max$|Y^X_{ab}|$=max$\frac{\sqrt{2}\left( m_D\right) _{ab}}{v_2}=$ max$\frac{M_0\sqrt{2}\left( {\hat{x}}_{\nu }^{1/2}U^\dagger _{{\mathrm {PMNS}}}\right) _{ab}}{vc_{\beta }}<3.5$. Consequently, $x_k=M_0^2/m^2_{H_k}$ should not be too large so that $m_{H_k}$ are larger than the lower bounds from experiments.

Fig. 2

The dependence of $\Delta a^{\mathrm {ISS}}_{\mu ,0}(H^\pm )$ and $\left[ -\Delta a^{\mathrm {ISS}}_{e,0}(H^\pm ) \right]$ as functions of $x_1$ with different fixed $x_2$. The red lines show the $1\sigma$ allowed ranges of $\Delta a^{\mathrm {NP}} _{\mu }$ and $\Delta a^{\mathrm {NP}} _{e}$ given in Eqs. (1) and (2), respectively

In general, the allowed regions of parameter space depend strongly on the ${\hat{x}}_{\nu }$, namely larger ${\hat{x}}_{\nu }$ will allow larger $t_{\beta }$, and smaller values of other parameters including $s_{2\alpha }\equiv 2 s_{\alpha }c_{\alpha }$, $Y^{d}_{1}$, and $Y^{d}_{2}$. Defining that ${\hat{x}}_{\nu }= \left( {\hat{m}}_{\nu }/m_{n_3}\right) \times {\hat{x}}_{\nu 3}$ with ${\hat{x}}_{\nu 3}=\frac{m_{n_3}}{\mu _X}$ and fixed $m_{n_1}=0.01$ eV, $a^{\mathrm {ISS}}_{e,\mu }(H^\pm )$ given in Eq. (61) depends strongly on ${\hat{x}}_{\nu 3}$. With large ${\hat{x}}_{\nu 3} \in \left[ 10^{-3},\; 5\times 10^{-3} \right]$ the allowed ranges of free parameters are given in Table 1. We note that $s_{\alpha }$ never vanishes, namely large ${\hat{x}}_{\nu 3} \le 5\times 10^{-3}$ can allow rather small $|s_{\alpha }|\ge 10^{-3}$, provided that $t_{\beta }\rightarrow 0.3$. This property distinguishes completely to the conclusion given in Ref. [76], where the allowed regions with fixed $s_{\alpha }=0$ require a necessary condition of large $t_{\beta }>30$.

Table 1 Allowed ranges of free parameters with large $\;10^{-3}\le {\hat{x}}_{\nu 3} = \frac{m_{n_3}}{\mu _X} \le 5\times 10^{-3}$, the notations − ($+$) denote the negative (positive) ranges of the allowed regions

In this last discussion we will focus on the allowed regions consisting of light masses of heavy neutrinos and singly charged Higgs bosons. Namely in the ISS realization, heavy neutrinos can be detected by future searches at colliders such as Large Hadron Collider (LHC) and the International Linear Collider (ILC), and Large Hadron electron Collider (LHeC) [121], where the heavy neutrinos mass range from ${\mathcal {O}}(10^2)$ GeV to few TeV were discussed [122,123,124,125,126]. Namely, because of the not too small mixing $\sim \sqrt{x_{\nu ,3}}$ between ISS and active neutrinos $\nu _{aL}$, the main production channel of heavy neutrinos $n_I$ (I = $4,\ldots ,9$) with mass $M_0$ at LHC is $u{\bar{d}}\rightarrow n_Ie^+_a$ through the s channel exchanging W boson. Then the decay channel of $n_I$ may be $n_I\rightarrow e_a^-W^+,n_{a} Z,\; n_a h$, where h is the standard model-like Higgs boson. The ILC can produce heavy neutrino in the processes $e^+e^-\rightarrow {\bar{n}}_a n_I$ through t and s-channels exchanging the W and Z bosons, respectively. The model under consideration also predicts a channel producing two heavy neutrinos $e^+e^-\rightarrow {\bar{n}}_I n_I$ through exchanging $H^\pm _k$. In the following numerical discussion, the allowed regions are defined as they result in the two values of $\Delta a^{\mathrm {ISS}}_{\mu }$ and $\Delta a^{\mathrm {ISS}}_{e}$ satisfying both AMM experimental data of $\mu$ and electron at 1 $\sigma$ discrepancy level, and all Yukawa couplings satisfy perturbative limits, $|(Y^{h}_2)_{ab}|,|Y^X_{ab}|\le \sqrt{4\pi }$ with $a,b\le 3$. The region of parameter space used to scan is chosen as follows:

$$\begin{aligned}&\; m_{H^\pm _1},\; m_{H^\pm _2} \ge 800 ~\text {GeV}; 10~\text {GeV}\le M_0\le \; 5\times 10^3~\text {GeV}; \nonumber \\&0.01\le x_1,x_2\le 100,\nonumber \\&0.3 \le t_{\beta }\le 50;\; \left| s_{\alpha }\right| \le 1.; \; |Y^{d}_1|,\;|Y^{d}_2|\le 4.5;\; 10^{-7} \nonumber \\&\le \; {\hat{x}}_{\nu 3}=\frac{m_{n_3}}{\mu _X} \le 10^{-3}. \end{aligned}$$

(69)

The scanning range of ${\hat{x}}_{\nu 3}$ satisfies the non-unitary constraint given in Eq. (37). The numerical results confirm that $\left| a^{\mathrm {ISS}}_{\mu ,1}(H^\pm )/ a^{\mathrm {ISS}}_{\mu }(H^\pm ) \right| <4\%$, and $\left| a^{\mathrm {ISS}}_{\mu ,2}(H^\pm )/ a^{\mathrm {ISS}}_{\mu }(H^\pm ) \right| <10^{-5}$. Therefore, these suppressed values are not shown in detail. The allowed regions are more strict than the scanned region given in (69), see Table 2. In addition, values of $|s_{\alpha }|$, $|Y^{d}_{1}|$, and $|Y^{d}_{2}|$ are bounded from below:

$$\begin{aligned} s_{\alpha }&\in [-0.99,\; -0.029] \cup [0.026,\; 0.996] \rightarrow s_{2\alpha } \nonumber \\&\in [-1,\; -0.058] \cup [0.051,\; 1] ,\nonumber \\ \; Y^{d}_{1}&\in [-0.361,\; -0.024] \cup [0.015,\;0.352], \; Y^{d}_{2} \nonumber \\&\in [-2.95,\;-0.176] \cup [0.523,\; 2.95], \end{aligned}$$

(70)

where we define $s_{2\alpha }=2s_{\alpha } c_{\alpha }$. Here although the lower bound of $t_{\beta }$ is the perturbative limit chosen in the scanned range, the upper bound is more strict than the largest value of the scanned range $t_{\beta }<21.42<50$. In general, the allowed regions require all lower bounds for free parameters ${\hat{x}}_{\nu 3}\ge 3.9\times 10^{-7}$, $M_0 \ge 318.$ GeV, $|Y^{d}_{2}| \ge 0.176$, and $|s_{2\alpha }|>0.051$. Values of $Y^{d}_{1}$ are bounded in a more strict range of $0.015<|Y^{d}_{1}|<0.352$. The lower bound of $M_0$ supports many promoting channels to search for heavy neutrinos at both LHC and ILC [122,123,124,125].

Table 2 Allowed ranges of free parameters corresponding to the scanning region (69)

Fig. 3

The correlations of free parameters vs. $\Delta a^{\mathrm {ISS}}_{\mu }(H^\pm )$ and $t_{\beta }$ in the allowed regions

The correlations between free parameters and $a^{\mathrm {ISS}}_{\mu }(H^\pm )$ in the allowed regions are illustrated in Fig. 3 with 2000 allowed points collected. The correlations of $\Delta a_{e}(H^\pm )$ vs. $\Delta a_{\mu }(H^\pm )$ can be seen from the correlations relating with $Y^{d}_{1}$ shown in the lower right panel of Fig. 3. We can see that very large $t_{\beta }$ allows only small $\Delta a_{\mu }(H^\pm )$. The dependence of $s_{\alpha }$, $Y^{d}_{1}$, and $Y^{d}_{2}$ on $\Delta a_{\mu }(H^\pm )$ is rather weak. The Fig. 3 shows the consistent approximation we discussed above that $a^{\mathrm {ISS}}_{e_a}\simeq a^{\mathrm {ISS}}_{e_a,0}(H^\pm )\sim t^{-1}_{\beta } {\hat{x}}_{\nu _3}^{1/2} Y^d_as_{2\alpha }$. Namely, the two left panels prefer allowed points with small $t_{\beta }$ and large ${\hat{x}}_{\nu _3}$. While large values of $\Delta a_{\mu }$ near the upper allowed bound exclude large $t_{\beta }$ and too small ${\hat{x}}_{\nu _3}$. In the upper right panel, large $t_{\beta }$ requires large $|s_{2\alpha }|$ so that the ratio $s_{2\alpha }/t_{\beta }$ is large enough to keep $\Delta a_{\mu }$ in the allowed range. In the lower right panel, we cannot realize the linear dependence of $Y^d_2$ on $\Delta a_{\mu }$ because many values of $Y^d_2$ are excluded by the perturbative limit of $Y^h_2$. On the other hand, this property can be seen for $Y^d_1$ because of the relations $|Y^d_1| \sim |\Delta a_{\mu ,0}/\Delta a_{e,0}|$. In particularly, based on the two dominant contribution of AMM given in Eqs. (64) and (68), $\Delta a^{\mathrm {ISS}}_{e_a,0}\simeq \Delta a^{\mathrm {ISS}}_{e_a}$, it is easily to show that $|\Delta a^\mathrm {ISS}_{\mu }/\Delta a^{\mathrm {ISS}}_{e}|\simeq$ $|\Delta a^\mathrm {ISS}_{\mu ,0}/\Delta a^{\mathrm {ISS}}_{e,0}| =|m_{\mu }Y^d_2/(m_{e}Y^d_1)|$. Identifying these with the experimental data will lead to a consequence that $|Y^d_1|= |m_{\mu }Y^d_2\Delta a^{\mathrm {NP}}_e/(m_{e}\Delta a^{\mathrm {NP}}_\mu )| \simeq {\mathcal {O}}(10^{-2})|Y^d_2|$. Therefore, $|Y_1|$ can get small values of ${\mathcal {O}}(10^{-2})$. Illustrations are shown in Fig. 4, where the left panel shows that $|Y^{d_1}|\sim \Delta a^{\mathrm {ISS}}_{e_a,0}$ depends nearly linearly on $\Delta a^{\mathrm {ISS}}_{e}$. The right panel shows the valid of the relation $\Delta a^{\mathrm {ISS}}_{e}\simeq a^{\mathrm {ISS}}_{e,0} \sim Y^d_1/Y^d_2$ we mentioned above. The band widths appear in the plots originate from the $1\sigma$ ranges of AMM experimental data.

Fig. 4

The correlations between $Y^d_1$ and $Y^d_1/Y^d_2$ vs $\Delta a^{\mathrm {ISS}}_{e_a,0}$

It is also emphasized that $Y^d_{1,2}$ may give loop corrections to lepton masses $m_{e,\mu }$ [127, 128], where large $|Y^d_{1,2}|$ may lead to the fine-tuning problem that loop corrections $\delta m_{\mu ,e}\gg m_{\mu ,e}$. Our model considered here has the same property with the models in class I with new neutral lepton having $Y_\psi =0$. Discussions in Ref. [128] suggest that the allowed regions we discussed above may consist of points with small $|Y^d_{1,2}|$ enough to avoid this fine-tuning. Determining exactly these regions of the parameter space should be done in the future.

Fig. 5

The correlations between different masses vs ${\hat{x}}_{\nu 3}$ (left panel) and $m_{H^\pm _1}$ (right panel)

The mass parameters $M_{0}$ and $m_{H^\pm _{1,2}}$ are independent with $\Delta a_{\mu }(H^\pm )$ in the allowed regions. It is more interesting to see the relations between two singly charged Higgs boson masses and $M_0$, and between ${\hat{x}}_{\nu 3}$ and two charged Higgs boson masses and $M_0$, see Fig. 5. In the left panel, small ${\hat{x}}_{\nu 3}$ is disfavored and allowed with only large $M_0$ up to the upper bound of the scanned range. In the right panel, the allowed region favors both small values of $m_{H^\pm _1}$ and $m_{H^\pm _2}$, but requires $|m_{H^\pm _1}-m_{H^\pm _2}|\ge 252.4$ GeV.

Other interesting correlations between different free parameters versus ${\hat{x}}_{\nu 3}$ are shown in Fig. 6. First, the allowed regions favor large ${\hat{x}}_{\nu 3}$, which supports small $s_{2\alpha }$ and large $t_{\beta }$. In addition, careful numerical investigations show that the recent constraint on ${\hat{x}}_{\nu 3}$ given in Eq. (37) does not allow $s_{\alpha }=0$ or $t_{\beta }>30$. This conclusion excludes completely the allowed regions indicated in Ref. [76], where large $t_{\beta }>30$ is one of the necessary requirements to explain the experimental $(g-2)_{\mu }$ data. This important difference appears because of the different Higgs triplets in the Yukawa term generating $M_D$ and Higgs couplings, depending on which models 331ISS or 331$\beta$. The future update on ${\hat{x}}_{\nu 3}$ will lead to a significant lower bound of $s_{2\alpha }$, for example ${\hat{x}}_{\nu 3}\le 10^{-4}$ will result in $|s_{2\alpha }|\ge 0.07$, $t_{\beta }\le 15$, and $|Y^d_2|\ge 0.4$. On the other hand, small ${\hat{x}}_{\nu 3} <5. 10^{-7}$ requires simultaneously small $t_{\beta }>0.3$, large $|s_{2\alpha }|\rightarrow 1$, and large $Y^d_2$ corresponding to max$| \left( Y^{h}_{2}\right) _{ab}| \rightarrow 3.0$. This is the reason why ${\hat{x}}_{\nu 3}$ must be bounded from below.

Fig. 6

The correlations between different free parameters vs ${\hat{x}}_{\nu 3}$

5 Conclusion

The two models $331\beta$ under consideration and 331ISS given in Ref. [76] have two identical Yukawa couplings generating masses to charged leptons and top quarks, therefore keep the same lower bound $t_{\beta }>0.3$. But they predict two opposite ranges of $t_{\beta }$ in the regions explaining successfully the experimental data of $(g-2)_{e,\mu }$. Namely, the regions predicted by the 331$\beta$ model requires small $t_{\beta }<30$, in contrast to the requirement of $t_{\beta }>30$ indicated for the 331ISS model. These opposite predictions of allowed $t_{\beta }$ depend on which Higgs triplets appear in the Yukawa terms needed to generate Dirac neutrino mass matrix and couplings of singly charged Higgs bosons.

We have indicated that the $331\beta$ model adding heavy neutrinos and singly charged Higgs bosons $h^\pm$ as $SU(3)_L$ singlets can explain both experimental data of $(g-2)_{\mu ,e}$ in the ISS framework. Apart from the well-known property that this mechanism generates active neutrino masses and mixing consistent with neutrino oscillations data, it also allows both large values of non-unitary mixing parameters and heavy neutrino masses larger than order of ${\mathcal {O}}(10^2)$ GeV. The new singly charged Higgs bosons as $SU(3)_L$ singlets will mix with the other Higgs components predicted by the 331$\beta$ model, leading to new free couplings $Y^{h}$ of singly charged Higgs bosons with heavy ISS neutrinos and charged leptons. All of these features result in the chirally-enhanced one-loop contributions from heavy ISS neutrino exchanges to the AMM of electron and muon. These contributions can be large up to the order of $\Delta a^{\mathrm {NP}}_{e,\mu }$. We have confirmed this conclusion from numerical illustrations in the limits of the simplest forms of the total neutrino mass matrix and the Yukawa coupling matrix $Y^{h}$ needed to avoid large Br$(e_b\rightarrow e_a\gamma )$. The phenomenology of the model $331\beta$ will be richer when these limits are relaxed, and should be studied in more detail.

Appendices

Appendix A: One loop contribution to the form factor $c_{(ba)R}$ for cLFV decays $e_b\rightarrow e_a\gamma$ and $\Delta a_{e_a}$

We collect here the results given in Ref. [36], which were used directly to construct our analytic formulas corresponding to the particular properties of the 3-3-1 models. The general Lagrangian for needed interactions ($b\equiv i$, $a\equiv f$):

$$\begin{aligned} {\mathcal {L}}_{\Phi }&= {\overline{\Psi }} \left( \Gamma _{\Psi \Phi }^{aL}P_L + \Gamma _{\Psi \Phi }^{aR}P_R\right) e_a \Phi ^*+\text {h.c.}, \nonumber \\ {\mathcal {L}}_{V}&= {\overline{\Psi }} \left( \Gamma _{\Psi V}^{aL} \gamma ^{\mu }P_L + \Gamma _{\Psi V}^{aR} \gamma ^{\mu } P_R\right) e_a V_{\mu }^*+\text {h.c.} \end{aligned}$$

(A1)

The form factors $c_{(ab)R}$ corresponding to the one-loop contribution of a boson X coupling with a fermion $\psi$ and usual charged $e_a$ are:

$$\begin{aligned} c^X_{(ab)R}&\equiv \frac{e}{16\pi ^2 m^2_{X}} \left\{ \Gamma ^{aL*}_{\Psi X} \Gamma ^{bR}_{\Psi X} m_{\Psi } \left[ f_X \left( t_X\right) +Q g_X \left( t_X\right) \right] \right. \nonumber \\&\quad + \left. \left[ m_{e_b} \Gamma ^{aL*}_{\Psi X} \Gamma ^{bL}_{\Psi X} + m_{e_a} \Gamma ^{aR*}_{\Psi X} \Gamma ^{bR}_{\Psi X} \right] \right. \nonumber \\&\quad + \left. \left[ {\tilde{f}}_X \left( t_X\right) +Q {\tilde{g}}_X \left( t_X\right) \right] \right\} , \end{aligned}$$

(A2)

where $X=\Phi ,V$, $t_X\equiv m^2_{\Psi }/m^2_X$, $Q\equiv Q_\Psi$ is the electric charge of the fermion $\Psi$, and the master functions are

$$\begin{aligned} f_\Phi (x)&= 2{\tilde{g}}_\Phi (x)=\frac{x^2-1 -2x\ln x}{4(x-1)^3},\nonumber \\ g_\Phi&=\frac{x-1 -\ln x}{2(x-1)^2}, \nonumber \\ {\tilde{f}}_\Phi (x)&= \frac{2x^3 +3x^2 -6x +1 -6x^2 \ln x}{24(x-1)^4}, \nonumber \\ f_V(x)&= \frac{x^3 -12 x^2 +15 x -4 +6 x^2\ln x}{4(x-1)^3}, \nonumber \\ g_V(x)&=\frac{x^2-5x +4+3 x \ln x}{2(x-1)^2}, \nonumber \\ {\tilde{f}}_V(x)&= \frac{-4x^4 +49x^3 -78 x^2 +43x -10 -18x^3\ln x}{24(x-1)^4}, \nonumber \\ {\tilde{g}}_V(x)&= \frac{-3(x^3 -6x^2 +7x -2 +2x^2\ln x)}{(x-1)^3}. \end{aligned}$$

(A3)

We note that except $g_{\Phi }(x)$, all of the remaining master functions given in (A3) are bounded in finite ranges, namely $0\le f_\Phi (x), g_\Phi (x), {\tilde{f}}_\Phi (x), f_V(x), g_V(x), -{\tilde{f}}_V(x), -{\tilde{g}}_V(x)\le a \le 2$. Regarding $g_{\Phi }(x)$, although $\lim _{x\rightarrow 0}g_{\Phi }(x)= \infty$, the appearance of the factor $m_{\Psi }/m^2_{X}$ along with this function will result in the fact that the relevant contributions should be calculated by the modified function $g(x)\rightarrow \sqrt{x}g_{\Phi }(x)$ that is always finite and have bound $0\le \sqrt{x}g_{\Phi }(x)\le \frac{1}{4}$. In the 3-3-1 models discussed in this work, the modified function is $xg_{\Phi }(x)$ mentioned in Eq. (45) is also finite for all x. Furthermore, $\Gamma ^{bR}_{\Psi V}=0$ for all charged gauge bosons $V=W,\; Y$ hence $f_V(x)$ and $g_V(x)$ do not appear in our calculation.

Appendix B: Detailed steps of calculation

The one-loop contributions of the singly charged Higgs bosons to AMM is

$$\begin{aligned} a_{e_a} (H^\pm _k)&= \frac{-f_a}{m^2_{H^\pm _k} } \sum _{i=1}^{K+3} \left[ \lambda ^{L,k*}_{ia } \lambda ^{R,k}_{ia }m_{n_i} f_{\Phi }(x_{i,k})\right. \nonumber \\&\quad \left. + m_{e_a} \left( \lambda ^{L,k*}_{ia } \lambda ^{L,k}_{ia } + \lambda ^{R,k*}_{ia } \lambda ^{R,k}_{ia}\right) {\tilde{f}}_{\Phi }(x_{i,k}) \right] \nonumber \\&= \frac{-f_a}{m^2_{H^\pm _k} } \left\{ \sum _{i=1}^{3} \left[ \lambda ^{L,k*}_{ia } \lambda ^{R,k}_{ia }m_{n_i} f_{\Phi }(0)\right. \right. \nonumber \\&\quad \left. \left. + m_{e_a} \left( \lambda ^{L,k*}_{ia } \lambda ^{L,k}_{ia } +\lambda ^{R,k*}_{ia } \lambda ^{R,k}_{ia }\right) {\tilde{f}}_{\Phi }(0) \right] \right. \nonumber \\&\quad \left. +\sum _{i=4}^{K+3} \left[ \lambda ^{L,k*}_{ia } \lambda ^{R,k}_{ia }M_0 f_{\Phi }(x_{k}) \right. \right. \nonumber \\&\quad \left. \left. + m_{e_a} \left( \lambda ^{L,k*}_{ia } \lambda ^{L,k}_{ia } + \lambda ^{R,k*}_{ia }\lambda ^{R,k}_{ia }\right) {\tilde{f}}_{\Phi }(x_{k}) \right] \right\} , \end{aligned}$$

(B1)

where $x_{i,k}\equiv m^2_{n_i}/m^2_{H^\pm _k}$, $f_a= \frac{g^2m_{e_a}\;}{8 \pi ^2 m^2_W} >0$. Using the approximations that $m^2_{n_i}/m^2_{H^\pm _k}\simeq 0$ for $i\le 3$, otherwise $m^2_{n_i}/m^2_{H^\pm _k}\simeq M_0^2/m^2_{H^\pm _k}=x_k$, we have $f_{\Phi }(x_{i,k})\simeq f_{\Phi }(0)$ for $i\le 3$ and $f_{\Phi }(x_{i,k})\simeq f_{\Phi }(x_k)$ for $i>3$, leading to the precise analytic formulas for different left-right parts as follows

$$\begin{aligned}&\sum _{i=1}^{K+3}\lambda ^{L,1*}_{ia } \lambda ^{R,1}_{ia }m_{n_i} f_{\Phi }(x_{i,1})\nonumber \\&\quad = \left\{ -m_{e_a} c^2_{\alpha } \left[ M_D^{\dagger }R^Tm_{\nu }\left( I_3-\frac{1}{2}RR^{\dagger }\right) \right] _{aa}\right. \nonumber \\&\qquad \left. +\frac{v}{\sqrt{2}}t_{\beta }^{-1} s_{\alpha }c_{\alpha } \left[ M_D^{\dagger }R^Tm_{\nu }R Y^{h}\right] _{aa} \right\} f_{\Phi }(0)\nonumber \\&\qquad + \left\{ m_{e_a}c^2_{\alpha }\left[ M_D^{\dagger }\left( I_K-\frac{1}{2}R^TR^{*} \right) V^*V^{\dagger }R^{\dagger }\right] _{aa}\right. \nonumber \\&\qquad \left. +\frac{v}{\sqrt{2}}t^{-1}_{\beta }c_{\alpha }s_{\alpha }\right. \nonumber \\&\qquad \times \left. \left[ M_D^{\dagger }\left( I_K-\frac{1}{2}R^TR^*\right) V^*V^{\dagger }\right. \right. \nonumber \\&\qquad \times \left. \left. \left( I_K-\frac{1}{2}R^{\dagger }R\right) Y^{h}\right] _{aa} \right\} M_0f_{\Phi }(x_{1}),\nonumber \\&\sum _{i=1}^{K+3} m_{e_a}\lambda ^{L,1*}_{ia }\lambda ^{L,1}_{ia } {\tilde{f}}_{\Phi }(x_{i,k})\nonumber \\&\quad = m_{e_a}t_{\beta }^{-2}c_{\alpha }^2 \left\{ \left( M_D^{\dagger }R^TR^*M_D\right) _{aa} {\tilde{f}}_{\Phi }(0)\right. \nonumber \\&\qquad \left. + \left[ M_D^{\dagger }\left( I_K-\frac{1}{2}R^TR^*\right) ^2M_D\right] _{aa}{\tilde{f}}_{\Phi }(x_1)\right\} ,\nonumber \\&\sum _{i=1}^{K+3} \lambda ^{R,1*}_{ia } \lambda ^{R,1}_{ia }{\tilde{f}}_{\Phi }(x_{i,k})= m^2_{e_a}t^2_{\beta }c^2_{\alpha }\nonumber \\&\quad \times \left[ \left( I_3-\frac{1}{2}RR^{\dagger }\right) ^2_{aa} {\tilde{f}}_{\Phi }(0)+ \left( RR^{\dagger }\right) _{aa}{\tilde{f}}_{\Phi }(x_1) \right] \nonumber \\&\qquad +\frac{v^2s^2_{\alpha }}{2} \left\{ \left( Y^{h\dagger }R^{\dagger } R Y^{h}\right) _{aa}{\tilde{f}}_{\Phi }(0)\right. \nonumber \\&\qquad \left. + \left[ Y^{h\dagger } \left( I_K-\frac{1}{2}R^{\dagger }R\right) ^2Y^{h}\right] _{aa} {\tilde{f}}_{\Phi }(x_1) \right\} \nonumber \\&\qquad +\frac{v m_{e_a}t_{\beta } s_{2\alpha }}{\sqrt{2}}{\mathrm {Re}}\left\{ -\left[ \left( I_3-\frac{RR^{\dagger }}{2}\right) RY^{h} \right] _{aa}{\tilde{f}}_{\Phi }(0)\right. \nonumber \\&\qquad \left. + \left[ R\left( I_K-\frac{R^{\dagger }R}{2}\right) Y^{h}\right] _{aa}{\tilde{f}}_{\Phi }(x_1) \right\} , \end{aligned}$$

(B2)

where $s_{2\alpha }=2s_{\alpha }c_{\alpha }$, $U\equiv U_{{\mathrm {PMNS}}}$, and $m_{\nu }\equiv U^*{\hat{m}}_{\nu }U^{\dagger }$. Ignoring suppressed term proportional to ${\mathcal {O}}(R^3)$ and setting ${\tilde{f}}_{\Phi }(0) =\frac{1}{24}$, we have

$$\begin{aligned} a_{e_a} (H^\pm _1)&=- \frac{g^2 m_{e_a}^2}{8\pi ^2 m^2_{W} }{\mathrm {Re}}\left\{ \left[ c^2_{\alpha } \left( M_D^{\dagger } V^*V^{\dagger }R^{\dagger }\right) _{aa}\right. \right. \nonumber \\&\quad \left. + \frac{vt_{\beta }^{-1}c_{\alpha }s_{\alpha }}{\sqrt{2}m_{e_a}} \left( M_D^{\dagger }V^*V^{\dagger } Y^{h}\right) _{aa}\right] \frac{M_0 f_{\Phi }(x_1)}{m^2_{H^\pm _1}} \nonumber \\&\quad +t_{\beta }^{-2} c^2_{\alpha } \left[ \frac{\left( M_D^{\dagger }R^TR^*M_D \right) _{aa}}{m^2_{H^\pm _1}} \left( \frac{1}{24} -{\tilde{f}}_{\Phi }(x_1) \right) \right. \nonumber \\&\quad \left. + \frac{\left( M_D^{\dagger }M_D \right) _{aa}}{m^2_{H^\pm _1}}{\tilde{f}}_{\Phi }(x_1) \right] \nonumber \\&\quad + \frac{m^2_{e_a} t_{\beta }^2c_{\alpha }^2}{m^2_{H^\pm _1}} \left[ \frac{1}{24} - (RR^{\dagger })_{aa}\left( \frac{1}{24} -{\tilde{f}}_{\Phi }(x_1) \right) \right] \nonumber \\&\quad + \frac{v^2s_{\alpha }^2}{2 m^2_{H^\pm _1}} \left[ \left( Y^{h\dagger }R^{\dagger }RY^{h}\right) _{aa}\left( \frac{1}{24} -{\tilde{f}}_{\Phi }(x_1) \right) \right. \nonumber \\&\quad \left. + \left( Y^{h\dagger }Y^{h}\right) _{aa} {\tilde{f}}_{\Phi }(x_1)\right] \nonumber \\&\quad -\left. \frac{v m_{e_a} t_{\beta } s_{2\alpha }}{\sqrt{2} m^2_{H^\pm _1}} \left[ \left( RY^{h}\right) _{aa}\left( \frac{1}{24}- {\tilde{f}}_{\Phi }(x_1) \right) \right] \right. \nonumber \\&\quad \left. +{\mathcal {O}}(R^3) \right\} , \end{aligned}$$

(B3)

and

$$\begin{aligned} a_{e_a} (H^\pm _2) =a_{e_a} (H^\pm _1)\left[ x_1 \rightarrow \; x_2,\;s_{\alpha }\rightarrow -c_{\alpha }, c_{\alpha } \rightarrow s_{\alpha }\right] . \nonumber \\ \end{aligned}$$

(B4)

The total mixing matrices of neutrino corresponding to the MSS and ISS frameworks are

$$\begin{aligned} U^{\nu }= \left( \begin{array}{cc} U_{{\mathrm {PMNS}}}\left( 1 -\frac{{\hat{m}}_{\nu }}{2M_0} \right) &{} -i U_{{\mathrm {PMNS}}}\left( \frac{{\hat{m}}_{\nu }}{M_0} \right) ^{1/2} \\ -i\left( \frac{{\hat{m}}_{\nu }}{M_0} \right) ^{1/2} &{} 1 -\frac{{\hat{m}}_{\nu }}{2M_0}\\ \end{array} \right) \end{aligned}$$

(B5)

and

$$\begin{aligned} U^{\nu }=\begin{pmatrix} U_{{\mathrm {PMNS}}} \left( I_3-\frac{1}{2} {\hat{x}}_\nu \right) &{} \quad i U_{{\mathrm {PMNS}}} \frac{{\hat{x}}_\nu ^{1/2}}{\sqrt{2}} &{} \quad U_{{\mathrm {PMNS}}}\frac{{\hat{x}}_\nu ^{1/2}}{\sqrt{2}}\\ 0_{3\times 3} &{} -\frac{i I_3}{\sqrt{2}} &{} \frac{I_3}{\sqrt{2}} \\ -{\hat{x}}_\nu ^{1/2} &{}\frac{ i}{\sqrt{2}}\left( I_3 - \frac{{\hat{x}}_\nu }{2} \right) &{} \frac{ 1}{\sqrt{2}}\left( I_3 - \frac{{\hat{x}}_\nu }{2} \right) \end{pmatrix}, \end{aligned}$$

(B6)

respectively. They satisfy the unitary condition: $U^{\nu \dagger } U^{\nu }= U^{\nu } U^{\nu \dagger }=I_3 +{\mathcal {O}}\left( \left[ \frac{{\hat{m}}_{\nu }}{M_0}\right] ^{2} \right)$ and $U^{\nu \dagger } U^{\nu }= U^{\nu } U^{\nu \dagger }=I_9 +{\mathcal {O}}\left( {\hat{x}}_{\nu }^{2} \right)$.

Appendix C: Masses and mixing of the singly charged Higgs bosons

From the three relations corresponding to the minimal conditions of the Higgs potential (21), three parameters $\mu _{1,2,3}$ are written in terms of the remaining Higgs potential couplings and non-zero vevs of the neutral Higgs components, namely

$$\begin{aligned} \mu _1^2&= -\frac{f c_{\beta } u}{s_{\beta }}-\frac{1}{2} c_{\beta }^2 \lambda _{12} v^2-\frac{\lambda _{13} u^2}{2}-\lambda _1 s_{\beta }^2 v^2, \nonumber \\ \mu _2^2&= -\frac{f s_{\beta } u}{c_{\beta }}-c_{\beta }^2 \lambda _2 v^2-\frac{\lambda _{23} u^2}{2}-\frac{1}{2} \lambda _{12} s_{\beta }^2 v^2,\nonumber \\ \mu _3^2&-\frac{f c_{\beta } s_{\beta } v^2}{u}-\frac{1}{2} c_{\beta }^2 \lambda _{23} v^2-\lambda _3 u^2-\frac{1}{2} \lambda _{13} s_{\beta }^2 v^2. \end{aligned}$$

(C1)

Inserting these relations into the Higgs potential (21), we obtain the squared mass matrix of the singly charged Higgs bosons in the basis $(\rho ^\pm ,\; \eta ^\pm ,\; h^\pm )^T$ as follows

$$\begin{aligned}&{\mathcal {M}}^2_c= \left( \begin{array}{ccc} \frac{c_{\beta } \left( c_{\beta } {\tilde{\lambda }}_{12} s_{\beta } v^2-2 f u\right) }{2 s_{\beta }} &{} \frac{1}{2} c_{\beta } {\tilde{\lambda }}_{12} s_{\beta } v^2-f u &{} \frac{c_{\beta } f_h v}{\sqrt{2}} \\ \frac{1}{2} c_{\beta } {\tilde{\lambda }}_{12} s_{\beta } v^2-f u &{} \frac{s_{\beta } \left( c_{\beta } {\tilde{\lambda }}_{12} s_{\beta } v^2-2 f u\right) }{2 c_{\beta }} &{} \frac{f_h s_{\beta } v}{\sqrt{2}} \\ \frac{c_{\beta } f_h v}{\sqrt{2}} &{} \frac{f_h s_{\beta } v}{\sqrt{2}} &{} \frac{1}{2} \left( \lambda _3^h u^2 +\left( \lambda _2^h c_{\beta }^2+s_{\beta }^2 \lambda _1^h\right) v^2+2 \mu _4^2\right) \\ \end{array} \right) .\nonumber \\ \end{aligned}$$

(C2)

Diagonalizing this matrix will result in a zero eigenvalue and two massive ones denoted as $m^2_{H^\pm _{12}}$, which correspond to a goldstone boson $\phi ^\pm _W$ and two physical singly charged Higgs bosons $H^\pm _{1,2}$. The mixing matrix C used to diagonalize $C{\mathcal {M}}^2_cC^T= {\mathrm {diag}}\left( 0,\; m^2_{H^\pm _1},\;m^2_{H^\pm _2}\right)$ can be written as a product of the two unitary transformations $C=C_2C_1$, where

$$\begin{aligned} C_1\equiv \left( \begin{array}{ccc} -s_{\beta } &{} c_{\beta } &{} 0 \\ c_{\beta } &{} s_{\beta } &{} 0 \\ 0 &{} 0 &{} 1 \\ \end{array} \right) ,\; C_2\equiv \left( \begin{array}{ccc} 1 &{} 0 &{} 0 \\ 0 &{} c_{\alpha } &{} -s_{\alpha } \\ 0 &{} s_{\alpha } &{} c_{\alpha } \\ \end{array} \right) . \end{aligned}$$

(C3)

The $C_1$ was introduced previously [64, 67] corresponding to the decoupling limit between $h^\pm$ and two Higgs triplets $\rho ^\pm$ and $\eta ^\pm$, namely the matrix

$$\begin{aligned}&C_1 {\mathcal {M}}^2_cC_1^T\nonumber \\&\quad ={\mathcal {M}}^2_{c,1}=\left( \begin{array}{ccc} 0 &{} 0 &{} 0 \\ 0 &{} \frac{{\tilde{\lambda }}_{12} v^2}{2}-\frac{f u}{c_{\beta } s_{\beta }} &{} \frac{f_h v}{\sqrt{2}} \\ 0 &{} \frac{f_h v}{\sqrt{2}} &{} \frac{1}{2} \left( \lambda _3^h u^2+\left( \lambda _2^h c_{\beta }^2+s_{\beta }^2 \lambda _1^h\right) v^2+2 \mu _4^2\right) \\ \end{array} \right) , \end{aligned}$$

(C4)

which is diagonal when triple coupling $f_h=0$. In contrast, $C_2$ presents the mixing between the singlet $h^\pm$ and two singly charged Higgs components of the two Higgs triplets, $C_2 {\mathcal {M}}^2_{c,1}C_2^T= {\mathrm {diag}}\left( 0,\; m^2_{H^\pm _1},\;m^2_{H^\pm _2}\right)$. It can be proved that

$$\begin{aligned}&\tan (2\alpha )\nonumber \\&\quad = \frac{2 \sqrt{2} c_{\beta } f_h s_{\beta } v}{2 f u+c_{\beta }^3 s_{\beta } \lambda _2^h v^2+c_{\beta } s_{\beta } \left[ v^2 \left( s_{\beta }^2 \lambda _1^h-\tilde{\lambda }_{12}\right) {+}\lambda _3^h u^2{+}2 \mu _4^2\right] },\nonumber \\ \end{aligned}$$

(C5)

and $m_{H^\pm _{1,2}}$ are functions of parameters included in the matrix ${\mathcal {M}}^2_{c,1}$ including $f_h,\;f$, and $\mu _4$. On the other hand, we can choose $m_{H^\pm _{1,2}}$ and $\alpha$ as free parameters while $f_h,\;f$, and $\mu _4$ are dependent parameters, their analytic formulas are given in Eq. (23).