A radiatively induced inverse seesaw model with hidden U(1) gauge symmetry

Abstract

We propose an inverse seesaw scenario under hidden U(1) gauge symmetry, having rather natural hierarchies among neutral fermion mass scales. The hierarchies are derived from theory and experimental constraints. The theory requires Majorana exotic masses have to be induced at one-loop level. The experimental side requests that the Dirac mass terms has to be highly suppressed to satisfy the constraints from the lepton flavor violations (LFVs) such as $\mu \rightarrow e\gamma$. In order to induce such a small Majorana exotic masses we introduce exotic fermions and bosons that also provide us additional intriguing phenomenologies such as muon anomalous magnetic moment, dark matter candidate as well as LFVs and deviations in leptonic Z decays. We analyze these phenomenologies including neutrino oscillation data numerically, and show allowed region. Finally, we discuss the DM candidate in both the cases of fermion and scalar boson, where we focus on rather lighter range that is equal or less than 10 GeV. The dominant contributions originate from interactions of hidden gauge sector, and we show allowed ranges for both cases.

1 Introduction

Understanding electrically neutral fields is our common issues to be resolved in both theoretical and experimental view points. Typically, it is considered in the framework of physics beyond the standard model (SM) since the active neutrino masses as well as dark matter(DM) candidate cannot be explained in the SM unless new scale or fields are supplied. Even though several experiments have partially discovered nature of the active neutrinos measuring three mixing angles and two squared mass differences, we only have few ideas about CP phases etc.; Dirac CP phase, Majorana phases, neutrino mass eigenvalues, Majorana or Dirac mass type, and so on. This is because the mechanism to generate neutrino masses is not uniquely determined yet. On the other hand there are almost null direct information on nature of DM in spite of huge efforts by experimentalists on direct searches such as XENON1T [1] and LUX [2], indirect searches such as Fermi-LAT [3], AMS-02 [4], and CALET [5], and collider searches at LHC [6]. These experiments especially focus on weakly interacting massive particles (WIMP) whose typical mass would be the order 100 GeV to 1000 GeV. Although it does not suggest WIMP has totally been excluded, it would be worthwhile to consider a lighter DM candidate below e.g., 10 GeV. It would be more natural in view of relaxed hierarchies among neutral fields, since the active neutrino masses are minuscule that is the order $10^{-9}$ GeV. In order to realize such a tiny mass of neutrinos, we propose an improved inverse seesaw model with a hidden gauged U(1) symmetry that plays crucial roles in carrying out our neutrino model and assuring the stability of our DM due to remnant $Z_2$ symmetry after spontaneous symmetry breaking. Hidden U(1) symmetry has an advantage that the mass of hidden gauge boson $(m_{Z'})$ and its gauge coupling $(g'')$ are almost taken to be free, since the $Z'$ boson does not directly couple to the SM fields and there are almost no bounds by LEP [7] and LHC [8].¹ It is unlikely to flavorful gauge symmetries such as $U(1)_{B-L}$. This is why this kind of symmetry is frequently applied to lighter DM scenarios [9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24].

The original inverse seesaw mass matrix in basis of $[\nu _L,N_R^C,N_L]^T$ [25, 26] can be sketched as follows:

$$\begin{aligned} \left( \begin{array}{ccc} 0 &{} M_D &{} 0 \\ M_D &{}0 &{} M_N \\ 0 &{} M_N &{} \mu \end{array} \right) , \end{aligned}$$

(1)

then the active neutrino mass matrix is given by $\mu \left( \frac{M_D}{M_N}\right) ^2$ after the block-diagonalization. Notice here that it requires mass hierarchies $\mu< M_D<M_N$ at least in order to generate the appropriate neutrino mass scale. Usually, these mass hierarchies are imposed by hand.

To achieve the mass hierarchies, one of the interesting scenario is generation of Majorana $\mu$ mass term at loop level; here we call it as radiative inverse seesaw. In Ref. [27], a radiative inverse seesaw model is proposed applying local $U(1)_\chi$ motivated by SO(10) and adding some scalar fields; a model with local $U(1)_{B-L}$ is found in Ref. [28]. Moreover a model with lepton triplet [29] is considered where $\mu$ mass term at tree level is forbidden by the nature of gauge symmetry. $Z_4$ symmetry can be also used to forbid tree level $\mu$ mass term [30]. A radiative inverse seesaw model with a new gauge symmetry is phenomenologically interesting since it provides new phenomena at experiments.² Here we are interested in a scenario with hidden U(1) gauge symmetry where the SM fields are not charged under it. A $Z'$ boson from hidden U(1) would provide different phenomenology from non-hidden U(1) symmetry such as $U(1)_{B-L}$. Remarkably $Z'$ can have interactions with lepton flavor violation (LFV) when the SM lepton and exotic fermions with hidden U(1) charge mixes. It is thus interesting to consider a radiative inverse seesaw model with hidden U(1) gauge symmetry and discuss various lepton flavor physics.

In this paper we propose a radiative inverse seesaw model with hidden U(1) gauge symmetry. In our model, we explain the hierarchies among $\{ \mu , M_D, M_N\}$ through theoretical and experimental manners; $\mu$ is suppressed by generating it at one-loop level,³ while $M_D/M_N$ has to be tiny enough to satisfy lepton flavor violations such as $\mu \rightarrow e\gamma$ process, as can be seen in our analysis. Here $M_N$ is introduced as a mass of vector-like lepton doublet $L'$ and it is not suppressed. Majorana mass $\mu$ is forbidden at tree level due to the nature of gauge symmetry. We realize it at one loop level where we introduce singlet fermion N, inert scalar doublet $\eta$ and singlet $\chi$ with nonzero hidden charges, in order to induce relevant loop diagrams. Here $\{N, \eta , \chi \}$ fields will have remnant $Z_2$ odd parity after spontaneous symmetry breaking of hidden U(1) and the lightest neutral particle among them can be our DM candidate. Thanks to these additional fields, we can also discuss flavor physics such as muon anomalous magnetic moment(muon $g-2$), LFVs and deviations in leptonic Z boson decays as well as DM phenomenologies.

This paper is organized as follows. In Sect. 2, we review our model and formulate the Higgs sector, and gauge sector. In Sect. 3, we discuss the neutrino masses, LFVs, muon $g-2$, leptonic Z boson decays, and show numerical analysis satisfying all the constraints except DM. In Sect. 4, we study two types of our DM; bosonic and fermionic one, and demonstrate the allowed region to satisfy the relic density for each candidate. Notice here that the dominant processes are via new gauge interactions that would not be independent of neutrino and flavor physics. In Sect. 5, we summarize and conclude.

2 Model

In this model we consider hidden $U(1)_H$ gauge symmetry in addition to the SM gauge symmetry. Here we introduce the vector like extra lepton doublet $L' = (N', E')$ and SM singlet fermions $N_{L,R}$ which have $U(1)_H$ charge x and x/2 respectively. In scalar sector, $SU(2)_L$ doublet $\eta$, singlet $\varphi$ and singlet $\chi$ are introduced whose $U(1)_H$ charges are x/2, x and x/2. The field contents and their charge assignments are summarized in Table 1. In this model, $\varphi$ and the SM Higgs field $\Phi$ develop VEVs spontaneously breaking $U(1)_H$ and electroweak symmetry. The scalar fields are written by

$$\begin{aligned} \Phi&= \begin{pmatrix} G^+ \\ \frac{1}{\sqrt{2}} (v + {{\tilde{h}}} + i G_Z) \end{pmatrix}, \quad \eta = \begin{pmatrix} \eta ^+ \\ \frac{1}{\sqrt{2}} ( \eta _R + i \eta _I) \end{pmatrix} \nonumber \\ \varphi&= \frac{1}{\sqrt{2}} (v_\varphi + \varphi _R + i G_{Z^\prime }), \quad \chi = \frac{1}{\sqrt{2}} ( \chi _R + i \chi _I), \end{aligned}$$

(2)

where v and $v_{\varphi }$ are VEVs of $\Phi$ and $\varphi$, and $G^\pm$, $G_{Z}$ and $G_{Z^\prime }$ are Nambu–Goldstone (NG) bosons absorbed by $W^\pm$, Z and $Z^\prime$ gauge bosons. Note that $Z_2$ symmetry remains after $U(1)_H$ symmetry breaking where $N_{L/R}$, $\eta$ and $\chi$ are parity odd under the $Z_2$ while the other particles are parity even. Thus the lightest $Z_2$ odd particle can be our DM candidate.

Table 1 Field contents and charge assignment in the model

In this model, Yukawa interactions for leptons⁴ and scalar potential are given by

$$\begin{aligned}&{{\mathcal {L}}} \supset \ y_\ell {{\bar{L}}}_L \Phi e_R + M_{L'} {{\bar{L}}}'_L L'_R + M_N {{\bar{N}}}_R N_L + y_D {{\bar{L}}}_L L'_R \varphi \nonumber \\&\qquad + y_{R} \bar{L}'_L {{\tilde{\eta }}} N_R + y_L {{\bar{L}}}'_R {{\tilde{\eta }}} N_L \nonumber \\&\qquad + y_{\varphi _R} {{\bar{N}}}^c_R N_R \varphi ^* + y_{\varphi _L} {{\bar{N}}}^c_L N_L \varphi ^* + h.c. \, , \end{aligned}$$

(3)

$$\begin{aligned}&V = \ m_\Phi ^2 \Phi ^\dagger \Phi + m_\eta ^2 \eta ^\dagger \eta + m_\varphi ^2 \varphi ^* \varphi + m_\chi ^2 \chi ^* \chi \nonumber \\&\qquad + \mu (\chi ^2 \varphi ^* + h.c.) \nonumber \\&\qquad + \uplambda _0((\Phi ^\dagger \eta ) \chi ^* \varphi ^* + h.c.) + \uplambda _\Phi (\Phi ^\dagger \Phi )^2 + \uplambda _\eta (\eta ^\dagger \eta )^2 \nonumber \\&\qquad + \uplambda _\chi (\chi ^* \chi )^2 + \uplambda _\varphi (\varphi ^* \varphi )^2 \nonumber \\&\qquad + \uplambda _{\Phi \eta } (\Phi ^\dagger \Phi )(\eta ^\dagger \eta ) + {{\tilde{\uplambda }}}_{\Phi \eta } (\Phi ^\dagger \eta )(\eta ^\dagger \Phi ) \nonumber \\&\qquad + \uplambda _{\Phi \varphi } (\Phi ^\dagger \Phi )(\varphi ^* \varphi ) + \uplambda _{\Phi \chi } (\Phi ^\dagger \Phi )(\chi ^* \chi ) \nonumber \\&\qquad + \uplambda _{\eta \varphi }(\eta ^\dagger \eta )(\varphi ^* \varphi ) + \uplambda _{\eta \chi } (\eta ^\dagger \eta )(\chi ^* \chi )\nonumber \\&\qquad + \uplambda _{\varphi \chi } (\varphi ^* \varphi )(\chi ^* \chi ). \end{aligned}$$

(4)

2.1 Scalar sector

Here we consider scalar sector in the model and formulate mass spectrum and corresponding mass eigenvalues. Firstly the VEVs of $\Phi$ and $\varphi$ are calculated from scalar potential Eq. (4) by solving the condition $\partial V/\partial v = \partial V/\partial v_\varphi = 0$. Then we obtain the conditions

$$\begin{aligned}&m_\Phi ^2 + \uplambda _\Phi v^2 + \frac{1}{2} \uplambda _{\Phi \varphi } v_\varphi ^2 = 0, \end{aligned}$$

(5)

$$\begin{aligned}&m_\varphi ^2 + \uplambda _\varphi v_\varphi ^2 + \frac{1}{2} \uplambda _{\Phi \varphi } v^2 = 0. \end{aligned}$$

(6)

Note that the conditions $m_\Phi ^2, m_\varphi ^2 < 0$ are required to make square of VEVs positive definite. In addition we assume $m_\chi ^2, m_{\eta }^2 >0$ so that $\chi$ and $\eta$ do not develop non-zero VEVs.

After spontaneous symmetry breaking, we obtain mass matrix for CP-even neutral scalar fields with $Z_2$-even parity such that

$$\begin{aligned} {{\mathcal {L}}} \supset \frac{1}{2} \begin{pmatrix} {{\tilde{h}}} \\ \varphi _R \end{pmatrix}^T \begin{pmatrix} 2 \uplambda _\Phi v^2 &{} \uplambda _{\Phi \varphi } v v_\varphi \\ \uplambda _{\Phi \varphi } v v_\varphi &{} 2 \uplambda _\varphi v_\varphi ^2 \end{pmatrix}. \begin{pmatrix} {{\tilde{h}}} \\ \varphi _R \end{pmatrix} \end{aligned}$$

(7)

Diagonalizing the mass matrix, the mass eigenstates are given by

$$\begin{aligned} m_{h,H}^2 = \uplambda _\Phi v^2 + \uplambda _\varphi v_\varphi ^2 \pm \sqrt{(\uplambda _\Phi v^2 - \uplambda _\varphi v_\varphi ^2)^2 + \uplambda _{\Phi \varphi }^2 v^2 v_\varphi ^2 }, \end{aligned}$$

(8)

where we identify $m_h = 125$ GeV as the SM Higgs mass. The mass eigenstates and mixing are written as

$$\begin{aligned}&\begin{pmatrix} {{\tilde{h}}} \\ \varphi _R \end{pmatrix} = \begin{pmatrix} \cos \beta &{} - \sin \beta \\ \sin \beta &{} \cos \beta \end{pmatrix} \begin{pmatrix} h \\ H \end{pmatrix}, \nonumber \\&\tan 2\beta = \frac{v v_{\varphi }\uplambda _{\Phi \varphi }}{v^2 \uplambda _\Phi - v^2_{\varphi }\uplambda _\varphi }, \end{aligned}$$

(9)

where h is identified as the SM Higgs boson.

The mass for $Z_2$-odd charged scalar field is given by

$$\begin{aligned} m_{\eta ^\pm }^2 = m_\eta ^2 + \frac{1}{2} \uplambda _{\Phi \eta } v^2 + \frac{1}{2} \uplambda _{\eta \varphi } v_\varphi ^2, \end{aligned}$$

(10)

where the corresponding mass eigenstate is $\eta ^\pm$ given in Eq. (2). We also obtain mass matrix for $Z_2$-odd CP-odd scalar fields such as

$$\begin{aligned}&{{\mathcal {L}}} \supset \frac{1}{2} \begin{pmatrix} \eta _I \\ \chi _I \end{pmatrix}^T \begin{pmatrix} m_{\eta ^\pm }^2 + \frac{1}{2} {{\tilde{\uplambda }}}_{\Phi \eta } v^2 &{} -\frac{1}{2} \uplambda _{0} v v_\varphi \\ -\frac{1}{2} \uplambda _{0} v v_\varphi &{} m_{\chi '}^2 \end{pmatrix} \begin{pmatrix} \eta _I \\ \chi _I \end{pmatrix}, \nonumber \\&m^2_{\chi '} \equiv m_\chi ^2 + \frac{1}{2} \uplambda _{\Phi \chi } v^2 + \frac{1}{2} \uplambda _{\varphi \chi } v_\varphi ^2 - \sqrt{2} \mu v_\varphi . \end{aligned}$$

(11)

Diagonalizing the mass matrix, the mass eigenstates are given by

$$\begin{aligned}&m_{A_1,A_2}^2 = \frac{1}{4}\left[ 2m_{\eta ^\pm }^2 + 2 m_{\chi ^\prime }^2 + v^2 {\tilde{\uplambda }}_{\Phi \eta } \pm \sqrt{(2m_{\eta ^\pm }^2 + 2m_{\chi ^\prime }^2 + v^2 {\tilde{\uplambda }}_{\Phi \eta })^2 -4(4m_{\eta ^\pm }^2 m_{\chi ^\prime }^2 -v^2v_{\varphi }^2 \uplambda _0^2 + 2m_{\chi ^\prime }^2 v^2 {\tilde{\uplambda }}_{\Phi \eta })} \right] , \end{aligned}$$

(12)

The mass eigenstates and mixing are written as

$$\begin{aligned}&\begin{pmatrix} \eta _I \\ \chi _I \end{pmatrix} = \begin{pmatrix} \cos \gamma &{} - \sin \gamma \\ \sin \gamma &{} \cos \gamma \end{pmatrix} \begin{pmatrix} A_1 \\ A_2 \end{pmatrix}, \nonumber \\&\tan 2 \gamma = \frac{v v_{\varphi }\uplambda _0}{m^2_{\chi ^\prime } - m^2_{\eta ^\pm } - \frac{1}{2}v^2 {\tilde{\uplambda }}_{\Phi \eta }}. \end{aligned}$$

(13)

Furthermore mass matrix for $Z_2$-odd CP-even scalar fields are obtained as follows

$$\begin{aligned}&{{\mathcal {L}}} \supset \frac{1}{2} \begin{pmatrix} \eta _R \\ \chi _R \end{pmatrix}^T \begin{pmatrix} m_{\eta ^\pm }^2 + \frac{1}{2} {{\tilde{\uplambda }}}_{\Phi \eta } v^2 &{} \frac{1}{2} \uplambda _{0} v v_\varphi \\ \frac{1}{2} \uplambda _{0} v v_\varphi &{} m_{\chi ''}^2 \end{pmatrix} \begin{pmatrix} \eta _R \\ \chi _R \end{pmatrix}, \nonumber \\&m^2_{\chi ''} \equiv m_\chi ^2 + \frac{1}{2} \uplambda _{\Phi \chi } v^2 + \frac{1}{2} \uplambda _{\varphi \chi } v_\varphi ^2 + \sqrt{2} \mu v_\varphi . \end{aligned}$$

(14)

Diagonalizing the mass matrix, the mass eigenstates are given by

$$\begin{aligned} m_{H_1,H_2}^2 = \frac{1}{4}\Big [ 2 m_{\eta ^\pm }^2 + 2 m_{\chi ^{\prime \prime }}^2 + v^2 {\tilde{\uplambda }}_{\Phi \eta } \pm \sqrt{(2m_{\eta ^\pm }^2 + 2m_{\chi ^{\prime \prime }}^2 + v^2 {\tilde{\uplambda }}_{\Phi \eta })^2 -4(4m_{\eta ^\pm }^2 m_{\chi ^{\prime \prime }}^2 - v^2 v_{\varphi }^2 \uplambda _0^2 + 2m_{\chi ^{\prime \prime }}^2v^2{\tilde{\uplambda }}_{\Phi \eta })} \Big ], \end{aligned}$$

(15)

The mass eigenstates and mixing are written as

$$\begin{aligned}&\begin{pmatrix} \eta _R \\ \chi _R \end{pmatrix} = \begin{pmatrix} \cos \alpha &{} - \sin \alpha \\ \sin \alpha &{} \cos \alpha \end{pmatrix} \begin{pmatrix} H_1 \\ H_2 \end{pmatrix}, \end{aligned}$$

(16)

$$\begin{aligned}&\tan 2 \alpha = \frac{v v_\varphi \uplambda _0}{m^2_{\eta ^\pm } - m^2_{\chi ^{\prime \prime }} + \frac{1}{2}v^2 {\tilde{\uplambda }}_{\Phi \eta }}. \end{aligned}$$

(17)

We find that the mixing $\sin \alpha$ and $\sin \gamma$ can be written in terms of mass eigenvalues such that

$$\begin{aligned} \cos ^2\alpha= & {} \frac{(m^2_{A_2} - m^2_{H_2})(m^2_{H_2} - m^2_{A_1})}{(m^2_{A_2} + m^2_{A_1} - m^2_{H_2} - m^2_{H_1})(m^2_{H_2} - m^2_{H_1})} \nonumber \\ \cos ^2 \gamma= & {} \frac{(m^2_{A_2} - m^2_{H_2})(m_{A_2}^2 - m^2_{H_1})}{(m^2_{A_2} + m^2_{A_1} - m^2_{H_2} - m^2_{H_1})(m^2_{A_2} -m^2_{A_1})}. \end{aligned}$$

(18)

Note that we can write some parameters in the scalar potential by VEVs, scalar masses and mixings as summarized in Appendix where the independent parameters are $\{m_h, m_H, m_{H_1}, m_{H_2}, m_{A_1}, m_{A_2}, m_{\eta ^\pm }, v, v_{\varphi }, \cos \beta$, $\uplambda _{\Phi \chi },\uplambda _{\varphi \chi },\uplambda _{\Phi \eta }, \uplambda _{\eta \varphi }, \uplambda _{\eta \chi }, \uplambda _{\eta }, \uplambda _{\chi } \}$.

2.2 Gauge sector

The most general U(1) gauge Lagrangian including the kinetic mixing term is

$$\begin{aligned} {{\mathcal {L}}}_{\text {gauge}}= -\frac{1}{4}B_{\mu \nu }B^{\mu \nu } - \frac{1}{4}B^\prime _{\mu \nu }B^{\prime \mu \nu } - \frac{1}{2}\epsilon B_{\mu \nu }B^{\prime \mu \nu } \end{aligned}$$

(19)

where $B_{\mu \nu }$ and $B^\prime _{\mu \nu }$ are the field strength tensors of $U(1)_Y$ and $U(1)_H$ gauge symmetries. We can diagonalize Eq. (19) by the following transformation

$$\begin{aligned} \left( \begin{array}{c} {\tilde{B}}^\prime _\mu \\ {\tilde{B}}_\mu \\ \end{array}\right) =\left( \begin{array}{cc} \sqrt{1 - \epsilon ^2} &{} 0 \\ \epsilon &{} 1 \\ \end{array}\right) \left( \begin{array}{c} B^\prime _\mu \\ B_\mu \\ \end{array}\right) \end{aligned}$$

(20)

where $\epsilon$ is a dimensionless quantity ($\epsilon \ll 1$) and we parameterize $\rho =-\frac{\epsilon }{\sqrt{1-\epsilon ^2}}$. Under the transformation Eq. (20), the gauge Lagrangian can be written as

$$\begin{aligned} {{\mathcal {L}}}_{\text {gauge}}=-\frac{1}{4}{\tilde{B}}_{\mu \nu }{\tilde{B}}^{\mu \nu }-\frac{1}{4}{\tilde{B}}^\prime _{\mu \nu }{\tilde{B}}^{\prime \mu \nu } \end{aligned}$$

(21)

where ${\tilde{B}}_{\mu \nu }=\partial _\mu {\tilde{B}}_\nu - \partial _\nu {\tilde{B}}_\mu$ and ${\tilde{B}}^\prime _{\mu \nu }=\partial _\mu {\tilde{B}}^\prime _\nu - \partial _\nu {\tilde{B}}^\prime _\mu$.

The kinetic term of the scalar fields are

$$\begin{aligned} {{\mathcal {L}}}_{\text {kin}}= & {} (D_{\mu }\Phi )^\dagger (D^{\mu } \Phi ) + (D_{\mu }\eta )^\dagger (D^{\mu } \eta ) + (D_{\mu }\varphi )^\dagger (D^{\mu } \varphi ) \nonumber \\&+ (D_{\mu }\chi )^\dagger (D^{\mu } \chi ). \end{aligned}$$

(22)

The covariant derivatives of scalar fields are written by

$$\begin{aligned} D_{\mu }\Phi= & {} \Big (\partial _\mu + ig\frac{\tau ^a}{2}W_{\mu }^a + i\frac{g^\prime }{2}{\tilde{B}}_\mu + i\frac{g^\prime }{2}\rho {\tilde{B}}^\prime _\mu \Big )\Phi , \nonumber \\ D_{\mu }\eta= & {} \Big ( \partial _\mu + i g \frac{\tau ^a}{2} W_{\mu }^a + i\frac{g^\prime }{2}{\tilde{B}}_\mu + \Big ( i \frac{g^\prime }{2} \rho -i g^{\prime \prime } \frac{3x}{2} \frac{\rho }{\epsilon } \Big ){\tilde{B}}^\prime _\mu \Big )\eta , \nonumber \\ D_{\mu }\varphi= & {} \Big ( \partial _\mu -ig^{\prime \prime }x \frac{\rho }{\epsilon }{\tilde{B}}^\prime _\mu \Big )\varphi , \nonumber \\ D_\mu \chi= & {} \Big ( \partial _\mu - i \frac{g^{\prime \prime }}{2}x\frac{\rho }{\epsilon }{\tilde{B}}^\prime _mu \Big ) \chi , \end{aligned}$$

(23)

where g, $g'$ and $g''$ are gauge couplings of $SU(2)_L$, $U(1)_Y$ and $U(1)_H$, $W^a$ is the $SU(2)_L$ gauge field, and $\tau ^a$ is the Pauli matrix.

The scalar fields H and $\varphi$ get VEVs whereas other fields do not develop VEVs to preserve the $Z_2$ symmetry. The masses of the gauge bosons come from the first and third term of Eq. (22). The mass term of the neutral gauge bosons written in the basis of neutral gauge fields $(W_\mu ^3, {\tilde{B}}_\mu , {\tilde{B}}^\prime _\mu )$ is

$$\begin{aligned} {{\mathcal {L}}}_{\text {gauge}}^{\text {mass}} = \frac{1}{2} \left( \begin{array}{c} W_\mu ^3 \\ {\tilde{B}}_\mu \\ {\tilde{B}}^\prime _\mu \end{array}\right) ^T M^2_{\text {gauge}} \left( \begin{array}{c} W_\mu ^3 \\ {\tilde{B}}_\mu \\ {\tilde{B}}^\prime _\mu \end{array}\right) \end{aligned}$$

(24)

where

$$\begin{aligned} M^2_{\text {gauge}}=\frac{1}{4} \left( \begin{array}{ccc} g^2 v^2 &{} -gg^\prime v^2 &{} -gg^\prime v^2 \rho \\ -g g^\prime v^2 &{} g^{\prime 2} v^2 &{} g^{\prime 2} v^2 \rho \\ -g g^\prime v^2 \rho &{} g^{\prime 2} v^2 \rho &{} g^{\prime 2} v^2 \rho ^2 + 4 g^{\prime \prime 2}v^2_{\varphi }x^2\frac{\rho ^2}{\epsilon ^2} \\ \end{array}\right) .\nonumber \\ \end{aligned}$$

(25)

Here we parameterize

$$\begin{aligned} M_{Z^\prime }^2 = \frac{1}{4} g^{\prime 2} v^2 \rho ^2 + g^{\prime \prime 2} v_{\varphi }^2 x^2 \frac{\rho ^2}{\epsilon ^2}. \end{aligned}$$

(26)

We rotate the fields $(W_\mu ^3,{\tilde{B}}_\mu )$ by Weinberg angle $\theta _W$ to obtain the massless photon field $A_\mu$

$$\begin{aligned} \left( \begin{array}{c} W^3_\mu \\ {\tilde{B}}_\mu \\ \end{array}\right) = \left( \begin{array}{cc} \cos \theta _W &{} \sin \theta _W \\ -\sin \theta _W &{} \cos \theta _W\\ \end{array}\right) \left( \begin{array}{c} {\tilde{Z}}_\mu \\ A_\mu \\ \end{array}\right) , \end{aligned}$$

(27)

and the mass matrix for the massive neutral gauge bosons

$$\begin{aligned}&{{\mathcal {L}}}^{\text {mass}}_{\text {gauge}} = \frac{1}{2} \left( \begin{array}{c} {\tilde{Z}}_\mu \\ {\tilde{B}}^\prime _\mu \\ \end{array}\right) ^T \left( \begin{array}{cc} M_{Z,SM}^2 &{} -\Delta ^2 \\ -\Delta ^2 &{} M_{Z'}^2 \end{array}\right) \left( \begin{array}{c} {\tilde{Z}}_\mu \\ {\tilde{B}}^\prime _\mu \\ \end{array}\right) \end{aligned}$$

(28)

where $\Delta ^2 = \frac{1}{4} g^\prime v^2 \rho \sqrt{g^2 +g^{\prime 2}}$ and $M_{Z,SM}^2 = \frac{1}{4} v^2 (g^2 + g^{\prime 2})$. The physical masses of the neutral gauge bosons are

$$\begin{aligned} m_Z^2= & {} \frac{1}{2}\Big [M_{Z,SM}^2 + M_Z^{\prime 2} + \sqrt{(M_{Z,SM}^2 - M_{Z^\prime }^2)^2 + 4\Delta ^4} \Big ] \nonumber \\ m_{Z^{\prime }}^2= & {} \frac{1}{2}\Big [M_{Z,SM}^2 + M_Z^{\prime 2} - \sqrt{(M_{Z,SM}^2 - M_{Z^\prime }^2)^2 + 4\Delta ^4} \Big ].\nonumber \\ \end{aligned}$$

(29)

In the limit $\epsilon \rightarrow 0$, we have $m_{Z} \approx M_{Z,SM}$ and $m_{Z^\prime } \approx M_{Z^\prime }$. The mass matrix in Eq. (28) can be diagonalized by rotation matrix

$$\begin{aligned} \left( \begin{array}{c} {\tilde{Z}}_\mu \\ {\tilde{B}}^\prime _\mu \\ \end{array}\right)= & {} \left( \begin{array}{cc} \cos \theta &{} \sin \theta \\ -\sin \theta &{} \cos \theta \\ \end{array}\right) \left( \begin{array}{c} Z_\mu \\ Z^\prime _\mu \\ \end{array}\right) \end{aligned}$$

(30)

$$\begin{aligned} \tan 2\theta= & {} \frac{2\Delta ^2}{M_{Z,SM}^2 - M_{Z^\prime }^2} \end{aligned}$$

(31)

where $Z_\mu$ and $Z^\prime _\mu$ are the two physical gauge bosons correspond to the SM Z boson and extra gauge boson. The $Z-Z^\prime$ mixing would disappear in the limit $\epsilon \rightarrow 0$. In summary, the original unphysical gauge fields are transformed into mass eigenstates by

$$\begin{aligned} \left( \begin{array}{c} W_\mu ^3 \\ B_\mu \\ B^\prime _\mu \\ \end{array}\right)= & {} \left( \begin{array}{ccc} R^w_{11} &{} R^w_{12} &{} R^w_{13}\\ R^w_{21} + R^w_{31}\rho &{} R^w_{22} + R^w_{32}\rho &{} R^w_{23} + R^w_{33}\rho \\ -R^w_{31}\frac{\rho }{\epsilon } &{} -R^w_{32}\frac{\rho }{\epsilon } &{} -R^w_{33}\frac{\rho }{\epsilon }\\ \end{array}\right) \nonumber \\&\times \left( \begin{array}{c} Z_\mu \\ A_\mu \\ Z^\prime _\mu \\ \end{array} \right) \end{aligned}$$

(32)

$$\begin{aligned} R^w_{ab}= & {} \begin{pmatrix} \cos {\theta _w}\cos {\theta } &{} \sin {\theta _w} &{} \cos {\theta _w}\sin {\theta } \\ -\sin {\theta _w}\cos {\theta } &{} \cos {\theta _w} &{} -\sin {\theta _w}\sin {\theta } \\ -\sin {\theta }&{} 0 &{}\cos {\theta } \end{pmatrix}_{ab},\nonumber \\ \end{aligned}$$

(33)

where $a(b) = 1,2,3$.

2.3 Constraints in boson sector

Here we consider constraints in boson sector such as perturbativity, unitarity, vacuum stability, T-parameter and Higgs invisible decay. We mainly focus on two Higgs doublet $\{\Phi , \eta \}$ and $\varphi$ since parameters associated with $\chi$ are completely free and phenomenological constraints can be easily satisfied. The constraints from unitarity and perturbativity are given by [38, 39]

$$\begin{aligned}&|\uplambda _{\Phi ,\eta ,\varphi , \Phi \eta }| \le 4 \pi , \quad |\uplambda _{\Phi \varphi , \Phi \eta }| \le 8 \pi , \quad |\uplambda _{\Phi \eta } \pm {{\tilde{\uplambda }}}_{\Phi \eta }| \le 8 \pi , \nonumber \\&|\uplambda _{\Phi \eta } + 2 {{\tilde{\uplambda }}}_{\Phi \eta } | \le 8 \pi , \nonumber \\&\sqrt{|\uplambda _{\Phi \eta } (\uplambda _{\Phi \eta } + 2 {\tilde{\uplambda }}_{\Phi \eta })|} \le 8 \pi , \nonumber \\&\left| \uplambda _\Phi + \uplambda _\eta \pm \sqrt{(\uplambda _\Phi - \uplambda _\eta )^2 + {{\tilde{\uplambda }}}^2_{\Phi \eta }} \right| \le 8 \pi , \nonumber \\&\left| \uplambda _\Phi + \uplambda _\eta \pm \sqrt{(\uplambda _\Phi - \uplambda _\eta )^2 } \right| \le 8 \pi , \quad a_{1,2,3} \le 8 \pi , \end{aligned}$$

(34)

where $a_{1,2,3}$ are obtained as the solutions of the following equation

$$\begin{aligned}&x^3 - 2x^2 (3 \uplambda _\Phi + 3 \uplambda _\eta + 2 \uplambda _{\varphi }) \nonumber \\&\quad - x (2 \uplambda _{\Phi \varphi }^2 + 2 \uplambda _{\eta \varphi }^2 - 36 \uplambda _\Phi \uplambda _\eta - 24 \uplambda _\Phi \uplambda _{\Phi \varphi } \nonumber \\&\quad - 24 \uplambda _1 \uplambda _{\Phi \varphi } + 4 \uplambda _{\Phi \eta }^2+ 4\uplambda _{\Phi \eta } {{\tilde{\uplambda }}}_{\Phi \eta } + {\tilde{\uplambda }}_{\Phi \eta }^2 ) \nonumber \\&\quad + 4 (3 \uplambda _{\Phi \varphi }^2 \uplambda _\eta - \uplambda _{\Phi \varphi } \uplambda _{\eta \varphi } (2 \uplambda _{\Phi \eta } + {\tilde{\uplambda }}_{\Phi \eta }) + 3 \uplambda _{\eta \varphi }^2 \uplambda _\Phi \nonumber \\&\quad + \uplambda _{\varphi } ((2 \uplambda _{\Phi \eta }+ {{\tilde{\uplambda }}}_{\Phi \eta })^2 - 36 \uplambda _\Phi \uplambda _\eta )) =0. \end{aligned}$$

(35)

We can also writhe the conditions to guarantee vacuum stability such that [38]:

$$\begin{aligned}&\Omega _1 \cup \Omega _2 \end{aligned}$$

(36)

$$\begin{aligned}&\Omega _1 = \biggl \{ \uplambda _{\Phi ,\eta ,\varphi }>0, \ 2 \sqrt{\uplambda _\Phi \uplambda _{\varphi } } + \uplambda _{\Phi \varphi }>0, \nonumber \\&\qquad \qquad \ 2 \sqrt{\uplambda _\eta \uplambda _{\varphi } } + \uplambda _{\eta \varphi }>0, 2\sqrt{\uplambda _\Phi \uplambda _\eta } + \uplambda _{\Phi \eta } > 0,\nonumber \\&\qquad \qquad \ \uplambda _{\Phi \varphi } + \sqrt{\frac{\uplambda _\Phi }{\uplambda _\eta }} \uplambda _{\Phi \varphi } \ge 0 \biggr \} \end{aligned}$$

(37)

$$\begin{aligned}&\omega _2 = \biggl \{ \uplambda _{\Phi ,\eta ,\varphi }>0, \ 2 \sqrt{\uplambda _\Phi \uplambda _{\varphi } } \ge \uplambda _{\eta \varphi }> -2 \sqrt{\uplambda _\eta \uplambda _{\varphi } },\nonumber \\&\qquad \qquad \ \ 2 \sqrt{\uplambda _\Phi \uplambda _{\varphi } }> -\uplambda _{\Phi \varphi } \ge \sqrt{\frac{\uplambda _\Phi }{\uplambda _\eta }} \uplambda _{\eta \varphi }, \nonumber \\&\qquad \qquad \sqrt{(\uplambda _{\Phi \varphi }^2 - 4 \uplambda _\Phi \uplambda _{\varphi })(\uplambda _{\eta \varphi }^2 - 4 \uplambda _\eta \uplambda _{\varphi }) }\nonumber \\&\quad \qquad > \uplambda _{\Phi \varphi } \uplambda _{\eta \varphi } - 2 \uplambda _{\Phi \eta } \uplambda _{\varphi } \biggr \}. \end{aligned}$$

(38)

In addition we impose inert condition for $\eta$ requiring the potential be bounded below [40]

$$\begin{aligned}&-\frac{2}{\sqrt{21}}\sqrt{\uplambda _\Phi \uplambda _\eta }< \uplambda _{\Phi \eta }+ {{\tilde{\uplambda }}}_{\Phi \eta },\ -\sqrt{6} \uplambda '_{\Phi \eta }<|\uplambda _{\Phi \eta }+ {{\tilde{\uplambda }}}_{\Phi \eta }|, \nonumber \\&-\frac{2}{\sqrt{21}}\sqrt{\uplambda _\eta \uplambda _\varphi } < \uplambda _{\eta \varphi }. \end{aligned}$$

(39)

The contribution to T-parameter from scalar loops can be written as [41]

$$\begin{aligned} \Delta T&= \frac{1}{32 \alpha v^2} \left( G[m_{\eta ^\pm }^2, m_{A_1}^2] \cos ^2 \gamma + G[m_{\eta ^\pm }^2, m_{A_2}^2] \sin ^2 \gamma \right. \nonumber \\&\quad \left. + G[m_{\eta ^\pm }^2, m_{H_1}^2] \cos ^2 \alpha + G[m_{\eta ^\pm }^2, m_{H_2}^2] \sin ^2 \alpha \right) , \end{aligned}$$

(40)

$$\begin{aligned}&G[m_1^2,m_2^2] \equiv \frac{m_1^2 +m_2^2}{2} - \frac{m_1^2 m_2^2}{m_1^2 - m_2^2} \ln \left[ \frac{m_1^2}{m_2^2} \right] , \end{aligned}$$

(41)

where $\alpha$ is the fine structure constant. Here we adopt the constraint [8]

$$\begin{aligned} -0.01< \Delta T < 0.11, \end{aligned}$$

(42)

where vanishing U-parameter is assumed.

Fig. 1

Allowed region on $\{v_\varphi , m_H\}$ and $\{v_\varphi , \sin \beta \}$ planes

In our analysis below, we consider relatively small $v_\varphi (\ll v)$ to satisfy constraints from $L'$–L mixing induced by $\bar{L}_L L'_R \varphi$ Yukawa interaction. Thus $Z'$ and H masses tend to be light and we need to consider decay processes of $h \rightarrow Z'Z'$ and $h \rightarrow H H$. The decay widths are given by

$$\begin{aligned}&\Gamma (h \rightarrow Z' Z') = \frac{1}{8 \pi } \frac{m_{Z'}^4}{v_\varphi ^2 m_h} s^2_\beta \left( 2 + \frac{m_h^4}{4 m_{Z'}^4} \left( 1- \frac{2 m_{Z'}^2}{m_h^2} \right) ^2 \right) \nonumber \\&\qquad \qquad \qquad \qquad \times \sqrt{1 - \frac{4 m_{Z'}^2}{m_h^2}}, \end{aligned}$$

(43)

$$\begin{aligned}&\Gamma (h \rightarrow HH) = \frac{C_{hHH}^2}{16 \pi m_h} \sqrt{1 - \frac{4 m_{H}^2}{m_h^2}}, \end{aligned}$$

(44)

$$\begin{aligned}&C_{hHH} \equiv \frac{\uplambda _{\Phi \varphi }}{2} \left( v (c^2_\beta - 2 c_\beta s^2_\beta ) + v_\varphi (s^3_\beta - 2 c^2_\beta s_\beta ) \right) \nonumber \\&\qquad \qquad \quad + \frac{9}{2} s_\beta c_\beta (\uplambda _\Phi s_\beta v + \uplambda _\varphi c_\beta v_\varphi ), \end{aligned}$$

(45)

where $c_\beta = \cos \beta (s_\beta = \sin \beta )$. Here H mainly decays into $Z'Z'$ mode and $Z'$ decays into the SM particles via kinetic mixing or Lepton mixing effect. In our scenario we consider kinetic mixing is tiny and $Z'$ is long lived to escape detector at the LHC experiments; $Z'$ could decay into leptons via $L'$–L mixing discussed below and it is also considered to be tiny. Then we apply constrains from invisible decay of h for these decay branching ratio (BR) where the upper bound is taken to be $BR(h \rightarrow inv) <0.19$ [42, 43]. We obtain BRs of $h \rightarrow Z'Z'$ and $h \rightarrow HH$ modes in the case of $m_{Z' H}^2 \ll m_{h}^2$ as follows

$$\begin{aligned}&BR(h \rightarrow Z'Z') \simeq 4.6 \times 10^{-4} \left( \frac{s_\beta }{10^{-5}} \right) ^2 \left( \frac{1 \ \mathrm{GeV}}{v_\varphi } \right) ^2, \end{aligned}$$

(46)

$$\begin{aligned}&BR(h \rightarrow HH) \simeq 9.3 \times 10^{-4} \left( \frac{s_\beta }{10^{-5}} \right) ^2 \left( \frac{1 \ \mathrm{GeV}}{v_\varphi } \right) ^2, \end{aligned}$$

(47)

where we used the SM Higgs decay width $\Gamma _h \simeq 4.2$ MeV. Thus very small $s_\beta$ is required to avoid constraints from the SM Higgs decay.

Finally we discuss allowed region from the above constraints where we assume $v_\varphi \in [1, 5]$ GeV, $g'' \in [0.001,1]$, $m_{H} \in [0.1, 50]$ GeV, $m_{H_1} \in [150, 1500]$ GeV, $m_{H_2} \in [0.1, 1500]$ GeV, $m_{A_1} \in [150, 1500]$ GeV and $m_{A_2} \in [0.1, 1500]$ GeV. We also assume small mixing among inert scalar bosons as $\sin \alpha < 0.1$ and $\sin \gamma < 0.1$. We find that mass of H is light as $m_H \lesssim 20$ GeV due to small $v_\varphi$ and unitarity condition as shown in left plot of Fig. 1. Also $\Delta T$ constraint requires $|m_{H_1} - m_{A_1}| \lesssim 4$ GeV and $|m_{\eta ^{\pm }} - m_{H_1}| \lesssim 120$ GeV. The mixing between the SM Higgs and H should be suppressed to avoid constraint from invisible decay as shown in right plot of Fig. 1. The other scalar mass values are not constrained.

3 Neutrino mass and lepton flavor violation

In this section we formulate neutrino mass matrix, LFV processes and muon $g-2$. We then carry out numerical analysis to search for allowed parameter ranges taking into account experimental constraints.

3.1 Neutrino mass generation

We obtain neutral fermion mass terms at tree level, after scalar fields developing VEVs, such that

$$\begin{aligned} L&\supset {{\bar{\nu }}}_L M_D N'_R + {{\bar{N}}}'_L M_{L'} N'_R \nonumber \\&\quad + {\bar{N}}_R M_N N_L + {{\bar{N}}}^c_R M_R N_R + {{\bar{N}}}^c_L M_L N_L + h.c., \end{aligned}$$

(48)

where $M_D \equiv y_D v_\varphi /\sqrt{2}$ and $M_{L/R} \equiv y_{\varphi _{L/R}} v_\varphi /\sqrt{2}$. Then neutrino mass matrices at tree level are obtained as

$$\begin{aligned}&L_{M_\nu }^{\text {tree } } = \frac{1}{2} \left( \begin{array}{c} \bar{\nu }^c_L \\ {{\bar{N}}}'_R \\ {{\bar{N}}}'^c_L \end{array} \right) ^T \left( \begin{array}{ccc} 0 &{} M^*_D &{} 0 \\ M_D^\dagger &{} 0 &{} M_{L'}^\dagger \\ 0 &{} M_{L'}^* &{} 0 \end{array} \right) \left( \begin{array}{c} \nu _L \\ N'^c_R \\ N'_L \end{array} \right) + h.c. \nonumber \\ \end{aligned}$$

(49)

$$\begin{aligned}&L_{M_N}^{\text {tree}} = \frac{1}{2}\left( \begin{array}{c} {\bar{N}}^c_L \\ {\bar{N}}_R \end{array} \right) ^T \left( \begin{array}{cc} 2M_L &{} M^T_N \\ M_N &{} 2M^\dagger _R \\ \end{array} \right) \left( \begin{array}{c} N_L \\ N^c_R \end{array} \right) + h.c. \end{aligned}$$

(50)

The 22 and 33 components of the mass matrix in Eq. (49) are generated at one-loop level by diagrams in Fig. 2. Calculating the diagrams, we obtain one-loop contributions

$$\begin{aligned} \delta \mu _{22(33)}= & {} -\sum _\alpha \frac{{Y_R}_{2(3)\alpha }{Y^T_R}_{\alpha 2(3)}}{32 \pi ^2} \nonumber \\&\times M_{N_\alpha }\int _0^1 dx\Big [ \cos ^2 \alpha \log Q^2_{H_1} + \sin ^2\alpha \log Q^2_{H_2} \nonumber \\&- \cos ^2\gamma \log Q^2_{A_1} - \sin ^2\gamma \log Q^2_{A_2} \Big ] \end{aligned}$$

(51)

where $Q_{\phi }^2=x M_{N_\alpha }^2 + (1-x)m_{\phi }^2$ and $M_{N_\alpha }$ is the mass obtained after diagonalizing Eq. (50).

Fig. 2

The one-loop diagram generating Majorana mass terms of $N'_{L(R)}$

Then the inverse seesaw mass matrix becomes

$$\begin{aligned} L_{M_\nu }^{\text {tree + 1-loop} }= & {} \frac{1}{2} \left( \begin{array}{c} {{\bar{\nu }}}^c_L \\ {{\bar{N}}}'_R \\ {{\bar{N}}}'^c_L \end{array} \right) ^T \left( \begin{array}{ccc} 0 &{} M^*_D &{} 0 \\ M_D^\dagger &{} \delta \mu _{22} &{} M_{L'}^\dagger \\ 0 &{} M_{L'}^* &{} \delta \mu _{33} \end{array} \right) \left( \begin{array}{c} \nu _L \\ N'^c_R \\ N'_L \end{array} \right) \nonumber \\&+ h.c. \end{aligned}$$

(52)

Since our model has the following hierarchies $\delta \mu _{22(33)}, M_D<< M_{L'}$,⁵ the inverse seesaw mass matrix can be block diagonalized by unitary matrix $U_{\nu }$ to obtain the mass matrix of the light (active) neutrinos and heavy neutrinos as

$$\begin{aligned} M_{\nu }= & {} -\left( \begin{array}{cc} M^*_D &{} 0\\ \end{array}\right) \left( \begin{array}{cc} \delta \mu _{22} &{} M^\dagger _{L'} \\ M^*_{L'} &{} \delta \mu _{33} \\ \end{array} \right) ^{-1} \left( \begin{array}{c} M_D^\dagger \\ 0\\ \end{array} \right) \nonumber \\= & {} M_D^* {M^*_{L'}}^{-1} \delta m_{33} {M^\dagger _{L'}}^{-1} M_{D}^\dagger \end{aligned}$$

(53)

and

$$\begin{aligned} M_{N'}=\left( \begin{array}{cc} \delta \mu _{22} &{} M^\dagger _{L'} \\ M^*_{L'} &{} \delta \mu _{33}\\ \end{array} \right) \end{aligned}$$

(54)

such that

$$\begin{aligned} U_\nu ^T \left( \begin{array}{ccc} 0 &{} M^*_D &{} 0 \\ M_D^\dagger &{} \delta \mu _{22} &{} M_{L'}^\dagger \\ 0 &{} M_{L'}^* &{} \delta \mu _{33} \end{array} \right) U_\nu = \left( \begin{array}{cc} M_{\nu } &{} 0 \\ 0 &{} M_{N'}\\ \end{array}\right) \end{aligned}$$

(55)

where the unitary matrix $U_\nu$ is given by

$$\begin{aligned}&U_\nu = \left[ \begin{array}{cc} 1_{3\times 3}&\left( \begin{array}{cc} M_D^* &{} 0 \end{array}\right) \left( \begin{array}{cc} \delta \mu _{22} &{} M^\dagger _{L'}\\ M^*_L &{} \delta \mu _{33} \\ \end{array}\right) ^{-1} \\ - \left( \begin{array}{cc} \delta \mu _{22} &{} M^\dagger _{L'}\\ M^*_L &{} \delta \mu _{33} \\ \end{array}\right) ^{-1}\left( \begin{array}{c} M_D^\dagger \\ 0\\ \end{array}\right)&1_{6\times 6} \end{array} \right] \nonumber \\&\quad = \left[ \begin{array}{cc} 1_{3\times 3}&\begin{array}{cc} -M_D^*(M_{L'}^*)^{-1}\delta \mu _{33} (M_{L'}^\dagger )^{-1} &{} M_D^* (M_{L'}^*)^{-1} \end{array}\\ \begin{array}{c} (M_{L'}^*)^{-1}\delta \mu _{33} (M_{L'}^\dagger )^{-1}M_D^\dagger \\ -(M_{L'}^\dagger )^{-1}M_{D}^\dagger \end{array}&1_{6\times 6} \end{array}\right] \end{aligned}$$

(56)

Using the Cholesky factorization procedure, the symmetric matrix $N={M^*_{L'}}^{-1} \delta m_{33} {M^\dagger _{L'}}^{-1}$ can be written as $N=R R^T$ where R is an upper-right triangle matrix. The light neutrino mass matrix $M_{\nu }$ can be diagonalized by Pontecorvo–Maki–Nakagawa–Sakata (PMNS) matrix $U_{PMNS}$ to obtain the physical masses of the light neutrinos $D_\nu =\mathrm{diag}(m_{\nu _1}, m_{\nu _2}, m_{\nu _3})$

$$\begin{aligned} D_\nu = U_{PMNS}^T M_{\nu } U_{PMNS}. \end{aligned}$$

(57)

The standard parameterization of $U_{PMNS}$ in terms of mixing angles $\theta _{12},~\theta _{13},~\theta _{23}$, Dirac CP phase $\delta$, and Majorana phases $\alpha _1,~\alpha _2$ is

$$\begin{aligned}&U_{PMNS}\nonumber \\&=\left( \begin{array}{ccc} c_{12}c_{13}e^{i\alpha _1/2} &{} s_{12}c_{13}e^{i\alpha _2/2} &{} s_{13}e^{-i\delta }\\ (-s_{12}c_{23} -c_{12}s_{23}s_{13}e^{i\delta })e^{i\alpha _1/2} &{} (c_{12}c_{23}-s_{12}s_{23}s_{13}e^{i\delta })e^{i\alpha _2/2} &{} s_{23}c_{13}\\ (s_{12}s_{23}-c_{12}c_{23}s_{13}e^{i\delta })e^{i\alpha _1/2} &{} (-c_{12}s_{23}-s_{12}c_{23}s_{13}e^{i\delta })e^{i\alpha _2/2} &{} c_{23}c_{13}\\ \end{array} \right) \nonumber \\ \end{aligned}$$

(58)

where $s(c)_{ij} = \sin (\cos )\theta _{ij}$ and without any loss of generality we can consider $\theta _{ij} \in [0,\pi /2]$. Following Casas–Ibarra parameterization, we get

$$\begin{aligned} M^*_{D}= & {} U^*_{PMNS} D_{\nu }^{1/2} {{\mathcal {O}}} R^{-1} \nonumber \\ \text {which implies} ~~ y^*_{D}= & {} \frac{\sqrt{2}}{v_{\varphi }} U^*_{PMNS} D_{\nu }^{1/2} {{\mathcal {O}}} R^{-1} \end{aligned}$$

(59)

where ${{\mathcal {O}}}$ is an orthogonal matrix with three independent complex angles. In our numerical analysis we will consider the perturbative limit $|y_D| \le \sqrt{4\pi }$.

The mass matrix for the heavy neutrino $M_{N'}$ is diagonalized perturbatively by an unitary matrix $U_{N'}$ to obtain the masses $D_{N'}=\mathrm{diag}(m_{N'_1},m_{N'_2},m_{N'_3},m_{N'_4},m_{N'_5},m_{N'_6})$ of the six heavy Majorana neutrinos

$$\begin{aligned} D_{N'}= & {} U_{N'}^T M_{N'}U_{N'}\nonumber \\= & {} \left( \begin{array}{cccccc} m_{E'_i}-\delta {\mu _{22}}^{ii} - \delta \mu _{33}^{ii} &{} 0 \\ 0 &{} m_{E'_i}+ \delta {\mu _{22}}^{ii} + \delta \mu _{33}^{ii} \\ \end{array} \right) .\nonumber \\ \end{aligned}$$

(60)

In the limit $\delta _{\mu _{22(33)}} \rightarrow 0$ we can consider the heavy neutrinos to be Dirac type fermions. In our case we identify $N'$ flavour state as three heavy Dirac neutrinos.

3.2 Lepton flavor violation and muon $g-2$

The charged lepton mass term obtained from the Yukawa interaction is

$$\begin{aligned} L^{\text {mass}}_{\text {lepton}}={\bar{e}}_L M_{l'}e_R + \bar{E'}_L M_{L'}E'_R + {\bar{e}}_L M_D E'_R + h.c. \end{aligned}$$

(61)

which can be written in matrix form as

$$\begin{aligned} L^{\text {mass}}_{\text {lepton}} = \left( \begin{array}{cc} {\bar{e}}_L &{} \bar{E'}_L\\ \end{array} \right) \left( \begin{array}{cc} M_{l'} &{} M_D \\ 0 &{} M_{L'} \\ \end{array} \right) \left( \begin{array}{c} e_R\\ E'_R\\ \end{array} \right) + h.c. \end{aligned}$$

(62)

where $M_{l'} = y_l v/\sqrt{2}$. The mass matrix can be diagonalized by bi-unitary transformation

$$\begin{aligned} M^{\text {diag}}= U_L^\dagger \left( \begin{array}{cc} M_{l'} &{} M_D \\ 0 &{} M_{L'} \\ \end{array} \right) U_R \end{aligned}$$

(63)

where $U_L$ and $U_R$ are $6 \times 6$ unitary matrices such that the field transformations are

$$\begin{aligned} \left( \begin{array}{c} e_L \\ E'_L \end{array} \right) =U_L~l_{L}, \quad \left( \begin{array}{c} e_R \\ E'_R \end{array} \right) =U_R~l_{R}. \end{aligned}$$

(64)

Here we consider the elements of the submatrix $M_D$ to be very small in comparison to diagonal heavy lepton mass matrix $M_{L'}$; $M_D \ll M_{L'}$. The smallness of the submatrix $M_D$ is required by the constraints from lepton flavor violating (LFV) processes like $BR(l_i \rightarrow l_j \gamma )$. In such a case, unitary matrices $U_L$ and $U_R$ are approximated as

$$\begin{aligned} U_L \simeq \begin{pmatrix} {\varvec{1}} &{} {\varvec{0}} \\ - \left( M_D M_{L'}^{-1} \right) ^\dagger &{} {\varvec{1}} \end{pmatrix} \equiv \begin{pmatrix} {\varvec{1}} &{} {\varvec{0}} \\ - \delta U &{} {\varvec{1}} \end{pmatrix}, \quad U_R \simeq \begin{pmatrix} {\varvec{1}} &{} {\varvec{0}} \\ {\varvec{0}} &{} {\varvec{1}} \end{pmatrix},\nonumber \\ \end{aligned}$$

(65)

where ${\varvec{0}}$ and ${\varvec{1}}$ are $3\times 3$ zero and unity matrices respectively.

To estimate LFV BRs the relevant Yukawa interaction terms are

$$\begin{aligned} L&\supset&{\bar{\nu }}_i \frac{U^{*}_{PMNS_{ji}} y_D^{j\alpha }}{\sqrt{2}}N'_\alpha (\sin \beta h + \cos \beta H) \nonumber \\&+ \bar{N'_\alpha } \frac{y_D^{*j\alpha } U_{PMNS_{ji}}}{\sqrt{2}}P_L \nu _i (\sin \beta h + \cos \beta H) \nonumber \\&+ {\bar{e}}_i \frac{y_D^{i\alpha }}{\sqrt{2}}E'_\alpha (\sin \beta h\nonumber \\&+\cos \beta H) + \bar{E'_\alpha } \frac{y_D^{*i\alpha }}{\sqrt{2}}P_L e_i (\sin \beta h + \cos \beta H), \nonumber \\&+ \frac{i}{\sqrt{2}} y_D^{i \alpha } {{\bar{e}}}_{i} P_R E'_{\alpha } G_{Z'} + h.c., \end{aligned}$$

(66)

where summation over repeated index is implied. In this Lagrangian the light neutrinos $\nu _i$ are three Majorana neutrinos and $N'_\alpha$ are three heavy Dirac neutrinos. Note that we write interaction among $e^i$, $E^\alpha$ and NG boson $G_{Z'}$ to calculate muon $g-2$ in Feynman gauge. Flavor violating $Z'$ interaction terms are also given by

$$\begin{aligned} L&\supset -x g'' \left[ \delta U_{\alpha i} \bar{E}'_\alpha \gamma _\mu P_L e_i + (\delta U^\dagger )_{i \alpha } \bar{e}_i \gamma _\mu P_L E'_\alpha \right] Z'^\mu \nonumber \\&\quad + x g'' (\delta U^\dagger \delta U)_{ij} {{\bar{e}}}_i \gamma _\mu P_L e_j Z'^\mu , \end{aligned}$$

(67)

where $\delta U$ is $3\times 3$ matrix given in Eq. (65). From the interaction we obtain $Z' \rightarrow \ell _i {{\bar{\ell }}}_j$ decay width for vanishing kinetic mixing such that

$$\begin{aligned}&\Gamma _{Z' \rightarrow \ell _i {{\bar{\ell }}}_j}\nonumber \\&\quad = \frac{|x g'' (\delta U^\dagger \delta U)_{ij}|^2}{24 \pi } m_{Z'} \uplambda ^{\frac{1}{2}}\left( 1,\frac{m_{\ell _1}^2}{m_{Z'}^2},\frac{m_{\ell _2}^2}{m_{Z'}^2} \right) \nonumber \\&\qquad \times \left[ 1 - \frac{m_{\ell _1}^2}{m_{Z'}^2} - \frac{m_{\ell _2}^2}{m_{Z'}^2}\right. \nonumber \\&\qquad \left. + \sqrt{ \left( \uplambda \left( 1,\frac{m_{\ell _1}^2}{m_{Z'}^2},\frac{m_{\ell _2}^2}{m_{Z'}^2} \right) + \frac{4 m_{\ell _1}^2}{m_{Z'}^2} \right) \left( \uplambda \left( 1,\frac{m_{\ell _1}^2}{m_{Z'}^2},\frac{m_{\ell _2}^2}{m_{Z'}^2} \right) + \frac{4m_{\ell _2}^2}{m_{Z'}^2} \right) } \right] , \end{aligned}$$

(68)

where $\uplambda (1,x,y) \equiv 1 + x^2+ y^2 -2x -2y -2xy$. If light $Z'$ can only decay through this lepton mixing effect its lifetime will be long; for example $\tau _{Z'} \sim 5 \times 10^{-4} s$ for $|(\delta U^\dagger \delta U)_{22}| = 10^{-8}$, $g'' \sim 0.1$ and $m_{Z'} = 1$ GeV.

Considering the one-loop diagram with scalars h,  H and leptons $E'_\alpha$ running in the loop, we obtain [44, 45]

$$\begin{aligned} BR(l_i \rightarrow l_j \gamma )= & {} \frac{3(4\pi )^3\alpha _{em}}{4G_F^2}\Big [ |A^M_{ji}|^2 + |A^E_{ji}|^2 \Big ]\nonumber \\&\times BR(l_i \rightarrow l_j\nu _i{\bar{\nu }}_j) \end{aligned}$$

(69)

where

$$\begin{aligned} A^M_{ji}= & {} \frac{m_i + m_j}{16\pi ^2 m_i}\sin ^2\beta \sum _\alpha F(m_{E'_{\alpha }},m_h)y^{j\alpha }_D y^{*i\alpha }_D \nonumber \\&+ \frac{m_i + m_j}{16\pi ^2 m_i}\cos ^2\beta \sum _\alpha F(m_{E'_{\alpha }},m_H)y^{j\alpha }_Dy^{*i\alpha }_D \nonumber \\&- \frac{m_i + m_j}{8 \pi ^2 m_i} x^2 g''^2 \sum _\alpha {{\hat{F}}}(m_{E'_{\alpha }},m_{Z'}) \delta U_{\alpha j} (\delta U^\dagger )_{i \alpha } \nonumber \\&+ \frac{m_i + m_j}{16\pi ^2 m_i} \sum _\alpha F(m_{E'_{\alpha }},m_{Z'})y^{j\alpha }_Dy^{*i\alpha }_D \nonumber \\ A^E_{ji}= & {} \frac{m_i - m_j}{16\pi ^2 m_i}\sin ^2\beta \sum _\alpha F(m_{E'_{\alpha }},m_h)y^{j\alpha }_D y^{*i\alpha }_D \nonumber \\&+ \frac{m_i - m_j}{16\pi ^2 m_i}\cos ^2\beta \sum _\alpha F(m_{E'_{\alpha }},m_H) y^{j\alpha }_D y^{*i\alpha }_D \nonumber \\&- \frac{m_i - m_j}{8 \pi ^2 m_i} x^2 g''^2 \sum _\alpha {{\hat{F}}}(m_{E'_{\alpha }},m_{Z'}) \delta U_{\alpha j} (\delta U^\dagger )_{i \alpha } \nonumber \\&+ \frac{m_i - m_j}{16\pi ^2 m_i} \sum _\alpha F(m_{E'_{\alpha }},m_{Z'}) y^{j\alpha }_D y^{*i\alpha }_D, \end{aligned}$$

(70)

where the last two terms in the RHS are respectively contributions from $Z'$ (transverse mode) and NG boson, and loop integration factor is

$$\begin{aligned}&F(m_a,m_b) \nonumber \\&\quad = \frac{m_a^6 + 2 m_b^6 - 6 m_a^4 m_b^2 + 3 m_a^2 m_b^4 + 12 m_a^2 m_b^4 \log (\frac{m_a}{m_b})}{12(m_a^2 - m_b^2)^4}, \nonumber \\&{{\hat{F}}}(m_a,m_b) \nonumber \\&\quad = \frac{5 m_a^6 + 4 m_b^6 - 9 m_a^2 m_b^4 -12 m_a^2 m_b^2 (2 m_a^2 - m_b^2) \log (\frac{m_a}{m_b})}{12(m_a^2 - m_b^2)^4},\nonumber \\ \end{aligned}$$

(71)

Note that the second term of RHS in Eq. (67) also contributes to $\ell _i \rightarrow \ell _j \gamma$ process through $Z'$ loop. However this effect is suppressed since we take $\delta U = -(M_D/M_{L'})^\dagger$ to be very small. In our numerical calculation, we use explicit form of Eq. (69) to impose constraints from the LFV decay BRs.

In addition we consider $\mu$-e conversion through LFV interaction associated with Z boson induced by the lepton mixing effect [48]. The effective interaction for $\mu$-e conversion is written as

$$\begin{aligned}&L_{eff} = \sum _{q=u,d} (C^{LL}_q O_q^{LL} + C^{LR}_q O_q^{LR}) + (L \leftrightarrow R) +h.c. \ , \end{aligned}$$

(72)

$$\begin{aligned}&O_q^{XX} = ({{\bar{e}}} \gamma ^\mu P_X \mu )({{\bar{q}}} \gamma _\mu P_X q), \quad C_q^{XX} = \Gamma ^{ X}_{e \mu } \frac{1}{m_Z^2} \Gamma _{qq}^{X}, \end{aligned}$$

(73)

where $X = L, R$ and $\Gamma ^X_{qq(e\mu )}$ is Z couplings given by

$$\begin{aligned}&\Gamma _{uu}^{L} = - \frac{g}{c_W} \left( \frac{1}{2} -\frac{2}{3} s_W^2 \right) , \quad \Gamma _{uu}^R = \frac{2}{3} \frac{g s_W^2}{c_W}, \end{aligned}$$

(74)

$$\begin{aligned}&\Gamma _{dd}^{L} = - \frac{g}{c_W} \left( -\frac{1}{2} +\frac{1}{3} s_W^2 \right) , \quad \Gamma _{dd}^R = -\frac{1}{3} \frac{g s_W^2}{c_W}, \end{aligned}$$

(75)

$$\begin{aligned}&\Gamma _{e\mu }^{L} \simeq - \frac{g}{c_W} \left( -\frac{1}{2} + s_W^2 \right) \frac{ M_{D_{ea}} M^\dag _{D_{a\mu }}}{M^2_{L'_a}} , \quad \Gamma _{e\mu }^R \simeq 0, \end{aligned}$$

(76)

with $s_W(c_W) = \sin \theta _W (\cos \theta _W)$. Then $\mu$–e conversion ratio is given by [49,50,51]

$$\begin{aligned} \Gamma ^N_{\mu \rightarrow e}= & {} 4 m_\mu ^5 \left| \sum _{q=u,d} \left( C_q^{RL} + C_q^{RR} \right) \left( f_p^q V_N^p + f_n^q V_N^n \right) \right| ^2 \nonumber \\&+ (L \leftrightarrow R), \end{aligned}$$

(77)

where $f_p^u =2$, $f_n^u =1$, $f_p^d =1$, $f_n^d =2$ and $V_N^{p(n)}$ is the overlap integrals for nuclear N. For gold, we obtain $V_{Au}^{p(n)} = 0.0974(0.146)$ [51]. We need to normalize the conversion ratio by capture rate $\Gamma _{Au}^\mathrm{capt} = 8.7 \times 10^{-18}$ GeV [52], and the constraint is given by [53]

$$\begin{aligned} \frac{\Gamma ^{Au}_{\mu \rightarrow e}}{\Gamma _{Au}^{\mathrm{capt}}} < 7.0 \times 10^{-13}. \end{aligned}$$

(78)

We then obtain the constraint on lepton mixing such that

$$\begin{aligned} \frac{ M_{D_{ea}} M^\dag _{D_{a\mu }}}{M^2_{L'_a}} \lesssim 2.8 \times 10^{-7}. \end{aligned}$$

(79)

The constraint is weaker than $\mu \rightarrow e \gamma$ but it will be tested in future experiments with more precision. Hence the lepton mixing should be very small, and for a good approximation we can consider no lepton mixing when we consider SM lepton interactions.

Muon anomalous magnetic moment

It is known as hint of physics beyond the SM since observed value is deviated from the SM prediction. The deviation from the SM prediction $\Delta a_\mu$ is about 3.7$\sigma$ [59, 60];

$$\begin{aligned} \Delta a_\mu =(2.706\pm 0.726)\times 10^{-9}.~ \end{aligned}$$

(80)

It suggests that new physics would be needed. Our new contribution to the muon $g-2$ is obtained to be

$$\begin{aligned} \Delta a_\mu= & {} \frac{m_{\mu }^2}{8\pi ^2}\sin ^2\beta \sum _\alpha F(m_{E'_\alpha },m_h) y_D^{2\alpha } y_D^{*2\alpha } \nonumber \\&+ \frac{m_{\mu }^2}{8\pi ^2}\cos ^2\beta \sum _\alpha F(m_{E'_\alpha },m_H) y_D^{2\alpha } y_D^{*2\alpha } \nonumber \\&- \frac{m_{\mu }^2}{4 \pi ^2} x^2 g''^2 \sum _\alpha {{\hat{F}}} (m_{E'_\alpha },m_{Z'}) \delta U_{\alpha 2} (\delta U^\dagger )_{2 \alpha } \nonumber \\&+ \frac{m_{\mu }^2}{8\pi ^2} \sum _\alpha F(m_{E'_\alpha },m_{Z'}) y_D^{2\alpha } y_D^{*2\alpha }, \end{aligned}$$

(81)

where the last two terms in the RHS are respectively contributions from $Z'$ (transverse mode) and NG boson.

Flavor violating $\ell _i \rightarrow \ell _j Z'$ decay We also have this kind of LFV process induced by the second term of Eq. (67). For $m_{Z'} < m_i - m_j$, decay width is given by

$$\begin{aligned}&\Gamma (\ell _i \rightarrow \ell _j Z')\nonumber \\&\quad = x^2 g''^2 (\delta U^\dagger \delta U)^2_{ij}\frac{k}{8 \pi m_i} \nonumber \\&\qquad \times \left( \sqrt{k^2 + m_j^2} + \frac{1}{m_{Z'}^2} (m_i^2 - m_j^2 -m^2_{Z'})\sqrt{k^2 + m^2_{Z'}} \right) , \nonumber \\&k = \frac{m_i}{2} \uplambda ^{\frac{1}{2}}\left( 1, \frac{m_j^2}{m^2_i}, \frac{m^2_{Z'}}{m^2_i} \right) . \end{aligned}$$

(82)

Then $Z'$ decays into leptons through the mixing effect giving three charged lepton final states. Thus when these modes are kinematically allowed there are strong constraint for $\delta U = M_D^\dagger /M_{L'}$. For $m_{Z'} < m_\mu$, it is constrained by $BR(\mu \rightarrow 3 e) \lesssim 10^{-12}$ [54] which results in

$$\begin{aligned} \left| g'' \frac{ M_{D_{ea}} M^\dag _{D_{a\mu }}}{M^2_{L'_a}} \right| \lesssim 9 \times 10^{-15} \end{aligned}$$

(83)

where we used $m_{Z'} = 0.05$ GeV as a reference value. For $m_\mu< m_{Z'} < m_\tau$, it is constrained by , for example $BR(\tau \rightarrow 3 \mu ) \lesssim 2 \times 10^{- 8}$ [55], which results in

$$\begin{aligned} \left| g'' \frac{ M_{D_{\mu a}} M^\dag _{D_{a\tau }}}{M^2_{L'_a}} \right| \lesssim 4 \times 10^{-10} \end{aligned}$$

(84)

where we used $m_{Z'} = 0.5$ GeV as a reference value. Thus we find that strong constraints are imposed from $\ell _i \rightarrow \ell _j Z'$ when these decay modes are kinematically allowed. In our analysis, we consider $Z' > m_{\tau }$ to avoid these strong LFV constraints.

3.3 Leptonic Z boson decays

The $Z_\mu$ to the leptonic (charged and uncharged) interaction terms in the limit $\epsilon \rightarrow 0$ i.e. $(\sin \theta \rightarrow 0)$ is given by [56,57,58]

$$\begin{aligned}&L \supset \frac{g}{c_W}\Big [ {\bar{l}}\gamma _\mu \Big (\frac{P_L}{2} - s_W^2\Big )l + \bar{E'} \gamma _\mu \Big (\frac{1}{2} -s_W^2\Big )E' \Big ]Z^\mu \nonumber \\&\qquad - \frac{g}{c_W} \Big [ {\bar{\nu }} \gamma _\mu \frac{P_L}{2}\nu + \bar{N'}\gamma _\mu \frac{1}{2}N' \Big ]Z^\mu \end{aligned}$$

(85)

where the sine of the Weinberg angle is $s_W \sim 0.23$. The equation is written in the mass basis where $\nu$ are the three light Majorana neutrinos and $N'$ are three heavy Dirac neutrinos. The deviation of Z boson branching ratios from SM prediction up to 1-loop level is given by

$$\begin{aligned}&\Delta \mathrm{BR} (Z \rightarrow f_{i}\bar{f_{j}})\nonumber \\&\quad = \frac{\Gamma (Z \rightarrow f_{i}\bar{f_{j}})_{\mathrm{SM+ New}} - \Gamma (Z \rightarrow f_{i}\bar{f_{j}})_{\mathrm{SM}}}{\Gamma ^{tot}_Z} \nonumber \\&\quad \approx \left\{ ~\begin{array}{cc} \frac{\Gamma (Z\rightarrow f_i{{\bar{f}}}_j)_{\mathrm{SM+New}} - \Gamma (Z\rightarrow f_i{{\bar{f}}}_j)_{\mathrm{SM}}}{\Gamma ^{\text {tot}}_Z} &{} (i =j)\\ \frac{ \Gamma (Z\rightarrow f_i{{\bar{f}}}_j)_{\mathrm{New}}}{\Gamma ^{\text {tot}}_Z} &{} (i \ne j)\\ \end{array}\right. , \end{aligned}$$

(86)

where $\Gamma _Z^{\mathrm{{tot}}}=2.4952 \pm 0.0023$ [8].

Then our new contributions for charged-lepton and neutrino final modes are respectively given by

$$\begin{aligned} \Delta \mathrm{BR}(Z \rightarrow l_i \bar{l_j})= & {} \frac{m_Z g^2}{12 \pi \Gamma _Z^{\mathrm{{tot}}} c^2_W} \Big [ \frac{|B^l_{ij}|^2}{2} \nonumber \\&- s^2_W \delta _{ij} Re [B^{l*}_{ij}] - \Big ( - \frac{s_W^2}{2}+ \frac{1}{8}\Big )\delta _{ij}\Big ] ,\nonumber \\ \Delta \mathrm{BR}(Z \rightarrow \nu _i \bar{\nu _j})= & {} \frac{m_Z g^2}{24 \pi \Gamma _Z^{\mathrm{{tot}}} c^2_W} \Big [ |B^\nu _{ij}|^2 -\frac{\delta _{ij}}{4} \Big ], \end{aligned}$$

(87)

with

$$\begin{aligned} B^l_{ij}= & {} \frac{\delta _{ij}}{2} - \frac{1}{2}\sum _{\alpha } \frac{y_D^{i\alpha }y_D^{*j\alpha }}{(4\pi )^2} G^l(m_{E'_\alpha },m_{h})\sin ^2\beta \nonumber \\&- \frac{1}{2}\sum _\alpha \frac{y_D^{i\alpha }y_D^{*j\alpha }}{(4\pi )^2}G^l(m_{E'_\alpha },m_{H})\cos ^2\beta , \nonumber \\ B^\nu _{ij}= & {} \frac{\delta _{ij}}{2} + \frac{1}{2}\sum _{\alpha ,k,l} \frac{U^*_{PMNS_{ki}} U_{PMNS_{lj}} y_D^{k\alpha }y_D^{*l\alpha }}{(4\pi )^2}\nonumber \\&\times G^{\nu }(m_{N'_\alpha },m_{h})\sin ^2\beta \nonumber \\&+ \frac{1}{2}\sum _{\alpha ,k,l} \frac{U^*_{PMNS_{ki}} U_{PMNS_{lj}} y_D^{k\alpha }y_D^{*l\alpha }}{(4\pi )^2}\nonumber \\&\times G^{\nu }(m_{N'_\alpha },m_{H})\cos ^2\beta , \nonumber \\ G^l(m_a,m_b)= & {} -\Big ( -\frac{1}{2} + s^2_W \Big )^2 H_1(m_a,m_b) \nonumber \\&+ \Big ( -\frac{1}{2} + s^2_W \Big ) H_2(m_a,m_b) \nonumber \\ G^{\nu }(m_a,m_b)= & {} -\frac{1}{2}H_1(m_a,m_b) - \frac{1}{2}H_2(m_a,m_b) , \nonumber \\ H_1(m_a,m_b)= & {} \frac{\begin{array}{ll} &{}m_b^4 - 4m_a^2 m_b^2 + 3 m_a^4-4m_b^2(m_b^2 - 2 m_a^2)\\ &{}\quad \times \log (m_b) - 4 m_a^4 \log (m_a) \end{array}}{4(m_a^2 -m_b^2)^2} ,\nonumber \\ H_2(m_a,m_b)= & {} m_a^2 \Big [ \frac{m_a^2 - m_b^2 + 2 m_b^2 \log (\frac{m_b}{m_a})}{(m_a^2 + m_b^2)^2} \Big ]. \end{aligned}$$

(88)

3.4 Numerical analysis

In our numerical analysis we consider the Yukawa couplings $|y_D^{1\alpha }|\in [10^{-8},10^{-3}]$, $|y_D^{2\alpha }|\in [10^{-3},1.0]$ and $|y_D^{3\alpha }|\in [10^{-6},10^{-1}]$, $m_H \in [1,10]$ GeV, $m_{E'_\alpha } \in [100,1500]$ GeV, $m_{N'_\alpha } \in [100,1500]$ GeV and $\sin \beta \sim 10^{-4}$. The allowed parameter spaces are obtained which satisfy the neutrino data of recent global fit by NuFIT 5.0 [61]. For simplicity we consider normal ordering and the Majorana phases as free parameters in the range $[0,2 \pi ]$.

$$\begin{aligned}&s^2_{12} \in [0.269,0.343], \quad s^2_{23} \in [0.415,0.616],\nonumber \\&s^2_{13}\in [0.02032,0.02410],\quad \delta \in \left[ \frac{2}{3}\pi , 2\pi \right] , \nonumber \\&\Delta m_{21}^2 \in [6.82,8.04]\times 10^{-5}~ \mathrm{eV}\quad \text {and} ~~ \nonumber \\&\Delta m_{31}^2 \in [2.435,2.598]\times 10^{-3}~ \mathrm{eV}. \end{aligned}$$

(89)

Fig. 3

Allowed region of $|y_D^{21}|$, $|y_D^{22}|$ and $|y_D^{23}|$

Fig. 4

Allowed region for LFVs and the correlation to muon $g-2$. The left one shows muon $g-2$ in terms of BR($\mu \rightarrow e\gamma )$, while the right one BR($\tau \rightarrow e\gamma )$ in terms of BR($\tau \rightarrow \mu \gamma )$

Under these parameter space, we have randomly selected and show plots to satisfy all the constraints as discussed above. Figure 3 shows the allowed spaces for relevant components of the Dirac Yukawa couplings $|y^{{21}}_{D}|$, $|y^{{22}}_{D}|$ and $|y^{{23}}_{D}|$, which are characterized by Casas–Ibarra parametrization and restricted by leptonic Z boson decays as well as LFVs. It implies that these components are 0.6 at most that would be reasonable in our model.

Fig. 5

Allowed regions of parameter space satisfying the observed DM relic density constraint. Left: The DM candidate is $N_1$ and right: the DM candidate is $H_2$

Figure 4 shows the possibly allowed regions for muon $g-2$, BR($\mu \rightarrow e\gamma )$ in the left figure, and BR($\tau \rightarrow e\gamma )$ and BR($\tau \rightarrow \mu \gamma )$ in the right one. In the left one, new contribution to the muon $g-2$ reaches at $3\times 10^{-10}$ at around the upper bound on BR($\mu \rightarrow e\gamma )=4.2\times 10^{-13}$ [46, 47]. Even though the scale of muon $g-2$ is smaller than the expected value in Eq. (80) by 1 magnitude, it will still be verifiable to be tested in near future.

In the right one, BR($\tau \rightarrow e\gamma )$ is much smaller than the current upper bound while BR($\tau \rightarrow \mu \gamma )$ reaches the current upper bound. Therefore, BR($\tau \rightarrow \mu \gamma )$ could also be tested in near future.

4 Dark matter physics

For the dark matter (DM) analysis we consider two scenarios, fermionic DM scenario and the scalar DM scenario. In the fermionic DM scenario, we consider the lightest of the $Z_2$ odd Majorana fermions $N_{1,\ldots ,6}$ obtained by diagonalizing Eq. (50) as the DM candidate. For simplicity we consider the mass ordering $m_{N_1}<m_{N_2}<m_{N_3}<m_{N_4}<m_{N_5}<m_{N_6}$ and therefore the lightest DM candidate is $N_1$. For numerical analysis we consider $m_{N_1} \in [1,10]$ GeV, $m_{Z'} \in [2,10]$ GeV, the gauge kinetic mixing parameter $\epsilon \sim 10^{-4}$ and the $U(1)_X$ charge, $x=1$, $m_H \in [1,10]$ GeV, $v_{\varphi } \in [1,5]$ GeV and $\sin \beta \sim 10^{-4}$. The relic density is computed using the public code Micromegas-5.2.4 [64] implementing our model. The left panel of Fig. 5 shows the allowed region of $m_{Z'}-m_{N_1}$ space with $g''$ in colour bar satisfying the observed relic density; $0.11< \Omega h^2 < 0.13$. Above the dashed line, the dominant contribution to $\langle \sigma v \rangle$ comes from the $N_1 N_1 \rightarrow Z' Z'$ channel. Below the dashed line, the dominant contribution to $\langle \sigma v \rangle$ comes from $N_1 N_1 \rightarrow Z'\rightarrow f_{SM} {\bar{f}}_{SM}$ processes. Along the dashed line slightly below it, the cross sections are enhanced due to resonance which leads to a sharp fall in the relic density [63].

For the scalar DM scenario, we have two candidates; $H_1$ or $H_2$, where one can select the lighter one between CP even and CP odd bosons. When the mixing is not so large, $H_1$ is $\eta _R$ dominant and $H_2$ is $\chi _R$ one. In case of $H_1$, the detailed behavior of DM has been known unless the Yukawa interactions $y_{L,R}$ are dominant. Below 100 GeV, a solution has at the pole of half of DM Higgs mass $\sim 63$ GeV. Above 100 GeV, another solution is at around 534 GeV that is induced by gauge interactions in kinetic terms. In details, see, e.g., Ref. [62]. However since our focus region is that the DM mass is less than 1 GeV, these solution is not valid. Even though we have new channels via $H_1-A_1-Z'$ interactions, the DM mass wouldn’t be below 1 GeV because of the electroweak precision test. Thus, we consider $H_2$ as the lightest $Z_2$ odd particle. For numerical analysis we consider $m_{H_2} \in [1,10]$ GeV, $m_{Z'} \in [2,10]$ GeV and the remaining parameters same as above. The right panel of Fig. 5 shows the allowed region of $m_{Z'}-m_{H_2}$ space with $g''$ in color bar satisfying the observed relic density. The dominant contribution to the relic density comes from $H_2 H_2 \rightarrow Z' \rightarrow f_{SM} {\bar{f}}_{SM}$ channel.

5 Summary and discussion

We have proposed an inverse seesaw scenario under hidden U(1) gauge symmetry, having natural hierarchies among mass scales of neutral fermions. The hierarchies are derived from theory and experimental constraints. The theoretical aspect tells us $\delta \mu _{22,33}\ll M_{L'}$ since $\delta \mu _{22,33}$ is induces at one-loop level, while $M_{L'}$ is bare mass. The experimental side requests that $M_D\ll M_{L'}$ since $M_D$ is highly suppressed by the LFVs such as $\mu \rightarrow e\gamma$. In order to induce small $\delta \mu _{22,33}$, we introduce exotic fermions and bosons that provide us additional intriguing phenomenologies such as muon anomalous magnetic moment, dark matter candidate as well as LFVs and leptonic Z decays. We have analyzed these phenomenologies including neutrino oscillation data numerically, and shown allowed region. Here, Dirac Yukawa coupling $y_D$ possesses the information of neutrino data derived by modified Casas–Ibarra parametrization. We have found that our new contribution to the muon $g-2$ is about $3\times 10^{-10}$ which is smaller than 1 order magnitude, but it would still be verifiable in near future experiments. BR$(\mu \rightarrow e\gamma )$ and BR$(\tau \rightarrow \mu \gamma )$ would be tested soon since these upper bounds reaches at the current experimental bounds. Finally, we have discussed the DM candidate in both the cases of fermion and boson, where we have focussed on rather lighter range that is equal or less than 1 GeV. The dominant contributions originate from interactions of hidden gauge sector, and we have found allowed ranges for both the case as can be seen in Fig. 5.

In our scenario, we consider light $Z'$ and extra scalar H since it is preferred to provide sizable muon $g-2$ relaxing tension between experimental data and theoretical prediction. In such a case the SM Higgs boson can decay into HH and $Z'Z'$ mode where H further decays into $Z'Z'$ and $Z'$ decays into charged leptons via lepton mixing effect. Thus we would have multi-lepton signals from rare Higgs decay as $h \rightarrow Z' Z' \rightarrow 4 \ell$ and/or $h \rightarrow HH \rightarrow 4Z' \rightarrow 8 \ell$ which can be distinguishable signature of our scenario. Interestingly, leptonic $Z'$ decays includes LFV modes. We need detailed simulation analysis to estimate discovery potential of these signals since H and $Z'$ are light and some final state particles are collimated. Therefore we leave the analysis in future work.

Before closing our paper, we will briefly mention collider phenomenologies. In our scenario, extra charged particles $\eta ^\pm$ and $E^\pm$ can be produced by electroweak interactions at the LHC. Inert charged scalar $\eta ^\pm$ dominantly decays into $W^{\pm (*)} H_{1,2}$ assuming $E^\pm$ has heavier mass; $H_{1,2}$ would be DM or it decays into the states including DM with other particles. If $\eta ^\pm$ decays into $W^{\pm (*)} DM$ the situation is the same as inert Higgs doublet scenario, and it is well studied, e.g. in Ref. [65]. Exotic lepton $E^\pm$ can decay into $\ell ^\pm H$ and $\eta ^\pm N_i$ where H decays into $Z'Z'$ and $\eta ^\pm$ decays as described above. Thus new signals from these exotic charged particles provide us complicated final states with many particles and analysis of them is beyond the scope of this paper. Analysis of collider signals would be given in future work.

Appendix A: Couplings in scalar potential

The parameters in the scalar potential can be written by scalar masses and mixings:

$$\begin{aligned} \uplambda _\Phi= & {} \frac{m^2_h\cos ^2\beta + m_H^2\sin ^2\beta }{2v^2} \nonumber \\ \uplambda _\varphi= & {} \frac{m_h^2\sin ^2\beta + m_H^2\cos ^2\beta }{2v_\varphi ^2} \nonumber \\ \uplambda _{\Phi \varphi }= & {} -\frac{(m_H^2-m_h^2)\cos \beta \sin \beta }{v v_\varphi } \nonumber \\ \uplambda _0= & {} -\frac{2(m^2_{H_2} - m^2_{H_1})\cos \alpha \sin \alpha }{v v_\varphi } \nonumber \\= & {} \frac{2(m^2_{A_2} - m^2_{A_1})\cos \gamma \sin \gamma }{v v_\varphi } \nonumber \\ {\tilde{\uplambda }}_{\Phi \eta }= & {} -\frac{2(m_{\eta ^\pm }^2 -m_{H_1}^2\cos ^2\alpha - m_{H_2}^2\sin ^2\alpha )}{v^2} \nonumber \\= & {} -\frac{2(m_{\eta ^\pm }^2 -m_{A_1}^2\cos ^2\gamma - m_{A_2}^2\sin ^2\gamma )}{v^2} \nonumber \\ m_{\chi ^\prime }^2= & {} m_{A_1}^2 \sin ^2\gamma + m_{A_2}^2 \cos ^2\gamma \nonumber \\ m_{\chi ^{\prime \prime }}^2= & {} m_{H_1}^2 \sin ^2\alpha + m_{H_2}^2 \cos ^2\alpha \nonumber \\ \mu= & {} \frac{m_{\chi ^{\prime \prime }}^2 - m_{\chi ^\prime }^2}{2\sqrt{2}v_\varphi } \nonumber \\ m_{\chi }^2= & {} \frac{1}{2}\Big [ m_{\chi ^\prime }^2 + m_{\chi ^{\prime \prime }}^2 -v^2\uplambda _{H\chi } -v_{\varphi }^2\uplambda _{\varphi \chi } \Big ]. \end{aligned}$$

(A1)

Thus the independent parameters are $m_h, m_H, m_{H_1}, m_{H_2}, m_{A_1}, m_{A_2}, m_{\eta ^\pm }, v, v_{\varphi }, \cos \beta , \uplambda _{\Phi \chi },\uplambda _{\varphi \chi },\uplambda _{\Phi \eta }, \uplambda _{\eta \varphi }, \uplambda _{\eta \chi }, \uplambda _{\eta }, \uplambda _{\chi }$.