Bounds on LFV Higgs decays in a vector-like lepton model and searching for doubly charged leptons at the LHC

Abstract

The Higgs-portal lepton flavor violation is studied in a vector-like lepton model. To avoid the constraints from rare $Z\rightarrow \ell ^\pm _i \ell ^\mp _j$ decays, we introduce two triplet vector-like leptons, $(1,3)_{-1}$ and $(1,3)_{0}$. The resultant branching ratio for $h\rightarrow \mu \tau$ can be up to $10^{-4}$ when the constraints from the invisible Z decays are applied. As a result, the signal strength for the $\tau \tau$ channel has a $12~\%$ deviation from the standard model prediction, while the muon $g-2$ is two orders of magnitude smaller than the data, and $\mathrm{BR}(\tau \rightarrow \mu \gamma )$ is of the order of $10^{-12}$. A predicted doubly charged lepton in pp collisions at $\sqrt{s}=13$ TeV is analyzed, and it is found that the interesting production channels are $pp\rightarrow (\Psi ^{-\,-}_{1} \Psi ^{++}_1, \Psi ^{\pm \pm }_1 \Psi ^\mp _1)$. Both single and pair production cross sections of $\Psi ^{++}_1$ are comparable, and can be a few hundred fb. The main decay channels for the doubly charged lepton are $\Psi ^{\pm \pm } \rightarrow \ell ^\pm W^\pm$, and for the heavy singly charged lepton they are $\Psi ^\pm _1 \rightarrow \nu W^\pm , \ell ^\pm Z$. The numerical analysis is carried out with regard to 13 TeV LHC with 100 fb$^{-1}$ luminosity.

Due to a number of unsolved issues, such as the origin of neutrino mass, dark matter (DM), and matter–antimatter asymmetry, it is believed that the standard model (SM) of particle physics is an effective theory at the electroweak scale. Since rare decays are important in the development of new physics, the loop-induced flavor-changing neutral current (FCNC) processes are generally used to examine the SM. However, most hadronic processes involve very uncertain non-perturbative quantum chromodynamic (QCD) effects, and thus, even if new physics exists, it is not easy to distinguish this from the SM results due to QCD uncertainty.

Leptons do not carry a color charge, and QCD uncertainty is thus much smaller in this case. However, due to the Glashow–Iliopoulos–Maiani (GIM) mechanism, lepton FCNC processes in the SM (e.g., $\mu \rightarrow e \gamma$ and $\tau \rightarrow (e,\mu )\gamma$) are highly suppressed; if any signal of lepton flavor violation (LFV) is observed, it is certainly strong evidence for new physics. It is thus important to search for new physics through the lepton sector [1–3].

With the discovery of the SM Higgs in the ATLAS [4] and CMS [5] experiments, we are moving toward better understanding the process of electroweak symmetry breaking (EWSB) through the spontaneous symmetry breaking (SSB) mechanism in the scalar sector. The next mission for the High Luminosity Large Hadron Collider (LHC) is to explore not only the detailed properties of the SM Higgs, but also the new physics effects.

Since the SM Higgs has been discovered, it is of interest to search for new physics through the Higgs portal. For instance, an excess of events with a significance of $2.4\sigma$ in $h\rightarrow \mu \tau$ decay was reported by CMS in pp collisions at $\sqrt{s}=8$ TeV, where the branching ratio (BR) with the best fit is given by [6]:

$$\begin{aligned} \mathrm{BR}(h\rightarrow \mu \tau )=(0.84 ^{+0.39}_{-0.37})\% \ \ [\text {CMS}]. \end{aligned}$$

(1)

ATLAS also reported the same measurement and found no significant excess, where the best fit is [7]:

$$\begin{aligned} \mathrm{BR}(h\rightarrow \mu \tau )=(0.77 \pm 0.62)~\% \ \ [\text {ATLAS}]. \end{aligned}$$

(2)

Although the measurements of $\mathrm{BR}(h\rightarrow \mu \tau )$ are not conclusive yet, inspired by the Higgs-portal events, a number of the possible new physics effects that could explain the large BR for $h\rightarrow \mu \tau$ decay have been studied [8–34]. In this study, we explore the LFV in the framework of a vector-like lepton model.

FCNCs are quite a common phenomenon in the quark sector, such as neutral meson oscillations and $b\rightarrow s \gamma$. However, with the exceptions of the neutrino oscillations we have no concrete and solid signals to verify the LFV in the lepton sector, thus limiting our knowledge about this. In this context, the measurements from ATLAS and CMS of the Higgs-portal LFV provide a good chance to better understand the lepton sector. Following the hint of the SM with regard to whether the Higgs couplings to the fermions appear in the Yukawa sector, a possible minimal extension of the SM for Higgs-mediated LFV is to include exotic heavy leptons or to add a new Higgs doublet without imposing any symmetry [30]. In this work, we study the implications of adding heavy leptons. In order to avoid the gauge anomaly, we investigate the model with vector-like leptons (VLLs).

To achieve mixing with the SM leptons, the representations of VLL under $SU(2)\times U(1)_Y$ gauge symmetry can be singlet, doublet [35–43], and triplet [44]. The VLLs from a composite model are discussed in earlier work [45, 46]. In order to avoid the constraints from rare $Z\rightarrow \ell ^\pm _i \ell ^\mp _j$ decays, we study the triplet representations $(1,3)_{-1}$ and $(1,3)_{0}$ with hypercharges $Y=-1$ and $Y=0$, respectively. The new Yukawa couplings are thus written

$$\begin{aligned} -\mathcal{L}_{Y}= & {} \bar{L} \mathbf{Y}_{1} \Psi _{1R} H + \bar{L} \mathbf{Y}_{2} \Psi _{2R} \tilde{H} + m_{\Psi _1} Tr\bar{\Psi }_{1L} \Psi _{1R} \nonumber \\&+\, m_{\Psi _2} Tr\bar{\Psi }_{2 L} \Psi _{2R} + H.c., \end{aligned}$$

(3)

where we have suppressed the flavor indices; H is the SM Higgs doublet, $\tilde{H}= i\tau _2 H^*$, the neutral component of Higgs field is $H^0 = (v + h)/\sqrt{2}$, and the representations of two VLLs are:

$$\begin{aligned}&\Psi _{1} = \left( \begin{array}{cc} \Psi ^-_1/\sqrt{2} &{} \Psi ^0_1 \\ \Psi ^{-\,-}_1 &{} -\Psi ^-_1/\sqrt{2} \end{array} \right) ,\nonumber \\&\Psi _{2} = \frac{1}{\sqrt{2}} \left( \begin{array}{cc} \Psi ^0_{2}/\sqrt{2} &{} \Psi ^{+}_{2} \\ \Psi ^{-}_{2} &{} -\Psi ^0_{2}/\sqrt{2} \end{array} \right) \end{aligned}$$

(4)

with $\Psi ^+_{2} = C\bar{\Psi }^{-}_{2}$ and $\Psi _{2}^{0} = C \bar{\Psi }_{2}^{0}$. Since $\Psi _2$ is a real representation of SU(2), the factor of $1/\sqrt{2}$ in $\Psi _{2}$ is to obtain the correct mass term for Majorana fermion $\Psi ^0_2$. Due to the new Yukawa terms of $\mathbf{Y}_{1,2}$, the heavy neutral and charged leptons mix with the SM leptons; after EWSB, the lepton mass matrices become $5\times 5$ matrices and are expressed by

$$\begin{aligned} M_{\ell } \!= \!\left( \begin{array}{cc} \mathbf{m}_{\ell } &{} \mathbf{Y}^\ell v \\ 0 &{} \mathbf{m}_{\Psi } \end{array} \right) , \quad M_{\nu } \!=\! \left( \begin{array}{cc} \mathbf{m}_{\nu } &{} \mathbf{Y}^\nu v \\ 0 &{} \mathbf{m}_{\Psi } \end{array} \right) , \end{aligned}$$

(5)

where we have chosen the basis such that the SM leptons are in diagonalized states, $\mathbf{m}_{\ell }$ is the SM charged lepton mass matrix, $\mathbf{m}_{ \Psi }=$diag$(m_{\Psi _1}, m_{\Psi _2})$, and

$$\begin{aligned} \mathbf{Y}^\ell = \frac{1}{2} \left( \begin{array}{cc} -Y_{11} &{} Y_{21} \\ - Y_{12} &{} Y_{22} \\ -Y_{13} &{} Y_{23} \end{array} \right) , \quad \mathbf{Y}^\nu = \sqrt{2} \left( \begin{array}{cc} Y_{11} &{} Y_{21}/2 \\ Y_{12} &{} Y_{22}/2 \\ Y_{13}&{} Y_{23}/2 \end{array} \right) . \end{aligned}$$

(6)

We note that the elements of $\mathbf{Y}^\chi$ should be read as $Y_{ij}= (\mathbf{Y}_i)_j$, where the index $i=1,2$ distinguishes the Yukawa couplings of the different VLLs and the index $j=1,2,3$ stands for the flavors of the SM leptons. Since the origin of neutrino mass has not been concluded yet and is still model dependent, we directly put the Majorana type of neutrino mass term to the Yukawa sector. Since the detailed effects of neutrino physics are irrelevant to this study, we do not further pursue issues related to this and $\mathbf{m}_\nu = 0$.

To diagonalize $M_{\ell }$ and $M_{\nu }$, we introduce the unitary matrices $V^{\chi }_{R,L}$ with $\chi =\ell , \nu$ so that $M^\mathrm{dia}_\chi = V^\chi _L M_{\chi } V^{\chi \dagger }_R$. The information of $V^\chi _L$ and $V^\chi _R$ can be obtained through $M_\chi M^\dagger _\chi$ and $M^\dagger _\chi M_\chi$, respectively. According to Eq. (5), it can easily be found that the flavor mixings between heavy and light leptons in $V^\chi _{R}$ are proportional to the lepton masses. Since the neutrino masses are small, it is a good approximation to take $V^\nu _R \approx 1$. If we further set $m_e = m_\mu =0$ in the phenomenological analysis, only $\tau$-related processes have significant contributions. Unlike $V^\chi _R$, the off-diagonal elements in flavor-mixing matrices $V^\chi _L$ are associated with $\mathbf{Y}_{1,2} v/\mathbf{m}_\Psi$; in principle, the mixing effects can be of the order of 0.1. In this study, we examine these effects on $h\rightarrow \tau \mu$ and rare tau related decays. To be more specific, we parametrize the unitary matrices in terms of $\mathbf{Y}_{1,2}$ as:

$$\begin{aligned}&V^\chi _{L} \approx \left( \begin{array}{cc} \mathbbm {1}_{3\times 3} - \pmb \varepsilon ^\chi _L \pmb \varepsilon ^{\chi \dagger }_L / 2 &{} - \pmb \varepsilon ^\chi _L \\ \pmb \varepsilon ^{\chi \dagger }_L &{} \mathbbm {1}_{3\times 3} - \pmb \varepsilon ^{\chi \dagger }_L \pmb \varepsilon ^{\chi }_L/2 \end{array} \right) ,\nonumber \\&V^\ell _{R} \approx \left( \begin{array}{cc} \mathbbm {1}_{3\times 3} &{} - \pmb \varepsilon ^\ell _R \\ \pmb \varepsilon ^{\ell \dagger }_R &{} \mathbbm {1}_{3\times 3} \end{array} \right) , \end{aligned}$$

(7)

where $V^\nu _R \approx 1$ is implied, $\pmb \varepsilon ^\chi _L \approx v \mathbf{Y}^\chi /\mathbf{m}_\Psi$, and $\pmb \varepsilon ^\ell _R \approx v \mathbf{m}^\dagger _{\ell } \mathbf{Y}^\ell /\mathbf{m}^2_{\Psi }$.

Combining the SM Higgs couplings and the new Yukawa couplings of Eq. (3), the Higgs couplings to all singly charged leptons are given by

$$\begin{aligned} -\mathcal{L}_{h\ell ' \ell '}&= h \bar{\ell }'_L V^\ell _L \left( \begin{array}{cc} \mathbf{m}_\ell /v &{} Y^\ell \\ 0 &{} 0 \end{array} \right) V^{\ell \dagger }_{R} \ell '_R+ H.c., \end{aligned}$$

(8)

where $\ell '^T = (e, \mu , \tau , \tau ', \tau '')$ is the state of a physical charged lepton in lepton flavor space. We use the notations of $\tau '$ and $\tau ''$ to denote the heavy charged VLLs. Using the parametrization of Eq. (7), the Higgs couplings to the SM leptons can be formulated by

$$\begin{aligned} -\mathcal{L}_{h \ell \ell }&= C^{h}_{ij} \bar{\ell }_{iL} \ell _{jR} h + H.c., \nonumber \\ C^{h}_{ij}&= \frac{m_{\ell j}}{v} \left[ \delta _{ij} - \frac{3}{8} \left( \frac{v^2 Y_{1i} Y_{1j}}{m^2_{\Psi _1}} +\frac{v^2 Y_{2i} Y_{2j}}{m^2_{\Psi _2}}\right) \right] . \end{aligned}$$

(9)

If we set $m_e=m_\mu =0$, it is clear that in addition to the $h\tau \tau$ coupling being modified, the tree-level flavor-changing h–$\tau$–$\mu$ and h–$\tau$–e couplings are induced, and the couplings are proportional to $m_\tau /v\approx 7.2\times 10^{-3}$. In order to study the VLL contributions to $h\rightarrow \gamma \gamma$, the couplings for $h\tau '\tau '$ and $h\tau '' \tau ''$ are expressed as:

$$\begin{aligned} -\mathcal{L}_{h\Psi \Psi }&= \frac{v \sum _{i} Y^2_{1i}}{2 m_{\Psi _1}} h \tau ' \tau ' + \frac{v \sum _{i} Y^2_{2i} }{2 m_{\Psi _2}} h \tau '' \tau ''. \end{aligned}$$

(10)

Due to the mixture between VLLs and the SM leptons, the same effects also influence the gauge couplings of Z and W to the SM leptons. To understand the modifications, we discuss the gauge interactions below. With the covariant derivative for triplet VLLs [51], we first write the Z-boson gauge interactions with VLLs,

$$\begin{aligned} \mathcal{L}_Z&= -\frac{g}{c_W} Z_\mu \left[ \phantom {\left. + \bar{N}_R \gamma ^\mu \left( \begin{array}{cc} 1/2 &{} 0 \\ 0 &{} 0 \end{array} \right) N_R + \bar{L}'_R \gamma ^\mu \left( \begin{array}{cc} 0 &{} 0 \\ 0 &{} -1 \end{array} \right) L'_R \right] } \bar{\Psi }_L \gamma ^\mu \left( I^\Psi _3 -s^2_W Q_\Psi \right) \Psi _L + \bar{\Psi }_R \gamma ^\mu ( -s^2_W Q_\Psi ) \Psi _R\right. \nonumber \\&\quad \left. + \overline{\Psi ^{-\,-}_1} \gamma ^\mu (-1 + 2 s^2_W) \Psi ^\mathrm{-\,-}_1 + \bar{\Psi }_L \gamma \mu \left( \begin{array}{cc} 1/2 &{} 0 \\ 0 &{} -1/2 \end{array} \right) \Psi _L \right. \nonumber \\&\quad \left. + \bar{N}_R \gamma ^\mu \left( \begin{array}{cc} 1/2 &{} 0 \\ 0 &{} 0 \end{array} \right) N_R + \bar{L}'_R \gamma ^\mu \left( \begin{array}{cc} 0 &{} 0 \\ 0 &{} -1 \end{array} \right) L'_R \right] , \end{aligned}$$

(11)

where we have expressed the forms of couplings to be the same as those in the SM, $I^\Psi _3 =1/2 (-1/2)$ and $Q_\Psi =0 (-1)$ for neutral (charged) VLLs, $\Psi ^T=(\Psi ^0_1, \Psi ^0_2)$ or $(\Psi ^-_1 , \Psi ^-_2)$, $N^T_R = (\Psi ^0_{1R}, \Psi ^0_{2R})$, and $L^{'T}_R = (\Psi ^-_{1R}, \Psi ^{-}_{2R})$. It is clear that the first two terms in Eq. (11) provide the flavor-conserving effects; however, the last three terms lead to Z-mediated FCNC couplings at the tree level. As mentioned earlier, the flavor mixings in $V^\chi _R$ are associated with the lepton masses; if we ignore the small effects, the gauge couplings of the Z-boson to the neutral and singly charged leptons can be expressed as

$$\begin{aligned} \mathcal{L}_{Z\ell ' \ell '}&= -\frac{g}{c_W} C^{\ell '_L}_{ij} \bar{\ell }'_L \gamma ^\mu \ell '_L Z_\mu - \frac{g}{c_W} C^{\ell '_R}_{ij} \bar{\ell }'_L \gamma ^\mu \ell '_R Z_\mu , \\ C^{\ell '_L}_{ij}&=( I^{\ell '}_3 -s^2_W Q_{\ell '}) \delta _{ij} + \frac{1}{2} \left( V^{\ell '}_{Li4} V^{\ell ' *}_{Lj4} - V^{\ell '}_{Li5} V^{\ell ' *}_{Lj5} \right) , \nonumber \\ C^{\ell '_R}_{ij}&\approx -s^2_W Q_{\ell '} \delta _{ij} + \left\{ \begin{array}{ll} \frac{1}{2}\delta _{i4} \delta _{j4} &{}\quad \text {for } Q_{\ell '}=0 \\ - \delta _{i5} \delta _{j5} &{}\quad \text {for } Q_{\ell '}=-1 \end{array} \right. \nonumber \end{aligned}$$

(12)

with $\ell '^T = ( \nu _1, \nu _2, \nu _3, \nu _4, \nu _5)$ or $(e, \mu , \tau , \tau ', \tau '')$. As a result, the tree-level Z-mediated FCNCs only occur in the left-handed currents.

The new gauge interactions of the W-boson with VLLs are given by

$$\begin{aligned} \mathcal{L}_W= & {} - g \left( \overline{ \Psi ^0_1} \gamma ^\mu \Psi ^-_1 + \overline{ \Psi ^-_1 } \gamma ^\mu \Psi ^{-\,-}_1 \right) W^+_\mu \nonumber \\&- g \left( \overline{\Psi ^0_2} \gamma ^\mu \Psi ^-_2 \right) W^+_\mu + H.c.. \end{aligned}$$

(13)

With $V^\chi _R \approx 1$, the W-mediated interactions of neutral and singly charged leptons are expressed as:

$$\begin{aligned} \mathcal{L}_{W\nu ' \ell '}= & {} - \frac{g}{\sqrt{2}} \overline{\mathcal{N}_L} \gamma ^\mu V^\nu _L \left( \begin{array}{cc} V'_\mathrm{PMNS} &{} 0 \\ 0 &{} \sqrt{2} \end{array} \right) \nonumber \\&\times V^{\ell \dagger }_L \ell '_L W^+_\mu + H.c., \end{aligned}$$

(14)

where $V'_\mathrm{PMNS}$ is the Pontecorvo–Maki–Nakagawa–Sakata (PMNS) matrix without triplet VLLs, which can be regarded as a model-dependent result, and is uncertain. Since $\mathbf{m}_\nu =0$, in this study $V'_\mathrm{PMNS}=1$. With the flavor mixings in Eq. (7), from Eq. (14) the W-boson interactions with the SM leptons are formulated by

$$\begin{aligned} \mathcal{L}_{W\nu \ell }&= -\frac{g}{\sqrt{2}} ( \bar{\nu }_1, \bar{\nu }_2, \bar{\nu }_3)_L \gamma ^\mu V_\mathrm{PMNS} \left( \begin{array}{c} e \\ \mu \\ \tau \end{array} \right) _L W^+_\mu \!+ \!H.c., \end{aligned}$$

(15)

$$\begin{aligned} V_\mathrm{PMNS}&= \!V'_\mathrm{PMNS} \!-\! V'_\mathrm{PMNS} \frac{{\pmb \varepsilon }^\ell _L {\pmb \varepsilon }^{\ell \dagger }_{L}}{2} \!-\! \frac{{\pmb \varepsilon }^\nu _L \pmb \varepsilon ^{\nu \dagger }_L}{2} V'_\mathrm{PMNS} \!+\! \sqrt{2} {\pmb \varepsilon }^\nu _L {\pmb \varepsilon }^{\ell \dagger }_L.\nonumber \\ \end{aligned}$$

(16)

The $V_\mathrm{PMNS}$ is the $3\times 3$ PMNS matrix, which can be extracted from the matrix product of $V^\nu _L (...) V^{\ell \dagger }_L$ in Eq. (14). Since the minimal value of the PMNS matrix element is around 0.15 [47], the limits of $Y_{1i}$ and $Y_{2i}$ from the charged current interactions may not be as clear as those from the rare Z decays. Therefore, to constrain the free parameters, we focus on the rare Z decays, such as $Z\rightarrow \ell ^{\pm }_i \ell ^{\mp }_j$, invisible Z decays, and so on.

Fig. 1

Contours as a function of Y and $m_{\Psi }$: a for $\Gamma (Z\rightarrow \nu _i \nu _j)$ and $\mathrm{BR}(h\rightarrow \mu \tau )$ and b for signal strength $\mu _{\gamma \gamma }$, where the numbers on the plot a for $\mathrm{BR}(h\rightarrow \mu \tau )$ are in units of $10^{-5}$

Before studying the LFV-related phenomenologies, we discuss the possible constraints on the free parameters $Y_{1i,2i}$. From Eq. (12), it can be seen that the Z-mediated lepton flavor-violating effects can contribute to $Z\rightarrow (e \mu , e\tau , \mu \tau )$, where the current upper limits of the data are [47]

$$\begin{aligned}&Br(Z\rightarrow e^\pm \mu ^\mp )< 7.5\times 10^{-7}, \nonumber \\&Br(Z\rightarrow e^\pm \tau ^\mp )< 9.8 \times 10^{-6}, \nonumber \\&Br(Z\rightarrow \mu ^\pm \tau ^\mp ) < 1.2 \times 10^{-5}. \end{aligned}$$

(17)

The severe constraints make the BR of $h\rightarrow \tau \mu$ decay far smaller than the CMS measurements. In order to avoid the bounds from the rare Z decays, we set $Y_{1i} = Y_{2i}$ and $m_{\Psi _1} = m_{\Psi _2}$. As a result, the second term of $C^{\ell '_L}_{ij}$ in Eq. (12) for charged leptons vanishes. However, the cancellations are not complete in the $Z\rightarrow \bar{\nu }_i \nu _j$ decays, due to the structure of $\mathbf{Y}^\nu$ in Eq. (6). That is, the invisible Z-boson decays can directly constrain the parameters, where the current measurement is $\Gamma ^Z_\mathrm{inv}=499 \pm 1.5$ MeV [47] and the SM prediction is around 500 MeV. With $Y_{1i} = Y_{2i} = Y_i$ and $m_{\Psi _1} = m_{\Psi _2} = m_{\Psi }$, the partial decay rates for $Z\rightarrow \bar{\nu }_i \nu _j$ and $h\rightarrow \mu \tau$ are given as:

$$\begin{aligned} \Gamma (Z\rightarrow \bar{\nu }_i \nu _j)&\approx \frac{m_Z}{24 \pi } \frac{g^2}{c^2_W} \sum _{ij} |C^\nu _{ij}|^2, \end{aligned}$$

(18)

$$\begin{aligned} \Gamma (h\rightarrow \mu \tau )&\approx \frac{m_h}{16\pi } \frac{m^2_\tau }{v^2} \left| \frac{ 3v^2 Y_2 Y_3}{2 m^2_\Psi }\right| ^2. \end{aligned}$$

(19)

Accordingly, we present the contours for $\Gamma (Z\rightarrow \bar{\nu }_i \nu _j)$ and $\mathrm{BR}(h\rightarrow \tau \mu )$ as a function of $Y_i$ and $m_\Psi$ in Fig. 1a, where numerically we adopt $Y_2=Y_3=Y$ and $Y_1\ll 1$, the solid line stands for $Z\rightarrow \bar{\nu }_i \nu _j$, the dashed line is for $h\rightarrow \mu \tau$ and the values on the plot are in units of $10^{-5}$, and $\Gamma _h \approx 4.21$ MeV is used. The results clearly show that the lepton flavor-violating Higgs decay can only be up to $10^{-4}$ when the data for invisible Z decays are applied.

Next, we discuss the influence of new flavor mixings on other Higgs decays. From Eq. (10), it can be seen that the induced couplings of Higgs to VLLs can contribute to $h\rightarrow \gamma \gamma$ through the loop diagrams; the decay rate can be formulated as:

$$\begin{aligned}&\Gamma (h\rightarrow \gamma \gamma ) \approx \Gamma ^\mathrm{SM}(h\rightarrow \gamma \gamma ) \left| 1 + \frac{C_{VLL}}{A_W + N_C Q^2_t A_{t}} \right| ^2, \end{aligned}$$

(20)

$$\begin{aligned}&C_{LLV} = \frac{v^2\sum _i Y^2_{i} }{2 m^2_{\Psi }} A_{1/2}(\tau _{\Psi }) \end{aligned}$$

(21)

where $N_C =3$, $Q_t = 2/3$, $A_W \approx 8.3$, $A_t \approx -1.38$, and the loop integral from VLL is [48]

$$\begin{aligned} A_{1/2}(\tau ) = -2 \tau [1+(1-\tau ) f(\tau )^2] \end{aligned}$$

with $\tau _\Psi = 4 m_{\Psi }^2/m_h^2$ and $f(x)=\sin ^{-1}(1/\sqrt{x})$. We note that although the coupling $h\tau \tau$ is modified by the new flavor-mixing effects, we ignore its small contribution to the loop-induced $h\rightarrow \gamma \gamma$ decay. The signal strength, which is used to show the Higgs measurement, is defined as:

$$\begin{aligned} \mu _{\gamma \gamma } = \frac{\sigma (pp\rightarrow h)}{\sigma (pp\rightarrow h)_\mathrm{SM}} \times \frac{\mathrm{BR}(h\rightarrow f)}{\mathrm{BR}(h\rightarrow f)_\mathrm{SM}}, \end{aligned}$$

(22)

where f denotes the possible decay channel. Taking $\Gamma _h \approx 4.21$ MeV and $\sigma (pp\rightarrow h) = \sigma (pp\rightarrow h)_\mathrm{SM}$, we plot the contours for $\mu _{\gamma \gamma }$ as a function of Y and $m_\Psi$ in Fig. 1b. For comparison, we also show the constraint from $\Gamma ^Z_\mathrm{inv}$ in the same plot. In these results we see that the deviation from the SM prediction is about $4~\%$ and is consistent with $\mu _{\gamma \gamma }=1.17\pm 0.27$ and $1.13\pm 0.24$, as measured by ATLAS [52] and CMS [53], respectively.

From Eq. (9), it can be seen that the modified Higgs couplings to the SM leptons are still proportional lepton masses. By comparison with other lepton channels, it can be seen that the $\tau \tau$ mode is more significant, and thus we study the influence on $h\rightarrow \tau ^+ \tau ^-$. Using the values that satisfy $\mathrm{BR}(h\rightarrow \mu \tau ) \approx 10^{-4}$, the deviation of $\Gamma (h\rightarrow \tau ^+ \tau ^-)$ from the SM results can be obtained:

$$\begin{aligned} \kappa _{\tau \tau } \equiv \frac{\Gamma (h\rightarrow \tau ^+ \tau ^-)}{\Gamma ^\mathrm{SM} (h\rightarrow \tau ^+ \tau ^-)} = \left| 1- \frac{6 v^2 Y^2_3}{8 m^2_\Psi } \right| ^2 \approx 0.88. \end{aligned}$$

(23)

If the SM Higgs production cross section is not changed, the signal strength for $pp\rightarrow h\rightarrow \tau ^+ \tau ^-$ in this model is $\mu _{\tau \tau } \approx 0.88$, where the measurements from ATLAS and CMS are $1.44^{+0.42}_{-0.37}$ [52] and $0.91 \pm 0.27$ [53], respectively. Although the current data errors for the $\tau \tau$ channel are still large, the precision measurement of $\mu _{\tau \tau }$ can test the model or give strict limits on the parameters.

In the following text we investigate the contributions of new couplings in Eq. (9) to the rare tau decays and to the flavor-conserving muon anomalous magnetic moment. We first examine the muon $g-2$, denoted by $\Delta a_\mu$. The lepton flavor-changing coupling $h\mu \tau$ can contribute to the $\Delta a_\mu$ through the Higgs-mediated loop diagrams. However, as shown in Eq. (9), the induced couplings are associated with $m_{\ell j}/v \bar{\ell }_{Li} \ell _{Rj}$; that is, only the right-handed tau-lepton has a significant contribution. The induced $\Delta a_\mu$ is thus suppressed by $m^2_\mu m_\tau /(v m^2_h)$ so that the value of $\Delta a_\mu$ is two orders of magnitude smaller than current data $\Delta a_\mu = a^\mathrm{exp}_\mu - a^\mathrm{SM}_\mu =(28.8\pm 8.0)\times 10^{-10}$ [47]. A similar situation happens in $\tau \rightarrow 3\mu$ decay. Since the couplings are suppressed by $m_\tau /v$ and $m_\mu /v$, the BR for $\tau \rightarrow 3\mu$ is of the order of $10^{-14}$. We also examine the process $\tau \rightarrow \mu \gamma$ via the h-mediation. The effective interaction for $\tau \rightarrow \mu \gamma$ is expressed by

$$\begin{aligned} \mathcal{L}_{\tau \rightarrow \mu \gamma } =\frac{ e }{16\pi ^2} m_\tau \bar{\mu }\sigma _{\mu \nu } \left( C_L P_L + C_R P_R\right) \tau F^{\mu \nu }, \end{aligned}$$

(24)

where $C_L=0$ and the Wilson coefficient $C_R$ from the one-loop is formulated as:

$$\begin{aligned} C_R&\approx \frac{ C^h_{23} C^h_{33} }{2m^2_h } \left( \ln \frac{m^2_{h} }{m^2_\tau } - \frac{4}{3}\right) . \ \end{aligned}$$

(25)

Accordingly, the BR for $\tau \rightarrow \mu \gamma$ is expressed as

$$\begin{aligned} \frac{\mathrm{BR}(\tau \rightarrow \mu \gamma )}{\mathrm{BR}(\tau \rightarrow e \bar{\nu }_e \nu _\tau ) } = \frac{3 \alpha _e}{4\pi G^2_F} |C_R|^2. \end{aligned}$$

(26)

We present the contours for $\mathrm{BR}(\tau \rightarrow \mu \gamma )$ as a function of Y and $m_\Psi$ in Fig. 2, where the numbers on the plots are in units of $10^{-12}$. It can be seen that the resultant $\mathrm{BR}(\tau \rightarrow \mu \gamma )$ can be only up to $10^{-12}$, where the current experimental upper bound is $\mathrm{BR}(\tau \rightarrow \mu \gamma )< 4.4 \times 10^{-8}$ [47].

Fig. 2

Contours for $\mathrm{BR}(\tau \rightarrow \mu \gamma )$ (dashed) as a function of Y and $m_\Psi$, where the constraint from $\Gamma ^Z_\mathrm{inv}$ (solid) is included

In this model, we have two new neutral leptons, two new singly charged leptons, and one doubly charged lepton. Since a particle carrying an electrical charge of 2 can have less background and a clearer signature in colliders, we discuss the potential for discovering the doubly charged lepton $\Psi ^{-\,-}_1$. By electroweak interactions, $\Psi ^{-\,-}_{1}$ can be produced singly and in pairs through the channels $\Psi ^{\mp \mp }_1 \Psi ^{\pm }_1$ and $\Psi ^{-\,-}\Psi ^{++}$, where the former is from charged weak interactions while the latter is from Z and electromagnetic interactions. In addition, due to the flavor-mixing effects, the W couplings to $\Psi ^{-\,-}_1$ and the SM leptons can be written

$$\begin{aligned} \mathcal{L}_{W\Psi ^{-\,-}_{1} \ell } = - g {\bar{\ell }}_i \gamma ^\mu \left( \pmb \varepsilon ^{\ell }_{L i 4} P_L + \pmb \varepsilon ^{\ell }_{R i4} P_R\right) \Psi ^{-\,-}_1 W^{+} + H.c. \nonumber \\ \end{aligned}$$

(27)

By the induced gauge couplings, the doubly charged lepton can be produced through the $\Psi ^{-\,-}_{1} \ell$ channels. We note that due to $\pmb \varepsilon ^{\ell }_{R} \propto \mathbf{m}_{\ell }$, the right-handed current contributions can be neglected. In order to discuss the production cross section in pp collisions, we implement our model in CalcHEP [49] and use CTEQ6L PDF [50] to do the numerical calculations. In Fig. 3, we show the single and pair production cross sections of $\Psi ^{\pm \, \pm }_1$ as a function of $m_\Psi$ in pp collisions at $\sqrt{s}=13$ TeV, where $vY/m_{\Psi } = 0.3$ is used for the $\Psi ^{++}\ell$ production. It can be seen that the production cross sections for $\Psi ^{\pm \pm } \ell$ modes are one order of magnitude smaller than those for other modes. For $m_\Psi < 400$ GeV, the production cross sections for $\Psi ^{-\,-} \Psi ^{++}$ and $\Psi ^{++} \Psi ^{-}$, which only depend on gauge couplings, can be over 50 fb.

Fig. 3

Doubly charged lepton production cross section (in units of fb) as a function of $m_\Psi$ in pp collisions at $\sqrt{s}=13$ TeV, where X in the y axis denotes the possible channel

Next, we discuss the decays of $\Psi ^{-\,-}_1$ and $\Psi ^{-}_1$. From Eq. (3), we see that before EWSB, the triplet VLLs in $\Psi _1$ are degenerate; however, the masses are split when the $\mathbf{Y}_{1}$ effects are involved. Since $\Psi ^{-\,-}_{1}$ cannot mix with other leptons, the mass splittings occur in neutral and singly charged leptons. According to Eqs. (5) and (7), the mass of $\Psi ^-_1$ shifted from $\Psi ^{-\,-}_1$ is:

$$\begin{aligned} m_{\Psi ^{-}_1} \approx m_{\Psi _1} \left( 1 + \frac{ v^2 \sum _i Y^2_{1i} }{8 m^2_{\Psi _1}} \right) . \end{aligned}$$

(28)

With the bound from $\Gamma ^Z_\mathrm{inv}$, the mass splitting is only $3~\%$. Therefore, we still take $m_{\Psi ^-_1} \approx m_{ \Psi ^{-\,-}_1} \approx \Psi ^0_1$ in this study.

Table 1 Number of events for the processes in Eqs. (32) and (33), where a luminosity of 100 fb$^{-1}$ and the center-of-mass energy of 13 TeV are used

From Eq. (27), we see that the $\Psi ^{-\,-}_{1}$ decay pattern depends on each value of Yukawa coupling $Y_{1i}$. If we adopt $Y_{11}\approx Y_{12}\approx Y_{13}=Y$, the better channels to search for the doubly charged lepton are $\Psi ^{-\,-}_1 \rightarrow (e,\mu ,\tau ) W^-$ and the corresponding BRs are fixed as:

$$\begin{aligned} \mathrm{BR}(\Psi ^{-\,-}_1 \rightarrow (e,\mu ,\tau ) W^-) \approx 1/3, \end{aligned}$$

(29)

where the lepton masses are ignored. Since $\tau$ has hadronic and leptonic decays and accompanies the neutrino when it decays, the clear signal for probing the doubly charged lepton should be $\Psi ^{-\,-}_1 \rightarrow \ell W^- \rightarrow \ell (\ell \nu )$, where $\ell =e,\mu$ and the final states are the same-sign dilepton. The Higgs and gauge boson couplings of $\Psi ^{-}_{1}$ to the SM leptons are given by

$$\begin{aligned} {I}_{\Psi ^-_1}= & {} \frac{v Y_{1i}}{2} \bar{\ell }_{iL} \Psi ^-_{1R} h + \frac{g}{c_W} \frac{v Y_{1i}}{2m_\Psi } \bar{\ell }_{iL} \gamma ^\mu \Psi ^{-}_{1L} Z_\mu \nonumber \\&+ g \frac{3v Y_{1i} }{2m_\Psi } \bar{\nu }_{i L} \gamma ^\mu \Psi ^{-}_{1L} W^+_\mu +H.c. \end{aligned}$$

(30)

It is found that the BRs for $\Psi ^-_1 \rightarrow (\ell _i h, \ell _i Z, \nu W)$ are insensitive to $Y=Y_{1i}$ and $m_{\Psi }$, and their ratios are

$$\begin{aligned} \ell _i h : \ell _i Z : \nu W \approx 0.02/3 : 0.1/3 : 0.89, \end{aligned}$$

(31)

where $\ell _i$ denotes one of the SM leptons and the $\nu W$ channel includes all SM neutrinos. It is clear that the BR for $\nu W$ is one order of magnitude larger than other decay modes.

According to the analysis, there are two ways to search for the doubly charged lepton in the model. In pair production, the search channel is

$$\begin{aligned} pp \rightarrow {\Psi }^{-\,-}_{1} \Psi ^{+\,+}_{1} \rightarrow ({\ell }^{-} W^{-}) ({\ell }^{+} W^{+}), \end{aligned}$$

(32)

where $\ell =e,\mu$, the W-boson can decay to leptons or jets, and their corresponding cross section with $m_\Psi =300$ GeV is 76 fb at $\sqrt{s}=13$ TeV. The expected events with a luminosity of 100 fb$^{-1}$ are shown in Table 1. In single production, the search channels are

$$\begin{aligned} pp\rightarrow \Psi ^{\pm \, \pm }_1 \Psi ^{\mp }_1 \rightarrow \left\{ \begin{array}{c} ({\ell }^\pm W^\pm ) ( \nu W^\mp ), \\ ({\ell }^\pm W^\pm ) ({\ell }^\mp Z ), \end{array}\right. \end{aligned}$$

(33)

where the associated cross sections for $m_\Psi =300$ GeV at $\sqrt{s}=13$ TeV are $\sigma ({\ell }^{+} \nu W^{+} W^{-})=122$ fb, $\sigma (\ell ^{-} \nu W^{-} W^{+})=58$ fb, $\sigma ({\ell }^{+} \ell ^{-} W^{+} Z)=9$ fb, and $\sigma ({\ell }^{-} {\ell }^{+} W^{-} Z)=4$ fb. Without event selection criteria and event kinematic cuts, we naively show the expected number of events with 100 fb$^{-1}$ in Table 1. In this work, we just show the potential for discovering the VLLs, and the detailed event simulation with kinematic cuts will be given elsewhere.

We also examine $\Psi ^{\pm \, \pm }$ production via flavor-changing interactions: $p p \rightarrow \Psi _{1}^{\pm \, \pm } \ell ^\mp$. Although the cross section is $O(1)-O(10)$ fb for $m_\Psi = 300-400$ GeV, as shown in Fig. 3, it would be possible to find the signal with a sufficiently large luminosity. This process will thus also be important, since we could test the flavor-changing coupling $v Y/m_\Psi$ in collider experiments. However, detailed analysis of this is left for future work.

In summary, we investigated Higgs-portal lepton flavor violation by introducing two triplet vector-like leptons to the SM; one is the hypercharge $Y=-1$ and the other is $Y=0$. The model has the Higgs-mediated and Z-mediated flavor-changing neutral currents at the tree level. When the bounds from rare $Z\rightarrow \bar{\ell }_i \ell _j$ decays are smeared out, the invisible Z decays become the dominant constraints. As a result, the branching ratio for $h\rightarrow \mu \tau$ can be up to $10^{-4}$, muon $g-2$ is two orders of magnitude smaller than the current data, and $\mathrm{BR}(\tau \rightarrow \mu \gamma )$ is of $O(10^{-12})$. The deviation of signal strength from the SM prediction in $\tau \tau$ mode is $12~\%$. We analyze the production channels for the predicted doubly charged lepton. We find that the interesting production channels in pp collisions are $pp\rightarrow (\Psi ^{-\,-}_{1} \Psi ^{++}_1, \Psi ^{\pm \pm }_1 \Psi ^\mp _1)$. Both single and pair production cross sections of $\Psi ^{++}_1$ are comparable and can be a few hundred fb. The main decay channel for the doubly charged lepton is $\Psi ^{\pm \pm } \rightarrow \ell ^\pm W^\pm$, while the heavy singly charged lepton is $\Psi ^\pm _1 \rightarrow \nu W^\pm , \ell ^\pm Z$. We then summarize possible signatures of our model with the expected number of events for 100 fb$^{-1}$ luminosity at the 13 TeV LHC.