Lepton flavor violation and collider searches in a type I + II seesaw model

Abstract

Neutrinos are massless in the Standard Model. The most popular mechanism to generate neutrino masses are the type I and type II seesaw, where right-handed neutrinos and a scalar triplet are augmented to the Standard Model, respectively. In this work, we discuss a model where a type I + II seesaw mechanism naturally arises via spontaneous symmetry breaking of an enlarged gauge group. Lepton flavor violation is a common feature in such setup and for this reason, we compute the model contribution to the $\mu \rightarrow e\gamma$ and $\mu \rightarrow 3e$ decays. Moreover, we explore the connection between the neutrino mass ordering and lepton flavor violation in perspective with the LHC, HL-LHC and HE-LHC sensitivities to the doubly charged scalar stemming from the Higgs triplet. Our results explicitly show the importance of searching for signs of lepton flavor violation in collider and muon decays. The conclusion about which probe yields stronger bounds depends strongly on the mass ordering adopted, the absolute neutrino masses and which much decay one considers. In the 1–5 TeV mass region of the doubly charged scalar, lepton flavor violation experiments and colliders offer orthogonal and complementary probes. Thus if a signal is observed in one of the two new physics searches, the other will be able to assess whether it stems from a seesaw framework.

1 Introduction

The observation of neutrino oscillations implies that neutrinos are massive particles. Their masses are much smaller than any other in the Standard Model (SM) spectrum. Several mechanism have surfaced trying to explain the smallness of the neutrino masses [1,2,3,4,5]. In the type I seesaw, the existence of heavy right-handed neutrinos is evoked, whereas in the type II seesaw a scalar triplet is added to the SM. This scalar triplet couples to the SM lepton doublets and features a neutral scalar that develops a small vacuum expectation value giving rise to tiny neutrino masses [6,7,8,9,10]. This mechanism can elegantly explain the masses of the three active neutrinos in the SM and lead to several phenomenological imprints in collider, low energy observables and leptogenesis [11, 12, 12,13,14,15,16,17,18].

On the other side, we do not know why we have three generations of fermions in the SM. Theoretically speaking, it would nice to have a model where these two problems are simultaneously addressed. Several models have been proposed where the number of generations is addressed by anomaly cancellation and asymptotic freedom arguments, and they are known as 3-3-1 models [19,20,21]. These models have an enlarged gauge sector, $SU(3)_c \times SU(3)_L \times U(1)_N$  (3-3-1 for short). Notice that $SU(2)_L\times U(1)_Y$ in the SM gives place to $SU(3)_L \times U(1)_N$. Therefore, the fermions will now be arranged in the fundamental representation of $SU(3)_L$, i.e. triplets. The same happens for scalars fields, also sorted in triplets in order to generate fermion masses. Several versions based on the 3-3-1 symmetry have been proposed in the literature where the replication of fermion generations are explained [22,23,24,25,26].

Not all of these realizations are capable of explaining neutrino masses at the same time, though. In this work, we focus on a model which can account for neutrino masses, known as 3-3-1 model with right-handed neutrinos [21, 27,28,29]. Originally the model has only three scalar triplets. With scalar triplets one can generate two mass degenerate neutrinos and a massless one [30], which is in conflict with existing data [31]. The use of high dimensional effective operators has been put forth in the attempt to break the degeneracy [32, 33]. In this setup, the smallness of the neutrino masses and the oscillation pattern is not successfully explained. The most simple way to nicely solve this issue is by adding a scalar sextet [34, 35]. The interesting aspect of this scalar sextet is that after spontaneous symmetry breaking it breaks down to a scalar triplet, two doublet scalars and a scalar singlet field. The scalar triplet is exactly the one desired to perform the type II seesaw mechanism. Therefore, in this way, the type II seesaw arises as a result of the spontaneous symmetry breaking. Several phenomenological aspects of the 3-3-1 with right-handed neutrinos have been explored in the past [36,37,38,39,40,41], but our work differs from those because,

• We explicitly compute the $\mu \rightarrow e\gamma$ and $\mu \rightarrow 3e$ decays;

• We explore their connection to neutrino mass ordering;

• We put our results into perspective with collider and lepton flavor violation bounds.

This work is structured as follows: In Sect. 2 we briefly describe the model and explain how neutrino masses are incorporated, in Sect. 3 we present the relevant collider bounds, in Sect. 4 we derive the lepton flavor violating muon decays and then draw our conclusions.

2 Model with right-handed neutrinos and scalar sextet

There are several models based on the 3-3-1 gauge group. The model discussed here differs from the original proposal [19,20,21] because of the presence of three right-handed neutrinos. This model does not suffer from a Landau pole at the TeV scale as the previous version. For this reason, it has attracted lots of attention in the past decade [42,43,44,45,46,47,48,49,50,51,52,53,54]. In this section, we will briefly discuss the model to ease our reasoning. We start with the fermion content.

2.1 Fermion content

In this model, the left-handed leptons are arranged in the fundamental representation of $SU(3)_L$, whereas right-handed leptons as a singlet, as follows,

$$\begin{aligned} \psi _{aL}=\left( \begin{array}{c} \nu _{a} \\ e_{a} \\ \nu _{a}^{c} \end{array} \right) _{L}\sim \left( \mathbf {1},\mathbf {3},-1/3\right) , e_{aR}\sim (\mathbf {1},\mathbf {1},-1), \end{aligned}$$

(2.1)

where $a=1,2,3$ runs through the fermion generations. The numbers in parenthesis represent the quantum number of the fields under $SU(3)_C$, $SU(3)_L$ and $U(1)_N$, respectively. For instance, the $\psi _{aL}$ is a singlet under $SU(3)_C$, triplet under $SU(3)_L$ and has a charge equal $-1/3$ under $U(1)_N$. Notice that the third component of the leptonic triplet is a right-handed neutrino since $\left( \nu _{a}^{c}\right) _{L}=\nu _{R}^{c}$, explaining why this model is known as 3-3-1 model with right-handed neutrinos.

In a similar vein, the quarks are arranged as follows,

$$\begin{aligned} Q_{\alpha L}= & {} \left( \begin{array}{c} d_{\alpha } \\ -u_{\alpha } \\ D_{\alpha } \end{array} \right) _{L}\sim \left( \mathbf {3},\mathbf {3}^{*},0\right) , \end{aligned}$$

(2.2)

$$\begin{aligned} u_{\alpha R}\sim & {} \left( \mathbf {3},\mathbf {1},\frac{2}{3}\right) ,\quad d_{\alpha R}\sim \left( \mathbf {3},\mathbf {1},-\frac{1}{3}\right) ,\nonumber \\ D_{\alpha R}\sim & {} \left( \mathbf {3},\mathbf {1},-\frac{1}{3}\right) . \end{aligned}$$

(2.3)

$$\begin{aligned} Q_{3L}= & {} \left( \begin{array}{c} u_{3} \\ d_{3} \\ U \end{array} \right) _{L}\sim \left( \mathbf {3},\mathbf {3},1/3\right) , \end{aligned}$$

(2.4)

$$\begin{aligned} u_{3R}\sim & {} \left( \mathbf {3},\mathbf {1},\frac{2}{3}\right) ,\quad d_{3R}\sim \left( \mathbf {3},\mathbf {1},-\frac{1}{3}\right) ,\nonumber \\ U_{R}\sim & {} \left( \mathbf {3},\mathbf {1},\frac{2}{3}\right) . \end{aligned}$$

(2.5)

where $\alpha =1,2$. We highlight that the first two generations of quarks are placed in the anti-triplet representation of $SU(3)_L$. This has to be the case in order to cancel gauge anomalies. The quarks $D_{1,2}$ are down-type heavy quarks, in the sense, they have the same hypercharge as the SM quark down. A similar logic applies to the heavy up-quark U. These quarks have masses proportional to the scale of symmetry breaking of the 3-3-1 symmetry, which should lie at several TeV, to be consistent with the null results of such exotic quarks [55, 56]. The scale of symmetry breaking of the 3-3-1 symmetry will be kept sufficiently high to be consistent with these constraints. We will address the fermion masses below.

2.2 Fermion masses

To generate fermion masses we need to invoke three scalar triplets of the following form,

$$\begin{aligned} \chi= & {} \left( \begin{array}{c} \chi _{1}^{0} \\ \chi _{2}^{-} \\ \chi _{3}^{0} \end{array} \right) \sim \left( \mathbf {1},\mathbf {3},-1/3\right) ,\quad \eta =\left( \begin{array}{c} \eta _{1}^{0} \\ \eta _{2}^{-} \\ \eta _{3}^{0} \end{array} \right) \sim \left( \mathbf {1},\mathbf {3},-1/3\right) ,\nonumber \\ \rho= & {} \left( \begin{array}{c} \rho _{1}^{+} \\ \rho _{2}^{0} \\ \rho _{3}^{+} \end{array} \right) \sim \left( \mathbf {1},\mathbf {3},2/3\right) . \end{aligned}$$

(2.6)

These scalar triplets are sufficient to successfully yield masses to all fermions, except neutrinos. The spontaneous symmetry breaking goes as: first $SU(3)_{L}\otimes U(1)_{X}\overset{\left\langle \chi \right\rangle }{ \rightarrow }SU(2)_{L}\otimes U(1)_{Y},\$while$\ \ SU(2)_{L}\otimes U(1)_{Y} \overset{\left\langle \eta \right\rangle ,\left\langle \rho \right\rangle }{ \rightarrow }U(1)_{Q}$. In other words, when $\chi$ develops vacuum expectation values the 3-3-1 symmetry is broken reproducing the SM gauge group, which then breaks into quantum electrodynamics after $\eta$ and $\rho$ acquire a vacuum expectation value. For the purpose of this work, we will work on the broken phase where,

$$\begin{aligned} \left\langle \chi \right\rangle= & {} \frac{1}{\sqrt{2}}\left( \begin{array}{c} 0 \\ 0 \\ v_{\chi } \end{array} \right) ,\quad \left\langle \eta \right\rangle =\frac{1}{\sqrt{2}}\left( \begin{array}{c} v_{\eta } \\ 0 \\ 0 \end{array} \right) ,\nonumber \\ \left\langle \rho \right\rangle= & {} \frac{1}{\sqrt{2}}\left( \begin{array}{c} 0 \\ v_{\rho } \\ 0 \end{array} \right) . \end{aligned}$$

(2.7)

For simplicity we will adopt $v_{\eta }=v_{\rho }$. This assumption is typically made to simplify the diagonalization of the mass matrices involving the scalars [44]. Moreover, we take $v_{\eta }^2+v_{\rho }^2 \sim 246^2\, \mathrm{GeV^2}$, this ought to be enforced to generate the correct masses for the W and Z bosons in the SM, as occurs in models with extra scalars contributing to gauge boson masses [57, 58].

We have explained in detail how each mass is generated in the Appendix, for this reason, in what follows we will just briefly consider each sector separately.

2.2.1 Charged leptons masses

The masses for the charged leptons are generated via the lagrangian,

$$\begin{aligned} \mathcal {L}_{Y}^{l}=h_{ab}^{l}\overline{\psi }_{aL}\rho e_{bR} +h.c.. \end{aligned}$$

(2.8)

Notice that this term conserves lepton number. One possible term that one could write down is $\epsilon _{ijk}\, \overline{\psi ^c}_{L\,i} \psi _{Lj} \rho _k$, which does not conserve lepton flavor. The reason why we did not include this term in Eq. (2.8) is that there is a set of discrete symmetries that will invoke later on to prevent mixing between the SM quarks and the exotic ones. One of them requires $\rho \rightarrow -\rho$, which forbids the term above. However, we need to impose $e_R \rightarrow - e_R$ to engender masses for charged leptons. When the field $\rho _2^0$ develops a vacuum expectation value, $v_\rho$, we find,

$$\begin{aligned} m_l= h_{ab}^l \frac{v_\rho }{\sqrt{2}}. \end{aligned}$$

(2.9)

2.2.2 Quarks masses

The quark masses could stem from two sources, one where flavor is conserved (LFC) and other where it is violated (LFV) as follows,

$$\begin{aligned} \mathcal {L}_{LFC}= & {} h_{\alpha a}^{u}\overline{Q}_{\alpha L}\rho ^{*}u_{aR}+h_{\alpha a}^{d}\overline{Q}_{\alpha L}\eta ^{*}d_{aR}\nonumber \\&+h^{U} \overline{Q}_{3L}\chi U_{R} +h_{a}^{d}\overline{Q}_{3L}\rho d_{aR}+h_{a}^{u}\overline{Q}_{3L}\eta u_{aR}\nonumber \\&+ h_{\alpha \beta }^{D}\overline{Q}_{\alpha L}\chi ^{*}D_{\beta R}+h.c., \end{aligned}$$

(2.10)

and

$$\begin{aligned} \mathcal {L}_{LFV}= & {} s_{a}^{u}\overline{Q}_{3L}\chi u_{aR}+s_{\alpha a}^{d}\overline{Q} _{\alpha L}\chi ^{*}d_{\beta R}\\&+s^{U}\overline{Q}_{3L}\eta U_{R} +s_{\alpha a}^{D}\overline{Q}_{\alpha L}\eta ^{*}D_{aR}\\&+s_{\alpha }^{D} \overline{Q}_{3L}\rho D_{\alpha R}+s_{\alpha }^{U}\overline{Q}_{\alpha L}\rho ^{*}U_{R}+h.c.. \end{aligned}$$

Nevertheless, the LFV lagrangian is problematic because it induces mixing between the SM quarks and the exotic ones. This mixing could alter the properties of the SM quarks. Thus we need to eliminate them. To do so, we invoke a set of $Z_2$ symmetries where some fields are odd under: $u_{aR} \rightarrow -u_{aR}$, $d_{aR}\rightarrow - d_{aR}$, $D_{aR} \rightarrow - D_{aR}$, $\eta \rightarrow -\eta$, $\rho \rightarrow -\rho$. The reaming fields transform trivially. We emphasize that these discrete symmetries do not affect the other sectors of the model that is still capable of reproducing the SM features at low energy scales with no prejudice.

In summary, with these discrete symmetries the SM quarks get masses through Eq. (2.10) yielding,

$$\begin{aligned} m_{u,c} = \frac{-h^u_{11,22} v_{\rho } }{\sqrt{2}},\quad m_{t} = \frac{h^u_{33} v_{\eta } }{\sqrt{2}}, \end{aligned}$$

(2.11)

and,

$$\begin{aligned} m_{d,s} = \frac{h^d_{11,22} v_{\eta } }{\sqrt{2}},\quad m_{b} = \frac{h^d_{33} v_{\rho } }{\sqrt{2}}. \end{aligned}$$

(2.12)

It is visible that their masses are proportional to $v_{\eta }$ and $v_{\rho }$ which are set at the electroweak scale to avoid flavor changing interactions and return the correct quark masses [46, 59,60,61,62,63,64,65].

Moreover, from Eq. (2.10) we find the exotic quarks masses,

$$\begin{aligned} m_U =\frac{h^U v_\chi }{\sqrt{2}},\quad m_D =\frac{h^D_{11,22} v_\chi }{\sqrt{2}} \end{aligned}$$

(2.13)

which are proportional to the scale of the 3-3-1 symmetry breaking, $v_\chi$, assumed to be sufficiently high to be in agreement with null results from collider searches for exotic quarks [56, 66, 67]. We will now turn our attention to neutrino masses.

2.2.3 Neutrino masses-type $I+II$ seesaw

As far as neutrino masses are concerned, the scalar triplets do not suffice to successfully generate neutrino masses. Using Lie algebra one may notice that $\bar{\psi _L^c} \psi _L \sim (\mathbf {3}^{*} + \mathbf {6}, -2/3)$. Thus, one may generate neutrino masses via the scalar triplet $\rho$ and a scalar sextet [21, 68, 69] with,

$$\begin{aligned} S=\left( \begin{array}{lll} S_{11}^{0} &{} S_{12}^{-} &{} S_{13}^{0} \\ S_{12}^{-} &{} S_{22}^{--} &{} S_{23}^{-} \\ S_{13}^{0} &{} S_{23}^{-} &{} S_{33}^{0} \end{array} \right) \sim \left( \mathbf {1},\mathbf {6},-2/3\right) . \end{aligned}$$

(2.14)

However, the discrete symmetries mentioned previously prohibit the Yukawa term proportional to $\rho$ and for this reason, only the scalar sextet contributes leading to,

$$\begin{aligned} \mathcal {L}_{Y}^{l}=f_{ab}^{\nu }(\overline{\psi }_{aL} )_{m}(\psi _{bL}^{c})_{n}S_{mn}+h.c. \end{aligned}$$

(2.15)

Neutrino masses arise after the neutral scalars develop vacuum expectation value as follows,

$$\begin{aligned} \left\langle S\right\rangle =\frac{1}{\sqrt{2}}\left( \begin{array}{lll} v_{s_{11}} &{} 0 &{} v_{s_{13}} \\ 0 &{} 0 &{} 0 \\ v_{s_{13}} &{} 0 &{} \Lambda \end{array} \right) \end{aligned}$$

(2.16)

Notice that $v_{s11},v_{s13}$, and $\Lambda$ are the vacuum expectation value of the neutral scalars in the sextet.

With this vacuum structure we generate a mass matrix of the form,

$$\begin{aligned} \mathcal {L}_{mass}^{\nu }=-\frac{1}{2}\left( \begin{array}{cc} \overline{\nu }_{aL}&\overline{\nu }_{aR}^{c} \end{array} \right) M_{\nu }\left( \begin{array}{c} \nu _{bL}^{c} \\ \nu _{bR} \end{array} \right) +h.c. \end{aligned}$$

(2.17)

where $M_{\nu }$,

$$\begin{aligned} M_{\nu }=\left( \begin{array}{cc} M_{L} &{} M_{D} \\ M_{D}^{T} &{} M_{R} \end{array} \right) \end{aligned}$$

(2.18)

with,

$$\begin{aligned} M_{L}= & {} \sqrt{2}v_{s_{11}}f^{\nu }, \end{aligned}$$

(2.19)

$$\begin{aligned} M_{D}= & {} \sqrt{2} v_{s_{13}}f^{\nu } , \end{aligned}$$

(2.20)

$$\begin{aligned} M_{R}= & {} \sqrt{2}\Lambda f^{\nu }. \end{aligned}$$

(2.21)

We remind the reader that $M_{\nu }$ is a $6\times 6$ matrix and $M_{L},M_{D}$ and $M_{R}$ as $3\times 3$ matrices. After the diagonalization procedure we get for the active neutrinos,

$$\begin{aligned} (m_{\nu })_{ab}=\sqrt{2}\left( v_{s_{11}}-\frac{v_{s_{13}}^{2}}{\Lambda }\right) (f^{\nu })_{ab}, \end{aligned}$$

(2.22)

and for the right-handed ones,

$$\begin{aligned} (M_{\nu _R})_{ab}=\sqrt{2}\Lambda (f^{\nu })_{ab}. \end{aligned}$$

(2.23)

It is good timing to highlight a few things:

• (i) the mass ratio $m_\nu /m_{\nu _R}$ is independent of the Yukawa coupling;

• (ii) a scalar sextet breaks down to a triplet plus doublet plus singlet scalar, i.e. $\mathbf {6}\rightarrow \mathbf {3}+\mathbf {2}+\mathbf {1}$. In order words, the scalar sextet generates the triplet scalar used in the type II seesaw, the doublet scalar that induces Dirac masses, $M_D$, and a singlet scalar that yields right-handed Majorana masses, $M_R$. Therefore, the scalar sextet naturally gives rise to a type I + II seesaw mechanism via spontaneous symmetry breaking;

• (iii) Setting $v_{s13}^2/\Lambda \propto v_{s11}$, say $v_{s13}^2/\Lambda =0.1 v_{s11}$ the smallness of the active neutrino masses are justified by taking $v_{s11}$ to be around 1 eV.

There are different possibilities to successfully generate neutrino masses. Since the vacuum expectation values of the fields in the scalar sextet are in principle arbitrary one can play with them and find different Yukawa couplings leading to the same neutrino masses. Be that as it may, one can draw important and insightful conclusions by adopting some simplifications. We will assume throughout that $v_{s13}^2/\Lambda -v_{s11}\sim v_{s11}$. Thus the neutrino masses are basically $(m_\nu )_{ab} \sim \sqrt{2} v_{s11} f^\nu _{ab}$. In our study we will consider two cases, one where $m_\nu \sim 0.1$ eV and the another where $m_\nu \sim 0.01$ eV. Since the doubly charged scalar, $S_{22}^{--}$ in Eq. 2.14 is the key player in our reasoning and it has mass proportional to $\Lambda$, we are not particularly interested in the case which $\Lambda$ is very large, at the GUT scale for instance. Therefore, $v_{s13}$ has to be sufficiently small. Assuming $\Lambda$ to be around $10^5$ GeV, we get $v_{s13}$ between 0.01–0.1 GeV for $v_{s11}=1$–100 eV respectively. It is important to note that such small value for $v_{s13}$ is not problematic because the masses of the scalars in the model are proportional to $\Lambda$. Moreover, $v_{s13}$ will enter in the W gauge boson mass, thus we can enforce it to be small to keep the $\rho$ parameter unaltered, similarly to what occurs for the triplet field in the type II seesaw. Later on, we will present several benchmark scenarios taking $v_{s11}=1$ eV and $v_{s11}=100$ eV to investigate the impact of the neutrino mass ordering on the lepton flavor violating muon decays. With this information, one can now have an estimate of the right-handed neutrino masses since they are proportional to $\Lambda$. Notice, that as long as $v_{s13}^2/\Lambda - v_{s11} \sim v_{s11}$ this choice for $\Lambda =10^5$ GeV brings no impact to our conclusions. In other words we could take $\Lambda$ to be larger because the mass of the doubly charged scalar is proportional to $\Lambda$ and quartic couplings in the scalar potential involving the scalar sextet. If $\Lambda$ were very large, we could simply take the quartic couplings to be sufficiently small to bring down the doubly charged scalar mass to TeV scale with no prejudice. We will not explicitly consider the latter scenario here and we will simply assume $\Lambda$ to be around $10^5$ GeV, for concreteness.

In summary, we will adopt $v_{s13}^2/\Lambda - v_{s11} \sim v_{s11}$ so that $(m_{\nu })_{ab} =\sqrt{2} v_{s11} f^{\nu }_{ab}$, with $\Lambda$ not too large to naturally have a doubly charged scalar at the TeV scale. In this way, we have a dominant type II seesaw mechanism with the neutrino masses governed by the vacuum expectation value of $S^0_{11}$. One can think of it as the vacuum expectation value of the triplet under $SU(2)_L$ in the broken phase in the usual type II seesaw study.

Looking at Eq. (2.15) we notice that $S^0_{33},S^0_{11},S^-_{12},S^{--}_{22}$ carry two units of lepton number and for this reason are called bileptons. The lepton flavor (number) is violated after the neutral components $S^0_{11}$ and $S^0_{33}$ acquire a vacuum expectation value. The singly charged scalar in the scalar sextet will not mix with other singly charged scalars that do not carry lepton number. The presence of flavor violation will be explored in this work in the context of muon decays.

2.3 Gauge sector

We turn our attention to the gauge sector. The main point is to show that there are new gauge bosons with masses are proportional to the 3-3-1 scale of symmetry breaking and they are subject to stringent collider bounds. Showing this in a pedagogical manner requires us to start with the covariant derivative of $SU(3)_{L}\otimes U(1)_{N}$,

$$\begin{aligned} \mathcal {D}_{\mu }\varphi =\left[ \partial _{\mu }-ig_{L}W_{\mu }^{m}\frac{ \lambda ^{m}}{2}-ig_{N}N_{\varphi }W_{\mu }^{N}\right] \varphi , \end{aligned}$$

(2.24)

where $g_{L}$ is the coupling constant of $SU_{L}\left( 3\right)$ group, $g_{N}$ is the coupling constant of $U(1)_{N}, \lambda ^{m}$ are the Gell-Mann matrices with $m=1,\ldots ,8,$ $W_{\mu }^{m}$ are the gauge bosons in the adjoint representation of $SU(3)_{L}$, $W_{\mu }^{N}$ is the gauge field associated to $U(1)_{N}$, $N_{\varphi }$ is the hypercharge associated to $U(1)_{N}$, and $\varphi =\phi ,\ \eta ,\ \chi$ are the scalar triplets. Writing down the term proportional to $\lambda ^m$ one finds,

$$\begin{aligned} \frac{g_{L}}{2}W_{\mu }^{m}\lambda ^{m}=\frac{g_{L}}{2}\left( \begin{array}{ccc} W_{\mu }^{3}+\frac{1}{\sqrt{3}}W_{\mu }^{8} &{} \sqrt{2}W_{\mu }^{+} &{} \sqrt{2}U_{\mu }^{0} \\ \sqrt{2}W_{\mu }^{-} &{} -W_{\mu }^{3}+\frac{1}{\sqrt{3}} W_{\mu }^{8} &{} 2\,W_{ \mu }^{\prime -} \\ \sqrt{2}U_{\mu }^{0*} &{} \sqrt{2}W_{\mu }^{\prime +} &{} -\frac{2}{\sqrt{3}} W_{\mu }^{8} \end{array} \right) , \end{aligned}$$

(2.25)

where $W_{\mu }^{\pm }=\left( W_{\mu }^{1}\mp iW_{\mu }^{2}\right) /\sqrt{2}$ are the SM charged gauge bosons, $U_{\mu }^{0}=\left( W_{\mu }^{4}-iW_{\mu }^{5}\right) \sqrt{2}$, and $U_{\mu }^{0* }=\left( W_{\mu }^{4}+i W_{\mu }^{5}\right) /\sqrt{2},$ $W_{\mu }^{\prime \pm }=\left( W_{\mu }^{6}\pm i W_{\mu }^{7}\right) /\sqrt{2}$ are the new gauge bosons, respectively the non-hermitian neutral and charged ones.

Therefore, the 3-3-1 model with right-handed neutrinos predicts the existence of new charged gauge bosons, $W^{\prime \pm }$, which are subject to intense searches at the LHC [70, 71], and exotic neutral gauge bosons $U^{0}$ and $U^{0*}$ [72]. Moreover, from a combination of the $W_3$, $W_4$, $W_8$ and $W_N$ fields, we extract the SM photon and Z bosons, as well as a massive $Z^{\prime }$ field. It is clear the model add five new gauge bosons to the SM, as a direct result of the extended gauge sector which predicted $N^2-1$ bosons, where $N=3$ in this case.

The masses of SM gauge bosons are slightly altered by the presence of the scalar sextet. This change is proportional to the vacuum expectation value $v_{s11}$ and $v_{s13}$ which are meant to be small. A similar conclusion is found in the usual type II seesaw mechanism. The bound that rises from the $\rho$ parameter enforces at $3\sigma$ [57],

$$\begin{aligned} v_{s_{11}}+v_{s_{13}} \le 2 ~ GeV. \end{aligned}$$

(2.26)

The masses of the new gauge bosons are all proportional to $\Lambda$. We present their masses in the Appendix. The relevant gauge bosons for our reasoning are the $Z^\prime$ and $W^\prime$ fields whose mass read,

$$\begin{aligned} M^2_{Z^{\prime }}\approx & {} \frac{g^2}{4h_W\cos ^2{\theta _W}}\left[ v^2_\rho +2v^2_{s_{13}} +\left( v^2_\eta +4v^2_{s_{11}}\right) \cos ^2{(2\theta _W)}\right. \nonumber \\&\left. +4(v^2_\chi +4\Lambda ^2) \cos ^4\theta _W\right] , \end{aligned}$$

(2.27)

and,

$$\begin{aligned} M^2_{W^{\prime \pm }}\approx \frac{g^2}{4}(v^2_\rho +v^2_\chi +4v^2_{s_{13}}+2\Lambda ^2), \end{aligned}$$

(2.28)

where,

$$\begin{aligned} \cos ^2{\theta _W}= & {} \frac{3+t^2}{3+4t^2}, \end{aligned}$$

(2.29)

$$\begin{aligned} \sin ^2{\theta _W}= & {} \frac{3t^2}{3+4t^2}, \end{aligned}$$

(2.30)

and

$$\begin{aligned} h_W=3-4\sin ^2{\theta _W}. \end{aligned}$$

(2.31)

with $t=\frac{g_N}{g}$, where $g_N$ (g) is the gauge coupling of the $U(1)_N$ ( $SU(2)_L$) group.

As aforementioned we will assume $\Lambda =10^5$ GeV, therefore the gauge bosons will have sufficiently large masses to avoid the collider bounds we address further.

3 Collider bounds

3.1 Searches for $Z^\prime$ bosons

Collider searches for heavy $Z^\prime$ bosons are quite popular because they typically feature a clear signal. If they couple to fermions and have a narrow width, they give rise to pronounced bumps in the dilepton or dijet invariant mass [73]. If the couplings to leptons are not very suppressed, the use of dilepton data is more promising because it is subject to a smaller SM background. In this model, the $Z^\prime$ couplings to leptons are not small. Using LHC data at 13 TeV center-of-mass energy with 3.2 $fb^{-1}$ of integrated luminosity the authors in [74] placed a lower mass bound of 3 TeV. Later, in [75] this limit was improved using 36 $fb^{-1}$ and 3 $ab^{-1}$ of integrated luminosity finding,

$$\begin{aligned} m_{Z^\prime }> & {} 4~\mathrm{TeV}\,\, (\mathcal {L}=36~fb^{-1}) ,\nonumber \\ m_{Z^\prime }> & {} 6.4~\mathrm{TeV} \,\, (\mathcal {L}=3~ab^{-1}). \end{aligned}$$

(3.1)

We emphasize that this limit of 4 TeV relies on the dielectron plus dimuon data with invariant mass in the 500–6000 GeV mass range as recommended, with the cuts in transverse energy and momentum as recommended by ATLAS collaboration in [76, 77]. The projected bound of 6.4 TeV assumes a similar detector with the same trigger efficiency running at 14 TeV center-of-mass energy and with $3 ab^{-1}$ of integrated luminosity. This lower mass bound of 4 TeV is rather robust and important because the $Z^\prime$ mass is tied to the scale of symmetry breaking of the model, $\Lambda$. Therefore, our choice for $\Lambda =10^5$ GeV safely obeys this bound.

Therefore, our entire discussion of lepton flavor violation is fully consistent with these bounds on the $Z^\prime$ mass. Now we will address collider searches for lepton flavor violation in what follows.

3.2 Searches for a doubly charged scalar at the LHC

The presence of a doubly charged scalar in the spectrum is typical signature of a type II seesaw mechanism [78,79,80,81,82,83,84,85,86,87]. Typically, this type II seesaw is realized by the addition of a scalar triplet. This popular extension triggered several phenomenological analyses. As we mentioned previously, the scalar triplet that arises after the spontaneous symmetry breaking is key to the type II seesaw mechanism in our model. Strictly speaking our model has a type I + II seesaw because we do also have Dirac neutrino masses, but we will assume type II seesaw dominance as discussed previously. After spontaneous symmetry breaking, we can single out the Yukawa term involving the scalar triplet and SM particles which reads [57],

$$\begin{aligned} \mathcal {L} = f_{ab}^\nu \bar{L}^c_{a} \Delta ^{*} L_b \end{aligned}$$

(3.2)

where L is the SM lepton doublet with,

$$\begin{aligned} \Delta =\left( \begin{array}{cc} S_{}^0 &{} S_{}^- \\ S_{}^- &{} S^{--}\\ \end{array}\right) , \end{aligned}$$

(3.3)

where $S^0\equiv S^0_{11}$, $S^-\equiv S^-_{12}$, $S^{--}\equiv S_{22}^{--}$, as defined in Eq. (2.14).

The doubly charged scalar decay width into charged leptons is found to be [88],

$$\begin{aligned} \Gamma \left( S^{\pm \pm } \rightarrow l^{\pm }_a l^{\pm }_b \right) = 1/(2 \pi x) |f^\nu _{ab}|^2 m_{S^{\pm \pm }}^2, \end{aligned}$$

(3.4)

where $x= 1$ for $i \ne j$ and $x=2$ for $i=j$. As we pointed out earlier, the mass of the doubly charged scalar will be proportional to $\Lambda$ and the quartic coupling of the scalar sextet that appears in the scalar potential, which has been omitted. As we are assuming $\Lambda \sim 10^5$  GeV, we can bring down the doubly charged scalar mass to the TeV scale by taking this quartic coupling to be sufficiently small.

It is clear that the branching ratio into charged leptons can change depending on the Yukawa couplings. Different choices lead to different branching ratios and consequently different lower mass bounds. The dominant decay mode determines cuts, detector efficiency, and backgrounds which the signal is subject to, and consequently yielding different lower mass bounds. Dielectron and dimuon channels offer a cleaner environment and thus yield stronger bounds. If lepton flavor violation is assumed the SM background is suppressed, which again strengthens the limits. This is reasoning behind the limits derived using LHC data.

We used the code fastlim described in [89], and adopted a parton distribution function at next-to-next leading [90, 91] in order to project the LHC sensitivity for a high-luminosity (HL) setup, following the recommendations presented in [92]. We stress that the HL-LHC limit refers to a detector similar to LHC running at 14 TeV with $3~ab^{-1}$ of integrated-luminosity. The High-Energy LHC configurations represents a 27 TeV colliding beam with $15~ab^{-1}$ of data. We highlight that these bounds are based on the simulated signal $q\bar{q} \rightarrow Z,\gamma \rightarrow \phi ^{++}\phi ^{--}$ as outlined in [88], which features bounds stronger than previous studies [93,94,95]. In summary we derived,

$$\begin{aligned} m_{S^{\pm \pm }}> & {} 943\,\mathrm{GeV} \,\, (\text {LHC}),\nonumber \\ m_{S^{\pm \pm }}> & {} 2.5\, \mathrm{TeV}\,\, (\text {HL-LHC})\nonumber \\ m_{S^{\pm \pm }}> & {} 4.9\, \mathrm{TeV}\,\, (\text {HE-LHC}) \end{aligned}$$

(3.5)

The latest LHC search for doubly charged scalars was with $12.9~fb^{-1}$ of data, but notice that with $36~fb^{-1}$ we expect LHC to already rule out doubly charged scalars with masses around 940 GeV, assuming a normal mass ordering for the active neutrinos. If we had considered an inverted mass ordering a weaker bound would have been found, lying around 900 GeV. This difference in the lower mass bound from LHC for normal and inverted ordering will not cause a meaningful impact on our conclusions and with this understanding in mind, we will simply quote those in Eq. (3.5). There are other important limits on this scenario [83, 96,97,98,99] but the ones we quote are the most relevant. For a more complete discussion of the collider bounds on the type II seesaw we refer to [100].

Moreover, it is exciting to see that HE-LHC can potentially probe doubly charged scalar with masses up to $\sim 5$ TeV. These limits are quite important and serve as an orthogonal test to the type I + type II seesaw scenario we are investigating because that restricts the region in which the seesaw mechanism is viable. Having in mind that the doubly charged scalar is key to the lepton flavor violation observables we are about to discuss, such collider bounds stand as a complementary and important cross-check to lepton flavor violation signatures.

4 Lepton flavor violation

Lepton flavor violation is one of the most interesting probes of physics beyond the SM. The main lepton flavor violation signatures of the seesaw mechanism stem from muon decay namely, $\mu \rightarrow e\gamma$ and $\mu \rightarrow 3e$. There are other sources of lepton flavor violation such as $\mu -e$ conversion but they are subdominant [101,102,103,104]. Other lepton flavor violating decays involving the $\tau$ lepton are less promising, unless one invokes a mechanism to significantly suppress $\mu \rightarrow e\gamma$ [105,106,107,108,109,110,111,112,113,114]. Anyway, going back to the relevant muon decays, $\mu \rightarrow e\gamma$ and $\mu \rightarrow 3e$, one can check that the current bounds read $BR(\mu \rightarrow e\gamma )< 4.2 \times 10^{-13}, BR(\mu \rightarrow 3e) < 10^{-12}$, and future experiments aim $BR(\mu \rightarrow e\gamma )< 4 \times 10^{-14}, BR(\mu \rightarrow 3e) < 10^{-16}$. Therefore, we expect an important experimental improvement in the near future. Eventually, we will superimpose these limits with the model’s contribution. That said, having in mind that we have a dominant type II seesaw setup, the first contribution to $\mu \rightarrow e\gamma$ in our models stems from 1-loop processes involving the doubly and singly charged scalars which lead to,

$$\begin{aligned} BR(\mu \rightarrow e\gamma )\simeq \frac{\alpha _{EM}\left| (f_{ab}^{*}\,f_{ab})_{e\mu }\right| ^{2}}{192\pi G_{F}^{2}\,}\left( \frac{1}{m_{S^{\pm \pm }}^{2}}+\frac{8}{m_{S ^{\pm }}^{2}}\right) ^{2}, \end{aligned}$$

(4.1)

where $\alpha _{EM}$ is the fine-structure constant, $G_{F}$ the Fermi constant, $m_{S^{\pm \pm }}$ the mass of doubly charged scalar, $m_{S^{\pm }}$ the mass of singly charged scalar. In what follows will assume that the doubly charged and singly charged scalar have the same mass. This assumption will allow us to connect $\mu \rightarrow e\gamma$ directly to $\mu \rightarrow$ 3e and LHC limits on the doubly charged scalar mass.

There is an additional source of lepton flavor violation that rises from charged current which reads,

$$\begin{aligned} \mathcal {L} = \frac{g}{2\sqrt{2}} \bar{l} \gamma ^\mu (1-\gamma _5) (\nu _R)^c W^{\prime -}, \end{aligned}$$

(4.2)

where $W^{\prime }$, is a charged gauge boson defined in Eq. (2.25). The $W^\prime$ boson. This interaction results in the following branching ratio,

$$\begin{aligned} \mathrm{BR} (\mu \rightarrow e\gamma )= \frac{3 (4\pi )^3 \alpha _{em}}{4 G_F^2}\left( |A^M_{e\mu }|^2 + |A^{E}_{e\mu }|^2\right) \end{aligned}$$

(4.3)

where,

$$\begin{aligned} A^M_{e\mu }= & {} \frac{-1}{(4\pi )^2}\sum _f \left( {g_{v\,1}^{fe}}^*g_{v\,1}^{f\mu } I_{f,\, 3}^{+\,+}+{g_{a\,1}^{fe}}^*g_{a\,1}^{f\mu } I_{f,\, 3}^{+\,-}\right) ,\nonumber \\ \end{aligned}$$

(4.4)

$$\begin{aligned} A^E_{e\mu }= & {} \frac{i}{(4\pi )^2}\sum _f \left( {g_{a\,1}^{fe}}^*g_{v\,1}^{f\mu } I_{f,\, 3}^{-\,+} + {g_{v\,1}^{fe}}^*g_{a\,1}^{f\mu } I_{f,\, 3}^{-\,-}\right) ,\nonumber \\ \end{aligned}$$

(4.5)

with $g_v$, $g_a$ being the couplings constants that encompass the constants in Eq. (4.2) and the neutrino mixing matrices, and $I_{f,\,3}^{\pm \,\pm }$ are functions defined in [75].

If right-handed neutrinos and the $W^\prime$ boson are very heavy, with masses much larger than the TeV scale one can plugging the numbers in Eq. (4.3) to show that $\mathrm{BR} (\mu \rightarrow e\gamma ) < 10^{-15}$, which is beyond reach current and projected experiments. This is the scenario we will consider throughout. In other words, the $\mathrm{BR} (\mu \rightarrow e\gamma )$ will be governed by the type II seesaw mechanism which features doubly charged and singly charged scalars contributions.

Bearing in mind that the right-handed neutrinos in our model will be heavy, and one can neglect the $W^\prime$ contribution to the $\mu \rightarrow e\gamma$ decay, we may focus on the seesaw component. However, another observable is important to our discussion, namely the $\mu \rightarrow 3e$ decay [75]. Since only the doubly charged Higgs contributes to this decay the calculation is simpler and leads to,

$$\begin{aligned} \mathrm{BR} (\mu \rightarrow 3e) =\frac{| f^{\nu \dagger }_{ee} f^{\nu }_{\mu e}|^2 }{G_F^2 m_{S^{\pm \pm }}^4}. \end{aligned}$$

(4.6)

On one hand we can see that the Yukawa couplings $f^{\nu }_{ab}$ dictate the lepton flavor violation observables, but on the other hand these Yukawa couplings enter in the neutrino mass matrix. Therefore, neutrino masses and lepton flavor violation observables are correlated.

Going back to Eq. (2.22), if $v_{s13}^2/\Lambda - v_{s11} \sim v_{s11}$ as we will assume throughout, we have a dominant type II seesaw setup with the active neutrino masses set by vacuum expectation value, $v_{s11}$, i,e. $(m_\nu )_{ab} \simeq \sqrt{2} v_{s11}\, f^\nu _{ab}$. The picture is not so simple because neutrinos oscillate and we need to reproduce the oscillation pattern. Therefore, these Yukawa couplings are found to be [102],

$$\begin{aligned} f^{\nu }_{ab}=\frac{1}{\sqrt{2}v_{s11}}(U^{*} diag(m_{\nu 1},m_{\nu 2},m_{\nu 3}) U^{\dagger })_{ab} \end{aligned}$$

(4.7)

where U is the Pontecorvo, Maki, Nakagawa, Sakata (PMNS) neutrino mixing matrix of dimension $3\times 3$, parametrized as follows [31],

$$\begin{aligned}&\left( \begin{array}{ccc} 1 &{} 0 &{} 0 \\ 0 &{} \cos \theta _{23} &{} \sin \theta _{23} \\ 0 &{} -\sin \theta _{23} &{} \cos \theta _{23} \end{array} \right) \left( \begin{array}{ccc} \cos \theta _{13} &{} 0 &{} \sin \theta _{13} e^{-i\delta } \\ 0 &{} 1 &{} 0 \\ -\sin \theta _{13} e^{i\delta } &{} 0 &{} \cos \theta _{13} \end{array} \right) \nonumber \\&\quad \left( \begin{array}{ccc} \cos \theta _{12} &{} \sin \theta _{12} &{} 0 \\ -\sin \theta _{12} &{} \cos \theta _{12} &{} 0 \\ 0 &{} 0 &{} 1 \end{array} \right) \end{aligned}$$

(4.8)

with the mixing parameters as shown in Table 1 [115].

Table 1 Table with the best-fit parameters that enter in the neutrino mass mixing according to [116]

In summary, one needs to incorporate neutrino oscillations to have more solid predictions for the lepton flavor violation observables. We will explore this fact by investigating several benchmark points which encompass normal and inverted mass ordering and different absolute neutrino masses. We will show that the neutrino mass spectrum is rather relevant to the overall lepton flavor violation signatures, a fact that has not been explored in detail in the context of 3-3-1 models. With this input from neutrino oscillations, our reasoning goes as follows:

• We choose a neutrino mass ordering;

• Then we pick a neutrino mass $m_{\nu 1}$, which then basically fixes $m_{\nu 2}$ and $m_{\nu 3}$, for a given vev, $v_{s11}=1-100$ eV. From this, we find the Yukawa couplings that reproduce this spectrum taking into account the oscillation patterns using Eq. (4.7);

• With these Yukawa couplings we use Eqs. (4.1)–(4.6) to compute the lepton flavor violating muon decays.

In Table 2 we summarize our findings using this logic. The table will allow the reader to easily follow our reasoning as we discuss the results in Fig. 1 where we display four of these benchmark points for $m_{\nu 1}=0.1$ eV and $m_{\nu 1}=0.01$ eV including normal and inverted mass ordering. The blue (red) lines are the model predictions for the muon decays for inverted (normal) neutrino mass hierarchies. The difference between the solid and dashed lines is the absolute mass for the neutrino flavor $\nu _1$. For instance, the red solid line in the left-panel of Fig. 1 accounts for the $BR(\mu \rightarrow e\gamma )$ for $m_{\nu 1}=0.01$ eV within a normal mass ordering. The gray region represents the region currently excluded by LHC based on the search for doubly charged scalars within a type II seesaw framework with normal mass ordering. The dashed and dotted vertical gray lines are the HL-LHC and HE-LHC projected exclusion limits. The black horizontal lines are the current and projected bounds on the $\mu \rightarrow e\gamma$ (left-panel) and $\mu \rightarrow 3e$ (right-panel) decays.

One can notice that the neutrino mass ordering has a great impact on the lepton flavor violating muon decays. Looking at the first benchmark scenario with $m_{\nu 1}=0.1$ eV and $v_{s11}=1$ eV, which assumes an inverted mass ordering (IO) we get BR$(\mu \rightarrow e\gamma )= 0.024/m_{S^{\pm \pm }}^4$, BR$(\mu \rightarrow 3e)= 7.5/m_{S^{\pm \pm }}^4$. Having in mind that the current experimental limits, we conclude that we can probe doubly charged scalars with masses of 600 GeV and 1.3 TeV using from $\mu \rightarrow e\gamma$ and $\mu \rightarrow 3e$ decays. It is exciting to see that in this setup $\mu \rightarrow 3e$ provides stronger bounds than the LHC.

Table 2 Table with the theoretical predictions for the $\mu \rightarrow e\gamma$ and $\mu \rightarrow 3e$ decay for the type $I+II$ seesaw setup encompassing normal (NO) and inverted (IO) mass ordering. We highlight that the mass of the doubly charged scalar is in GeV units. The neutrino masses in the IO scenario are given by $m_{\nu 2}^2=m_{\nu 1}^2-\Delta m_{21}^2$, $m_{\nu 3}^2=m_{\nu 2}^2-\Delta m_{23}^2$, whereas in NO case are $m_{\nu 2}^2=m_{\nu 1}^2-\Delta m_{21}^2$, $m_{\nu 3}^2=m_{\nu 1}^2+\Delta m_{31}^2$

Fig. 1

Figures showing the predictions for $\mu \rightarrow e\gamma$ (left-panel) and $\mu \rightarrow 3e$ (right-panel) decays in our model overlaid with the existing and projected bounds from collider and lepton flavor violation searches. We used the labels IO (NO) for inverted (normal) mass ordering for the neutrinos. LHC limit refers to the current LHC bound with $36~fb^{-1}$ of data. We also show the HL-LHC, and HE-LHC sensitivities which represent the LHC running with the $14~TeV-3~ab^{-1}$ and $27~TeV-15~ ab^{-1}$ configurations. It is clear from the figures that the neutrino mass ordering has a great impact on the theoretical predictions for lepton flavor violation. See text for a detailed discussion

However, the larger the vacuum $v_{s11}$ the smaller the Yukawa couplings needed to reproduce the same neutrino masses. Hence, when we set $v_{s11}=100$ eV, the predictions change drastically to BR$(\mu \rightarrow e\gamma )= 2.4\times 10^{-10}/m_{S^{\pm \pm }}^4$, BR$(\mu \rightarrow 3e)= 7.5\times 10^{-10}/m_{S^{\pm \pm }}^4$. For these scenarios where the vacuum of is much larger than 1 eV, LHC constitute the best probe. For concreteness, in this second case described above, taking $m_{S^{\pm \pm }}= 1000$ GeV, would lead to muon decays much smaller than current and projected sensitivity [75], making HL-LHC and HE-LHC the best laboratories, since HE-LHC will probe masses of about $\sim 5$ TeV, for instance. All these conclusions are quite visible in Fig. 1.

Considering the normal mass ordering (NO) we conclude that the qualitative statements do not change. Taking $m_{\nu 1}=0.1$ eV, $v_{s11}=1$ eV, quantitatively we notice that while the inverted hierarchy gives BR$(\mu \rightarrow e\gamma ) =0.024/m_{S^{\pm \pm }}$ we find BR$(\mu \rightarrow e\gamma ) =0.44/m_{S^{\pm \pm }}$, which is a factor of 20 larger. A larger much decay into 3e is also found for the NO compared to the IO (see Fig. 1). An orthogonal way to look at this is by noticing that the region between the current and projected limits delimit a signal region of lepton flavor violation. Looking at both panels of Fig. 1 we conclude that doubly charged scalars with masses around 1–2 TeV might be spotted at the $\mu \rightarrow e\gamma$ decay, whereas the $\mu \rightarrow 3e$ will be able to detect such scalars with masses of up to $\sim 30$ TeV. The experimental progress on the search for the $\mu \rightarrow 3e$ decay is remarkable and it will surpass even the HE-LHC regardless of the mass ordering for benchmark scenarios where $v_{s11}\sim 1$ eV.

In the left-panel of Fig. 1 one can clearly see the impact of changing the value of the neutrino masses in the $\mu \rightarrow e\gamma$ decay. This can be checked by comparing the ratio between the dashed blue and red lines with the solid blue and red lines. This is not true for the $\mu \rightarrow 3e$ decay though (see right-panel of Fig. 1). Comparing the NO and IO predictions for $m_{\nu 1}=0.01$ eV and $v_{s11}=1$ eV, we find no much difference, this is because the Yukawa couplings relevant for this observable are similar regardless of the neutrino mass ordering. This would continue to be true if we had taken even smaller values for $m_{\nu 1}$ because when $m_{\nu 1}$ is sufficiently small, the measured mass differences of the neutrino flavors govern by the neutrino mixings and thus the $\mu \rightarrow 3e$ decay. When we take $m_{\nu 1}=0.1$  eV, then the difference in the predictions for NO and IO is noticeable. One can easily use Table 2 and Fig. 1 to validate our conclusions.

We can conclude that regardless of the mass ordering and absolute value of the active neutrino masses that HE-LHC will solidly probe the type II seesaw model with respective to its contributions to the $\mu \rightarrow e\gamma$ decay. Concerning the $\mu \rightarrow 3e$ decay the situation changes due to the fantastic experimental sensitivity aimed in the near future. The $\mu \rightarrow 3e$ decay will be able to probe this model for doubly charged scalar masses up to 30 TeV, which is way beyond HE-LHC reach, no matter the neutrino mass ordering. If the absolute neutrino masses are much smaller than 0.01 eV, then $\mu \rightarrow 3e$ decay becomes smaller making HE-LHC still the best probe.

Our conclusions explicitly show the importance of searching for signs of lepton flavor violation in collider and muon decays. The conclusion about which probe yields stronger bounds depends strongly on the mass ordering adopted, the absolute neutrino masses and which much decay one considers. In the 1–5 TeV mass region of the doubly charged scalar lepton flavor violation experiments and colliders offer orthogonal and complementary probes. Thus if a signal is observed in one of the two new physics searches, the other will be able to assess whether is stems from a seesaw framework.

In summary, within the 3-3-1 model with right-handed neutrinos a type I + II seesaw naturally emerges. In the scenario where we have a dominant type II seesaw, the model offers a clear prediction for the $\mu \rightarrow e\gamma$ and $\mu \rightarrow 3e$ decays. One may wonder how one could discriminate our model from other type II seesaw proposal and a plausible answer would go as follows: setting aside the type II seesaw, our model predicts the existence of $W^\prime$ and $Z^\prime$ gauge bosons, as well as heavy exotic quarks. The detection of multiple signals consistent with all these particles could serve a discriminator and favor our models over others.

5 Conclusions

We have discussed a model which promotes $SU(2)_L\times U(1)_Y$ to $SU(3)_L\times U(1)_N$. In this extended gauge sector, all fermions get masses via a spontaneous symmetry breaking mechanism that encompasses three scalar triplets but neutrinos. A scalar sextet is added to incorporate neutrino masses. After spontaneous symmetry breaking this scalar sextet breaks down to a doublet and scalar triplet, which play a role in the type I and type II seesaw mechanism. We focus on a scenario of type II seesaw dominance where the relevant lepton flavor violation observables namely, $\mu \rightarrow e\gamma$ and $\mu \rightarrow 3e$, are directly tied to neutrino mass ordering and collider bounds on the doubly charged scalar.

We have explicitly shown how the absolute mass scale and neutrino mass ordering change the model predictions for lepton flavor violation within the type II seesaw framework. Combining the LHC, HL-LHC, HE-LHC sensitivity to doubly charged scalars and the experimental sensitivity to these rare muon decays we concluded that for doubly charged scalar with masses around 1–5 TeV, these probes are rather complementary. Moreover, regardless of the mass ordering HE-LHC is expected to solidly test any possible signal seen in these muon decays.

One may wonder how one could discriminate our model from other types II seesaw proposals and a plausible answer would rely on the existence of $W^\prime$ and $Z^\prime$ gauge bosons, as well as heavy exotic quarks, all predicted in our model. The detection of multiple signals consistent with all these particles could serve a discriminator and favor our models over others.

Appendix

In this appendix will describe in more detail and pedagogical manner some sections of the model which are relevant to our reasoning.

1.1 A.1 Lepton Masses

The leptons masses could in principle be generated via the Yukawa lagrangian,

$$\begin{aligned} \mathcal {L}_{Y}^{l}= & {} h_{ab}^{l}\overline{\psi }_{aL}\rho e_{bR}+h_{ab}^{\nu } \overline{\psi }_{aL}^{c}\psi _{bL}\rho \nonumber \\&+f_{ab}^{\nu }(\overline{\psi }_{aL} \,)_{m}(\psi _{bL}^{c}\,)_{n}S_{mn}+h.c., \end{aligned}$$

(A.1)

where $h_{ab}^{\nu }$ is an antisymmetric constant coupling matrix and $f_{ab}^{\nu }$ is a symmetric constant coupling matrix. The first and third terms conserve the lepton flavor, while the second term violate it. Some the scalars inside the scalar sextet S do have lepton number, for this reason the third term conserves lepton number as well.

The second term is not problematic because a set of discrete symmetries will be invoked, and consequently eliminate this term. This set of $Z_2$ symmetries are needed to avoid mixing between the SM quarks and the exotic ones. One of them requires $\rho \rightarrow \rho$, which forbids the second term above. Although, we also need to impose $e_R \rightarrow -e_R$ to generate masses for charged leptons.

In particular, the charged lepton masses arise from,

$$\begin{aligned} L_{mass}^{l}= & {} -h_{ab}^{l}\overline{\psi }_{aL}\rho e_{bR}+h.c.\nonumber \\= & {} -h_{ab}^{l}\left( \begin{array}{ccc} \overline{\nu }_{aL}&\overline{e}_{aL}&\overline{\nu }_{aR}^{c} \end{array} \right) \frac{1}{\sqrt{2}}\left( \begin{array}{c} 0 \\ v_{\rho } \\ 0 \end{array} \right) e_{bR}+h.c. \nonumber \\= & {} -h_{ab}^{l}\frac{v_{\rho }}{\sqrt{2}}\overline{e}_{aL}e_{bR}+h.c. \end{aligned}$$

(A.2)

In this way, in the flavor space the Dirac mass matrix for the charged leptons, at tree level, is given by

$$\begin{aligned} m_{l}=h_{ab}^{l}\frac{v_{\rho }}{\sqrt{2}}, \end{aligned}$$

(A.3)

where $v_{\rho }$ is the vacuum expectation value of the neutral scalar $\rho _{0}$ and $h_{ab}^{l}$ is the coupling constant matrix.

The neutrino masses come from the first second term only when the neutral components of scalar sextet acquire a vacuum expectation value as follows,

$$\begin{aligned}&L_{mass}^{\nu S} =-f_{ab}^{\nu }\left( \begin{array}{ccc} \overline{\nu }_{aL}&\overline{e}_{aL}&\overline{\nu }_{aR}^{c} \end{array} \right) \\&\qquad \times \frac{1}{\sqrt{2}}\left( \begin{array}{ccc} v_{s_{11}} &{} 0 &{} v_{s_{13}} \\ 0 &{} 0 &{} 0 \\ v_{s_{13}} &{} 0 &{} \Lambda \end{array} \right) \left( \begin{array}{c} \left( \nu _{bL}\right) ^{c} \\ \left( e_{bL}\right) ^{c} \\ \left[ \left( \nu _{b}^{c}\right) _{L}\right] ^{c} \end{array} \right) +h.c. \\&\quad =-f_{ab}^{\nu }\frac{v_{s_{11}}}{\sqrt{2}}\overline{\nu }_{aL}\left( \nu _{bL}\right) ^{c}-f_{ab}^{\nu }\frac{v_{s_{13}}}{\sqrt{2}}\overline{\nu }_{aL} \left[ \left( \nu _{b}^{c}\right) _{L}\right] ^{c}\\&\qquad -f_{ab}^{\nu }\frac{ v_{s_{13}}}{\sqrt{2}}\overline{\nu }_{aR}^{c}\left( \nu _{bL}\right) ^{c}-f_{ab}^{\nu }\frac{\Lambda }{\sqrt{2}}\overline{\nu }_{aR}^{c}\left[ \left( \nu _{b}^{c}\right) _{L}\right] ^{c}+h.c. \\&\quad =-f_{ab}^{\nu }\frac{v_{s_{11}}}{\sqrt{2}}\overline{\nu }_{aL}\nu _{bL}^{c}-f_{ab}^{\nu }\frac{v_{s_{13}}}{\sqrt{2}}\overline{\nu }_{aL}\nu _{bR}\\&\qquad -f_{ab}^{\nu }\frac{v_{s_{13}}}{\sqrt{2}}\overline{\nu }_{aR}^{c}\nu _{bL}^{c}-f_{ab}^{\nu }\frac{\Lambda }{\sqrt{2}}\overline{\nu }_{aR}^{c}\nu _{bR}+h.c. \end{aligned}$$

We may arrange the mass terms as,

$$\begin{aligned} L_{mass}^{\nu S}&=-\left( \begin{array}{cc} \overline{\nu }_{aL}&\overline{\nu }_{aR}^{c} \end{array} \right) \frac{1}{\sqrt{2}}\left( \begin{array}{cc} f_{ab}^{\nu }v_{s_{1}} &{} f_{ab}^{\nu }v_{s_{3}} \\ f_{ab}^{\nu }v_{s_{3}} &{} f_{ab}^{\nu }\Lambda \end{array} \right) \left( \begin{array}{c} \nu _{bL}^{c} \\ \nu _{bR} \end{array} \right) +h.c. \\&=-\left( \begin{array}{cc} \overline{\nu }_{aL}&\overline{\nu }_{aR}^{c} \end{array} \right) \frac{\sqrt{2}}{2}\left( \begin{array}{cc} v_{s_{1}}f_{ab}^{\nu } &{} v_{s_{3}}f_{ab}^{\nu } \\ v_{s_{3}}f_{ab}^{\nu } &{} \Lambda f_{ab}^{\nu } \end{array} \right) \left( \begin{array}{c} \nu _{bL}^{c} \\ \nu _{bR} \end{array} \right) +h.c. \\&=-\frac{1}{2}\left( \begin{array}{cc} \overline{\nu }_{aL}&\overline{\nu }_{aR}^{c} \end{array} \right) \left( \begin{array}{cc} \sqrt{2}v_{s_{1}}f_{ab}^{\nu } &{} \sqrt{2}v_{s_{3}}f_{ab}^{\nu } \\ \sqrt{2}v_{s_{3}}f_{ab}^{\nu } &{} \sqrt{2}\Lambda f_{ab}^{\nu } \end{array} \right) \left( \begin{array}{c} \nu _{bL}^{c} \\ \nu _{bR} \end{array} \right) +h.c. \\&=-\frac{1}{2}\left( \begin{array}{cc} \overline{\nu }_{aL}&\overline{\nu }_{aR}^{c} \end{array} \right) M_{\nu }\left( \begin{array}{c} \nu _{bL}^{c} \\ \nu _{bR} \end{array} \right) +h.c. \end{aligned}$$

With this result one can now easily understand Eq. (2.17).

1.2 A.2 Yukawa interactions of quarks

We mentioned in the paper that we needed to invoke some discrete symmetries to prevent mixing between the SM and exotic quarks. We will explain this statement in more detail now.

The lepton number conserving terms in the renormalizable Yukawa Lagrangian for the quarks sector are

$$\begin{aligned} L_{LNC}= & {} h_{\alpha a}^{u}\overline{Q}_{\alpha L}\rho ^{*}u_{aR}+h_{\alpha a}^{d}\overline{Q}_{\alpha L}\eta ^{*}d_{aR}\nonumber \\&+h^{U} \overline{Q}_{3L}\chi U_{R} +h_{a}^{d}\overline{Q}_{3L}\rho d_{aR}+h_{a}^{u}\overline{Q}_{3L}\eta u_{aR}\nonumber \\&+h_{\alpha \beta }^{D}\overline{Q}_{\alpha L}\chi ^{*}D_{\beta R}+h.c., \end{aligned}$$

(A.4)

while the lepton number violating terms of quarks are

$$\begin{aligned} L_{LNV}= & {} s_{a}^{u}\overline{Q}_{3L}\chi u_{aR}+s_{\alpha a}^{d}\overline{Q} _{\alpha L}\chi ^{*}d_{\beta R}+s^{U}\overline{Q}_{3L}\eta U_{R} \end{aligned}$$

(A.5)

$$\begin{aligned}&+s_{\alpha a}^{D}\overline{Q}_{\alpha L}\eta ^{*}D_{aR}+s_{\alpha }^{D} \overline{Q}_{3L}\rho D_{\alpha R}+s_{\alpha }^{U}\overline{Q}_{\alpha L}\rho ^{*}U_{R}+h.c.,\nonumber \\ \end{aligned}$$

(A.6)

where h and s are constant couplings.

One might notice that the terms in Eq. (A.5A.6) will give rise to mass mixing terms involving the SM and exotic quarks, which can be problematic because they will lead to changes in the properties of the SM quarks. Therefore, one needs to prevent that and to do so invoke some discrete symmetries. The discrete symmetries have to be such that keep all the desired mass terms for the SM quarks and neutrino masses but forbid these ones. The set of discrete symmetries is,

$$\begin{aligned} e_{aR}\rightarrow & {} -e_{aR}, \\ u_{aR}\rightarrow & {} -u_{aR}, \\ d_{aR}\rightarrow & {} -d_{aR}, \\ U_{R}\rightarrow & {} U_{R}, \\ D_{aR}\rightarrow & {} D_{aR}, \\ \eta\rightarrow & {} -\eta , \\ \rho\rightarrow & {} -\rho , \\ \chi\rightarrow & {} \chi , \end{aligned}$$

where $\alpha =1,2; a=1,2,3.$

From Eq. (A.4), after spontaneous symmetry breaking we find,

$$\begin{aligned} L_{mass}^{Q}= & {} \frac{v_{\rho }}{\sqrt{2}}h_{\alpha a}^{u}\overline{u} _{\alpha L}u_{aR}-\frac{v_{\eta }}{\sqrt{2}}h_{\alpha a}^{d}\overline{d} _{\alpha L}d_{aR}-\frac{v_{\eta }}{\sqrt{2}}h_{a}^{u}\overline{u}_{3L}u_{aR} \nonumber \\&-\frac{v_{\rho }}{\sqrt{2}}h_{a}^{u}\overline{d}_{3L}d_{aR}-\frac{v_{\chi } }{\sqrt{2}}h_{U}\overline{U}_{L}U_{R}\nonumber \\&-\frac{v_{\chi }}{\sqrt{2}}h_{\alpha \beta }^{D}\overline{D}_{\alpha L}D_{\beta R}+h.c.. \end{aligned}$$

(A.7)

Thus, we can write the SM quark mass as follows,

$$\begin{aligned}&L_{mass}^{u}=-\frac{1}{\sqrt{2}}\left( \begin{array}{ccc} \overline{u}_{1L}&\overline{u}_{2L}&\overline{u}_{3L} \end{array} \right) \nonumber \\&\quad \times \left( \begin{array}{ccc} -v_{\rho }h_{11}^{u} &{} -v_{\rho }h_{12}^{u} &{} -v_{\rho }h_{13}^{u} \\ -v_{\rho }h_{21}^{u} &{} -v_{\rho }h_{22}^{u} &{} -v_{\rho }h_{23}^{u} \\ v_{\eta }h_{31}^{u} &{} v_{\eta }h_{32}^{u} &{} v_{\eta }h_{33}^{u} \end{array} \right) \left( \begin{array}{c} u_{1R} \\ u_{2R} \\ u_{3R} \end{array} \right) +h.c., \end{aligned}$$

(A.8)

which leads to,

$$\begin{aligned} M_{u}=\frac{1}{\sqrt{2}}\left( \begin{array}{ccc} -v_{\rho }h_{11}^{u} &{} -v_{\rho }h_{12}^{u} &{} -v_{\rho }h_{13}^{u} \\ -v_{\rho }h_{21}^{u} &{} -v_{\rho }h_{22}^{u} &{} -v_{\rho }h_{23}^{u} \\ v_{\eta }h_{31}^{u} &{} v_{\eta }h_{32}^{u} &{} v_{\eta }h_{33}^{u} \end{array} \right) . \end{aligned}$$

(A.9)

Similarly, for the down-type quarks we get,

$$\begin{aligned}&L_{mass}^{d}=-\frac{1}{\sqrt{2}}\left( \begin{array}{ccc} \overline{d}_{1L}&\overline{d}_{2L}&\overline{d}_{3L} \end{array} \right) \nonumber \\&\quad \times \left( \begin{array}{ccc} v_{\eta }h_{11}^{d} &{} v_{\eta }h_{12}^{d} &{} v_{\eta }h_{13}^{d} \\ v_{\eta }h_{21}^{d} &{} v_{\eta }h_{22}^{d} &{} v_{\eta }h_{23}^{d} \\ v_{\rho }h_{31}^{d} &{} v_{\rho }h_{32}^{d} &{} v_{\rho }h_{33}^{d} \end{array} \right) \left( \begin{array}{c} d_{1R} \\ d_{2R} \\ d_{3R} \end{array} \right) +h.c., \end{aligned}$$

(A.10)

with,

$$\begin{aligned} M_{d}=\frac{1}{\sqrt{2}}\left( \begin{array}{ccc} v_{\eta }h_{11}^{d} &{} v_{\eta }h_{12}^{d} &{} v_{\eta }h_{13}^{d} \\ v_{\eta }h_{21}^{d} &{} v_{\eta }h_{22}^{d} &{} v_{\eta }h_{23}^{d} \\ v_{\rho }h_{31}^{d} &{} v_{\rho }h_{32}^{d} &{} v_{\rho }h_{33}^{d} \end{array} \right) . \end{aligned}$$

(A.11)

We can see that the type-up and type-down quarks are associated with the VEVs $v_{\eta }$ and $v_{\rho },$ related to the electroweak scale.

From the Eq. (A.7), we can write the U extra quark mass term

$$\begin{aligned} M_{U}=\frac{1}{\sqrt{2}}v_{\chi }h_{U}. \end{aligned}$$

(A.12)

From the Eq. (A.7), we can write the D extra quark mass lagrangian

$$\begin{aligned} L_{mass}^{D}=-\frac{1}{\sqrt{2}}\left( \begin{array}{cc} \overline{D}_{1L}&\overline{D}_{2L} \end{array} \right) \left( \begin{array}{cc} v_{\chi }h_{11}^{D} &{} v_{\chi }h_{12}^{D} \\ v_{\chi }h_{21}^{D} &{} v_{\chi }h_{22}^{D} \end{array} \right) \left( \begin{array}{c} D_{1R} \\ D_{2R} \end{array} \right) +h.c.. \end{aligned}$$

The D quark mass matrix in the basis ($D_{1},D_{2})$ is

$$\begin{aligned} M_{D}=\frac{1}{\sqrt{2}}\left( \begin{array}{cc} v_{\chi }h_{11}^{D} &{} v_{\chi }h_{12}^{D} \\ v_{\chi }h_{21}^{D} &{} v_{\chi }h_{22}^{D} \end{array} \right) . \end{aligned}$$

Notice that the masses of D quarks will depend the VEV $v_{\chi },$ related to the TeV-scale.

1.3 A.3 Scalar sector

Considering the three scalar fields ($\chi , \eta , \rho )$, the Higgs potential more general, renormalizable and invariant on the $SU(3)_{L}\otimes U(1)_{X}$ symmetry group is

$$\begin{aligned}&V\left( \eta ,\chi ,\rho \right) =\mu _{\chi }^{2}\chi ^{\dagger }\chi +\mu _{\eta }^{2}\eta ^{\dagger }\eta +\mu _{\rho }^{2}\rho ^{\dagger }\rho \nonumber \\&\quad +\lambda _{1}\left( \chi ^{\dagger }\chi \right) ^{2}+\lambda _{2}\left( \eta ^{\dagger }\eta \right) ^{2}+\lambda _{3}\left( \rho ^{\dagger }\rho \right) ^{2} \nonumber \\&\quad +\lambda _{4}\left( \chi ^{\dagger }\chi \right) \left( \eta ^{\dagger }\eta \right) +\lambda _{5}\left( \chi ^{\dagger }\chi \right) \left( \rho ^{\dagger }\rho \right) \nonumber \\&\quad +\lambda _{6}\left( \eta ^{\dagger }\eta \right) \left( \rho ^{\dagger }\rho \right) +\lambda _{7}\left( \chi ^{\dagger }\eta \right) \left( \eta ^{\dagger }\chi \right) \nonumber \\&\quad +\lambda _{8}\left( \chi ^{\dagger }\rho \right) \left( \rho ^{\dagger }\chi \right) +\lambda _{9}\left( \eta ^{\dagger }\rho \right) \left( \rho ^{\dagger }\eta \right) \nonumber \\&\quad -\frac{f}{\sqrt{2}}\varepsilon ^{ijk}\eta _{i}\rho _{j}\chi _{k}+h.c. \nonumber \\&\quad +\mu _{4}^{2}\left( \chi ^{\dag }\eta +\eta ^{\dag }\chi \right) +\lambda _{10}\left( \chi ^{\dagger }\chi \right) \left( \chi ^{\dag }\eta +\eta ^{\dag }\chi \right) \nonumber \\&\quad +\lambda _{11}\left( \eta ^{\dagger }\eta \right) \left( \chi ^{\dag }\eta +\eta ^{\dag }\chi \right) +\lambda _{12}\left( \rho ^{\dagger }\rho \right) \left( \chi ^{\dag }\eta +\eta ^{\dag }\chi \right) \nonumber \\&\quad +\lambda _{13}\left[ \left( \chi ^{\dag }\eta \right) \left( \chi ^{\dag }\eta \right) +\left( \eta ^{\dag }\chi \right) \left( \eta ^{\dag }\chi \right) \right] \nonumber \\&\quad +\lambda _{14}\left[ \left( \rho ^{\dag }\chi \right) \left( \eta ^{\dag }\rho \right) +\left( \rho ^{\dag }\eta \right) \left( \chi ^{\dag }\eta \right) \right] . \end{aligned}$$

(A.13)

where $\mu _{i}$ are constants, $\lambda _{i}$ and f are constant couplings. Furthermore, the additional terms of potential with sextet scalar, as combinations with the others scalar triplets, are

$$\begin{aligned}&V_{S}=\mu _{S}^{2}Tr\left( S^{\dagger }S\right) +\lambda _{10}Tr[\left( S^{\dagger }S\right) ^{2}]+\lambda _{11}\left[ Tr\left( S^{\dagger }S\right) \right] ^{2} \nonumber \\&\quad +\left[ \lambda _{15}\left( \eta ^{\dagger }\eta \right) +\lambda _{16}\left( \rho ^{\dagger }\rho \right) +\lambda _{17}\left( \chi ^{\dagger }\chi \right) \right] Tr\left( S^{\dagger }S\right) \nonumber \\&\quad +\lambda _{18}\left( \chi ^{\dag }\eta +\eta ^{\dag }\chi \right) Tr\left( S^{\dagger }S\right) \nonumber \\&\quad +\lambda _{S}\left( \varepsilon ^{ijk}\varepsilon ^{imn}\rho _{n}\rho _{k}S_{li}S_{mj}+h.c.\right) +\lambda _{19}\left( \chi ^{\dagger }S\right) \left( S^{\dagger }\chi \right) \nonumber \\&\quad +\lambda _{20}\left( \eta ^{\dagger }S\right) \left( S^{\dagger }\eta \right) +\lambda _{21}\left( \rho ^{\dagger }S\right) \left( S^{\dagger }\rho \right) \nonumber \\&\quad +\lambda _{22}(\varepsilon ^{ijk}\eta _{m}^{*}S_{mi}\rho _{j}\chi _{k}+h.c.)\nonumber \\&\quad +\lambda _{23}(\varepsilon ^{ijk}\chi _{m}^{*}S_{mi}\rho _{j}\eta _{k}+h.c.) \nonumber \\&\quad +M_{1}\eta ^{T}S^{\dagger }\eta +M_{2}\chi ^{T}S^{\dagger }\chi +M_{3}\eta ^{\dagger }S\eta ^{*}\nonumber \\&\quad +M_{4}\chi ^{\dagger }S\chi ^{*}. \end{aligned}$$

(A.14)

The permitted terms of the scalar potential by the discrete symmetry are

$$\begin{aligned}&V_{1}\left( \eta ,\chi ,\rho ,S\right) =\mu _{\chi }^{2}\chi ^{\dagger }\chi +\mu _{\eta }^{2}\eta ^{\dagger }\eta +\mu _{\rho }^{2}\rho ^{\dagger }\rho \nonumber \\&\quad +\lambda _{1}\left( \chi ^{\dagger }\chi \right) ^{2}+\lambda _{2}\left( \eta ^{\dagger }\eta \right) ^{2}+\lambda _{3}\left( \rho ^{\dagger }\rho \right) ^{2} \nonumber \\&\quad +\lambda _{4}\left( \chi ^{\dagger }\chi \right) \left( \eta ^{\dagger }\eta \right) +\lambda _{5}\left( \chi ^{\dagger }\chi \right) \left( \rho ^{\dagger }\rho \right) \nonumber \\&\quad +\lambda _{6}\left( \eta ^{\dagger }\eta \right) \left( \rho ^{\dagger }\rho \right) +\lambda _{7}\left( \chi ^{\dagger }\eta \right) \left( \eta ^{\dagger }\chi \right) \nonumber \\&\quad +\lambda _{8}\left( \chi ^{\dagger }\rho \right) \left( \rho ^{\dagger }\chi \right) +\lambda _{9}\left( \eta ^{\dagger }\rho \right) \left( \rho ^{\dagger }\eta \right) \nonumber \\&\quad -\frac{f}{\sqrt{2}}\varepsilon ^{ijk}\eta _{i}\rho _{j}\chi _{k}+h.c. \nonumber \\&\quad +\lambda _{13}\left[ \left( \chi ^{\dag }\eta \right) \left( \chi ^{\dag }\eta \right) +\left( \eta ^{\dag }\chi \right) \left( \eta ^{\dag }\chi \right) \right] \nonumber \\&\quad +\mu _{S}^{2}Tr\left( S^{\dagger }S\right) +\lambda _{10}Tr[\left( S^{\dagger }S\right) ^{2}]+\lambda _{11}\left[ Tr\left( S^{\dagger }S\right) \right] ^{2} \nonumber \\&\quad +\left[ \lambda _{15}\left( \eta ^{\dagger }\eta \right) +\lambda _{16}\left( \rho ^{\dagger }\rho \right) +\lambda _{17}\left( \chi ^{\dagger }\chi \right) \right] Tr\left( S^{\dagger }S\right) \nonumber \\&\quad +\lambda _{S}\left( \varepsilon ^{ijk}\varepsilon ^{imn}\rho _{n}\rho _{k}S_{li}S_{mj}+h.c.\right) +\lambda _{19}\left( \chi ^{\dagger }S\right) \left( S^{\dagger }\chi \right) \nonumber \\&\quad +\lambda _{20}\left( \eta ^{\dagger }S\right) \left( S^{\dagger }\eta \right) +\lambda _{21}\left( \rho ^{\dagger }S\right) \left( S^{\dagger }\rho \right) \nonumber \\&\quad +\lambda _{22}(\varepsilon ^{ijk}\eta _{m}^{*}S_{mi}\rho _{j}\chi _{k}+h.c.)\nonumber \\&\quad +\lambda _{23}(\varepsilon ^{ijk}\chi _{m}^{*}S_{mi}\rho _{j}\eta _{k}+h.c.) \nonumber \\&\quad +M_{1}\eta ^{T}S^{\dagger }\eta +M_{2}\chi ^{T}S^{\dagger }\chi +M_{3}\eta ^{\dagger }S\eta ^{*}\nonumber \\&\quad +M_{4}\chi ^{\dagger }S\chi ^{*}. \end{aligned}$$

(A.15)

In this model, the scalar triplets develop non-trivial vacuum expectation values (VEV), in order to engender correct spontaneous symmetry breaking (SSB), as written below

$$\begin{aligned} \left\langle \chi \right\rangle= & {} \frac{1}{\sqrt{2}}\left( \begin{array}{c} 0 \\ 0 \\ v_{\chi } \end{array} \right) ,\quad \left\langle \eta \right\rangle =\frac{1}{\sqrt{2}}\left( \begin{array}{c} v_{\eta } \\ 0 \\ 0 \end{array} \right) ,\nonumber \\ \left\langle \rho \right\rangle= & {} \frac{1}{\sqrt{2}}\left( \begin{array}{c} 0 \\ v_{\rho } \\ 0 \end{array} \right) . \end{aligned}$$

(A.16)

Observe that the scalar triplet $\chi$ develops VEV only on the third neutral component, while $\eta$ develops VEV only on the first neutral component. The steps of symmetry breaking transition is given by $SU(3)_{L}\otimes U(1)_{X}\overset{\left\langle \chi \right\rangle }{ \rightarrow }SU(2)_{L}\otimes U(1)_{Y}$, while $SU(2)_{L}\otimes U(1)_{Y} \overset{\left\langle \eta \right\rangle ,\left\langle \rho \right\rangle }{ \rightarrow }U(1)_{Q},$ i.e., the triplet $\chi$ develops VEV breaking the $3-3-1$ gauge symmetry to SM, while the triplets $\eta$ and $\rho$ develop VEV breaking the SM gauge symmetry to the QED (Quantum Electrodynamics).

Regarding the scalar sextet, the three neutral components develop VEVs in the following way:

$$\begin{aligned} \left\langle S\right\rangle =\frac{1}{\sqrt{2}}\left( \begin{array}{lll} v_{s_{1}} &{} 0 &{} v_{s_{3}} \\ 0 &{} 0 &{} 0 \\ v_{s_{3}} &{} 0 &{} \Lambda \end{array} \right) \end{aligned}$$

(A.17)

We will see that $v_{s_{1},}$ $v_{s_{3}}$ and $\Lambda$ are responsible for the mass for the left-handed neutrinos, while $\Lambda$ is responsible for the right-handed neutrinos Dirac masses. After the SSB of $SU(3)_{L}\otimes U(1)_{X}$ to $SU(2)_{L}\otimes U(1)_{Y}$, the scalar sextet results in a triplet plus a doublet and a singlet ($\mathbf {6}\rightarrow \mathbf {3}+ \mathbf {2}+\mathbf {1})$, as follows

$$\begin{aligned} S\rightarrow S_{1(\mathbf {1},\mathbf {3},-2)}+S_{2(\mathbf {1},\mathbf {2} ,-1)}+S_{3(\mathbf {1},\mathbf {1},0)}, \end{aligned}$$

(A.18)

where

$$\begin{aligned} S_{1}=\left( \begin{array}{ll} S_{11}^{0} &{} S_{12}^{-} \\ S_{12}^{-} &{} S_{22}^{--} \end{array} \right) ,\,S_{2}=\left( \begin{array}{l} S_{13}^{0} \\ S_{23}^{-} \end{array} \right) ,\,S_{3}=S_{33}^{0}. \end{aligned}$$

(A.19)

In this model, the lepton number distribution of the scalars is

$$\begin{aligned} L(\eta _{3}^{0},S_{33}^{0},\rho _{3}^{+})= & {} -2, \end{aligned}$$

(A.20)

$$\begin{aligned} L(\chi _{1}^{0},\chi _{2}^{-},S_{11}^{0},S_{12}^{-},S_{22}^{--})= & {} +2. \end{aligned}$$

(A.21)

We can see that $S_{12}^{-}$ and $S_{22}^{--}$ carry two lepton numbers. Both are essential in our analysis of charged lepton flavor violating decay of muon.

1.4 A.4 Charged lepton flavor interactions

The Yukawa Lagrangian with charged lepton flavor violation (CLFV) is given by

$$\begin{aligned} L_{CLFV}\supset f_{ab}^{\nu }(\overline{\psi }_{aL}^{c})_{m}(\psi _{bL})_{n}(S^{*})_{mn}+h.c., \end{aligned}$$

(A.22)

where $a,b=1,2,3$ indicates the lepton generations and $m,n=1,2,3$ indicates the entries of the sextet, $f_{ab}$ is symmetric. The CLFV interactions in the $\mu -e$ sector results from taking $a = 1 , b = 2$ in the above equation,

$$\begin{aligned} f_{ab}\overline{\left( \psi _{aL}\right) _{m}^{c}}\left( \psi _{bL}\right) _{n}S_{mn}^{*}\overset{a=1,b=2}{\rightarrow }f_{12}\overline{\left( e_{L}\right) ^{c}}\left( \mu _{L}\right) S^{++}. \end{aligned}$$

Using the following relations:

$$\begin{aligned} \left( \Psi ^{c}\right) _{L}&=\left( \Psi _{R}\right) ^{c} \\ \left( \Psi ^{c}\right) _{R}&=\left( \Psi _{L}\right) ^{c} \\ \left( \overline{\Psi }^{c}\right) _{L}&=\left( \overline{\Psi }_{R}\right) ^{c} \\ \left( \overline{\Psi }^{c}\right) _{R}&=\left( \overline{\Psi }_{L}\right) ^{c}. \end{aligned}$$

we have,

$$\begin{aligned}&f_{12} \overline{\left( e_{L}\right) ^{c}}\left( \mu _{L}\right) S^{++} =f_{12}\overline{\left( e^{c}\right) _{R}}\left( \mu _{L}\right) S^{++} \nonumber \\&\quad =f_{12}\left[ \left( e^{c}\right) _{R}\right] ^{\dag }\gamma ^{0}\left( \mu _{L}\right) S^{++} \nonumber \\&\quad =f_{12}\left[ \frac{1}{2}\left( 1+\gamma _{5}\right) e^{c}\right] ^{\dag }\gamma ^{0}\left( \mu _{L}\right) S^{++} \nonumber \\&\quad =f_{12}\left[ \frac{1}{2}\left( 1+\gamma _{5}\right) e^{c}\right] ^{\dag }\gamma ^{0}\frac{1}{2}\left( 1-\gamma _{5}\right) \mu S^{++} \nonumber \\&\quad =f_{12}\left( e^{c\dag }\right) \frac{1}{2}\left( 1-\gamma _{5}\right) \gamma ^{0}\frac{1}{2}\left( 1-\gamma _{5}\right) \mu S^{++} \nonumber \\&\quad =f_{12}\left( e^{c\dag }\gamma ^{0}\right) \frac{1}{2}\left( 1-\gamma _{5}\right) \mu S^{++} \nonumber \\&\quad =f_{12}\overline{e^{c}}\frac{1}{2}\left( 1-\gamma _{5}\right) \mu S^{++} \nonumber \\&f_{12} \overline{\left( e_{L}\right) ^{c}}\left( \mu _{L}\right) S^{++} =\frac{1}{2}f_{12}\overline{e^{c}}\mu S^{++}-\frac{1}{2}f_{12}\overline{ e^{c}}\gamma _{5}\mu S^{++}. \end{aligned}$$

(A.23)

We see that these terms directly induce $\mu \rightarrow e\gamma$ lepton flavor violating decay mediated by the doubly charged scalar $S^{++}$. Proceeding in an analogous way, we can easily obtain the relevant CLFV terms mediated by the singly charged scalar $S ^+$.