Search for $Z'$ pair production from scalar boson decay in minimal $U(1)_{L_\mu - L_\tau }$ model at the LHC

Abstract

We consider a model with gauged $L_\mu ^{} - L_\tau ^{}$ symmetry in which the symmetry is spontaneously broken by a scalar field. The decay of the scalar boson into two new gauge bosons is studied as a direct consequence of the spontaneous symmetry breaking. Then, a possibility of searches for the gauge and scalar bosons through such a decay at the LHC experiment is discussed. We consider the case that the mass range of the gauge boson is ${\mathcal {O}}(10)$ GeV, which is motivated by anomalies reported by LHCb. We will show that the signal significance of the searches for the gauge boson and scalar boson reach to 3 and 5 for the scalar mixing 0.012 and 0.015, respectively.

1 Introduction

The muon Number minus the tau Number ($L_\mu ^{} - L_\tau ^{}$) gauge symmetry [1,2,3,4] has been gaining an attention as an extension of the Standard Model (SM) of particle physics. It has been shown that some observed anomalies can be explained by a new gauge boson $Z'$ from the symmetry. One can explain the anomalous magnetic of muon (muon $g-2$) by the interaction between $Z'$ and muon at one-loop level [5,6,7], and cosmic neutrino spectrum observed at IceCube can be settled by the interaction between $Z'$ and neutrinos [8,9,10,11,12]. Moreover the anomalies observed in $B \rightarrow K^{(*)} \ell ^+ \ell ^-$ decay (B anomaly) can be explained when we include some extra field contents [13,14,15,16,17,18,19]. Searches for such $Z'$ boson have been studied for on-going and future electron collider [20,21,22,23,24,25,26,27], muon collider [28], hadron collider [29,30,31], beam dump [32, 33], neutrino beam experiments [34,35,36] and meson decay [37], atmospheric neutrino observations [38], astrophysical observations [39].

Although the favored range of the $Z'$ mass is different for explaining each anomaly, the gauge boson is required to be massive to explain them. Therefore the origin of the gauge boson mass should be implemented in full models. Spontaneous symmetry breaking is one of the natural mechanisms to generate the gauge boson mass. Then, the scalar sector of the SM also needs to be enlarged by introducing new scalar field(s). Observations of such the scalar boson as well as the gauge boson will provide important hints of the origin of particle mass and interaction.

In spontaneous symmetry breaking mechanism, the gauge boson $Z'$ acquires a mass after a new scalar field breaks the $L_\mu ^{} - L_\tau ^{}$ gauge symmetry by developing a vacuum expectation value (vev). The gauge boson mass term emerges from the interaction term of two scalar bosons ($\phi$) and two gauge bosons ($\phi -\phi -Z'-Z'$) by replacing the scalar field with its vev. Then, as a direct consequence of the spontaneous symmetry breaking, the interaction of the massive scalar boson with two gauge bosons ($\phi -Z'-Z'$) also emerges. By this interaction term, the scalar boson can decay into two gauge bosons if kinematically allowed. Such a decay is a clear signature of the spontaneous symmetry breaking or the mass generation. An interesting feature of this interaction is that the decay of $\phi$ into two $Z'$ can be enhanced when the gauge boson is lighter than the scalar boson. This is because the longitudinal component of $Z'$, which is the Nambu–Goldstone mode of $\phi$, is inversely proportional to its mass. In the case of very light $Z'$ compared with $\phi$, the decay width into two $Z'$ can dominate other decay widths. Then, $\phi$ dominantly decays into two $Z'$.

The scalar boson, on the other hand, can be produced at collider experiments through a mixing with the SM Higgs boson. In general such a mixing is expected to exist because quartic terms of the SM Higgs boson and $\phi$ are allowed by the symmetry. The mixing can be simultaneously generated through the quartic terms after the spontaneous symmetry breaking. Once the scalar boson is produced, it can dominantly decay into the lighter gauge bosons when the decay widths into the SM particles are suppressed. In the case that the gauge boson further decays into charged SM particles, the mass of the scalar boson and gauge boson can be reconstructed from the energies and momenta of the SM particles.

Searches for the scalar boson and gauge boson through the above decay at the LHC and ILC have been studied for the $L_\mu ^{} - L_\tau ^{}$ gauge boson with mass lighter than 200 MeV in ref. [40] where the mass region is favored to explain muon $g-2$. We found that such a light gauge boson decaying into charged leptons is difficult to identify in the LHC experiments while the scalar and gauge boson masses can be reconstructed at the ILC experiment using energy-momentum conservation. Thus, the confirmation of the $\phi -Z'-Z'$ interaction will be possible at the ILC. Similar studies for dark photon from a light scalar boson decay at the FASER experiment have been done in [41]. Since the above studies are focused on light gauge bosons, it is also important to search for the signal when the $L_\mu ^{} - L_\tau ^{}$ gauge boson is heavier than ${\mathcal {O}}(1)$ GeV, which is motivated to explain B anomaly. In this paper, we consider such a heavier $Z'$ with mass ${\mathcal {O}}(10)$ GeV in a minimal $L_\mu ^{} - L_\tau ^{}$ model, and study a possibility of the search for both the gauge and scalar bosons at the LHC.

This paper is organized as follows. In Sect. 2, we review a minimal gauged $L_\mu - L_\tau$ model summarizing interactions and mass spectrum. In Sect. 3, we show decay widths of new bosons and the SM-like Higgs bosons. In Sect. 4, we discuss experimental constraints for new gauge coupling, $Z'$ mass and scalar mixing. In Sect. 5, we carry out numerical simulation for our signal and background, and the estimate discovery significance. Our conclusion is given in Sect. 6.

2 Minimal $L_\mu - L_\tau$ model

We start our discussion with introducing a minimal gauged $L_\mu ^{} - L_\tau ^{}$ model, where $L_\mu$ and $L_\tau$ represent the muon $(\mu )$ and tau $(\tau )$ number, respectively. The gauge symmetry of the model is defined by adding $U(1)_{L_\mu - L_\tau }$ to that of the SM. Under the $L_\mu ^{} - L_\tau ^{}$ symmetry, only muon and tau flavour leptons among the SM particles are charged. As a minimal setup, the scalar sector is also extended by introducing one complex scalar, $\varphi$, which is charged under the $L_\mu ^{} - L_\tau ^{}$ symmetry and is singlet under the SM gauge symmetry.¹ The gauge charge assignment for the fermions and scalars under the weak $SU(2)_W$ and the hypercharge $U(1)_Y$ symmetries is shown in Table 1. In the table, $Q_L$ and $u_R,~d_R$ are $SU(2)_W$ doublet left-handed quarks and singlet right-handed up-type, down-type quarks, and $l_\alpha$ and $\alpha _R$ $(\alpha = e,~\mu ,~\tau )$ are $SU(2)_W$ doublet left-handed and singlet right-handed charged leptons, respectively. Here, e stands for electron flavour. The $SU(2)_W$ doublet scalar, H, is responsible for the EW symmetry breaking while the singlet scalar, $\varphi$, is for the $L_\mu ^{} - L_\tau ^{}$ symmetry breaking. The strong interaction part of the model is the same as that of the SM.

Table 1 The gauge charge assignment of a gauged minimal $U(1)_{L_\mu - L_\tau }$ model

2.1 Lagrangian

The Lagrangian of the model takes the form of

$$\begin{aligned} {\mathcal {L}}&= {\mathcal {L}}_{\mathrm {SM}}- \frac{1}{4} Z'_{\mu \nu } Z'^{\mu \nu } + g' Z'_\mu J^\mu _{L_\mu ^{} - L_\tau ^{}} \nonumber \\&\quad - \frac{\epsilon }{2} B_{\mu \nu } Z'^{\mu \nu } +|D_\mu \varphi |^2 - V, \end{aligned}$$

(1)

where ${\mathcal {L}}_{\mathrm {SM}}$ represents the SM Lagrangian except for the Higgs potential, and $Z'$ and B stand for the $L_\mu ^{} - L_\tau ^{}$ and hypercharge gauge boson and its field strength in interaction basis, respectively. The $L_\mu ^{} - L_\tau ^{}$ gauge coupling constant is denoted as $g'$ and its gauge current $J_{L_\mu ^{} - L_\tau ^{}}^\mu$ is given by

$$\begin{aligned} J^\mu _{L_\mu ^{} - L_\tau ^{}} = {\bar{l}}_\mu \gamma ^\mu l_\mu + {\bar{\mu }}_R \gamma ^\mu \mu _R - {\bar{l}}_\tau \gamma ^\mu l_\tau - {\bar{\tau }}_R \gamma ^\mu \tau _R. \end{aligned}$$

(2)

The gauge kinetic mixing term between two U(1) gauge bosons is allowed by the symmetry, which is parametrized by $\epsilon$. The covariant derivative for the kinetic term of $\varphi$ is given by

$$\begin{aligned} D_\mu = \partial _\mu - i g' Z'_\mu . \end{aligned}$$

(3)

The scalar potential V is given by

$$\begin{aligned} V&= -\mu _H^2 H^\dagger H - \mu _\varphi ^2 \varphi ^* \varphi + \frac{\lambda _H}{2} (H^\dagger H)^2 \nonumber \\&\quad + \frac{\lambda _\varphi }{2} (\varphi ^* \varphi )^2 + \lambda _{H \varphi } (H^\dagger H)(\varphi ^* \varphi ). \end{aligned}$$

(4)

where $\mu _H$ and $\mu _\varphi$ are the mass parameters of H and $\varphi$, and $\lambda _H,~\lambda _\varphi$ and $\lambda _{H\varphi }$ are the quartic couplings, respectively.

In the following discussion, we assume that the values of $\mu _H^2$ and $\mu _\varphi ^2$ as well as $\lambda _H,~\lambda _\varphi$ and $\lambda _{H\varphi }$, are set appropriately so that the scalar fields develop vevs to break the EW and $L_\mu ^{} - L_\tau ^{}$ symmetries on a stable vacuum. Furthermore, to simplify our analysis, we assume that the gauge kinetic mixing term is absent at tree-level. Even though, such a kinetic mixing can be generated radiatively via muon and tau loops. The loop-induced kinetic mixing with photon at one loop level is given by

$$\begin{aligned} \epsilon (q^2)&= \frac{8 e g'}{(4 \pi )^2} \int _0^1 dx x(1-x) \log \left[ \frac{m_\tau ^2 -x (1-x) q^2}{m_\mu ^2 -x (1-x) q^2} \right] , \end{aligned}$$

(5)

where $q^2$ is the four momentum squared carried by $Z'$ and e is the electric charge of proton.² Masses of muon and tau leptons are denoted as $m_{\mu }$ and $m_{\tau }$, respectively. When $Z'$ is on-shell, $q^2$ can be replaced with the $Z'$ boson mass, $m_{Z'}^2$.

Fig. 1

Loop induced kinetic mixing normalized by $g'$ in $L_\mu - L_\tau$ model. The blue and orange dashed curves represent the absolute value of the real and imaginary part of $\epsilon (q^2 = m_{Z'}^2)$, and the green solid one is the absolute value of $\epsilon (q^2 = m_{Z'}^2)$

Figure 1 shows the ratio, $\epsilon (m_{Z'}^2)/g'$, as a function of $m_{Z'}$. Blue and orange curves represent the absolute value of the real part and imaginary part, and green one represents the absolute value of the ratio, respectively. The imaginary part is comparable to or larger than the real part for $2 m_\mu \le m_{Z'}\le 2 m_\tau$. Above a tau pair production threshold, both of the real and imaginary parts monotonically decrease as $m_{Z'}$ increases. The imaginary part decreases faster than the real part does. This is because the imaginary part is inversely proportional to $q^4$ while the real part is to $q^2$.

The mass range of our interest is 20 GeV $\lesssim m_{Z'}\lesssim 60$ GeV, and $\epsilon (m_{Z'}^2)/g'$ varies from $10^{-4}$ to $10^{-5}$ in this range. Therefore the loop-induced kinetic mixing is negligible compared with $g'$ in the $Z'$ decays into $\mu \bar{\mu }$ and $\tau \bar{\tau }$ pairs. On the other hand, $Z'$ can decay into electron-positron ($e^- e^+)$ and quark–antiquark ($q\bar{q})$ pairs only through this mixing. However, as we will show later, the branching ratios of $Z' \rightarrow \mu \bar{\mu }/\tau \bar{\tau }$ are ${\mathcal {O}}(10^4{-} 10^{10})$ times larger than that of $Z' \rightarrow e^- e^+$ and $q\bar{q}$. The expected numbers of $Z'$ decay into these pairs are negligible. Therefore, we can safely ignore the gauge kinetic mixing in our analysis of the scalar production and $Z'$ decays at one-loop level. Higher-order loop contributions to the kinetic mixing also exist. Those would be UV sensitive and could be large when a cutoff scale is very high. In this paper, we assume a similar setup given in [44] that $U(1)_{L_\mu ^{} - L_\tau ^{}}$ symmetry emerges from a $SU(2)_{L_\mu ^{} - L_\tau ^{}}$ symmetry and its breaking scale is high but not so far from the EW scale.

2.2 Gauge boson mass

The gauge bosons acquire masses after the EW and $L_\mu ^{} - L_\tau ^{}$ symmetries are spontaneously broken by the vevs of the scalar fields, H and $\varphi$,

$$\begin{aligned} \langle H \rangle = \frac{v}{\sqrt{2}}, \quad \langle \varphi \rangle = \frac{v_\varphi }{\sqrt{2}}. \end{aligned}$$

(6)

Inserting Eq. (6) into the kinetic term of H and $\varphi$, the mass terms of the gauge bosons are obtained as

$$\begin{aligned} {\mathcal {L}}_{{\mathrm {gauge, mass}}} =\frac{1}{2}m_{Z}^2 Z_\mu Z^\mu + m_{W}^2 W^+_\mu W^{-\mu } + \frac{1}{2} m_{Z'}^2 Z'_\mu Z'^\mu , \end{aligned}$$

(7)

where A,  Z and $W^\pm$ are the SM photon, Z and W boson are defined by

$$\begin{aligned} A_\mu&= \sin \theta _W W^3_\mu + \cos \theta _W B_\mu , \end{aligned}$$

(8a)

$$\begin{aligned} Z_\mu&= \cos \theta _W W^3_\mu - \sin \theta _W B_\mu , \end{aligned}$$

(8b)

$$\begin{aligned} W^\pm _\mu&= \frac{1}{\sqrt{2}} (W^1_\mu \mp i W^2_\mu ), \end{aligned}$$

(8c)

and the masses are given by

$$\begin{aligned} m_{Z}^2&= \frac{1}{2} (g_1^2 + g_2^2) v^2, \end{aligned}$$

(9a)

$$\begin{aligned} m_W^2&= g_2^2 v^2, \end{aligned}$$

(9b)

$$\begin{aligned} m_{Z'}^2&= g'^2 v_\varphi ^2. \end{aligned}$$

(9c)

Here $W^a_\mu ~(a=1,2,3)$ and $g_2$ are the $SU(2)_W$ gauge bosons and coupling constant, respectively, and $\theta _W$ is the Weinberg angle defined by $\sin \theta _W =g_1/\sqrt{g_1^2+g_2^2}$ in the SM. It should be noticed that A,  Z and $Z'$ do not mix because we ignore the gauge kinetic mixing.

2.3 Scalar boson mass and mixing

The scalar bosons also acquire their masses after spontaneous symmetry breaking. We expand H and $\varphi$ around its vev as

$$\begin{aligned} H = \begin{pmatrix} w^+ \\ \frac{1}{\sqrt{2}} (v + {\tilde{h}} + i \xi ) \end{pmatrix}, \quad \varphi&= \frac{1}{\sqrt{2}} (v_\varphi + {\tilde{\phi }} + i \eta ), \end{aligned}$$

(10)

where ${\tilde{h}}$ and ${\tilde{\phi }}$ are CP-even scalar bosons as physical degree of freedom while $w^+$, $\xi$ and $\eta$ are the Nambu–Goldstone bosons which are absorbed by the weak gauge boson W,  Z and $Z'$. The scalar boson masses are obtained by inserting Eq. (10) into the potential. The mass matrix of the CP-even scalar bosons is given by

$$\begin{aligned} M^2_{\mathrm {even}} = \begin{pmatrix} \lambda _H v^2 &{} \lambda _{H\varphi } v v_\varphi \\ \lambda _{H\varphi } v v_\varphi &{} \lambda _\varphi v_\varphi ^2 \end{pmatrix}, \end{aligned}$$

(11)

where we used the stationary conditions,

$$\begin{aligned} \frac{\partial V}{\partial H}= \frac{\partial V}{\partial \varphi } =0. \end{aligned}$$

(12)

The mass matrix can be diagonalized by an orthogonal matrix $U_{\mathrm {even}}$ as

$$\begin{aligned} U^T_{\mathrm {even}} M^2_{\mathrm {even}} U_{\mathrm {even}} = {\mathrm {diag}}(m_h^2, m_H^2), \end{aligned}$$

(13)

where

$$\begin{aligned} U_{\mathrm {even}} = \begin{pmatrix} \cos \alpha &{} \sin \alpha \\ -\sin \alpha &{} \cos \alpha \end{pmatrix}. \end{aligned}$$

(14)

The mass eigenvalues are obtained as

$$\begin{aligned} m_h^2&= \lambda _H v^2 c_\alpha ^2 + \lambda _\varphi v_\varphi ^2 s_\alpha ^2 + 2 \lambda _{H\varphi } v v_\varphi s_\alpha c_\alpha , \end{aligned}$$

(15a)

$$\begin{aligned} m_\phi ^2&= \lambda _\varphi v_\varphi ^2 s_\alpha ^2 + \lambda _H v^2 s_\alpha ^2 - 2 \lambda _{H\varphi } v v_\varphi s_\alpha c_\alpha , \end{aligned}$$

(15b)

where the scalar mixing angle $\alpha$ is expressed as

$$\begin{aligned} \tan 2\alpha = \frac{2 \lambda _{H \varphi } v v_\varphi }{\lambda _H v^2 - \lambda _\varphi v_\varphi ^2}. \end{aligned}$$

(16)

The corresponding mass eigenstates are given by

$$\begin{aligned} \begin{pmatrix} h \\ \phi \end{pmatrix} = U^T_{\mathrm {even}} \begin{pmatrix} {\tilde{h}} \\ {\tilde{\phi }} \end{pmatrix} = \begin{pmatrix} \cos \alpha &{} -\sin \alpha \\ \sin \alpha &{} \cos \alpha \end{pmatrix} \begin{pmatrix} {\tilde{h}} \\ {\tilde{\phi }} \end{pmatrix}. \end{aligned}$$

(17)

Note that h becomes the SM Higgs boson in the limit of $\alpha \rightarrow 0$.

In this work, we employ the masses and mixing angle as input parameters and express the quartic coupling in terms of the inputs. From Eqs. (15) and (16), the quartic couplings can be given by

$$\begin{aligned} \lambda _H&= \frac{1}{2 v^2}\big ( m_h^2 + m_\phi ^2 + (m_h^2 - m_\phi ^2) \cos 2\alpha \big ), \end{aligned}$$

(18a)

$$\begin{aligned} \lambda _\varphi&= \frac{1}{2 v_\varphi ^2}\big ( m_h^2 + m_\phi ^2 - (m_h^2 - m_\phi ^2) \cos 2\alpha \big ), \end{aligned}$$

(18b)

$$\begin{aligned} \lambda _{H\varphi }&= \frac{1}{2 v v_\varphi } (m_h^2 - m_\phi ^2) \sin 2\alpha . \end{aligned}$$

(18c)

where v and $v_\varphi$ are given by $m_W/g_2$ and $m_{Z'}/g'$ from Eqs. (9), respectively. Note that $\lambda _H$ and $\lambda _\varphi$ are always positive because positive mass squared of h and $\phi$ while $\lambda _{H\varphi }$ can be negative.

3 Decay widths

In this section, we present the partial decay widths of $\phi$ and $Z'$ for estimating the signal events while those of the SM-like Higgs, h, for constraints.

3.1 Decays of $\phi$

In our analysis the signal of the new bosons is obtained from the decays $\phi \rightarrow Z'Z'$ followed by $Z' \rightarrow \mu \mu$. Note also that $\phi$ can decay into the SM fermions (f), the SM-like Higgs (h) and the SM vector bosons (V) depending on its mass,

$$\begin{aligned} \phi&\rightarrow f {\overline{f}}, \end{aligned}$$

(19a)

$$\begin{aligned} \phi&\rightarrow h h, \end{aligned}$$

(19b)

$$\begin{aligned} \phi&\rightarrow V V, \end{aligned}$$

(19c)

$$\begin{aligned} \phi&\rightarrow V V^*\rightarrow V f {\overline{f}} (f \overline{f'}). \end{aligned}$$

(19d)

The partial widths of the decays Eq. (19d) can be sizable when $m_\phi$ is smaller than $2m_W$ and/or $2m_Z$, although these are next-leading order.

The relevant interaction Lagrangian for the decays is given by

$$\begin{aligned} {\mathcal {L}}_{\phi -{\mathrm {int}}}&= - \frac{\sin \alpha }{v} m_f {\overline{f}} f \phi -\lambda _{\phi 2h} h^2 \phi \nonumber \\&\quad +\left( -2 \frac{m_W^2}{v}\sin \alpha W^+_\mu W^{-\mu } - \frac{m_Z^2}{v} \sin \alpha Z_\mu Z^\mu \nonumber \right. \\&\quad \left. + \frac{m_{Z'}^2}{v_\varphi } \cos \alpha Z'_\mu Z'^\mu \right) \phi , \end{aligned}$$

(20)

where $m_f$ is the mass of fermion, f, and

$$\begin{aligned} \lambda _{\phi 2h}&= \frac{\sin 2\alpha (2 m_h^2 + m_\phi ^2) ( \cos \alpha v_\varphi - \sin \alpha v)}{4 v v_\varphi }. \end{aligned}$$

(21)

The interaction of $\phi$ with Z and $Z'$ is absent without the tree-level gauge kinetic mixing. It can be generated at loop-level, but it is much suppressed as we have explained. Therefore, we do not consider the decay $\phi \rightarrow Z Z'$. Then, using Eq. (20), the partial decay width of $Z'Z'$ mode is obtained as

$$\begin{aligned} \Gamma (\phi \rightarrow Z'Z')&= \frac{m_{Z'}^4 \cos ^2\alpha }{8 \pi v_\varphi ^2 m_\phi }\beta (x_{Z'}) \left[ 2 + \frac{m_\phi ^4}{4 m_{Z'}^4} \left( 1 - \frac{2 m_{Z'}^2}{m_\phi ^2} \right) ^2 \right] , \nonumber \\ \end{aligned}$$

(22)

and those into the SM particles are given by

$$\begin{aligned} \Gamma (\phi \rightarrow f {\overline{f}})&= \frac{N_c \sin ^2 \alpha }{8 \pi } \frac{m_f^2}{v^2} \beta (x_f)^3 m_\phi , \end{aligned}$$

(23a)

$$\begin{aligned} \Gamma (\phi \rightarrow hh)&= \frac{\lambda _{2h \phi }^2}{8 \pi m_\phi } \beta (x_h), \end{aligned}$$

(23b)

$$\begin{aligned} \Gamma (\phi \rightarrow WW)&= \frac{m_W^4 \sin ^2\alpha }{4 \pi v^2 m_\phi }\beta (x_W)\nonumber \\&\qquad \times \left[ 2 + \frac{m_\phi ^4}{4 m_W^4} \left( 1 - \frac{2 m_W^2}{m_\phi ^2} \right) ^2 \right] , \end{aligned}$$

(23c)

$$\begin{aligned} \Gamma (\phi \rightarrow ZZ)&= \frac{m_Z^4 \sin ^2\alpha }{8 \pi v^2 m_\phi }\beta (x_Z) \nonumber \\&\qquad \times \left[ 2 + \frac{m_\phi ^4}{4 m_Z^4} \left( 1 - \frac{2 m_Z^2}{m_\phi ^2} \right) ^2 \right] , \end{aligned}$$

(23d)

where $N_c = 3~(1)$ is the color factor for quarks (leptons) and $\beta (x) = \sqrt{1 - 4 x}$ with $x_i = m_i^2/m_\phi ^2$ $(i=Z',f,h,W,Z)$. For the decay modes in Eq. (19d), the decay widths are given by summing over fermions with neglecting their masses,

$$\begin{aligned} \Gamma (\phi \rightarrow W W^*\rightarrow W f \bar{f'})&= \frac{3 m_W^4 \sin ^2\alpha }{2^{6} \pi ^3 v^4} m_\phi S(x_W), \end{aligned}$$

(24a)

$$\begin{aligned} \Gamma (\phi \rightarrow Z Z^*\rightarrow W f \bar{f'})&= \frac{3 m_Z^2 m_W^2\sin ^2 \alpha }{2^6 \pi ^3 v^4 \cos ^2\theta _W} m_\phi R_W S(x_Z) \end{aligned}$$

(24b)

where $\theta _W$ is the Weinberg angle, and

$$\begin{aligned} S(x)&= 47 x^2 - 60 x + 15 - \frac{2}{x} - 3 (4 x^2 - 6 x + 1) \log x \nonumber \\&\quad + \frac{6( 20 x^2 - 8 x + 1) }{\sqrt{4x-1}} \cos ^{-1} \left( \frac{3x-1}{2 x^{3/2} } \right) , \end{aligned}$$

(25a)

$$\begin{aligned} R_W&= \frac{7}{12} - \frac{10}{9} \sin ^2\theta _W + \frac{40}{27} \sin ^4\theta _W. \end{aligned}$$

(25b)

Fig. 2

The branching ratios of $\phi$ decays as a function of $m_\phi$. The final states are indicated near each curves

Figure 2 shows numerical results of the decay branching ratios of $\phi$, ${\mathrm {Br}}(\phi )$, as a function of $m_\phi$. The $Z'$ boson mass is taken to be 20 and 40 GeV in top and bottom panels as reference values, respectively. Other parameters, $g'$ and $\alpha$, for each $m_{Z'}$ are indicated in each panels. These parameters are chosen so as to avoid the present experimental bounds on $\sin \alpha \lesssim 0.25$ and/or $h \rightarrow Z'Z' \rightarrow 4l$, which we will discuss in the next section. In the figures, blue, green and yellow solid curves represent the branching ratios into $Z'Z',~ZZ$ and WW, and light blue, brown, purple and red dashed curves represent those into hh, top (tt), bottom (bb), and tau pairs, respectively. From the figures, the branching ratio of $\phi \rightarrow Z'Z'$ is larger for smaller value of $\alpha$ to each $m_{Z'}$. This is simply because the interaction of $\phi$ with $Z'$ comes from the $L_\mu ^{} - L_\tau ^{}$ gauge interaction and hence is proportional to $\cos \alpha$, as presented in Eq. (20). On the other hand, the interactions with the SM particles, ZZ,  WW and $hh,~tt,~bb,~\tau \tau$ is proportional to $\sin \alpha$ because it is generated through the mixing with the SM Higgs. Thus, the decay widths into $Z'Z'$ can be dominate for smaller $\alpha$. However, in such a case, the production cross section of $\phi$ becomes much suppressed since $\phi$ is mainly produced via gluon-fusion which is also proportional to $\sin ^2 \alpha$. Therefore, there are certain parameter regions where $\phi$ and $Z'$ will be detected at the LHC when both production cross section and BR of $Z'Z'$ mode is sizable. We will search for such parameter regions performing numerical simulation analysis in Sect. 5.

3.2 Decays of $Z'$

In the mass range of our interest, $m_{Z'}< 60$ GeV, the $Z'$ boson decays mainly into muon and tau lepton pairs through the $L_\mu ^{} - L_\tau ^{}$ gauge interaction while it decays partly into electron and quarks through the loop-induced kinetic mixing.³ Although these decay widths are suppressed, we include the effects of the loop-induced kinetic mixing for completeness only in this subsection.

The relevant interaction Lagrangian with the $Z'$ decays into fermion f is given by

$$\begin{aligned} {\mathcal {L}}_{Z'-{\mathrm {int}}} =\frac{1}{2} \bar{f} \gamma ^\mu (v'_f - a'_f \gamma _5) f Z'_\mu , \end{aligned}$$

(26)

where

$$\begin{aligned} v'_f&= 2 e Q_f U_{13} + \frac{g_2}{\cos \theta _W}(T_{3f}\nonumber \\&\quad - 2 Q_f \sin ^2\theta _W) U_{23} + 2 g' X_f U_{33}, \end{aligned}$$

(27a)

$$\begin{aligned} a'_f&= \frac{g_2}{\cos \theta _W}(T_{3f} ) U_{23}. \end{aligned}$$

(27b)

In Eqs. (27), $Q_f,~T_{3f}$ and $X_f$ are the electric, weak and $L_\mu ^{} - L_\tau ^{}$ charges of f, respectively. The elements of the gauge mixing matrix, U, is given by

$$\begin{aligned} U_{13}&= \epsilon r \cos \theta _W \cos \chi , \nonumber \\ U_{23}&= -\sin \chi - \epsilon r \sin \theta _W \cos \chi , ~~U_{33} = r \cos \chi , \end{aligned}$$

(28)

where $\epsilon$ is given by Eq. (5) and $\chi$ is the mixing angle of Z and $Z'$ defined by

$$\begin{aligned} \tan 2\chi = - \frac{2 \epsilon r \sin \theta _W m_{Z}^2}{m_Z^2 - m_{Z'}^2}. \end{aligned}$$

(29)

Here we assume that the mixing angle $\chi$ is much smaller than unity, and hence the masses and eigenstates of the gauge bosons are approximately the same as those without the kinetic mixing.

The decay width of $Z' \rightarrow f\bar{f}$ is given by

$$\begin{aligned} \Gamma (Z' \rightarrow f \bar{f}) = \frac{m_{Z'}}{48 \pi } \beta (y_f) \left[ {v'_f}^2 + {a'_f}^2 + 2({v'_f}^2 - 3 {a'_f}^2) \frac{m_f^2}{m_{Z'}^2}\right] , \nonumber \\ \end{aligned}$$

(30)

where $y_f = m_f^2/m_{Z'}^2$. In the limit of $\epsilon \rightarrow 0$, $v_f'$ is given by $2g' X_f$ and $a_f'$ vanishes. In such a situation, the decay widths Eq. (30) is proportional to $g'^2 m_{Z'}$.

Fig. 3

Decay width of $Z'$ normalized by ${g'}^2 m_{Z'}$, $\Gamma (Z')/({g'}^2 m_{Z'})$ for various $g'$ as a function of $m_{Z'}$. The blue, yellow and green curves represent the decay into muons, taus and neutrinos, respectively. The brown curve corresponds to the sum of the decay widths. In the figure, the gauge coupling constant is fixed to $10^{-3}$

Figure 3 shows the $m_{Z'}$ dependence of the decay widths normalized by $g'^2 m_{Z'}$. In the figure, $g'$ is fixed to be $10^{-3}$ and the kinetic mixing is set to be $10^{-3} g'$ which correspond to the loop induced kinetic mixing for $m_{Z'}= 20$ GeV. The blue, yellow and green curves represent the decays into $\mu \mu ,~\tau \tau$ and $\nu _\mu \nu _\mu + \nu _\tau \nu _\tau$. The brown one denotes the total widths. From the figure, one can see that $Z'$ mainly decays into $\mu \mu ,~\tau \tau$ and $\nu _\mu \nu _\mu +\nu _\tau \nu _\tau$, and the normalized decay widths are almost constant above the threshold of muon and tau. This is due to the fact that $v_f' \sim 2 g' X_f \gg a_f'$ in Eq. (27). Compared with $U_{33}$, the mixing matrix $U_{13}$ is much suppressed with $\epsilon \sim 10^{-3} g'$ and $U_{23}$ is also suppressed because of the cancellation in $\sin \chi +\epsilon r \sin \theta _W \cos \chi$. Thus, the partial decay widths into $\mu \mu ,~\tau \tau$ and $\nu _{\mu ,\tau } \nu _{\mu ,\tau }$ are almost proportional to $g'^2m_{Z'}$. The normalized decays widths into quarks and electrons are below about $10^{-7}$, and hence the branching ratio of these decays are below $10^{-6}$. Such a small portion of the decays will not be seen due to large background of the SM processes. Therefore, we only consider the decays into $\mu \mu$ as the signal in the following sections. Before closing this subsection, we should comment on the gauge mixing angle Eq. (29). The gauge mixing angle increases as $m_{Z'}$ get close to $m_Z$ and becomes maximal when $m_{Z'}\simeq m_Z$. In such situation, $U_{23}$ is not suppressed and the $Z'$ interactions with quarks and electron is significant. Then, the decays into quarks and electrons can dominate the $Z'$ decays. However, the mixing angle is kept below $\chi /g' \simeq 1.2 \times 10^{-3}$ for $m_{Z'}\lesssim 60$ GeV and $\epsilon /g' = 10^{-3}$. Therefore, the increase of the gauge mixing angle can be safely ignored in the decays of $Z'$.

3.3 Decays of h

The SM-like Higgs can decay into $Z'Z'$ and $\phi \phi$ when these are kinematically allowed.

The relevant Lagrangian of those new decay modes is given by

$$\begin{aligned} {\mathcal {L}}_{h~{\mathrm {int}}} = \frac{m_{Z'}}{v_\varphi } \sin \alpha h Z'_\mu Z^{'\mu } -\lambda _{h 2\phi } h \phi ^2, \end{aligned}$$

(31)

where

$$\begin{aligned} \lambda _{h 2\phi } =- \frac{\sin 2\alpha (m_h^2 + 2 m_\phi ^2) ( \cos \alpha v + \sin \alpha v_\varphi )}{4 v v_\varphi }. \end{aligned}$$

(32)

The decay widths are given by

$$\begin{aligned} \Gamma (h \rightarrow Z' Z')&= \frac{g_{Z'Z'h}^2}{8 \pi m_h} \beta (z_{Z'}) \left[ 2 + \frac{m_h^4}{4 m_{Z'}^4} \left( 1 - \frac{2 m_{Z'}^2}{m_h^2} \right) ^2 \right] ,\nonumber \\ \end{aligned}$$

(33a)

$$\begin{aligned} \Gamma (h \rightarrow \phi \phi )&= \frac{\lambda _{2h \phi }^2}{8 \pi m_h} \beta (z_\phi ), \end{aligned}$$

(33b)

where $z_{Z'} = m_{Z'}^2/m_h^2$.

The invisible decay branching ratio of h is constrained at the LHC experiment. In this model, the invisible decays are

$$\begin{aligned} h&\rightarrow Z' Z' \rightarrow 4 \nu , \end{aligned}$$

(34a)

$$\begin{aligned} h&\rightarrow ZZ' \rightarrow 4 \nu , \end{aligned}$$

(34b)

$$\begin{aligned} h&\rightarrow \phi \phi \rightarrow 4 Z' \rightarrow 8 \nu . \end{aligned}$$

(34c)

The branching ratio of these decays is given by

$$\begin{aligned}&{\mathrm {Br}}(h \rightarrow {\mathrm {invisibles}}) = \mathrm {Br}(h \rightarrow Z'Z') \big [ {\mathrm {Br}}(Z' \rightarrow \nu \nu ) \big ]^2\nonumber \\&\quad + {\mathrm {Br}}(h \rightarrow ZZ') \big [ {\mathrm {Br}}(Z \rightarrow \nu \nu ) {\mathrm {Br}}(Z' \rightarrow \nu \nu ) \big ] \nonumber \\&\quad + {\mathrm {Br}}(h \rightarrow \phi \phi ) \big [ {\mathrm {Br}}(\phi \rightarrow Z' Z') \big ]^2\big [ {\mathrm {Br}}(Z' \rightarrow \nu \nu ) \big ]^4, \end{aligned}$$

(35)

where its current bound is less than 0.25 and 0.19 by the CMS [48] and the ATLAS [49] experiments. To be conservative, we employ the bound from the CMS experiment in the following discussion.

The decays of h into charged leptons are also constrained at the LHC. Its constraint is more stringent when the branching ratio of $Z' \rightarrow l\bar{l}$ is sizable.

The new decay processes into charged leptons in this model are

$$\begin{aligned} h&\rightarrow Z' Z' \rightarrow l_1 \overline{l_1} + l_2 \overline{l_2}, \end{aligned}$$

(36a)

$$\begin{aligned} h&\rightarrow Z Z' \rightarrow l_1 \overline{l_1} + l_2 \overline{l_2}, \end{aligned}$$

(36b)

$$\begin{aligned} h&\rightarrow \phi \phi \rightarrow 4 Z' \rightarrow l_1 \overline{l_1} + l_2 \overline{l_2} + l_3 \overline{l_3} + l_4 \overline{l_4}, \end{aligned}$$

(36c)

where $l= e,~\mu ,~\tau$.

In [50], a light gauge boson search was performed and impose stringent constraint on $m_{Z'}$ and $g'$. The branching ratio of (36a) and (36a) is given by

$$\begin{aligned} {\mathrm {Br}}(h \rightarrow 4l)&= {\mathrm {Br}}(h \rightarrow Z'Z') \big [2 {\mathrm {Br}}(Z' \rightarrow l_1 \overline{l_1}) {\mathrm {Br}}(Z' \rightarrow l_2 \overline{l_2}) \big ] \nonumber \\&\quad + {\mathrm {Br}}(h \rightarrow ZZ') \big [ {\mathrm {Br}}(Z \rightarrow l_1 \overline{l_1}) {\mathrm {Br}}(Z' \rightarrow l_2 \overline{l_2})\nonumber \\&\quad + {\mathrm {Br}}(Z' \rightarrow l_1 \overline{l_1}) {\mathrm {Br}}(Z \rightarrow l_2 \overline{l_2}) \big ]. \end{aligned}$$

(37)

4 Constraints

We explain the relevant constraints on the gauge coupling $g'$, the $Z'$ boson mass $m_{Z'}$ and the scalar mixing $\alpha$, and show the allowed region for these parameters. In the following discussion, we assume that the extra scalar boson is heavier than the Higgs boson so that the decay of $h \rightarrow \phi \phi$ is kinematically forbidden. The ATLAS collaboration searched for new light gauge bosons via the Higgs boson decays into 4 leptons, $h \rightarrow Z'Z' \rightarrow 2l 2l'$ and $h \rightarrow ZZ' \rightarrow 2l 2l'$ [50, 51]. In our setup, the latter decay is much suppressed by the loop-induced kinetic mixing, and hence the bound does not restrict the parameters. The former decay, on the other hand, occurs through the $L_\mu ^{} - L_\tau ^{}$ gauge interaction and the scalar mixing, which constrains the parameters. The $1\sigma$ and $2\sigma$ upper limit (UL) on the decay branching ratio of $h \rightarrow Z'Z' \rightarrow 4l$ are shown in Table 2.

Table 2 Upper limit on the branching ratios of the Higgs boson decays into $Z'Z'$ followed by $Z' \rightarrow l\bar{l}$ and the gauge coupling constant $g'$. Numbers of the branching ratio and the gauge coupling are taken from Figure 10.(a) of [51] and Figure 8 of [52], respectively

The CMS collaboration also searched the $Z'$ boson using 77.3 $\hbox {fb}^{-1}$ data recorded in 2016 and 2017 [52]. The search was performed for the bremsstrahlungs of $Z'$ from $\mu$ or ${\overline{\mu }}$ produced in pp collisions. The results put the constraint on $g'$ in the mass range $5 \le m_{Z'}\le 70$ GeV. The upper bound on $g'$ is shown in Table 2.

From the experimental bounds explained above, we calculate the branching ratio of $h \rightarrow 4l$, Eq. (37), and showed the allowed region for $g'$ and $\alpha$ in Fig. 4. Red, green, blue and orange filled region are the exclusion region for $m_{Z'}=20,~30,~40$ and 50 GeV, from the search for $h \rightarrow 4l$ at $2\sigma$, respectively. Dashed lines are the contour of the upper limit of the branching ratio. Gray region is excluded by analysis of data regarding the SM Higgs signals from the LHC experiments [53, 54].

Fig. 4

Allowed region of the parameters in $\log (g')$-$\log (\alpha )$ plane. Red, green, blue and orange colored region are the exclusion region for $m_{Z'}=20,~30,~40$ and 50 GeV from $h \rightarrow 4l$ search at $2\sigma$. Dashed lines represent the contour of the upper bound of Br$(h \rightarrow 4l)$ given in Table 2. Gray shaded region is excluded by analysis of data regarding the SM Higgs signals from the LHC experiments

5 $Z'$ and $\phi$ search at LHC

In this section we carry out numerical simulation for $\phi$ production followed by $\phi \rightarrow Z'Z'$ and $Z' \rightarrow \mu ^+ \mu ^-$ decays at the LHC. The background (BG) process, $pp \rightarrow \mu ^+ \mu ^- \mu ^+ \mu ^-$, is also considered in the SM. Then significance of the signal is discussed applying relevant kinematical cuts where experimental constraints in previous section are also taken into account.

Fig. 5

The cross section for $pp \rightarrow \phi$ as a function of $m_\phi$ which is multiplied by scaling factor $\kappa _\alpha = (\sin \alpha /0.01)^{-2}$, and $\sqrt{s} = 14$ TeV is applied

5.1 Production cross section

Here we discuss the dominant $\phi$ production process at the LHC. This new scalar boson can be produced by gluon fusion process $gg \rightarrow \phi$ through mixing with the SM Higgs boson. We obtain the relevant effective interaction for the gluon fusion as [55]

$$\begin{aligned} {{\mathcal {L}}}_{\phi gg} = \frac{\alpha _s}{16 \pi } \frac{\sin \alpha }{v} A_{1/2}(\tau _t) \phi G^a_{\mu \nu } G^{a \mu \nu }, \end{aligned}$$

(38)

where $A_{1/2}(\tau _t) = -\frac{1}{4} [\ln [(1+\sqrt{\tau _t})/(1-\sqrt{\tau _t})] - i \pi ]^2$ with $\tau _t = 4 m_{t}^2/m_\phi ^2$ and $G^a_{\mu \nu }$ is the field strength for gluon. This effective interaction is dominantly induced from ${\bar{t}} t \phi$ coupling via the mixing effect where we take into account only top Yukawa coupling omitting the other subdominant contributions for simplicity. In Fig. 5, we show the production cross section as a function of $m_\phi$ estimated by use of MADGRAPH5 [56] implementing the effective interaction wit FeynRules 2.0 [57], which is multiplied by scaling factor $\kappa _\alpha \equiv (\sin \alpha /0.01)^{-2}$ as the cross section is proportional to $\sin ^2 \alpha$. In addition we included K-factor $K_{gg} = 1.6$ for gluon fusion process which represent NLO correction effect [58].

Fig. 6

Kinetic distributions of final state muons for signal

Fig. 7

Kinetic distributions of final state muons for BG

5.2 Kinematical cuts

We carry out numerical simulation for our signal and BG processes where the events are generated using MADGRAPH/MADEVENT 5 [56] implementing the necessary Feynman rules and relevant parameters of the model via FeynRules 2.0 [57], the PYTHIA 8 [59] is applied to deal with hadronization effects, the initial-state radiation (ISR) and final-state radiation (FSR) effects and the decays of SM particles, and Delphes [60] is used for detector level simulation. In generating signal and BG events basic cuts are implemented in MADGRAPH/MADEVENT 5 as

$$\begin{aligned} p_T(\ell ^\pm ) > 7~ \mathrm{GeV}, \quad \eta (\ell ^\pm ) < 2.5, \end{aligned}$$

(39)

where $p_T$ denotes transverse momentum and $\eta = 1/2 \ln (\tan \theta /2)$ is the pseudo-rapidity given by $\theta$ being the scattering angle in the laboratory frame. We then chose events which has two muon anti-muon pair in final states.

To impose additional cuts we produce kinematical distributions for signal and BG. In Figs. 6 and 7 we show several distributions for signal and BG where we fix $m_\phi = 200$ GeV, $m_{Z'} = 30$ GeV and $\sin \alpha =0.012$ as reference values for signal events. In addition we chose integrated luminosity as 3000 $\hbox {fb}^{-1}$ in estimating number of events. The upper-left and -right plots in the figures show distributions for transverse momentum of $\mu ^-$ where $\mu _1^-$ and $\mu ^-_2$ are distinguished by $p_T(\mu ^-_1) > p_T(\mu ^-_2)$ for each event; the distributions for $\mu ^+$ are the same as $\mu ^-$. We find that the signal and BG provide similar distribution for transverse momentum of muon. Thus we do not impose further cuts for $p_T(\mu )$. The lower-left plots in the figures show distributions for invariant mass of $\mu _1^- \mu _1^+$. For the signal we find clear peak corresponding to $Z'$ mass where another bump comes from combinations of muon from different $Z'$ decays. On the other hand peak at Z mass is found for BG. The lower-right plots in the figures show distributions for invariant mass of four muons $M_{\mu ^+\mu ^+\mu ^-\mu ^-}$. For the signal the distribution is clearly concentrated at the mass of $\phi$ while the distribution for BG shows peak at Z mass and continuous region. To reduce the BG events, we thus eliminate events which has $\mu ^+ \mu ^-$ and $\mu ^+\mu ^+\mu ^-\mu ^-$ invariant masses within the range of

$$\begin{aligned}&80~ \mathrm{GeV}< M_{\mu ^+ \mu ^-} < 100~ \mathrm{GeV}, \end{aligned}$$

(40)

$$\begin{aligned}&80 \ \mathrm{GeV}< M_{\mu ^+\mu ^+\mu ^-\mu ^-} < 100 \ \mathrm{GeV}. \end{aligned}$$

(41)

We estimate numbers of the signal and BG events after applying these cuts.

Fig. 8

Significance on $m_{Z'}$–$g'$ plane where red dashed lines are for $BR(\phi \rightarrow Z' Z')$ and its values are indicated on the lines

Fig. 9

Significance on $\sin \alpha$–$g'$ plane

5.3 Significance

We estimate discovery significance of the signal after imposing the kinematical cuts discussed in the previous subsection. The significance is given by

$$\begin{aligned} S = \frac{N_S}{\sqrt{N_{BG}}}, \end{aligned}$$

(42)

where $N_S$ and $N_{BG}$ are respectively the number of events for the signal and total BG. In estimating number of events we assume integrated luminosity of 3000 $\hbox {fb}^{-1}$ as in the previous subsection. In Fig. 8 we show contours of discovery significance on $\{m_{Z'}, g' \}$ plane where we fix $\{m_\phi , \sin \alpha \}=\{200 \ \mathrm{GeV}, 0.012\}$, $\{300 \ \mathrm{GeV}, 0.012\}$, $\{200~\mathrm{GeV}, 0.015\}$ and $\{300 \ \mathrm{GeV}, 0.015\}$ as reference values for upper-left, upper-right, lower-left and lower-right plots. In addition value of $BR(\phi \rightarrow Z'Z')$ is shown by red dashed curve, gray region is excluded by the constraint from $h \rightarrow Z' Z' \rightarrow \ell ^+ \ell ^- \ell ^+ \ell ^-$ decay, and light gray region is excluded by CMS data for $pp \rightarrow \mu ^+ \mu ^- Z'(\rightarrow \mu ^+ \mu ^-)$ signal search. We find that $S \gtrsim 3$ can be achieved on parameter space with $g' \gtrsim 10^{-3(-2)}$ for $m_\phi = 200(300)$ GeV case when $\sin \alpha =0.012$. For $\sin \alpha = 0.015$, we can obtain $S > 5$ with $g' \gtrsim 10^{-3} - 10^{-2}$ as the $\phi$ production cross section becomes large. Moreover we show contours of the significance on $\{\sin \alpha , g' \}$ plane fixing $m_{Z'} = 33$ GeV and $m_{\phi } = 200(300)$ GeV in left(right) plot of Fig. 9. To obtain sizable significance $\sin \alpha$ should be larger than ${\mathcal {O}}(0.01)$ to achieve sufficiently large $\phi$ production cross section. Also we cannot obtain sizable significance for parameter region with too small $g'$ since BR of $\phi \rightarrow Z'Z'$ becomes tiny in such region. For very small $g'$ region, it will be more promising to search for the signal that $\phi$ decays into SM particles, but analysis of such signals is beyond the scope of this paper.

6 Conclusion

We have considered a minimal gauged $L_\mu ^{} - L_\tau ^{}$ model where the $L_\mu ^{} - L_\tau ^{}$ gauge symmetry is spontaneously broken by a SM singlet scalar field, and studied the possibility of discovering the new gauge boson $Z'$ and scalar boson $\phi$ at the LHC experiments. We considered the case in which $\phi$ is heavier than $Z'$ so that it dominantly decays into $Z'Z'$. The produced $Z'$ decays into a pair of muons, taus and their corresponding neutrinos. Then, the signal significance of such $\phi$ and $Z'$ decays against the SM backgrounds is analyzed focusing on the four muon final states.

We firstly showed the branching ratio of $\phi \rightarrow Z'Z'$ decay becomes larger as the scalar mixing is smaller. The scalar boson dominantly decays into $Z'Z'$ for $\alpha \sim {\mathcal {O}}(10^{-2})$ for $m_\phi = 20$ and 50 GeV as reference parameters. The decay width of $Z'$ is also showed that those into muons, taus and neutrinos are the almost the same. Thus, the branching ratio of $Z'$ into muons is 1/4 in our setup. The gauge coupling constant and the scalar mixing have been constrained by the searches for the four lepton decay and the invisible decays of the Higgs boson. Based on these constraints, the allowed region of $g'$ and $\alpha$ was derived for the analyses of the signal significance at LHC.

Then, the production cross section of $\phi$ through gluon fusion at LHC was calculated and the kinematical distributions of muons for the signal and backgrounds were analyzed. We found that the mass of $Z'$ and $\phi$ can be clearly reconstructed as a peak in the distributions of the invariant mass of $\mu ^+ \mu ^-$ and $\mu ^+ \mu ^+ \mu ^- \mu ^-$, respectively, for the signal events. On the other hand, for the background, the mass of Z boson can be reconstructed in the same distributions. By applying cuts vetoing 80 GeV $< M_{\mu ^+ \mu ^-},~M_{\mu ^+ \mu ^ +\mu ^- \mu ^-} < 100$ GeV, the background events can be reduced. We showed the signal significance S can reach to 3 for $m_\phi = 200~(300)$ GeV and $g' \ge 10^{-3~(-2)}$, respectively, when the scalar mixing is 0.012. For $\alpha = 0.015$, $S > 5$ can be achieved. We also showed the signal significance in $\sin \alpha$-$g'$ plane. The region for $\sin \alpha \ge 10^{-2}$ and $g' \ge 10^{-3}$ can be explored with $S > 3$.