Investigation of $(g-2)_{\mu }$ anomaly in the $\mu$-specific 2HDM with vector like leptons and the phenomenological implications

Abstract

The anomalous magnetic moment of muons has been a long-standing problem in SM. The current deviation of experimental value of the $(g-2)_{\mu }$ from the standard model prediction is exactly $4.2\sigma$. Two Higgs Doublet Models can accommodate this discrepancy but such type of model naturally generate flavor changing neutral current(FCNC). To prevent this it was postulated that 2HDM without FCNC required that all fermions of a given charge couple to the same Higgs boson but the rule breaks in Muon Specific Two Higgs Doublet Model where all fermions except muon couple to one Higgs doublet and muon with the other Higgs doublet. The muon specific 2HDM provides an explanation for muon anomaly for extremely large $\tan \beta$ with a particular value of $m_{12}^2$. As a result, the parameter space of $m_{12}^2$ drastically decreased for the muon anomaly solution. To evade the limitation of this model we have extended this model with a vector like lepton generation which could explain the muon anomaly at low $\tan \beta$ value with a heavy pseudo scalar Higgs boson under the shadow of current experimental and theoretical constraints. Moreover, with the help of the cut based analysis and multivariate analysis methods, we have also attempted to shed some light on the potential experimental signature of vector lepton decay to the heavy Higgs boson in the LHC experiment. We have showed that a multivariate analysis can increase the vector like leptons signal significance even in the high VLL mass and low $\tan \beta$ region than that of a cut based analysis.

1 Introduction

The Standard Model (SM) contribute an amazing interpretation of nature persisting draconian test at both the current energy and precision frontiers. Due to lack of any direct signal for new particles at the LHC puts stringent bounds for different new particles up to several TeV. The remarkable consistency between the predictions from the standard model (SM) and the experimental data from the LHC so far has indicated that the SM is the appropriate effective theory of electroweak (EW) symmetry breaking. The measurement of the magnetic moment of the muon deviates from the SM prediction by more than three standard deviations. The $(g-2)_{\mu }$ Collaboration of Fermilab recently published a new result from Run 1 experiment measuring the anomalous magnetic moment of the muon [1,2,3,4]. Before this result the discrepancy between the experimental measurement $a_{\mu }^{exp}$ [5] and the Standard Model $a_{\mu }^{SM}$ prediction [6] was

$$\begin{aligned} \Delta a_{\mu }^{exp} = a_{\mu }^{exp} - a_{\mu }^{SM} =(279 \pm 76) \times 10^{-11} (3.7 \sigma ) \nonumber \\ \end{aligned}$$

(1)

while the new combined result is [1]

$$\begin{aligned} \Delta a_{\mu }^{exp} =(251 \pm 59) \times 10^{-11} (4.2 \sigma ) \end{aligned}$$

(2)

From long time people tried very hard to explain the $(g-2)_{\mu }$ and there are many papers with different models such as supersymmetric models [7, 8], left-right symmetric models [9], scotogenic models [10], 331 models[11], $L_{\mu } - L_{\tau }$ models [12], seesaw models [13], the Zee–Babu model [14, 15] whose detail discussions can be found in [16]. The expansion of SM lepton sector with vector leptons, is of particular interest [17, 18] can explain the discrepancy.In this type of SM extension with vector like leptons (VLLs), muon mixing with the VLLs is required to explain $(g-2)_{\mu }$, and this mixing will change the coupling of Higgs with muon, which will affect not only the Higgs dimuon decay branching ratio, but also the Higgs diphoton decay, which is strongly disfavored by recent collider Higgs data.

The $(g-2)_{\mu }$ can also be explained using the minimal SM scalar extension Two Higgs Doublet Model (2HDM) [19, 20]. A discrete $Z_2$ symmetry can be used to block the flavor-changing neutral current that occurs in 2HDM, leading to the emergence of four different types of 2HDM, Type-I, Type-II, Type-X and Type-Y (flipped) [21]. Among them only Type-X and Type-Y variants are effective to explain the $(g-2)_{\mu }$. The corresponding model has an enhanced coupling of lepton with new heavy scalar of 2HDM it can solve the muon anomaly including the usual one-loop and two-loop contribution from the Barr Zee type diagrams [22, 23]. The Type-II model is severely constrained by flavor physics and direct searches of extra Higgs bosons because both the charged lepton and down type quark coupling with the new heavy scalar are proportional to $\tan \beta$. In Type-II model the $(g - 2)_{\mu }$ required high value of $\tan \beta$ and light pseudo scalar mass which is disallowed by B-physics observables [24]. The flavour limitations are weaker in Type-X 2HDM [25]than in Type-II 2HDM because the lepton couplings are boosted while the quark couplings are suppressed. Among the two variants, the Type-X only fit to explain the existing muon anomaly escaping the flavor constraint without any fermionic extension. But there is a problem with Type-X model that is to satisfy the low energy data it requires very light pseudo Higgs boson and large $\tan \beta$ [22,23,24, 26,27,28] which is also not allowed by B-physics observables. The Type-X parameter space is also highly constrained by the experimental measurement of leptonic tau decay. As a result, the parameter region which can explain the discrepancy in the $(g-2)_{\mu }$ at the $1\sigma$ level is excluded by the constraint from the tau decay [27].

Due to the shortcomings of Type-II and Type-X models to explain the $(g-2)_{\mu }$ a new type of 2HDM was proposed by ASY (Tomohiro Abe, Ryosuke Sato and Kei Yagyu) to probe the $(g-2)_\mu$ problem [29]. The model was structured in such a way that without losing the advantage of type-X model it could accommodate the solution of $(g-2)_{\mu }$. By the implementation of $Z_4$ symmetry they managed to stop the flavor changing neutral current process and simultaneously constrained the model in such a way that the only second generation of SM leptons couple with one doublet and all others quarks and leptons couple to other doublet. The details of couplings and quantum numbers can be found in [29]. The muon specific 2HDM ($\mu$2HDM) is also important from another point of view as CMS and ATLAS have performed [30, 31] a search for the dimuon decay; the most recent study by ATLAS [31] finds a branching ratio of $0.5\pm 0.7$ times the Standard Model branching ratio (the uncertainty is one standard deviation). This value is consistent with SM value but if the dimuon decay is not discovered in near future then it will be certain that the branching ratio is substantially below of SM. As the ditau decay is expected to be like SM which means that the muon and tau must couple with different Higgs doublet [32].

Above all, these positive aspects, the $\mu$2HDM has a drawback, it requires very high $\tan \beta$ typically of O(1000) to explain muon anomaly. As usual such large $\tan \beta$ value causes problem with perturbation theory, unitarity, electroweak precision observables, etc. Though ASY shows that this problem can be bypassed if one chooses the free parameters carefully. Because of this, even though $\mu$2HDM is a more acceptable explanation than the Type-II and Type-X models, the very large $\tan \beta$ decrease the parameter space. However, the inclusion of a vector lepton doublet and singlet in the $\mu$2HDM model (referred to as $\mu$2HDM+VLLs) has the potential to expand the parameter space and account for the muon anomaly even at low $\tan \beta$ values. In this model we have applied the ASY mechanism to extended lepton sector and assigned the respective quantum numbers to vector like leptons. Here the new leptons couple to $\Phi _{1}$, so only muon can mix with the vector like lepton. Though in [33] they have studied the muon anomaly including muon mixing with vector like lepton but they have considered the 2HDM Type-II model with the assumption of muon mixing. We have shown that depending on the ASY mechanism the vector like leptons can naturally mix with muon only, which is a unique situation and also it does not carry 2HDM Type-II enhanced quark coupling limitation. Furthermore the charged VLL and charged Higgs will contribute in $h \rightarrow \gamma \gamma$ decay so we have also shown the allowed parameter space by $h \rightarrow \gamma \gamma$ within the experimental limit.

In this work we have also discussed the signal of all possible VLLs detection channels at LHC. As we know Leptons lighter than 100 Gev are excluded in the earlier search at LEP experiment [34] and the LHC, ATLAS rejected VLLs that underwent singlet transformation under $SU(2)_L$ in the energy range of 114–176 GeV at 95% CL [35]. Recent CMS experiment with luminosity 77.4 fb$^{-1}$ at 13 TeV, look for doublet VLLs coupling to third-generation leptons only, discarded all VLLs with masses between 120 and 790 GeV at 95$\%$ CL [36]. However, those VLL limits were determined using simplified models. In this work, we investigate the collider signature of extended $\mu$2HDM with vector like leptons (VLLs), which includes both SU(2) doublet and a singlet. We analyze this $\mu$2HDM+VLLs in its multi-lepton final state using cut-based analysis as well as multivariate analysis (MVA). This two-way search offers us a comparative examination of the model and provides us the optimised cut value of parameters to find a sensitivity that is compatible to understanding the model.

This paper is organized as follows: In Sect. 2 we have discussed the extended $\mu$2HDM model coupled with VLL and also the contribution of this model in the $(g-2)_\mu$. Section 3 contain all the experimental and the theoretical constraints that have been used to constrain our model. In the Sect. 4 we have discussed about the parameter space allowed by all the constraints. The collider phenomena of the VLL in the light of recent collider data and MVA analysis has been discussed in the Sect. 5. In the final Sect. 6 we conclude all our findings and some relevant formulas have been showcased in the Appendix.

2 The model

The scalar sector of the $\mu$2HDM is made of two $SU(2)_L$ doublet scalar fields $\Phi _1$ and $\Phi _2$. To prevent the FCNCs we apply $Z_4$ symmetry under which the fields transform as $\Phi _1 \rightarrow -\Phi _1$ and $\Phi _2 \rightarrow \Phi _2$. We have extended $\mu$2HDM with vector like leptons (VLLs) including both SU(2) doublet $L_{L,R}$ and singlet representation $E_{L,R}$. The quantum numbers of SM leptons, Higgs doublets and vector-like fields are represented in Table 1.

Table 1 Quantum numbers of Standard Model leptons, Higgs doublets, and vector like leptons under $Z_4$

The Yukawa interaction for the muon terms following $Z_4$ symmetry under this charge assignment are given by

$$\begin{aligned} {\mathcal {L}}= & {} - y_{\mu }{\bar{l}}_L \mu _R \Phi _{1} - \lambda _L{\bar{L}}_L\mu _R \Phi _{1} - \lambda _E {\bar{l}}_LE_R \Phi _{1} \nonumber \\{} & {} \quad -\lambda {\bar{L}}_LE_R \Phi _{1} - {\bar{\lambda }}\Phi _{1}^\dagger {\bar{E}}_L L_R \nonumber \\{} & {} \quad - M_l {\bar{L}}_L L_R - M_e {\bar{E}}_L E_R+ h.c \end{aligned}$$

(3)

The lepton and scalar doublets can be written as,

$$\begin{aligned} l_L= & {} \begin{pmatrix} \nu _{\mu } &{}\\ \mu _{L}^- \end{pmatrix} ,\quad L_{L,R} = \begin{pmatrix} L_{L,R}^0 &{}\\ L_{L,R}^- \end{pmatrix},\nonumber \\ \Phi _{1}= & {} \begin{pmatrix} \Phi _{1}^+ &{}\\ \Phi _{1}^0 \end{pmatrix}\nonumber \\ \Phi _{2}= & {} \begin{pmatrix} \Phi _{2}^0 &{}\\ \Phi _{2}^- \end{pmatrix} \end{aligned}$$

(4)

As usual in 2HDM, we have,

$$\begin{aligned} \Phi _{1}^0= & {} v_1 + \frac{1}{\sqrt{2}}(-h \sin \alpha + H \cos \alpha ) \nonumber \\{} & {} \quad + \frac{i}{\sqrt{2}} (G \cos \beta - A \sin \beta ) \end{aligned}$$

(5)

$$\begin{aligned} \Phi _{2}^0= & {} v_2 + \frac{1}{\sqrt{2}}(h \cos \alpha + H \sin \alpha )\nonumber \\{} & {} \quad - \frac{i}{\sqrt{2}} (G \sin \beta + A \sin \beta ) \end{aligned}$$

(6)

$$\begin{aligned} \Phi _{1}^{\pm }= & {} \cos \beta G^{\pm } - \sin \beta H^{\pm } , \Phi _{2}^{\pm } \nonumber \\= & {} - \sin \beta G^{\pm } - \cos \beta H^{\pm } \end{aligned}$$

(7)

The 2HDM possesses five physical Higgs bosons a charged pair ($H^{\pm }$) two neutral CP-even scalars (h and H) and a neutral CP-odd scalar (A), often called a pseudoscalar. In paradigm of alignment limit the h is like SM Higgs boson. The charged gauge eigenstate $\phi ^{+}_1$ and $\phi ^{+}_2$ will give rise to one charged Higgs $H^+$ and a charged Goldstone boson($G^\pm$). The scalar sector of 2HDM is described in detail in [19]. After symmetry breaking the vacuum expectation value associated to the neutral components $<\phi ^{0}_1>=v_1$ and $<\phi ^{0}_2>=v_2$. Here, we have taken into account the following condition $\sqrt{v_1^2 + v_2^2}=v=174$ GeV between the vevs and we have define $\tan \beta = \frac{v_2}{v_1}$. As a result, the mass matrix is transformed to

$$\begin{aligned}{} & {} \begin{pmatrix} {\bar{\mu }}_L&{\bar{L}}^-_L&{\bar{E}}_L \end{pmatrix} M_e \begin{pmatrix} \mu _R \\ L^-_R \\ E_R \end{pmatrix} \nonumber \\{} & {} \quad = \begin{pmatrix} {\bar{\mu }}_L&{\bar{L}}^-_L&{\bar{E}}_L \end{pmatrix} \begin{pmatrix} y_{\mu } v_1 &{} 0 &{} \lambda _E v_1 \\ \lambda _L v_1 &{} M_l &{} \lambda v_1 \\ 0 &{} {\bar{\lambda }} v_1 &{} M_e \end{pmatrix} \begin{pmatrix} \mu _R \\ L^-_R \\ E_R \end{pmatrix} \end{aligned}$$

(8)

We need to diagonalize the mass matrix using

$$\begin{aligned}{} & {} U^{e\dagger }_L \begin{pmatrix} y_{\mu } v_1 &{} 0 &{} \lambda _E v_1 \\ \lambda _L v_1 &{} M_l &{} \lambda v_1 \\ 0 &{} {\bar{\lambda }} v_1 &{} M_e \end{pmatrix} \,\, U^e_R = \begin{pmatrix} m_\mu &{} 0 &{} 0 \\ 0 &{} m_{e_4} &{} 0 \\ 0 &{} 0 &{} m_{e_5} \end{pmatrix} \end{aligned}$$

(9)

The relevant mass eigenvalues are provided in Appendix A.

We have two charged vector lepton eigenstates with masses $M_L$ and $M_E$, respectively, where $e_4$ corresponds to $L^+$ and $e_5$ corresponds to $E^+$ and a neutral $\nu _4$ corresponds to LN. Simultaneously one part of the muon mass came from the yukawa term, and the other part cames from mixing with VLLs. For the sake of simplicity, we will use the neutral vector lepton ($\nu _4$) mass, which is provided by $M_L$.

We can obtain the effective lagrangian from Eq. (3) in the heavy mass limit of VLLs, which will put some light on the correlation of muon mass from VLLs mixing and the impact on $(g-2)_{\mu }$.

$$\begin{aligned} {\mathcal {L}} = - y_{\mu } {\bar{l}}_{L} \mu _R H_1 - \frac{\lambda _L {\bar{\lambda }} \lambda _E}{M_L M_E}{\bar{l}}_{L} \mu _R H_1 H_1^{\dagger } H_1 + h.c \end{aligned}$$

(10)

where the second term of this equation quantify a new source of muon mass due to VLLs [33]. By following this route, we may roughly estimate the contributions from each diagram in Fig 1 to obtain the transparent effect of the new Muon mass source on $(g-2)_{\mu }$.

The contribution from all 1-loop diagrams can be expressed in a single formula as

$$\begin{aligned} \Delta a^i_{\mu } \simeq \frac{k^i}{16\pi ^2}\frac{m_{\mu } m_{\mu }^{LE}}{v^2}, m_{\mu }^{LE} \equiv \frac{\lambda _L {\bar{\lambda }} \lambda _E}{M_L M_E}v^3 \cos ^3\beta \end{aligned}$$

(11)

where $k^W = 1$,$k^Z = -\frac{1}{2}$, $k^h = -\frac{3}{2}$,$k^H = -\frac{11}{12} \tan ^2\beta$, $k^A = -\frac{5}{12}\tan ^2\beta$ and $k^{H^{\pm }} = \frac{1}{3}\tan ^2\beta$ these are good approximation with heavy VLLs and $M_L \simeq M_E \simeq m_{H,A,H^{\pm }}$. To explain the $(g-2)_{\mu }$ in $\mu$2HDM requires very high value of $\tan \beta$ typically of O(1000). Usually such large value of $\tan \beta$ causes concern with perturbative theory, unitarity and electroweak precision observables etc. So to circumvent the large $\tan \beta$ problem in $\mu$2HDM we have extended the lepton sector with VLLs. Further we have restricted the VLLs mixing with muon only. Then from the original lagrangian Eq. (3) we have built an effective lagrangian (10). So now we have two sources of muon mass, one from the typical Yukawa term, and the other from the mixing with VLLs, which is a novel source of muon mass. This new mass parameter will linearly modify the muon yukawa couplings and generate a remarkable effect on $(g-2)_{\mu }$ correction. Now that we have a rough idea of the heavy Higgs contribution in (11) we can see that the factor $m_\mu ^{LE}$ related to VLLs is not present in the $\mu 2HDM$ and that it plays an intriguing role in the heavy Higgs contribution.

Fig. 1

The diagrams contributing the muon anomalous magnetic moment with $W, Z, h, H, A, H^{\pm }$

2.1 Contribution in $(g - 2)_{\mu }$

The contribution for $(g-2)_{\mu }$ anamoly is completely derived in the present model and other variation of 2HDM have presented in [37]. The mixing of vector like leptons with muon generate new diagrams for $(g-2)_{\mu }$ at one loop level which shown in Fig. 1. The one-loop diagramms are dominant compared to two loop Bar-Zee (BZ) diagramms with heavy fermions in the loop.The mixing of muon with new vector like lepton generate an additional contribution to $(g-2)_\mu$.The contribution coming from W and Z bosons [17] are.

The W boson contribution is

$$\begin{aligned} \Delta a_{\mu }^W= & {} \frac{m_{\mu }}{16 \pi ^2 M_W^2} \sum _{a=4,5} \Big [m_{\mu }\Big (|g_R^{W\nu _{a}\mu }|^2 {+} |g_L^{W\nu _{a}\mu }|^2 \Big )F_W(x_W^a) \nonumber \\{} & {} -m_{\nu _{a}} Re[g_R^{W\nu _{a}\mu }( g_L^{W\nu _{a}\mu })^*] G_W(x_W^a)\Big ] \end{aligned}$$

(12)

The Z-boson contribution to $(g-2)_{\mu }$ is then given by

$$\begin{aligned} \Delta a_{\mu }^{Z}= & {} \frac{-m_{\mu }}{8 \pi ^2 M_Z^2}\sum _{a=4,5}\Big [m_{\mu }\Big (|g_R^{Z_{\mu e_a}}|^2 + |g_L^{Z_{\mu e_a}}|^2 \Big )F_Z(x_W^a) \nonumber \\{} & {} -m_{e_{a}} Re[g_R^{Z_{\mu e_a}}(g_R^{Z_{\mu e_a}})^*] G_Z(x_Z^a)\Big ] \end{aligned}$$

(13)

The contribution from neutral Higgs bosons h, H and A are identical in nature except their couplings factors. For $\phi = h,H,A$ we can define the couplings of charged leptons to neutral Higgses by

$$\begin{aligned} \Delta a_{\mu }^{\phi }= & {} \frac{m_{\mu }}{32 \pi ^2 m_{\phi }^2}\sum _{a=4,5}\Big [m_{\mu } \Big (|g^{\phi }_{\mu e_a}|^2 + |g^{\phi }_{ e_a \mu }|^2 \Big ) F_{\phi }(x_{\phi }^a)\nonumber \\{} & {} + m_{e_{a}} Re[g^{\phi }_{\mu e_a} g^{\phi }_{ e_a \mu }] G_{\phi }(x_{\phi }^a)\Big ] \end{aligned}$$

(14)

The contribution to $(g-2)_{\mu }$ from loops with the charged Higgs is then given by

$$\begin{aligned} \Delta a^{H^{\pm }}_{\mu }= & {} \frac{- m_{\mu }}{16 \pi ^2 m_{H^{\pm }}^2}\!\sum _{a=4,5}\!\Big [m_{\mu } \Big (|g^{H^{\pm }}_{\nu _a \mu }|^2 {+} |g^{H^{\pm }}_{ \mu \nu _a }|^2 \Big )F_{H^{\pm }}(x_{H^{\pm }}^a)\nonumber \\{} & {} + m_{\nu _{a}} Re[g^{H^{\pm }}_{\nu _a \mu } g^{H^{\pm }}_{ \mu \nu _a}] G_{H^{\pm }}(x_{H^{\pm }}^a)\Big ] \end{aligned}$$

(15)

3 Constraints on the model parameters

3.1 Constraints from the Z pole measurements

The $\mu$2HDM+VLLs model allowed the mixing of only second generation lepton with the vector like leptons leading to modifications of muon couplings to W and Z bosons, this modification can affect the different observables like $\mu$ lifetime, the forward-backward and left-right asymmetries involving muons,the Z width into $\mu ^+ \mu ^-$ and $\nu _{\mu } {\bar{\nu }}_{\mu }$. The EW measurements constrain possible modification of couplings of the muon to the Z and W bosons [38, 39]. For the $\mu$2HDM+VLLs, the leptons as well as the VLLs couple exclusively to $\Phi _{1}$ so that the global electroweak fit for the vector like leptons which gives the following limit [40, 41].

$$\begin{aligned} \Big |\frac{\lambda _E v_1}{M_l}\Big |\lesssim 0.03 \end{aligned}$$

(16)

$$\begin{aligned} \Big |\frac{\lambda _L v_1}{M_e}\Big |\lesssim 0.04 \end{aligned}$$

(17)

3.2 Higgs diphoton decays with vector lepton

In the domain of alignment limit the tree level couplings to leptons and gauge bosons become exactly like SM. Along with the charged scalar $H^{\pm }$ of the 2HDM, the charged VLLs can contribute to the loop-induced decay mode of the Higgs into $\gamma \gamma$. As a result, our model must be compatible with the present Higgs to diphoton decay limit. The Higgs to diphoton decay width is expressed including the contribution coming from new particles (vector like leptons) in the loop as

$$\begin{aligned} \Gamma _{h\rightarrow \gamma \gamma }= & {} \frac{G_{F} \alpha ^2 m_h^3}{128\sqrt{2}\pi ^3}\Bigg | \kappa _V F_1(x_W)+\frac{4}{3}\kappa _{ht{\bar{t}}} F_{\frac{1}{2}}(x_t) \nonumber \\{} & {} + F_{\frac{1}{2}}(x_l) \kappa _{hl{\bar{l}}}+ \kappa _{hH^{+} H^{-}} F_{+}(x_{H^{\pm }})\Bigg |^2 \end{aligned}$$

(18)

where $x_j = (\frac{2m_j}{ m_h })^2, (j = W,t,f, H_{\pm }$), $m_h$ is the SM Higgs mass, $\kappa _{hl{{\bar{l}}}}$ ($\kappa _{hH^+ H^-}$) are the couplings of SM Higgs boson to vector-like fermions (charged Higgs) with mass $M_L (M_{H^\pm })$ respectively. The corresponding loop functions $F_1, F_{1/2}$ and $F_+$ which appear in the calculation are

$$\begin{aligned}{} & {} F_1(x)=2 + 3 x +3 x (2-x)f(x) \nonumber \\{} & {} F_{1/2}(x)=-2x[ 1+ (1-x)f(x)] \nonumber \\{} & {} F_{+}(x)=-x[1-x f(x)] \nonumber \\{} & {} f(x)=\Bigg \{[\sin ^{-1} (1/\sqrt{x})]^2, \ x \ge 1\nonumber \\{} & {} f(x)= {\left\{ \begin{array}{ll} (\sin ^{-1} (1/\sqrt{x}))^2,&{} x\ge 1\\ -\frac{1}{4}\left( \ln (\frac{1+\sqrt{1-x}}{1-\sqrt{1-x}})-i \pi \right) ^2, &{} x<1 \end{array}\right. } \end{aligned}$$

(19)

and, the charged Higgs couplings to the SM Higgs is given by [42]

$$\begin{aligned}{} & {} \!\!\!k_{hH^+ H^-}\nonumber \\{} & {} \quad =-\frac{1}{2M_{H^{\pm }}^2} \nonumber \\{} & {} \qquad \times \Big [\frac{(m_h^2 {-} 2M_{H^{\pm }}^2)\cos (\alpha {-} 3\beta ){+} (3m_h^2 {+} 2M_{H^{\pm }}^2 {-}4 m_0^2)\cos (\alpha {+} \beta )}{4\sqrt{2} \sin \beta \cos \beta }\Big ]\nonumber \\ \end{aligned}$$

(20)

where

$$\begin{aligned} m_0^2 = \frac{m_{12}^2}{\sin \beta \cos \beta } \end{aligned}$$

(21)

Table 2 Parameter values used to generate the allowed region in a plane

Therefore, we must carefully scrutinize how the model’s parameters affect higgs diphoton decay. We utilized the existing experimental limit to accomplish this. The present experimental limit on the strength of the Higgs to diphoton signal is quite close to its SM value and stands at $\mu _{\gamma \gamma } = \frac{\mu _{\gamma \gamma }^{exp}}{\mu _{\gamma \gamma }^{SM}}=1.18^{+0.17}_{-0.14}$ [43]. We may now define the ratio of decay width as describing the enhancement and suppression in $h \rightarrow \gamma \gamma$ channel because the VLLs no longer contributes to Higgs production.

Fig. 2

Restriction on ($m_0 - M_{H^{\pm }}$ ) (left) and ($m_0$ - $M_L$ ) (right). The light red is 1$\sigma$ and gray is 2$\sigma$ allowed region of $h\rightarrow \gamma \gamma$ signal strength. Here $m_0$ is the soft breaking parameter defined in the text, and $M_L$ is the mass for vector-like charged lepton

The constraints on mass parameters are demonstrated in Fig. 2 for a specific set of parameters (as given in Table 2). These graphs demonstrate how the experimental results can be satisfactorily explained by carefully adjusting the soft-symmetry breaking term $m_0$, the charged Higgs mass $M_{H^{\pm }}$, and the VLL mass $M_L$. It should be emphasised that, while we set a certain value for the VLL Yukawa coupling and $\tan \beta$, the correlation between the mass parameters is unaffected by our choice.

3.3 Constraints from electroweak precision observables

Important limitations come from the electroweak oblique parameters, since the extra scalars and leptons contribute to gauge boson masses via loop corrections, in addition to the Higgs data and the theoretical constraints established in the preceding subsections. The scalar contributions to the oblique T and S parameters are well-known, as shown in [44, 45]. The Z and W couplings with VLLs can be written as,

$$\begin{aligned} {\mathcal {L}}^Z= & {} \Big ({\bar{f}}_{La} \gamma ^{\mu } g_L^{Z f_a f_b} f_{Lb} + {\bar{f}}_{Ra} \gamma ^{\mu } g_R^{Z f_a f_b} f_{Rb}\Big )Z_{\mu } \end{aligned}$$

(22)

$$\begin{aligned} {\mathcal {L}}^W= & {} \Big ( \bar{{\hat{\nu }}}_{La} \gamma ^{\mu } g^{W\nu _a e_b}_L {\hat{e}}_{Lb} + \bar{{\hat{\nu }}}_{Ra} \gamma ^{\mu } g^{W\nu _a e_b}_R {\hat{e}}_{Rb} \Big )W^+_{\mu } + h.c \nonumber \\ \end{aligned}$$

(23)

The complete expression of $g_L$ and $g_R$ are given in Appendix A.

3.3.1 T-parameter

The additional fermion contribution formula [46]

$$\begin{aligned} \Delta T_F= & {} \frac{1}{8 \pi s_W^2 c_W^2}\!\sum _{a,b=2,4,5}\!\Big [ \big (|g_L^{W\nu _a e_b}|^2 {+} |g_R^{W\nu _a e_b}|^2\big ) \theta _{+}(f_a,f_b) \nonumber \\{} & {} + 2Re\big (g_L^{W\nu _a e_b} g_R^{W\nu _a e_b*}\big ) \theta _{-}(f_a,f_b) \nonumber \\{} & {} -\frac{1}{2} \big (|g_L^{Ze_a e_b}|^2 + |g_R^{Z e_a e_b}|^2\big ) \theta _{+}(f_a,f_b) \nonumber \\{} & {} + 2Re\big (g_L^{Z e_a e_b} g_R^{Z e_a e_b *}\big ) \theta _{-}(f_a,f_b) \Big ] \end{aligned}$$

(24)

where, $f_a =\frac{m_{e_{a}}^2}{M_z^2}$and the functions are defined as

$$\begin{aligned} \theta _{+}(x,y)= & {} {\left\{ \begin{array}{ll} \frac{x+y}{2}-\frac{x y}{x-y} \ln (\frac{x}{y}),&{} \text {if } x\ne y \\ 0, &{} \text {if } x = y \end{array}\right. }\nonumber \\ \theta _{-}(x,y)= & {} {\left\{ \begin{array}{ll} \sqrt{x y} \Big [ \frac{x+y}{x -y} \ln (\frac{x}{y}) -2 \Big ],&{} \text {if } x\ne y \\ 0, &{} \text {if } x = y \end{array}\right. } \end{aligned}$$

3.3.2 S-parameter

The general expression for the S-parameter contribution from additional fermions is [46],

$$\begin{aligned} \Delta S_F= & {} \frac{1}{2 \pi }\sum _{a,b=2,4,5}\Big [ \big (|g_L^{W\nu _a e_b}|^2 + |g_R^{W\nu _a e_b}|^2\big ) \psi _{+}(f_a,f_b) \nonumber \\{} & {} + 2Re\big (g_L^{W\nu _a e_b} g_R^{W\nu _a e_b*}\big ) \psi _{-}(f_a,f_b) \nonumber \\ {}{} & {} -\frac{1}{2} \big (|g_L^{Ze_a e_b}|^2 + |g_R^{Z e_a e_b}|^2\big ) \xi _{+}(f_a,f_b) \nonumber \\{} & {} + 2Re\big (g_L^{Z e_a e_b} g_R^{Z e_a e_b *}\big ) \xi _{-}(f_a,f_b) \Big ] \end{aligned}$$

(25)

where, $f_a =\frac{m_{e_{a}}^2}{M_z^2}$and the functions are defined as

$$\begin{aligned} \psi _{+}(x,y)= & {} \frac{1}{3} -\frac{1}{9}\ln \left( \frac{x}{y}\right) \nonumber \\ \psi _{-}(x,y)= & {} -\frac{x + y}{6 \sqrt{x y}}\nonumber \\ \xi _{+}(x,y)\!&{=}&\! {\left\{ \begin{array}{ll} \frac{5 (x^2 + y^2) {-} 22 x y }{9 (x- y)^2} {+} \frac{ 3 x y (x + y ) {-} x^3 - y^3 }{3 (x-y)^3} \ln \left( \frac{x}{y}\right) ,&{} \text {if } x{\ne } y \\ 0, &{} \text {if } x {=} y \end{array}\right. }\nonumber \\ \xi _{-}(x,y)\!&{=}&\! {\left\{ \begin{array}{ll} \sqrt{x y} \Big [ \frac{x+y}{6 x y} {-} \frac{x + y }{(x - y)^2} {+} \frac{2 x y }{(x - y)^3}\ln (\frac{x}{y}) \Big ],&{} \text {if } x{\ne } y \\ 0, &{} \text {if } x{=} y \end{array}\right. }\nonumber \\ \end{aligned}$$

(26)

Recently, [47] provided the values of these parameters based on an analysis of precision electroweak data, including the PDG-2021 and new result of the W-mass.

$$\begin{aligned} \Delta T&= 0.11 \pm 0.12 \nonumber \\ \Delta S&= 0.06 \pm 0.10 \nonumber \\ \Delta U&= -0.02\pm 0.09 \end{aligned}$$

(27)

Fig. 3

Allowed region of vector leptons couplings and permitted mass difference between $M_L$ and $M_E$ the S and T parameters bound

The precision observables can confirm the mixing of SM leptons and VLLs. The coupling $\lambda$ and ${\bar{\lambda }}$, which mixes the VLLs among themselves, is not constrained by the Z pole observables. Because the mass eigenstates of the heavy charged leptons are dependent on these couplings, these couplings can generate a mass gap between the neutral and charged components of the doublet. This can give correction to oblique T parameter [48, 49] and can be constrained. We have constrained the allowable parameter space with the most recent (PDG 2021) value of S,T and U that is provided in [47] as shown in Fig. 3. However, if we use the CDF(2022) value, as in [47] the parameter space is now allowed at $2 \sigma$ rather than $1 \sigma$.

The interesting thing is that for large mass gaps, all values of $\lambda$s are allowed, but when the mass gap between charged and neutral lepton $(M_L -M_E) = \Delta m =$3 GeV) is less than $\Delta m$ the parameters space is disallowed. But if we take the $\lambda$ value into account, the charged vector leptons, degenerates. We have demonstrated the impact of $\lambda$ on the mass degeneracy of charged vector leptons in Fig. 3d. When the value of the $\lambda$ reaches 0.5, the $\Delta m$ becomes zero.The scalar also has an impact on oblique parameters that must be considered. Even so, it has been demonstrated that by making the charged Higgs degenerate with the heavy scalar or the pseudo-scalar, the oblique corrections from the scalar sector of 2HDM can be minimised [44, 45]. Moreover, the tree level contribution of the charged Higgs exchange diagram to the leptonic decay process is insignificant in the existing $\mu$2HDM model. This is because of the cancellation of the $\tan \beta$ dependency, according to the muon-specific feature of this model.

4 Results

The analysis and interpretation of our study are the focus of this section. In order to describe the $(g- 2)_{\mu }$, we will first explore the promising out come and efficiency of the existing $\mu$2HDM model. The $\mu$2HDM model which is singular from all existing variant of 2HDM regarding the yukawa structure of leptons.

Due to chiral amplification in the closed fermion loop, two-loop contributions to $(g-2)_\mu$ from Barr–Zee (BZ) diagrams can comparable with one-loop predictions. The contribution from BZ-type diagrams with a neutral Higgs and photon in the internal legs is almost ${\mathcal {O}}$($10^{-4}-10^{-5}$) suppressed [37]. Therefore, these diagrams are irrelevant in comparison to one-loop contribution. For the purpose of further discussion and illustrative numerical analysis we have selected some benchmark the parameters (see Table 2) which satisfy all coupling constraints, Higgs to diphoton data, and the oblique parameter constraints, as mentioned in the past sections. Moreover, one may choose $\sqrt{4\pi }$ as the limit from perturbativity at the scale of new physics to keep the Yukawa coupling under perturbative control for large energy scales. However, We have taken into account all Yukawa couplings up to the value 1 in our numerical analysis for the same purpose. For the all numerical analysis we have considered $M_{L} > 600$ GeV and $M_{E} > 500$ GeV to typically satisfy constraints from direct searches for new leptons [36, 50]. Further we have considered $m_H = m_A = m_{H^{\pm }}$ and applied limits on $H(A) \rightarrow \tau ^+ \tau ^-$ [51] and $H^+ \rightarrow t {\bar{b}}$ [52] which are currently the strongest at small and large $\tan \beta$ respectively. These limits are also sufficient to satisfy indirect constraints from flavour physics observables [53].We have also imposed the constraints from muon electroweak data such as Z-pole observables,the W partial width and the muon life time and constraints from oblique corrections, [39]. The $h \rightarrow \mu \mu$ can be affected by variations in the Higgs muon coupling caused by the mixing of VLLs and muon. The constraint obtained from the experimentally determined value of $h \rightarrow \mu \mu$ must therefore be taken into account. We used the current experimentally observed signal strength $\mu _{\mu \mu } = 1.2 \pm 0.6$ [54] to confine the parameter space. We have also used the experimentally $h \rightarrow \tau \tau$ signal strength $\mu _{\tau \tau } = 1.09^{+0.27}_{-0.26}$ [55] to observe the impact on the permitted parameter space by $(g - 2)_\mu$. We have discovered that the parameter space that can explain the $(g - 2)_\mu$ at $1\sigma$ and $2\sigma$, are also permitted by $h \rightarrow \mu \mu$ and $h \rightarrow \tau \tau$.

In order to explain the $(g-2)_{\mu }$ we have to take those values of the parameters $m_{\mu }^{LE}$ (see Eq. 11) and $\tan \beta$, so that they satisfy the relation $\tan ^2\beta m_{\mu }^{LE} =-m_{\mu }$. This condition will enhance the contribution that are coming from the H, A and $H^{\pm }$ in the $(g-2)_\mu$ [33]. The enhancement caused by $m_{\mu }^{LE}$ for the low $\tan \beta$ region ($\tan \beta \sim 6$–12) is significantly greater than the improvement for the $\tan ^2\beta$ value. For this behavior at low $\tan \beta$ value the heavy Higgs contribute reasonably to accommodate the $(g-2)_{\mu }$. Also due to the $\tan ^2\beta$ enhancement when we increase the value of $\tan \beta$, the contribution due to heavy Higgs increases but simultaneously the value of $m_{\mu }^{LE}$ decreases. Therefore the modification of muon yukawa coupling and the gauge coupling remains under control.

In Fig. 4 we have plotted the allowed $1\sigma$ and $2\sigma$ region of parameters space satisfying experimental result $(g-2)_\mu$ in the heavy neutral Higgs mass - $\tan \beta$ plane. It is needed to mention here that the limit vary significantly with the assumed pattern of branching ratios of new leptons to W, Z and h [56].

Fig. 4

The allowed parameter space in heavy scalar–$\tan \beta$ plane for reproducing the correct value for the muon anomalous magnetic moment in $\mu$2HDM+VLLs. We show the constraints imposed by agreement with the $(g-2)_{\mu }$ at 1$\sigma$ (light red) and 2$\sigma$ (gray)

Fig. 5

The allowed parameter space in vector like leptons for reproducing the correct value for the muon anomalous magnetic moment in $\mu$2HDM+VLLs. We show the constraints imposed by agreement with the $(g-2)_{\mu }$ at 1$\sigma$ (light red) and 2$\sigma$ (gray)

Taking forward our analysis we have created the parameter spaces as a result of our investigation shown in Fig. 5 where we have illustrated the acceptable region after the restriction of $(g-2)_{\mu }$ for the $\mu$2HDM + VLLs architecture in several parametric planes. The colour conventions are same as in the previous figure, with the grey region representing the 2$\sigma$ permissible region and the bright red coloured region representing the 1$\sigma$ allowed region of the following parameter space. We can see from Fig. 5b that the higher values of $\tan \beta$, are not allowed. This is because of the fact that after that certain value of $\tan \beta$ the decreasing effect of $m_{\mu }^{LE}$ is so high that it can annihilate the enhancement effect due to $\tan \beta$ and decrease the overall contribution coming from heavy Higgs. The balancing effect of $\tan ^2\beta$ and $m^{LE}_{\mu }$ generate specific parameters space for $(g-2)_{\mu }$. One more thing we have to mention here is that in our model we successfully generate the parameter space in low $\tan {\beta }$ region which is obviously much better than the work done by [29]. Also in the previous work [29] they had only shown the allowed parameter space of $M_H$ with $\tan _\beta$, where as we can show all other allowed parameter space relevant to our study. If we look at the Fig. 5a, b it is showing the anti-correlation between the $\tan \beta$ and $M_L$ ($M_E$), for higher value of $M_L$ ($M_E$) we need lower values of $\tan \beta$. In Fig. 5c the correlation between the VLLs mass gives us an idea that to satisfy the $(g-2)_\mu$ anomalies, we can take only a certain combination of VLLs masses. At last, our findings in Fig. 5 highlight the characteristics of the model parameters that account for the $(g-2)_{\mu }$ anomaly.

5 Collider study

In this section we shall discuss the detector level simulation corresponding to VLLs (vector like leptons) in the multi-lepton channels. We have compared the outcomes of the cut based analysis and also multivariate analysis methods while analysing this model. There are many phenomenological studies [57,58,59] on VLLs in the literature, but for the simulation purpose we took the inspiration from the CMS study [36] for the search of VLLs. For the simulation purpose we have generate both signal and background events and compute the cross section in MG5 aMC@NLO [60] at the leading order (LO). The default PDF set NNPDF2.3LO [61] has been used for the event generation and computing the cross section. We have consider the main SM background for the multi-lepton channels are VVV, VV, $t{\bar{t}}V$, $t{\bar{t}}h$ and hV (where $V= W,Z$ boson). For each background and as well as signal processes we generate as many as 4 million events. The showering and hadronisation of the produced events are then processed within the Pythia 8.2 [62]. Then for the purpose of detector level simulation we use Delphes 3.4 [63], a fast detector simulation package. We use CMS card to reconstruct jets, electrons, muons, missing energy within the Delphes 3.4. We use the anti-kt algorithm for clustering the jets with a radius parameter $R= 0.4$ applying the FastJet package [64]. We have shown our simulation analysis for the centre of mass energy of 13 TeV and integrated luminosity of 139 $fb^{-1}$.

In our analysis we consider the lighter leptons i.e. e and $\mu$ to get the multi-lepton signal and put the object cuts on them. These objects are reconstructed with the identification efficiency of default CMS card. The events with multi-lepton signal then sorted in two different final state comprising of either two leptons or three leptons. The leading lepton must have to pass the trigger criteria mentioned in Ref. [36] to qualify as selected event. The restriction on missing energy, ( GeV) gives us a better discrimination of signal than SM background. The events with two opposite sign same flavour leptons are labeled as “OS” and the three lepton events are labeled as “3l”. Here we imposed a condition on invariant mass of the leptons, $M_{ll}$ for the signal region comprising of two opposite sign leptons. We have discarded the pair with $M_{ll}$ within 15 GeV of $M_Z$ for the cut based analysis. This reduces the SM background events which contains leptons coming from Z boson.

After all these cuts implemented on each of the above mentioned signal region, a minimum bound on $L_T$ has been set to get a significant excess of signal events over the SM background. $L_T$ is defined as the sum of the transverse momentum of all the signal leptons.

$$\begin{aligned} L_T = \sum _{l=e,\mu } p_{T}(l). \end{aligned}$$

We have chosen different minimum cuts on $L_T$ for different signal regions to get clear signal. This optimization of $L_T$ is given based on our simulation. We note in passing that this Monte Carlo simulations are only an approximation of the actual experimental capability. In general, in cut based analysis our findings shows that for a higher value of VLLs mass, at edge of the reach, the lower bound on $L_T$ can be increased to optimize the signal.

Table 3 The parameter values we have considered in order to construct the BPs

We use transverse mass ($m_T$) as a distinguishing variable in multivariate analysis along with all of these variables mentioned above. This variable is only used in the three-lepton final state because it is a good discriminator in that SR. The variable ($m_T$) is defined as $m_T = \sqrt{2 p_T^{miss} p_T^l [1 - \cos (\Delta \Phi _{m_T})]}$, where $p_T^l$ refers to the $p_T$ of the lepton that is not part of the OS pair closest to the Z boson mass and $\Delta \Phi _{m_T}$ is the difference in azimuth angle between $\textbf{p}_T^{~miss}$ and $\textbf{p}_T^{~l}$.

5.1 Cut based analysis (CBA)

Fig. 6

The production cross section of different VLLs. We have mentioned the different parameter values which have been considered in order to produce the vector like leptons

In the Fig. 6 we have shown the change in production cross section of different VLL with respect to their masses. From the Fig. 6 we can infer that the production cross section of $L^+ LN$ is the greatest among all the VLL production. Despite the fact that process $pp \rightarrow W^- \rightarrow L^- {\overline{LN}}$ and process $pp \rightarrow W^+ \rightarrow L^+ LN$ is conjugate, the production cross section of $L^- {\overline{LN}}$ is relatively small. This is due to the fact that the production cross section of $W^-$ is nearly 3 nb less than the production cross section of $W^+$ [65]. Here $L^-$ and LN are the charge and neutral particle from the VLL doublet and $E^-$ is the particle from the singlet. For this reason further detector level simulations have been done only taking this process into account. $L^+$ dominantly decays into $\mu ^+ H$, $\mu ^+ A$, $H^+ \nu _{\mu }$ and $E^+ \gamma$ channels, where H, A, $H^+$ are the heavy CP even, CP odd and heavy charged Higgs respectively. Likewise, the particle LN decays mainly into $\mu ^- H^+$, $W^+ \mu ^- H$ etc. channels. The heavy neutral CP even Higgs decays to SM model like Higgs and other channels containing jets. Charged Higgs decays dominantly in the final state comprising of jets. This decay topology gives us a dominant two leptons final state with a lesser amount of final state comprising of three leptons. From the decay topology one can understand that, for obtaining two or three lepton in the final state we need to allowed multi-jet in the final state. So our final state is the leptons with multi-jet and . From now on we use the term two or three lepton for the final state which contains all these object present in the signal.

We now introduce some benchmark points (BPs) to analyse the results further. For the purpose of illustration BPs are taken for the two different $\tan \beta$ values and for each $\tan \beta$ we have taken two different mass of VLL. All other parameters are shown in Table 3. All BPs are chosen such that, they satisfy all constraints coming from S, T, U parameter, $b \rightarrow s \gamma$ decay etc. which are mentioned in the previous Sect. 3. We have also checked the current limit [66] obtaining by the ATLAS collaboration on Chargino-Neutralino pair production to the final state of 2 leptons and missing energy. We have recasted the original analysis into CheckMATE [67] to check the bounds on our BPs. All our BPs are allowed by the limit given by the collaboration on the simplified SUSY model. However, we keep note in passing that these exclusion limit can be varied if we move from the simplified model to the realistic pMSSM models (for instance see [68]).

Fig. 7

We have displayed $L_T$ distribution in this plot for the final state comprising of three and two opposite sign leptons. The shaded region represent the SM background for this final state. The distribution of $L_T$ for the BPs are shown in different colors

In Fig. 7a we have shown the distribution of $L_T$ for the final state comprising of 3 leptons. We can determine from this distribution that a minimum cut must be applied to $L_T$ ($L_T >400$ GeV) in order to obtain a significant VLL signal event. We have shown the normalised event number after passing all the cuts in the Table 4 for the 3 lepton final state. There was no statistically significant excess of signal over background event because of the small BR to the three lepton channel. We have adopted the approach used by the reference [69] when calculating the signal significance. We assume a $10\%$ systematic uncertainty for further collider analysis while calculating the significance.

Table 4 In this table we have displayed the normalised event number ($N_{Sig}$) and the significance at 139 $fb^{-1}$ for the signal (BPs) and background event for the final state comprising of 3 leptons and two opposite sign leptons

Next we have discussed the signature of final state comprising of 2 leptons. In Fig. 7b we have displayed the $L_T$ distribution with normalised event comprising of the final state of 2 opposite sign leptons. We have put a cut on $M_{ll}$ to exclude the event that can emerge from Z boson. In this signal region we have put a minimum bound on $L_T$ ($L_T > 300$ GeV) to obtain the signal event. The normalised event numbers are shown in Table 4. From this Table 4 and the plots (Fig. 7a, b) we can say that, the SR 2 l-OS shows the substantial signal significance for VLLs signature at the low $\tan \beta$($\sim 10$) region compare to the high $\tan \beta$ region. Whereas the SR 3 l performs better at the high $\tan \beta$($\sim 30$) region as far as the significance concern. The CBA shows that we can minimize background for significant cut on $L_T (L_T > 1500$) GeV but with nominal number of events with this luminosity. As a result, this approach is inadequate for VLL signal analysis. It is very hard to get that 5$\sigma$ significance for discovery from that low signal significance. From Table 4 one can see that only BP1 and BP3 has the potential to be discovered in the two lepton final state. For reaching the potentially discover region we need 296 fb$^{-1}$ and 664 fb$^{-1}$ respectively for the BP1 and BP3 in the two lepton channel. In order to get a appreciable amount of signal significance for even high VLL mass with low $\tan \beta$ value we move to multivariate analysis in the following subsection.

5.2 Multivariate analysis

Table 5 The relative importance of the input variables utilised in MVA with $M_L= 1007$ GeV with $\tan \beta$= 10 and $M_E = 850$ GeV at $\sqrt{s}=13$ TeV. This could be different for different sets of parameters

Fig. 8

The signal (blue) and background (red) distributions of the input variables used for MVA have been shown in this figure. For the final state comprising of three leptons we have used the transverse mass ($m_T$) as one of the input discriminating variable. The invariant mass of two final lepton is only used as discriminating variable for two lepton final state. Other variables, which are described in the earlier section are same for these two SRs

Fig. 9

Over-training check of the BDT response for the parameter set $M_L= 1007$ GeV, $\tan \beta$= 10, $M_E = 850$ GeV at $\sqrt{s}=13$ TeV

We present a multivariate analysis in this subsection for better signal to background differentiation, which leads to an increment in significance. In the TMVA [70] framework within the ROOT [71], we implement the Boosted Decision Tree (BDT) algorithm for the multivariate analysis (MVA). To achieve substantial significance in the cut-based analysis discussed in the preceding subsection, we have identified an cut value for the variables. The MVA technique is a powerful tool for obtaining the optimal sensitivity for a given set of parameters for this purpose. We use four variables for each signal region for the MVA. In the Table 5 we have shown those variables along with the importance of those variables in the BDT response. These variables have been determined by comparing the background distributions with the signal trained for $M_L= 1007$ GeV with $\tan \beta$= 10 and $M_E = 850$ GeV at $\sqrt{s}=13$ TeV. We have incorporated a new variable, transverse mass ($m_T$), as the discriminating input variable for the BDT for the final state comprised of three leptons, in addition to the variables presented in the cut based analysis.

As seen in Fig. 8, where we have presented the normalised signal and background event distributions, each of these variables has a respectable level of discriminating power. It’s worth noting that the four variables utilised here may not be the best, and there’s always the possibility of improving the analysis by making better variable choices. We utilised these simple kinematic variables in our study since they are less correlated and have significant discriminating power. To better the analysis, a more focused MVA can be incorporated with distinct sets of variables for different parameter points.

The Kolmogorov–Smirnov (KS) test can be used to determine whether or not a test sample is over-trained. In general, if the KS probability is somewhere between 0.1 and 0.9, the test sample is not over-trained. A critical KS probability value greater than 0.01 assures that the samples are not over-trained in most scenarios. The KS probability values for the signal and background of the BDT response are presented in Fig. 9, indicating that neither the signal nor the background samples have been over-trained. We have made sure that we do not get over-trained on any of the parameter points we have mentioned. As seen in Fig. 9, the signal and background samples in this BDT output are well separated, allowing us to considerably enhance the signal significance by applying an appropriate BDT cut.

We can observe that in the high $\tan \beta$ region, the cut based analysis gives us a better outcome over SM background, whereas even in the low $\tan \beta$ region, BDT gives us a substantial signal significance. The MVA is significantly more effective to analyze the $\mu$2HDM+VLLs model relative to cut based analysis. We compute the significance given in Table 6 for various $M_L$ at $\tan \beta = 10$, $\sqrt{s}$ = 13 TeV and $L = 139 fb^{-1}$, and also shows the significance for different $\tan \beta$ values at a given $M_L = 1200$ GeV with same $\sqrt{s}$ and L. From this tables of MVA we can also conclude that 2 l-OS is substantially good for analysis as the signal significance is higher. The MVA analysis and thus the signal significance does depend on the VLL mass and $\tan \beta$ value as the corresponding signal strength is different. The BDT cut inherently adjust itself to discriminant the signal from the SM background with the significant precision and minimise the background. For that we have to specify some variables with reasonable importance. In this analysis we choose almost same sets of variables for 3 l and 2 l-OS except the $m_T$ for 3 l is replaced by $M_{ll}$ in case of 2 l-OS. Moreover at the high VLL mass where the cross section is small and makes the significance of CBA analysis is nominal. Whereas in MVA (see Table 6) the BDT cut helps to probe a better significance even in the high VLL mass (i.e. $M_L=1606$ GeV) region. We have also presented the required luminosity for the BPs to make a potential discovery in the Table 6. From the Table 6 we can infer that all the chosen points in the 2 l-OS search channel can be probed in the upcoming future colliders like HL-LHC. From the prospect of the discovery perspective we also can see the MVA analysis provide us a better result than CBA to probe this search channel in the upcoming and proposed collider.

Table 6 Numbers of signal ($N_S$) and the background events with 10$\%$ systematic uncertainty have been shown for different $M_L$ and $\tan \beta$ after passing the BDT cut. ’– ’ denotes the negligible background ($N_B\rightarrow 0$) compared to the signal. In case the background is negligible 5 signal events are considered as the requirement for the discovery

6 Summary and conclusion

The constant alignment between the prediction from the standard model (SM) and the experimental data from LHC so far has posed strong challenges for new physics (NP) scenarios beyond the standard model (BSM). But in this smooth path way the measurement of anomalous magnetic moment of the muon remains one of the continuing deviation from SM expectation [6]. Stretching this deviation forward the $(g-2)_\mu$ experiment [72, 73] consolidate the ground for BSM. Without vector-like leptons, the type-I and type-Y models cannot explain the discrepancy and the type II requires light pseudoscalar that are in conflict with BR$(B\rightarrow X_s \gamma )$ [74, 75]. The type-X model can accommodate the discrepancy but requires large values of $\tan \beta$ and also very light pseudoscalar [76]. The $\mu$2HDM can explain $(g-2)_{\mu }$ but it also requires very large $\tan \beta$ [27]. After revisiting $\mu$2HDM in light of the new result reported by the $(g-2)_{\mu }$ collaboration at Fermilab, we have studied the discrepancy of the $(g-2)_{\mu }$ in the $\mu$2HDM+VLLs with mixing. We restricted our analysis to the alignment limit, where the SM and lightest CP-even Higgs bosons coincide, ensuring agreement with LHC result. Firstly we have analysed the charged VLLs and extra charged scalar effect on the higgs diphoton decay. Also we have ensured the effect of modified higgs muon coupling on $h \rightarrow \mu ^+ \mu ^-$ decay. The parameter space of $\mu$2HDM+VLLs chosen to analyze the $(g-2)_{\mu }$ passed through the constraints from precision electroweak parameters, S,T and U. Additionally, the Higgs potential’s perturbativity, unitarity, and vacuum stability must be respected by keeping the coupling constants within certain bounds. Using these limits we have studied the discrepancy of $(g-2)_{\mu }$ in $\mu$2HDM+VLLs and tried to revel the effect of extra heavy scalar and the vector leptons. Here, we have taken into account the 1-loop contribution from additional scalar and vector leptons. In order to maintain consistency with the perturbative limit, we have taken into account all coupling values up to 1. To explain the discrepancy of $(g-2)_\mu$, the existing $\mu$2HDM needed a very large value of $\tan \beta$ where as by including the vector lepton with $\mu$2HDM, we can bring down the $\tan \beta$ value to a range between (5–10) for heavy extra scalar and vector leptons masses of the order of (TeV). We do not take into account additional BZ loops since we are only concerned with the effect in $(g-2)_\mu$ caused by VLLs mixing in $\mu$2HDM from 1-loop contribution. Even so, there’s a chance that the BZ-diagrams with charge higgs and a muon VLLs mixing loop can make a substantial difference. This issue will be addressed in our next project.

In the collider section we have focused our study to the VLL production and its decay to heavy Higgs boson. As $pp \rightarrow L^+ LN$ has the greater production cross section, all the collider study and results are showcased considering this process only. To distinguish the signal from the background, we have used two alternative strategies. Firstly we present a traditional cut-based analysis considering the various kinematic properties of the signal over background. Then, to improve the separation of the signal from the background, we perform a multivariate analysis using a boosted decision tree approach which enables us to obtain a better signal significance even in the high VLL mass region.

Appendix A

1.1 A. Diagonalizing mass matrices

If we consider the limit $\lambda _L v_1,\lambda _E v_1,\lambda v_1,{\bar{\lambda }} v_1 \ll M_L, M_E$ the approximate analytic formulas for diagonalization matrices can be obtained (Table 7) [17, 77]

$$\begin{aligned} U_L= & {} \begin{pmatrix} 1-\frac{v_1^2}{2} \frac{\lambda _E^2}{ M_E^2} &{} \frac{v_1^2}{2}\Big (\frac{\lambda _E}{M_L} \frac{{{\bar{\lambda }}} M_E + \lambda M_L}{M_E^2 - M_L^2} - \frac{y_{\mu } \lambda _L}{M_L^2}\Big ) &{} v_1\frac{\lambda _E}{M_E} \\ v_1^2 \frac{{{\bar{\lambda }}} \lambda _E M_L -y_{\mu }\lambda _L M_E}{M_L^2 M_E} &{} 1-v_1^2 \frac{(\lambda M_E + \lambda M_L)^2}{(M_E^2 - M_L^2)^2} &{} v_1 \frac{{{\bar{\lambda }}} M_L + \lambda M_E}{M_E^2 - M_L^2} \\ -v_1\frac{\lambda _E}{M_E} &{} -v_1 \frac{{{\bar{\lambda }}} M_L + \lambda M_E}{M_E^2 - M_L^2} &{} 1-v_1^2 \frac{\lambda _E^2}{ M_E^2} - v_1^2 \frac{(\lambda M_E + \lambda M_L)^2}{(M_E^2 - M_L^2)^2} \end{pmatrix} \end{aligned}$$

(28)

$$\begin{aligned} U_R= & {} \begin{pmatrix} 1-v_1^2 \frac{\lambda _L^2}{2 M_L^2} &{} v_1\frac{\lambda _L}{M_L} &{} v_1^2 \Big (\frac{\lambda _L}{M_E} \frac{{{\bar{\lambda }}} M_L + \lambda M_E}{M_E^2 - M_L^2} + \frac{y_{\mu } \lambda _L}{M_E^2}\Big ) \\ -v_1\frac{\lambda _L}{M_L} &{} 1-v_1^2 \frac{\lambda _L^2}{ M_L^2} - v_1^2 \frac{(\lambda M_E + \lambda M_L)^2}{(M_E^2 - M_L^2)^2} &{} v_1 \frac{{{\bar{\lambda }}} M_E + \lambda M_L}{M_E^2 - M_L^2} \\ v_1^2 \frac{{{\bar{\lambda }}} \lambda _L M_E -y_{\mu }\lambda _E M_L}{M_L M_E^2} &{} -v_1 \frac{{{\bar{\lambda }}} M_E + \lambda M_L}{M_E^2 - M_L^2} &{} 1 - v_1^2 \frac{({{\bar{\lambda }}} M_E + \lambda M_L)^2}{(M_E^2 - M_L^2)^2} \end{pmatrix} \end{aligned}$$

(29)

Table 7 The formal definition of the coupling between the W and Z bosons and the VLLs

1.2 B. Loop functions

The loop functions for W boson contribution

$$\begin{aligned} F_W(x)&=\frac{4 x^4 - 49 x^3 + 78 x^2 - 43 x +10 + 18 x^3 \ln (x)}{6 (1-x)^4} \\ G_W(x)&=\frac{- x^3 + 12 x^2 - 15 x + 4 - 6 x^2 \ln (x)}{ (1-x)^3} \end{aligned}$$

The loop functions for Z boson contribution

$$\begin{aligned} F_Z(x)&=\frac{5 x^4 - 14 x^3 + 39 x^2 - 38 x + 8 - 18 x^3 \ln (x)}{12 (1-x)^4} \\ G_Z(x)&= - \frac{ x^3 + 3 x - 4 - 6 x \ln (x)}{ 2(1-x)^3} \end{aligned}$$

The loop functions for scalar bosons where $\phi =\{ h,H,A \}$

$$\begin{aligned} F_{\phi }(x)&=\frac{x^3 - 6 x^2 + 3 x + 2 + 6 x^3 \ln (x)}{6 (1-x)^4} \\ G_{\phi }(x)&= \frac{ - x^2 + 4 x - 3 - 2 \ln (x)}{ (1-x)^3} \end{aligned}$$

The loop functions for $H^{\pm }$ contribution

$$\begin{aligned} F_{H^{\pm }}(x)&=\frac{2x^3 + 3 x^2 - 6 x + 1 - 6 x^2 \ln (x)}{6 (1-x)^4} \\ \nonumber G_{H^{\pm }}(x)&= \frac{ -x^2 + 1 + 2 x \ln (x)}{ (1-x)^3} \end{aligned}$$