Heavy neutrino search via semileptonic higgs decay at the LHC

Abstract

In the inverse see-saw model the effective neutrino Yukawa couplings can be sizable due to a large mixing angle between the light $(\nu )$and heavy neutrinos (N). When the right handed neutrino (N) can be lighter than the Standard Model (SM) Higgs boson (h). It can be produced via the on-shell decay of the Higgs, $h\rightarrow N\nu$ at a significant branching fraction at the LHC. In such a process N mass can be reconstructed in its dominant $N\rightarrow W \ell$ decays. We perform an analysis on this channel and its relevant backgrounds, among which the $W+$jets background is the largest. Considering the existing mixing constraints from the Higgs and electroweak precision data, the best sensitivity of the heavy neutrino search is achieved for benchmark N mass at 100 and 110 GeV for upcoming high luminosity LHC runs.

1 Introduction

The current experimental results on the neutrino oscillation phenomena [1], including the recent measurements of the so-called reactor angle [2,3,4,5,6,7], have established the existence of neutrino masses and flavor mixings, which require us to extend the Standard Model (SM). The seesaw extension of the SM [8,9,10,11,12,13,14] is probably the simplest idea for explaining the very small neutrino masses naturally, where the SM-singlet heavy right-handed Majorana neutrinos induce the dimension five operators leading to very small Majorana neutrino masses (the seesaw mechanism [8,9,10,11,12,13,14]). The seesaw scale varies from the intermediate scale to the electroweak scale as we change the neutrino Dirac Yukawa coupling ($Y_D$) from the scale of top quark Yukawa coupling ($Y_D \sim 1$) to the scale of electron Yukawa coupling ($Y_D \sim 10^{-6}$).

In high energy collider experimental point of view, it is interesting if the heavy neutrino mass lies at the TeV scale or smaller, because such heavy neutrinos could be produced at high energy colliders, such as the Large Hadron Collider (LHC) and the Linear Collider (LC) being projected as energy frontier physics in the future. However, since the heavy neutrinos are singlet under the SM gauge group, they obtain the couplings with the weak gauge bosons only through the mixing via the Dirac Yukawa coupling. For the seesaw mechanism at the TeV scale or smaller, the Dirac Yukawa coupling is too small ($Y_D \sim 10^{-6}-10^{-5}$) to produce the observable amount of the heavy neutrinos at the colliders.

There is another type of seesaw mechanism so-called the inverse seesaw [15, 16], where the small neutrino mass is obtained by tiny lepton-number-violating parameters, rather than the suppression by the heavy neutrino mass scale in the ordinary seesaw mechanism. In the inverse seesaw scenario, the heavy neutrinos are pseudo-Dirac particles and their Dirac Yukawa couplings with the SM lepton doublets and the Higgs doublet can be even order one, while reproducing the small neutrino masses. Thus, the heavy neutrinos in the inverse seesaw scenario can be produced at the high energy colliders through the sizable mixing with the SM neutrinos.

Since any number of singlets can be added to a gauge theory without introducing anomalies, one could exploit this freedom to find a natural alternative low-scale realization of the seesaw mechanism. In the low scale seesaw,¹ the SM is extended by $n_1$ SM singlet RHNs $N_R$ and $n_2$ sterile neutrinos S. For the simplicity we consider a basis where the charged leptons are identified with their mass eigenstates. Hence before the electroweak symmetry breaking (EWSB) we write the general interaction Lagrangian as

$$\begin{aligned} -{\mathcal {L}}_{int}= & {} Y_1 \overline{\ell _L} H N_R+ Y_2 \overline{\ell _L} H S + M_N \overline{N_R^c} S + \frac{1}{2} \mu \overline{S^c} S \nonumber \\&+ \frac{1}{2} M_R \overline{N^c_R} N_R+ h. c. \end{aligned}$$

(1)

where $\ell _L$ and H are the SM lepton and Higgs doublets, respectively. $Y_1$ and $Y_2$ are the Yukawa coupling matrices of dimensions $3\times n_1$ and $3\times n_2$ respectively. $M_R$ and $\mu$ are Majorana mass matrices for $N_R$ and S of dimensions $n_1\times n_1$ and $n_2\times n_2$, respectively. Due to the presence of $\mu$ and $M_R$ mass parameters the lepton number is broken. After the EWSB breaking, from Eq. 1 we get

$$\begin{aligned} -{\mathcal {L}}_{mass}= & {} M_{D} \overline{\nu _L} N_R + M \overline{\nu _L} S + M_N \overline{N^c_R} S+ \frac{1}{2} \mu \overline{S^c} S \nonumber \\&+ \frac{1}{2} M_R \overline{N^c_R} N_R+ h. c. \end{aligned}$$

(2)

where $M_D=Y_1 \frac{v}{\sqrt{2}}$, $M=Y_2\frac{v}{\sqrt{2}}$ and $<H>=\frac{v}{\sqrt{2}}$. Hence the neutral fermion mass matrix can be written as

$$\begin{aligned} -{\mathcal {L}}_{mass} =\frac{1}{2} \begin{pmatrix} \overline{\nu _L}&\overline{N^c_R}&\overline{S^c} \end{pmatrix} \begin{pmatrix}0&{}M_D&{}M\\ M_D^T&{}M_R&{}M_N\\ M^T&{}M_N^T&{}\mu \end{pmatrix} \begin{pmatrix}\nu _L^c\\ N_R\\ S \end{pmatrix}. \end{aligned}$$

(3)

From Eq. 3 we can get a variety of the seesaw scenarios by setting respective terms to be zero.² The simplest scenario is the inverse seesaw [15, 16] model which has been studied in [18, 19] using vacuum stability and fitting the neutrino oscillation data considering M and $M_R$ to be zero [15, 16]. Sub-matrices $M_N$ and $\mu$ did not arrive from the $SU(2)_L$ symmetry breaking whereas $\mu$ is the lepton number violating mass term. Hence they might follow the hierarchy $M_N>> M_D>>\mu$. The value of $\mu$ can be small by ’t Hooft’s naturalness criteria [20] since the expected degree of lepton number violation becomes naturally small. In a common scenario each of $M_N$, $M_D$ and $\mu$ are $3\times 3$ matrices (See, Ref. [21] where a minimal scenario has been studied. In this article we consider a minimal scenario where two generations of the RHNs are involved such a scenario can satisfy the neutrino oscillation data. The effective light neutrino mass matrix can be written under the seesaw approximation as

$$\begin{aligned} M_{\nu }^\mathrm{{light}} \sim M_D (M_N^T)^{-1} \mu M_N^{-1} M_D^T \end{aligned}$$

(4)

where as in the heavy sector we will have the three pairs of degenerate pseudo-Dirac neutrinos of masses of order $M_N\mp \mu$. The smallness of $M_\nu ^\mathrm{{light}}$ is naturally obtained from both of the smallness of $\mu$ and $\frac{M_D}{M_N}$. Hence $M_\nu ^\mathrm{{light}}\sim {\mathcal {O}}(0.1)$ eV can be obtained from $\frac{M_D}{M_N}\sim 0.01$ and $\mu \sim {\mathcal {O}}(100)$ eV. Thus the seesaw scale can be lowered considering $Y_1\sim {\mathcal {O}}(0.1)$ which implies $M_D \sim 10$ GeV and $M_N \sim 1$ TeV. The inverse seesaw scenario has also been discussed in the supersymmetric context in Ref. [22] (and also the references there in). The inverse seesaw scenario has been discussed under the general parametrization in [23] using Casas-Ibarra conjecture for general $Y_D$. In [25, 26] the Casas- Ibarra parametrization has been used to study the inverse seesaw scenario. A generalized scenario of the inverse seesaw has been discussed under the left-right scenario has been discussed in Ref. [24]).

We rather simplify the scenario a bit further with respect to [19]. In a simplified scenario $M_D$ and $M_N$ can be the diagonal matrices where as the flavors are encoded in the $\mu$ matrix. This is called the Flavor Diagonal (FD) scenario. Explicit numerical fits are also given in [23] using the neutrino oscillation data, non-unitarity effects and lepton flavor violation measurements. In the collider analysis we consider a minimal set up where both of $M_N$ are proportional to the $2\times 2$ unit matrix $(\mathbf {1}_{2\times 2})$ where the entire flavor mixing structure lies in $\mu$ which is another $2\times 2$ matrix keeping $Y_D$ as a diagonal matrix proportional to $\mathbf {1}_{2\times 2}$. Such a scenario can also reproduce the neutrino oscillation data. It means that there are two degenerate generations of each of $N_R$ and S whose mass can be considered at the TeV scale. Such a scenario has also been used in Ref. [23]. Such heavy neutrinos can be observed at the LHC from a variety of production processes [27]. We first study a model-independent search for high luminosity LHC runs and then interpret the search prospects with a benchmark FD inverse seesaw scenario. Due to flavor dependence in electroweak precision and Higgs decay constraints, we consider benchmark FD case in which both the first two flavor (electron and muon) heavy pseudo-Dirac pairs are at the TeV scale. Due to the degeneracy we consider that both of the electron and muon flavor RHNs (N) have the same mass, and their decays into electron and muons contribute to our collider signal.

For LHC production we focus on the $pp\rightarrow h j$ channel, where the Higgs boson subsequently decays as $h\rightarrow N \nu$ via the $Y_1{\bar{L}}HN_R$ interaction term. The Higgs boson can be copiously produced by gluon fusion at the LHC, and due to its relatively narrow $\sim$MeV scale decay width, the Higgs boson decay branchings are more sensitively affected by the presence of a new $h\rightarrow N \nu$ channel, if compared to the decay of W, Z bosons. When N decay leptonically the $h \rightarrow 2l2\nu$ channel has been previously studied in [28, 32, 34], and here we will examine the $h \rightarrow 2jl\nu$ channel from the semileptonic N decay, where a N mass peak is reconstructible in the final state. As we will discuss later, an associated jet is necessary for this Higgs decay channel both for event triggering and the SM background veto.

Our paper is arranged in the following way. In Sect. 2 we discuss the recent experimental bounds on the heavy neutrino searches. In Sect. 3 we discuss about the $h+j$ production and the decays of the Higgs boson into the heavy neutrino. In Sect. 4 we focus on the semileptonic Higgs decay channel and study the LHC search. A model-independent constraint is derived on the heavy-active neutrino mixing angle, and we comment on its effectiveness in the Inverse Seesaw model. Then we conclude in Sect. 5.

2 Bounds on the mixings

Being the SM gauge singlets, the heavy mass eigenstate of neutrinos can interact with the W and Z bosons via its mixings into the SM neutrino. Due to such mixing, the SM neutrino flavor eigenstate $(\nu )$ can be expressed as a linear combination of the light $(\nu _m)$ and heavy $(N_m)$ mass eigenstates,

$$\begin{aligned} \nu \simeq U_{\ell m}\nu _m + V_{\ell N} N_m, \end{aligned}$$

(5)

where U is the $3\times 3$ light neutrino mixing matrix being identical to the PMNS matrix at the leading order if we ignore the non-unitarity effects. Where as $V_{\ell N} \simeq m_D M_N^{-1}$ is the mixing between the SM neutrino and the SM gauge singlet heavy neutrino assuming $|V_{\ell N}| \ll 1$. The charged current (CC) and neutral current (NC) interactions can be expressed in terms of the mass eigenstates of the neutrinos as

$$\begin{aligned} {{\mathcal {L}}_{CC} \supset -\frac{g}{\sqrt{2}} W_{\mu } {\bar{e}} \gamma ^{\mu } P_L V_{\ell n} N_n + \mathrm {h.c}.}, \end{aligned}$$

(6)

where e denotes the three generations of the charged leptons, and $P_L =\frac{1}{2} (1- \gamma _5)$ is the projection operator. Similarly, in terms of the mass eigenstates the neutral current interaction is written as

$$\begin{aligned}&{\mathcal {L}}_{NC} \supset -\frac{g}{2 c_w} Z_{\mu } \left[ {\overline{N}}_m \gamma ^{\mu } P_L (V^{\dagger } V)_{mn} N_n \right. \nonumber \\&\quad \left. + \left\{ {\overline{\nu }}_m \gamma ^{\mu } P_L (U^{\dagger }V)_{mn} N_n + \mathrm {h.c.} \right\} \right] , \end{aligned}$$

(7)

where $c_w=\cos \theta _w$ with $\theta _w$ being the weak mixing angle. We notice from Eqs. 6 and 7 that the production cross section of the heavy neutrinos at the high energy collider is proportional to $|V_{\ell N}|^2$. However, the Yukawa coupling in Eq. 1 can also be directly measured from the decay mode of the Higgs boson such as $h\rightarrow N \nu$. The corresponding Yukawa coupling can be written as

$$\begin{aligned} {\mathcal {L}} \supset Y_D \frac{v}{\sqrt{2}} {\overline{\nu }}_{L} h N_R \end{aligned}$$

(8)

using $<H> = \begin{pmatrix} \frac{v+h}{\sqrt{2}}\\ 0 \end{pmatrix}$ where $V_{\ell N} =\frac{M_D}{M_N} =\frac{ Y_D v}{\sqrt{2}M_N}$. Applying the bounds obtained from the invisible Higgs boson decay widths we can measure the allowed parameter regions for $Y_D$ and $V_{\ell N}$. The recent and the projected bounds on the mixing angle as a function of $M_N$ from different experiments are shown in Figs. 1 and 2.

For $M_N < M_Z$, the RHN can be produced from the Z-decay through through the NC interaction with missing energy. The heavy neutrino can decay according CC and NC interactions. Such processes have been discussed in [29,30,31]. In [31,32,33,34], a scale dependent production cross section at the Leading Order (LO) and Next-to-Leading-Oder QCD (NLO QCD) of $N\nu$ at the LO and NLO have been studied at the 14 TeV LHC and 100 TeV hadron collider.

Fig. 1

Experimental upper bounds on $|V_{e N}|^2$ as a function of $M_N$

Fig. 2

Experimental upper bounds on $|V_{\mu N}|^2$ as a function of $M_N$

The L3 collaboration [35] has performed a search on such heavy neutrinos directly from the LEP data and found a limit on ${\mathcal {B}}(Z\rightarrow \nu N)< 3\times 10^{-5}$ at the $95\%$ CL for the mass range up to 93 GeV. The exclusion limits from L3 are given in Figs. 1 and 2 where the red dashed line stands for the limits obtained from e $(L3-e)$ in Fig. 1 and the red dotted line stands for the exclusion limits coming from $\mu$ $(L3-\mu )$ in Fig. 2.

The corresponding exclusion limits on $|V_{(\ell =e) N}|^2$at the $95\%$ CL [36, 37] have been drawn from the LEP2 data in Fig. 1. This is denoted by the dark magenta line. In this analysis they searched for $80~\mathrm {GeV} \le { M}_{ N} \le 205~\mathrm {GeV}$ with a center of mass energy between 130 GeV to 208 GeV [37]. The LEP2 [37] has studied the $e^+e^- \rightarrow N\nu$ process followed by the $N\rightarrow e W$ mode to study the bounds on the corresponding mixing angle involved in the analysis. The bounds denoted by LEP2 have been taken from [37] where the data collected with the L3 detector for 208 GeV center of mass energy.

The DELPHI collaboration [38] had also performed the same search from the LEP-I data which set an upper limit for the branching ratio ${\mathcal {B}} (Z\rightarrow N \nu )$ about $1.3\times 10^{-6}$ at $95\%$ CL for 3.5 GeV $\le M_{N} \le 50$ GeV. Outside this range the limit starts to become weak with the increase in $M_N$. In both of the cases they have considered $N\rightarrow W\ell$ and $N\rightarrow Z \nu$ decays after the production of the heavy neutrino was produced. The exclusion limits for $\ell =e$ and $\mu$ are depicted by the blue dotted (dashed) lines for $e(\mu )$in Fig. 1 (2).

The heavy neutrinos can participate in many electroweak (EW) precision tests due to the active-sterile couplings. For comparison, we also show the $95\%$ CL indirect upper limit on the mixing angle, $|V_{\ell N}| < 0.030 ~\mathrm {and}~0.041$ for $\ell =e~(\mu )$ respectively derived from a global fit to the electroweak precision data (EWPD), which is independent of $M_N$ for $M_N > M_Z$, as shown by the horizontal purple dot-dashed (dashed ) lines respectively in Fig. 1 (2) [40,41,42]. For the mass range, $M_N < M_Z$, it is shown in [43] that the exclusion limit on the mixing angle remains almost unaltered, however, it varies drastically at the vicinity of $M_N =1~\mathrm {GeV}.$ For the flavor universal case the bound on the mixing angle is given as $|V_{\ell N}|^{2}=0.025$ from [40] which has been depicted in Figs. 1 and 2 with a purple solid line. Improvements in the EWPD has been observed in [39] for the general seesaw and three extra heavy neutrino cases. The $2\sigma$ bound allowed for $|V_{eN}|^2$ is below $2.5\times 10^{-3}$ for the lepton flavor conserving case for the general seesaw described by [39] and the bound for $|V_{\mu N}|^2$ is $4.4\times 10^{-4}$. In the three extra heavy neutrino case the $2-\sigma$ bound is shown as the same for the general seesaw case irrespective of the neutrino mass hierarchies. Where as the bounds on $|V_{\mu N}|^2$ for the NH case is $< 4.0\times 10^{-4}$. That for the IH case is $< 5.3 \times 10^{-4}$. These limits are all under good agreement with the parameter spaces shown for the different mixing matrix elements applied in [23] for the inverse seesaw and calculated in [44] for the seesaw cases with appropriate general parametrization.

The relevant $95\%$ CL upper limits are also shown to compare with the experimental bounds using the LHC Higgs boson data in [28] (also see, [27]) using the $2\ell 2\nu$ final state from the $WW^*$ data at the LHC [45,46,47,48,49] for $\ell =e$ and $\mu$ combined. In this case $h \rightarrow N \nu , N \rightarrow W \ell , W \rightarrow \ell \nu ~(h \rightarrow N \nu , N \rightarrow Z \nu , Z \rightarrow 2\ell )$ mode has been considered to probe the mixing in [27, 28]. The darker green solid line named Higgs boson shows the relevant bounds on the mixing angle in Figs. 1 and 2. In this analysis we will compare our results taking this line as one of the references. We have noticed that the $|V_{\ell N}|^2$ can be as low as $4.86\times 10^{-4}$ while $M_N=60$ GeV and the bound becomes stronger at $M_N=100$ GeV as $3.73\times 10^{-4}$. When $M_N > 100$ GeV, the bounds on $|V_{\ell N}|^2$ become weaker.

LHC has also performed the direct searches on the Majorana heavy neutrinos. The ATLAS detector at the 7 TeV with a luminosity of 4.9 fb$^{-1}$ [50] studied the $\mu ^{\pm }\mu ^{\pm }+\mathrm {jets}$ in the type-I seesaw model framework for 100 GeV $\le M_{N} \le 500$ GeV. They performed the analyses at the 8 TeV LHC with a luminosity of 20.3 fb$^{-1}$ in [51] and interpreted the limit in terms of the mixing angle, $|V_{\mu N}|^2$ which is shown in the Fig. 2. The corresponding bounds for the $\mu$ are shown by the dashed orange line and marked as ATLAS8-$\mu$ in Fig. 2.³

Fig. 3

Production processes of the heavy neutrino via Higgs boson decay with one associated jet. The extra jet originates from either the initial state or part of the hard process

The CMS also studied the type-I seesaw model from the $e^{\pm }e^{\pm }+\mathrm {jets}$ and $\mu ^{\pm }\mu ^{\pm }+\mathrm {jets}$ final states in [52] at the 8 TeV LHC with a luminosity of 19.7 fb$^{-1}$ with 30 GeV$\le M_{N} \le 500$ GeV. The limits from the CMS in the for $\mu$ is roughly comparable to the DELPHI result while $M_N < 70$ GeV. The CMS limits are denoted by CMS8-$\mu$ and CMS8-e with the magenta dashed and dotted lines respectively in Fig. 2. The prospective high luminosity limits have been shown in [44, 53]. In Eq. 7, there is a part where the heavy neutrino can produced in a pair from the NC interaction where the production cross section will be proportional to $|V_{\ell N}|^4$. The corresponding limits for the electrons are given in Fig. 1. The 8 TeV limits for the muons (electrons) are denoted as CMS8-$\mu$ (CMS8-e).

A detailed scale dependent LO and NLO-QCD studies of this process followed by various multi-lepton decays of the heavy neutrino have been studied in [54]. It is shown that 95 GeV$\le M_{N} \le 160$ GeV could be probed well at the high energy colliders at very high luminosity while the results will be better than the results from EWPD.

The updated limits at the 13 TeV LHC with a luminosity of $35.9\hbox { fb}^{-1}$ have been shown in Figs. 1 and 2 from [55] for electron and muon respectively. The corresponding limit for the e ($\mu$) is shown by the black dotted (dashed) line which marked as CMS13-e (CMS13-$\mu$). Recently the CMS has performed the trilepton search from the Majorana RHNs [56] at the 13 TeV LHC with a luminosity of 35.9 fb$^{-1}$. The corresponding bounds for the e ($\mu$) flavors are shown by the brown dotted (dashed) lines which are marked with CMS13-$3\ell$-e (CMS13-$3\ell$-$\mu$).

In this work we consider the heavy neutrino from the on-shell decay of the Higgs boson. Therefore we choose ‘benchmark’ heavy neutrino masses below the Higgs boson mass, and adopt the experimental bounds on the mixing angles to forecast a maximally allowed production rate. We also give the production rates for a generic range of the mixing $|V_{\ell N}|^2= 10^{-3}$ to $10^{-8}$ that are relevant to the current and prospective bounds.

Fig. 4

Upper bounds on the leading-order production cross sections of $N \nu$ from $h+j$ process with maximally allowed mixing angles, with $p_{T}^{j} > 200$ GeV at $\sqrt{s}= 13$ TeV. The electron and muon flavor curves deviate due to different EWPD and LHC constraints. The right panel shows the single-flavor signal cross-section at fixed mixing angle values. For the FD case, the signal cross-section doubles as two flavors can contribute

3 Higgs boson + jet cross-sections

The Higgs boson can decay into a right handed pseudo-Dirac heavy neutrino and a SM neutrino via the $\nu -N$ mixing. If $M_N$ lies below the Higgs boson mass, the Higgs boson can decay on-shell into the heavy neutrino through a single production channel shown in Fig. 3.

The Higgs boson’s SM decay width is taken as $\Gamma _{h}^\mathrm{{SM}}=$ 4.1 MeV, with allowance to fit in BSM physics where the Higgs boson can decay into the SM singlet heavy neutrino in association with missing energy. The partial decay width is given by

$$\begin{aligned} \Gamma (h\rightarrow N \nu )= \frac{Y_{N}^{2}}{8 \pi m_h^3} (m_h^2-M_N^2)^2 \end{aligned}$$

(9)

and it sums $h\rightarrow N{\overline{\nu }}$ and $h \rightarrow {\overline{N}}\nu$ cases. The branching fraction of the Higgs boson to each heavy neutrino is⁴

$$\begin{aligned} {\mathcal {B}}_{h\rightarrow N\nu } =\frac{\Gamma (h\rightarrow N \nu )}{\Gamma _{h}^\mathrm{{SM}}+\Gamma (h\rightarrow N \nu )} \end{aligned}$$

(10)

We focus on the signal channel of single Higgs boson production with an associated jet, and utilize the consequent decay of the Higgs boson. The inclusion of an extra jet is necessary due to the requirement of experimentally triggering on the event, and also due to the fact that most of the Higgs boson decay products are not very energetic without a transverse boost from the associated initial state jet.

The search channel $pp\rightarrow h j$ needs a large $p_T$ jet as event trigger and to reduce the amount of the SM background. Due to a large jet $p_T$, the hj production is generated at one-loop with a next-to-leading order model, see Sect. 4 for details. Including the Higgs boson decay branching ratios, the signal cross-section for a single heavy neutrino can be written as

$$\begin{aligned} \sigma =\sigma (h+j) {\mathcal {B}}_{h\rightarrow N\nu }, \end{aligned}$$

(11)

where the Higgs boson decay branching fraction ${\mathcal {B}}_{h\rightarrow N\nu }$ depends upon $M_N$ and the size of $|V_{lN}|^2$. For each $m_N$, we will consider the current experimental bounds on $|V_{lN}|^2$ and use the maximal experimentally allowed ${\mathcal {B}}_{h\rightarrow N\nu }$ for the optimal signal rate. The maximally allowed production cross section is shown in Fig. 4 at 13 TeV LHC, with the requirement of the leading jet $p_T^j > 200$ GeV.

To calculate the prospective cross section in this channel, we consider the experimental mixing angles constraint from leptonic Higgs channel, as discussed in [27, 28]. While the Higgs boson bound is most stringent in a large N mass range, at N mass between 100 and 110 GeV, the EWPD bound [40] becomes stronger. We use the stronger of the two constraints to produce an upper bound of $|V_{lN}|^2$, and the heavy neutrino production cross section for the $h+j$ channel.

For the convenience of estimating generic signal rates, we also show the signal cross sections at fixed mixing angle values in Fig. 4. Note that $|V_{\ell N}|^2=10^{-5}$ will be nearly ${\mathcal {O}}(1)$ magnitude below the constraint obtained in [27, 28, 40] in case only a single lepton flavor considered. Note that The FD case for the ‘benchmark’ mixing angles can be nearly twice as large as the corresponding single flavor cases.

The produced heavy neutrino will then decay via the SM weak bosons such as W, Z (and h for heavier N). The corresponding decay widths are given in [23, 31]. N lighter than W and Z bosons will decay into three-body channels through the virtual W and Z bosons. The corresponding partial decay widths are given in [58, 59]. Note that the W channel will typically dominate both two-body decay, shown in Fig. 5. In our analysis, we require the reconstruction of both dijet mass at $M_W$ and ljj invariant mass at $M_N$ to veto against SM backgrounds. Note the ljj system’s mass window cut is $M_N$ dependent, and should be tried for each choice of the $M_N$ in the relevant parameter range. In case of a signal, if present, the determination of $M_N$ may either come from $M_{ljj}$ reconstruction or more sophisticated $M_N$-dependent template fits on the final state kinematics.

Fig. 5

Decay of the heavy neutrino in the $\ell j j$ mode through the W boson

4 Collider signals and backgrounds

For successful triggering and background suppression, we require the leading jet $p_T^j$ in $pp\rightarrow h j$ event to be at least 200 GeV. This jet is also more energetic than Higgs decay products and it assumes the role of triggering jet. At the same time, this jet transversely boosts the Higgs boson system so that the Higgs boson decay products acquire larger $p_T^j$ and become more visible.

The Higgs boson then can decay into an $N-\nu$ pair. We focus on the $N\rightarrow \ell jj$ channel in which all three daughter particles are visible. The two jets from N arises from the on-shell decay of a W boson, so that their invariant mass would reconstruct to $M_W$. The lepton + dijet invariant mass would also reconstruct to $M_N$. These two invariant mass window cuts greatly suppress the SM backgrounds.

Fig. 6

Invariant dijet (left) and lepton+dijet (right) masses out of the three jets in signal events. N ($M_N=100$ GeV) decay jets are mostly represented by $j_2$ and $j_3$. In these histograms, the signal events only assume selection cuts $N_j\ge 3$ and $N_{\ell }\ge 1$

The after-cut cross-section is inferred from the $pp\rightarrow h j$ cross-section, decay branching ratios, and the selection efficiencies, as

$$\begin{aligned} \sigma = \sigma (hj){\mathcal {B}}_{h\rightarrow N\nu } {{\mathcal {B}}_{N\rightarrow \ell j j}} A_{\text {eff}}. \end{aligned}$$

(12)

For the selection efficiency $A_{\text {eff}}$, we consider the following detector-level cuts:

• 1. leading jet $p_T>$200 GeV;

• 2. Additional two or more jets with $p_T > 30$ GeV and exactly one lepton with $p_T > 15$ GeV;

• 3. $|M(j_2 j_3)-M_W|<20$ GeV;

• 4. $|M(l_1 j_2 j_3)-M_N|<20$ GeV;

• 5. GeV.

The selection cuts are designed to reconstruct the characteristic heavy neutrino mass as well as the physical W boson from N decay. These cuts are implemented at detector-level on Monte-Carlo simulated events. The leptons and jets pass basic detector pseudorapidity and $p_T$ cuts (specified later), and they are ordered descendingly by $p_T$. The large leading jet $p_T^j$ is important in suppressing weak boson + jets backgrounds. Vetoing a second lepton removes backgrounds with Z bosons. Here we focus on the hadronic W decay in order to reconstruct both the W boson and the N masses. These cuts greatly reduces SM backgrounds while retaining signal events at a much higher acceptance rate. Note that a fully leptonic decay of N can yield more leptons and suffer fewer SM background channels, but it also yields a neutrino and makes it impossible to reconstruct $M_N$.

Compared to the triggering jet, the N decay jets are mostly the second and third by $p_T$ ordering. As illustrated in Fig. 6, an $M_W$ peak is the most statistically pronounced between $j_2$ and $j_3$ among the three leading jets.

In the list of requirements, a few comments are due for the transverse mass $M_T$ cut. After reconstructing the W and heavy neutrino N masses, significant SM background, esp. the W+jets channel, can still fake a heavy neutrino from a leptonically decayed W boson and two additional jets. To further remove such contamination, we make use of $M_T$ of the lepton and missing energy system, defined as,

(13)

In signal events, l and would originate from the limit mass gap between W and h bosons, while for Wj background they are from the physical W boson. This $M_T$ nicely separates the signal and the leading Wj background, as illustrated in Fig. 7.

A number of the SM backgrounds are relevant for the $3j+\ell$ final state. The leading background channels typically arise from the presence of a W boson, from either direct production or top quark decay, along with extra jets. The leading background include $W+$jets, and top-quark producing channels. A large leading jet $p_T$ is the most effective selection against the $W+$jets channel, but it would also suppress the signal rate. Top quark included backgrounds can be efficiently controlled by the N mass-window cut.

Fig. 7

The transverse mass can effective separate semileptonic $h\rightarrow N\nu$ decay and a W decay, the latter being the leading SM Wj (solid) and $t{\bar{t}}$ (dashed) backgrounds. The cross-sections are normalized to better demonstrate the respective spectral shape. Here the heavy neutrino mass $m_N=110\hbox { GeV}$

In order to obtain the selection efficiencies we use the NLO model of the RHN as described in [31] and perform a 1-loop level simulation of $pp\rightarrow h j$ events with MadGraph5$\_$aMC@NLO [60] code and its the Pythia-PGS package for event showering and detector simulation. Pile-up is not included. The 1-loop level calculation gives the leading-order cross-section for high jet-$p_T$ Higgs production via gluon fusion. Additional jets and radiation are handled by Pythia. For a detector setup, we require a jet pseudo-rapidity $|\eta ^{j}|<2.5$, lepton pseudo-rapidity $|\eta ^{\ell }|<2.4$, minimal jet and lepton transverse momenta $p_T^j$ and $p_T^{\ell }$ at 30 GeV and 15 GeV. respectively.

For background simulation, we use an ‘MLM’ jet-matched [61, 62] cross-section for the inclusive for the W / Z+jets process with up to three additional jets. The $t{\bar{t}}$ channel uses a jet-matched cross-section for up to two additional jets. Other background channels are sub-leading and we only show-case their leading-order cross-sections. CMS has recently reported measurements on 13 TeV inclusive $t{\bar{t}}$ and $W+$jets channels. We adopt 746 pb [63] for $t{\bar{t}}$ production and 69 pb for $W+$jets production at $p_T^{j_1}\ge 100$ GeV [64]. Experimental measurements on the $Z+$jets channel [65] is more complicated to infer as it contains virtual photon contamination. We use the same measurement-to-simulation ratio as in $W+$jets to correct the $Z+$jets channel due to the kinematic similarity between the two channels.

Table 1 The SM background (left) and signal (right) cross-sections after selection cuts 1–3 (upper), and after selection cuts (4–5) with different $M_N$ windows (lower). The inclusive cross-sections for $t{\bar{t}}$ and W / Z+jets are corrected to recent 13 TeV measurements. Other background channels are sub-leading and given at their lowest order. The signal cross-section is at LO and is given without the Higgs decay branching ratio, i.e. $\sigma _\mathrm{sig.}/{{{\mathcal {B}}}}_{h\rightarrow N\nu }$, as a model independent result. The signal cross-section with $M_N=100$ and 110 GeV assume a maximal mixing parameter at $|V_{lN}|^2=\,$3.9$\times$10$^{\text {-4}}$ and 6.3$\times$10$^{\text {-4}}$, respectively

The significant background channels are listed in Table 1 that shows the efficiency flow of the event selection cuts. For signal rates, we list two benchmark N masses at 100 and 110 GeV that optimize these selection efficiencies. Lower N masses would observe a reduced selection efficiency due to softer lepton energy and/or lower rate in reconstruction of a physical W mass.

We found the a residue total background cross-section of 0.1–0.16 pb. For a generic estimate with ${{{\mathcal {B}}}}_{h\rightarrow N\nu }$ at $\{5\%,3\%,1\%\}$ at future LHC with 3000 fb$^{-1}$ luminosity, the sensitivity $S/\sqrt{S+B}$ is $\{2.1, 1.3, 0.4\}$ and $\{2.2, 1.3, 0.4\}$ for $M_N=100$ GeV and $M_N=110$ GeV. This sensitivity may improve by including NLO signal contribution in future studies. Note our selection cuts (1–5), in particular the leading jet trigger, are based on the current LHC design. For now we will assume similar trigger and cuts to estimate the sensitivity for future high lumonisity. These cuts can be further optimized in case design upgrades become available.

In the 100–110 GeV mass range, this upper limit on $|V_{\ell N}|^2$ is dominated by leptonic Higgs search from LHC and it is $M_N$ dependent. EWPD is most stringent in the $M_N<100$ GeV range. The $|V_{\ell N}|^2$ bound assume flavor-blind coupling to all three lepton generations. We only consider the first two lepton generations and do not include the tau lepton channel due to lower tagging efficiency, plus the fact that only a fraction of the tau energy is visible. Both $h\rightarrow N_\mu \nu _\mu , N_e\nu _e$ channels contribute equally to our search. By Eq. 10 the corresponding total $\mathcal{B}_{h\rightarrow N\nu }$ in the FD case of the inverse seesaw model is 4% and 3% for $M_N=100$ and 110 GeV. The LO sensitivity $S/\sqrt{S+B}$ at 3000 fb$^{-1}$ luminosity will be 1.7 at $M_N=100$ GeV, and 1.3 at $M_N=110$ GeV.

5 Conclusion

We investigated the prospects of probing the single-production of a heavy RHN from the on-shell decay of the SM Higgs boson at the LHC at 13 TeV. In the framework of the inverse see-saw model, a sizable neutrino mixing angle can be allowed. Due to the small decay width of the SM Higgs boson, a significant $h\rightarrow N \nu$ branching fraction can be allowed within the current bounds on the $N\nu$ mixing.

We adopt the $pp\rightarrow h j$ process as the search channel where the SM Higgs boson decays into the RHN followed by $N\rightarrow W\ell$ and $W\rightarrow j j$. One high $p_T$ associated jet is required for triggering and also to transversely boost h decay products as well as better background suppression. A leading order calculation of the $pp\rightarrow h j$ process is carried out at one-loop level in signal event generation. The N mass is reconstructed in $N\rightarrow \ell W$, followed by $W \rightarrow q {\overline{q}}^{\prime }$. A transverse mass cut is further introduced to reduce the SM $t{\bar{t}}, W+$jets contributions.

We found a selection efficiency at 1–3% for $M_N$ close to the Higgs boson mass and a reduced efficiency for lighter N. For a few benchmark N masses at 100 and 110 GeV, a leading order signal cross-section seems to be sub-fb after relevant selection requirements, compared with a total background of 0.1–0.16 pb. The significance at $2\sigma$ can be achieved at 3000 fb$^{-1}$ runs for a 5% branching ratio for $h\rightarrow N\nu$ decay. At the maximally allowed $N\nu$ mixing angle, the inverse model gives 4% and 3% at $h\rightarrow N\nu$ branching ratio and 1.7$\sigma$ and 1.3$\sigma$ signal significance at 3000 fb$^{-1}$. Note $pp\rightarrow h j$ is a QCD dominated process and future NLO calculations may enhance these significance prospects.