Probing sterile neutrinos in B (D) meson decays at Belle II (BESIII)

We present, how a systematic study of B→DℓN\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$B \rightarrow D\ell N$$\end{document} (D→KℓN\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D \rightarrow K \ell N$$\end{document}) decays with ℓ=μ,τ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell =\mu ,\tau $$\end{document}, at Belle II (BESIII) can provide unambiguous signature of a heavy neutrino N and/or constrain its mixing with active neutrinos νℓ\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nu _\ell $$\end{document}, which is parameterized by |UℓN|2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$| U_{\ell N} |^2$$\end{document}. Our constraint on |UμN|2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vert U_{\mu N} \vert ^2$$\end{document} that can be achieved from the full Belle II data is comparable with what can be obtained from the much larger data set of the upgraded LHCb. Additionally, our method offers better constraint on |UμN|2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vert U_{\mu N} \vert ^2$$\end{document} for mass of sterile neutrino mN<2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m_N < 2$$\end{document} GeV. We can also probe the Dirac and Majorana nature of N by observing the sequential decay of N, including suppression from observation of a displaced vertex as well as helicity flip, for Majorana N.

Our main significant result in this paper is the stringent constraint that can be put on |U N | 2 , especially on |U μN | 2 , from non-observation of the decays B → D N , without considering the sequential decay of N . This simple strategy has, however, remained unexplored in the currently existing literature. Instead of considering two-body leptonic decays B + → + N , similar to existing studies on π + or K + → + N which look for mono-energetic + to constrain |U N | 2 [88][89][90][91][92][93][94][95][96][97][98][99][100][101][102][103][104][105][106][107], we have considered the threebody semileptonic decays B → D N which have bigger branching ratios in a larger mass range. The reach of our study to constrain |U μN | 2 and |U τ N | 2 is better by an order of magnitude from existing experimental constraints in certain mass ranges of interest. Interestingly, our constraint on |U μN | 2 obtained by considering only ∼ 4.8 × 10 8 events of fully reconstructed B → DμN decays at Belle II [108] is comparable with the constraint achievable from 4.8 × 10 12 events of B → Dμμπ decays at upgraded LHCb [36]. Although the missing sterile neutrino search gives the stringent constraint on |U N | 2 , it can not distinguish Dirac and Majorana neutrinos. Therefore, we also study the sequential decay of N with a displaced vertex signature for probing its Majorana nature, and consider the important but otherwise overlooked effect of helicity flip for sterile neutrinos. Despite the suppression coming from observation of displaced vertices as well as the helicity flip, we find that heavier and less energetic neutrinos have a bigger chance of decaying inside a detector with decay length 1 m, provided they exist. Finally, we present an estimate of |U μN | 2 in the case of observation of LNC B → Dμ + μ − π + decay in Belle II. The LNV mode B → Dμ + μ + π − receives additional suppression from helicity flip. This paper is organized as follows. In Sect. 2 we provide the logical basis for considering the decays B → D N or D → K N and the advantages they offer over choosing any other processes. In Sect. 3 we show how well the mixing parameters |U μN | 2 and |U τ N | 2 can be probed by using these decays at Belle II. We also provide a short discussion, via example, on possible SM background processes and how they can be distinguished from the signal events. This is followed by a discussion in Sect. 4 on how the Majorana nature of neutrino could be probed, and whether it is possible to do such a study. Finally we conclude in Sect. 5 highlighting the important features of our paper.

Choosing appropriate production modes
We aim to find a process that would (1) unambiguously probe the presence of sterile neutrino N , free from any other new physics possibilities, and (2) constrain the mixing of N with active neutrinos. In this regard, we find it helpful to keep in mind the following four cardinal aspects of sterile neutrino. Any candidate for sterile neutrino would have (i) electric charge = 0, (ii) spin = 1 /2, (iv) mass > 0, and (v) possibly long life-time. The fact that the sterile neutrino would most likely remain undetected at its point of production, is in fact the experimental consequence of its possibly long life-time and electrically neutral nature. This manifests as 'missing momentum' in any process which would have sterile neutrino(s) in the final state, just like the case with ordinary active neutrino(s) in any final state.
As examples of processes with 'missing momentum' we can consider meson decays such as B → D ( * ) μX , D → K ( * ) μX , B → K X etc., where X denotes missing (i.e. undetected) particle(s) other than active neutrinos. A simple analysis of spin would suffice to illustrate the fact that for decays such as B → D * μX and D → K * μX , the spin of X is ambiguous: it could be 1 /2 or 3 /2. Similarly, in the decay B → K X, the 'missing' X would indeed be made of (at least) two invisible particles and their individual spins could be 0, 1 /2 or 3 /2. In order to avoid such ambiguities, we shall refrain from considering these types of decays in this paper. Thus we are left to consider decays of the type B → DμN and D → K μN as decay modes suitable for discovery of sterile neutrino N .
It should be noted that the literature dealing with sterile heavy neutrino searches is replete with LNC and LNV processes mediated by sterile neutrino N , such as B → D π, B → π, τ → π 1 2 ν etc. where , 1,2 = e, μ. These processes, can also take place via other new physics possibilities, such as exotic scalars, vectors or lepto-quarks. Nevertheless, the LNV processes mediated by N constitute the only known reliable methodology to probe the conjectured Majorana nature of N . Notwithstanding the importance, these neutrino mediated decays are suppressed from associated displaced vertices, branching ratio of sequential decay of N as well as helicity flip of N (relevant in case of LNV modes only). Therefore, when concerned with the discovery prospect of sterile neutrino, we shall refrain from considering the sequential decay of N . Once an unambiguous signature of existence of sterile neutrino N is obtained, study of its Dirac or Majorana nature becomes highly relevant, and in this context we would consider the feasibility of the complete LNC and LNV modes which take into account the sequential decay of N .
Let us now analyze the decays B → DμN (or D → K μN ) with N remaining undetected in the detector. We are interested in the scenario where there is one light sterile neutrino with mass m N 3.3 GeV. The criteria of massive sterile neutrino helps us to eliminate background events to our processes. The decay B → DμN , for example, can receive background from the decay B → Dμν μ νν or B → D * (→ Dπ soft /γ soft )μν, where the soft pion (π soft ) or soft photon (γ soft ) arising from sequential decay of D * are not detected by the detector. However, the invariant mass obtained from the missing 4-momenta of background process would vary significantly from one event to another unlike the signal case which would be centered about the fixed mass of N .
In order to obtain the 'missing mass' (m miss ) in the processes, say B → Dμ + 'missing' that includes the signal events for B → DμN also, we need to know the 4-momenta of initial B meson ( p B ), final D meson ( p D ) and μ ( p μ ): For signal events only m miss = m N . This methodology is applicable in experiments such as Belle II or BESIII where B and D mesons are pair produced along with B and D from the decays ϒ(4S) → B B and ψ(3770) → D D respectively, and the 4-momentum of B, D can be precisely measured by full hadronic reconstruction. It should be, therefore, clear that our methodology is not applicable for experiments such as LHCb where the initial 4-momentum of the B or D meson can not be inferred without measuring the 4-momenta of the final particles arising from the B or D meson decay. Furthermore, the minimum value of mass m N = 0 that can be probed in our approach is, therefore, constrained only by the experimental accuracy of measurement of 4-momenta of B, D and μ. In the next section we would provide a numerical comparison of a few observables (including m N ), for the SM background decay B → Dμνπ soft /γ soft and the signal decay B → DμN , specifically in context of Belle II. It is important to note that N may or may not decay inside a detector, depending on its mass, energy and the size of the detector. If it decays inside the detector, with noticeable displaced vertex, we can not only measure its 4-momentum directly from its decay products, but also probe its Dirac or Majorana nature as well as veto any background events for the decays under consideration.
Note that the decays B + → τ + N and B → Dτ N , where the 4-momentum of the tau lepton is reconstructed from its further sequential decay, are less promising for our study, due to the presence of at least one neutrino (or antineutrino) in the final state of all tau decays. Nevertheless, taking into account that the tau 4-momentum could be measured accurately with a smaller probability, we shall constrain |U τ N | 2 from B → Dτ N .
3 Determining or constraining the value of |U μN | 2 and |U τ N | 2 The branching ratios of all the decay modes under our consideration are directly proportional to the appropriate activesterile mixing parameter |U N | 2 . We obtain the canonical branching ratio of a decay, e.g. B → D N , by factoring out |U N | 2 from the theoretically calculable branching ratio [86], Given the value of canonical branching ratio Br (B → D N ), the number of such decays observable in the detector (N B→D N ) and the total number of fully reconstructed parent particles (N B ), we can estimate |U N | 2 by where D , denote the efficiency to reconstruct the D, in the signal side. In our numerical study discussed below we have assumed D = μ = 1, but τ = 0.001 (the reason of which is given later).
For a numerical study we consider the decays B → DμN and B → Dτ N in context of Belle II experiment, which is poised to detect 10 11 B decay events [108]. Out of these, about 0.61% of charged B events and 0.34% of neutral B events can be fully reconstructed from hadronic tagging [108], so that only about 4.8 × 10 8 events of B decays get fully reconstructed. Considering only these B decays, we are able to estimate the value of |U μN | 2 , as shown in Fig. 1a, from possible observation of 50 events or less for B → DμN . It is easy to observe that in the mass range ∼ 2 − 3 GeV our approach can provide stronger constraint, by about one order of magnitude, than the existing experimental upper-limit (exclusion region at ∼ 90% C.L. from various experiments is shown by the shaded region in gray). Figure 1a also shows that our constraint is comparable with the 95% C.L. upper-limit on |U μN | 2 , shown as a thick solid line, predicted in Ref. [36] based on 4.8 × 10 12 B decay events at upgraded LHCb (with the decay B → Dμμπ ). For m N < 2 GeV (important for light sterile neutrino searches) our constraint significantly surpasses the above-mentioned constraint predicted for LHCb upgrade. This is primarily due to the suppression factors affecting the observation of B → Dμμπ decays inside a finite-sized detector for smaller values of m N (see the next section and Figs. 4 and 6). It is worth mentioning that the method proposed in this paper cannot be applied to LHCb, for it requires full reconstruction of the rest of the event so that missing energy-momentum can be used to extract the information about N .
Similarly, we can constrain |U τ N | 2 from number of observed B → Dτ N decays if the 4-momentum of the final τ could be measured accurately. In Fig. 1b, Fig. 2 we compare the signal event distributions vs. the SM events and SM background events, with respect to the energy of muon (E μ ), the missing energy (E miss , with E miss = E N for signal events), the missing mass (m miss , with m miss = m N for signal events) and the invariant mass-square s = ( p B − p D ) 2 . It is clear from Fig. 2 that   1 Those numbers of signal events can be easily seen from Fig. 1a. 2 See Fig. 2, and please note that we have considered for each mass m N the number of events/0.1 GeV. the missing mass distribution is the most useful one among all the observables, as the NP scenarios with m N > 1 GeV are easily discernible. Nevertheless, combinations of all the observables could be used to look for the sterile neutrino signature. Since the SM and SM background processes have much larger statistics and they are well understood both theoretically and experimentally, one could, in principle, implement multi-variate analysis or likelihood studies to figure out NP cases for m N < 1 GeV.
We have demonstrated how the active-sterile mixing parameters can be probed without considering any sequential decay of sterile neutrino, provided the 4-momenta of all the other particles are well measured. However, as mentioned before and as is well known, the Dirac and Majorana nature of the sterile neutrino can be probed only when its sequential decay inside detector is considered.

Probing the Dirac and Majorana nature of the neutrino N
If the sterile neutrino N decays inside a detector, we can probe lepton number violation in the entire process (which includes both the production of N and its sequential decay) to ascertain its Majorana nature. As an example, let us consider the sequential decay, The meson-level Feynman diagrams for these decays are shown in Fig. 3. It is very clear that observation of the lepton number violating mode B 0 → D − μ + μ + π − would imply that the sterile neutrino has Majorana nature. While reconstructing the sterile neutrino from the final states μ ∓ π ± in the detector, we must also include an observable spatial separation between the point of production and the point of decay of the sterile neutrino. We can also consider the decay N → τ ∓ π ± if allowed by kinematics. Similar analysis as above can be done for D 0 → K − μ + μ ∓ π ± and related decays as well. For a detector of finite size, say L D , the observation of displaced vertex with decay length L necessarily demands that L < L D , and this depends on the lifetime and energy of the sterile neutrino.
The feasibility of studying the Dirac and Majorana signatures of the sterile neutrino N (of mass m N , energy E N and total decay rate N ) via B → Dμμπ decays by using a detector of finite size L D depends on two important factors, (1) P decay (L), the probability of decay of N within L < L D , and (2) P flip , the probability of helicity flip required for observation of the LNV decays which characterize the Majorana neutrino, and these are given by We can quantify the feasibility of observing the full decay B → Dμμπ at Belle II, by the following distribution of events with respect to E N , where N L NC , N L N V , respectively, denote the number of LNC and LNV events, N B is the total number of B mesons produced/analyzed in the experiment, B is the total decay rate of the B meson, and D , μ,1 , μ,2 , π are the various efficiency factors corresponding to the reconstruction of final D meson, μ from the first vertex, μ from the second vertex and the π , respectively. For our numerical study we have assumed all these efficiency factors to be 1. It should be noted that inclusion of P flip (m N , E N ) is an effective way of introducing the helicity flip. The most accurate way is to consider the full decay B → Dμμπ along with the propagator for the intermediate N and that would automatically lead to m N dependence (similar to the mass-dependence (a) (b) Fig. 3 Meson-level Feynman diagrams contributing to the decays The sterile neutrino is produced at the first vertex and decays at the second vertex, which is at an observable distance away from the first vertex. The circular blobs connote the contributions from the corresponding hadronic form factors and decay constants. The cross in the Majorana scenario denotes the helicity flip involved in the decay Fig. 4 Distribution of the number of events of the lepton number conserving B → Dμ ± μ ∓ π decays. Here we have considered the displaced vertices not larger than 1 m so that the events can be observed at Belle II found in neutrinoless double-beta decay, a classic example of a LNV process). By including helicity-flip factor in dN L N V (m N , E N , L) /d E N it is clear that for m N → 0, the difference between Dirac and Majorana cases vanish as per the 'practical Dirac-Majorana confusion theorem' [110,111].
In the numerical study shown in Fig. 4 we have studied the energy distribution of the μπ system originating from the decay of N . It is noticeable that for m N 2 GeV scenarios, no B → Dμ ± μ ∓ π decays with characteristic displaced vertices 1 m are expected to be observable at Belle II (considering the size of the central drift chamber [108]) if we consider the existing constraints on |U μN | 2 (see Refs. [35,36] for details of existing constraints). However, if we consider |U μN | 2 = 10 −5 , the smaller mass scenarios are also feasible. It must be noted that the number of events shown here are larger than the ones shown in Fig. 2c simply because we have considered the full sample of B mesons here whereas for Fig. 2c only the smaller number of fully reconstructed B decays were considered. Figure 4 shows that as the mass of neutrino gets smaller the mean distance of displaced ver-tices gets bigger which would put most of the decays outside the Belle II detector, unless there is compensation by a substantial increase in the value of |U μN | 2 which facilitates an appreciable number of events still happening inside the detector. This is also clearly discernible from Fig. 5 where we have varied the decay length L within which the observed displaced vertices would lie. As per the existing constraint on |U μN | 2 if we wish to observe B → Dμ ∓ μ ± π events in a detector for m N 2 GeV, our existing detectors are clearly not suitable. Nevertheless, our approach elaborated in previous sections might come handy in the search for discovery of such sterile neutrino(s). It should be noted that the energy E N in both Figs. 4 and 5 are measured in the rest frame of the parent B meson. Figure 5 also illustrates an interesting behavior that the probability of decaying within a smaller decay length is larger when the neutrino is less energetic or equivalently more non-relativistic. Moreover, as we go to higher masses, the neutrino decays faster with larger decay width and shorter life time.
Finally assuming that one can observe the full decays at Belle II, we can estimate the value of the active-sterile mixing parameter |U μN | 2 as a function of observed number of events. In Fig. 6 we have analyzed how the number of events varies for different values of m N and |U μN | 2 . We have also considered the additional suppression from helicity flip factor P flip while considering the LNV mode. For the total decay rate of N which enters P decay (L) we have used Eqs. (30)(31)(32) of Ref. [86] and it depends on |U eN | 2 , |U μN | 2 and |U τ N | 2 . Since |U eN | 2 is already constrained by 0νββ experiments to be much smaller than |U μN | 2 and |U τ N | 2 (see Ref. [35]), we can safely neglect its contribution. It can be inferred from Fig. 6 that for |U μN | 2 smaller than the existing experimental upper limit, one can still aspire to observe more than a handful of B → Dμμπ decays at Belle II with displaced vertex signatures.

Conclusion
In this paper we offer a new strategy that could lead to the discovery of a heavy sterile neutrino N of mass m N 3.3 GeV and having appreciable mixing with active neutrinos, at the Belle II experiment. Consideration of the defining properties of a sterile neutrino such as it being an electrically neutral, spin-1 /2 fermion with non-zero mass and possibly long lifetime, irrespective of its Dirac or Majorana nature, points out that decays such as B → D N , with = μ, τ and without considering the sequential decay of N , could play a crucial role in the discovery of N . Once N is discovered, it is possible to probe the Majorana nature via searching for the lepton number violating processes.
We have shown that at Belle II, with only about 4.8 × 10 8 events of fully reconstructed B → DμN decays, one can  It is certainly alluring to consider the full decay final states of B → Dμμπ as they can be used to probe Dirac or Majorana nature of N . However, these decays can have other new physics contributions in addition to contribution from N . Considering the contribution from N as the major one, such decays still suffer suppression from displaced vertices that must lie within the finite sized detectors, as well as from branching ratio of the sequential N decay and possible spinflip associated with the Majorana signature. Thus, despite their undisputed strength, these suppressed decays would be difficult to observe and discovery of N might be missed if such decays are considered. For some masses, especially the heavier mass of N , however, we can still expect to observe a few such events even at Belle II, albeit the lower number of B mesons produced as compared to LHCb.
In summary, in this paper we propose the most systematic approach to probe a sterile neutrino N of mass m N 3.3 GeV, by using the decays B → D N (with = μ, τ ) rather than the suppressed B → 1 2 π, X 1 2 π decays (with 1,2 = e, μ and X = π, D) previously used at Belle and Fig. 6 Numerical study of feasibility of observing Dirac (LNC) signal B → Dμ ± μ ∓ π and Majorana (LNV) signal B → Dμ ± μ ± π , inside Belle II detector with decay lengths less than 1 m. Here we have neglected the contributions from both |U eN | 2 and |U τ N | 2 when compared with |U μN | 2 LHCb [112,113]. Our proposal does not require the sequential decay of N . The constraint on |U μN | 2 thus achievable from Belle II is not only better than the existing experimental constraints in certain mass range of m N (viz. 0.4-1 GeV and 2-3 GeV), but also better than the constraint that is achievable at upgraded LHCb for m N < 2 GeV. The minimum value of m N that can be probed is constrained only by experimental accuracy of measurement of 4-momenta of B, D and μ. The sequential decay of N , useful to distinguish its Dirac or Majorana nature by observing the LNC or LNV modes respectively, is suppressed from observation of displaced vertices (for both LNC and LNV cases) and helicity flip (for LNV case only). Our numerical study shows if no decays of N get observed within decay lengths 1 m, the existing experimental upper limit on |U μN | 2 can be improved.

Data Availability Statement
This manuscript has no associated data or the data will not be deposited. [Authors' comment: This is a theo-retical study and sources of all experimental data used are cited at the appropriate places in the article.] Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecomm ons.org/licenses/by/4.0/. Funded by SCOAP 3 .