Sterile neutrino portal dark matter in $\nu$THDM

Abstract

In this paper, we propose the sterile neutrino portal dark matter in $\nu$THDM. This model can naturally generate tiny neutrino mass with the neutrinophilic scalar doublet $\Phi _{\nu }$ and sterile neutrinos N around TeV scale. Charged under a $Z_2$ symmetry, one Dirac fermion singlet $\chi$ and one scalar singlet $\phi$ are further introduced in the dark sector. The sterile neutrinos N are the mediators between the DM and SM. Depending on the coupling strength, the DM can be either WIMP or FIMP. For the WIMP scenario, pair annihilation of DM into NN is the key channel to satisfy various bounds, which could be tested at indirect detection experiments. For the FIMP scenario, besides the direct production of DM from freeze-in mechanism, contributions from late decay of NLOP is also important. When sterile neutrinos are heavier than the dark sector, NLOP is long-lived due to tiny mixing angle between sterile and light neutrinos. Constrains from free-streaming length, CMB, BBN and neutrino experiments are considered.

1 Introduction

For many years, the standard model (SM) of particle physics has been proved very successful. But it could not explain some phenomena, such as the origin of tiny neutrino mass and particle dark matter (DM). Searching for the connections between these two parts has become an interesting topic at present, where new physics beyond the SM is required [1,2,3,4,5].

The right-handed neutrinos N are introduced in the traditional Type-I seesaw mechanism [6, 7] to solve the tiny neutrino mass problem. Although a keV-scale right-handed neutrino could both explain the active neutrino mass and act as DM candidate [8,9,10,11], the parameter space is now tightly constrained by X-ray searches [12]. To avoid such constraints, one may introduce an exact $Z_2$ symmetry to make the lightest right-handed neutrino $N_1$ stable [13,14,15,16]. This $Z_2$ symmetry also forbids the direct Yukawa interaction $y{\overline{L}}{\tilde{\Phi }}N$, thus neutrino masses are not appear at tree-level. Provided an additional inert scalar doublet, the right-handed neutrinos can mediate the generation of tiny neutrino mass at one-loop level, which is known as the Scotogenic model [3].

Another pathway is assuming the right-handed neutrinos N as the messenger between the dark sector and standard model [17,18,19,20,21,22,23,24,25,26,27,28,29,30]. With sizable coupling between DM and N, sufficient annihilation rate of DM into NN pairs is viable. When this channel is dominant, the DM-nucleon scattering cross section is suppressed, hence satisfy the constraints from direct detection. Meanwhile, observable signature is still expected by indirect detection [31,32,33,34], although the large astrophysical uncertainties will impact the presented limits. On the other hand, the coupling between DM and N can be tiny, thereby producing DM via the freeze-in mechanism [35,36,37,38,39,40,41]. Leptogenesis in the frame work of right-handed neutrino portal DM is also discussed in Ref. [42,43,44,45,46,47,48].

To naturally obtain tiny neutrino mass, the right-handed neutrinos should be at the scale of $10^{14}$ GeV with ${\mathcal {O}}(1)$ Yukawa coupling [49, 50]. However, phenomenological studies usually favor the right-handed neutrinos below TeV scale. In this scenario, the Yukawa coupling y is at the order of $10^{-7}$. Therefore, the right-handed neutrinos may not be fully thermalized [51, 52], which affects the relic density of DM. TeV scale right-handed neutrinos with large Yukawa coupling y is possible in low-scale seesaw, e.g., inverse seesaw [53, 54] and neutrinophilic two Higgs doublet model ($\nu$THDM) [55].

In this paper, we propose the sterile neutrino portal DM in the $\nu$THDM. A neutrinophilic Higgs doublet $\Phi _\nu$ with lepton number $L_{\Phi _\nu }=-1$ is introduced. By assuming the sterile neutrinos N with vanishing lepton number $L_N=0$, they couple exclusively to $\Phi _{\nu }$ via the Yukawa interaction ${\overline{L}}{\tilde{\Phi }}_\nu N$, while interaction with standard model Higgs doublet $\Phi$ is forbidden. The soft term $\mu ^2 \Phi ^\dag \Phi _{\nu }$ then induces a naturally small vacuum expectation value of $\Phi _{\nu }$, resulting light sterile neutrinos with large Yukawa coupling. The dark sector consists of one Dirac fermion singlet $\chi$ and one scalar singlet $\phi$. Both are charged under a $Z_2$ symmetry, of which the lightest one serves as DM candidate. As a mediator, the sterile neutrinos couple with the dark sector through the Yukawa interaction $\lambda _{ds} {\bar{\chi }}\phi N$. Depending on the strength of $\lambda _{ds}$ and other relevant couplings, both WIMP and FIMP DM scenario are possible. A comprehensive analysis of the DM phenomenology is carried out in the following studies.

The structure of the paper is organized as follows. In Sect. 2, we briefly introduce the model. A detail study of the WIMP scenario in aspect of relic density, Higgs invisible decay, direct and indirect detection is performed in Sect. 3. Then we study the FIMP scenario in Sect. 4. Finally, we summarize our results in Sect. 5.

2 The model

Table 1 Relevant particle contents and corresponding charge assignments

This model extends the $\nu$THDM with a dark sector. The particle contents and corresponding charge assignments are listed in Table 1. Under the global $U(1)_L$ symmetry, sterile neutrinos N can not couple to standard Higgs doublet $\Phi$, but to the new one $\Phi _{\nu }$. The $Z_2$ symmetry is exact to stabilize DM. In this paper, both the fermion and scalar DM scenario are considered. The scalar potential under above symmetry is given by

$$\begin{aligned} V= & {} m_{\Phi }^2 \Phi ^\dag \Phi + m_{\Phi _\nu }^2 \Phi ^\dag _\nu \Phi _\nu +m_\phi ^{\prime 2} \phi ^\dag \phi \nonumber \\{} & {} +\frac{\lambda _1}{2} (\Phi ^\dag \Phi )^2+\frac{\lambda _2}{2} (\Phi ^\dag _\nu \Phi _\nu )^2 \nonumber \\{} & {} +\lambda _3 (\Phi ^\dag \Phi )(\Phi ^\dag _\nu \Phi _\nu )+\lambda _4(\Phi ^\dag \Phi _\nu )(\Phi ^\dag _\nu \Phi ) \nonumber \\{} & {} - (\mu ^2 \Phi ^\dag \Phi _\nu +\mathrm{h.c.}) \nonumber \\{} & {} +\frac{\lambda _5}{2} (\phi ^\dag \phi )^2 + \lambda _6 (\phi ^\dag \phi )(\Phi ^\dag \Phi )+\lambda _7 (\phi ^\dag \phi ) (\Phi _\nu ^\dag \Phi _\nu ), \end{aligned}$$

(1)

where the $\mu ^2$ term breaks the $U(1)_L$ symmetry explicitly but softly. Since this term is the only source of $U(1)_L$ breaking, it is naturally small and stable under radiative corrections [56, 57]. The small $\mu ^2$ term may originate from the spontaneous breaking of global $U(1)_L$ [58]. The electroweak symmetry is broken spontaneously by $\Phi$ as usual. With the condition $m_{\Phi }^2<0, m_{\Phi _\nu }^2>0, |\mu ^2|\ll m_{\Phi _\nu }^2$, one could be obtained hierarchical vacuum expectation values as

$$\begin{aligned} v\simeq \sqrt{\frac{-2 m_{\Phi }^2}{\lambda _1}}, v_\nu \simeq \frac{\mu ^2 v}{m_{\Phi _\nu }^2+(\lambda _3+\lambda _4)v^2/2}, \end{aligned}$$

(2)

where $v=246$ GeV, and $v_\nu \sim 10$ MeV with $\mu \sim 1$ GeV, $m_{\Phi _{\nu }}\sim 200$ GeV typically. Notably, the constraints from lepton flavor violation require $v_\nu \gtrsim 1$ MeV [59].

After the symmetry breaking, we have six physical Higgs bosons. They are the standard model Higgs boson h [60, 61], the charged Higgs $H^\pm$, the CP-even Higgs H, the CP-odd Higgs A, and the inert Higgs $\phi$. The mixing angles among the CP even neutral physical Higgs bosons and the Goldstone fields with charged scalar are at the order of ${\mathcal {O}}(v_\nu /v)$, which are suppressed by the small value of $v_\nu$ [59]. Ignoring the terms of ${\mathcal {O}}(v_\nu ^2)$ and ${\mathcal {O}}(\mu ^2)$, masses of physical scalars are [55]

$$\begin{aligned} m_h^2\simeq & {} \lambda _1 v^2,~ m_{H^\pm }^2\simeq m_{\Phi _\nu }^2\!+\frac{1}{2}\lambda _3v^2,\nonumber \\ m_A^2\simeq & {} m_H^2\simeq m_{H^\pm }^2\!+\frac{1}{2}\lambda _4v^2,\nonumber \\ m_\phi ^2\simeq & {} m_\phi ^{\prime 2}+\frac{1}{2}\lambda _6 v^2. \end{aligned}$$

(3)

Such simplified spectrum is equivalent to ignore all mixing angles among physical scalars. For simplicity, we adopt a degenerate mass spectrum of $\Phi _ \nu$, $m_{H^+}\!=\!m_{H}\!=\!m_A\!=\!m_{\Phi _\nu }$, which is allowed by various constraints [62].

Currently, the LEP experiment has excluded $m_{H^\pm }<80$ GeV [63]. Meanwhile, the limits at LHC heavily depends on the decay mode of $H^\pm$. If $H^\pm \rightarrow \ell ^\pm \nu$ is the decay mode when $m_N>m_{\Phi _\nu }$ and $v_\nu \lesssim 1$ MeV, we need $m_{H^\pm }\gtrsim 700$ GeV [64]. Fixing $v_\nu =10$ MeV, then $H^\pm \rightarrow tb/cb$ is the dominant decay channel when $m_N>m_{\Phi _\nu }$, while $H^\pm \rightarrow \ell ^\pm N$ becomes the dominant one when $m_N<m_{\Phi _\nu }$ [65]. The $H^\pm \rightarrow tb$ mode is only promising with large $v_\nu$ above 110 GeV, and no search in the $H^\pm \rightarrow \ell ^\pm N$ decay mode is performed. Anyway, $m_{\Phi _\nu }=200$ GeV with $v_\nu =10$ MeV is still allowed by current collider searches [66]. The $\lambda _7$ term generates a new decay mode $H\rightarrow \phi \phi$, which is invisible at colliders. Such decay mode might be tested via the process at LHC.

Fig. 1

The dominant annihilation channels for WIMP DM. Feynman diagrams a–c are for scalar DM, and Feynman diagram d is for fermion DM. Here, SM denotes all the particles in the standard model

Besides providing masses for light neutrinos via Yukawa coupling with $\Phi _\nu$ and lepton doublet L, the sterile neutrinos also contribute to the production of DM through coupling with dark particles $\phi$ and $\chi$. The the new Yukawa interaction and mass terms can be written as

$$\begin{aligned}{} & {} -{\mathcal {L}}_Y\supset y{\overline{L}} {\widetilde{\Phi }}_\nu N +\lambda _{ds} {\bar{\chi }}\phi N +\frac{1}{2}\overline{N ^c}m_{N } N \nonumber \\{} & {} \quad + m_\chi {\bar{\chi }} \chi + \mathrm{h.c.}, \end{aligned}$$

(4)

where ${\widetilde{\Phi }}_\nu =i\sigma _2 \Phi _\nu ^*$. The tiny neutrino mass is generated via Type-I seesaw like mechanism with $\Phi$ simply replaced by $\Phi _\nu$,

$$\begin{aligned} m_\nu = - \frac{v_\nu ^2}{2} y~ m_{N}^{-1} y^T. \end{aligned}$$

(5)

$m_\nu \sim 0.1$ eV can be obtained with $v_\nu \sim 10$ MeV, $y\sim 0.01$ and $m_N\sim 200$ GeV. With such large Yukawa coupling, the sterile neutrinos are fully thermalized through interaction with $\Phi _{\nu }$ and L. Therefore, the conventional calculation of relic density are not affected in our studies.

In this paper, we focus on the DM phenomenology. For sufficient large Yukawa coupling $\lambda _{ds}$ or other relevant couplings, the DM is a WIMP candidate produced via the freeze-out mechanism. In contrast, tiny $\lambda _{ds}$ and other relevant couplings lead to FIMP candidate produced via the freeze-in mechanism. These two scenarios are both considered in the following studies. The complete model file is written by FeynRules package [67], then micrOMEGAs [68] is used to calculate the relic density, DM-nucleon scattering and annihilation cross section.

3 WIMP dark matter

3.1 Relic density

When $m_\phi <m_\chi$, the scalar $\phi$ is treated as the DM candidate. In the case of WIMP DM, the most common way for scalar DM $\phi$ is annihilating to the SM final state via the standard Higgs portal coupling term $\lambda _6 (\phi ^\dag \phi )(\Phi ^\dag \Phi )$, which has been extensively studied in previous paper [69,70,71,72,73]. In our scenario, there are two extra ways for scalar DM annihilation. One is annihilating to neutrinophilic scalars $\Phi _{\nu }^\dag \Phi _{\nu }$ through the coupling $\lambda _7 (\phi ^\dag \phi ) (\Phi _\nu ^\dag \Phi _\nu )$ [74]. The other one is annihilating to sterile neutrinos NN via the Yukawa interaction $\lambda _{ds} {\bar{\chi }}\phi N$. The fermion DM is realized with $m_\chi <m_\phi$. It can only annihilate to sterile neutrinos NN with a t-channel mediator $\phi$. The dominant annihilation channels are shown in Fig. 1. For the process $\phi \phi \rightarrow hh$, there is also contribution from $\phi$ exchange in the t-channel. In addition, the scattering processes with tiny contributions are ignored, i.e., s-channel $\phi \phi \rightarrow h^* \rightarrow \Phi _\nu \Phi _{\nu }$, $\phi \phi \rightarrow \Phi _{\nu }^* \rightarrow \Phi _\nu \Phi _{\nu }$ due to very small $\lambda _{3,4}$ and $v_\nu$ respectively.

According to the nature of WIMP, DM is in thermal equilibrium with other particles in the early universe. Then it starts to freeze-out and decouple from thermal bath as the temperature decreases. The evolution of abundance Y is obtained by solving the Boltzmann equation

$$\begin{aligned} \frac{dY}{dz} = - \frac{k}{z^2}\left\langle \sigma v \right\rangle (Y^2-Y_{\text {eq}}^{2}), \end{aligned}$$

(6)

where we use the definition $z\equiv m_{\text {DM}}/T$. Here, $m_{\text {DM}}$ is the mass of DM and T is the temperature. The parameter k is denoted as $k=\sqrt{\pi g_{\star }/45}m_{\text {DM}} M_{Pl}$, where $g_{\star }$ is the effective number of degrees of freedom of the relativistic species and $M_{Pl}=1.2 \times 10^{19}$ GeV is the Planck mass. $\langle \sigma v \rangle _{\phi \phi \rightarrow {\textrm{SMSM}}, \Phi _{\nu }\Phi _{\nu }, NN}$ for WIMP DM $\phi$ and $\langle \sigma v \rangle _{\chi \chi \rightarrow NN}$ for WIMP DM $\chi$ represent thermally averaged cross sections multiply by velocity.

The evolution of DM abundance for various benchmark scenarios are shown in Fig. 2, which correspond to the individual contribution of annihilation channels in Fig. 1. During the calculation, we have fixed $m_N=250$ GeV, $m_\text {DM}=500$ GeV, and $|m_\phi -m_\chi |=500$ GeV. When the scalar DM $\phi$ annihilates via the Higgs or neutrinophilic scalar portal channels, correct relic density is predicted with couplings $\lambda _{6,7}\sim 0.1$. When the sterile neutrino portal becomes the dominant channel, the coupling $\lambda _{ds}$ should be slightly larger than 0.1 to realize correct relic density. Meanwhile, $\lambda _{ds}\sim 1$ is required for fermion DM $\chi$ with correct relic density.

Fig. 2

The evolution of DM abundance for the four annihilation channels. Subfigures a–c are for scalar DM, and subfigure d is for fermion DM. The orange horizontal lines predict the Planck observed relic density $\Omega _\text {DM}h^2=0.12$ [75] for $m_\text {DM}=500$ GeV. The black dashed lines describe the evolution of $Y_\text {eq}$

Fig. 3

Allowed parameter space with correct relic density. Relative contributions of various annihilation channels to the total abundance of the scalar DM $\phi$ are illustrated in subfigure a–c. Subfigure d is for fermion DM $\chi$, which is colored by the mass splitting $\Delta _{\phi \chi }=m_\phi -m_\chi$

To obtain the parameter space with correct relic density in $3\sigma$ range of the Planck observed result [75], i.e., $\Omega _\text {DM} h^2\in [0.117,0.123]$, we perform a random scan on relevant parameters in the following range based on previous benchmark scenarios

$$\begin{aligned} \begin{aligned} m_{\phi ,\chi ,N}\in [10,1000]~\textrm{GeV}, \lambda _{6,7,ds}\in [0.0001,1]. \end{aligned} \end{aligned}$$

(7)

Meanwhile, we fix the masses of neutrinophilic scalars as $m_{\Phi _{\nu }}=200$ GeV for simplicity. The results are shown in Fig. 3, where the relative contribution of annihilation channels of the scalar DM are defined as

$$\begin{aligned} R_{\phi h}= & {} \frac{\Omega _{\textrm{DM}}h^2}{\Omega _{\phi \textrm{SM}}h^2}, R_{\phi \Phi _\nu }=\frac{\Omega _\textrm{DM}h^2}{\Omega _{\phi \Phi _{\nu }}h^2}, R_{\phi N}=\frac{\Omega _\textrm{DM}h^2}{\Omega _{\phi N}h^2}. \end{aligned}$$

(8)

Here, $\Omega _{\phi \textrm{SM}}h^2$, $\Omega _{\phi \Phi _{\nu }}h^2$ and $\Omega _{\phi N}h^2$ denote the relic density for the SM Higgs portal $\phi \phi \rightarrow \textrm{SM}{\overline{\textrm{SM}}}$, neutrinophilic scalar portal $\phi \phi \rightarrow \Phi _\nu \Phi _{\nu }^\dag$ and sterile neutrino portal $\phi \phi \rightarrow NN$ channel respectively. As for the fermion DM, since the sterile neutrino portal $\chi {\bar{\chi }}\rightarrow NN$ is the only possible annihilation channel, we instead color the samples by the mass splitting $\Delta _{\phi \chi }=m_\phi -m_\chi$.

Fig. 4

The predicted and constraints from invisible Higgs decay. The upper(lower) two panels are for scalar(fermion) DM. The purple, blue, and red points express the parameter spaces excluded by ATLAS Higgs invisible result, direct detection experiment XENON1T [77], and indirect detection experiments [33], respectively. The cyan points satisfy all current constraints but are within the reach of the future LZ experiment [78, 79]. The green points satisfy all constraints mentioned. For fermion DM, the direct detection experiments XENON1T or even LZ do not exclude any points for the parameter space scanned in Eq. 7

For the scalar DM, green samples indicate that the corresponding annihilation channel has a relative large contribution to the total abundance. In contrast, contributions of the red and blue samples are negligible. It is clear in Fig. 3a that there is a sharp dip around $m_\phi \sim m_h/2$ in the $\lambda _{6}-m_\phi$ plane due to the resonance production of SM Higgs h in the s-channel. Typically, the $\phi \phi \rightarrow \Phi _\nu \Phi _{\nu }^\dag$ and the $\phi \phi \rightarrow NN$ channel become the dominant one with $\lambda _7\gtrsim 0.01$ and $\lambda _{ds}\gtrsim 0.1$ respectively. When $m_\phi$ is close to $m_{\Phi _\nu }$, the cross section is suppressed by the final state phase space, thus a relative larger coupling $\lambda _7$ is required. As for the fermion DM, $\lambda _{ds}\gtrsim 0.1$ should be satisfied to generate correct abundance. The lower bound on $\lambda _{ds}$ becomes larger when $m_\chi$ is heavier. It is also obvious that a smaller mass splitting $\Delta _{\phi \chi }$ is accompanied by a smaller $\lambda _{ds}$.

3.2 Higgs invisible decay

Now, we consider the possible constraints on the above parameter space for correct abundance. When the SM Higgs decays into DM, the invisible Higgs decay rate will be enhanced. Currently, the invisible branching ratio is constrained by the ATLAS experiment [76], namely, $\textrm{BR}_\textrm{inv}=\Gamma _\textrm{inv}/\left( \Gamma _\textrm{inv}+\Gamma _\textrm{SM}\right) <0.11$, where the SM Higgs width is $\Gamma _\textrm{SM} \approx 4$ MeV. In the case of scalar DM $\phi$, the invisible Higgs decay $h\rightarrow \phi \phi$ via the coupling $\lambda _6 (\phi ^\dag \phi )(\Phi ^\dag \Phi )$ is kinematically allowed when $m_\phi <m_h/2$. The resulting invisible decay width can be expressed as

$$\begin{aligned} \Gamma _{h\rightarrow {\phi \phi }}=\frac{\lambda _6^{2}v^{2}}{8\pi {m_h}}\sqrt{1-\frac{4m_\phi ^2}{m_h^2}}. \end{aligned}$$

(9)

In the case of fermion DM $\chi$, although $\chi$ does not couples to h at tree level, the dimension-5 operator $({\bar{\chi }}\chi )({\Phi ^\dagger }\Phi )$ can generate an effective coupling between $\chi$ and h at one-loop level. The partial decay width of $h\rightarrow {\chi {\bar{\chi }}}$ with the condition of $m_\chi <m_h/2$ can be written as

$$\begin{aligned} \Gamma _{h\rightarrow {\chi {\bar{\chi }}}}=\frac{\lambda _{h\chi }^{2}{m_h}}{8\pi }{\left( 1-\frac{4m_\chi ^2}{m_h^2}\right) ^{3/2}}, \end{aligned}$$

(10)

with the effective coupling $\lambda _{h\chi }$ expressed as [18]

$$\begin{aligned} \lambda _{h\chi }=\frac{{\lambda _6}\lambda _{ds}^{2}}{16{\pi }^2}\left( \frac{m_N}{m_\phi ^2-m_N^2}+\frac{2m_N^3\log \frac{m_N}{m_\phi }}{(m_\phi ^2-m_N^2)^2}\right) . \end{aligned}$$

(11)

The predicted branching ratio of Higgs invisible decay and constraints on the parameter spaces under correct abundance are shown in Fig. 4. The shape of right limiting curve in Fig. 4a corresponds to the upper limit on $\lambda _6$ as shown in Fig. 3. The disallowed parameter space of scalar DM $\phi$ satisfy $\lambda _6\gtrsim 0.01$ with the constraint of ATLAS result. However, current constraint from direct detection experiment XENON1T is approximately one order of magnitude tighter than Higgs invisible decay. Moreover, the indirect detection experiments have already exclude the region $m_\phi \lesssim 60$ GeV in this sterile neutrino portal model. The current allowed samples predict $\textrm{BR}_\textrm{inv}\lesssim 10^{-2}$ in the $h\rightarrow \phi \phi$ decay mode. If no signal is observed in the future LZ experiments, $\textrm{BR}_\textrm{inv}\lesssim 3\times 10^{-4}$ in this mode is expected, which will be smaller than the SM value $\textrm{BR}_\textrm{inv}\sim 10^{-3}$ in the $h\rightarrow ZZ^*$ with $Z\rightarrow \nu {\bar{\nu }}$ mode. For fermion DM $\chi$, the Higgs invisible decay mode $h\rightarrow \chi {\bar{\chi }}$ is heavily suppressed by the loop-factor. In the scanned parameter space, it has a range of $\textrm{BR}_\textrm{inv}\lesssim 10^{-9}$, which is at least eight orders of magnitude smaller than current measured bound. Hence, no sample is excluded by Higgs invisible decay for fermion DM.

3.3 Direct detection

Nowadays, the DM direct detection experiments have already set tight constraints on the DM-nucleon scattering cross sections. Results from the XENON1T [77] and the future LZ [78, 79] experiments are considered in this paper. The spin-independent scattering cross section of scalar DM $\phi$ is mediated by SM Higgs h, which originates from the Higgs portal coupling $\lambda _6(\phi ^\dag \phi )(\Phi ^\dag \Phi )$. The result of fermion DM $\chi$ is also mediated by h with the one-loop generated effective coupling $\lambda _{h\chi } ({\bar{\chi }}\chi )(\Phi ^\dag \Phi )$, where $\lambda _{h\chi }$ is given in Eq. 11. The corresponding scattering cross sections are then calculated as [73]

$$\begin{aligned} \sigma _\text {SI}^{\phi }= & {} \frac{\lambda _6^2 f_n^2}{\pi m_h^4} \frac{m_n^4}{(m_\phi +m_n)^2}, \end{aligned}$$

(12)

$$\begin{aligned} \sigma _\text {SI}^{\chi }= & {} \frac{\lambda _{h\chi }^2 f_n^2}{\pi m_h^4} \frac{m_n^4 m_\chi ^2}{(m_\chi +m_n)^2}, \end{aligned}$$

(13)

where $f_n\approx 0.3$ is the nucleon matrix element and $m_n\simeq 0.939$ GeV is the averaged nucleon mass.

Fig. 5

Constraints from the direct detection experiments. Labels are the same as Fig. 4

The spin-independent scattering cross sections as a function of DM mass are shown in Fig. 5. Since the Higgs portal coupling $\lambda _{6}$ plays a vital important role in direct detection, we also show the parameter space in the $\lambda _6-m_\text {DM}$ plane. Currently, the XENON1T experiment has excluded the samples with a cross section larger than $4\times 10^{-11}$ pb for $m_\text {DM}\sim 30$ GeV. The future LZ experiment can push this limit down to about $10^{-12}$ pb. Although LZ could probe the scalar DM mass region $m_\textrm{DM}\lesssim$ 60 GeV, such region has already been exclude by current indirect experiments. For the scalar DM $\phi$, the parameter space with $\lambda _6\gtrsim 0.001$ has been excluded by XENON1T. Combined with the relative contribution in Fig. 3, it is clear that the SM Higgs portal channel could be the dominant one only at the narrow resonance region $m_\phi \sim m_h/2$ or the high mass region $m_\phi \sim 1000$ GeV at present. However, this high mass region could be further excluded when there is no signal at the future LZ experiment. This indicates that for $m_\phi \lesssim m_{\Phi _\nu }=200$ GeV, the sterile neutrino portal channel $\phi \phi \rightarrow NN$ is usually the dominant one when we take into account the present XENON1T constraint. Meanwhile, $\phi \phi \rightarrow NN,\Phi ^\dag _\nu \Phi _\nu$ are the dominant channels for $m_{\phi }\gtrsim m_{\Phi _\nu }$ under current direct detection limit. For the fermion DM $\chi$, the effective Higgs portal coupling $\lambda _{h\chi }$ is suppressed by the loop factor, leading to a natural suppression of the scattering cross section. The predicted value $\sigma _\text {SI}\lesssim 10^{-13}$ pb is beyond the reach of XENON1T and LZ experiments, therefore no sample is excluded by direct detection for fermion DM.

3.4 Indirect detection

Fig. 6

Constraints from the indirect detection experiments. In the left panel, the red and yellow curves represent the bounds of antiproton-to-proton flux ratio from AMS-02 [80] and gamma-rays in the Milky Way dSphs from Fermi-LAT [81] with $\langle \sigma v\rangle =2.2\times 10^{-26}~ \textrm{cm}^3/s$, respectively. In the right panel, the black and gray dotted lines express the upper limits obtained using the $\gamma$ ray data from Fermi-LAT and H.E.S.S. for the muon sterile neutrino final state with $m_N=10$ GeV [32]. The orange dotted curve is upper bound from prospective CTA [83] with $W^+W^-$ final state. Other labels are the same as Fig. 4

One distinct property of the sterile neutrino portal DM is that the present day annihilation of DM into NN pair will lead to observable signature at indirect detection experiments. In this work, we consider the indirect detection constraints from the antiproton observations of AMS-02 [80] and gamma-rays observations of Fermi-LAT [81]. These experiments are able to probe the DM mass below about 100 GeV [20]. Although there are three possible annihilation channels for scalar DM $\phi$ in our model, the sterile neutrino portal $\phi \phi \rightarrow NN$ is the dominant one for $m_\phi \lesssim 100$ GeV under the XENON1T constraint. Currently, the indirect limits we considered in the $\phi \phi \rightarrow NN$ channel is $m_\phi \lesssim 60$ GeV. The neutrinophilic scalars should be heavier than 80 GeV. So varying $m_{\Phi _\nu }\in [80,200]$ GeV does not affect current indirect limits.

The exclusion limits of indirect detection experiments are shown in Fig. 6. For both the scalar and fermion DM, the thermal average annihilation cross section of NN channel is about $\langle \sigma v\rangle \simeq 2.2\times 10^{-26}\,{ \textrm{cm}^3/\textrm{s}}$ for most samples, therefore we simply use the exclusion region in the $m_\textrm{DM}-m_N$ plane obtained by Ref. [33]. Currently, Fermi-LAT experiment is more sensitive for light $m_N$ with $m_\text {DM}\lesssim 60$ GeV been excluded. Meanwhile AMS-02 is more sensitive for heavy $m_N$ with $m_\text {DM}\lesssim 80$ GeV been excluded. In this way, the indirect detection experiments have set a more stringent constrain than the direct detection experiments in the low mass region of sterile neutrino portal model. Notably, the observed Galactic Center gamma ray excess [82] might be explained by the parameter set $\langle \sigma v\rangle =3.08\times 10^{-26}~\text {cm}^3/s, m_\text {DM}=41.3~\textrm{GeV},m_N=22.6$ GeV. However, such result is conflict with the indirect limits. For the high mass region, current H.E.S.S. limit does not exclude any samples. In the future, the CTA experiment [83] is able to probe the region above 200 GeV. Moreover, it is worth noting that the large astrophysical uncertainties have not been taken into account at all, which may modify substantially the quoted limits, in particular those from the antiproton flux observed by AMS-02.

4 FIMP dark matter

In the case of FIMP DM, we consider that the neutrinophilic particles $\Phi _\nu$ and sterile neutrinos N are in thermal equilibrium invariably. Meanwhile, we assume that the DM interacts with tiny couplings and never reaches the equilibrium [84, 85]. With only renormalisable operators, the DM is produced via InfraRed (IR) freeze-in [86]. In this paper, we consider the reheating temperature $T_R\gg {\mathcal {O}}$(TeV), thus the FIMP DM abundance is set by IR physics and is independent of the reheating temperature.

4.1 Relic density

Fig. 7

The dominant Feynman diagrams for scalar FIMP DM $\phi$

If the scalar $\phi$ is FIMP DM, it can be generated from the annihilation of SM particles via Higgs boson h related to $\lambda _6(\phi ^\dag \phi )(\Phi ^\dag \Phi )$, neutrinophilic scalars connected with $\lambda _7 (\phi ^\dag \phi ) (\Phi _\nu ^\dag \Phi _\nu )$, and sterile neutrinos associated with $\lambda _{ds} {\bar{\chi }}\phi N$. Due to the small value of $v_\nu$, the scattering process $\Phi _\nu ^\dag \Phi _\nu \rightarrow \phi \phi$ has the dominant contribution within neutrinophilic portals. The decay of $N\rightarrow \phi \chi$ followed by $\chi \rightarrow \phi \nu$ is possible when $m_N>m_\phi +m_\chi$, where the $\chi \rightarrow \phi \nu$ decay is induced by the mixing between N and $\nu$. Otherwise, for a heavier $\chi$, $\phi$ and $\chi$ are associated produced by the annihilation of $\Phi _\nu$ and L, which is mediated by the off-shell N. The dominant Feynman diagrams are shown in Fig. 7. For the process $hh\rightarrow \phi \phi$, there is also t-channel contribution mediated by $\phi$.

In the case of fermion FIMP DM $\chi$, it is mainly produced by the decay of N or $\phi$. If $m_N>m_\phi +m_\chi$, then $\chi$ can be produced by the decay $N\rightarrow \chi \phi$ with further decay $\phi \rightarrow \chi \nu$. For the opposite case $m_\phi >m_N$, the decay $\phi \rightarrow \chi N$ or $\phi \rightarrow \chi \nu$ generates the relic abundance of $\chi$. The annihilation of $\Phi _\nu$ and L mediated by the off-shell N can also generate the fermion DM $\chi$ for heavier $\chi$. The dominant Feynman diagrams are shown in Fig. 8. The feeble nature of $\chi$ only requires tiny $\lambda _{ds}$, and other couplings of the dark scalar $\phi$, i.e., $\lambda _{6,7}$, are not limited. One scenario is that $\lambda _{6,7}$ is large enough to keep $\phi$ in thermal equilibrium, and the other scenario is that $\phi$ is also feeble with tiny $\lambda _{6,7}$.

Fig. 8

The dominant Feynman diagrams for fermion FIMP DM $\chi$

The evolution of abundances of the dark scalar $\phi$ and dark fermion $\chi$ are determined by the following Boltzmann equations

$$\begin{aligned} \frac{dY_\phi }{dz}= & {} \frac{k}{z^2} \left\langle \sigma v \right\rangle _{\text {SM}\text {SM}\rightarrow \phi \phi }\left( (Y_\text {SM}^\textrm{eq})^2\!-\!\left( \frac{Y_\text {SM}^\textrm{eq}}{Y_{\phi }^\textrm{eq}}\right) ^2\! Y_{\phi }^{2}\right) \nonumber \\{} & {} +\frac{k}{z^2} \left\langle \sigma v \right\rangle _{\Phi _{\nu }\Phi _{\nu }\rightarrow \phi \phi }\left( (Y_{\Phi _\nu }^\textrm{eq})^2\!-\!\left( \frac{Y_{\Phi _\nu }^\textrm{eq}}{Y_{\phi }^\textrm{eq}}\right) ^2 \!Y_{\phi }^{2}\right) \nonumber \\{} & {} +k^{\star }z{\tilde{\Gamma }}_{N\rightarrow \phi \chi }\left( Y_{N}^\textrm{eq}-\frac{Y_{N}^\textrm{eq}}{Y_{\phi }^\textrm{eq}Y_{\chi }^\textrm{eq}}Y_\phi Y_\chi \right) \nonumber \\{} & {} +\frac{k}{z^2} \left\langle \sigma v \right\rangle _{\Phi _\nu L\rightarrow \phi \chi }\left( Y_{\Phi _\nu }^{\textrm{eq}}Y_L^{\textrm{eq}}-\left( \frac{Y_{\Phi _\nu }^\textrm{eq}Y_L^\textrm{eq}}{Y_{\phi }^\textrm{eq}Y_{\chi }^\textrm{eq}} \right) Y_{\phi }Y_{\chi }\right) \nonumber \\{} & {} +k^{\star }z{\tilde{\Gamma }}_{\chi \rightarrow \phi N,\phi \nu }\left( Y_{\chi }-\frac{Y_{\chi }^\textrm{eq}}{Y_{\phi }^\textrm{eq}}Y_\phi \right) \nonumber \\{} & {} - k^{\star }z {\tilde{\Gamma }}_{\phi \rightarrow \chi N,\chi \nu }\left( Y_{\phi }-\frac{Y_{\phi }^\textrm{eq}}{Y_{\chi }^\textrm{eq}}Y_\chi \right) , \end{aligned}$$

(14)

$$\begin{aligned} \frac{dY_\chi }{dz}= & {} k^{\star }z{\tilde{\Gamma }}_{N\rightarrow \phi \chi }\left( Y_{N}^\textrm{eq}-\frac{Y_{N}^\textrm{eq}}{Y_{\phi }^\textrm{eq}Y_{\chi }^\textrm{eq}}Y_\phi Y_\chi \right) \nonumber \\{} & {} +k^{\star }z {\tilde{\Gamma }}_{\phi \rightarrow \chi N,\chi \nu }\left( Y_{\phi }-\frac{Y_{\phi }^\textrm{eq}}{Y_{\chi }^\textrm{eq}}Y_\chi \right) \nonumber \\{} & {} + \frac{k}{z^2} \left\langle \sigma v \right\rangle _{\Phi _\nu L\rightarrow \phi \chi }\left( Y_{\Phi _\nu }^{\textrm{eq}}Y_L^{\textrm{eq}}-\left( \frac{Y_{\Phi _\nu }^\textrm{eq}Y_L^\textrm{eq}}{Y_{\phi }^\textrm{eq}Y_{\chi }^\textrm{eq}} \right) Y_{\phi }Y_{\chi }\right) \nonumber \\{} & {} -k^{\star }z{\tilde{\Gamma }}_{\chi \rightarrow \phi N,\phi \nu }\left( Y_{\chi }-\frac{Y_{\chi }^\textrm{eq}}{Y_{\phi }^\textrm{eq}}Y_\phi \right) , \end{aligned}$$

(15)

and the parameter $k^{\star }$ satisfies $k^{*}=\sqrt{45/4\pi ^{3}g_{\star }}M_{Pl}/m_\text {DM}^2$. Here, we need to emphasize that some $2 \rightarrow 2$ scattering processes related to the generation of dark particles have been ignored due to the suppressed cross sections, i.e., t-channels $NN \rightarrow \phi \phi ,\chi \chi$, s-channels $NL \rightarrow \Phi _{\nu }^* \rightarrow \phi \phi$, and the conversion processes $\phi \phi \longleftrightarrow \chi \chi$. We use micrOMEGAs [68] to numerically calculate the thermal average cross sections $\langle \sigma v \rangle$. The thermal decay width ${\tilde{\Gamma }}_X$ is defined as $\Gamma _X {\mathcal {K}}_1/{\mathcal {K}}_2$ with ${\mathcal {K}}_{1,2}$ being the first and second modified Bessel Function of the second kind. Corresponding decay widths are given by

$$\begin{aligned} \Gamma _{{N\rightarrow }{\phi \chi }}= & {} \frac{\lambda _{ds}^{2}}{16\pi {m_N}}{\left( \frac{(m_N+m_\chi )^2-m_\phi ^2}{m_N^2}\right) }\nonumber \\{} & {} \times {\lambda ^{1/2}({m_N}^2,{m_\phi }^2,{m_\chi }^2)}, \nonumber \\ \Gamma _{\chi \rightarrow \phi N}= & {} \frac{\lambda _{ds}^{2}}{16\pi {m_\chi }}{\left( \frac{(m_N+m_\chi )^2-m_\phi ^2}{m_\chi ^2}\right) }\nonumber \\{} & {} \times {\lambda ^{1/2}({m_N}^2,{m_\phi }^2,{m_\chi }^2)},\nonumber \\ \Gamma _{\chi \rightarrow \phi \nu }= & {} \frac{\lambda _{ds}^{2}\sin ^2\theta \, m_\chi }{16\pi }{\left( \frac{m_\chi ^2-m_\phi ^2}{m_\chi ^2}\right) ^2},\nonumber \\ \Gamma _{\phi \rightarrow \chi N}= & {} \frac{\lambda _{ds}^{2}}{8\pi {m_\phi }}{\left( \frac{m_\phi ^2-(m_N+m_\chi )^2}{m_\phi ^2}\right) }\nonumber \\{} & {} \times {\lambda ^{1/2}({m_N}^2,{m_\phi }^2,{m_\chi }^2)},\nonumber \\ \Gamma _{\phi \rightarrow \chi \nu }= & {} \frac{\lambda _{ds}^{2}\sin ^2\theta \, m_\phi }{8\pi }{\left( \frac{m_\phi ^2-m_\chi ^2}{m_\phi ^2}\right) ^2}, \end{aligned}$$

(16)

The kinematic function $\lambda (a,b,c)$ is

$$\begin{aligned} \lambda (a,b,c)=a^2+b^2+c^2-2ab-2ac-2bc. \end{aligned}$$

(17)

Here the mixing angle $\theta$ can be simply expressed as $\theta =y v_\nu /\sqrt{2}m_N$. Typically, mixing angle $\theta =1.4\times 10^{-7}$ is obtained with $y=0.01$, $v_\nu =0.01$ GeV and $m_N=500$ GeV.

Fig. 9

The evolution of abundance for scalar FIMP $\phi$. The orange horizontal lines correspond to the Planck observed abundance for $m_\phi =0.1$ GeV. The other dashed lines in subfigure c and d show the evolution of $Y_\chi$

The evolutionary phenomena of scalar FIMP DM is shown in Fig. 9, where we have assume $m_{\phi }=0.1$ GeV. Different from the WIMP scenario, a larger coupling $\lambda _{6,7,ds}$ leads to a larger abundance $Y_\phi$. In panel (a), $\phi$ is mainly generated through the decay $h\rightarrow \phi \phi$. And correct relic density is obtained with $\lambda _{6}\sim {\mathcal {O}}(10^{-11})$. For the $\Phi _\nu \Phi _\nu \rightarrow \phi \phi$ channel, $\lambda _7\sim {\mathcal {O}}(10^{-9})$ is required to produce $\Omega _\phi h^2\simeq 0.12$, which is about two orders of magnitude larger than the coupling $\lambda _6$ for the $h\rightarrow \phi \phi$ channel. In Fig. 9c, both $Y_\phi$ and $Y_\chi$ from the $N\rightarrow \phi \chi$ channel are shown, where we assume $m_N=500$ GeV and $m_\chi =10$ GeV. At the very beginning, same amount of $Y_\phi$ and $Y_\chi$ are generated via the $N\rightarrow \phi \chi$ decay. Then the late time decay $\chi \rightarrow \phi \nu$ translates $Y_\chi$ into $Y_\phi$. This late decay rate is heavily suppressed by the sterile and active neutrino mixing, which might be tested at neutrino experiments [38]. In this scenario, the observed abundance is generated with $\lambda _{ds}\sim {\mathcal {O}}(10^{-10})$. In panel (d), we show the scenario with $m_\chi =150$ GeV, $m_N=100$ GeV and $y=0.01$. Comparing with the pair production channel $NN\rightarrow \phi \phi$, the new associated production channel $\Phi _\nu L\rightarrow N^*\rightarrow \phi \chi$ followed by $\chi \rightarrow \phi N$ becomes the dominant one. Correct relic abundance is realized with $\lambda _{ds}\sim {\mathcal {O}}(10^{-8})$, which is about two orders of magnitudes larger than the scenario of on-shell decay $N\rightarrow \phi \chi$. Different from the scenario in panel (c), here the $\chi \rightarrow \phi N$ decay is not suppressed by the small mixing angle $\theta$. Therefore, after $\chi$ is associated produced, it quickly decays into $\phi N$.

Fig. 10

The evolution of abundance for fermion FIMP $\chi$. The orange horizontal lines correspond to the Planck observed abundance for $m_\chi =0.1$ GeV. The other dashed lines are the evolution of $Y_\phi$. The black dashed line represent the evolution of $Y_\phi$ in thermal equilibrium state

The evolution of $Y_\chi$ and $Y_\phi$ for fermion FIMP scenario are shown in Fig. 10, where we also fix $m_\chi =0.1$ GeV. Panels (a) and (b) correspond to FIMP DM $\chi$ produced by direct decay of $N\rightarrow \phi \chi$ with cascade decay $\phi \rightarrow \chi \nu$, which is shown in Fig. 8a. During the calculation, we have fixed $m_\chi =0.1$ GeV, $m_N=500$ GeV and $m_\phi =10$ GeV. With vanishing scalar coupling $\lambda _{6,7}=0$, both $\phi$ and $\chi$ are dominant from the decay $N\rightarrow \phi \chi$. And the transition $Y_\phi \rightarrow Y_\chi$ happens via the late decay $\phi \rightarrow \chi \nu$. Correct relic density is obtained with $\lambda _{ds}\sim {\mathcal {O}}(10^{-10})$ when $N\rightarrow \phi \chi$ contributes solely. However, the other scalar interaction $\lambda _{6,7}$-terms of the dark scalar $\phi$ have great impact on the evolution of $Y_\chi$. First, we assume that $\phi$ is also feeble interacting. The result is shown in panel (b) of Fig. 10. Since the direct decay $h\rightarrow \phi \phi$ is kinematically allowed for $m_\phi =10$ GeV, the production of $\phi$ is subdominant by the decay $N\rightarrow \phi \chi$ for $\lambda _6=\lambda _7=\lambda _{ds}$. So when $\lambda _6=\lambda _7\gtrsim \lambda _{ds}$, the final abundance $Y_\chi$ is actually controlled by the dark scalar produced from the scalar portal channels. In another case of $m_\phi >m_N$, the generation of DM $\chi$ is mainly determined by the decay of $\phi$. The coupling $\lambda _{ds}$ only determines the decay rate of $\phi$, and has no effect on the final DM abundance. Evolution of $Y_\phi$ is similar with Fig. 9a, b, but finally $Y_\phi$ is translated into $Y_\chi$ via decay $\phi \rightarrow \chi N, \chi \nu$.

Fig. 11

Allowed parameter space with correct relic density for scalar FIMP DM $\phi$. Panel a–c correspond to the scenario $m_N>m_\chi$, and panel d corresponds to $m_N<m_\chi$

Next, let’s consider that the quartic coupling $\lambda _{6,7}$ are relative large to keep $\phi$ in thermal equilibrium. The results are shown in panel (c) and (d) of Fig. 10 with $m_\chi =0.1$ GeV, $m_\phi =500$ GeV and $m_N=100$ GeV, where the abundance of $\phi$ is controlled by the annihilation process $\phi \phi \rightarrow \textrm{SM}\, \textrm{SM},\Phi _\nu \Phi _\nu$ and $\phi \rightarrow \chi N$ is the dominant production mode of fermion DM. For $\lambda _6=\lambda _7=0.01$, the abundance of $\phi$ after freeze-out is smaller than the abundance of $\chi$ from freeze-in with $\lambda _{ds}>10^{-11}$. By decreasing the quartic coupling $\lambda _{6,7}$, the abundance of $\phi$ will increase. The decay of $\phi$ after freeze-out is the dominant contribution when $\lambda _6=\lambda _7\lesssim 0.01$ and $\lambda _{ds}=10^{-11}$. As for the case of $m_N>m_\phi$, evolution of $Y_\phi$ is similar with Fig. 10c, d, but the lifetime of $\phi$ is much longer due to the tiny late decay rate of $\phi \rightarrow \chi \nu$.

4.2 Scanning results

Corresponding to the above benchmark situations, we perform a random scan to obtain the parameter space of FIMP DM within $3\sigma$ range of the observed abundance. We first consider the scalar DM $\phi$. Depending on the mass relation between $m_N$ and $m_\chi$, the sterile neutrino portal channel requires quite different values of $\lambda _{ds}$. So we divide the scan into two parts. For the scenario with $m_N>m_\chi$, we scan the following parameter space:

$$\begin{aligned} \begin{aligned}&m_\phi \in [0.01,1]~\textrm{GeV}, m_\chi \in [1,100]~\textrm{GeV},\\&\lambda _{6,7,ds}\in [10^{-12}, 10^{-8}]. \end{aligned} \end{aligned}$$

(18)

During the scan, we fix $m_N=500$ GeV for simplicity. The results are shown in Fig. 11a–c, where the relative contribution of individual channels are defined as

$$\begin{aligned} R_{h\phi }= & {} \frac{\Omega _{h\phi }h^2}{\Omega _\textrm{DM}h^2}, R_{\Phi _{\nu }\phi }=\frac{\Omega _{\Phi _{\nu }\phi }h^2}{\Omega _\textrm{DM}h^2}, \nonumber \\ R_{N\phi }= & {} \frac{\Omega _{N\phi }h^2}{\Omega _\textrm{DM}h^2}. \end{aligned}$$

(19)

Hence $R_{h\phi }$, $R_{\Phi _{\nu }\phi }$ and $R_{N\phi }$ represent the contribution of $h\rightarrow \phi \phi$, $\Phi _{\nu }\Phi _{\nu }\rightarrow \phi \phi$ and $N\rightarrow \phi \chi$ channel as shown in Fig. 7a–c, respectively. It is clear that the DM mass is inversely proportional to the coupling coefficient for the same contribution. The $h\rightarrow \phi \phi$, $\Phi _{\nu }\Phi _{\nu }\rightarrow \phi \phi$ and $N\rightarrow \phi \chi$ channel become the dominant one when the corresponding coefficients individually satisfy $\lambda _6\in [6.5 \times 10^{-12},6.8 \times 10^{-11}]$, $\lambda _7\in [1.3 \times 10^{-11},1.3 \times 10^{-9}]$, $\lambda _{ds}\in [2.0 \times 10^{-11},2.0 \times 10^{-10}]$ with $m_\phi$ decreasing from 1 GeV to 0.01 GeV.

For the scenario with $m_N<m_\chi$, the dominant contribution of annihilation channel $\Phi _\nu L\rightarrow N^*\rightarrow \phi \chi$ requires that the coupling $\lambda _{ds}$ is about two to three orders of magnitudes larger than the $m_N>m_\chi$ scenario, so we scan the parameter space

$$\begin{aligned} \begin{aligned}&m_\phi \in [0.01,1]~\textrm{GeV}, m_\chi \in [100,200]~\textrm{GeV},\\&\lambda _{6,7}\in [10^{-12}, 10^{-8}], \lambda _{ds}\in [10^{-9}, 10^{-5}], \end{aligned} \end{aligned}$$

(20)

where $m_N=100$ GeV and $y=0.01$ are fixed for illustration. $m_\chi$ is not greater than $m_{\Phi _\nu }$ to make the scattering process occur naturally. In addition, we take $m_\phi +m_N<m_\chi$ to ensure the decay channel $\chi \rightarrow \phi N$ is kinematically allowed during scanning. Notably, the relative contributions of $h\rightarrow \phi \phi$ and $\Phi _\nu \Phi _\nu \rightarrow \phi \phi$ channel are the same as previous scenario with $m_N>m_\chi$, so we only need to show the result of $\Phi _\nu L\rightarrow \phi \chi$ channel in Fig. 11d. We use the notation $R_{\Phi _{\nu }L}=\Omega _{\Phi _{\nu }L\rightarrow \phi \chi }h^2/\Omega _\textrm{DM}h^2$ to express the relative contribution of this off-shell channel. The coupling $\lambda _{ds}\in [5.3\times 10^{-9},5.3\times 10^{-8}]$ will lead this channel to be the dominant one.

Fig. 12

Allowed parameter space with correct relic density for fermion FIMP DM $\chi$. Panel a and b correspond to the FIMP scalar $\phi$, and panel c and d correspond to the WIMP $\phi$

Next, we consider the fermion DM $\chi$. Depending on the quartic coupling $\lambda _{6,7}$, the dark scalar $\phi$ can be either FIMP or WIMP, so a two part scan is also performed here. Assume that $\phi$ is a FIMP, we take the decay mode $N\rightarrow \phi \chi$ with $\phi \rightarrow \chi \nu$ for illustration. We scan the parameter space

$$\begin{aligned} \begin{aligned}&m_\chi \in [0.01,1]~\textrm{GeV}, m_\phi \in [1,100]~\textrm{GeV}, \lambda _{6,ds}\\&\in [10^{-12}, 10^{-8}], \end{aligned} \end{aligned}$$

(21)

where we have fix $m_N=500$ and $\lambda _6=\lambda _7$. The results are shown in Fig. 12a, b, where $R_{N\chi }=\Omega _{N\chi }h^2/\Omega _\textrm{DM}h^2$ denotes the relative contribution from direct decay $N\rightarrow \phi \chi$. When this channel is the dominant one, $\lambda _{ds}\in [2.0\times 10^{-11},2.0\times 10^{-10}]$ should be satisfied. Meanwhile, $R_{\phi \nu }=\Omega _{\phi \nu }h^2/\Omega _\textrm{DM}h^2$ represents the relative contribution from direct decay $\phi \rightarrow \chi \nu$, where $\phi$ is generated via the quartic scalar couplings $\lambda _{6,7}$. It is worth noting that $\phi$ is mainly produced by the decay of Higgs h when $m_\phi <m_h/2$, so $\lambda _6=\lambda _7\in [6.5\times 10^{-12},6.5\times 10^{-11}]$ is enough to make this channel dominant. For $m_\phi \ge m_h/2$, $\phi$ is produced via the $2\rightarrow 2$ scattering of SM particles and $\Phi _\nu$. And $\lambda _6=\lambda _7\in [7.4\times 10^{-11},7.4\times 10^{-10}]$ is required to make the late decay $\phi \rightarrow \chi \nu$ dominant.

Then for a WIMP $\phi$, we consider the decay mode $\phi \rightarrow N\chi$ and scan the parameter space

$$\begin{aligned} \begin{aligned} m_\chi \in [0.01,1]~\textrm{GeV}, \lambda _6\in [10^{-4}, 1], \lambda _{ds}\in [10^{-12}, 10^{-8}], \end{aligned}\nonumber \\ \end{aligned}$$

(22)

where we have fix $m_\phi =500$ GeV, $m_N=100$ GeV and $\lambda _6=\lambda _7$. The results are shown in Fig. 12c, d, where $R_{\phi N}=\Omega _{\phi N}h^2/\Omega _\textrm{DM}h^2$ represent the relative contribution from dominant freeze-in decay $\phi \rightarrow N\chi$ with $\phi$ in the thermal equilibrium state. Contribution of the freeze-in $\phi \rightarrow N\chi$ is proportional to $\lambda _{ds}$, which set an upper bound $\lambda _{ds}\in [3.1\times 10^{-11}, 3.1\times 10^{-10}]$ with $m_\chi$ from 1 GeV to 0.01 GeV. $R_{\phi \chi }=\Omega _{\phi \chi }h^2/\Omega _\textrm{DM}h^2$ indicates the relative contribution of the decay of WIMP $\phi$ after freeze-out. $R_{\phi \chi }$ is inversely proportional to $\lambda _{6,7}$. And $\lambda _6=\lambda _7\in [1.6\times 10^{-4}, 2.0\times 10^{-3}]$ with $m_\chi$ from 0.01 GeV to 1 GeV has been obtained by requiring correct relic density.

4.3 Constraints

For FIMP DM, its couplings are too small to be detected at the traditional direct and indirect DM detection experiments. We consider a major constraint named free-streaming length for FIMP DM, it represents the average distance a particle travels without a collision [43]

$$\begin{aligned} r_\textrm{FS}=\int _{a_\textrm{rh}}^{a_\textrm{eq}}\frac{\langle v\rangle }{a^2H}da \approx \frac{a_\textrm{nr}}{H_0\sqrt{\Omega _R}} \left( 0.62+\ln \left( \frac{a_\textrm{eq}}{a_\textrm{nr}}\right) \right) , \end{aligned}$$

(23)

where $a_\textrm{eq}$, ${a_\textrm{rh}}$ and ${a_\textrm{nr}}$ represent the Friedmann–Robertson–Walker scale factors in equilibrium at matter/radiation equality, reheating and when DM becomes nonrelativistic, respectively. We use the approximate expression

$$\begin{aligned} r_\textrm{FS} \simeq 2.3\times 10^{-6}\left( \frac{\textrm{GeV}}{m_\textrm{DM}}\right) \text {Mpc} \end{aligned}$$

(24)

for a simple estimation. For DM mass in the range of [0.01, 1] GeV, the predicted value is approximately $r_\text {FS}\in [2.3\times 10^{-6},2.3\times 10^{-4}]$ Mpc. Such a small $r_\text {FS}$ will satisfy the most stringent bound comes from small structure formation $r_\text {FS}<0.1$ Mpc [87], therefore all the DM particles in our analysis are cold DM.

In our FIMP DM scenarios, the next lightest odd particle (NLOP) decays into DM and sterile neutrinos N when N is light enough. For $\lambda _{ds}\gtrsim 10^{-12}$, NLOP decays fast to avoid constraints from BBN [39]. On the other hand, NLOP will decay to DM and light neutrinos through $\chi \rightarrow \phi \nu$ or $\phi \rightarrow \chi \nu$ when the N is heavier than the dark sector. The tiny mixing angle $\theta$ between sterile and light neutrinos will lead to a delayed decay of NLOP, so the energetic final states neutrinos can be constrained and captured by certain experiments [38]. We consider the scenario in Fig. 7c for scalar DM $\phi$ and the scenario in Fig. 8a for fermion DM $\chi$. Typically, the lifetime of NLOP is about $\tau _\textrm{NLOP}\sim 10^{12}$ s with $m_\textrm{NLOP}$, $\lambda _{ds}\sim 10^{-11}$ and $\theta \sim 10^{-7}$, which is close to the time of recombination.

The final state neutrinos from long-lived NLOP can induce electromagnetic and hadronic showers, which will affect the CMB and BBN. In Fig. 13, we show the CMB and BBN constraints in the parameter space of $m_\textrm{NLOP}$ and $\tau _\textrm{NLOP}$ [88], and the scanning results of the considered two scenarios. The cosmological constraints depend on the fractional abundance $f_\textrm{NLOP}=\Omega _\textrm{NLOP}/\Omega _\textrm{DM}$. The constraints in Fig. 13 correspond to the most stringent one with $f_\textrm{NLOP}=10^{4}$ in version one of Ref. [88], because in the scanning parameter space $f_\textrm{NLOP}$ could reach $10^{4}$ in the extreme case when $m_\textrm{NLOP}=100$ GeV and $m_\textrm{DM}=0.01$ GV. It is obvious that the mass of NLOP should be less than 40 GeV. And the larger lifetime is usually satisfied only when the mass of NLOP is smaller. It is notable that the constraints from BBN heavily depend on $f_\textrm{NLOP}$, e.g., $m_\textrm{NLOP}\sim 100$ GeV can be allowed when $f_\textrm{NLOP}=1$ and $\tau _\textrm{NLOP}\sim 10^{11}$ s.

Fig. 13

The cosmological constraints for two specified scenarios. Left panel: DM $\phi$. Right panel: DM $\chi$. The pink curves represent the constraint from CMB and BBN [88]. The pink and green points represent the excluded and allowed parameter spaces respectively

Fig. 14

The results of neutrino flux constraints for two benchmark points. The orange and green solid lines represent the results from DM $\phi$ and $\chi$, respectively. The red dotted line represent the thermal solar neutrinos flux [89]. The blue dotted lines are the solar neutrino flux for nuclear production processes [89]. The black squares and purple triangles represent diffuse supernova neutrino background (DSNB) flux of the electron anti-neutrinos with the KamLAND [90] and SK [91] data, respectively

The energetic neutrinos from delayed NLOP decay might be detectable at the neutrino experiments. The neutrino flux from delayed NLOP decay is calculated as [38],

$$\begin{aligned} \Phi _\textrm{cos}\equiv E_\nu ^2 \frac{d\varphi }{dE_\nu }=E_\nu \left( \frac{n_\text {NLOP}^0}{\tau _\text {NLOP}}\right) \left( \frac{e^{-t(x)/\tau _\text {NLOP}}}{H(x)}\right) \theta ^{'}(x),\nonumber \\ \end{aligned}$$

(25)

where $E_\nu$ is the observed neutrino energy, $d\varphi /dE_\nu$ is the predicted neutrino flux, $n_\textrm{NLOP}^0$ is the NLOP number density when it acts as a decaying particle, $\theta ^{'}(x)$ is the Heaviside theta function. We have $n_\textrm{NLOP}^0=\rho _\textrm{DM}/m_\textrm{DM}$ with the observed DM energy density $\rho _\textrm{DM}=0.126\times 10^{-5}$ ${\mathrm{GeV/cm}^3}$. The cosmic time t(x) at red-shift $1+x$ and the Hubble parameter H(x) in the standard cosmology are given by

$$\begin{aligned} t(x)\approx & {} \frac{4}{3H_0}\left( \frac{\Omega _\textrm{r}^{3/2}}{\Omega _\textrm{m}^{2}}\right) \left( 1-\left( 1-\frac{\Omega _\textrm{m}}{2(1+x)\Omega _\textrm{r}}\right) \right. \nonumber \\{} & {} \times \left. \sqrt{1+\frac{\Omega _\textrm{m}}{(1+x)\Omega _\textrm{r}}}\right) , \end{aligned}$$

(26)

$$\begin{aligned} H(x)= & {} H_0\sqrt{\Omega _\Lambda +(1+x)^3\Omega _\textrm{m}+(1+x)^4\Omega _\textrm{r}}, \end{aligned}$$

(27)

where $x=E_0/E_\nu -1$ with initial energy of NLOP $E_0=(m_\textrm{NLOP}^2-m_\textrm{DM}^2)/2m_\textrm{NLOP}$, the Hubble constant $H_0=100\,h$ $\mathrm{km/s/Mpc}$ with $h=0.6727$ [75], the dark energy, matter and radiation (CMB photons and neutrinos) fractions are $\Omega _\Lambda =0.6846, \Omega _\textrm{m}=0.315$ and $\Omega _\textrm{r}=9.265\times 10^{-5}$.

The predicted neutrino flux is shown in Fig. 14. For both scalar and fermion DM, we assume $m_\textrm{NLOP}=10$ GeV, $m_\textrm{DM}=0.1$ GeV. These two benchmark points meet $\tau _\chi =2.38\times 10^{10}$ s for DM $\phi$ and $\tau _\phi =1.16\times 10^{10}$ s for DM $\chi$, which are allowed by the cosmological constraints in Fig. 13. Although the initial neutrino energy from NLOP decay is $E_0\sim m_\textrm{NLOP}/2=5$ GeV, the observed energy $E_\nu$ is red-shifted to below 1 MeV, which makes these neutrinos are hard to detect at current neutrino experiments.

5 Conclusion

In this paper, we study the phenomenology of sterile neutrino portal DM in the $\nu$THDM. This model introduces one neutrinophilic scalar doublet $\Phi _{\nu }$ and sterile neutrinos N. The soft-term $\mu ^2 \Phi ^\dag \Phi _\nu$ induces a tiny VEV for $\Phi _\nu$, which leads to naturally tiny neutrino mass with $\Phi _\nu$ and N around TeV scale. In this way, the Yukawa coupling between $\Phi _{\nu }$ and N is large enough to keep N in thermal equilibrium. The dark sector consists of one Dirac fermion singlet $\chi$ and one scalar singlet $\phi$, which are odd under a $Z_2$ symmetry. The sterile neutrinos N are the mediator between the DM and SM. Depending on the coupling strength, the DM can be either WIMP or FIMP.

For the WIMP scenario, the scalar candidate $\phi$ can annihilate into SM final states, into neutrinophilic scalars, and into sterile neutrinos. The Higgs portal interaction is tightly constrained by direct detection. Under the constraints from Higgs invisible decay, direct and indirect detection, $m_\phi$ should be larger than about 60 GeV. For the fermion candidate $\chi$, $\chi {\bar{\chi }}\rightarrow NN$ is the only possible annihilation channel. Current Higgs invisible decay and direct detection limits do not exclude any samples. Meanwhile, the indirect detection has exclude $m_\chi \lesssim 60$ GeV. In the future, the indirect detection experiment as CTA is able to probe the region with DM mass less than 1 TeV for both scalar and fermion candidate.

For the FIMP scenario, we consider the direct production of DM from freeze-in mechanism, as well as contributions from late decay of NLOP. For scalar FIMP, it can be generated from decay of SM Higgs boson h or sterile neutrinos N, annihilation of neutrinophilic scalars via $\Phi _\nu \Phi _{\nu }\rightarrow \phi \phi$ or via $\Phi _{\nu } L\rightarrow \phi \chi$. For fermion FIMP, it is mainly generated from decay of sterile neutrino N or NLOP $\phi$. Both the FIMP and WIMP case of $\phi$ are discussed. Although the traditional direct and indirect DM detection experiments are hard to probe FIMP DM, the energetic neutrinos from delayed decay of NLOP will lead to constraints from CMB, BBN, and neutrino experiments when the sterile neutrinos are heavier than the dark sector. Under these constraints, the NLOP mass should be less than about 40 GeV.