1 Introduction

The discovery of Higgs boson with mass about 125 GeV by ATLAS and CMS collaborations [1, 2] in 2012 stands for the beginning of a new era for high-energy physics. So far, the properties of the Higgs boson, such as the coupling with other Standard Model (SM) particles, have been measured and the measured properties are in accord with the prediction of SM. Significant improvements are expected from the upcoming runs of the Large Hadron Collider (LHC) or future \(e^{+}e^{-}\)colliders. However, the Higgs signal near 125 GeV observed at the LHC may also come from the contributions of two or more nearly mass-degenerate neutral Higgs bosons, and the Higgs resonances cannot be individually resolved by experiment. References [3,4,5] have developed different methods to detect whether two or more resonances exist. Some studies have also been done in the scenario with mass-degenerate Higgs states in the two-Higgs-doublet model (2HDM) [6, 7] and Next-to-Minimal Supersymmetric Standard Model (NMSSM) [8,9,10,11,12,13].

Compared with the minimal supersymmetric standard model (MSSM), the NMSSM introduces an extra singlet field \(\hat{S}\). An effective \(\mu \)-term is generated dynamically when \(\hat{S}\) acquires the vacuum expectation value (vev), and \(\mu \) problem in the MSSM can be solved elegantly [14, 15]. Meanwhile, the Higgs sector of the NMSSM consists of three CP-even neutral scalars and two CP-odd neutral scalars, which is more complex in comparison to the MSSM. In addition, some experimental constraints on the NMSSM can be weaker due to the mixing between the singlet superfield and the doublet Higgs fields. As a result, some unique phenomenological properties of the Higgs boson may appear in the NMSSM [16, 17].

Due to the interaction \(\lambda \hat{S}\hat{H_u}\cdot \hat{H_d}\) among the Higgs fields, the squared mass of SM-like Higgs boson in the NMSSM receives a positive contribution at tree-level [18,19,20,21,22], and it can be further enhanced by the mixing between singlet field and doublet field when the next-to-lightest CP-even Higgs boson is SM-like [23,24,25]. For larger values of \(\lambda \) and smaller \(\tan \beta \), the positive contribution is substantial and large radiative correction to the Higgs boson mass is unnecessary and the little hierarchy problem can be avoidable. However, in some cases, for example smaller values of \(\lambda \) and larger \(\tan \beta \), the positive contribution at tree-level may be less than the loop-corrected contributions to achieve a mass around 125 GeV. About the loop corrections to the Higgs boson mass and Higgs boson decays in the NMSSM, a lot of works [26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42] have been done in the recent years. At present there are lots of public codes available for predictions in the NMSSM, such as NMSSMTools [43,44,45], NMSSMCalc [46], SARAH/SPheno [47,48,49,50,51,52,53,54], SoftSUSY [55] and FlexibleSUSY [56,57,58]. Considering the current experimental constraints, especially the constraints from the latest Dark Matter (DM) direct detection, we aim to study this scenario with two nearly mass-degenerate CP even Higgs bosons in the NMSSM, i.e. the observed Higgs signal at the LHC may be due to the superposition of this two resonances.

The paper is organized as follows. In Sect. 2 we will briefly review the basics of the scale invariant NMSSM, and perform a comprehensive scan over the parameter space considering the current experimental constraints. In Sect. 3, we will show the impact of constraints from DM direct detection on the parameter space of NMSSM. In Sect. 4 we show the numerical results of two mass-degenerate states in Higgs boson signals and discuss the main production channels and decay channels relevant for LHC data. Eventually, we give a summary in Sect. 5.

2 Model and scan strategies

2.1 Basics of the NMSSM

Compared with the MSSM, the Higgs sector of the NMSSM is rather complicated by adding an extra singlet field \(\hat{S}\). The superpotential and the tree-level Higgs potential of the model are given by [14]:

$$\begin{aligned} W= & {} W_{F}+\lambda \hat{H}_{u}\cdot \hat{H}_{d} \hat{S} + \frac{1}{3}\kappa \hat{S}^3, \end{aligned}$$
(1)
$$\begin{aligned} V= & {} V_{F}+V_{D}+V_\mathrm{{soft}} \end{aligned}$$
(2)
$$\begin{aligned} V_{F}= & {} |\lambda S|^2 (|H_u|^2 + |H_d|^2) + |\lambda H_u \cdot H_d + \kappa S^2|^2 \nonumber \\ V_{D}= & {} \frac{1}{8} ( {g_1}^2 + {g_2}^2) (|H_d|^2 - |H_u|^2)^2 + \frac{1}{2} {g_2}^2 | H_u^{\dagger } \cdot H_d |^2 \nonumber \\ V_\mathrm{{soft}}= & {} m_{H_u}^2 |H_u|^2 + m_{H_d}^2 |H_d|^2 + m_S^2 |S|^2 \nonumber \\&+ (\lambda A_{\lambda } S H_u \cdot H_d + \frac{1}{3} \kappa A_{\kappa } S^3 + h.c.), \end{aligned}$$
(3)

where \(W_{F} \) represents the MSSM superpotential without the \(\mu \)-term, the fields \(\hat{H}_u\), \(\hat{H}_d\) and \(\hat{S}\) are Higgs superfields and their scalar components are \(H_u\), \(H_d\) and S respectively, the dimensionless parameters \(\lambda \) and \(\kappa \) are the coupling strength in the Higgs sector, \(g_1\) and \(g_2\) are the \(U(1)_Y\) and \(SU(2)_L\) gauge couplings, and the remaining quantities \(m_{H_u,H_d,S}^2\) and \(A_{\lambda ,\kappa }\) are soft breaking parameters.

The tree-level Higgs potential of the NMSSM is made up of F-term and D-term of the superfields, and also the soft breaking terms. Using the minimization condition of the scalar potential, the soft breaking masses \(m_{H_u}^2\), \(m_{H_d}^2\) and \(m_S^2\) can be replaced by \(v_u\), \(v_d\) and \(v_s\), and \(v_u\), \(v_d\), \(v_s\) are the vevs of the fields \(H_u\), \(H_d\) and S, respectively. As a result, in the Higgs sector, six independent parameters in the following are needed,

$$\begin{aligned} \lambda , \quad \kappa , \quad \tan {\beta } = \frac{v_u}{v_d}, \quad \mu =\lambda v_s, \quad A_{\lambda }, \quad A_{\kappa } \end{aligned}$$
(4)

We define \(h_0=\cos {\beta } H_u+\varepsilon \sin {\beta } H_d^*\) and \(H_0=\sin {\beta } H_u-\varepsilon \cos {\beta } H_d^*\) with \(\varepsilon \) being two-dimensional antisymmetric tensor and \(\varepsilon _{12}=-\varepsilon _{21}=1\) and \(\varepsilon _{11}=\varepsilon _{22}=0\), then the Higgs fields in the CP-conserving NMSSM can be written as [59]:

$$\begin{aligned}&h_0 = \left( \begin{array}{c} H^+ \\ \frac{S_1 + \mathrm {i} P_1}{\sqrt{2}} \end{array} \right) ,~~ H_0 = \left( \begin{array}{c} G^+ \\ v + \frac{S_2 + \mathrm {i} G^0}{\sqrt{2}} \end{array} \right) ,\nonumber \\&S = v_s +\frac{1}{\sqrt{2}} \left( S_3 + \mathrm {i} P_2 \right) , \end{aligned}$$
(5)

where \(G^+\) and \(G^0\) stand for Goldstone bosons and \(v^2 = v_u^2 + v_d^2\).

The charged fields \(H^\pm \) are already physical mass eigenstates, and their masses at tree-level can be given by

$$\begin{aligned} M^2_{H^\pm }= & {} M^2_A+m^2_W-\lambda ^2v^2 \end{aligned}$$
(6)

with \(M_A^2=2\mu (A_\lambda +\kappa v_s)/\sin 2\beta \).

The \(3\times 3\) symmetric CP-even Higgs mass matrix \(M^2\) at tree level under the basis \((S_1,S_2,S_3)\) is described as [60]

$$\begin{aligned} M_{S_1S_1}^2= & {} M^2_A + (m^2_Z -\lambda ^2 v^2) \sin ^2 2\beta ,\\ M_{S_1S_2}^2= & {} -\frac{1}{2}(m^2_Z-\lambda ^2 v^2)\sin 4\beta , \\ M_{S_1S_3}^2= & {} -\left( \frac{M^2_A}{2\mu /\sin 2\beta }+\kappa v_s\right) \lambda v\cos 2\beta , \\ M_{S_2S_2}^2= & {} m_Z^2\cos ^2 2\beta +\lambda ^2v^2\sin ^2 2\beta , \\ M_{S_2S_3}^2= & {} 2\lambda \mu v\left[ 1-\left( \frac{M_A}{2\mu /\sin 2\beta }\right) ^2 -\frac{\kappa }{2\lambda }\sin 2\beta \right] , \\ M_{S_3S_3}^2= & {} \frac{1}{4}\lambda ^2 v^2\left( \frac{M_A}{\mu /\sin 2\beta }\right) ^2 +\kappa v_s A_{\kappa }+4(\kappa v_s)^2\\&-\frac{1}{2}\lambda \kappa v^2 \sin 2\beta . \end{aligned}$$

By diagonalizing the squared mass matrix \(M^2\) with rotation matrix V, we can get the physical mass eigenstates \(H_{i} = \sum \nolimits _{j=1}^3 V_{ij} S_j\) (\(i = 1, 2, 3\)). Similarly, one can also get the CP-odd mass eigenstates \(A_1\) and \(A_2\). In general, we assume \(m_{H_1}<m_{H_2}<m_{H_3} \) and \(m_{A_1} < m_{A_2} \), and call \(H_i \) the SM-like Higgs boson if it is dominated by the \(S_2\) field. Without the mixing among \(S_i\) fields, the squared mass of SM-like Higgs boson receives an additional contribution \(\lambda ^2 v^2 \sin ^2 2\beta \) compared with the case in MSSM. Furthermore, the mass of SM-like boson can also be enhanced by the (\(S_2,S_3\)) mixing effect if \(M_{S_3S_3}^2 < M_{S_2S_2}^2\). Therefore, the mass of the SM-like Higgs boson can get its measured value without large radiative corrections [61,62,63].

For large \(\tan \beta \) and large \(M_A\), the tree-level physical masses of the two lightest CP-even Higgs bosons may be approximately expressed as follows [59],

$$\begin{aligned} M^2_{H_{2/1}}\approx & {} \frac{1}{2}\left\{ m^2_Z+\frac{\kappa \mu }{\lambda } \left( \frac{4\kappa \mu }{\lambda }+A_\kappa \right) \right. \nonumber \\&\left. \pm \sqrt{\left[ m^2_Z-\frac{\kappa \mu }{\lambda } \left( \frac{4\kappa \mu }{\lambda }+A_\kappa \right) \right] ^2 +\frac{\lambda ^2v^2}{2\mu ^2}(4\mu ^2-M_A^2\sin ^22\beta )^2}~ \right\} . \end{aligned}$$
(7)

Hence the mass difference between the tree-level masses should be as small as possible in order to obtain two mass-degenerate Higgs bosons.

2.2 Scan strategies and constraints on the parameter space of NMSSM

From Eq. (6), we can see that the parameter \(A_\lambda \) determines the size of \(M_{H^\pm }\). If the mixing among the doublet fields should be negligible, then \(M_{H^\pm }\) should be large. So we fix \(A_\lambda \) to be 2 TeV.Footnote 1 Considering the loop-corrected contributions to the Higgs boson mass, the two nearly mass-degenerate CP-even Higgs states strongly depend on the parameters \(\lambda , \kappa , \tan \beta , \mu , A_\kappa \) and \(A_t\) (\(A_t\) is stop trilinear coupling), so we perform a random scan over these parameters in the following ranges:

$$\begin{aligned} \begin{aligned}&0<\lambda<0.75,\quad \left| \kappa \right|<0.75,\quad 1<\tan {\beta }<60,\\&100~{\mathrm{GeV}}\le \mu \le 1~{\mathrm{TeV}},\quad \left| A_\kappa \right| \le 1~{\mathrm{TeV}},\quad \left| A_t\right| \le 5~{\mathrm{TeV}}. \end{aligned} \end{aligned}$$
(8)

For other unimportant parameters including the soft breaking masses of gauginos, the soft breaking parameters in the slepton sector and squark sector except \(A_t\), we fix them to be 2 TeV.

In the calculation, we use the package NMSSMTools [43, 44] to generate the particle mass spectrum, the relevant couplings and the decay branching ratios of Higgs bosons. We also consider the constraints on direct search for Higgs bosons from LEP, Tevatron and LHC with the package HiggsBounds [64, 65], and perform fits for the 125 GeV Higgs data with the package HiggsSignals [66,67,68]. Moreover, we use the package micrOMEGAs [69, 70] to compute the DM relic density and also the spin-independent (SI) and spin-dependent (SD) cross sections. We take the limits from LUX-2017 [71] for SD cross section and XENON1T-2018 [72] for SI cross section, respectively. We only require the DM relic density is less than the measured central value, i.e. \(\Omega h^2 < 0.1187\). Since we assume DM in the NMSSM is only one of DM candidates [73], the SI and SD cross sections for DM-nucleon scattering should also be scaled by a factor \((\Omega h^2)/0.1187\).

3 Impact of constraints from DM direct detection on the parameter space of NMSSM

In contrast to limits from direct searches for Higgs bosons and DM relic density, DM direct detection experiments has a strong constraint on the parameter space of NMSSM when we require there exist two nearly mass-degenerate Higgs bosons. At present, the strong limits on the SI and SD cross sections of DM-nucleon scattering come from the XENON-1T experiment in 2018 and the LUX measurement in 2017, respectively. Considering the constraints from the DM relic density, Higgs data from LEP, Tevatron and LHC experiments, we calculate the SI and SD cross sections separately and project the samples on the \(\sigma -m_{\tilde{\chi }_1^0}\) plane in Fig. 1. In the left plane, samples with green color above (below) the red dashed line have been excluded by the XENON-1T experiment (LUX experiment). In the right plane, samples with cyan color above (below) the magenta dash-dotted line have been excluded by the LUX experiment (XENON-1T experiment). The samples with red color in the left plane and samples with magenta color in the right plane are still experimentally allowed. We can see that the two experiments are complementary to limit the parameter space of NMSSM. In the following, we will discuss the features of the parameter space in the NMSSM, which is limited tightly by the DM direct detection experiments.

Fig. 1
figure 1

SI (SD) cross section for DM-nucleon scattering versus DM mass considering the constraints from the DM relic density, Higgs data from LEP, Tevatron and LHC experiments. In the left plane, the red dashed line represents the limit from XENON-1T-2018. In the right plane, the magenta dash-dotted line represents the limit from LUX-2017

Full size image
Fig. 2
figure 2

Samples projected on \(\lambda - \kappa \) plane, \(\lambda -A_t\) plane and \(\mu -\tan \beta \) plane. The green color indicates samples excluded by the DM direct detection experiments, and the red ones are still experimentally allowed

Full size image

In Fig. 2, we project the samples on \(\lambda - \kappa \) plane, \(\lambda -A_t\) plane and \(\mu -\tan \beta \) plane. The samples marked by green color have been excluded by DM direct detection experiments and the red ones are still experimentally allowed. We see that the samples with large \(\lambda \) and small \(\kappa \) shown in Fig. 2a and also samples with large \(\mu \) shown in Fig. 2c are excluded by the DM direct searches. This is because that DM corresponding to these samples are composed mainly of higgsinos and the couplings of Higgs-DM-DM are relatively large. The figure also shows that the parameter space of NMSSM shrinks somewhat after considering the constraints of DM direct detection. The surviving parameter space of NMSSM is featured by \(100~{\mathrm{GeV}} \lesssim \mu \lesssim 250~{\mathrm{GeV}}\) and \(6\lesssim \tan \beta \lesssim 25\). For most of the surviving samples, \(\lambda \) is not too large, \(\tan \beta \) is relatively large and the value of parameter \(A_t\) is relatively large. Hence, the tree-level contributions for the Higgs boson mass from \(\lambda ^2v^2\sin ^2 2\beta \) is not large and the contributions from the stop-loop radiative corrections are important to obtain two mass-degenerate Higgs boson near 125 GeV.

Fig. 3
figure 3

The double ratios \(D_1\), \(D_2\), \(D_3\) defined in Eq. (10), as functions of the mass difference \(m_{H_2}-m_{H_1}\). The color-bar in all the panels corresponds to the total \(\chi ^2\) calculated by HiggsSignals

Full size image
Fig. 4
figure 4

The field components of \(H_1\) (\(H_2\)) versus \(m_{H_1}\) (\(m_{H_2}\)). The samples with dark blue color represent the component of singlet field \(S_3\), the samples with yellow color represent the component of field \(S_2\), and the samples with light blue color represent the component of field \(S_1\)

Full size image

4 Searching for two mass-degenerate Higgs bosons near 125 GeV

At the LHC, the main production channels for Higgs boson are gluon-gluon fusion and vector boson fusion (VBF). We focus on these two main production modes (denoted by \(Y=gg\) or \(Y= \hbox {VBF}\)) and consider the Higgs decay channels (denoted by X) to \(\gamma \gamma , ZZ^*, b\bar{b}\) and \(\tau ^+\tau ^-\). We define the following ratio

$$\begin{aligned} \begin{aligned} R^{H_i}_{gg}(X)&\equiv \frac{\sigma (gg\rightarrow H_i )}{\sigma (gg\rightarrow h_{\mathrm{SM}})}\times \frac{{\mathrm{BR}}(H_i\rightarrow X)}{{\mathrm{BR}}(h_{\mathrm{SM}}\rightarrow X)}\\&= |C_{H_igg}|^2\times \frac{|C_{H_iX}|^2}{\Gamma _{H_i}/\Gamma _{h_{\mathrm{SM}}}} \\ R^{H_i}_{\mathrm{VBF}}(X)&\equiv \frac{\sigma ({\mathrm{VBF}}\rightarrow H_i)}{\sigma ({\mathrm{VBF}}\rightarrow h_{\mathrm{SM}})}\times \frac{{\mathrm{BR}}(H_i\rightarrow X)}{{\mathrm{BR}}(h_{\mathrm{SM}}\rightarrow X)}\\&=|C_{H_iWW}|^2\times \frac{|C_{H_iX}|^2}{\Gamma _{H_i}/\Gamma _{h_{\mathrm{SM}}}} \end{aligned} \end{aligned}$$
(9)

where \(C_{H_iX}\), \(C_{H_igg}\), \(C_{H_iWW}\) are the predictions of \(H_iX\), \(H_igg\), \(H_iWW\) couplings in the NMSSM normalized to its SM predictions, \(\Gamma _{H_i}\) and \(\Gamma _{h_{\mathrm{SM}}}\) are the total decay widths of \(H_i\) and \(h_{\mathrm{SM}}\), respectively. Due to the custodial symmetry, i.e. \(C_{H_iWW}=C_{H_iZZ}\), the relation \(R^{H_i}_Y(WW)=R^{H_i}_Y(ZZ)\) applies. We use the package NMSSMTools to calculate the normalized couplings \(C_{H_iX}\), \(C_{H_igg}\), \(C_{H_iWW}\) and the total decay widths \(\Gamma _{H_i}\), \(\Gamma _{h_{\mathrm{SM}}}\).

To test the existence of two mass-degenerated Higgs states, Ref. [3] defined the following double ratios,

$$\begin{aligned} D_1= & {} \frac{R^h_\mathrm{VBF}(\gamma \gamma )/R^h_{gg}(\gamma \gamma )}{R^h_{\mathrm{VBF}}(b \bar{b})/R^h_{gg}(b\bar{b})}\,\nonumber \\ D_2= & {} \frac{R^h_\mathrm{VBF}(\gamma \gamma )/R^h_{gg}(\gamma \gamma )}{R^h_\mathrm{VBF}(WW)/R^h_{gg}(WW)}\, \\ D_3= & {} \frac{R^h_{\mathrm{VBF}}(WW)/R^h_{gg}(WW)}{R^h_{\mathrm{VBF}}(b\bar{b})/R^h_{gg}(b\bar{b})}\,\nonumber \end{aligned}$$
(10)

where \(R^h_Y(X)=R^{H_1}_Y(X)+R^{H_2}_Y(X)\). Each of these double ratios should be unity if only a single Higgs boson contributes to the 125 GeV SM-like Higgs signal. However, these ratios may be deviated from 1 if two Higgs bosons contribute to the observed Higgs signal.

After implementing all the constraints mentioned above, we perform a global \(\chi ^2\) fit to a total of 102 most up-to-date Higgs boson observables at the LHC by the package HiggsSignals-2.2.3beta. Then in Fig. 3 we display the double ratios \(D_1\), \(D_2\), \(D_3\) as functions of the mass difference \(m_{H_2}-m_{H_1}\) with the color-bar representing the total \(\chi ^2\). From the figure, we can see that the three ratios are very close to unity, and the maximum deviation can reach to 10%. Both \(D_1\) and \(D_3\) can be larger or smaller than 1 because they depend on the same decay channel \(H_i\rightarrow b\bar{b}\), while \(D_2\) is not bigger than 1 because it relies on only the bosonic signal strengths. Meanwhile, the small values of \(\chi ^2\) correspond to samples with small deviation of each ratios and the samples with large deviation correspond to relatively larger \(\chi ^2\), such as for samples with deviation reach to 10% (4%), the value of \(\chi ^2\) is about 125 (100) in Fig. 3a, c, and samples with deviation of 3% (1%), the value of \(\chi ^2\) is about 120 (100) in Fig. 3b.

Figure 4 shows the field components of \(H_1\) and \(H_2\), respectively. For the left (right) upper corner samples, \(H_1\) (\(H_2\)) is dominated by singlet field \(S_3\) and \(H_2\) (\(H_1\)) consists mainly of SM-like Higgs field \(S_2\). Their corresponding Higgs signal characters \(D_i(i=1,2,3)\) are around one, which stands for the singlet dominated Higgs make little contribution to the Higgs signal. But for the middle samples, the mixing between \(S_2\) and \(S_3\) fields is large, both \(H_1\) and \(H_2\) have relatively large components of SM-like Higgs filed \(S_2\) and their double ratios \(D_i (i=1,2,3)\) slightly deviate from one. That is to say, the observed Higgs signal may be due to the contributions from the nearly mass-degenerate Higgs bosons near 125 GeV.

5 Summary

Considering the current experimental limits from LHC Higgs data and DM relic density, especially limits from DM direct detection experiments, there still exist samples predicting two nearly mass-degenerate neutral CP-even Higgs bosons lie in the mass range between 122 and 128 GeV. The surviving samples are featured with relatively small \(\mu \) and large \(\tan \beta \). We can use the double ratios defined in Eq. (10) to distinguish the existence of two mass-degenerate Higgs bosons. The maximum deviation of the double ratio from 1 can reach to about 10%. The samples with large deviation of each ratios correspond to large mixing between the \(S_2\) and \(S_3\) fields.

6 Note added

We note that varying the bino parameter \(M_1\) would affect the properties of DM somewhat and we will talk this scenario in detail in our next work.