1 Introduction

Pair-production of the Higgs boson state, \(h_{\mathrm{obs}}\), discovered in 2012 [1, 2] is a key process for measuring the Higgs self-coupling at the Run 3 of the large hadron collider (LHC), as well as at its now approved high-luminosity upgrade (HL-LHC). This process represents a direct probe of the mechanism of electroweak symmetry breaking (EWSB), since the Higgs self-coupling enters the Higgs potential directly. Accessing it experimentally will thus be of extreme importance in order to understand whether mass generation in nature occurs within the Standard Model (SM) or in some scenario incorporating new physics.

In a beyond-the-SM (BSM) framework containing an extended Higgs sector, the phenomenology of the pair-production process of the \(h_{\mathrm{obs}}\) candidate, i.e., the Higgs boson with mass lying near 125 GeV, can deviate significantly from that in the SM due to two main reasons. First, the \(h_{\mathrm{obs}}\) self-coupling gets modified from its predicted value in the SM owing to the mixing between various interaction eigenstates. Secondly, the additional Higgs states also enter the resonant channel, so that the other Higgs trilinear couplings appearing in the Lagrangian of the model also come into play. While there exists plenty of literature on \(h_{\mathrm{obs}}\) pair-production in BSM scenarios at the various energy and luminosity stages of the LHC, most often this is limited to frameworks wherein only one CP-even companion to the SM-like Higgs state exists, like the 2-Higgs Doublet Model (2HDM) [3] or the Minimal Supersymmetric Standard Model (MSSM) [4, 5].

In the MSSM, the requirement for the lighter of its two scalar states, \(H_1\), to be a good \(h_{\mathrm{obs}}\) candidate pushes the heavier scalar, \(H_2\), as well as the solo pseudoscalar, A, up into the so-called decoupling regime [6], where they have identical masses.Footnote 1 If the MSSM Higgs sector is CP-violating, all the interaction eigenstates can mix together to yield three CP-indefinite physical states, with the two nearly mass-degenerate heavy states now labelled \(H_2\) and \(H_3\). When the mass-splitting between these two is comparable to the sum of their widths, a description of the intervening propagators which takes into account the imaginary parts of the one-loop self-energies, alongside the customary real parts, becomes necessary [10, 11]. This is because the imaginary off-diagonal entries of the Higgs propagator matrix can induce quantum interference between these states, so that the one produced in, for example, gluon-fusion can potentially oscillates into the other one before decaying into a given SM final state. This can significantly alter not only the total production cross section but also the shape of the differential cross section distribution for that final state [12, 13].

In the 2-Higgs-doublet model (2HDM), obtained by simply adding an additional Higgs doublet to the SM, which results again in two scalar and one pseudoscalar Higgs states, a mass-degeneracy between \(H_2\) and A is not a precondition for the \(H_1\) to have properties identical to the \(h_{\mathrm{obs}}\). It is nonetheless a possibility not ruled out by any experimental results, and the aforementioned interference effects can become significant in this model also if it has a CP-violating Higgs sector with \(m_{H_2}\approx m_{H_3}\). In a BSM scenario containing three or more CP-even Higgs bosons, the quantum interference effects can appear in processes involving Higgs propagators without the need to invoke CP-violation. A minimal realisation of such a scenario would be the extension of the two models mentioned above by a singlet Higgs field, resulting in an extra scalar state in their Higgs sectors.

In the context of supersymmetry, adding a complex singlet Higgs field to the MSSM can address some of its theoretical and phenomenological shortcomings, resulting in the so-called Next-to-MSSM (NMSSM) [14,15,16,17]. In this model, some particular configurations of the free parameters can yield a SM-like \(H_1\) along with \(H_2\) and \(H_3\) that are nearly mass-degenerate. We have previously investigated the aforementioned interference effects in the NMSSM, in the scenario where \(H_1\) and \(H_2\) are mass-degenerate [18], as well as in the alternative scenario with \(m_{H_2}\approx m_{H_3}\) [19]. The first study pertained to the production process for the \(\gamma \gamma \) final state and the second to that of \(\tau ^+\tau ^-\) at the LHC. Both these studies found the results from the calculation embedding the full Higgs propagator matrix to be notably different from the ones using the standard approximation where only one term containing a Breit–Wigner (BW) propagator corresponding to each of the Higgs bosons appears in the amplitude expression. It was also shown in those papers that the expected mass resolutions of the respective final states at the LHC may, however, not allow it to disentangle the two Higgs states from each other.

In this article, we investigate the implications of the quantum interference on the gluon-initiated pair-production of the SM-like \(H_1\) state of the NMSSM at the LHC with \(\sqrt{s}=14\) TeV, and also of its non-supersymmetric counterpart, the next-to-2HDM (N2HDM). The latter model is obtained by introducing a real singlet Higgs field into the 2HDM, and while it is phenomenologically similar to the NMSSM, a crucial advantage the N2HDM has is that the physical Higgs boson masses can themselves be the input parameters. This grants us the freedom of setting the \(H_2\) and \(H_3\) masses exactly equal and assessing the impact of this maximal mass-degeneracy on the said process. This model additionally allows us to analyse how the various Higgs couplings govern the relative sizes of the interference effects, so that the general inferences can be extended to other multi-Higgs BSM scenarios.

The article is organised as follows. In the next section we briefly revisit the Higgs pair-production process at the LHC. In Sect. 3 we discuss some details of the NMSSM and the N2HDM, as well as of our numerical computational tool. In Sect. 4 we present our analysis and discuss its results. In Sect. 5 we conclude our findings.

2 Higgs pair-production at the LHC

The cross section for the (inclusive) process \(pp \rightarrow H_iH_j\), where \(i,j=1,...,N\) for a model with N CP-even Higgs bosons but without any additional particle content beyond the SM, can be written at the leading order (LO) as

$$\begin{aligned}&\sigma _\text {LO}(pp\rightarrow H_iH_j) \nonumber \\&\quad = \int _{0}^{1} d\tau \int _{\tau }^{1} \frac{dx_1}{x_1} \frac{g(x_1) g(\tau /x_1)}{1024 \pi {\hat{s}}^3} {{\mathcal {A}}}_{gg \rightarrow H_iH_j}^2\,, \end{aligned}$$
(1)

where \(g(x_1)\) and \(g(x_2)\) are the parton distribution functions (PDFs) of the two incoming gluons having squared centre-of-mass (CM) energy \({{\hat{s}}} = x_1 x_2 s\), given in terms of the total CM energy, s, of the pp system, and by defining \(\tau \equiv \frac{{\hat{s}}}{s} = x_1 x_2\). The amplitude-squared in Eq. (1) can be written, following the notation of Ref.  [20], as

$$\begin{aligned} {{\mathcal {A}}}^2_{gg \rightarrow H_iH_j}=\Big |C_\vartriangle F_\vartriangle + C_\Box F_\Box \Big |^2 + \Big |C_\Box G_\Box \Big |^2, \end{aligned}$$
(2)

where \(\vartriangle \) denotes the contribution from the Higgs-mediated triangle loop diagram, Fig. 1 (left), and \(\Box \) refers to the quark-box diagram, Fig. 1 (right).

The coefficient corresponding to the box contributions in Eq. (2) is written in terms of the Yukawa couplings as

$$\begin{aligned} C_\Box = \displaystyle \sum _{q} g_{H_i{\bar{q}}q}g_{H_j{\bar{q}}q}\,. \end{aligned}$$
(3)

The form factor \(F_\Box \) corresponds to the case when the gluons have a combined total spin of \(S_z = 0\) along the proton beam, while \(G_\Box \) refers to the case with \(S_z = 2\). The full expressions for \(F_\Box \) and \(G_\Box \) within the SM can be found in the appendix of Ref.  [20].

Fig. 1
figure 1

The Feynman diagrams contributing to Higgs boson pair-production in a model with an extended Higgs sector, but no additional particle content besides the SM

Full size image

The Higgs-mediated triangle loop diagram contributes only to the \(S_z = 0\) case. The corresponding form factor, for state \(H_l\) attached to the triangle, is written as

$$\begin{aligned} F^l_\vartriangle = \frac{\alpha _s {\hat{s}}}{4 \pi v} \Bigl \{ S_l^g + i \lambda P_l^g\Bigr \}\,, \end{aligned}$$
(4)

where the scalar and pseudoscalar components, \(S_l^g\) and \(P_l^g\), respectively, can be found in, e.g., Refs.  [21, 22]. In case of a single Higgs boson, as in the SM, the triangle coefficient in Eq. (2) is given as

$$\begin{aligned} C_\vartriangle = \frac{M_Z^2}{{\hat{s}} - M_h^2}~\lambda _{hhh}, \end{aligned}$$
(5)

where \(\lambda _{hhh}\) is the Higgs trilinear self-coupling. In multi-Higgs models like the ones we intend to explore here, the above coefficient is generalised to

$$\begin{aligned} C^l_\vartriangle \equiv \displaystyle \sum _{k=1}^N {{\mathcal {D}}}_{kl}({\hat{s}})\lambda _{H_iH_jH_k}\,. \end{aligned}$$
(6)

Here, \(\lambda _{H_iH_jH_k}\) are the Higgs trilinear couplings and \({{\mathcal {D}}}_{kl}({\hat{s}})\), with \(k,l=1,...,N\), are the entries of the Higgs propagator matrix. This modified \(C^l_\vartriangle \) allows the possibility of interference between two different Higgs intermediate states, induced by higher order quantum effects, as illustrated by Fig. 1 (left).

The main focus of this study is to investigate the above-mentioned quantum effects in the specific scenario with \(N=3\), which permits the resonant pair-production of the lightest Higgs state via the two, mutually interfering, heavier states. In this scenario, the (symmetric) propagator matrix is written as

$$\begin{aligned} {{\mathcal {D}}}({{\hat{s}}}) = {\hat{s}} \left( \begin{array}{@{}ccc@{}} {{\hat{s}}}-m_{H_1}^2 +i{{\mathfrak {I}}\mathrm{m}{\hat{\Pi }}}_{11}({\hat{s}}) &{} i{{\mathfrak {I}}\mathrm{m}{\hat{\Pi }}}_{12}({\hat{s}}) &{} i{{\mathfrak {I}}\mathrm{m}{\hat{\Pi }}}_{13}({\hat{s}})\\ i{{\mathfrak {I}}\mathrm{m}{\hat{\Pi }}}_{21}({\hat{s}}) &{} {{\hat{s}}}-m_{H_2}^2 +i{{\mathfrak {I}}\mathrm{m}{\hat{\Pi }}}_{22}({\hat{s}}) &{} i{{\mathfrak {I}}\mathrm{m}{\hat{\Pi }}}_{23}({\hat{s}}) \\ i{{\mathfrak {I}}\mathrm{m}{\hat{\Pi }}}_{31}({\hat{s}}) &{} i{{\mathfrak {I}}\mathrm{m}{\hat{\Pi }}}_{32}({\hat{s}}) &{} {{\hat{s}}}-m_{H_3}^2 +i{{\mathfrak {I}}\mathrm{m}{\hat{\Pi }}}_{33}({\hat{s}})\\ \end{array}\right) ^{-1}, \end{aligned}$$
(7)

where the \({{\mathfrak {I}}\mathrm{m}\hat{\Pi }}_{kl}(\hat{s})\) are the absorptive parts of the Higgs self-energies, and \(m_{H_k}\) is the renormalised mass of the k-th Higgs boson. The explicit expressions for \({{\mathfrak {I}}\mathrm{m}{\hat{\Pi }}}_{kl}({\hat{s}})\) can be found in the Appendix of Ref.  [18]. In general, however, the off-diagonal absorptive terms in the propagator are assumed to be negligible, in which case the \({{\mathcal {D}}}({{\hat{s}}})\) becomes a diagonal matrix and \(C_\vartriangle \) can, to a good approximation, be reduced to a sum over three terms containing BW propagators corresponding to each \(H_l\), as in Eq. (5).

3 Models and computational tools

Two new physics models that are consistent with the \(N=3\) scenario are the NMSSM and N2HDM. In both these models, we identified the lightest of the three scalars, \(H_1\), with \(h_{\mathrm{obs}}\), and analysed the impact of the interference between the heavier \(H_2\) and \(H_3\) on its pair-production at the \(\sqrt{s}=14\) TeV LHC. It has previously been emphasised in literature [11, 18, 23, 24] that these effects are more pronounced for large (combined) total widths of the intermediate Higgs states compared to the mass splitting between these. Using this as a guideline, we numerically scanned the parameter spaces of the two models to find their potentially relevant configurations. Below we discuss some details of the two models as well as of our calculation of the \(H_1H_1\) production cross section.

3.1 NMSSM

As a follow-up of our previous analyses, the first model that we investigate is the \(Z_3\)-symmetric NMSSM. The Higgs potential in this model is written in terms of the two \(SU(2)_L\) doublets \(H_u\) and \(H_d\), with \(Y = \pm 1\), and the singlet S as

$$\begin{aligned} V_{\mathrm{NMSSM}}= & {} {| \lambda \left( H_u^+ H_d^- - H^0_u H^0_d \right) + \kappa S^2 |}^2 + m_S^2 {| S |}^2 \nonumber \\&+ \left( m_{H_u}^2 + {| \lambda S |}^2 \right) \left( {| H^0_u |}^2 + {| H_u^+ |}^2 \right) \nonumber \\&+ \left( m_{H^0_d}^2 + {|\lambda S |}^2 \right) \left( {| H^0_d |}^2 + {| H_d^- |}^2 \right) \nonumber \\&+ \frac{g_1^2+g_2^2}{8} \left( {| H^0_u |}^2 + {| H_u^+ |}^2 - {| H^0_d |}^2 - {| H_d^- |}^2 \right) ^2 \nonumber \\&+ \frac{g_2^2}{2} {| H_u^+ H_d^{0*} + H^0_u H_d^{-*} |}^2 \nonumber \\&+ \left[ \lambda A_{\lambda } \left( H_u^+ H_d^- - H^0_u H^0_d \right) S + \frac{1}{3} \kappa A_{\kappa } S^3 + \mathrm{h.c.} \right] . \end{aligned}$$
(8)

Here \(\lambda \) and \(\kappa \) are dimensionless Higgs trilinear couplings and \(A_{\lambda }\) and \(A_{\kappa }\) are their respective soft SUSY-breaking counterparts, \(m_{H_d}\), \(m_{H_u}\) and \(m_S\) are the soft Higgs masses, while \(g_1\) and \(g_2\) are the \(U(1)_Y\) and \(SU(2)_L\) gauge coupling constants, respectively.

The neutral components of the fields \(H_d\), \(H_u\) and S are developed around their respective Vacuum Expectation Values (VEVs) \(v_d\), \(v_u\) and \(v_S\), when EW symmetry is broken, as

$$\begin{aligned} H^0_d= & {} \left( \begin{array}{c} v_d+H_{dR}+iH_{dI} \\ H_d^- \end{array} \right) ,~~~ \nonumber \\ H^0_u= & {} \left( \begin{array}{c} H_u^+ \\ v_u+H_{uR}+iH_{uI} \end{array} \right) ,~~~ S = v_S+S_R+iS_I\,. \end{aligned}$$
(9)

By taking the second derivative of \(V_{\mathrm{NMSSM}}\), one then obtains the tree-level \(3 \times 3\) neutral CP-even Higgs mass-squared matrix, \(M_H^2\), in the \((H_{dR},H_{uR},S_R)^T\) basis. The orthogonal matrix \({{\mathcal {R}}}\) rotates these interaction eigenstates into the physical states as

$$\begin{aligned} \left( H_1,H_2,H_3 \right) ^T={{\mathcal {R}}} \left( H_{dR},H_{uR},S_R \right) ^T\,. \end{aligned}$$
(10)

The matrix \(M_H^2\) thus gets diagonalised as

$$\begin{aligned} {{\mathcal {R}}}M_H^2{{\mathcal {R}}}^T=\mathrm{diag}\left( m_{H_1}^2,m_{H_2}^2,m_{H_3}^2\right) , \end{aligned}$$
(11)

with Higgs boson masses in the ascending order, i.e., \(m_{H_1}<m_{H_2}<m_{H_3}\).

Our current analysis pertains to the ‘phenomenological’ NMSSM, wherein all the free parameters, including the above Higgs sector ones, are input at the EW scale. Since variations in non-Higgs sector parameters are expected to have little impact on our particular phenomenological scenario, we fixed the soft squark masses as \(M_{Q_{1,2,3}} = M_{U_{1,2,3}} = M_{D_{1,2,3}} = 3\) TeV, the slepton masses as \(M_{L_{1,2,3}} = M_{E_{1,2,3}}= 2\) TeV, the soft gaugino masses as \(2M_1 = M_2 = \frac{1}{3}M_3 = 1\) TeV. This resulted in \(\tan \beta \) \((\equiv \frac{v_u}{v_d})\), \(\mu _{\mathrm{eff}}\) \((\equiv \lambda v_s)\), \(\lambda \), \(\kappa \), \(m_P\), \(m_A\), and the unified trilinear coupling of the charged sfermions, \(A_0 \equiv A_{\tilde{u},\tilde{c},\tilde{t}} = A_{\tilde{d},\tilde{s},\tilde{b}} = A_{\tilde{e},{\tilde{\mu }},{\tilde{\tau }}}\), as the complete set of inputs. The parameters \(m_P\) and \(m_A\) are the bare masses of the two pseudoscalars, which are a trade-off for \(A_\lambda \) and \(A_\kappa \) using the minimisation conditions of the Higgs potential.

Table 1 Ranges of the NMSSM input parameters scanned for obtaining \(H_2\) and \(H_3\) with large mass-degeneracy. The third column shows the parameter space yielding \(m_{H_{2,3}}\le 500\) GeV
Full size table

We used the public code NMSSMTools-v5.5.2 [25,26,27] for numerically generating the Higgs boson mass spectra and branching ratios (BRs) corresponding to each set of values of the seven model input parameters, randomly selected from the ranges shown in the second column of Table 1. Each parameter space point was required to satisfy all the theoretical and experimental constraints defined in NMSSMTools, which include limits from the Higgs searches at the Large Electron–Positron (LEP) collider, the TeVatron and the LHC, from the direct and indirect searches for neutralino Dark Matter (DM) and estimates of its relic abundance and from B-physics measurements. In our scenario, since the \(H_1\) plays the role of the \(h_{\mathrm{obs}}\), NMSSMTools intrinsically imposes 2\(\sigma \) bounds on its couplings from the most relevant recent LHC results, while also requiring \(m_{H_1}\) to lie within the \(122 - 128\) GeV range, allowing a \(\pm 3\) GeV theoretical uncertainty on the measured mass of \(\sim 125\) GeV. Output points satisfying all these constraints were further run through HiggsBounds-v5.7.0 [28,29,30,31,32] to test the Higgs sector observables of the model against the latest exclusion bounds from the LHC that might not (yet) have been included in NMSSTools itself.

In Fig. 2 we show the resulting successful points with \(m_{H_{2,3}}\le 500\) GeV, which are obtained for the input parameter ranges given in the third column of Table 1. One notices that the limits from the direct searches at the LHC rule out a mass below \(\sim 405\) GeV for the (predominantly doublet-like) \(H_3\), over the entire parameter space explored. Our initial scan with wide input ranges of the parameters yielded only one point (out of nearly two thousand violet points in the figure) with \(\Delta m_H \equiv m_{H_3}-m_{H_2}\) less than 5 GeV, lying just above the LHC exclusion bound for \(m_{H_3}\). In order to find solutions with larger \(H_2\text {-}H_3\) mass-degeneracy, we therefore performed another scan of the narrowed-down parameter space region around the said point. Indeed, several points with \(\Delta m_H <1\) GeV were obtained with this secondary scan, which are plotted in blue colour in the figure. The coordinates of the point with the smallest \(\Delta m_H\) are

$$\begin{aligned} \tan \beta= & {} 6,~~A_0=-5000\,\mathrm{GeV},~~\lambda =0.005,~~\kappa =0.0071, \nonumber \\ \mu _{\mathrm{eff}}= & {} 148.24\,\mathrm{GeV},~~m_P=147.59\,\mathrm{GeV},~~\nonumber \\ m_A= & {} 431.25\,\mathrm{GeV}, \end{aligned}$$
(12)

which result in the following Higgs mass spectrum:

$$\begin{aligned} m_{H_1}= & {} 122.23\,\mathrm{GeV},~~m_{H_2}=409.33\,\mathrm{GeV},~~\nonumber \\ m_{H_3}= & {} 410.13\,\mathrm{GeV},\nonumber \\ m_{A_1}= & {} 147.59\,\mathrm{GeV},~~m_{A_2}=408.23\,\mathrm{GeV},~~\nonumber \\ m_{H^\pm }= & {} 416.13\,\mathrm{GeV}. \end{aligned}$$
(13)

The total widths of the three scalars yielded by the above parameter space point are \(\Gamma _{H_1}=4.76\) MeV, \(\Gamma _{H_2}=535.4\) MeV and \(\Gamma _{H_3}=24.78\) MeV. \(H_2\) in this point is doublet-like, while \(H_3\) is singlet-like. For this reason, the latter has much weaker couplings to the SM, and hence much smaller partial decay widths, than the former. We also point out that for many points obtained in the initial wider scan, \(m_{H_1}\) easily reaches up to 125 GeV. Its lying close to the enforced lower limit of 122 GeV for the point in Eq. (13) is simply a consequence of the very narrow parameter space scanned to obtain maximally degenerate \(m_{H_2}\) and \(m_{H_3}\), especially with the soft squark and gaugino masses fixed. For this point, the \(A_0\) parameter, larger magnitudes of which were generally preferred by the points in order to push \(m_{H_1}\) above 122 GeV, sits right at the upper end of its scanned range.

Fig. 2
figure 2

The masses of \(H_2\) and \(H_3\) for the points obtained in the scans of the extended parameter space of the NMSSM (violet) and of its narrow region yielding \(\Delta m_H<5\) GeV (blue)

Full size image

3.2 N2HDM

Since the scans for the NMSSM did not generate any points with a \(H_3\) lighter than \(\sim 405\) GeV, we extended our analysis to the N2HDM also. In this model the physical masses of the three scalar Higgs bosons, \(m_{H_i}\), are input parameters, as opposed to the NMSSM, wherein they are derived quantities, which allows greater freedom in the selection of the other free parameters relevant to the process under investigation. The N2HDM is obtained by adding a real singlet scalar field, S, to the (CP-conserving) 2HDM, and its Higgs potential reads

$$\begin{aligned} V_{\mathrm{N2HDM}}= & {} m_{H_u}^2|H_u|^2+m_{H_d}^2|H_d|^2-m_{12}^2\left( H_u^\dagger H_d+\mathrm{h.c.}\right) \nonumber \\&+\frac{\lambda _1}{2}\left( H_u^\dagger H_u\right) ^2+\frac{\lambda _2}{2}\left( H_d^\dagger H_d\right) ^2 \nonumber \\&+\lambda _3\left( H_u^\dagger H_u\right) \left( H_d^\dagger H_d\right) \nonumber \\&+\lambda _4\left( H_u^\dagger H_d\right) \left( H_d^\dagger H_u\right) +\frac{\lambda _5}{2}\left\{ \left( H_u^\dagger H_d\right) ^2+\mathrm{h.c.}\right\} \nonumber \\&+\frac{m_S^2}{2}S^2+\frac{\lambda _6}{8}S^4+\frac{\lambda _7}{2}\left( H_u^\dagger H_u\right) S^2\nonumber \\&+\frac{\lambda _S}{2}\left( H_d^\dagger H_d\right) S^2, \end{aligned}$$
(14)

where \(H_u\) and \(H_d\) are doublet fields similar to the NMSSM ones. This potential has a generic form and observes two symmetries: i) a \(Z_2\)-symmetry, \(H_u \rightarrow H_u,\, H_d \rightarrow - H_d,\,S \rightarrow S\), which is softly broken by the term containing \(m_{12}^2\), and ii) a spontaneously broken \(Z_2^\prime \)-symmetry, \(H_u \rightarrow H_u,\, H_d \rightarrow H_d,\, S \rightarrow -S\).

The charge assignments of the fermions under the \(Z_2\) symmetry define the four types of the underlying (N)2HDM. Our adopted notation for the doublet Higgs fields is intended to indicate the Type-II N2HDM specifically, wherein the fermions have \(Z_2\) charges such that the doublet \(H_u\) couples only to the up-type quarks and \(H_d\) to the down-type quarks and charged leptons. Upon EWSB, the two doublet fields are expanded around their respective VEVs according to Eq. (9), while the real singlet is expanded in this model as \(S = v_S+S_R\). After minimisation of the potential and rotation of the scalar mass matrix, as in the NMSSM, the masses of the three physical CP-even Higgs states are obtained, with \(m_{H_1}<m_{H_2}<m_{H_3}\). Besides these, the Higgs sector of the model also contains a CP-odd Higgs boson A. The Type-II N2HDM is thus essentially the non-Supersymmetic counterpart of the NMSSM, with fewer symmetries impinging on the properties of the CP-even Higgs sector, which makes it a more suitable fit for our comparative investigation than the other N2HDM types. For details of the Higgs sector of the N2HDM, we refer the reader to Refs. [33, 34].

There are twelve free parameters in the potential in Eq. (14): \(\lambda _{1,\cdots ,~7, S},~ m_{H_u}^2,~m_{H_d}^2,~m_{S}^2,~m_{12}^2\). Relations between these parameters and the VEVs, arising from the minimisation conditions of the Higgs potential, allow us to trade \(m_{H_u}^2,~m_{H_d}^2\), and \(m_{S}^2\) for \(\tan \beta ,~v \left( \equiv \sqrt{v_u^2+v_d^2}\right) \) and \(~v_S\). Moreover, the eight quartic couplings can be traded for the physical masses, \(m_{H_{1,2,3}},~m_{H^\pm },~m_A\), and the three independent parameters of the mixing matrix \(\mathcal {R}\) in Eq. (10). These parameters, taken to be \(\mathcal {R}_{11},~\mathcal {R}_{12}\) and \(\mathcal {R}_{23}\), can then further be replaced by the top-Yukawa and gauge couplings of the \(H_1\), defined in units of the corresponding couplings of the Higgs boson in the SM as

$$\begin{aligned} g_{H_1tt}=\frac{{{\mathcal {R}}}_{12}}{\sin \beta },~~~~~~g_{H_1VV}=\cos \beta ~{{\mathcal {R}}}_{11}+\sin \beta ~{{\mathcal {R}}}_{12}. \end{aligned}$$
(15)

Thus, for the purpose of this study, the following independent real parameters representing the N2HDM were randomly scanned in the given ranges using the public tool ScannerS-2 [35, 36]:

$$\begin{aligned}&m_A:500 \text {-} 1000\,\mathrm{GeV},~~m_{H^\pm }:500\text {-}1000\,\mathrm{GeV},~~\nonumber \\&m_{12}^2:10^4\text {-}10^5\,\mathrm{GeV}^2,~~\tan \beta :1\text {-}20, \nonumber \\&g_{H_1VV}^2,\,g_{H_1t{{\bar{t}}}}^2:0.64\text {-}1.44,~~\mathrm{sign}({{\mathcal {R}}}_{13}):\pm ,~~\nonumber \\&{{\mathcal {R}}}_{23}:-1\text {-}1,~~v_S:1500\text {-}2500\,\mathrm{GeV}, \end{aligned}$$
(16)

where sign\(({{\mathcal {R}}}_{13})\) takes into account the sign ambiguity in the neutral scalar mixing. In this model, the Higgs trilinear couplings \(g_{H_2H_1H_1}\) and \(g_{H_3H_1H_1}\), which are of particular relevance for the process of our interest here, are given as

$$\begin{aligned} g_{H_jH_iH_i}= & {} \frac{3}{v}\left[ -\frac{1}{2}{\tilde{\mu }}^2\left( \frac{{{\mathcal {R}}}_{i2}}{\sin \beta }-\frac{{{\mathcal {R}}}_{i1}}{\cos \beta }\right) \right. \nonumber \\&\times \left( 6{{\mathcal {R}}}_{i2}{{\mathcal {R}}}_{j2}+6{{\mathcal {R}}}_{i3}{{\mathcal {R}}}_{j3}\sin ^2\beta \right. \nonumber \\&\left. +\sum _k \epsilon _{ijk}{{\mathcal {R}}}_{k3}\sin 2\beta \right) \nonumber \\&+ \left. \frac{2m_{H_i}^2+m_{H_j}^2}{v_S}\left( {{\mathcal {R}}}_{i3}^2{{\mathcal {R}}}_{j3}v+{{\mathcal {R}}}_{i2}^2{{\mathcal {R}}}_{j2}\frac{v_S}{\sin \beta }\right. \right. \nonumber \\&\left. \left. +{{\mathcal {R}}}_{i1}^2{{\mathcal {R}}}_{j1}\frac{v_S}{\cos \beta }\right) \right] , \end{aligned}$$
(17)

where \(\tilde{\mu }^2\equiv \frac{m_{12}^2}{\sin \beta \cos \beta }\) and \(\epsilon _{ijk}\) is the totally antisymmetric tensor, with \(\epsilon _{123}=1\).

While the above ranges were mostly guided by existing literature on the model (see, e.g., Refs. [34, 37, 38]), the ones of \(g^2_{H_1VV}\) and \(g^2_{H_1t{{\bar{t}}}}\) were based loosely on the current 2\(\sigma \) error-bar on the measurements of the corresponding couplings for the \(h_{\mathrm{obs}}\) at the LHC [39]. Several scans were performed for this model, in all of which we fixed \(v=246\) GeV and \(m_{H_1} =125\) GeV. The values of \(m_{H_{2,3}}\), in contrast, were set to certain different values of interest in different scans (as will be explained in the next section). The purpose of the numerical scanning was to find configurations of the parameters in Eq. (16) that satisfied theoretical conditions such as unitarity and vacuum stability, and were at the same time consistent with precision EW and B-physics measurements. In addition to these checks performed internally by ScannerS, testing of the Higgs sector observables against the exclusion bounds from direct collider searches was also performed for each scanned point, by interfacing it with N2HDECAY [40] and HiggsBounds. Finally, ScannerS was also interfaced with the program HiggsSignals-2 [41, 42], which performs a \(\chi ^2\)-fit of the \(h_{\mathrm{obs}}\) properties for a given model point against the LHC measurements, and rules it out if \(\Delta \chi ^2=\chi ^2_{\mathrm{N2HDM}}-\chi ^2_{\mathrm{SM}}>6.18\) (assuming a \(2\sigma \) Gaussian error on the best-fit value).

3.3 Cross section calculation

Fig. 3
figure 3

Left – Cross sections corresponding to the cases a (top half) and b (bottom half) for the scanned NMSSM points, with those shown in blue and red in the respective halves being the ones with \(\Delta m_H<5\) GeV. Right – The ratio of the cross sections b and c as a function of the ratio of the sum of the widths of \(H_2\) and \(H_3\) and their mass difference, with the colour map showing the width of \(H_2\), for the points with \(\Delta m_H<5\) GeV

Full size image

For the output points from the scans, we proceeded to calculate the inclusive \(pp \rightarrow H_1 H_1\) cross section, using a FORTRAN code prepared in-house. For evaluating \(\sigma _{\mathrm{LO}}\) given in Eq. (1), the expressions corresponding to the triangle and box form factors were formulated following the public code HPAIR-v2.00 [20, 43, 44], which includes only the SM and the MSSM. The numerical computation of the next-to-LO (NLO) corrections to \(\sigma _{\mathrm{LO}}\), which can be expressed as [43]

$$\begin{aligned} \Delta \sigma = \Delta \sigma _\text {virt} + \Delta \sigma _{gg} + \Delta \sigma _{gq} + \Delta \sigma _{{\bar{q}}q}\,, \end{aligned}$$
(18)

were also imported from HPAIR, since they are generic to all models. Besides catering to models beyond the MSSM, another significant way that our cross section calculator differs from HPAIR, which evaluates individual BW propagators for each intermediate Higgs boson in the triangle diagram, is in the incorporation of the full propagator matrix of Eq. (7). This allows us to estimate the magnitude of the effects resulting from the off-diagonal terms in the matrix, by including or neglecting these during the cross section computation for a given point by our code.

Since the input parameters as well as the particle contents, and hence the Higgs self-energy contributions, of the NMSSM and N2HDM are mutually rather different, we prepared a separate code for each of these models. In order to check the accuracy and consistency of our base code, we compared the \(pp \rightarrow H_1H_1\) cross sections calculated in the MSSM limit of the NMSSM for a few test points with the ones obtained from HPAIR. We found the two sets of results to be in very good (within 1%) agreement. Note here that the higher order QCD corrections for this process have now been evaluated up to the next-to-next-to-next-to-LO  [45,46,47,48,49,50] in the SM. We presume that these can be extended straightforwardly to the multi-Higgs models discussed here, and their overall impact would amount to a simple rescaling of our NLO calculations.

4 Analysis results

To quantify the magnitude of the triangle-box interference arising in the \(S_z=0\) channel and, additionally, the full-propagator effects within the triangle diagram, we calculated the integrated cross sections corresponding to the following cases for each successful point from the scans for the two models:

  1. (a)

    without triangle-box interference, with diagonal-only propagator matrix,

  2. (b)

    with triangle-box interference, with diagonal-only propagator matrix,

  3. (c)

    with triangle-box interference, with full propagator matrix.

    Below, these three cross sections will be referred to as \(\sigma _{\mathrm{a}}\), \(\sigma _{\mathrm{b}}\), and \(\sigma _{\mathrm{c}}\), respectively. We also define \(R_\sigma \equiv \sigma _{\mathrm{b}}/\sigma _{\mathrm{c}}\).

4.1 The NMSSM

The top half of Fig. 3 (left) shows the cross sections \(\sigma _{\mathrm{a}}\) (top half) and \(\sigma _{\mathrm{b}}\) (bottom half) with diagonal-only propagator matrix, as functions of the \(H_3\) mass. One sees a large negative impact of the triangle-box interference, reducing the cross section uniformly by \(\sim 35\) fb for all the points. We note here that, in models with Supersymmetry, the box and triangle diagrams in principle include loops from squarks also. Here we take the view that the squarks are always too heavy to contribute significantly to either of these production processes (recall that we fixed the soft squark masses to 3 TeV in our parameter space scans, to prevent the physical sparticle masses from conflicting with the direct search results from the LHC), and thus retain only the quark loops. A detailed study of the impact of the inclusion of squarks in the MSSM and the NMSSM (without the Higgs propagator interference effects) can be found in Refs. [51,52,53]. The small blue and red islands near the lowest allowed \(m_{H_3}\) and with overall largest cross sections in the top and bottom halves, respectively, are the points with \(\Delta m_H < 5\) GeV obtained from the secondary scan.

In the numerical calculation of the propagator matrix, in contrast, the (one-loop) Higgs self-energies due to all the relevant NMSSM particles were included. The right panel of Fig. 3, however, shows negligible impact of introducing the full propagator matrix. This figure, restricted only to the points with \(\Delta m_H <5\) GeV, shows \(R_\sigma \) against the ratio of the sum of \(H_2\) and \(H_3\) widths, \(\sum \Gamma _H\), and \(\Delta m_H\). Note that, for a more accurate picture, the widths used for producing this plot are the higher order ones output by NMSSMTools, rather than the tree-level ones corresponding to the self-energies computed by our cross section code. \(\sum \Gamma _H\) ranges between 535 MeV and 565 MeV for all the points, implying that when \(\Gamma _{H_2}\), depicted by the colour map in the figure, reaches its maximum value, \(\Gamma _{H_3}\) is at its minimum, and vice versa. The fact that the lowest \(\Delta m_H\) obtained is 0.8 GeV, according to Eq. (13), implies that \(\sum \Gamma _H/\Delta m_H\) is always smaller than 1 and hence the above mentioned condition of larger \(\sum \Gamma _H\) than \(\Delta m_H\) for a sizeable enhancement in the propagator effects is never met. Still, one can notice a small gradual increase in \(R_\sigma \), meaning an increasing negative effect of the full propagator, as \(\sum \Gamma _H\) rises with respect to \(\Delta m_H\). This effect is more pronounced for points with \(H_2\) and \(H_3\) widths closer to each other in magnitude, as illustrated by the violet/red points in the top left quadrant of the figure. A larger gap between these two widths, in contrast, generally tends to slightly increase \(\sigma _{\mathrm{c}}\) compared to \(\sigma _{\mathrm{b}}\) (the points in the bottom left quadrant).

Fig. 4
figure 4

\(H_2\) and \(H_3\) couplings to \(t{\bar{t}}\) (left), \(b{\bar{b}}\) (centre) and VV (right) pairs in the NMSSM, with the colour map showing the ratio of the cross sections corresponding to cases b and c for the points with \(\Delta m< 5\) GeV

Full size image

The overall smallness of \(R_\sigma \) in the NMSSM can be attributed partly to the large squark and slepton masses, so that their contribution to the Higgs self-energies is diminished, and partly to the specific Yukawa and gauge coupling combinations of the \(H_2\) and \(H_3\) in the narrow parameter space region yielding large mass degeneracy between these. Fig. 4 shows \(R_\sigma \) as a function of these coupling combinations. One notices in these figures that the colour-mapped points, which correspond to the parameter space region with \(\Delta m_H <5\) GeV, mark the boundaries of the (black) points from the extended scan. Thus, the search results from the LHC, besides directly constraining the mass of the \(H_3\) to lie above \(\sim 405\) GeV, also restrict its top-Yukawa coupling to fairly small values, with either sign. The condition of mass degeneracy with \(H_3\) then also dictates the signs and sizes of the \(H_2\) couplings.

According to the left panel of Fig. 4, while the \(H_2\) and \(H_3\) top-Yukawa couplings can take up three different sign combinations in general, for points with \(\Delta m_H <5\) GeV, the sign of \(g_{H_2 t{{{\bar{t}}}}}\) is always negative, while that of \(g_{H_3 t{{{\bar{t}}}}}\) can be both negative or positive. However, only positive \(g_{H_3 t{{{\bar{t}}}}}\) values appear for large negative values of \(g_{H_2 t{{{\bar{t}}}}}\). As the magnitude of the latter drops, that of the former increases, with \(R_\sigma \) also rising slowly, until both reach equal values (with opposite signs). At that point, the sign of \(g_{H_3 t{{{\bar{t}}}}}\) flips to negative, giving the largest \(R_\sigma \) according to the colour map. A further increase in its magnitude, however, along with a decrease in the size of \(g_{H_2 t{{{\bar{t}}}}}\), leads to a lowering of \(R_\sigma \) again. In short, largest (allowed) values of one of the two top-Yukawa couplings, whether positive or negative, coupled with the smallest value of the other, results in \(\sigma _{\mathrm{c}}>\sigma _{\mathrm{b}}\) and, as the two tend towards each other, \(\sigma _{\mathrm{c}}\) starts to lower towards \(\sigma _{\mathrm{b}}\) and eventually below it.

The central panel of the figure likewise illustrates the impact of the variations in \(g_{H_2 b{{{\bar{b}}}}}\) and \(g_{H_3 b{{{\bar{b}}}}}\) on \(R_\sigma \). Note that the points in the bottom half of this plot correspond to the points in the top half of the left panel, and vice versa. Thus, the sign of the bottom-Yukawa coupling of a given Higgs boson is always opposite to that of its top-Yukawa coupling, so that \(g_{H_2 b{{{\bar{b}}}}}\) is positive only, conversely to \(g_{H_2 t{{{\bar{t}}}}}\). Furthermore, \(R_\sigma \) shows a similar trend with the variation in the sizes of \(g_{H_2 b{{{\bar{b}}}}}\) and \(g_{H_3 b{{{\bar{b}}}}}\) as with the top-Yukawa couplings – the largest (allowed) value of one bottom-Yukawa coupling paired with the smallest value of the other yields \(\sigma _{\mathrm{c}}>\sigma _{\mathrm{b}}\), while \(\sigma _{\mathrm{c}} \le \sigma _{\mathrm{b}}\) results from their comparable magnitudes. The dependence of \(R_\sigma \) on the relative signs and magnitudes of \(g_{H_2 VV}\) and \(g_{H_3 VV}\) follows the behaviour of the top-Yukawa couplings exactly, as seen in the right panel of the Fig. 4. Their allowed values are, however, much smaller than even those of the top-Yukawa couplings, pointing towards the decoupling regime of the (N)MSSM. As for the remaining couplings of the \(H_2\) and \(H_3\), even when the corresponding (s)particles have sufficiently low masses, including \(A_1\) as well as \(\chi _{1,2}^0\) and \(\chi _{1}^\pm \) (which are higgsino-like and thus have masses \(\sim \mu _{\mathrm{eff}}\sim 150\) GeV, see Eq. (13)), their influence on \(R_\sigma \) is too small to merit a discussion here.

4.2 The Type-II N2HDM

Fig. 5
figure 5

Left – Cross sections for the points obtained from the N2HDM parameter space scan with \(m_{H_2}=m_{H_2}=410\) GeV, corresponding to the cases a (blue) and b (red) as functions of \(\Gamma _{H_3}\). Right – The ratio of the cross sections b and c against the sum of the widths of \(H_2\) and \(H_3\) for the points obtained from the five N2HDM scans with different \(m_{H_2}\) and \(m_{H_3}\) configurations

Full size image

As indicated earlier, the Higgs boson masses are input parameters in the N2HDM, which allows us to investigate \(H_2\) and \(H_3\) with exactly equal masses, that can also be much lower than those obtained in the NMSSM. For a direct comparison with the NMSSM though, in our first scan for this model we set \(m_{H_2}=m_{H_3}=410\) GeV, and the \(\sigma _{\mathrm{a}}\) and \(\sigma _{\mathrm{b}}\) for the 100 successful points thus obtained are shown in blue and red, respectively, in the left panel of Fig. 5 against the width of \(H_3\). Contrary to the NMSSM, triangle-box interference does not reduce the cross section uniformly for all the points. While for most of the points \(\sigma _b\) is smaller by a few tens of fb than \(\sigma _a\), the former is larger than the latter by upto 10 fb for a few points. This is owing to the wider ranges of magnitudes as well as sign combinations for the \(H_2\) and \(H_3\) couplings being available in this model, as will be explained later. Notice also that \(\Gamma _{H_2}\) can reach a few GeVs and, in fact, \(\Gamma _{H_3}\) can simultaneously be quite large, as illustrated by the horizontal axis of the right panel of the figure. Once again, \(\Gamma _{H_{2,3}}\) here are the widths output by ScannerS, instead of the tree-level ones that can be obtained from the one-loop self-energies computed by our code. The vertical axis of this panel shows the impact of including the full propagator matrix in the cross section calculation.

For a number of points from the first scan with \(m_{H_2}=m_{H_3}=410\) GeV, seen in red in Fig. 5 (right), \(\sigma _{\mathrm{b}}\) is a few times larger than \(\sigma _{\mathrm{c}}\), but for some points it gets reduced by upto 35%, implying a net positive contribution from the off-diagonal terms in the matrix. For two of the red points, though, \(R_{\sigma }\) exceeds 100, meaning a two orders of magnitude reduction in \(\sigma _{\mathrm{b}}\). (We point out here that these two points were omitted from the left panel to keep the y-axis scale visually interesting, but the corresponding cross sections will be provided below.) To assess the effect of reducing the mass degeneracy, the points from scans with \(\Delta m_H=5\) GeV and \(\Delta m_H=10\) GeV, with \(m_{H_3}\) still fixed to 410 GeV, are also plotted in this figure in orange and green, respectively. Evidently, a larger \(\Delta m_H\) results in smaller fluctuations in \(\sigma _{\mathrm{b}}\), as the \(R_\sigma \) value lies very close to 1 for all the 100 green points. The violet points in the figure correspond to the scan with \(m_{H_2}=m_{H_3}=300\) GeV. While in general \(R_\sigma \) can deviate substantially from 1 for these points also, its maximum value does not exceed 3. The main reason for this is that the widths of \(H_2\) and \(H_3\) are always lower than 1 GeV in this case, unlike the \(m_{H_2}=m_{H_3}=410\) GeV case, owing to the fact that their masses lie below the \(t{\bar{t}}\) production threshold. Lowering \(m_{H_2}\) and \(m_{H_3}\) even further to under the \(H_1H_1\) threshold expectedly results in a vanishing impact of the off-diagonal propagator matrix terms, as demonstrated by the cyan points in the figure, which are all clustered together below \(\sum \Gamma _H \lesssim 200\) MeV.

Table 2 Values of the input parameters, couplings and widths of the Higgs bosons, together with the cross sections corresponding to the six selected BPs of the N2HDM
Full size table

For a detailed investigation, we selected six Benchmark Points (BPs) from our main scan with \(m_{H_2}=m_{H_3}=410\) GeV. The input parameters, the widths and couplings of \(H_2\) and \(H_3\) as well as the four cross sections corresponding to these points are given in Table 2. \(\sigma _{2\times 2}\) in the table implies the cross section obtained by setting \(m_{H_3}\rightarrow \infty \) (in order to decouple the \(H_3\)), with all the other input parameters fixed to their exact values for a given BP, and is quoted for reference. BP1 and BP2 are the two points with the highest \(R_\sigma \) in Fig. 5, for BP3 and BP4 the \(R_\sigma \) value lies very close to 1 while BP5 and BP6 are chosen from amongst the points for which \(\sigma _c\) is slightly enhanced compared to \(\sigma _b\).

In the 2HDM, and the N2HDM by extension, of the type-II, the B-physics measurement strongly constrain \(m_{H^\pm }\) [34, 54] and therefore the latter lies above 600 GeV for all the successful points from the scans, while \(\tan \beta \) is also pushed to smaller values, as can be noted in the table. \(m_A\) is then also restricted to values close to \(m_{H^\pm }\) by the EW precision constraints. One feature distinguishing the points with the largest \(R_\sigma \) (BP1 and BP2) from the rest of the BPs are the larger \(m_{H^\pm }\) and \(g_{H_1 t{\bar{t}}}\) values and relatively small \(\tan \beta \). Such parameter configurations result in specific combinations of the couplings of \(H_2\) and \(H_3\) for BP1 and BP2, which in turn lead to very high \(R_\sigma \) for these. For these two points, \(g_{H_2 t{\bar{t}}}\) and \(g_{H_3 t{\bar{t}}}\) are both positive and large while \(g_{H_2 b{\bar{b}}}\), \(g_{H_3 b{\bar{b}}}\), \(g_{H_2VV}\) and \(g_{H_3VV}\) are all negative. In the case of BP3, \(g_{H_2 t{\bar{t}}}\) and \(g_{H_3 t{\bar{t}}}\) have signs opposite to each other while \(g_{H_2 b{\bar{b}}}\) and \(g_{H_3 b{\bar{b}}}\) are both negative. We note here that, again in contrast with the NMSSM, \(g_{H_2 b{\bar{b}}}\) is negative for all the 100 N2HDM points for the \(m_{H_2}=m_{H_3}=410\) GeV scenario, and also that, for a majority of these points, three out of the four Yukawa couplings had the same signs. BP4 and BP5 are very similar points in that the two top-Yukawa couplings have signs that are opposite not only to each other but also to the signs of the corresponding bottom-Yukawa couplings. BP6 is the only point of its kind found in the scan, with \(g_{H_2 t{\bar{t}}}\), \(g_{H_3 t{\bar{t}}}\) and \(g_{H_2 b{\bar{b}}}\) all having negative signs, and it therefore uniquely exhibits a constructive triangle-box interference as well as constructive propagator interference, so that \(\sigma _{\mathrm{a}}<\sigma _{\mathrm{b}}<\sigma _{\mathrm{c}}\).

Fig. 6
figure 6

Various cross sections as functions of the couplings \(g_{H_2t{{\bar{t}}}}\) (left) and \(g_{H_2b{{\bar{b}}}}\) (right). The point on a line marks the actual value of the plotted coupling for the corresponding BP. See text for more details

Full size image
Fig. 7
figure 7

Various cross sections as functions of the couplings \(g_{H_2VV}\) (left) and \(g_{H_2H_1H_1}\) (right). The point on a line marks the actual value of the plotted coupling for the corresponding BP. See text for more details

Full size image

In Figs. 6 and 7 we show the four cross sections as functions of the most important \(H_2\) couplings in this context, for all the six BPs. The former corresponds to the couplings \(g_{H_2 t{\bar{t}}}\) and \(g_{H_2 b{\bar{b}}}\), and the latter to \(g_{H_2VV}\) and \(g_{H_2H_1H_1}\), while rows 1, 2, 3 and 4 in both the figures depict \(\sigma _{2\times 2}\), \(\sigma _{\mathrm{b}}\), \(\sigma _{\mathrm{c}}\) and \(R_\sigma \), respectively. The plotted ranges of the couplings are indicative of those observed across all the 100 points obtained for the \(m_{H_2}=m_{H_3}=410\) GeV scenario. Once again, for each BP in a given panel, all the remaining couplings are fixed to the values described in Table 2. The red lines in the figure correspond to BP1, green to BP2, olive to BP3, violet to BP4, blue to BP5 and orange to BP6. The point on a line in a given panel marks the actual value of the plotted coupling for that BP. The horizontal black lines in the panels in columns 1, 2 and 3, indicate the current experimental limit on \(h_{\mathrm{obs}}\) pair-production cross section  [55], which we approximate to be 1 pb for \(m_{H_2} =m_{H_3}=410\) GeV considered here. Note also that, since only the product of the corresponding \(H_2\) and \(H_3\) couplings enters the \(i{{\mathfrak {I}}\mathrm{m}{\hat{\Pi }}}_{23}({\hat{s}})\) element of the Higgs propagator matrix, the behaviour of \(\sigma _{\mathrm{c}}\) with varying \(H_3\) couplings should by and large mimic that with varying \(H_2\) couplings.

In the left column of the Fig. 6, one sees that the presence of an additional Higgs boson degenerate in mass with \(H_2\) eliminates the peaks appearing at specific values of \(g_{H_2 t{\bar{t}}}\) in \(\sigma _{2\times 2}\), so that the variations in \(\sigma _{\mathrm{b}}\) in the second row are smoother than in the first row. As expected, \(\sigma _{\mathrm{b}}\) shows a very similar behaviour for BP1 and BP2, reaching values even higher than the true ones for slightly different positive \(g_{H_2 t{\bar{t}}}\) (recall that \(g_{H_3 t{\bar{t}}}\) is also positive for these two points). Thus \(g_{H_2 t{\bar{t}}} > rsim 0.1\) would be ruled out by the LHC \(h_{\mathrm{obs}}\) \(h_{\mathrm{obs}}\)-production limits. When \(g_{H_2 t{\bar{t}}}\) switches sign to negative, \(\sigma _{\mathrm{b}}\) drops to much smaller values. The introduction of the full propagator matrix then largely mitigates the very strong dependence of the cross section on positive \(g_{H_3 t{\bar{t}}}\), as seen in the third row, bringing it down to values consistent with experimental bounds. And since \(\sigma _{\mathrm{c}}\) shows little variation with \(g_{H_2 t{\bar{t}}}\), the shapes of the red and green lines in the bottom row of this column (and also of the right column) are very similar to those in row 2, with \(R_\sigma \) reaching about 900 for the BP1.

Cross sections for BP3 and BP4, both of which have \(g_{H_3 t{\bar{t}}}\) with mutually opposite signs but very similar magnitudes, show similar trends to each other with the variations in \(g_{H_2 t{\bar{t}}}\) across the four panels on the left. For these two points, the peaks in \(\sigma _{2\times 2}\) are the tallest, while \(\sigma _{\mathrm{b}}\) and even \(\sigma _{\mathrm{c}}\) violates the experimental bound for large negative \(g_{H_2 t{\bar{t}}}\). BP4 and BP5, likewise mimic each other’s behaviour for positive \(g_{H_2 t{\bar{t}}}\), but since BP4 has a negative \(g_{H_3 t{\bar{t}}}\) larger in magnitude than that in BP5, its dependence on negative \(g_{H_2 t{\bar{t}}}\) is much more pronounced for both \(\sigma _{\mathrm{b}}\) and \(\sigma _{\mathrm{c}}\). The right column of the figure shows negligible dependence of \(\sigma _b\) on \(g_{H_2 b{\bar{b}}}\) for all the BPs expect 1 and 2 and, conversely, the least variation in \(\sigma _c\) for these two BPs. This is due to the fact that for these points \(g_{H_2 b{\bar{b}}}\) and \(g_{H_3 b{\bar{b}}}\) both have negative signs, opposite to the signs of the two top-Yukawa couplings which have a much more dominant effect.

The left column of Fig. 7 illustrates that the \(g_{H_2VV}\) coupling plays a role as crucial as the top-Yukawa couplings. Similarly to the NMSSM, these couplings originally have generally quite small magnitudes, as a consequence of the very SM-like properties of the \(H_1\). For this coupling, the two peaks appearing in \(\sigma _{2\times 2}\) are replaced by a tall narrow peak in \(\sigma _{\mathrm{b}}\), close to \(g_{H_2VV}=0\). The introduction of the full propagator matrix brings even the highest of all the peak values of \(\sigma _{\mathrm{b}}\), seen for BP1 and BP2, down to an experimentally acceptable sub-pb level. The shapes of all the lines are hence largely dictated by the interplay between the signs and sizes of the top-Yukawa and gauge couplings of \(H_2\) and \(H_3\). In the right column is depicted the dependence of the cross sections on \(g_{H_2 H_1H_1}\), which is the only coupling of significance other than the ones discussed above. Here, \(\sigma _{2\times 2}\) shows a sharp dip at the zero of this coupling, since it also enters the \(H_2\rightarrow H_1H_1\) decay besides the self-energies. This sharp dip shifts away from zero for \(\sigma _{\mathrm{b}}\), according to the relative sign of the diagonal \(H_3\) contributions to the propagator. It returns to zero when the off-diagonal terms are also turned on. Around the minimum, \(\sigma _{\mathrm{c}}\) shows a fairly symmmetric behaviour in both signs of \(g_{H_2 H_1H_1}\), as do \(\sigma _{\mathrm{b}}\) and \(\sigma _{\mathrm{b}}\). Unlike these two cross sections, however, \(\sigma _{\mathrm{c}}\) increases rather smoothly.

Finally, a negligible dependence of each of the cross sections on all of the remaining couplings given in Table 2 was noted, since the corresponding particle pairs are rather heavy. The contribution to the Higgs self-energies from even the relatively lighter pairs, such as AZ and \(H^+ W^-\), for any value of the coupling is vanishing. Plots illustrating variations in the rest of the couplings can therefore be safely dropped.

5 Conclusions

The commonly adopted approach of calculating the cross section for a given \(2\rightarrow 2\) process by factorising it into its production and decay parts cannot account for possible quantum interference amongst the propagators of several mass-degenerate states. This is because such an approach by construction assumes narrow widths for the resonant mediators. In some previous papers we explored such interference effects in the case of the gluon-fusion production of certain SM final states, via two highly mass-degenerate Higgs mediators. The mass-splitting between the two intermediate Higgs states being comparable to or smaller than the sum of their widths is a precondition for such effects to be sizeable, in both the integral and the differential cross section. The reason for their onset is that the imaginary off-diagonal elements of the Higgs propagator matrix become comparable to the imaginary parts of the diagonal elements, irrespectively of whether they are taken into account coherently or incoherently. These studies were performed within the illustrative theoretical framework of the NMSSM.

In this article, we have extended our investigation to the pair-production of the lightest of the three NMSSM neutral Higgs scalars, \(H_1\), at the 14 TeV LHC, taking into account the contributions of the triangle as well as the box diagram that this process proceeds through. We have investigated the impact of not only the interference between these two topologies, but also of the aforementioned propagator interference between \(H_2\) and \(H_3\) within the triangle topology on the cross section for \(H_1H_1\) production. Furthermore, since the lowest \(H_3\) mass attainable in the NMSSM is 405 GeV, owing to the bounds from the LHC searches, we have also included the N2HDM in our analysis. In this framework, since the physical Higgs boson masses are input parameters, we could choose any desired (unique) value for \(m_{H_2}\) and \(m_{H_3}\), which allowed us to study also the scenario where they contribute non-resonantly to the triangle topology for the studied process.

In the case of the NMSSM, we have found the effects of the inclusion of the full Higgs propagator matrix in the triangle topology to be similar in size to those established in our previous studies. However, given the various constraints imposed, since the minimal mass-splitting between \(H_2\) and \(H_3\) is obtained in a very narrow region of the parameter space where, however, the sum of their widths never exceeds it, the effect is largely subdued. In this region the box diagram and the triangle diagram with (an off-shell) \(H_1\) in the propagator contribute much more dominantly to the \(H_1\) pair-production process. The narrowness of this region also means that a nearly constant negative interference is always observed between the two topologies.

In the N2HDM, on the other hand, we have seen that the propagator interference effects can modify the cross section by more than two orders of magnitude. Of particular importance is the observation that these effects tend to ‘regulate’ the behavior of the total \(H_1H_1\) cross section, smoothing the peaks that appear in it for certain specific values of the \(H_2\) and \(H_3\) couplings, and generally bringing it down to values consistent with the current LHC limits. Moreover, herein the interference between the box and triangle topologies can be positive or negative, as a consequence of the relatively wider ranges of the magnitudes and sign combinations of the Higgs boson Yukawa couplings. Clearly, such a disparity between the results obtained for this model and those for the NMSSM is due to the fact that supersymmetry imposes strong limitations on the masses and couplings of the heavy Higgs states. In the N2HDM, these quantities are essentially free parameters. But even in this model, when \(m_{H_2}\) and \(m_{H_3}\) lie below the \(H_1H_1\) production threshold, the propagator interference effects tend to vanish. In this case the one-loop two-point functions corresponding to the off-diagonal elements in the Higgs propagator matrix are too small to be able to overcome the kinematic suppression.

Finally, we emphasise that we have reached the above conclusions on the basis of a detailed analysis at the level of the total cross section. As for their phenomenological relevance, the LHC may develop sensitivity to all such interesting dynamics already at its upcoming Run 3, at least in the N2HDM. Hence, for the purpose of aiding experimental efforts in establishing all the effects studied here, we have proposed some BPs, compliant with the latest theoretical and experimental constraints, that are amenable to dedicated probes by the ATLAS and CMS collaborations.