1 Introduction

Already five years have past since the discovery of a new scalar at the LHC [1, 2]. In the meantime, the properties of this particle have been measured with an astonishing precision. All coupling measurements agree very well with the expectations for the standard model (SM) Higgs boson. Thus, this particle is at least a SM-like Higgs. Maybe, it is the Higgs, i.e. the only fundamental scalar which exists in nature and which is involved in the breaking of the electroweak (ew) gauge symmetry. However, it is much too early to draw this conclusion and there is still plenty of space where new Higgs-like particles as predicted by models beyond the SM (BSM) might show up. The motivation to introduce new scalars in BSM models can be quite different and either stem from a top-down approach like a new proposed symmetry, or from a bottom-up approach where new scalars are needed to get some specific feature in the model. One can count supersymmetric models, models with a grand unified theory or models with an extended gauge sector to the first category, models like those with only additional scalar doublets or triplets to the second one. The interesting feature of these models is that they can offer very interesting, phenomenological effects like changes in the couplings of the SM-like Higgs to fermions or vector bosons, charged or even doubly-charged scalars, or signatures of additional light or heavy neutral scalars. Models with additional Higgs triplets are constrained by the \(\rho \)-parameter which relates \(M_W\), \(M_Z\) and the weak mixing angle. This parameter is measured to be very close to one. While additional doublets only contribute loop corrections to this parameter, vacuum expectation values (VEVs) of triplets, \(v_T\), usually enter already at the tree-level. This imposes an upper limit on \(v_T\) of just a few GeV [3, 4]. However, this constraint can be circumvented if several triplets are arranged in such a way that they obey a new (custodial) symmetry: in this case, the tree-level contributions to \(\delta \rho \) cancel and the triplet VEVs can be much larger. This was the original idea proposed in Refs. [5, 6]. The model known today as the ‘Georgi-Machacek’ (GM) model comes with one complex and one real triplet. The impact of loop corrections to the \(\rho \)-parameter in the presence of triplets were considered in literature and potentially large effects were found [7].

Many other aspects of the GM model were studied in detail in the last years like the properties of the couplings of scalars [8,9,10,11,12,13,14,15,16,17,18] or vector bosons [19], or the potential signatures at colliders [20,21,22,23,24,25,26,27,28]. Not only the phenomenological consequences of the new states were explored, but also the theoretical properties of the extended Higgs sector were studied. This was used to impose constraints on the parameters of the model. For instance, limits for the quartic couplings were derived from the unitarity of the tree-level \(2 \rightarrow 2\) scattering [29, 30]. Also the stability of the ew vacuum was checked, and dangerous regions in which the potential is unbounded-from-below (UFB) were singled out [31, 32]. However, all of these constraints have up to now only been studied at lowest order in perturbation theory and it has not been checked whether the conclusions still hold once loop corrections are included. Moreover, it was so far not clear if higher-order corrections could even be implemented in a sensible way or if some regions in parameter space cannot be treated perturbatively. If this were the case, Born-level results would not be good approximations of the ‘full’ result, but might be completely meaningless. This is actually an issue which has so far hardly been discussed in any non-supersymmetric BSM model, which might sound surprising as it was already shown in Ref. [33] for the SM that a naive limit like \(\lambda < 4\pi \) is too weak and that perturbation theory can break down at much smaller coupling values. Among the reasons why this breakdown of perturbation theory has not been checked exhaustively for many BSM models are the missing but necessary higher order results: the only rigorous way to claim a breakdown of perturbation theory is to calculate observables at different loop levels and compare the residual scale dependence which, if a perturbative treatment is possible, must shrink from every order to the next. This would make it necessary to calculate decays or scattering processes at least up to the two-loop level, which would cause a tremendous amount of work. However, some results can be obtained in an easier way which should give more sophisticated hints as to when perturbation theory is in trouble than the simple rules of thumb which say that a quartic coupling must be smaller than some factor times \(\pi \). A first idea in that direction was shown in Ref. [34] where the one- and two-loop corrections to the scalar masses in the \(\overline{\text {MS}}\) scheme were compared. If the two-loop corrections become larger than the one-loop corrections, this already points already towards a problem. Indeed, it has been shown in Ref. [34] at the example of the GM model that the two-loop corrections can be several times as large as the one-loop corrections.

We are going to investigate this potential breakdown of perturbation theory in the GM model in more detail in this work. We propose different conditions which could be used to check for dangerous regions in parameters space. These are not only the size of the one- and two-loop corrections in the \(\overline{\text {MS}}\) scheme, but also the values of the counter-terms (CTs) when performing an on-shell (OS) normalisation of the scalar sector at the one-loop level. Since these CTs will enter at all higher loop levels, they cannot be too large without disturbing the convergence of the loop series. The methodology which we develop to apply perturbativity constraints is not restricted to the GM model, but can be applied similarly to other BSM models.

A second main aim of this work is the promotion of the vacuum stability constraints to the loop level: as we will see, the loop corrections in the GM model are always sizeable. Even if perturbation theory might still be under control, tree-level results can be very misleading. For instance, it was shown in Ref. [35] that UFB directions in two Higgs doublet Models (THDM) usually disappear once loop corrections are included. We will find similar results for the GM model. We will show that many parameter regions which seem to pass all tree-level constraints are often in conflict with the constraints from perturbativity. On the other side, parameter regions which are unstable at tree-level become stable once the loop corrections are included. We will present some indications where these effects are most likely to happen in the parameter space of the GM model.

This work is organised as follows: in Sect. 2, we explain the basic ingredients of the GM model with and without the custodial symmetry, as well as the two different normalisation schemes, \(\overline{\text {MS}}\) and OS, which we use in this paper. In Sect. 3, the existing tree-level constraints on the model are summarised, and our new constraints are explained. In Sect. 4, the impact of our new constraints is discussed. We start with specific examples to discuss the impact of the perturbativity constraints as well as the loop-improved vacuum stability checks before we explore the consequences in the full parameter space. We conclude in Sect. 5. The appendix contains a long collection of additional material: all tree-level couplings of all scalars, the expressions for the one-loop corrections to self-energies and tadpoles as well as the necessary CTs to renormalise the scalar sector of the GM model on-shell.

2 The Georgi-Machacek model

In the GM model, the SM is extended by one real scalar \(SU(2)_L\)-triplet \(\eta \) with hypercharge \(Y=0\) and one complex scalar triplet \(\chi \) with \(Y=-1\) (using the notation \(Q_{\mathrm{em}} = T_{3L}+Y\)). Those can be written as

$$\begin{aligned} \eta= & {} \frac{1}{\sqrt{2}} \, \left( \begin{array}{cc} \eta ^0 &{} - \sqrt{2} \left( \eta ^-\right) ^*\\ - \sqrt{2} \, \eta ^- &{} - \eta ^0 \end{array} \right) , \quad \nonumber \\ \chi= & {} \frac{1}{\sqrt{2}} \, \left( \begin{array}{cc} \chi ^- &{} \sqrt{2} (\chi ^0)^* \\ -\sqrt{2}\, \chi ^{--} &{} -\chi ^- \end{array} \right) . \end{aligned}$$
(2.1)

with \((\eta ^0)^* = \eta ^0\).

2.1 Unbroken custodial symmetry

After imposing a global \(SU(2)_L \times SU(2)_R\) symmetry on the model, the most compact way to write the Lagrangian in a form which explicitly conserves this custodial symmetry is to re-express the Higgs doublet \(\Phi \) as a bi-doublet under \(SU(2)_L \times SU(2)_R\) and the two scalar triplets as a bi-triplet:

$$\begin{aligned} \Phi = \left( \begin{array}{cc} \phi ^{0*} &{} \quad \phi ^+ \\ \phi ^- &{} \quad \phi ^0 \end{array} \right) , \qquad \quad \Delta = \left( \begin{array}{ccc} \chi ^{0*} &{} \quad \eta ^+ &{} \quad \chi ^{++} \\ \chi ^- &{} \quad \eta ^0 &{} \quad \chi ^+ \\ \chi ^{--} &{} \quad \eta ^{-} &{} \quad \chi ^0 \end{array} \right) . \end{aligned}$$
(2.2)

The scalar potential can then be written as

$$\begin{aligned} V(\Phi , \Delta )&= \frac{\mu _2^2}{2} \mathrm{Tr} \Phi ^\dagger \Phi + \frac{\mu _3^2}{2} \mathrm{Tr} \Delta ^\dagger \Delta + \lambda _1 \left[ \mathrm{Tr} \Phi ^\dagger \Phi \right] ^2 \\&\quad + \,\lambda _2 \mathrm{Tr} \Phi ^\dagger \Phi \, \mathrm{Tr} \Delta ^\dagger \Delta \\&\quad + \,\lambda _3 \mathrm{Tr} \Delta ^\dagger \Delta \Delta ^\dagger \Delta + \lambda _4 \left[ \mathrm{Tr} \Delta ^\dagger \Delta \right] ^2 \\&\quad -\, \lambda _5 \mathrm{Tr} \left( \Phi ^\dagger \sigma ^a \Phi \sigma ^b \right) \, \mathrm{Tr} \left( \Delta ^\dagger t^a \Delta t^b \right) \\&\quad - \,M_1 \mathrm{Tr} \left( \Phi ^\dagger \tau ^a \Phi \tau ^b \right) (U \Delta U^\dagger )_{ab}\\&\quad -\, M_2 \mathrm{Tr} \left( \Delta ^\dagger t^a \Delta t^b \right) (U \Delta U^\dagger )_{ab} , \end{aligned}$$

where \(\tau ^a\) and \(t^a\) are the SU(2) generators for the doublet and triplet representations respectively, and U is for instance given in Ref. [30].

After EWSB, the neutral, complex fields \(\phi ^0\) and \(\chi ^0\) split in CP eigenstates,

$$\begin{aligned} X^0 \rightarrow \frac{1}{\sqrt{2}} \left( X_{\mathcal R}^0+i X_{\mathcal I}^0 \right) \end{aligned}$$
(2.3)

with \(X=\{\phi ,\chi \}\). The vacuum expectation values (VEVs) of the triplets read

$$\begin{aligned} \langle \eta \rangle = \frac{1}{\sqrt{2}} \, \left( \begin{array}{cc} v_\eta &{} 0\\ 0 &{} -v_\eta \end{array} \right) , \quad \langle \chi \rangle = \, \left( \begin{array}{cc} 0 &{} v_\chi \\ 0 &{} 0 \end{array} \right) , \end{aligned}$$
(2.4)

while the doublet VEV is \(\langle \phi ^0 \rangle = v_\phi /\sqrt{2}\) as usual. The gauge boson masses then read at tree level

$$\begin{aligned} M_W^2= & {} \frac{1}{4}\,g_2^2\, \left( v_\phi ^2 + 4\left( v_\eta ^2 + v_\chi ^2 \right) \right) ,\qquad \nonumber \\ M_Z^2= & {} \frac{1}{4} \left( g_1^2+g_2^2 \right) \left( v_\phi ^2 + 8 v_\chi ^2 \right) . \end{aligned}$$
(2.5)

If the custodial symmetry is preserved, the triplet VEVs are identical, \(v_\eta =v_\chi \), and there are no tree-level contributions to electroweak precision observables as a consequence. The electroweak VEV v can then be written as

$$\begin{aligned} v^2 = v_\phi ^2 + 8 v_\chi ^2 \ \simeq 246~\text {GeV}, \end{aligned}$$
(2.6)

so that the massive gauge bosons end up with the known SM tree-level masses, and it is common to define the angle

$$\begin{aligned} s_H \equiv \sin \theta _H = 2 \sqrt{2} \, \frac{v_\chi }{v},\qquad c_H \equiv \cos \theta _H = \frac{v_\phi }{v}. \end{aligned}$$
(2.7)

The minimisation conditions for the scalar potential read

$$\begin{aligned} 0 = \frac{\partial V}{\partial v_{\phi }}= & {} v_{\phi } \left( \mu _2^2 + 4 \lambda _1 v_{\phi }^2 + 6 \lambda _2 v_\chi ^2 \right. \nonumber \\&\left. - \, 3 \lambda _5 v_\chi ^2 - \frac{3}{2} M_1 v_\chi \right) , \end{aligned}$$
(2.8)
$$\begin{aligned} 0 = \frac{\partial V}{\partial v_{\chi }}= & {} 3 v_\chi \left( \mu _3^2 + 2 \lambda _2 v_\phi ^2 - \lambda _5 v_\phi ^2 - 6 M_2 v_\chi \right. \nonumber \\&\left. +\,4 \lambda _3 v_\chi ^2 + 12 \lambda _4 v_\chi ^2 \right) -\frac{3}{4} M_1 v_\phi ^2, \end{aligned}$$
(2.9)

which we solve for \(\mu _2^2\) and \(\mu _3^2\) to eliminate these parameters from the scalar potential. Note that we have assumed all parameters to be real, resulting in CP conservation in the scalar sector.

The mass eigenstates of the scalar fields can be divided into singlets, triplets and fiveplets under the custodial symmetry. At tree level, the masses within each triplet and fiveplet are degenerate, and, using the above equations, are given by

$$\begin{aligned} m_5^2= & {} \frac{M_1}{4 v_{\chi }} v_\phi ^2 + 12 M_2 v_{\chi } + \frac{3}{2} \lambda _5 v_{\phi }^2 + 8 \lambda _3 v_{\chi }^2 \nonumber \\= & {} v\left( s_H (3 \sqrt{2} M_2 + s_H \lambda _3 v) + c_H^2 (\frac{M_1}{\sqrt{2} s_H} + \frac{3}{2} \lambda _5 v) \right) ,\nonumber \\ \end{aligned}$$
(2.10)
$$\begin{aligned} m_3^2= & {} \frac{v_\phi ^2 + 8 v_\chi ^2}{4 v_\chi } \left( M_1 + 2 v_\chi \lambda _5 \right) = \frac{v M_1}{\sqrt{2}\, s_H} + \frac{1}{2} \lambda _5 v^2.\nonumber \\ \end{aligned}$$
(2.11)

The mass matrix of the CP-even scalars reads in the basis \((\phi _{\mathcal R}^0,\eta ^0,\chi _{\mathcal R}^0)\)

$$\begin{aligned} m_{h^0}^2 = \begin{pmatrix} 8 c_H^2 \lambda _1 v^2 &{} -\frac{v}{2} c_H {\tilde{M}} &{} -\frac{v}{\sqrt{2}} c_H {\tilde{M}} \\ \cdot &{} \frac{v \left( \sqrt{2} (c_H^2 M_1 + 3 s_H^2 M_2) + 2 s_H v (s_H^2 (\lambda _3+\lambda _4) + c_H^2 \lambda _5)\right) }{2 s_H} &{} \frac{v^2 \left( 2 s_H^2 \lambda _4-c_H^2 \lambda _5 \right) }{\sqrt{2}} - 3 s_H M_2 v \\ \cdot &{} \cdot &{} \frac{v\left( 2 s_H^3 v (\lambda _3+2 \lambda _4) + c_H^2 (\sqrt{2} M_1+s_H \lambda _5 v) \right) }{2 s_H} \end{pmatrix}, \end{aligned}$$
(2.12)

where

$$\begin{aligned} {\tilde{M}} = M_1 + \sqrt{2} s_H v (\lambda _5 - 2 \lambda _2). \end{aligned}$$
(2.13)

After the diagonalisation, one mass eigenstate corresponds to the fiveplet mass, Eq. (2.10), whereas the other two mix to form the mass eigenstates h and H.

They are given in the gauge basis by [30]

$$\begin{aligned} h= & {} \cos \alpha \, H_1^0 - \sin \alpha \, H_1^{0\prime }, \nonumber \\ H= & {} \sin \alpha \, H_1^0 + \cos \alpha \, H_1^{0\prime }, \end{aligned}$$
(2.14)

where

$$\begin{aligned} H_1^0 = \phi _{\mathcal R}^{0}, \qquad H_1^{0 \prime } = \sqrt{\frac{1}{3}}\, \eta ^0 + \sqrt{\frac{2}{3}}\, \chi _{\mathcal R}^{0}. \end{aligned}$$
(2.15)

2.2 Broken custodial symmetry

Without the custodial symmetry, the most general gauge-invariant and CP-conserving form of the Higgs potential is given by [36]

$$\begin{aligned} V&=m_\phi ^2(\phi ^\dagger \phi )+m_\chi ^2\text {Tr}(\chi ^\dagger \chi )+m_\eta ^2\text {Tr}(\eta ^2)\nonumber \\&\quad +\,\mu _1\phi ^\dagger \eta \phi +\mu _2 (\phi ^T(i\tau _2) \chi \phi +\text {h.c.}) \nonumber \\&\quad +\,\mu _3\text {Tr}(\chi \chi ^\dagger \eta )+\lambda (\phi ^\dagger \phi )^2 \nonumber \\&\quad +\rho _1\left( \text {Tr}(\chi ^\dagger \chi )\right) ^2+\rho _2\text {Tr}(\chi ^\dagger \chi \chi ^\dagger \chi ) +\rho _3\text {Tr}(\eta ^4) \nonumber \\&\quad +\rho _4 \text {Tr}(\chi ^\dagger \chi )\text {Tr}(\eta ^2) +\rho _5\text {Tr}(\chi ^\dagger \eta )\text {Tr}(\eta \chi )\nonumber \\&\quad +\,\sigma _1\text {Tr}(\chi ^\dagger \chi )\phi ^\dagger \phi +\sigma _2 \phi ^\dagger \chi ^\dagger \chi \phi +\sigma _3\text {Tr}(\eta ^2)\phi ^\dagger \phi \nonumber \\&\quad +\,\sigma _4 (\phi ^\dagger \chi ^\dagger \eta \phi ^c + \text {h.c.}). \end{aligned}$$
(2.16)

Note that the above equation differs from Ref. [36] in \(\chi \leftrightarrow \chi ^\dagger \) because of the differing definitions of \(\chi \). Here, \(\phi ^c = i\tau \phi ^*\). For easier comparison with the limit of conserved custodial symmetry, we re-write the potential as

$$\begin{aligned} V&= \mu ^2_2 (\phi ^\dagger \phi ) + \mu ^2_\chi \mathrm{Tr}(\chi ^\dagger \chi ) + \frac{1}{2} \mu ^2_\eta \mathrm{Tr}(\eta ^\dagger \eta ) + 4 \lambda _1 (\phi ^\dagger \phi )^2\nonumber \\&\quad +\, 2 (\phi ^\dagger \phi ) \left( 2\lambda _{2a} \mathrm{Tr}(\chi ^\dagger \chi ) + \lambda _{2b} \mathrm{Tr}(\eta ^\dagger \eta )\right) \nonumber \\&\quad + \, 2 \lambda _{3a} \mathrm{Tr}\left( (\eta ^\dagger \eta )^2\right) + 2 \lambda _{3b} \left( \mathrm{Tr} \left( ( \chi ^\dagger \chi )^2\right) \right. \nonumber \\&\quad \left. + \,3 \mathrm{Tr}(\chi ^\dagger \chi \chi \chi ^\dagger ) \right) + 4 \lambda _{3c} \mathrm{Tr}(\chi ^\dagger \chi \eta ^\dagger \eta \chi ^\dagger \eta ^\dagger \chi \eta ) \nonumber \\&\quad +\, \lambda _{4a} \left( \mathrm{Tr}(\eta ^\dagger \eta )\right) ^2 + 4 \lambda _{4b} \left( \mathrm{Tr}(\chi ^\dagger \chi )\right) ^2 \nonumber \\&\quad +\, 4\lambda _{4c} \mathrm{Tr}(\eta ^\dagger \eta ) \mathrm{Tr}(\chi ^\dagger \chi ) - \lambda _{5a} (\phi ^\dagger \chi ^\dagger \chi \phi - \phi ^\dagger \chi \chi ^\dagger \phi ) \nonumber \\&\quad +\, \sqrt{2} \lambda _{5b} (\phi ^\dagger \eta \chi ^\dagger \phi ^c +\mathrm{h.c.}) \nonumber \\&\quad +\,\frac{1}{\sqrt{2}}M_{1a} \phi ^\dagger \eta \phi + \frac{1}{2} M_{1b} \left( \phi ^T (i \tau _2)\chi \phi + \text {h.c.} \right) \nonumber \\&\quad +\, 3 \sqrt{2} M_2 \left( \mathrm{Tr}( \chi ^\dagger \chi \eta ) + \mathrm{h.c.} \right) . \end{aligned}$$
(2.17)

At the tree-level where custodial symmetry is conserved, we have

$$\begin{aligned} \lambda _{Na}=\lambda _{Nb}\equiv&\lambda _2, \quad N=\{2,5\}, \end{aligned}$$
(2.18)
$$\begin{aligned} \lambda _{Na}=\lambda _{Nb}=\lambda _{Nc}\equiv&\lambda _3, \quad N=\{3,4\}, \end{aligned}$$
(2.19)

as well as

$$\begin{aligned} M_{1a}=M_{1b}\equiv&M_1 , \end{aligned}$$
(2.20)
$$\begin{aligned} \mu _\chi ^2 = \mu _\eta ^2 \equiv&\mu _3^2 . \end{aligned}$$
(2.21)

The parameters of Eqs. (2.16) and (2.17) are related via

$$\begin{aligned} \lambda _1&= \frac{1}{4} \lambda , \qquad \lambda _{2a} = \frac{1}{4} \left( \sigma _1 + \frac{1}{2} \sigma _2\right) , \qquad \nonumber \\ \lambda _{2b}&= \frac{1}{2} \sigma _3, \end{aligned}$$
(2.22)
$$\begin{aligned} \lambda _{3a} + \lambda _{4a}&= \frac{1}{2} \rho _3, \qquad \lambda _{3b} = -\frac{1}{4} \rho _2 , \qquad \lambda _{3c} = \frac{1}{4} \rho _5,\qquad \nonumber \\ \lambda _{4b}&= \frac{1}{4} \left( \rho _1 + \frac{3}{2} \rho _2\right) ,\qquad \lambda _{4c} = \frac{1}{4} \rho _4 , \end{aligned}$$
(2.23)
$$\begin{aligned} \lambda _{5a}&= -\frac{1}{2} \sigma _2, \qquad \lambda _{5b} = -\frac{1}{\sqrt{2}} \sigma _4, \end{aligned}$$
(2.24)
$$\begin{aligned} \mu _2^2&= m_\phi ^2, \qquad \mu _\chi ^2 = m_\chi ^2,\qquad \mu _\eta ^2 = 2\, m_\eta ^2, \qquad \nonumber \\ M_{1a}&= \sqrt{2}\, \mu _1,\qquad M_{1b} = 2 \mu _2,\qquad M_2 = -\frac{\mu _3}{6 \sqrt{2}}. \end{aligned}$$
(2.25)

The tadpole equations in the case of broken custodial symmetry read

$$\begin{aligned} 0= & {} \frac{\partial V}{\partial v_\phi } = \frac{v_\phi }{2} (2 \mu _2^2 + 8 \lambda _1 v_\phi ^2 - M_{1a} v_\eta + 4 \lambda _{2b} v_\eta ^2 \nonumber \\&-\, 2 M_{1b} v_\chi - 4 \lambda _{5b} v_\eta v_\chi + 8 \lambda _{2a} v_\eta ^2 - 2 \lambda _{5a} v_\chi ^2), \end{aligned}$$
(2.26)
$$\begin{aligned} 0= & {} \frac{\partial V}{\partial v_\chi } = -\frac{M_{1b} v_\phi ^2}{2} - \lambda _{5b} v_\phi ^2 v_\eta \nonumber \\&+ \,v_\chi (2 \mu _{\chi }^2 + 4 \lambda _{2a} v_\phi ^2 - \lambda _{5a} v_\phi ^2 - 12 M_2 v_\eta \nonumber \\&+ \,8 \lambda _{4c} v_\eta ^2 + 8 \lambda _{3b} v_\chi ^2 + 16 \lambda _{4b} v_\chi ^2), \end{aligned}$$
(2.27)
$$\begin{aligned} 0= & {} \frac{\partial V}{\partial v_\eta } = -\frac{M_{1a} v_\phi ^2}{4} + \mu _\eta ^2 v_\eta + 2 \lambda _{2b} v_\phi ^2 v_\eta \nonumber \\&+ \, 4 \lambda _{3a} v_\eta ^3 + 4 \lambda _{4a} v_\eta ^3 \nonumber \\&- \, \lambda _{5b} v_\phi ^2 v_\chi - 6 M_2 v_\chi ^2 + 8 \lambda _{4c} v_\eta v_\chi ^2. \end{aligned}$$
(2.28)

After solving these equations for \(\mu _2^2,\,\mu _\chi ^2\) and \(\mu _\eta ^2\), the CP-even scalar mass matrix is at tree level given by

$$\begin{aligned} m_{h^0}^2 = \left( \begin{array}{ccc} 8 \lambda _1 v_\phi ^2 &{} \quad -\frac{1}{2} v_\phi \left( 4 v_\chi \lambda _{5 b}-8 v_\eta \lambda _{2 b}+M_{1a}\right) &{} \quad -\frac{v_\phi \left( -8 v_\chi \lambda _{2 a}+2 v_\chi \lambda _{5 a}+2 v_\eta \lambda _{5 b}+M_{1b}\right) }{\sqrt{2}} \\ \cdot &{} \quad \frac{32 v_\eta ^3 \lambda _{3 a}+32 v_\eta ^3 \lambda _{4 a}+4 v_\chi v_\phi ^2 \lambda _{5 b}+M_{1a} v_\phi ^2+24 M_{2} v_\chi ^2}{4 v_\eta } &{} \quad -\frac{v_\phi ^2 \lambda _{5 b}+4 v_\chi \left( 3 M_{2}-4 v_\eta \lambda _{4 c}\right) }{\sqrt{2}} \\ \cdot &{} \quad \cdot &{} \quad \frac{v_\phi ^2 \left( 2 v_\eta \lambda _{5 b}+M_{1b}\right) +32 v_\chi ^3 \lambda _{3 b}+64 v_\chi ^3 \lambda _{4 b}}{4 v_\chi } \\ \end{array} \right) , \end{aligned}$$
(2.29)

and the mass of the physical pseudo-scalar state is

$$\begin{aligned} m_{A}^2 = \left( 8 v_\chi ^2+v_\phi ^2\right) \left( \frac{v_\eta \lambda _{5 b}}{2 v_\chi }+\frac{M_{1b}}{4 v_\chi }\right) . \end{aligned}$$
(2.30)

The mass matrix for the singly-charged scalars reads, in Landau gauge,

$$\begin{aligned} m_{H^\pm }^2 \!= \!\left( \!\! \begin{array}{ccc} {\tilde{M}}_b v_\chi +{\tilde{M}}_a v_\eta &{} -\frac{{\tilde{M}}_a v_\phi }{2} &{} -\frac{{\tilde{M}}_b v_\phi }{2} \\ \cdot &{} \frac{4 {\tilde{M}}_2 v_\chi ^2+2 v_\phi ^2 \lambda _{5 b} v_\chi +{\tilde{M}}_a v_\phi ^2}{4 v_\eta } &{} -\frac{1}{2} \lambda _{5 b} v_\phi ^2-{\tilde{M}}_2 v_\chi \\ \cdot &{} \cdot &{} \frac{{\tilde{M}}_b v_\phi ^2+2 v_\eta \lambda _{5 b} v_\phi ^2+4 {\tilde{M}}_2 v_\chi v_\eta }{4 v_\chi } \\ \end{array} \!\!\right) \!, \end{aligned}$$
(2.31)

where

$$\begin{aligned} {\tilde{M}}_a= & {} M_{1a} + 2 v_\chi \lambda _{5b} ,\qquad {\tilde{M}}_b = M_{1b} + 2 v\chi \lambda _{5a}, \qquad \nonumber \\ {\tilde{M}}_2= & {} 6 M_2 + 4 v_\eta \lambda _{3c}, \end{aligned}$$
(2.32)

and the mass of the doubly-charged scalar is given by

$$\begin{aligned} m^2_{H^{\pm \pm }}= & {} v_\phi ^2 \lambda _{5 a}+8 v_\chi ^2 \lambda _{3 b}+\frac{v_\eta v_\phi ^2 \lambda _{5 b}}{2 v_\chi }\nonumber \\&+\frac{M_{1b} v_\phi ^2}{4 v_\chi }+12 M_2 v_\eta . \end{aligned}$$
(2.33)

2.3 Renormalisation of the scalar sector of the Georgi-Machacek model

So far, the scalar sector of the GM model has only been studied at tree level in the literature. The only exception is Ref. [34] which has pointed out that an on-shell renormalisation of the model is not possible without breaking the custodial symmetry. Moreover, it was found that the loop corrections to the scalar mass matrices can become huge. Here we are going to apply two different renormalisation schemes to this model. We start with the discussion of a \(\overline{\text {MS}}\) scheme, before we turn to an on-shell scheme.

2.3.1 \(\overline{\text {MS}}\) renormalisation

The easiest option to renormalise the scalar sector of the GM model is to use the \(\overline{\text {MS}}\) scheme which is applicable to any model. Another advantage of this scheme, beside its general applicability, is that it makes large loops corrections immediately visible without hiding them in counter-terms.Footnote 1 The renormalisation procedure is as follows:

  1. 1.

    We match the measured SM parameters (\(\alpha _s, \alpha _{ew}, m_{\!f}\!, G_F\)) to the running \(\overline{\text {MS}}\) parameters at the scale \(Q=M_Z\). The matching procedure is explained in Ref. [37].

  2. 2.

    We run the SM parameters using three-loop SM renormalisation group equations (RGEs) to the scale of new physics, which we set to either \(m_5\) or \(m_H\). At this scale, the model-specific input parameters (\(\lambda _1\),...\(\lambda _5\)\(M_1\)\(M_2\)\(s_H\)) are taken as input. In addition, we solve the tadpole equations to obtain the numerical values of the quadratic mass terms.

  3. 3.

    All tree-level masses are calculated and the shifts to the SM gauge and Yukawa couplings are included. This is done by imposing that the eigenvalues of the one-loop corrected mass matrix of the fermions

    $$\begin{aligned} m_f^{(1L)}(p^2_i)= & {} m_f^{(T)} - \tilde{\Sigma }_S(p^2_i) - \tilde{\Sigma }_R(p^2_i) m_f^{(T)} \nonumber \\&- \, m_f^{(T)} \tilde{\Sigma }_L(p^2_i) \end{aligned}$$
    (2.34)

    agree with the running \(\overline{\text {MS}}\) masses. In the gauge sector, the electroweak coupling is shifted as

    $$\begin{aligned} \alpha ^\mathrm{GM}_{ew}(Q)= & {} \frac{\alpha ^{SM}_{ew}(Q)}{1-\frac{\alpha _{ew}}{2\pi }\Delta _{ew}} \text {with} \quad \quad \nonumber \\ \Delta _{ew}= & {} \frac{4}{3} \log (m_{H^{\pm \pm }}/Q) + \frac{1}{3} \sum _{i=1}^2 \log (m_{H_i^{\pm }}/Q).\nonumber \\ \end{aligned}$$
    (2.35)

    The changes in the weak mixing angle are calculated from the Z and W self-energies, see Ref. [37] for more details of the matching procedure at the example of the MSSM.

  4. 4.

    In order to ensure that we are operating at the bottom of the potential at each order of perturbation theory, we demand that the loop-corrected tadpole equations fulfil

    $$\begin{aligned} T_i + \delta ^{(n)} t_i = 0. \end{aligned}$$
    (2.36)

    Here, \(T_i\) are the tree-level tadpoles, and \(\delta ^{(n)} t_i\) are the shifts at the n-loop level. These conditions introduce the only three finite CTs which we need:

    $$\begin{aligned} \delta ^{(n)} \mu _2^2&= - \,\delta ^{(n)} t_v , \end{aligned}$$
    (2.37)
    $$\begin{aligned} \delta ^{(n)} \mu _\eta ^2&= -\, \delta ^{(n)} t_\eta , \end{aligned}$$
    (2.38)
    $$\begin{aligned} \delta ^{(n)} \mu _\chi ^2&= - \,\delta ^{(n)} t_\chi . \end{aligned}$$
    (2.39)

    Note that the breaking of the custodial symmetry at the loop level already becomes visible at this stage due to \(\delta ^{(n)} t_\eta \ne \delta ^{(n)} t_\chi \) and therefore \(\mu _\eta \ne \mu _\chi \). We will make use of the functionality of the tools SARAH/SPheno which are able to calculate \(\delta ^{(n)} t_i\) up to two-loop.

  5. 5.

    We calculate the one-loop corrections \(\Pi \) to the scalar mass matrices. At the one-loop level, the full dependence on the external momenta is included, while at two-loop, the approximation \(p^2=0\) is used. The pole masses are the eigenvalues of the loop-corrected mass matrix calculated as

    $$\begin{aligned} M_{\phi }^{(2L)}(p^2) = \tilde{M}^{(2L)}_{\phi } - \Pi _{\phi }(p^2)^{(1L)} - \Pi _{\phi }(0)^{(2L)}.\nonumber \\ \end{aligned}$$
    (2.40)

    Here, \(\tilde{M}_{\phi }\) is the tree-level mass matrix including the shifts Eqs. (2.37)–(2.39). The two-loop self-energies are available for all real scalars in SARAH/SPheno. For charged scalars, the scalar masses are available at the one-loop level,

    $$\begin{aligned} M_{\phi }^{(1L)}(p^2) = \tilde{M}^{(1L)}_{\phi } - \Pi _{\phi }(p^2)^{(1L)} . \end{aligned}$$
    (2.41)

    The calculation of the one-loop self-energies in both cases is done iteratively for each eigenvalue i until the on-shell condition

    $$\begin{aligned} \left[ \text {eig} M_{\phi }^{(n)}(p^2=m^2_{\phi _i})\right] _i \equiv m_{\phi _i}^2 \end{aligned}$$
    (2.42)

    is fulfilled.

    We present the explicit expressions for the one-loop corrections to the tadpoles and self-energies in “Appendix B”. The two-loop corrections are too long to be presented in this work. However, they are available in the Mathematica format and can be sent on demand or generated automatically with SARAH.

In the \(\overline{\text {MS}}\) scheme, all masses receive finite corrections at the loop level. Thus, mass parameters used as input are only Lagrangian (\(\overline{\text {MS}}\)) parameters which are different from the values of the pole masses. This has the drawback that one can’t use physical parameters as input. On the other side, as we already mentioned, it makes the presence of large loop corrections immediately visible. Moreover, if one wants to draw the connection to a more fundamental theory which predicts the Lagrangian parameter at a higher scale, one must start with the running \(\overline{\text {MS}}\) parameters at the given scale and include the higher order corrections to all masses.

2.3.2 On-shell renormalisation

In an OS scheme, the tree-level masses and rotation angles are taken to be equivalent to the loop-corrected ones. Therefore, an OS scheme has the advantage that physical parameters can directly be used as input. This scheme is often the preferred option if a sufficient number of free parameters exists to absorb all finite corrections. However, one needs to be careful as this is not always possible. The best known exception are supersymmetric models: the SUSY relations among the terms in the potential reduce the number of free parameters and make a full OS calculation of the Higgs sector even in the simplest models impossible. Also the custodial symmetry of the GM model reduces the number of free parameters and there are not sufficient CTs to renormalise the scalar sector on-shell. Therefore, we need to give up this symmetry at the loop level and introduce CTs for all potential parameters

$$\begin{aligned}&x \rightarrow x + \delta _x, \qquad x \in \{\lambda _1,\lambda _{2a}, \lambda _{2b}, \lambda _{3a},\lambda _{3b},\lambda _{3c},\nonumber \\&\quad \lambda _{4a},\lambda _{4b},\lambda _{4c},\lambda _{5a},\lambda _{5b},M_{1a},M_{1b},M_2, \mu _2^2, m_\eta ^2, m_\chi ^2\}\nonumber \\ \end{aligned}$$
(2.43)

With this extended set of CTs, it is now possible to renormalise the scalar sector completely on-shell. We are doing this at the one-loop level using a similar ansatz as in Ref. [38] for the THDM. The CTs are fixed by the following renormalisation conditions:

$$\begin{aligned}&\delta T_i + \delta ^{(1L)} t_i \equiv 0 , \end{aligned}$$
(2.44)
$$\begin{aligned}&\delta M^h + \Pi ^h \equiv 0 , \end{aligned}$$
(2.45)
$$\begin{aligned}&\delta M^A + \Pi ^A \equiv 0 , \end{aligned}$$
(2.46)
$$\begin{aligned}&\delta M^{H^+} + \Pi ^{H^+} \equiv 0 , \end{aligned}$$
(2.47)
$$\begin{aligned}&\delta M^{H^{++}} + \Pi ^{H^{++}} \equiv 0 . \end{aligned}$$
(2.48)

Here, \(\delta T_i\) and \(\delta M^\Phi \) are the counter-term contributions to the tadpoles and mass matrices. \(\delta t_i\) are the one-loop corrections to the tadpoles, and \(\Pi ^\Phi \) are the one-loop self-energies. For simplicity we assume that \(\Pi ^\Phi \) are calculated with vanishing external momentum, i.e. \(p^2=0\). This approximation is justified because we are only interested in the overall size of the CTs and their impact on the vacuum stability constraints. The explicit expressions for the CTs stemming from these conditions are given in “Appendix C”.

In order to obtain the finite values for the CTs, we perform the following steps:

  1. 1–3.

    These steps to get the parameters at the renormalisation scale are identical to the \(\overline{\text {MS}}\) calculation.

  2. 4.

    The one-loop tadpoles and self-energies are calculated.

  3. 5.

    Equations. (C.1)–(C.14) are used to obtain the finite values for the CTs.

As explained, the custodial symmetry gets broken by the one-loop shifts in the parameters \(\lambda _i\). This effect is triggered by the hypercharge and one might expect that it is rather mild. This was for instance also found in Ref. [36] where the effects of custodial symmetry breaking through RGE evolution have been studied. However, if we compare the dominant contributions to the CTs of \(\delta \lambda _{3b}\) and \(\delta \lambda _{3c}\), see Eqs. (C.3) and (C.4), we find for small \(s_H\)

$$\begin{aligned} \delta \lambda _{3b}- \delta \lambda _{3c} \simeq \frac{1}{s_H^2 v^2}\left( \Pi ^{H^{++}} + \Pi ^A_{22} - 2\, \Pi ^{H^+}_{33}\right) . \end{aligned}$$
(2.49)

The divergence for \(s_H \rightarrow 0\) is caused because the trilinear coupling \(M_2\) becomes huge in this limit for fixed \(m_5\). Including only the contributions from the ew gauge couplings and assuming the new scalars to be degenerate, we get for \(m_5 \gg v\)

$$\begin{aligned} \delta \lambda _{3b}- \delta \lambda _{3c} \simeq \frac{g_1^2}{16\pi ^2}\frac{m_5^2}{s_H^2 v^2} , \end{aligned}$$
(2.50)

i.e. the effects are enhanced by a factor \(m^2_5/(s_H v)^2\).

3 Theoretical constraints

3.1 Tree-level unitarity constraints

The first, and already at tree level rather severe, constraint on the parameter space of the GM model is perturbative unitarity of the \(2 \rightarrow 2\) scalar field scattering amplitudes. This means that the 0th partial wave amplitude \(a_0\) must satisfy either \(|a_0| \le 1\) or \(|{\mathcal Re} [ a_0]| \le \frac{1}{2}\). The scattering matrix element \(\mathcal {M}\) is given by

$$\begin{aligned} \mathcal {M} = 16\pi \sum _J (2J+1) a_J P_J(\cos \theta ), \end{aligned}$$
(3.1)

where J is the angular momentum and \(P_J(\cos \theta )\) are the Legendre polynomials. At the tree level, the \(2 \rightarrow 2\) amplitudes are real, which is why one usually uses the more severe constraint \(|{\mathcal Re} [ a_0]| \le \frac{1}{2}\), which leads to \(|\mathcal {M}| < 8 \pi \). For analysing whether perturbative unitarity is given or not, it is common to work in the high energy limit, i.e. the dominant tree-level diagrams contributing to \(|\mathcal M|\) involve only quartic interactions. All other diagrams with propagators are suppressed by the collision energy squared and can be neglected. Moreover, effects of electroweak symmetry breaking (EWSB) are usually ignored, i.e. Goldstone bosons are considered as physical fields.

The condition \(|\mathcal {M}| < 8 \pi \) must be satisfied by all of the eigenvalues \(\tilde{x}_i\) of the scattering matrix \(\mathcal {M}\). \(\mathcal {M}\) must be derived by including each possible combination of two scalar fields in the initial and final states. The explicit expressions for the eigenvalues \(\tilde{x}\) for conserved custodial symmetry are for instance given in Refs. [29, 30]. They can be translated into the following tree-level unitarity conditions:

$$\begin{aligned}&\sqrt{ \left( 6 \lambda _1 - 7 \lambda _3 - 11 \lambda _4 \right) ^2 + 36 \lambda _2^2} + \left| 6 \lambda _1 + 7 \lambda _3 + 11 \lambda _4 \right|< 4 \pi , \nonumber \\&\sqrt{ \left( 2 \lambda _1 + \lambda _3 - 2 \lambda _4 \right) ^2 + \lambda _5^2} + \left| 2 \lambda _1 - \lambda _3 + 2 \lambda _4 \right|< 4 \pi , \nonumber \\&\left| 2 \lambda _3 + \lambda _4 \right|< \pi , \nonumber \\&\left| \lambda _2 - \lambda _5 \right| < 2 \pi . \end{aligned}$$
(3.2)

Without the custodial symmetry, the eigenvalues of the scattering matrix have not been calculated before, but are given here for the first time. Still most eigenvalues of the more complicated scattering matrix have simple, analytical expressions:

$$\begin{aligned}&\pm \,8 (2 {\lambda }_{3b} + {\lambda }_{4b}), \end{aligned}$$
(3.3)
$$\begin{aligned}&\pm \, 4 (2 {\lambda }_{4b} + {\lambda }_{3b}), \end{aligned}$$
(3.4)
$$\begin{aligned}&\pm \, 2 \left( 2 {\lambda }_{3a} + 2 ({\lambda }_{4a} + {\lambda }_{4b}) + 3 {\lambda }_{3b} \right. \nonumber \\&\quad \left. \pm \sqrt{(2 \lambda _{3a}-3 \lambda _{3b}+2 \lambda _{4a}-2 \lambda _{4b})^2+8 \lambda _{3c}^2} \right) , \end{aligned}$$
(3.5)
$$\begin{aligned}&\pm \,8 (2 {\lambda }_{3c} + {\lambda }_{4c}), \end{aligned}$$
(3.6)
$$\begin{aligned}&\pm \,4(2 {\lambda }_{4c} + {\lambda }_{3c}), \end{aligned}$$
(3.7)
$$\begin{aligned}&\pm (4 {\lambda }_{2a} - {\lambda }_{5a}) , \end{aligned}$$
(3.8)
$$\begin{aligned}&\pm \, 2 (2 {\lambda }_{2a} + {\lambda }_{5a}), \end{aligned}$$
(3.9)
$$\begin{aligned}&\pm \,2 \left( 2 {\lambda }_{1} + 2 {\lambda }_{4b} - {\lambda }_{3b} \pm \sqrt{(2 \lambda _1+\lambda _{3b}-2 \lambda _{4b})^2+\lambda _{5a}^2}\right) , \end{aligned}$$
(3.10)
$$\begin{aligned}&\pm \,2 \left( 2 {\lambda }_{1} + 2 {\lambda }_{4c} - {\lambda }_{3c} \pm \sqrt{(2 \lambda _1+\lambda _{3c}-2 \lambda _{4c})^2+\lambda _{5b}^2}\right) , \end{aligned}$$
(3.11)
$$\begin{aligned}&\pm \frac{1}{2} \left( -4 {\lambda }_{2a} -4 {\lambda }_{2b} - {\lambda }_{5a} \right. \nonumber \\&\left. \quad \pm \sqrt{(4 \lambda _{2a}-4 \lambda _{2b}+\lambda _{5a})^2+8 \lambda _{5b}^2} \right) , \end{aligned}$$
(3.12)
$$\begin{aligned}&\pm \left( 2 {\lambda }_{2a} + 2 {\lambda }_{2b} - {\lambda }_{5a}\right. \nonumber \\&\quad \left. \pm \sqrt{(-2 \lambda _{2a}+2 \lambda _{2b}+\lambda _{5a})^2+8 \lambda _{5b}^2}\right) . \end{aligned}$$
(3.13)

Three other eigenvalues are the solutions \(x_{1,2,3}\) of the polynomial

$$\begin{aligned} 0&= 384 \Big [2 {\lambda }_{1} \left( 5 ({\lambda }_{3a} + {\lambda }_{4a})(3 {\lambda }_{3b} + 4 {\lambda }_{4b} ) + (3 {\lambda }_{4c} + {\lambda }_{3c})^{2}\right) \nonumber \\&\quad - \, 2 {\lambda }_{2a} {\lambda }_{2b} (3 {\lambda }_{4c} + {\lambda }_{3c}) +5 {\lambda }_{2a}^{2} ({\lambda }_{3a} \nonumber \\&\quad + \, {\lambda }_{4a}) + {\lambda }_{2b}^{2} (3 {\lambda }_{3b} + 4 {\lambda }_{4b} )\Big ] \nonumber \\&\quad +\, x \left[ 16 (2 (3 {\lambda }_{1} (5 {\lambda }_{3a} + 5 {\lambda }_{4a} + 6 {\lambda }_{3b} + 8 {\lambda }_{4b} ) \right. \nonumber \\&\quad -\, (3 {\lambda }_{4c} + {\lambda }_{3c})^{2} + 5 ({\lambda }_{3a} + {\lambda }_{4a})(3 {\lambda }_{3b} + 4 {\lambda }_{4b} )) \nonumber \\&\quad \left. -\,3 {\lambda }_{2b}^{2} -6 {\lambda }_{2a}^{2} )\right] \nonumber \\&\quad + x^2 \left[ 4 (5 {\lambda }_{3a} + 5 {\lambda }_{4a} + 6 {\lambda }_{1} + 6 {\lambda }_{3b} + 8 {\lambda }_{4b} ) \right] + x^3. \end{aligned}$$
(3.14)

These expressions were extracted from SARAH as described in appendix D.

From these eigenvalues, we can derive the following tree-level unitarity conditions in the case of broken custodial symmetry

$$\begin{aligned}&|2 {\lambda }_{3b} + {\lambda }_{4b}| < \pi , \end{aligned}$$
(3.15)
$$\begin{aligned}&|2 {\lambda }_{3c} + {\lambda }_{4c}| < \pi ,\end{aligned}$$
(3.16)
$$\begin{aligned}&|2 {\lambda }_{3a} + 2 ({\lambda }_{4a} + {\lambda }_{4b}) + 3 {\lambda }_{3b}| \nonumber \\&\quad + \sqrt{(2 \lambda _{3a}-3 \lambda _{3b}+2 \lambda _{4a}-2 \lambda _{4b})^2+8 \lambda _{3c}^2} ) < 4\pi ,\end{aligned}$$
(3.17)
$$\begin{aligned}&|2 {\lambda }_{4b} + {\lambda }_{3b}| < 2\pi ,\end{aligned}$$
(3.18)
$$\begin{aligned}&|2 {\lambda }_{4c2} + {\lambda }_{3c}| < 2\pi ,\end{aligned}$$
(3.19)
$$\begin{aligned}&|4 {\lambda }_{2a} - {\lambda }_{5a}| < 8 \pi , \end{aligned}$$
(3.20)
$$\begin{aligned}&|2 {\lambda }_{2a} + {\lambda }_{5a}|< 4\pi ,\end{aligned}$$
(3.21)
$$\begin{aligned}&|4 {\lambda }_{2a} +4 {\lambda }_{2b} + {\lambda }_{5a}| \nonumber \\&\quad + \sqrt{(4 \lambda _{2a}-4 \lambda _{2b}+\lambda _{5a})^2+8 \lambda _{5b}^2} < 16\pi ,\end{aligned}$$
(3.22)
$$\begin{aligned}&|2 {\lambda }_{2a} + 2 {\lambda }_{2b} - {\lambda }_{5a} | \nonumber \\&\quad + \sqrt{(-2 \lambda _{2a}+2 \lambda _{2b}+\lambda _{5a})^2+8 \lambda _{5b}^2} < 8 \pi ,\end{aligned}$$
(3.23)
$$\begin{aligned}&|2 {\lambda }_{1} + 2 {\lambda }_{4b} - {\lambda }_{3b}| + \sqrt{(2 \lambda _1+\lambda _{3b}-2 \lambda _{4b})^2+\lambda _{5a}^2}< 4\pi ,\end{aligned}$$
(3.24)
$$\begin{aligned}&|2 {\lambda }_{1} + 2 {\lambda }_{4c} - {\lambda }_{3c}| + \sqrt{(2 \lambda _1+\lambda _{3c}-2 \lambda _{4c})^2+\lambda _{5b}^2}< 4\pi ,\end{aligned}$$
(3.25)
$$\begin{aligned}&\text {max}(x_{1,2,3}) < 8\pi . \end{aligned}$$
(3.26)

Even though we will always assume that the custodial symmetry is conserved at tree-level, we will make use of these ‘generalised’ unitarity constraints in combination with new perturbativity constraints as explained in Sect. 3.3.

3.2 Vacuum stability constraints

3.2.1 Tree-level considerations

Another theoretical constraint which has already been studied in the context of the GM model is the vacuum stability constraint at tree-level. In general, there are two possible situations which can cause an instability of the vacuum with correct EWSB: either directions in the scalar potential exist in which the potential is unbounded from below, or other local minima exist in the scalar potential which are deeper than the ew one.

Unbounded from below UFB directions exist if quartic couplings fulfil specific conditions. The simplest condition is \(\lambda _1<0\) since in this case the potential approaches \(-\,\infty \) for \(v_\phi \rightarrow \infty \). The full list of tree-level conditions to avoid unboundedness from below in the case of conserved custodial symmetry was derived in Ref. [30]. It reads:

$$\begin{aligned} \lambda _1> & {} 0, \end{aligned}$$
(3.27)
$$\begin{aligned} \lambda _4> & {} \left\{ \begin{array}{l l} - \frac{1}{3} \lambda _3, &{}\quad \ \lambda _3 \ge 0, \\ - \lambda _3, &{}\quad \lambda _3 < 0, \end{array} \right. \end{aligned}$$
(3.28)
$$\begin{aligned} \lambda _2> & {} \left\{ \begin{array}{l l} \frac{1}{2} \lambda _5 - 2 \sqrt{\lambda _1 \left( \frac{1}{3} \lambda _3 + \lambda _4 \right) }, &{} \quad \lambda _5 \ge 0 \wedge \lambda _3 \ge 0, \\ \omega _+(\zeta ) \lambda _5 - 2 \sqrt{\lambda _1 ( \zeta \lambda _3 + \lambda _4)}, &{} \quad \lambda _5 \ge 0 \wedge \lambda _3< 0, \\ \omega _-(\zeta ) \lambda _5 - 2 \sqrt{\lambda _1 (\zeta \lambda _3 + \lambda _4)}, &{} \quad \lambda _5 < 0, \end{array} \right. \nonumber \\ \end{aligned}$$
(3.29)

with

$$\begin{aligned} \omega _{\pm }(\zeta )= & {} \frac{1}{6}(1 - B(\zeta )) \nonumber \\&\pm \frac{\sqrt{2}}{3} \left[ (1 - B(\zeta )) \left( \frac{1}{2} + B(\zeta )\right) \right] ^{1/2}, \qquad \nonumber \\ B(\zeta )= & {} \sqrt{\frac{3}{2}\left( \zeta - \frac{1}{3}\right) } . \end{aligned}$$
(3.30)

Equation (3.29) must be satisfied for all values of \(\zeta \in \left[ \frac{1}{3}, 1 \right] \).

If the custodial symmetry is broken, more conditions need to be checked. These conditions were derived in Ref. [36] assuming two simultaneously non-vanishing field directions. We have re-derived these conditions using our parametrisation, shown below. Not derived in Ref. [36] were UFB conditions on the “custodial” direction with \(\langle \eta ^0\rangle = \langle \chi ^0 \rangle \ne 0\) and \(\langle \phi ^0 \rangle \ne 0\) which we also present here. The set of UFB conditions reads

$$\begin{aligned}&\lambda _1 > 0 , \end{aligned}$$
(3.31)
$$\begin{aligned}&\lambda _{3a} + \lambda _{4a} > 0 , \end{aligned}$$
(3.32)
$$\begin{aligned}&\lambda _{3b} + 2 \lambda _{4b} > 0 , \end{aligned}$$
(3.33)
$$\begin{aligned}&\lambda _{3b} + \lambda _{4b} > 0 , \end{aligned}$$
(3.34)
$$\begin{aligned}&\lambda _{3c} + \sqrt{2} \sqrt{(\lambda _{3a} + \lambda _{4a}) (\lambda _{3b} + 2 \lambda _{4b})} + 2 \lambda _{4c} > 0 , \end{aligned}$$
(3.35)
$$\begin{aligned}&\sqrt{2} \sqrt{(\lambda _{3a} + \lambda _{4a}) (\lambda _{3b} + 2 \lambda _{4b})} + 2 \lambda _{4c} > 0 , \end{aligned}$$
(3.36)
$$\begin{aligned}&\sqrt{(\lambda _{3a} + \lambda _{4a}) (\lambda _{3b} + \lambda _{4b})} + \lambda _{4c} > 0 , \end{aligned}$$
(3.37)
$$\begin{aligned}&\lambda _{3c} + \sqrt{(\lambda _{3a} + \lambda _{4a}) (\lambda _{3b} + \lambda _{4b})} + \lambda _{4c} > 0, \end{aligned}$$
(3.38)
$$\begin{aligned}&4 \lambda _{2a} + 4 \sqrt{2} \sqrt{\lambda _1 (\lambda _{3b} + 2 \lambda _{4b})} - \lambda _{5a} > 0 , \end{aligned}$$
(3.39)
$$\begin{aligned}&4 \lambda _{2a} + 4 \sqrt{2} \sqrt{\lambda _1 (\lambda _{3b} + 2 \lambda _{4b})} + \lambda _{5a} > 0 , \end{aligned}$$
(3.40)
$$\begin{aligned}&\lambda _{2a} + 2 \sqrt{ \lambda _1 (\lambda _{3b} + \lambda _{4b})} > 0 , \end{aligned}$$
(3.41)
$$\begin{aligned}&\lambda _{2b} + 2 \sqrt{ \lambda _1 (\lambda _{3a} + \lambda _{4a})} > 0, \end{aligned}$$
(3.42)
$$\begin{aligned}&\lambda _{3a} + 2 \lambda _{3b} + \lambda _{4a} + 4 \lambda _{4b} + 4 \lambda _{4c} > 0 , \end{aligned}$$
(3.43)
$$\begin{aligned}&2 \lambda _{2a} + \lambda _{2b} + 2 \sqrt{\lambda _1 (\lambda _{3a} + 2 \lambda _{3b} + \lambda _{4a} + 4 (\lambda _{4b} + \lambda _{4c}))} \nonumber \\&\quad - (\lambda _{5a}/2) - \lambda _{5b} >0. \end{aligned}$$
(3.44)

Note that after translating the parameters according to Eqs. (2.22)–(2.25), Eqs. (3.37) and (3.38) differ w.r.t. the fourth and fifth line in Eq. (12) of Ref. [36] in that we do not have a factor 2 in front of the square-root. The last two conditions are shown here for the first time.

Other minima Even if the potential is bounded from below in all possible field directions, additional minima are usually present. In these minima, the sum of all neutral VEVs \(\sqrt{v_\eta ^2+4(v_\chi ^2+v_\phi ^2)}\) usually doesn’t agree with the measured value of 246 GeV, i.e. those minima are not viable. Moreover, also minima can occur at which charge is broken spontaneously by the VEV of a charged scalar. If any of those minima is the global minimum of the scalar tree-level potential, then the ew minimum is unstable. If tunnelling is assumed to be instantaneous on cosmological scales – as it is usually done – this vacuum configuration is forbidden. All possible minima of the scalar potential at tree-level can be found by solving the minimisation conditions of the potential, i.e. in the most general case a set of seven coupled, cubic equations must be solved. Another method of finding all minima by using a re-parametrisation of the scalar potential is discussed in Ref. [30].

3.2.2 Loop effects

Up to now, the vacuum stability in the GM model has only been checked at tree level. However, using UFB conditions at tree level can be very misleading since these conditions involve very large field excursions – which demand a proper treatment of radiative corrections. Usually, the best method to deal with very large field excursions is to use the RGE-improved potential. In the limit of very large VEVs, the potential can be approximated very well by the tree-level potential where the running quartic couplings are inserted. Thus, to single out UFB directions, it is necessary to check that the conditions hold in the limit

$$\begin{aligned} \lambda _N \rightarrow \lambda _N(Q)|_{Q\rightarrow \infty }, \qquad N=1,\dots ,5 . \end{aligned}$$
(3.45)

It has already been pointed out in the context of the THDM [35] that UFB directions usually disappear in the RGE-improved potential in the presence of large quartic couplings. This can be understood from the general form of the RGEs: bosonic contributions increase the size of the quartic couplings with increasing energy while fermionic contributions decrease them. We can confirm that a similar observation holds in the GM model. If we forget for a moment the breaking of the custodial symmetry via hypercharge effects, the one-loop RGEs for the quartic couplings are given by [36]:

$$\begin{aligned} 16\pi ^2 \beta _{\lambda _1}&=\frac{9 g_2^4}{32}-9 g_2^2 \lambda _1\nonumber \\&\quad +\,\frac{3}{2} \left( 64 \lambda _1^2+8 \lambda _1 y_t^2+12 \lambda _2^2+\lambda _5^2-y_t^4\right) ,\end{aligned}$$
(3.46)
$$\begin{aligned} 16\pi ^2 \beta _{\lambda _2} =&\frac{3 g_2^4}{2}-\frac{33 g_2^2 \lambda _2}{2}\nonumber \\&\quad +\,2 \lambda _2 \left( 24 \lambda _1+8 \lambda _2+28 \lambda _3+44 \lambda _4+3 y_t^2\right) \nonumber \\&\quad +\,4 \lambda _5^2,\end{aligned}$$
(3.47)
$$\begin{aligned} 16\pi ^2 \beta _{\lambda _3} =&\frac{3 g_2^4}{2}-24 g_2^2 \lambda _3+80 \lambda _3^2+96 \lambda _3 \lambda _4-\lambda _5^2,\end{aligned}$$
(3.48)
$$\begin{aligned} 16\pi ^2 \beta _{\lambda _4} =&\frac{3 g_2^4}{2}-24 g_2^2 \lambda _4\nonumber \\&\quad +\,8 \left( \lambda _2^2+3 \lambda _3^2+14 \lambda _3 \lambda _4+17 \lambda _4^2\right) +\lambda _5^2,\end{aligned}$$
(3.49)
$$\begin{aligned} 16\pi ^2 \beta _{\lambda _5}&=\frac{1}{2} \lambda _5 \left( -33 g_2^2+8 (4 \lambda _1+8 \lambda _2\right. \nonumber \\&\quad \left. -\,2 \lambda _3+4 \lambda _4+\lambda _5)+12 y_t^2\right) . \end{aligned}$$
(3.50)

We show the running of the two coupling combinations \(\lambda _3+\lambda _4\) and \(4 \lambda _2-|\lambda _5| + 4 \sqrt{2} \sqrt{\lambda _1 (\lambda _3 + 2 \lambda _4)}\) in Fig. 1. The input values at \(m_t\) were chosen to be \(\lambda _1 = 0.05\), \(\lambda _2 = 0.5\), \(\lambda _3= -1.5\), \(\lambda _4 = 1\), \(\lambda _5=5\), i.e. both combinations of couplings are negative at \(m_t\). This would give the impression of two UFB directions. However, we already see at scales which are below one TeV that both combinations of couplings turn positive. Thus, the UFB directions disappear once the loop effects are included. Since the running of the quartic couplings is usually very fast and since the scale at which the couplings (or combinations of them) change their sign is not far above the input scale, we can assume that the dominant radiative effects are also covered by the effective potential without an RGE resummation. See also Ref. [35] for a similar discussion in the context of the THDM.

Fig. 1
figure 1

The scale dependence of two combinations of quartic couplings. The input parameters at \(m_t\) have been \(\lambda _1 = 0.05\), \(\lambda _2 = 0.5\), \(\lambda _3= -1.5\), \(\lambda _4 = 1\), \(\lambda _5=5\). Negative values would point towards UFB directions, i.e. it is shown that these directions disappear at large energy scales

We will use in our numerical studies the one-loop effective potential \(V_{EP}^{(1)}\) to check the vacuum stability. The different ingredients are

$$\begin{aligned} V_{EP}^{(1)} = V_{\mathrm{Tree}} + V^{(1)}_{\mathrm{CT}} + V^{(1)}_{\mathrm{CW}} . \end{aligned}$$
(3.51)

Here, \(V^{(1)}_{\mathrm{CT}}\) is the counter-term potential, and the sum \(V_{\mathrm{Tree}} + V^{(1)}_{\mathrm{CT}}\) is given by Eq. (2.17) and replacing

$$\begin{aligned} \lambda _N ~\rightarrow ~ \tilde{\lambda }_{Nx} \equiv \lambda _N + \delta \lambda _{Nx}. \end{aligned}$$
(3.52)

Note, the derived CTs depend on the ew VEVs, i.e. they result in a cancellation between \(V_{\mathrm{CT}}\) and \(V_{\mathrm{CW}}\) only at the ew minimum, but not at other positions of the potential. Thus, the conditions Eqs. (2.44)–(2.48) don’t imply that the full one-loop potential is in general identical to the tree-level potential. The Coleman-Weinberg potential \(V^{(1)}_{\mathrm{CW}}\) is given by [39]

$$\begin{aligned} V^{(1)}_{\mathrm{CW}} = \frac{1}{16\pi ^2} \sum _{i}^{\text {all fields}} r_i s_i C_i m_i^4 \left( \log \frac{m_i^2}{Q^2} - c_i \right) , \end{aligned}$$
(3.53)

with \(r_i = 1\) for real bosons or Majorana fermions, otherwise 2; \(C_i =3\) for quarks, otherwise 1; \(\{s_i,c_i\}=\{-\frac{1}{2},\frac{3}{2}\}\) for fermions, \(\{\frac{1}{4},\frac{3}{2}\}\) for scalars and \(\{\frac{3}{4},\frac{5}{6}\}\) for vector bosons. As for the other calculations, we choose \(Q=m_5\) or \(Q=m_H\) depending on the choice of the input as described in Sect. 4.2. It is important to stress that, for the check for spontaneous charge breaking via VEVs of the charged scalars at the loop level, the calculation of the physical masses must be adjusted. The reason is that the additional VEVs, which can be potentially present, also mix particles with different electric charge. If we assume no spontaneous CP violation, this results in a \(7\times 7\) mass matrix for CP-even (\(\Phi \)) and a \(6\times 6\) mass matrix for CP odd scalars (\(\Sigma \)). Analogously, both fermions and vector bosons of the same colour mix. Thus, in this case the more explicit expression for the CW potential is

$$\begin{aligned} V^{(1)}_{\mathrm{CW}}&= \frac{1}{16\pi ^2} \Bigg [ \frac{1}{4} \left( \sum _{i=1}^7 m_{\Phi _i}^4 \left( \log \frac{m_{\Phi _i}^2}{Q^2} - \frac{3}{2} \right) \right. \nonumber \\&\quad \left. + \sum _{i=1}^6 m_{\Sigma _i}^4 \left( \log \frac{m_{\Sigma _i}^2}{Q^2} - \frac{3}{2} \right) \right) \nonumber \\&\quad - 3 \sum _{i=1}^6 m_{Q_i}^4 \left( \log \frac{m_{Q_i}^2}{Q^2} - \frac{3}{2} \right) \nonumber \\&\quad - \frac{1}{2} \sum _{i=1}^9 m_{L_i}^4 \left( \log \frac{m_{L_i}^2}{Q^2} - \frac{3}{2} \right) \nonumber \\&\quad + \frac{3}{4} \sum _{i=1}^4 m_{V_i}^4 \left( \log \frac{m_{V_i}^2}{Q^2} - \frac{5}{6} \right) \Bigg ]. \end{aligned}$$
(3.54)

Because of the length of the mass matrices in the case of charge breaking VEVs, we don’t give them explicitly in this paper. Instead, we provide on request the SARAH model files for the charge breaking GM model to generate them.

3.3 Perturbativity constraints

As we have seen, the GM model provides in principle a sufficient number of CTs to renormalise all masses on-shell if the custodial symmetry is given up at the loop level. However, this does not yet ensure that such an on-shell calculation would also be trustworthy as one always needs to assume that the perturbative expansion is working. If this is not the case, then the calculation of the CTs and also all other loop calculations are not meaningful. Naively, one might expect that problems with perturbativity occur once quartic couplings \(\mathcal O(4\,\pi )\) are involved, or that at least the tree-level unitarity constraints are strong enough to filter out points which violate perturbation theory. However, it was shown for the SM that problems can occur much earlier [33] and that a better limit for the quartic coupling in the SM is \(2\,\pi \). In the specific case of the GM model, it was pointed out in Ref. [34] that problems with perturbation theory can show up for even smaller coupling values. It was observed that, in sizeable regions of the parameter space of the GM model, the two-loop corrections to masses can become larger than the one-loop contributions. This was demonstrated at the example of the SM-like Higgs mass for which the one- and two-loop corrections in the \(\overline{\text {MS}}\) scheme were compared. Of course, for a robust statement about whether perturbation theory is still working or has already broken down, it would be necessary to compare physical processes and their scale dependence at different loop levels. However, this is hardly possible in the GM model – or any other BSM model. Therefore, we want to use information which is easier accessible to get some indication whether loop calculations for a given parameter point are trustworthy or not. For this, we are going to check the effects of four different conditions which might point towards the breakdown of perturbation theory. These conditions are:

  1. 1.

    A parameter point is considered to violate perturbation theory if the two-loop corrections to at least one scalar mass are larger than the one-loop corrections, i.e.

    $$\begin{aligned} |(m_{\phi }^{2})_{\mathrm{Tree}} - (m_\phi ^{2})_{1\mathrm{L}}| < |(m_\phi ^{2})_{\mathrm{2L}} - (m_\phi ^{2})_{ \mathrm 1L}| . \end{aligned}$$
    (3.55)

    This is very close to the ansatz of Ref. [34]. However, we do not only consider the SM-like mass, but test all three neutral CP-even states, i.e. \(\phi =h_{1,2,3}\). In addition, we impose a lower threshold on \(|(m_{\phi }^{2})_{\mathrm{Tree}} - (m_\phi ^{2})_{1\mathrm{L}}|\) of \(20^2~\text {GeV}^2\) for this test. The reason for this exception is that the one-loop corrections might be very small due to an accidental cancellation which is not any more present at two-loop level – which would therefore otherwise lead to a constraint according to Eq. (3.55) without actually violating perturbation theory.

  2. 2.

    A parameter point is considered to violate perturbation theory if the CT to at least one parameter is larger than the tree-level value of this parameter times some constant, i.e.

    $$\begin{aligned} \left| \frac{\delta x}{x}\right| > v. \end{aligned}$$
    (3.56)

    For the most conservative choice, \(v=1\), this forbids points with \(|\delta x|>|x|\). We apply this constraint to all quartic couplings.

  3. 3.

    A parameter point is considered to violate perturbation theory if the CT of at least one quartic coupling becomes larger than some fixed value, i.e.

    $$\begin{aligned} |\delta x| > c \cdot \pi . \end{aligned}$$
    (3.57)

    We are going to test c within 1 and 4. Since \(\delta x\) enters the two-loop corrections, a CT as large as \(4\pi \) is for sure problematic. However, problems might occur even for smaller values as one has seen in the SM.

  4. 4.

    A parameter point is considered to violate perturbation theory if the generalised unitarity constraints Eqs. (3.15)–(3.26) are violated when inserting the renormalised couplings, i.e.

    $$\begin{aligned} |\mathcal {M}(\lambda _{Nx} \rightarrow \lambda _N + \delta \lambda _{Nx})| > 8 \pi . \end{aligned}$$
    (3.58)

    This condition is similar to the third condition but doesn’t involve any (arbitrary) upper limit on the quartic couplings. In addition, it indicates the robustness of the unitarity constraints under radiative corrections. Of course, to be sure if the unitarity constraints are really violated or not, one would need to calculate in addition the virtual and real corrections to all possible \(2\rightarrow 2\) scattering processes.

All four conditions are not rigorous in the sense that they can provide a definite answer if perturbation theory is still working or not. It is also in some sense a matter of taste which condition is considered as the most reasonable or reliable one. However, as we will see, one can get some very clear hints if problems with perturbation theory are present or not. In particular, if several conditions fail at the same time, one should be tempted to take results obtained from a calculation at Born- or even one-loop-level with care.

4 Results

4.1 Numerical setup

For our numerical study we used the Mathematica package SARAH [40,41,42,43,44] with the implementation of the GM Model discussed in Ref. [45]. In a first step, we used the model files to generate a spectrum generator based on SPheno [46, 47]. SPheno calculates by default the mass spectrum at the full one-loop level and includes all important two-loop corrections to the neutral scalar masses [34, 48, 49]. Special care is needed at the two-loop level to avoid the so-called Goldstone boson catastrophe [50]. In addition, SPheno calculates all decay modes of the particles at tree- and one-loop-level [51], including the modes discussed recently in Ref. [18]. This information is also used to write the files necessary to test a parameter point with HiggsBounds [52,53,54]. SPheno performs in addition a calculation of flavour and precision observables like \(\delta \rho \) or \(g-2\) [55].

For this project, we have extended the list of precision observables by the oblique parameters S, T and U [56]. The main reason for this was mainly that \(\delta \rho \), respectively the T parameter can’t be used to constrain the GM model: even if the tree-level contribution vanishes for \(v_\chi =v_\eta \), the one-loop correction is formally next-to-leading order. Thus, a fine-tuned CT can be added to cancel this contribution in principle. Therefore, we use the S parameter as main constraint as proposed in Ref. [32]. We also compared our numerical values of a full one-loop calculation of the S parameter with those obtained with gmcalc and usually found agreement within 5–10%.

We have modified one instance of SPheno to include the CTs to keep the scalar sector on-shell as explained in Sect. 2.3.2. A second version was kept unmodified and used to get the size of the one- and two-loop corrections to the masses in the \(\overline{\text {MS}}\) scheme, see Sect. 2.3.1 for more details.

We further used Vevacious [57] to test the stability of the one-loop effective potential. The necessary model files have been generated with SARAH. Here, we use two two different implementations: the standard one with only the three standard VEVs for the neutral scalars, and one with the possibility that all seven scalars can obtain a VEV. Since the check of the vacuum stability with seven VEVs is quite time consuming, we only test points which have passed all other constraints.Footnote 2 As input for Vevacious, we used the spectrum files written by SPheno. Vevacious automatically adjusts the counter-term potential based on CTs which are present in the SPheno spectrum file.Footnote 3

We made use of gmcalc [58] to double-check the tree-level constraints like tree-level unitarity, unboundedness from below and the presence of other minima as well as the calculation of the S parameter as already mentioned. In order to circumvent the command line input, we have modified gmcalc to read in a file with all necessary parameters (\(\lambda _i, M_i, \mu _i^2\)) as well as the running electroweak VEV as calculated by SPheno. In addition, we added a function to write the results for the tree-level unitarity check, the check for other minima as well as unbounded-from-below directions into an external file in a SLHA-like format. The different codes are combined in numerical scans using the tool SSP [59].

4.2 Input parametrisation

At the tree-level, i.e. with conserved custodial symmetry, and after applying the tadpole conditions, there are seven free Lagrangian parameters

$$\begin{aligned}&\lambda _1,\ \lambda _2,\ \lambda _3,\ \lambda _4,\ \lambda _5,\ M_1,\ M_2.&\end{aligned}$$

In addition, the relative size of the SU(2)-doublet and -triplet VEVs, controlled by \(s_H\), is a free parameter. In principle, these parameters could directly be used as input. However, the overall majority of randomly chosen points would then be ruled out by the requirement to have a CP-even scalar mass with \(\sim 125\) GeV. Therefore, it is convenient to use \(m_h\) directly as input. We have explored two different sets of input parameters:

  1. 1.

    Input I: here, we use the SM-like Higgs mass \(m_h\) as input, together with the mixing angle \(\alpha \) between the CP-even neutral SU(2)-doublet and -triplet components. In addition, the heavy Higgs mass \(m_H\) is used as input to set the overall mass scale of the new scalars. Using those input parameters, \(\lambda _1\), \(M_1\) and \(M_2\) are calculated according to

    $$\begin{aligned} M_1&= \frac{3 s_H \sqrt{2-2 s_H^2} \left( t_\alpha ^2+1\right) v^2 (2 \lambda _2-\lambda _5)-2 \sqrt{3} m_h^2 t_\alpha +2 \sqrt{3} m_H^2 t_\alpha }{3 \sqrt{1-s_H^2} \left( t_\alpha ^2+1\right) v}, \end{aligned}$$
    (4.1)
    $$\begin{aligned} M_2&= \frac{1}{9 s_H^2 \sqrt{1-s_H^2} \left( t_\alpha ^2+1\right) v^2} \Big [v \big (m_h^2 t_\alpha \left( -3 \sqrt{2-2 s_H^2} s_H t_\alpha \right. \nonumber \\&\quad \left. +\,2 \sqrt{3} s_H^2-2 \sqrt{3}\right) , -3 s_H \sqrt{2-2 s_H^2} \left( t_\alpha ^2+1\right) v^2 \left( 2 \lambda _2 \left( s_H^2-1\right) \right. \nonumber \\&\quad \left. +\,s_H^2 (-(\lambda _3+3 \lambda _4))-\lambda _5 s_H^2+\lambda _5\right) \big ) \nonumber \\&\quad +\,m_H^2 v \left( -2 \sqrt{3} s_H^2 t_\alpha -3 \sqrt{2-2 s_H^2} s_H+2 \sqrt{3} t_\alpha \right) \Big ], \end{aligned}$$
    (4.2)
    $$\begin{aligned} \lambda _1&= -\frac{m_h^2+m_H^2 t_\alpha ^2}{8 \left( s_H^2-1\right) \left( t_\alpha ^2+1\right) v^2}, \end{aligned}$$
    (4.3)

    where \(t_\alpha = \tan \alpha \). The full list of input parameters for this choice is

    $$\begin{aligned}&m_h,\ m_H,\ \alpha ,\ \lambda _2,\ \lambda _3,\ \lambda _4,\ \lambda _5,\ s_H.&\end{aligned}$$
    (4.4)

    The advantage of this input is that Higgs constraints can easily be kept under control by choosing small or moderate values of \(s_H\) and \(\alpha \) at the same time. The disadvantage is that the independent handling of \(s_H\) and \(\alpha \) implies a tuning of the other dependent tree-level parameters. As a result, loop corrections can have a significant impact, as we will see below.

  2. 2.

    Input II: here, we use \(m_h\) and \(m_5\) instead of \(\lambda _1\) and \(M_1\) as input. Moreover, \(M_2\) is set relative to \(M_1\) via a dimensionless parameter \(r_{12}\). The used relations are

    $$\begin{aligned} M_1&= \frac{s_H \left( v^2 \left( 3 \lambda _5 \left( s_H^2-1\right) -2 \lambda _3 s_H^2\right) +2 m_5^2\right) }{\sqrt{2} v \left( (6 r_{12}-1) s_H^2+1\right) } , \end{aligned}$$
    (4.5)
    $$\begin{aligned} M_2&= r_{12} M_1 , \end{aligned}$$
    (4.6)
    $$\begin{aligned} \lambda _1&= \frac{1}{32 \left( s_H^2\!-\!1\right) v^2 \left( \!v \left( \!\sqrt{2} s_H^3 v (\lambda _3\!+\!3 \lambda _4)\!-\!s_H^2 (M_1\!+\!3 M_2)\!+\!M_1\!\right) \!-\!\sqrt{2} m_h^2 s_H\!\right) } \nonumber \\&\quad \!\times \!\Big [3 s_H \left( s_H^2\!-\!1\right) v^2 \left( 4 M_1 s_H v (\lambda _5\!-\!2 \lambda _2)\!+\!2 \sqrt{2} s_H^2 v^2 (\lambda _5\!-\!2 \lambda _2)^2\!+\!\sqrt{2} M_1^2\right) \nonumber \\&\quad +4 m_h^2 v \left( -\sqrt{2} s_H^3 v (\lambda _3+3 \lambda _4)+s_H^2 (M_1+3 M_2)-M_1\right) +4 \sqrt{2} m_h^4 s_H\Big ]. \end{aligned}$$
    (4.7)

    Thus, the full list of free parameters for this input is

    $$\begin{aligned}&m_h, \ m_5,\ \lambda _2,\ \lambda _3,\ \lambda _4,\ \lambda _5,\ r_{12},\ s_H.&\end{aligned}$$
    (4.8)

    The advantage of this input is that one has direct control over the SM-like Higgs mass as well as the BSM scale \(\simeq m_5\). On the other side, the mixing between the SM-like Higgs and the other states is not an input, i.e. it can in principle become very large. This will then cause conflicts with Higgs coupling measurements.

Other proposed input sets, which we don’t explore further in the following, are: \(\{m_h,\lambda _2,\lambda _3,\lambda _4,\lambda _5,M_1,M_2\}\), \(\{m_h,m_H,m_3,m_5,\alpha ,M_1,M_2\}\), and \(\{m_h,m_5,\lambda _2,\lambda _3,\lambda _4,M_1,M_2\}\).

We are going to start now to investigate the loop constraints on the GM model using the two input sets defined above. We start with a discussion of the perturbativity constraints before we turn to the vacuum stability constraints. First, we concentrate on specific parameter regions to study the different effects. In a second step, we consider the global picture by performing random scans over large parameter ranges.

Fig. 2
figure 2

The (tree-level) masses of the three CP-even states as function of \(s_H\) for two different values of \(m_H\): 300 GeV (blue) and 800 GeV (dashed red). The other input parameters are: \(\lambda _2 =0.1\), \(\lambda _3 = 0.5\), \(\lambda _4=-0.02\), \(\lambda _5=0.1\), \(\alpha =20^\circ \)

4.3 Perturbativity constraints

4.3.1 Dependence on \(s_H\)

We start with discussing the role of \(s_H\) since it was shown in Ref. [34] that large \(s_H\) usually implies large radiative corrections. For this reason we consider the parameter point

$$\begin{aligned} \lambda _2= & {} 0.1,\ \lambda _3 = 0.5,\ \lambda _4=-\,0.02,\ \lambda _5=0.1,\ \\ \alpha= & {} 20^{\circ }, \quad \text {with} \quad m_H = 300~\text {GeV } \text {or } 800~\text {GeV}. \end{aligned}$$

The tree-level masses of the three CP-even scalars are shown in Fig. 2. As can be seen, a large separation between \(m_5\) and \(m_H\) can be present for both small and large \(s_H\). In general, the very heavy states for small values of \(s_H\) do also increase the size of the loop effects as we will see.

Fig. 3
figure 3

The relative (first row) and absolute (second row) size of the counter-terms to the quartic couplings \(\lambda _i\) as a function of \(s_H\). The third row gives the absolute value of different eigenvalues of the scattering matrix when using the renormalised quartic couplings as input. The fourth row shows the size of the one- and two-loop corrections to the scalar masses in the \(\overline{\text {MS}}\) scheme. The red line in the second row indicates values of \(\pi \), \(2\pi \) and \(4\pi \) and in the third row of \(8\pi \). On the left we set \(m_H=300\) GeV, on the right \(m_H=800\) GeV. The other parameters are analogous to Fig. 2

Consequently, we find that perturbativity constraints are not only important for large \(s_H\), but that they can also be significant for small \(s_H\). This is shown in Fig. 3 where we show the size of the different loop effects. In the first row of Fig. 3, we show the ratio of the different CTs normalised to the tree-level coupling. Here, we have defined

$$\begin{aligned} \delta \lambda _{n} / \lambda _{n}&\equiv \text {Max} \{|\delta \lambda _{na}/\lambda _n|,|\delta \lambda _{nb}/\lambda _n| \}, \quad n =2,5 , \end{aligned}$$
(4.9)
$$\begin{aligned} \delta \lambda _{n} / \lambda _{n}&\equiv \text {Max} \{|\delta \lambda _{na}/\lambda _n|,|\delta \lambda _{nb}/\lambda _n|,|\delta \lambda _{nc}/\lambda _n| \}, \quad n =3,4 . \end{aligned}$$
(4.10)

We see that for \(m_H=300\) GeV, the couplings \(\lambda _3\), \(\lambda _4\) and \(\lambda _5\) fail the constraint \(\delta \lambda /\lambda <1\) for values of \(s_H\) up to 0.2. And again for \(s_H > 0.4\), \(\lambda _3\) is in conflict with this constraint. For \(m_H = 800\) GeV, there is always a contribution which violates this bound over the entire range of \(s_H\). This choice to define perturbativity seems to be quite strong. It might also give a ‘false-positive’ result since large \(\delta \lambda /\lambda \) can easily occur if the tree-level quartic is very small. This means that there is some tuning of parameters at tree level which gets spoilt by the loop corrections. In this case, it can happen that the perturbative series behaves well and that higher order terms remain small corrections to the one-loop terms. Therefore, a more robust limit is to check the absolute size of the CTs: if those are very large, e.g. \(> 4\pi \), then higher order corrections are expected to become more and more important. Therefore, in the second row of Fig. 3 the absolute size of the counter-terms is shown. Here, we defined

$$\begin{aligned} \delta \lambda _{n}&\equiv \text {Max} \{|\delta \lambda _{na}|,|\delta \lambda _{nb}| \}, \qquad n =2,5 , \end{aligned}$$
(4.11)
$$\begin{aligned} \delta \lambda _{n}&\equiv \text {Max} \{|\delta \lambda _{na}|,|\delta \lambda _{nb}|,|\delta \lambda _{nc}| \}, \qquad n =3,4 . \end{aligned}$$
(4.12)

The qualitative behaviour of the different lines looks very similar to the case of \(\delta \lambda /\lambda \). Note that for \(m_H=800\) GeV and small \(s_H\), the CTs to some quartic couplings can become as large as \(\mathcal O(10^4~\mathrm{GeV})\). This demonstrates how bad the perturbation theory can behave and that an OS calculation, although formally possible, is not well defined in this parameter region. In this example, because of the steep increase of \(\delta \lambda _3\) towards small values of \(s_H\), the bounds on \(s_H\) are rather independent of the choice of the maximal value for the quartic CT, i.e. \(|\delta \lambda |<2\pi \) and \(|\delta \lambda |<4\pi \) result in approximately the same bounds.

Since the maximal value of \(|\delta \lambda |\) which we still consider as viable is arbitrary, we also test another condition to get an upper limit of \(|\delta \lambda |\) which is the behaviour of the scalar \(2\rightarrow 2\) scattering. For that, we calculate the eigenvalues of the scattering matrix with the renormalised couplings instead of the tree-level values. This not only gives hints for the perturbative behaviour of the parameter point but also indicates the robustness of the tree-level unitarity constraints under radiative corrections. We show in the third row of Fig. 3 the absolute values of the eigenvalues \(y_1\, \dots \, y_6\) defined as

$$\begin{aligned} y_1&= \text {Max}\Big \{8|2 {\tilde{\lambda }}_{3b} + {\tilde{\lambda }}_{4b}|,|2 {\tilde{\lambda }}_{3c} + {\tilde{\lambda }}_{4c}|, \nonumber \\&\quad 2\left( |2 {\tilde{\lambda }}_{3a} + 2 ({\tilde{\lambda }}_{4a} + {\tilde{\lambda }}_{4b}) + 3 {\tilde{\lambda }}_{3b}| \right. \nonumber \\&\quad \left. + \sqrt{(2 \tilde{\lambda }_{3a}-3 \tilde{\lambda }_{3b}+2 \tilde{\lambda }_{4a}-2 \tilde{\lambda }_{4b})^2+8 \tilde{\lambda }_{3c}^2} \right) \Big \} , \end{aligned}$$
(4.13)
$$\begin{aligned} y_2&= \text {Max}\left\{ 4|2 {\tilde{\lambda }}_{4b} + {\tilde{\lambda }}_{3b}|,4| 2 {\tilde{\lambda }}_{4c} + {\tilde{\lambda }}_{3c}|\right\} , \end{aligned}$$
(4.14)
$$\begin{aligned} y_3&= \text {Max}\left\{ |4 {\tilde{\lambda }}_{2a} - {\tilde{\lambda }}_{5a}|, \frac{1}{2}\left( |4 {\tilde{\lambda }}_{2a} +4 {\tilde{\lambda }}_{2b} + {\tilde{\lambda }}_{5a}| \right. \right. \nonumber \\&\quad \left. \left. + \sqrt{(4 \tilde{\lambda }_{2a}-4 \tilde{\lambda }_{2b}+\tilde{\lambda }_{5a})^2+8 \tilde{\lambda }_{5b}^2}\right) \right\} , \end{aligned}$$
(4.15)
$$\begin{aligned} y_4&= \text {Max}\left\{ 2|2 {\tilde{\lambda }}_{2a} + {\tilde{\lambda }}_{5a}|, |2 {\tilde{\lambda }}_{2a} + 2 {\tilde{\lambda }}_{2b} - {\tilde{\lambda }}_{5a} | \right. \nonumber \\&\quad \left. + \sqrt{(-2 \tilde{\lambda }_{2a}+2 \tilde{\lambda }_{2b}+\tilde{\lambda }_{5a})^2+8 \tilde{\lambda }_{5b}^2}\right\} ,\end{aligned}$$
(4.16)
$$\begin{aligned} y_5&= 2 \text {Max}\Big \{|2 {\tilde{\lambda }}_{1} + 2 {\tilde{\lambda }}_{4b} - {\tilde{\lambda }}_{3b}| \nonumber \\&\quad + \sqrt{(2 \tilde{\lambda }_1+\tilde{\lambda }_{3b}-2 \tilde{\lambda }_{4b})^2+\tilde{\lambda }_{5a}^2}, \nonumber \\&\quad |2 {\tilde{\lambda }}_{1} + 2 {\tilde{\lambda }}_{4c} - {\tilde{\lambda }}_{3c}| + \sqrt{(2 \tilde{\lambda }_1+\tilde{\lambda }_{3c}-2 \tilde{\lambda }_{4c})^2+\tilde{\lambda }_{5b}^2}\Big \}, \end{aligned}$$
(4.17)
$$\begin{aligned} y_6&= \text {Max}(x_{1,2,3}) , \end{aligned}$$
(4.18)

where \(x_{1,2,3}\) are the solutions of the polynomial of Eq. (3.14). All \(y_i\) should be smaller than \(8\pi \). We find that this in general results in stronger constraints than those from the condition \(|\delta \lambda |<2\pi \). For \(m_H=300\) GeV, these ‘loop corrected’ unitarity constraints are comparable to those from \(\delta \lambda /\lambda < 1\), while for \(m_H = 800\) GeV, they are a bit weaker: not the entire parameter range is in conflict with this condition in contrast to \(|\delta \lambda /\lambda < 1|\) which is violated everywhere.

Fig. 4
figure 4

Comparison of the perturbativity limits of the parameter point of Fig. 2. The left-hand side corresponds to \(m_H=300\) GeV, the right-hand side to \(m_H=800\) GeV. We show here the allowed parameter ranges which fulfil different sets of constraints. (i) \(h_i\): \(|(m_{h_i}^{2})_{\mathrm{Tree}} - (m_{h_i}^{2})_{1\mathrm{L}}| > |(m_{h_i}^{2})_{2\mathrm{L}} - (m_{h_i}^{2})_{1\mathrm{L}}|\); (ii) \(\delta \lambda _x\): \(|\delta \lambda _x| < \pi \ (2 \pi )\)  [solid line (dashed line)]; (iii) \(|\delta \lambda _x / \lambda _x| < 1\)., (iv) \(y_i\) are the absolute values of different eigenvalues of the scattering matrix when using \(\lambda _N+\delta \lambda _{Nx}\) as input

We now turn to the constraints using the \(\overline{\text {MS}}\) calculation. The size of the one- and two-loop corrections to the CP-even masses is shown in the fourth row in Fig. 3. We see that these corrections could cause shifts of hundreds of GeV in the masses, i.e. for small and large \(s_H\), they can be as large as the tree-level values. Moreover, we find that the two-loop corrections can be larger than the one-loop corrections. This reflects again a breakdown of perturbation theory.

Fig. 5
figure 5

On the left: the same as Fig. 3 for \(m_H=300\) GeV with the additional condition \(\alpha =75 s_H \frac{^{\circ }}{\text {rad}}\). On the right-hand side, we use input choice II as defined in Eq. (4.8), i.e. \((m_h,m_5)\) as input instead of \((m_h,m_H,\alpha )\) with \(m_5=300~\mathrm{GeV},\,\lambda _i=0.2\) and \(r_{12}=0.15\). The vertical orange dashed line shows the HiggsBounds limit

In Fig. 4, we summarise the limits on \(s_H\) using the different perturbativity limits. We see that for \(m_H=300\) GeV the overall limits from \(\delta \lambda _x/\lambda _x\) and \(|\delta \lambda _x|\) are quite similar. We further observe that the limits from the one- and two-loop \(\overline{\text {MS}}\) corrections are slightly weaker for small \(s_H\) but stronger for large \(s_H\). All in all, we find that for \(m_H=300\) GeV, a sizeable range of \(s_H\) is still allowed by all constraints. This is different to \(m_H=800\) GeV where \(|\delta \lambda _x/\lambda _x|\) is violated in the entire range, while for the other three sets of constraints still a window in \(s_H\) exists where these constraints are fulfilled. It is interesting to note that the constraint from \(h_3\) and \(\lambda _3\) give quite similar results. Thus, there is an obvious correlation between the size of the one-loop CTs in the OS scheme and the hierarchy between the one- and two-loop corrections in the \(\overline{\text {MS}}\) scheme.

Before we move to the impact of the other parameters on the perturbative behaviour of the model, we comment on the dependence on the different input choices. It has already been shown in Ref. [34] that the loop corrections are usually small for \(s_H\) if \(m_h\) and \(m_5\) are used as input, but not \(\alpha \). We also find this for our input choice II, cf. Eq. (4.8). As shown in Fig. 5 on the right column, one can go up to \(s_H=0.7\) for \(m_5=300\) GeV without running into obvious problems with perturbation theory. However, there is a strong correlation between \(s_H\) and \(\alpha \) and for large \(s_H\) the mixing between the SM-like Higgs and the triplets becomes so large that this causes conflicts with Higgs observables as indicated by the vertical dashed line. We find a similar behaviour for input choice I if we impose a correlation between \(\alpha \) and \(s_H\) ‘by hand’: if we demand \(\alpha =75 s_H \frac{^{\circ }}{\text {rad}}\), then we can also go up to very large values for \(s_H\) together with \(m_H=300\) GeV without running into trouble with perturbativity. However, again the region of \(s_H > 0.3\) is ruled out by the Higgs constraints. Hence, the overall picture between both input modes is comparable. We also learn from this comparison that large loop corrections occur if the chosen mixing angle \(\alpha \) is far away from a natural value which is correlated with \(s_H\). As a consequence, we use for the further examples with Input I values of \(s_H\) between 0.2 and 0.3 and take \(\alpha \) between 10\(^{\circ }\) and 20\(^{\circ }\).

4.3.2 Dependence on heavy scalar masses

Fig. 6
figure 6

Perturbativity limits as a function of \(m_H\). The other parameters are chosen as \(\lambda _2 =0.1\), \(\lambda _3 = 1\), \(\lambda _4=-\,0.1\), \(\lambda _5=0.1\), \(\alpha =20^{\circ }\), \(s_H=0.25\)

We have already seen in the last subsection during the discussion about the dependence of the loop corrections on \(s_H\) that the loop corrections usually become more important the heavier the new scalars are. The reason is that large scalar masses imply large values for the trilinear couplings \(M_1\) and \(M_2\) which enter the scalar loop corrections. One finds for instance that the one-loop corrections to the neutral CP-even Higgs with mass \(m_5\) scales as

$$\begin{aligned} \Delta m^2_5 \sim m^2_5 \left( 1 + C \frac{m_5^2}{v^2}\right) . \end{aligned}$$
(4.19)

Here, we have neglected all quartic couplings and expressed \(M_1\), \(M_2\) by \(m_5\). The coefficient C is a complicated function of \(s_H\) and of the ratio \(M_1/M_2\). The important point is that it is usually much larger than \(\frac{1}{16\pi ^2}\) and can become \(\mathcal O(1)\) for large \(s_H\) and/or large ratios.

We show the impact in Fig. 6a where we give the size of the loop corrections using a fully numerical calculation as a function of \(m_H\). The other parameters are set to

$$\begin{aligned} \lambda _2= & {} 0.1,\ \lambda _3 = 1,\ \lambda _4=-\,0.1,\ \\ \lambda _5= & {} 0.1,\ \alpha =20^{\circ },\ s_H=0.25. \end{aligned}$$

In all four different formulations of perturbativity constraints shown in Fig. 6a, we find that large \(m_H\) is generically more constrained than lower masses, i.e. the values of the CTs as well as the size of the loop-corrected \(\overline{\text {MS}}\) masses increase with increasing \(m_H\), up to a few specific values where cancellations among the different loop contributions are present. In the lower panel of Fig. 6a we present the size of the one- and two-loop corrections to the neutral CP-even \(\overline{\text {MS}}\) masses. We observe that, for values of \(m_H\) above 1.1 TeV (1.3 TeV), the two-loop corrections to \(h_2\) (\(h_3\)) are larger than the one-loop corrections. Thus, in this example, the strongest perturbativity constraints in the \(\overline{\text {MS}}\) scheme are actually not due to the loop corrections to the SM-like state but due to the new scalars. For the SM-like state we see a short range between 600 and 800 GeV where the two-loop corrections are larger than the one-loop corrections. However, this is obviously because the one-loop corrections are suppressed by an accidental cancellation. Therefore, we extend the \(\overline{\text {MS}}\) perturbativity limit by a threshold for the minimal size of the one-loop corrections, cf. Sect. 3.3: only if the one-loop corrections to the squared masses are larger than \((20 ~\text {GeV})^2\) and if the two-loop corrections are larger than the one-loop corrections, we consider this as breakdown of perturbation theory.

For the one-loop CTs in the OS scheme, shown in the upper two panels of Fig. 6a, we find again that the strongest limits on perturbativity stem from the constraint \(|\delta \lambda _x /\lambda _x|<1\). Looking at the unitarity constraints, shown in the third panel, we end up with a similar upper bound on \(m_H \le 800~\)GeV for this particular parameter point, more than \(\sim 200\) GeV lower than the limit from the \(\overline{\text {MS}}\) mass corrections. The constraint \(|\delta \lambda _x|<\pi \), in turn, leads to a limit on \(m_H\) of 1.2 TeV, which is in-between the one obtained from the loop corrections to \(m_{h_2}\) and \(m_{h_3}\). Using a weaker upper limit of \(2\pi \) or \(4\pi \) wouldn’t lead to any constraint in the tested parameter range. Qualitatively, however, we observe a clear correspondence of the perturbativity constraints formulated in the \(\overline{\text {MS}}\) scheme and the ones from the OS CTs. In Fig. 6b, we show the ranges of allowed \(m_H\) for this scenario, analogously to Fig. 4.

Fig. 7
figure 7

Perturbativity limits as function of \(\lambda _5\). The other parameters are chosen as: \(\lambda _2 =0.1\), \(\lambda _3 = 0.5\), \(\lambda _4=-\,0.1\), \(\alpha =20^{\circ }\), \(s_H=0.3\), \(m_H=750\) GeV

4.3.3 Dependence on large quartic couplings

So far, we have not considered the role of the quartic couplings on the perturbativity limits. Unlike in the THDM, the quartic couplings in the GM model are usually not taken very large, i.e. O(10). This is due to the tree-level unitarity limits in this model which already severely constrain combinations of couplings to be much smaller than \(4\pi \), see Eq. (3.2). For instance, if we assume only \(\lambda _3\) and \(\lambda _4\) to be non-negligible at tree-level, then large \(\lambda _{3}\) is only allowed if it cancels against a large \(\lambda _{4}\), confining both parameters to a narrow strip around \(\lambda _3 = -\frac{11 \lambda _4}{7}\pm \frac{2 \pi }{7}\) which is cut off at roughly \(\lambda _4 \simeq 2\).

On \(\lambda _5\), however, there are comparably weak tree-level unitarity constraints, i.e. \(|\lambda _5| \gg 1\) is easily possible without violating any unitarity limit. On the other hand, \(\lambda _5\) enters the one- and two-loop corrections, i.e. large effects are expected there. This is depicted in Fig. 7a where the loop effects are plotted as a function of \(\lambda _5\). The other parameter values are set to

$$\begin{aligned} \lambda _2= & {} 0.1,\ \lambda _3 = 0.5,\ \lambda _4=-\,0.1,\nonumber \\ \alpha= & {} 20^{\circ },\ s_H=0.3,\ m_H=750~\text {GeV}. \end{aligned}$$
(4.20)

For positive values of \(\lambda _5\), there is a fast increase in the size of the loop corrections. In particular the two-loop corrections to the SM-like Higgs grow very quickly and are as large as the tree-level mass for \(\lambda _5\simeq 2\). For the second-lightest Higgs, the two-loop corrections are larger than the one-loop corrections for \(\lambda _5 \simeq 2.25\). This roughly corresponds to the value at which \(\delta \lambda _3 \simeq \pi \), depicting again the correlation between the two-loop corrections in the \(\overline{\text {MS}}\) scheme and OS CTs. For positive values of \(\lambda _5\), similar constraints are also found when using the condition \(|\delta \lambda _x/\lambda _x|<1\) or the ‘loop improved’ unitarity constraints. For negative values of \(\lambda _5\), in turn, these two sets of conditions are much more restrictive: they would forbid values of \(\lambda _5\) below \(-2\), while the other two sets of conditions are fulfilled until \(\lambda _5 \simeq -\,4\).

Fig. 8
figure 8

Perturbativity limits as function of \(\lambda _2\). The other parameters are chosen as: \(\lambda _3 = 0.5\), \(\lambda _4=-\,0.1\), \(\lambda _5=0.1\), \(\alpha =20^{\circ }\), \(s_H=0.25\), \(m_H=600\) GeV

We now turn to the other quartic couplings. Even though those are constrained to smaller values than \(\lambda _5\) due to the tree-level unitarity bounds, the one-loop corrections to those quartics can turn out to be so large that we end up with stronger perturbativity constraints on \(\lambda _{1\cdots 4}\) than on \(\lambda _5\). An indication is already seen in Fig. 7 where it is actually the counter-terms to \(\lambda _3\) which become problematic much earlier than those to \(\lambda _5\). In Fig. 8a, we show the loop effects as a function of \(\lambda _2\). The other input values are set to

$$\begin{aligned} \lambda _3= & {} 0.5,\ \lambda _4=-\,0.1,\ \lambda _5=0.1,\\ \alpha= & {} 20^{\circ },\ s_H=0.25,\ m_H=600~\text {GeV}. \end{aligned}$$

We find almost continuously increasing CTs and loop corrected masses for increasing \(\lambda _2\). The two sets of constraints which have been the most restrictive ones on the other examples (\(|\delta \lambda _x / \lambda _x| < 1\) and the unitarity constraints) are already violated for \(\lambda _2\simeq 0.2\). In this case, this is quite similar to the upper limit which is obtained from the two-loop corrections to \(h_3\). On the other hand, when using the absolute size of the CTs or the two-loop corrections of the first two scalar masses as constraints, the limits are much weaker and range between \(\lambda _2 \simeq 1 \cdots 1.5\), comparable with the tree-level unitarity constraints. The constraints from the different perturbativity requirements are summarised in Fig. 8b.

Fig. 9
figure 9

Impact of the perturbativity constraints in the \((m_H,s_H)\) plane for different values of \(\lambda _2\) on regions which are allowed at tree-level (green areas). The other parameters are set to \(\lambda _3 = 0.5\), \(\lambda _4=-\,0.1\), \(\lambda _5=0.1\), \(\alpha =20^{\circ }\). The shaded area indicates the Higgs mass constraints (\(|(m_{h_i})^{2})_{\mathrm{Tree}} - (m_{h_i}^{2})_{ \mathrm 1L}| > |(m_{h_i}^{2})_{2\mathrm{L}} - (m_{h_i}^{2})_{ \mathrm 1L}|\)) [red: \(h_1\); blue: \(h_2\); brown: \(h_3\)]. The blue lines are the contours of constant values for \(\text {Max}(|\delta \lambda _x|) =c\pi \) [dotted: \(c=1\); dashed: \(c=2\); full \(c=4\)]. The black lines indicate \(\text {Max}(|\delta \lambda _x/\lambda _x|)>v\) [dotted: \(v=1\); dashed: \(v=2\); full: \(v=4\)]. The red lines show when the tree-level unitarity limits calculated with \(\tilde{\lambda }\)’s are violated when setting an upper limit on the scattering eigenvalues of \(4\pi \) (dotted), \(8\pi \) (dashed) or \(16\pi \) (full)

4.3.4 Impact on parameter regions

We have seen in the last subsections that the loop corrections in the GM model can become huge, indicating a breakdown of perturbation theory. Of course, it is difficult to define an absolute condition when this breakdown takes places. We have investigated four sets of conditions which ended up in different constraints on the parameters. Which conditions are applied depend on how conservative one wants to be. However, the important observation is that at some point all conditions point towards severe conflicts with the perturbative expansion: if the CT to a quartic coupling is O(100) and if the two-loop corrections are larger than the one-loop corrections by an order of magnitude, it is clear that one has entered the strongly coupled regime of the model.

We demonstrate at one example the impact of the different perturbativity constraints on the parameter space which seems to be valid at tree-level. We show in Fig. 9 an overlay of the allowed parameter space at tree-level and the different loop constraints in the \((m_H,s_H)\) plane for four different values of \(\lambda _2\). The other parameters have been set to

$$\begin{aligned} \lambda _3 = 0.5,\ \lambda _4=-\,0.1,\ \lambda _5=0.1,\ \alpha =20^{\circ }. \end{aligned}$$

The green shaded areas in Fig. 9 show the parameter space which is allowed at the tree level. The areas shaded in red (blue) [brown] indicate when the \(\overline{\text {MS}}\) two-loop correction to \(m_{h_1}\) (\(m_{h_2}\)) [\(m_{h_3}\)] becomes larger than the corresponding one-loop correction. The other contour lines show the OS perturbativity bounds as well as tree-level unitarity bounds calculated with the renormalised couplings \(\tilde{\lambda }\): the blue lines show the contours of constant Max\((|\delta \lambda _x|)\) of \(\pi \) (dotted), \(2 \pi \) (dashed) and \(4 \pi \) (full line). The black lines show the contours of constant ratios Max\((|\delta \lambda _x/\lambda _x|)\) of 1 (dotted), 2 (dashed) and 4 (full line). Finally, the unitarity bounds are represented by the red contours, showing constant scattering eigenvalues of \(4\pi \) (dotted), \(8 \pi \) (dashed) and \(16 \pi \) (full line).

The first observation is that in neither of the four subfigures, any of the parameter space features generalised scattering amplitudes with \(y_i < 4 \pi \). As in the previous examples, we will however always use the less restrictive bound of \(8\pi \), as defined in Eq. (3.58). In the left upper panel of Fig. 9, we present the case \(\lambda _2=0.5\). We observe that in this case, the loop corrections behave comparatively well – most of the parameter space which is allowed at tree level is still viable if the higher-order constraints are taken into account. The only exception is the ratio of CTs over the tree-level coupling which would exclude most of the valid parameter space if we were to apply the most restrictive bound of Max\((|\delta \lambda _x/\lambda _x|)<1\). We further observe that the other, less conservative OS bounds agree well with the \(\overline{\text {MS}}\) conditions.

For increasing values of \(\lambda _2\), the perturbativity constraints invade more and more the valid tree-level regions. Finally, for \(\lambda _2=1.25\), nearly the entire parameter space which is allowed at tree-level seems to demand a non-perturbative handling.

In all four examples in Fig. 9 we see that the strongest constraints always come from the ratios Max\((|\delta \lambda _x/\lambda _x|)<v\), especially if \(v=1\) is considered as the maximally-allowed ratio. However, also using \(v=2\) or 4 as bounds, these limits are always stronger than the ones from the absolute values of \(|\delta \lambda _x|<c \pi \), even if we apply \(c=1\) or 2 as condition. The exclusion regions when demanding that the tree-level unitarity constraints calculated with renormalised couplings should be fulfilled are comparable with those from the relative size of the CTs when imposing \(<8\pi \) for the maximal eigenvalue of the scattering matrix. If eigenvalues up to \(16\pi \) are accepted, the condition becomes more comparable to the ones on the absolute value of the CTs with \(c=1\).

Similarly, also the hierarchy between the one- and two-loop corrections to the scalar masses leads to quite severe constraints. It is in particular interesting to see that for different parameter points the corrections to different masses are more important. Because of this complementarity, the superposition of the constraints from all three masses cover a significant part of the parameter space. The constraints from the absolute size of the quartic couplings result, for this example, in the weakest limits.

Quite generically, however, we observe again clear correlations between the size of the CTs and the hierarchy between the one- and two-loop corrections in the \(\overline{\text {MS}}\) scheme.

Fig. 10
figure 10

Constraints on the parameter space in the \(m_5\)-\(s_H\) plane using the other parameters according to eq. (4.21). Points above the black (blue) thick line are excluded due to theoretical tree-level bounds (direct LHC searches [60]) according to Ref. [61]. The grey shaded area with the red contour lines shows the OS perturbativity constraints based on the relative size of the counter-terms. The red contours correspond to Max\((|\delta \lambda _x/\lambda _x|)=1,\,2,\,4,\,16\) and 64. The yellow shaded area corresponds to the unitarity constraints using renormalised parameters. The black dashed contours correspond to the associated scattering eigenvalues of \(4\,\pi \), \(8\,\pi \), \(16 \, \pi \) and \(64\,\pi \). As the green contour we show the \(\overline{\text {MS}}\) constraints from the size of the two-loop corrections vs. the one-loop corrections. Regions tainted green do not pass the constraint, i.e. Max\((|(m_{\phi _i}^{2})_{2\mathrm{L}} - (m_{\phi _i}^{2})_{1\mathrm{L}}|/|(m_{\phi _i}^{2})_{\mathrm{Tree}} - (m_{\phi _i}^{2})_{1\mathrm{L}}|)>1\), \(i=1,2,3\). Finally, the purple shaded area corresponds to the perturbativity constraints on the absolute size of the CTs. The purple dot-dashed contours correspond to Max\((|\delta \lambda _x|)=2\,\pi \) and \(4\,\pi \)

Finally, we want to show some constraints on a particular benchmark, the so-called ‘H5plane’, which has been promoted for the Georgi-Machacek model recently, see Refs. [61, 62]. This plane is characterised by only two free parameters \(m_5\) and \(s_H\). The other parameters are fixed according to

$$\begin{aligned} m_h&= 125~\mathrm{GeV},\quad \lambda _3 = -\,0.1,\quad \lambda _4 = 0.2, \quad \nonumber \\ \lambda _2&= 0.4 \frac{m_5}{1~\mathrm{TeV}}, \quad M_1 = \frac{\sqrt{2} \, s_H}{v} (m_5^2 + v^2),\quad M_2 = \frac{M_1}{6}. \end{aligned}$$
(4.21)

Obviously, the input parametrisation for this plane is different from the standard input choices I and II defined above. We have modified our code accordingly.

It has already been shown in Ref. [34] that there are, what appears to be, serious problems with perturbativity for large values of \(m_5\). This was shown using the \(\overline{\text {MS}}\) scheme, with the result that in large regions of the parameter space the 2-loop mass correction to the SM-like Higgs must are larger than the 1-loop correction. In Fig. 10, we now show the constraints arising from all other perturbativity conditions. As can be seen from there, the perturbativity constraints cut deeply into the parameter space of the H5plane. In particular, demanding in the OS scheme that \(|\delta \lambda _x/\lambda _x|<1\) (thick red contour line) only leaves valid parameter space below \(m_5 \lesssim 500~\)GeV. Only considering the perturbative unitarity cuts using renormalised couplings and demanding \(8\,\pi \) to be the upper limit (thick black dashed contour line), instead leaves points up to about 1.2 TeV. The loosest constraints come from the absolute size of the CTs. Only the parameter space below \(m_5 \simeq 2.1~\mathrm{TeV}~(2.5~\mathrm{TeV})\) violates this condition if the cut \(|\delta \lambda _x|<2\,\pi ~(4\,\pi )\) is applied. In addition to the \(\overline{\text {MS}}\) constraints already discussed in Ref. [34] for the lightest neutral scalar, we include the same condition for the two heavier neutral scalars. In fact, it turns out that the 2-loop corrections to the second mass eigenstate (corresponding to \(m_5\)) are the most dangerous. The parameter space in conflict with Max\((|(m_{\phi _i}^{2})_{2\mathrm{L}} - (m_{\phi _i}^{2})_{1\mathrm{L}}|/|(m_{\phi _i}^{2})_{\mathrm{Tree}} - (m_{\phi _i}^{2})_{1\mathrm{L}}|)>1\) is shown in green.

Also shown in the figure are the constraints extracted from Ref. [61]: only points below the black solid line are allowed from theoretical tree-level constraints. Note that for large \(s_H\), this line is in agreement with the loop-level unitarity bound – considerable deviations however appear below \(s_H \lesssim 0.4\). The parameter space above the blue line is excluded from the LHC direct searches for doubly-charged scalars [60]. In total, depending on how conservative a perturbativity cut is applied, the left-over parameter space of the H5plane is either small or even just a tiny strip.

4.4 Vacuum stability

So far, we have discussed one-loop perturbativity constraints on the parameter space of the GM model, a new kind of constraint which, to the best of our knowledge, has not been discussed in literature in the context of the GM model – or any other BSM model – before. However, also the impact of loop-corrections on well-known constraints which already exist at tree level is expected to be significant. Here we turn to the discussion of the vacuum stability constraints and show how the impact of the loop corrections can alter the conclusions drawn on that basis. A discussion of these effects for the THDM was done in Ref. [35], and we can find here quite similar features for the GM model.

4.4.1 Stabilising UFB directions

We start with unbounded-from-below directions which exist for the tree-level potential for many different field combinations. We have already discussed in Sect. 3.2.1 that these directions are very often expected to disappear once loop effects are included. While we have focused in Sect. 3.2.1 on the RGE-improved potential containing only quartic couplings – which is a valid approach in the limit of very large scales – we use here the one-loop effective potential. There are mainly two reasons for that: (i) the running of the quartic couplings is usually very fast, i.e. the scale at which the couplings or combinations of them change their sign is not for away from the input scale. Thus this scale is often not much higher than the scale of the dimensionful parameters in the potential. (ii) Even if the UFB conditions are satisfied at higher scales, this doesn’t mean that those points are necessarily stable: it can and will happen that the potential in the UFB directions is deformed to a local minimum which is deeper than the electroweak one. In order to check this, one needs to find all minima of the effective potential and compare their depths. We do this explicitly at one example in Fig. 11 where we compare the vacuum stability at tree level and at the one-loop level as a function of tree-level input value \(\lambda _3\) as well as \(m_H\). We used as further input

Fig. 11
figure 11

Comparison of the vacuum stability at tree-level and one-loop level. The shaded areas represent the stability of the potential at tree-level: unbounded from below direction exist (red), other minima deeper than the ew vacuum exist (orange), the ew vacuum is stable (green). The green hatching indicates a stable ew vacuum at one-loop whereas no hatching corresponds to a metastable vacuum at one-loop. Black lines show the value of constant \(\lambda _3/3+\lambda _4\) at tree level, and the blue ones of \(\text {min}\{\lambda _{3a} + 2 \lambda _{3b} + \lambda _{4a} + 4 \lambda _{4b} + 4 \lambda _{4c},\ \lambda _{3a}+\lambda _{4a},\ \lambda _{3b}+2\lambda _{4b},\ \lambda _{3b}+\lambda _{4b},\ \lambda _{3c}+\sqrt{2}\sqrt{(\lambda _{3a} +\lambda _{4a})(\lambda _{3b}+2\lambda _{4b})} + 2 \lambda _{4c}\}\) at one-loop. The black area is forbidden by the tree-level unitarity conditions. The grey shaded area indicates the perturbativity constraints. The other parameter values are \(\lambda _2 = 0.1\), \(\lambda _4=-\,0.1\), \(\lambda _5=0.1\), \(\alpha =15^{\circ }\), \(s_H=0.23\)

$$\begin{aligned} \lambda _2 = 0.1,\ \lambda _4=-0.1,\ \lambda _5=0.1,\ \alpha =15^{\circ },\ s_H=0.23. \end{aligned}$$

At tree level, the most constraining condition \(\lambda _3/3+\lambda _4<0\) for the presence of a UFB direction, cf. Eq. (3.43), therefore becomes

$$\begin{aligned} \lambda _3 < 0.3. \end{aligned}$$
(4.22)

This rules out a large fraction of parameter space in the shown plane. Moreover, also for \(\lambda _3 > 0.3\), other and deeper minima than the ew one are present at tree-level. As a consequence, the tree-level potential is only stable in a small region with \(\lambda _3 \simeq 1\) and \(m_H < 900\) GeV. At the one-loop level, there are finite corrections to the quartic couplings. Therefore, the most constraining conditions to not have a UFB direction in the combined tree-level and CT potential become

$$\begin{aligned} \text {min}&\{\lambda _{3a} + 2 \lambda _{3b} + \lambda _{4a} + 4 \lambda _{4b} + 4 \lambda _{4c},\quad \lambda _{3a}\nonumber \\&\quad +\lambda _{4a}, \quad \lambda _{3b}+2\lambda _{4b},\quad \lambda _{3b}+\lambda _{4b}, \nonumber \\&\quad \lambda _{3c}+\sqrt{2}\sqrt{(\lambda _{3a} +\lambda _{4a})(\lambda _{3b}+2\lambda _{4b})} + 2 \lambda _{4c}\} > 0, \end{aligned}$$
(4.23)

with \(\lambda _{Nx} = \lambda _N + \delta \lambda _{Nx}\). The contours for constant values of Eq. (4.23) are also shown in Fig. 11: negative values appear only for a rather small region with mainly \(\lambda _3 < 0\) and \(m_H<1\) TeV. Thus, the UFB directions in the other parts of the plane disappeared already just because of the CTs independently of the other one-loop corrections.Footnote 4 As expected, not the entire region where the loop potential doesn’t have a UFB direction also provides a stable ew vacuum. There is still a non-negligible region where the ew minimum is not the global minimum of the scalar potential. Nevertheless, the region with the ew minimum as the global minimum is significantly larger than at tree-level.

Fig. 12
figure 12

Comparison of the vacuum stability at tree-level and one-loop level. The shaded areas represent the stability of the potential at tree-level: unbounded from below direction exist (red), other minima deeper than the ew vacuum exist (orange), the ew vacuum is stable (green). Hatched regions feature a stable ew minimum at one-loop. Black lines show the value of constant \(\lambda _2 - \frac{1}{4} |\lambda _5| + \sqrt{2 \lambda _1 (2\lambda _4+\lambda _3)}\) using the tree-level input values, and the full blue ones of \(\text {Min}\{4 \lambda _{2a} - |\lambda _{5a}| + 4 \sqrt{2 \lambda _1 (2 \lambda _{4b} + \lambda _{3b})}, 2 \lambda _{2a} + \lambda _{2b} + 2 \sqrt{\lambda _{1} (\lambda _{3a} + 2 \lambda _{3b} + \lambda _{4a} + 4 (\lambda _{4b} + \lambda _{4c}))} - \frac{1}{2} \lambda _{5a} - \lambda _{5b} \}\). For the dashed blue lines, the term of Eq. (4.26) was added in addition to the tree-level + CT potential and the UFB condition of \(4 \lambda _{2a} +\lambda _{5a} + 4 \sqrt{2 \lambda _1 (2 \lambda _{4b} + \lambda _{3b})}>0\) has been re-derived. The grey shaded area indicates the perturbativity constraints. The other parameter values are \(\lambda _2 = 0.1\), \(\lambda _3=0.5\), \(\lambda _4=-0.1\), \(\alpha =20^{\circ }\), \(s_H=0.33\)

A similar mechanism also works for other UFB directions. In Fig. 12 we show the \(\lambda _5-m_H\) plane and consider the condition

$$\begin{aligned} \lambda _2 - \frac{1}{4} |\lambda _5| + \sqrt{2 \lambda _1 (2\lambda _4+\lambda _3)} > 0 \end{aligned}$$
(4.24)

which is violated in the depicted \((\lambda _5,m_H)\) plane at tree-level roughly for \(\lambda _5 < -1\) (up to some corrections stemming from the changes in \(\lambda _1\) in this plane). We show for comparison again the contours of the equivalent conditions for UFB directions in the tree-level plus CT potential, i.e.

$$\begin{aligned}&\text {min}\{4 \lambda _{2a} - |\lambda _{5a}| + 4 \sqrt{2 \lambda _1 (2 \lambda _{4b} + \lambda _{3b})}, \nonumber \\&\quad 2 \lambda _{2a} + \lambda _{2b} + 2 \sqrt{\lambda _{1} (\lambda _{3a} + 2 \lambda _{3b} + \lambda _{4a} + 4 (\lambda _{4b} + \lambda _{4c}))} \nonumber \\&\quad - \frac{1}{2} \lambda _{5a} - \lambda _{5b} \} < 0 , \end{aligned}$$
(4.25)

cf. Eqs. (3.39), (3.40) and (3.44). The difference between these lines is not as pronounced as in Fig. 4.23 – however, the regions with a stable minimum at the one-loop level are still significantly larger than at tree-level. This means that also in this example the loop corrections from the Coleman-Weinberg potential are very important to stabilise the UFB directions. In the entire plane the more stringent condition of Eq. (4.25) is \(4 \lambda _{2a} - |\lambda _{5a}| + 4 \sqrt{2 \lambda _1 (2 \lambda _{4b} + \lambda _{3b})}\), which corresponds to the direction \(\langle H^0\rangle =\langle \chi ^-\rangle = \langle \phi ^-\rangle =\langle \chi ^{--}\rangle =\langle \phi ^0\rangle =0, \langle H^+\rangle =x \langle \chi ^0\rangle \). In this direction, the terms proportional to \(\langle \chi ^0 \rangle ^4\) in the CW potential which don’t come together with a logarithm are

$$\begin{aligned} V_{\mathrm{CW}} \sim&\frac{1}{32 \pi ^2} \Big (64 (\lambda _2^2 + 8 \lambda _3^2 + 20 \lambda _3 \lambda _4 + 17 \lambda _4^2) \nonumber \\&\quad + 4 \lambda _5^2 + 4 (8 \lambda _2 (6 \lambda _1 + 2 \lambda _2 + 7 \lambda _3 + 11 \lambda _4) \nonumber \\&\quad + \, 2 (2 \lambda _1 + 4 \lambda _2 - \lambda _3 + 2 \lambda _4) \lambda _5 + 3 \lambda _5^2) x^2 \nonumber \\&\quad + 3 (64 \lambda _1^2 + 12 \lambda _2^2 + \lambda _5^2) x^4\Big ). \end{aligned}$$
(4.26)

In the next step, for achieving an insight into the question where the UFB directions go at one-loop, we add these terms to the tree-level + counter-term potential according to Eq. (3.51) and re-derive the UFB condition of Eq. (4.24) for the modified potential. The corresponding dashed blue contour lines are shown in Fig. 12. We see that the corresponding values are positive over the entire parameter range – which means that the tree-level UFB direction becomes bounded from below at the one-loop level.

4.4.2 Stabilising meta-stable regions

If no UFB direction exists, the ew vacuum could still be unstable due to the existence of other minima deeper than the one with correct EWSB. This case was to some extent already mentioned in the last subsection. Here we are going to investigate it in more depth. In Fig. 13, we show the difference \(\Delta V\) between the ew minimum and the panic vacuum, \(\Delta V = V_{\mathrm{EWSB}}-V_{\mathrm{panic}}\). The panic vacuum is defined as the minimum which is deeper than and closest in field space to the ew vacuum, i.e. the one to which the ew vacuum configuration would tunnel to eventually. \(\Delta V\) is a positive-semidefinite quantity: if the desired (i.e. ew) vacuum configuration corresponds to the global minimum of the potential, then such a panic vacuum does not exist (or is exactly as deep as the ew minimum) and we define \(\Delta V =0\). Positive values of \(\Delta V\) are, in turn, reached if there is a non-ew panic vacuum. The contour lines drawn in the figure correspond to lines of constant \(\sqrt{v_\phi ^2+4 (v_\eta ^2 +v_\chi ^2)}\). On the left-hand plane of Fig. 13, we present the result at the tree-level, and on the right-hand side with the full one-loop corrections included. The other parameters were set to

$$\begin{aligned} \lambda _2 = 0.1,\ \lambda _4=-\,0.1,\ \lambda _5=0.1,\ \alpha =20^{\circ },\ s_H=0.33. \end{aligned}$$

We see that at tree-level, only in one corner of the depicted parameter region, the ew minimum is also the global minimum of the potential. The depthFootnote 5 of this minimum is about \(-10^7 \text {GeV}^4\). For \(\lambda _3<0.8\) and/or \(m_H>600\) GeV, minima occur which are deeper than the ew one by several orders of magnitude. Note that the sum of all VEVs \(\sqrt{v_\phi ^2+4(v_\eta ^2 +v_\chi ^2)}\) for these other minima is in the TeV range, i.e. clearly larger than the ew scale. Nevertheless, those minima are not ‘too far’ away in field space, meaning that the tunnelling from the ew to the panic vacuum is very fast. We are going to quantify this statement below.

Fig. 13
figure 13

The scalar potential at the tree-level (left) and one-loop level (right): the colour indicates the difference between the panic minimum and the electroweak minimum. The contour lines are constant values for \(\sqrt{v_\phi ^2+4 (v_\eta ^2 +v_\chi ^2)}\). The other parameters are set to \(\lambda _2 = 0.1\), \(\lambda _4=-\,0.1\), \(\lambda _5=0.1\), \(\alpha =20^{\circ }\), \(s_H=0.33\)

Turning to the vacuum structure of the one-loop potential, shown on the right in Fig. 13, we observe that the ew vacuum corresponds to the global minimum of the potential for a much larger region of the depicted parameter space. Up to \(m_H \simeq 1200\) GeV, values of \(\lambda _3\) exist for which the ew potential is stable. Obviously, the one-loop corrections can be large and positive: they manage to overcompensate the potential difference at tree-level of 10 orders of magnitude. This large value is not surprising because one could estimate the corrections to be \(O(\frac{m_H^4}{16\pi ^2})\). Only for small values of \(\lambda _3\), the other minima remain deeper than the ew one.

We conclude this discussion with a brief analysis of the tunnelling rate. We have calculated the tunnelling from the desired to the panic vacuum using the code CosmoTransitions [63] in combination with Vevacious. The result is shown in Fig. 14, where we have zoomed into the region \(600~\text {GeV}<m_H<900~\text {GeV}\) of the right-hand side of Fig. 13. We find that there is only a narrow band where the life-time of the ew minimum is comparable to the age of universe when calculating the tunnelling rate at zero temperature. These points could in principle be considered as ‘long-lived’, i.e. viable. However, once the thermal corrections are included as explained in Ref. [64], they also usually become short-lived. Therefore, from now on, we are going to consider all points which feature a deeper panic vacuum as not viable. This is similar to the THDM where it was also found that the tunnelling to deeper minima is always very fast on cosmological time scales [35, 65].

Fig. 14
figure 14

Zoom into the parameter plane of the right-hand side of Fig. 13 and presenting the calculated life-time of the electroweak vacuum normalised to the life-time of the universe at zero temperature

Fig. 15
figure 15

Comparison of the vacuum stability at tree level and the one-loop level. The shaded areas represent the stability of the potential at tree level: unstable (orange) or stable (green). The green hatched area is stable at the one-loop level. The green, purple and red contour lines show the depth of the loop-corrected ew minimum: the values are \(\text {sign}(V)\log _{10}|V/\text {GeV}^4|\). The grey shaded area indicates the perturbativity constraints where we have used the \(\overline{\text {MS}}\) condition. The other parameter values are \(\lambda _2 = 1\), \(\lambda _3=0.9\), \(\lambda _4=-\,0.2\), \(\lambda _5=0.2\), \(\alpha =15^{\circ }\)

4.4.3 De-stabilising stable regions

So far, we have discussed the situation that the stability of the potential is increased once loop corrections are included. We found this situation to appear far more often than the opposite effect. However, it can also happen in some cases that the stability of the ew potential is decreased via the loop corrections. The reason is that the loop corrections are often positive. Thus, if they are larger than the depth of the ew minimum at tree-level, they can push it to positive values. Consequently, the minimum with all VEVs set to zero is deeper at the loop-level. In other words, it can happen that the ew symmetry is restored by the loop corrections. We show an example of this situation in Fig. 15. Here, the stability at tree level and the one-loop level is shown in the \((m_H,s_H)\) plane for

$$\begin{aligned} \lambda _2 = 1,\ \lambda _3=0.9,\ \lambda _4=-\,0.2,\ \lambda _5=0.2, \alpha =15^{\circ }. \end{aligned}$$

The tree-level vacuum is stable in a strip around the line \((s_H=0.7,m_H=250~\text {GeV})\) to \((s_H=0.35,m_H=450~\text {GeV})\). However, once the loop corrections are included, the depth of the ew potential in a large fraction of this area becomes positive. This is indicated by the green, purple and red contour lines which show the logarithmic values for negative, zero, and positive potential depth of the ew minimum. For \(m_H=250\) GeV, all points which are stable at the tree level are actually unstable at the loop level. For \(m_H=350\) GeV, still more than half of the \(s_H\) range which is allowed at tree level is forbidden after the inclusion of loop corrections. Only for \(m_H\ge 450\) GeV, all the points which are stable at tree level are also stable at the one-loop level. In general, we find that the restoration of the ew symmetry at the loop-level appears mainly in parameter regions with small \(m_H\) and not too small \(s_H\). On the other side, we find that the entire area which is ‘below’ the green band, i.e. where the tree-level ew vacuum is unstable, is stabilised at the loop level. Hence, the overall picture that loop corrections increase the parameter space in which the vacuum is stable, still holds.

4.5 The global picture

We have discussed so far the perturbativity constraints as well as the loop-improved vacuum stability constraints at selected examples. As the final step, we want to obtain an impression of the ‘global picture’, i.e. the impact of these constraints in a wide fraction of the full parameter space of the GM model. For this purpose, we performed random scans for both input choices according to Eqs. (4.4) and (4.8), in the following parameter ranges:

  1. 1.

    Input I:

    $$\begin{aligned}&a \lambda _2,\lambda _3,\lambda _4 \in [-1,1], \qquad \lambda _5 \in [-2.5,2.5], \\&\quad m_H \in [200~\text {GeV}, 2000~\text {GeV}],\qquad \\&\qquad \alpha \in [3^{\circ },50^{\circ }],\qquad s_H \in [0.05,0.95]. \end{aligned}$$
  2. 2.

    Input II:

    $$\begin{aligned}&\lambda _2,\lambda _3,\lambda _4 \in [-1,1], \qquad \lambda _5 \in [-2.5,2.5],\\&\quad m_5 \in [200~\text {GeV}, 2000~\text {GeV}], \qquad \\&\qquad r_{12} \in [10^{-2},10^2], \qquad s_H \in [0.05,0.95]. \end{aligned}$$

The values of the \(\lambda \)’s were chosen to ensure that a large fraction of them are in agreement with tree-level unitarity conditions, i.e. to make the scan more efficient. The scans were carried out until for each input choice, 250,000 points were collected which (i) have a tachyon-free particle spectrum, (ii) which pass the tree-level unitarity constraints and (iii) which are in agreement with the S parameter. Afterwards, these points were confronted with different cuts. First, the experimental constraints on the Higgs couplings were applied using HiggsBounds. In the second step, the consequences of the tree-level constraints (UFB, other minima) were compared to the loop-improved vacuum stability constraints and to the perturbativity constraints in addition. In the case of the perturbativity cuts, we made two choices:

  1. 1.

    weak condition (“Weak P.”): the condition that corrections to the scalar masses calculated in the \(\overline{\text {MS}}\) scheme must be smaller at two-loop than at one-loop is applied. In addition, an upper limit for the absolute value of the CTs for the quartic couplings of \(2\pi \) has been set.

  2. 2.

    strong condition (“Strong P.”): the condition that the generalised tree-level unitarity conditions must be fulfilled for the renormalised quartic couplings is applied.

Table 1 Summary of the results of a random scan using Input I and II. A sample with 250,000 points each has been generated which pass the tree-level unitarity constraints. The numbers give the points surviving the different cuts. As weak perturbativity cut we applied the \(\overline{\text {MS}}\) condition based on the hierarchy of the one- and two-loop corrections to the masses of the CP-even neutral scalars. The strong cut uses the generalised unitarity constraints with renormalised couplings. Under ‘Agreement’, we list the number of points which pass both the tree- and loop-level constraints. We show in brackets how many of the points passing the tree-level constraints also pass the loop constraints and vice versa

A summary of the number of points surviving the different cuts is given in Table 1. Passing the HiggsBounds constraints (column ‘HB’) is taken as a prerequisite for the subsequent columns. The numbers in the columns “Tree-Level” as well as in “Loop-Level” show a cutflow in itself, i.e. points which pass “Tree-Level Stability” also pass the HB as well as Tree-Level UFB constraints, and points which pass “Loop-Level Weak/Strong P.” also pass the HB and Loop-Level Stability constraints. Points which pass the loop-level constraints do not necessarily also pass the tree-level constraints and vice versa, in accordance with the previous observations that tree-level constraints are often changed significantly at the loop level. In the last column “Agreement”, we compare the numbers from the tree- and loop-level constraints: first, we show the number of points which pass both tree and loop constraints. In brackets, we show which percentage of points which pass tree-level constraints also pass the one-loop constraints and vice versa. For instance, 54% of those points which pass the tree-level cuts are still viable after imposing the loop-level “Stability” and “Weak P.” conditions. In turn, out of all points which pass the latter, only 35% would have also been considered viable at tree-level. A cross-check of the viability of the results is that the fraction of points which have passed the “Strong P.” cut and are also in agreement with the tree-level conditions is always larger than the analogue with the “Weak P.” cuts. This means that the stronger constraints are more stringent in filtering out points at which the loop corrections have a large impact.

The overall result is that the theoretical constraints at tree level have a misidentification rate of about 50%. That means that roughly half of the points which seem to be viable when applying tree-level conditions are in conflict with the loop conditions. On the other side, also a large fraction of points which look fine at the loop level would have been discarded if only tree-level conditions had been used. If the strong perturbativity cut is applied, only one third (Input I) respectively one fourth (Input II) of the points which appear to be valid at tree level also pass the loop constraints. Since these are quite large effects, it is important to understand them in some more detail. In the following subsections, we are therefore going discuss the effects and try to pin down where the differences between tree- and loop-level are most pronounced.

Fig. 16
figure 16

Results of the parameter scan. First row: the minimal value of \(\alpha \) found in the \((m_H,s_H)\)-plane for Input I (left) and the \((m_5,s_H)\)-plane for Input II (right). In the second row, we show the same figures after applying the HiggsBounds cuts, and in the third row after both HiggsBounds and weak perturbativity cuts

4.5.1 Perturbativity

The main difference between the two input modes I and II is the treatment of \(\alpha \): while it is an input parameter for Input I, it is dynamically calculated for Input II from the other parameters by diagonalisation the scalar mass matrix. One finds approximately the following dependence of \(\alpha \) on the other parameters

$$\begin{aligned} \sin 2\alpha \sim s_H\sqrt{1-s_H^2} \, \frac{m_3^2}{m_h^2-m_H^2} . \end{aligned}$$
(4.27)

Therefore, usually a correlation between \(s_H\) and \(\alpha \) is visible and \(\alpha \) is naturally large for large \(s_H\). On the other side, for Input I one can, in principle, choose \(\alpha \) arbitrarily small independently of \(s_H\). The only constraint is that this could lead to tachyons in the scalar sector. To get a picture from the possible values of \(\alpha \) for the two input modes, we present in the first row of Fig. 16 the minimal value of \(\alpha \) which we found in our random scans in the \((m_H,s_H)\) respectively \((m_5,s_H)\) planes. As expected, for Input II, the minimal values of \(\alpha \) tend to be larger for large \(s_H\) than for Input I. Of course, since both inputs are related by just a re-parametrisation, they should be equivalent at the end. However, in order to keep \(\alpha \) small for large \(s_H\), some tuning in the other parameters is required which is not easily achieved in a random scan. Thus, from this point of view, Input I looks much more promising to find points which pass the constraints on the Higgs couplings. This is also confirmed in the second row of Fig. 16 where we show the minimal value of \(\alpha \) in the same planes after the Higgs constraints have been applied. There are parameter regions like \(s_H>0.3, m_H>1200\) GeV which are hardly accessible via Input II when choosing the input randomly, but which are populated for Input I. However, we had already observed in Sect. 4.3.1 that the perturbativity constrains are in particular strong for cases in which some tension between the values of \(s_H\) and \(\alpha \) is present. This is also nicely confirmed in our random scan. If we apply in addition to the Higgs coupling constraints the perturbativity constraints, as done in the last row in Fig. 16, we can observe that the two input modes show quite similar results even in the random scans. All regions where only for Input I, points had survived the HiggsBounds cuts, are in conflict with the (weak) perturbativity constraints. Thus, the accessible regions as well as the minimal value of \(\alpha \) in these regions look very similar once the HiggsBounds constraints are combined with the perturbativity constraints.

Fig. 17
figure 17

Fraction of points passing the weak (first row) or strong (second row) perturbativity cut. Only points which have initially passed the HiggsBounds checks are taken into account. The left column is for Input I, the right one for Input II

We can confirm this picture also from another point of view. In Fig. 17, we show the fraction of points which pass the weak or strong perturbativity constraint in the \((m_{H,5},s_H)\) planes for Input I respectively Input II. One finds that these cuts are particularly strong for Input I in the case of (i) large \(s_H\) and \(m_H\) or (ii) very small \(s_H\). In these parameter regions, up to 100% of the points require a non-perturbative treatment. Is is remarkable that the ‘strong’ cut as we called it since it affects on average more points, can be even less restrictive than the ‘weak’ cut in some regions of parameter space. This is especially the case for Input I and small \(m_H\) in combination with large \(s_H\). However, one should note that the number of points per bin in these regions is also smaller than in other regions. For Input II, we find in contrast only a soft dependence on \(s_H\) as long as it is below 0.5 (strong cut) or 0.3 (weak cut). On the other side, a strong tendency is visible that the perturbativity cuts are more important for increasing \(m_5\).

In general, it is worth noticing that even in those parameter regions where the perturbativity cuts are the least constraining, they still affect at least 15% of the shuffled points.

4.5.2 Vacuum stability constraints

Fig. 18
figure 18

First row: fraction of those points which are stable at one-loop and also pass the tree-level constraints. Second row: fraction of those points which are stable at tree-level and pass also the loop constraints. The left columns shows the results for Input I, the right column for Input II

We now turn to the vacuum stability constraints and discuss the main differences between the tree-level and loop-improved constraints. A comparison between the tree-level and one-loop constraints in the \((m_H,s_H)\) and \((m_5,s_H)\) plane for Input I and II, respectively, is shown in Fig. 18. In this figure, we consider all points which are either stable at the tree- or one-loop-level and show the fraction of points which are also stable at loop- or tree-level, respectively. Thus, these plots present

$$\begin{aligned} R= & {} \frac{\#_{|\text {stable at loop}|\text {stable at tree}}}{\#_{|\text {stable at loop}}} \quad \text {or}\quad \nonumber \\ R= & {} \frac{\#_{|\text {stable at tree}|\text {stable at loop}}}{\#_{|\text {stable at tree}}} \end{aligned}$$
(4.28)

For Input I, we find that at most 60% of the points per bin which are stable at the one-loop level would also have passed the tree-level constraints. In other words, the misidentification rate that stable points are considered as unstable from tree-level considerations (‘false negative’) is at least as large as 40%. For Input II, the ‘false negative’ misidentification rate can go down to 20%. However, this is only the case for a small band of very small \(m_5\) of 200–300 GeV. Otherwise, this misidentificatio rate is often 70% and more. For Input I, parameter regions exist where nearly all points with a stable vacuum at the loop-level would be ruled out by the tree-level constraints. This is in particular the case for very small \(s_H\) or for somewhat large \(s_H\) (\(\sim 0.4\)) together with \(m_H\) above 500 GeV. For Input II, there is also one spot where the misidentification rate is close to 100% at \(s_H \sim 0.35\), \(m_5\sim \) 800–1000 GeV.

Fig. 19
figure 19

Fraction of those points out of the parameter scan which are stable at one-loop and also pass the tree-level constraints. The results are shown in the \((\lambda _2,\lambda _4)\) (upper row) and the \((\lambda _3,\lambda _5)\) plane (lower row). The left column uses the points with the input choice I, the right column uses Input II

Fig. 20
figure 20

Maximally allowed value for \(s_H\) using different sets of constraints. Upper left figure: maximal value of \(s_H\) in the \((m_5,r_{12})\) plane for Input II after the HiggsBounds cut and all loop constraints. For the perturbativity cuts, we have used the weak conditions. Upper right plane: the relative difference to the maximal value when using only tree-level constraints and not checking for perturbativity (depicted is \(100 \frac{s_H^\mathrm{Loop}-s_H^\mathrm{Tree}}{s_H^\mathrm{Tree}}\)). In the second row, the difference is shown if tree-level vacuum stability constraints are combined with the weak perturbativity cuts (left), or the one-loop vacuum stability is checked but the perturbativity constraints are not applied (right). The purple regions are not accessible if the tree-level cuts are applied. In the black bins, no point was found in our random scan after applying the one-loop perturbativity constraints on points with a stable tree-level vacuum

In contrast, the large majority of points which pass the tree-level constraints is also stable at the loop-level. The main exception is for Input I in the limit of large \(s_H\) and small \(m_H\) where the loop corrections restore the ew symmetry as explained in Sect. 4.4.3. In these regions, the misidentification rate that points are assumed to be stable based on tree-level consideration is up to 30%. In all other parameter regions, it is significantly smaller and at most a few percent. Since the effect of symmetry restoration comes with large \(s_H\) together with rather small values of \(\alpha \), it is much less pronounced for Input II. Consequently, the ‘false positive’ misidentification rate is, with the exception of one bin, always below 10% in the \((m_5,s_H)\) plane. Since for the vacuum stability constraints, and in particular for the check against UFB directions, the quartic couplings are crucial, we show in Fig. 19 the ‘false negative’ misidentification rate also in the \((\lambda _2,\lambda _4)\) and \((\lambda _3,\lambda _5)\) planes for Input I and II. The overall situation for both input modes is quite comparable. One finds in particular for negative \(\lambda _2\) and/or \(\lambda _4\) that a large majority is mis-categorised. Only for \(\lambda _2 \rightarrow 1\), \(\lambda _4 \rightarrow 1\), more than 50% of the points which are stable at the loop-level are also considered ‘stable’ from tree-level checks. The reason is that, although the loop corrections are in principle more important for larger quartic couplings, large \(\lambda _{2}\) and \(\lambda _4\) make it less likely that a given parameter point fails one of the tree-level checks for UFB directions.

In the lower row of Fig. 19, which gives the results for the \((\lambda _3,\lambda _5)\) plane, the picture changes. Here we find that most of the points for which the tree-level checks also return ‘stable’ if the point is deemed stable at one-loop reside around small absolute values of \(\lambda _3\) and \(\lambda _5\). In this case, the misidentification goes down below 60% for Input I and even below 30% for Input II. On the other side, if either \(|\lambda _3|\) or \(|\lambda _5|\) are large, less than 20% of the points are categorised correctly. The reason is that for large couplings, the loop corrections are even more important. In contrast to \(\lambda _2\) or \(\lambda _4\), the quartic coupling \(\lambda _5\) enters the UFB checks as \(|\lambda _5|\), i.e. UFB directions are as likely for large positive \(\lambda _5\) as they are for large negative values. In the case of \(\lambda _3\), points with large negative tree-level values for that coupling are likely to have a UFB direction at tree-level which becomes bounded from below after including the loop corrections, analogous to \(\lambda _2\) and \(\lambda _4\); recall also the example of Fig. 11. At large positive \(\lambda _3\), in turn, a point which is considered unstable at tree level because of a deeper non-ew minimum of the scalar potential can become stable at one-loop after the inclusion of the large loop corrections as we have seen at the example of Fig. 13.

Finally, we want to remark again what is particularly noticeable in Fig. 13: one clearly sees that, if a point passes the loop-level constraints, there is in general a higher chance that it also passes all tree-level tests if Input II is used rather than Input I. This is a result of the smaller amount of parameter tuning which is needed in average (i.e. for a randomly chosen parameter point) if \(\alpha \) is calculated rather than used as an input.

4.5.3 Maximal \(s_H\)

We want to close the discussion of our random scans by checking the maximal allowed value of \(s_H\) which is possible in the different parameter regions when applying the various tree- and loop-level constraints defined above. Here, we concentrate on Input II because of the larger number of valid points when using both tree- and loop-level checks. I.e., we consider a comparison between results of the lowest order and higher orders of perturbation theory as more robust for this input choice. In the first row of Fig. 20, we show the maximal value of \(s_H\) which we find after applying the weak perturbativity constraints and checking the stability of the one-loop effective potential. We see that very large values are only possible for \(r_{12}<0\) (i.e. opposite signs for \(M_1\) and \(M_2\)) and small \(m_5\) below 500 GeV. For increasing \(r_{12}\) or \(m_5\), the maximal possible value of \(s_H\) quickly goes down to 0.3 and less. We can now compare these values with \(s_H^\mathrm{max}\) which we would find in the same plane if we apply the tree-level vacuum conditions and if we don’t check for perturbativity. The results are depicted as well in the first row of Fig. 20, on the right-hand side. First of all, a band with large \(r_{12}\), shown in purple, wouldn’t have been accessible at all under these conditions. Close to this region, i.e. still for \(r_{12}\gtrsim 1\), the relative changes in \(s_H^\mathrm{max}\) can be 150% and more. Of course, in total numbers this means for this parameter region rather moderate shifts of 0.01–0.10 in the maximal possible value of \(s_H\). In the other parameter regions, the relative differences in \(s_H^\mathrm{max}\) are not too pronounced; they lie between \(-\,25\) and \(+100\)%. Nevertheless, these are effects which are not negligible even if one has averaged over thousands of points, i.e. the general behaviour of the model is affected. Of course, for a single point which should be for instance used for collider studies, it is even more important to apply robust checks to test whether the point is allowed or not.

In the second row of Fig. 20, we show the difference in \(s_H^\mathrm{max}\) when using the tree-level checks and either the weak perturbativity constraints (lower left plane) or the loop-corrected vacuum stability checks (lower right plane). If one would use the tree-level vacuum stability conditions and apply the weak perturbativity conditions in addition, the sample of points considered as ‘valid’ will shrink. Therefore, also \(s_H^\mathrm{max}\) becomes smaller. The differences can be up to -50% for large \(m_5\). On the other side, if only the loop-improved vacuum stability checks were used without perturbativity checks (right-hand plot), the sample of ‘valid’ points would increase significantly, resulting in larger positive shifts of \(s_H^\mathrm{max}\) up to 500% for large \(r_{12}\). Moreover, a rather large region in the parameter space would seem accessible in this case: the region at large \(r_{12}\) and large \(m_5\) which is purple in the lower right plane but white in the upper right figure appears to be allowed if the perturbativity constraints are disregarded.

This shows that the two proposals to improve the theoretical checks for parameter points in the GM model are complementary, and we stress that a check of the stability at the loop level is only meaningful if the perturbative series is trustworthy.

5 Summary and conclusions

We have investigated in this work the perturbative behaviour of the Georgi-Machacek model, focussing on the scalar sector. An on-shell renormalisation of the scalar masses and mixing angles is formally possible once the custodial symmetry is given up at the loop level. However, it has been shown that in large regions of parameter space, very large loop corrections can appear, pointing towards the breakdown of perturbation theory. Therefore, these regions are most likely strongly coupled although naive limits for the quartic couplings would not indicate this. For a definite answer to the question whether the perturbative expansion works or not, it would be necessary to check for the decrease in the scale dependence of different physical processes – which is so far hardly possible in most BSM models since one would need to calculate the two-loop results for several decay or scattering processes.

We have therefore proposed a set of easily-accessible checks which should help to get an impression of how well the perturbative expansion works. These checks either include conditions on the size of the counter-terms to the quartic couplings which should be fulfilled in an on-shell scheme, or use the hierarchy between the one- and two-loop corrections to the scalar masses in the \(\overline{\text {MS}}\) scheme. The methodology developed in this work therefore solely relies on calculations which are, as of late, possible in an automated way for many BSM models via the Mathematica package SARAH. The perturbativity conditions proposed here are thus a lot more sophisticated than the simple tree-level checks and at the same time already easily calculable with the help of modern particle physics computer tools.

We have shown at the example of the Georgi-Machacek model that those comparably easily-accessible conditions are, in many parameter regions, sufficient to be certain that perturbation theory is not applicable: we have found examples in which the counter-terms to quartic couplings are larger than 100 or in which the two-loop mass corrections are several orders of magnitude larger than the one-loop corrections. There are, however, also regions where the results are not that clear. For instance if the two-loop corrections are comparable to the one-loop corrections or if counter-terms are of order one. Here, it is still a matter of taste or conservatism if one assumes that Born-level or one-loop results for masses or processes are still reliable. Although in these cases, we cannot provide a definite yes/no answer to the question of perturbativity, we think that this work has pointed out the potential problems with perturbativity arising in particular in non-supersymmetric models in the presence of sizeable quartic couplings. We further hope that the presented ansätze are a step forward towards more reliable checks of this issue in the future.

In the specific case of the Georgi-Machacek model, we could identify several parameter regions where the loop corrections tend to be large. In particular, we found that a check for perturbativity is inevitable if (i) the new scalar masses are large, i.e. 1 TeV or more, (ii) if there is a mismatch between the actual values of \(s_H\) and the scalar mixing angle \(\alpha \) either due to the choice of the input parameters or due to accidental cancellations, or (iii) if quartic couplings become large (although still well below the limits from the tree-level unitarity conditions).

The second main outcome of this work is that we have revised the checks for vacuum stability, which have so far only been done at tree-level, by including loop corrections. We have shown that those tree-level checks are usually not reliable in the Georgi-Machacek model. In particular, in many cases, a vacuum configuration which appears to feature an unstable electroweak vacuum at the tree level turns out to be stable once the one-loop corrections are included. This is the case as (i) a change in hierarchy between different minima of the scalar potential can easily occur at the one-loop level, and (ii) because field directions which appear to be unbounded from below at the tree level turn out to be bounded after the inclusion of the loop corrections. The opposite case that the vacuum is destabilised at the loop level is much less likely. Depending on the parameter regions, we found misidentification rates of the tree-level checks compared to the more reliable loop-level checks of up to 100%. In the entire parameter region which we have checked, the misidentification rate was always above 15%. Therefore, we conclude that, for a reliable prediction if a point has a stable or unstable electroweak vacuum, tree-level conditions are not sufficient. On the other side, one needs to be sure that perturbation theory is working if loop corrected checks for the vacuum stability shall be applied. This shows the strong connection between the two topics discussed in this work.